| janders | dtantsur|afk regarding backport of https://review.opendev.org/#/c/747815/ - will do, on it. | 00:09 |
|---|---|---|
| patchbot | patch 747815 - bifrost - Explicitly set permissions on /httpboot contents (MERGED) - 8 patch sets | 00:09 |
| openstackgerrit | Jacob Anders proposed openstack/bifrost stable/ussuri: Explicitly set permissions on /httpboot contents https://review.opendev.org/748054 | 00:18 |
| *** uzumaki has quit IRC | 00:23 | |
| *** gyee has quit IRC | 00:26 | |
| openstackgerrit | Jacob Anders proposed openstack/bifrost stable/ussuri: Explicitly set permissions on /httpboot contents https://review.opendev.org/748054 | 00:27 |
| *** Lucas_Gray has quit IRC | 00:33 | |
| openstackgerrit | Jacob Anders proposed openstack/bifrost bugfix/8.3: Explicitly set permissions on /httpboot contents https://review.opendev.org/748058 | 00:50 |
| *** rloo has quit IRC | 00:59 | |
| janders | dtantsur|afk I hope ^^^^ make sense - let's chat when you're online | 01:00 |
| *** Goneri has quit IRC | 01:01 | |
| *** hjensas has quit IRC | 01:05 | |
| *** JamesBenson has joined #openstack-ironic | 01:38 | |
| *** JamesBenson has quit IRC | 01:42 | |
| *** Qianbiao has joined #openstack-ironic | 02:13 | |
| *** chenhaw has quit IRC | 02:51 | |
| *** chenhaw has joined #openstack-ironic | 02:52 | |
| *** rh-jelabarre has quit IRC | 03:08 | |
| *** stevebaker has quit IRC | 03:17 | |
| *** rcernin_ has joined #openstack-ironic | 03:25 | |
| *** rcernin has quit IRC | 03:25 | |
| *** JamesBenson has joined #openstack-ironic | 03:39 | |
| *** JamesBenson has quit IRC | 03:44 | |
| *** stevebaker has joined #openstack-ironic | 03:46 | |
| *** chenhaw has quit IRC | 04:00 | |
| *** chenhaw has joined #openstack-ironic | 04:00 | |
| *** ijw has quit IRC | 04:05 | |
| *** mkrai has joined #openstack-ironic | 04:05 | |
| *** tzumainn has quit IRC | 04:06 | |
| *** dmellado has quit IRC | 04:31 | |
| *** ijw has joined #openstack-ironic | 04:36 | |
| *** ijw has quit IRC | 04:41 | |
| *** Qianbiao has quit IRC | 04:45 | |
| *** Qianbiao has joined #openstack-ironic | 04:47 | |
| *** stevebaker has quit IRC | 04:52 | |
| openstackgerrit | wu.chunyang proposed openstack/ironic master: Remove the unused coding style modules https://review.opendev.org/747578 | 04:56 |
| *** ijw has joined #openstack-ironic | 05:07 | |
| *** ijw has quit IRC | 05:12 | |
| *** Qianbiao has quit IRC | 05:21 | |
| openstackgerrit | vinay kumar muddu proposed openstack/ironic master: Adds ilo-uefi-https boot interface to ilo5 https://review.opendev.org/745501 | 05:25 |
| *** ijw has joined #openstack-ironic | 05:39 | |
| *** JamesBenson has joined #openstack-ironic | 05:45 | |
| *** belmoreira has joined #openstack-ironic | 05:46 | |
| *** belmoreira has quit IRC | 05:47 | |
| *** ijw has quit IRC | 05:49 | |
| *** JamesBenson has quit IRC | 05:50 | |
| *** ijw has joined #openstack-ironic | 06:11 | |
| *** ijw has quit IRC | 06:16 | |
| *** Qianbiao has joined #openstack-ironic | 06:26 | |
| *** jtomasek has joined #openstack-ironic | 06:35 | |
| *** ijw has joined #openstack-ironic | 06:43 | |
| michchap | Anyone know what might cause failures like this? https://36e67bad41869ee0bc5d-99456abad161e1f28f9c3ec49cb7f2f5.ssl.cf1.rackcdn.com/745318/3/gate/ironic-tempest-ipa-wholedisk-direct-tinyipa-networking-ansible/64e3790/controller/logs/ironic-bm-logs/node-0_console_2020-08-25-18:51:55_log.txt | 06:44 |
| michchap | RAMDISK: Couldn't find valid RAM disk image starting at 0. | 06:44 |
| michchap | VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6 | 06:44 |
| michchap | Started seeing this in the networkin ansible gate job some time in the last week or two | 06:44 |
| *** mkrai_ has joined #openstack-ironic | 06:45 | |
| *** ijw has quit IRC | 06:48 | |
| *** mkrai has quit IRC | 06:48 | |
| arne_wiebalck | Good morning, ironic! | 07:05 |
| *** belmoreira has joined #openstack-ironic | 07:12 | |
| *** ijw has joined #openstack-ironic | 07:14 | |
| *** ijw has quit IRC | 07:19 | |
| rpittau | good morning ironic! o/ | 07:34 |
| *** tosky has joined #openstack-ironic | 07:40 | |
| *** ijw has joined #openstack-ironic | 07:45 | |
| *** ijw has quit IRC | 07:50 | |
| *** hjensas has joined #openstack-ironic | 07:55 | |
| iurygregory | good morning arne_wiebalck rpittau and Ironic o/ | 07:56 |
| iurygregory | TheJulia, ack I will look at it | 07:56 |
| rpittau | hey iurygregory :) | 07:58 |
| arne_wiebalck | Hey iurygregory o/ | 08:04 |
| Qianbiao | hello ironic. | 08:07 |
| Qianbiao | Hi, i have install a devstack without ironic plugin. | 08:07 |
| Qianbiao | And if i want to enable ironic part, how to do it. | 08:08 |
| Qianbiao | Or i just need to reinstall whole devstack with a new local.conf | 08:08 |
| *** mgoddard has quit IRC | 08:10 | |
| *** mgoddard has joined #openstack-ironic | 08:11 | |
| rpittau | Qianbiao: ideally you would install directly with ironic enabled | 08:11 |
| rpittau | Qianbiao: https://docs.openstack.org/ironic/latest/contributor/dev-quickstart.html#deploying-ironic-with-devstack | 08:11 |
| Qianbiao | rpittau I found this document too late. | 08:12 |
| Qianbiao | I have install a devstack without ironic. | 08:12 |
| Qianbiao | and now, i want to add ironic plugin. should i reinstall whole devstack, or there has a method that could only install addition ironic part. | 08:13 |
| ajya | Qianbiao: restack should be enough | 08:13 |
| Qianbiao | <ajya> ok | 08:13 |
| ajya | there is ./unstack.sh, then ./clean.sh and then ./stack.sh | 08:14 |
| Qianbiao | ok, I hoped there has a way to void unstack and clean :) | 08:14 |
| rpittau | Qianbiao: as ajya wrote, with the config for ironic before running stack.sh again :) | 08:14 |
| rpittau | Qianbiao: you could also try without clean.sh if you don't touch services that you already deployed | 08:16 |
| *** lucasagomes has joined #openstack-ironic | 08:16 | |
| Qianbiao | rpittau ajya sure, thanks. | 08:16 |
| *** ijw has joined #openstack-ironic | 08:17 | |
| *** dougsz has joined #openstack-ironic | 08:19 | |
| *** Lucas_Gray has joined #openstack-ironic | 08:20 | |
| *** jtomasek has quit IRC | 08:21 | |
| *** dougsz has quit IRC | 08:21 | |
| *** ijw has quit IRC | 08:22 | |
| *** dougsz has joined #openstack-ironic | 08:24 | |
| iurygregory | michchap, we had a few problems with the memory on tinyipa jobs, maybe you can try to increase and tell us how it goes? | 08:24 |
| michchap | iurygregory: yeah I bumped it to 512 the same as the ironic jobs but I seem to have gotten the same result so I figured I'd see if there was something I'd missed | 08:25 |
| iurygregory | michchap, do you have a link for the patch? | 08:26 |
| michchap | iurygregory: https://review.opendev.org/#/c/748100/1/zuul.d/jobs.yaml | 08:26 |
| patchbot | patch 748100 - x/networking-ansible - Tinyipa needs more memory in gate jobs - 1 patch set | 08:26 |
| *** dougsz has quit IRC | 08:27 | |
| rpittau | michchap: you might want to increase the size on L43 as well | 08:28 |
| michchap | rpittau: thanks | 08:30 |
| iurygregory | yeah | 08:30 |
| iurygregory | the job is not with 512 | 08:30 |
| iurygregory | michchap, also please increase then number of VMs | 08:31 |
| iurygregory | to 2 | 08:31 |
| iurygregory | IRONIC_VM_COUNT L40 | 08:31 |
| michchap | I think it was working with 1 before, but I can bump it and see. Thanks for the reivew :) | 08:32 |
| *** tosky has quit IRC | 08:32 | |
| iurygregory | please bump because we did changes on the tempest plugin | 08:32 |
| *** tosky has joined #openstack-ironic | 08:33 | |
| iurygregory | and now we require more VM's | 08:33 |
| *** johnhaan_ has quit IRC | 08:33 | |
| *** rajinir has quit IRC | 08:33 | |
| *** arne_wiebalck has quit IRC | 08:33 | |
| *** arne_wiebalck has joined #openstack-ironic | 08:35 | |
| janders | g'day Ironic o/ | 08:36 |
| iurygregory | hey janders o/\ | 08:37 |
| *** derekh has joined #openstack-ironic | 08:37 | |
| iurygregory | lol an extra \ | 08:37 |
| *** rajinir has joined #openstack-ironic | 08:37 | |
| rpittau | hey janders :) | 08:37 |
| iurygregory | any with some ideas on how I can make the uefi jobs happy on https://review.opendev.org/741886 ? no luck so far =( | 08:38 |
| patchbot | patch 741886 - ironic - Add tempest default_boot_option and altflavor - 12 patch sets | 08:38 |
| openstackgerrit | Merged openstack/ironic master: driver_internal_info in provision notifications https://review.opendev.org/747996 | 08:39 |
| *** ijw has joined #openstack-ironic | 08:48 | |
| *** strigazi has joined #openstack-ironic | 08:51 | |
| *** ijw has quit IRC | 08:52 | |
| *** dtantsur|afk is now known as dtantsur | 09:09 | |
| dtantsur | morning ironic | 09:09 |
| iurygregory | morning dtantsur | 09:11 |
| dtantsur | JayF: metalsmith is a user-friendly CLI tool for ironic[+neutron+glance]. the thing about its CI job, however, is that it's the only job on IPA that tests 1. local boot, 2. streaming raw images | 09:14 |
| dtantsur | I think the latter is the difference that is important for you | 09:14 |
| Qianbiao | hello, when install devstack with ironic plugin, it starts a node. And it has been in clean wait status for a long time. | 09:16 |
| Qianbiao | Can i see what the node is doing now? | 09:16 |
| Qianbiao | like remote login or something. | 09:16 |
| *** ijw has joined #openstack-ironic | 09:19 | |
| Qianbiao | And it seems it uses KVM to virtual node. But my server does not support virtual(it's a virtual itself, and not config to support virtual), not sure how it starts a virtual node. | 09:20 |
| janders | good morning dtantsur | 09:23 |
| *** ijw has quit IRC | 09:24 | |
| *** k_mouza has joined #openstack-ironic | 09:26 | |
| dtantsur | JayF: https://github.com/eventlet/eventlet/issues/147 \o/ | 09:29 |
| dtantsur | there are more issues. tl;dr: eventlet+multiprocessing=sadness | 09:31 |
| Qianbiao | it seems ironic start a node using qemu. And clean step fails, the error output in ironic-bm-logs is: end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) | 09:34 |
| Qianbiao | Any ideas to fix this issue | 09:34 |
| dtantsur | Qianbiao: do you have enough RAM? the default (DIB-based ramdisk) requires at least 3G | 09:36 |
| Qianbiao | oh god... | 09:37 |
| Qianbiao | dtantsur should RAM size the same as image file? i found it's 12G | 09:39 |
| Qianbiao | default qcow2 file is /var/lib/libvirt/images/node-0.qcow2 | 09:39 |
| *** rcernin_ has quit IRC | 09:39 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible https://review.opendev.org/746774 | 09:40 |
| *** k_mouza has quit IRC | 09:40 | |
| dtantsur | folks, could you please review https://review.opendev.org/#/c/747960/ ? | 09:41 |
| patchbot | patch 747960 - sushy - Less scary warning when GET /redfish/v1/SessionSer... - 1 patch set | 09:41 |
| dtantsur | Qianbiao: no, but IPA has to fit into it | 09:41 |
| dtantsur | meaning, at least 3G of RAM | 09:41 |
| Qianbiao | ok thanks | 09:41 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible https://review.opendev.org/746774 | 09:43 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy stable/train: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/748145 | 09:46 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy stable/stein: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/748147 | 09:47 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy stable/rocky: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/748149 | 09:49 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy stable/queens: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/748150 | 09:51 |
| *** ijw has joined #openstack-ironic | 09:51 | |
| *** ijw has quit IRC | 09:56 | |
| *** k_mouza has joined #openstack-ironic | 09:57 | |
| *** gryf has quit IRC | 09:58 | |
| *** mkrai_ has quit IRC | 09:58 | |
| *** mkrai__ has joined #openstack-ironic | 09:58 | |
| *** gryf has joined #openstack-ironic | 10:00 | |
| *** mkrai has joined #openstack-ironic | 10:02 | |
| *** mkrai__ has quit IRC | 10:02 | |
| rpittau | no ussuri? :) | 10:07 |
| dtantsur | has merged already? | 10:08 |
| dtantsur | https://review.opendev.org/#/c/747956/ | 10:08 |
| patchbot | patch 747956 - sushy (stable/ussuri) - Do not log passwords and auth tokens when using Se... (MERGED) - 1 patch set | 10:08 |
| rpittau | yeah, didn't see in the list and forgot about it | 10:09 |
| iurygregory | hehehe | 10:17 |
| *** ijw has joined #openstack-ironic | 10:22 | |
| openstackgerrit | Merged openstack/ironic master: Enforce autospec in ilo tests https://review.opendev.org/747983 | 10:24 |
| *** ijw has quit IRC | 10:27 | |
| *** vmud213 has joined #openstack-ironic | 10:30 | |
| openstackgerrit | Merged openstack/bifrost stable/ussuri: Explicitly set permissions on /httpboot contents https://review.opendev.org/748054 | 10:34 |
| openstackgerrit | Merged openstack/bifrost bugfix/8.3: Explicitly set permissions on /httpboot contents https://review.opendev.org/748058 | 10:40 |
| *** ijw has joined #openstack-ironic | 10:53 | |
| *** mkrai has quit IRC | 10:55 | |
| *** ijw has quit IRC | 10:58 | |
| dtantsur | rpittau: could you check https://review.opendev.org/#/c/747960/ please? | 11:12 |
| patchbot | patch 747960 - sushy - Less scary warning when GET /redfish/v1/SessionSer... - 1 patch set | 11:12 |
| dtantsur | and when it merges, we should request a master release | 11:12 |
| rpittau | dtantsur: ack | 11:12 |
| openstackgerrit | Merged openstack/sushy stable/stein: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/748147 | 11:21 |
| *** ijw has joined #openstack-ironic | 11:25 | |
| *** ijw has quit IRC | 11:30 | |
| *** JamesBenson has joined #openstack-ironic | 11:33 | |
| *** JamesBenson has quit IRC | 11:40 | |
| *** JamesBenson has joined #openstack-ironic | 11:40 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:50 | |
| *** sshnaidm|afk is now known as sshnaidm|mtg | 11:56 | |
| *** ijw has joined #openstack-ironic | 11:56 | |
| openstackgerrit | Merged openstack/sushy stable/train: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/748145 | 11:59 |
| *** dougsz has joined #openstack-ironic | 12:08 | |
| openstackgerrit | vinay kumar muddu proposed openstack/ironic master: Adds ilo-uefi-https boot interface to ilo5 https://review.opendev.org/745501 | 12:10 |
| janders | time to get some sleep - see you tomorrow Ironic o/ | 12:14 |
| janders | have a good day everyone | 12:14 |
| Qianbiao | dtantsur I have increase vm memory to 32Gb, still the same error occurs. | 12:15 |
| Qianbiao | end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) | 12:15 |
| iurygregory | bye janders | 12:25 |
| vmud213 | Hey iurygregory! | 12:26 |
| vmud213 | Hello Ironic | 12:26 |
| iurygregory | Qianbiao, how much memory for the virtual baremetal? | 12:26 |
| iurygregory | hey vmud213 o/ | 12:27 |
| *** aedc has joined #openstack-ironic | 12:27 | |
| Qianbiao | iurygregory dtantsur no worry, it seems i should update local.conf, but not just increase vm memory size. :) | 12:27 |
| Qianbiao | reinstall now. | 12:27 |
| iurygregory | yeah =) | 12:28 |
| iurygregory | IRONIC_VM_SPECS_RAM | 12:28 |
| Qianbiao | sure. | 12:32 |
| openstackgerrit | Merged openstack/sushy master: Less scary warning when GET /redfish/v1/SessionService fails https://review.opendev.org/747960 | 12:36 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: [WIP] Move ironic-standalone-ipa-src to Ubuntu Focal https://review.opendev.org/748213 | 12:36 |
| *** thiagop has joined #openstack-ironic | 12:38 | |
| *** rloo has joined #openstack-ironic | 12:39 | |
| rpittau | dtantsur: https://review.opendev.org/748214 | 12:41 |
| patchbot | patch 748214 - releases - Release Sushy 3.3.1 for victoria - 1 patch set | 12:41 |
| *** mkrai has joined #openstack-ironic | 12:42 | |
| *** alexmcleod has quit IRC | 12:52 | |
| *** alexmcleod has joined #openstack-ironic | 12:53 | |
| *** ijw has quit IRC | 12:55 | |
| *** ijw has joined #openstack-ironic | 12:55 | |
| dtantsur | thanks! | 12:56 |
| *** ftarasenko has quit IRC | 12:58 | |
| *** bnemec has joined #openstack-ironic | 12:59 | |
| *** ftarasenko has joined #openstack-ironic | 13:00 | |
| *** mkrai has quit IRC | 13:02 | |
| *** mkrai has joined #openstack-ironic | 13:02 | |
| iurygregory | dtantsur, if we are doing netboot for partition images the flavor needs an ephemeral partition? | 13:03 |
| TheJulia | good morning | 13:04 |
| iurygregory | good morning TheJulia =) | 13:06 |
| *** alexmcleod has quit IRC | 13:07 | |
| TheJulia | I may need to lay back down, I'msuper tired | 13:07 |
| dtantsur | morning TheJulia | 13:08 |
| dtantsur | yep, get some rest | 13:08 |
| TheJulia | I started hacking on a patch yesterday with the aim of preserving /boot/efi contents from a partition image and re-using them | 13:09 |
| TheJulia | next step would be to also add retention of the uefi partition to /etc/fstab. Any thoughts/objections on such changes? | 13:10 |
| *** alexmcleod has joined #openstack-ironic | 13:10 | |
| iurygregory | go get some rest =) | 13:10 |
| TheJulia | gah, okay. | 13:11 |
| * TheJulia takes the fresh cup of coffee and goes and hides in the darkness for a little while and hopes that cat gravity does not capture her | 13:11 | |
| * arne_wiebalck wonders if that was iurygregory 's answer to TheJulia 's question | 13:11 | |
| dtantsur | TheJulia: I guess we don't mind that it's linux-specific because partition images are linux specific? | 13:11 |
| dtantsur | I'd honestly read an RFE on it first before judging | 13:11 |
| iurygregory | arne_wiebalck, lol | 13:12 |
| iurygregory | I still need to look at the patch =) | 13:12 |
| iurygregory | it's on my list | 13:12 |
| TheJulia | Yeah, that was kind of what I was thinking but more as a bug handling sort of thing since people are trying ot apparently upgrade grub on partition images and are hitting issues | 13:12 |
| TheJulia | bbiab | 13:12 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Move services to internal_ip by default and refactor endpoint creation https://review.opendev.org/748230 | 13:31 |
| dtantsur | feels suitable for the upcoming major release ^^ | 13:31 |
| openstackgerrit | Isaac Prior proposed openstack/bifrost master: Allow custom elements by appending existing ELEMENTS_PATH var if exists https://review.opendev.org/748231 | 13:34 |
| openstackgerrit | Isaac Prior proposed openstack/bifrost master: Allow custom elements by appending existing ELEMENTS_PATH var https://review.opendev.org/748231 | 13:36 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add tempest default_boot_option and altflavor https://review.opendev.org/741886 | 13:36 |
| * TheJulia needs new sinuses | 13:40 | |
| dtantsur | :( | 13:41 |
| TheJulia | dtantsur: I'll put something in storyboard once I have a call with one of the grub folks in about an hour | 13:42 |
| dtantsur | ++ | 13:42 |
| dtantsur | oh, my naive attempt to create endpoints does not work :( | 13:43 |
| Qianbiao | hello, how to source ironic devstack envs in a fresh session. | 13:43 |
| *** tzumainn has joined #openstack-ironic | 13:43 | |
| iurygregory | Qianbiao, . openrc admin admin | 13:43 |
| TheJulia | dtantsur: doh | 13:43 |
| iurygregory | inside the devstack dir | 13:43 |
| * dtantsur shakes fist at no_log | 13:43 | |
| Qianbiao | iurygregory seems not work, echo $DEFAULT_IMAGE_NAME is empty | 13:45 |
| TheJulia | Do you guys see any issue if I take Thursday/Friday off? | 13:45 |
| TheJulia | I promise I'll do spec reviews later today :) | 13:45 |
| Qianbiao | echo $IRONIC_IMAGE_NAME is empty too, those should be provided by ironic devstack. iurygregory | 13:46 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Trivial: remove a reference to update-rc.d https://review.opendev.org/748237 | 13:46 |
| TheJulia | I have short shifters and replacement motor mounts arriving ?today? so a mechanical project should be kind of refreshing | 13:46 |
| iurygregory | the image is not uploaded or available you mean? | 13:46 |
| dtantsur | TheJulia: distraction is always helpful. we did end up going climbing yesterday, it was a huge relief. | 13:47 |
| rpittau | TheJulia: you should, we'll hold the fort against Zuul :) | 13:47 |
| Qianbiao | I am following doc: https://docs.openstack.org/ironic/train/contributor/dev-quickstart.html | 13:47 |
| TheJulia | dtantsur: awesome! | 13:47 |
| *** Goneri has joined #openstack-ironic | 13:47 | |
| Qianbiao | when run "image=$(openstack image show $DEFAULT_IMAGE_NAME -f value -c id)" could not load the env. | 13:47 |
| iurygregory | well you can try openstack image list | 13:48 |
| iurygregory | and check if you have a cirros image there | 13:49 |
| Qianbiao | <iurygregory> checked, yes | 13:49 |
| iurygregory | so you can just use the information instead of the variable | 13:49 |
| Qianbiao | I know the env value is: cirros-0.5.1-x86_64-uec, and it exists in image list command. | 13:50 |
| iurygregory | not sure why it didn't set but you have the img =) | 13:50 |
| Qianbiao | ~ ok | 13:50 |
| dtantsur | "Service baremetal does not exist". we... create it.. in the previous task... | 13:50 |
| iurygregory | huh | 13:51 |
| iurygregory | O.o | 13:51 |
| iurygregory | seems like ironic is not installed... | 13:52 |
| dtantsur | it's not about ironic, it's pure keystone | 13:52 |
| TheJulia | hmm | 13:52 |
| *** mkrai has quit IRC | 13:52 | |
| *** sshnaidm|mtg is now known as sshnaidm | 13:52 | |
| TheJulia | check endpoint names for unicode chars? | 13:53 |
| dtantsur | maybe? trying to print the result of the service creation now | 13:53 |
| dtantsur | TheJulia: do we have any reasons to keep rabbitmq support in bifrost? | 13:57 |
| iurygregory | I would say no | 13:59 |
| TheJulia | The hope was to eventually add neutron support | 13:59 |
| TheJulia | but I never got those patches out of WIP | 13:59 |
| TheJulia | so *shrug* | 14:00 |
| dtantsur | I hope with OVN we won't need rabbitmq | 14:00 |
| dtantsur | also, WTF?? http://paste.openstack.org/show/797172/ | 14:00 |
| * TheJulia shrugs | 14:00 | |
| TheJulia | could it be because your enabling it before it is created? | 14:01 |
| iurygregory | dtantsur, lol | 14:01 |
| Qianbiao | hello, when using nova to deploy bm, how nova know it should use ironic driver? | 14:05 |
| Qianbiao | could not find a doc talk about this. | 14:05 |
| Qianbiao | should any metadata be set | 14:05 |
| dtantsur | TheJulia: figured it, the module does not work with names, only IDs.. | 14:07 |
| TheJulia | ugh | 14:07 |
| openstackgerrit | Isaac Prior proposed openstack/bifrost master: Allow custom elements by appending existing ELEMENTS_PATH var https://review.opendev.org/748231 | 14:09 |
| *** jtomasek has joined #openstack-ironic | 14:10 | |
| *** mkrai has joined #openstack-ironic | 14:11 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Move services to internal_ip by default and refactor endpoint creation https://review.opendev.org/748230 | 14:14 |
| dtantsur | okay, this works locally ^^^ | 14:22 |
| dtantsur | works around the problem by using service ID | 14:22 |
| JayF | dtantsur: thanks for the updates to my patch, I read the changes... interesting about lower-requirements ending up differently, it was absolutely the same locally, I thought | 14:26 |
| dtantsur | sigh | 14:27 |
| dtantsur | unfortunately, my updates don't seem to work | 14:27 |
| JayF | dtantsur: they are a MUCH better failure | 14:28 |
| JayF | IPA in metalsmith didn't hard-freeze, it didn't pass verification | 14:28 |
| JayF | I put relevant logs in zuul comments | 14:28 |
| JayF | dtantsur: it's a little frustrating that we picked up an eventlet dep by proxy, but it looks like none of IPA has been written with eventlet-safety in mind :| | 14:30 |
| dtantsur | yep :) | 14:32 |
| *** derekh has quit IRC | 14:32 | |
| dtantsur | :( | 14:32 |
| *** derekh has joined #openstack-ironic | 14:32 | |
| JayF | dtantsur: are you sure in your change, threading=False is the right thing to do? | 14:33 |
| dtantsur | bool(are you sure.*eventlet.*) == False | 14:33 |
| JayF | reading those issues, it looks like the workaround was just to pass that threading state around | 14:33 |
| dtantsur | i.e. no, shooting in darkness | 14:33 |
| JayF | I'm glad I'm not the only one | 14:33 |
| JayF | I'm going to drop the thread=False | 14:33 |
| openstackgerrit | wu.chunyang proposed openstack/ironic-lib master: Remove the unused coding style modules https://review.opendev.org/748246 | 14:34 |
| JayF | hmm, it also seems that that problem was fixed | 14:34 |
| JayF | from the linked bug | 14:34 |
| JayF | dtantsur: WDYT about going the other direction: just only patching the modules in eventlet needed to make SSL work | 14:35 |
| dtantsur | JayF: as long as it works, I guess | 14:35 |
| JayF | I'm a little afraid that's going to leave little bugs throughout the code, but they'd be preexisting and I'm THRILLED to dodge them at this point | 14:35 |
| JayF | It sounds like the "real fix" is to audit IPA overall for uses of libraries that are not well-supported by eventlet, and purge them | 14:36 |
| JayF | which is a more significant undertaking than I think any of us want to take on | 14:36 |
| *** dustinc has joined #openstack-ironic | 14:37 | |
| *** janders has quit IRC | 14:37 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 14:41 |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible https://review.opendev.org/746774 | 14:44 |
| *** Qianbiao has quit IRC | 14:46 | |
| *** ijw has quit IRC | 14:50 | |
| * TheJulia needs to find a different coffeeeeeeeeee | 14:52 | |
| iurygregory | what coffee usually you have TheJulia ? | 14:53 |
| TheJulia | I vary from month to month | 14:53 |
| TheJulia | Right now I'm drinking Pete's something | 14:53 |
| TheJulia | house blend I think | 14:53 |
| iurygregory | nice =) | 14:53 |
| TheJulia | done as drip coffee | 14:53 |
| TheJulia | I need something less acidic | 14:54 |
| iurygregory | https://www.littlecoffeeplace.com/low-acid-coffee/ maybe it would help | 14:54 |
| *** ijw has joined #openstack-ironic | 14:56 | |
| *** tkajinam has quit IRC | 14:57 | |
| dtantsur | we're boring people and just buy locally roasted coffee from a supermarket | 14:57 |
| *** trandles has joined #openstack-ironic | 14:59 | |
| JayF | I recently switched to cold brew. | 14:59 |
| JayF | I haven't figured out how to do it with "good" coffee yet, I just buy the preground, prebagged stuff. It's alright. | 14:59 |
| TheJulia | dtantsur: I need locally roasted, but sadly this is like a myth here | 15:01 |
| *** ijw has quit IRC | 15:01 | |
| TheJulia | everytime my stomach says it needs low acid coffee, it typically switch to a kona/maui blend | 15:02 |
| *** janders has joined #openstack-ironic | 15:02 | |
| trandles | dtantsur, I rebooted my bifrost ironic host today, which put selinux back in enforcing mode from permissive. in.tftpd got whacked as a result. Details at http://paste.openstack.org/show/797177/ | 15:04 |
| trandles | dtantsur, I _think_ you were the one who asked to let them know if selinux caused any issues... | 15:04 |
| trandles | worked fine after the actions in the paste | 15:05 |
| dtantsur | yes, I was, thank you! | 15:07 |
| openstackgerrit | Riccardo Pittau proposed openstack/sushy master: Use Sessions URL from root service if it's provided https://review.opendev.org/748256 | 15:08 |
| dtantsur | If you want to allow tftp to read and write files in the user home directories, you must turn on the tftp_home_dir boolean. Disabled by default. | 15:10 |
| dtantsur | this is weird, we shouldn't require that.. | 15:10 |
| JayF | Where does bifrost keep the tftp files/root? Is it under the httpd-user's homedir? like /var/www/tftproot or similar? | 15:11 |
| * JayF has seen that configuration in places before, but I don't know how bifrost sets it up | 15:12 | |
| dtantsur | JayF: nope, /tftpboot | 15:12 |
| dtantsur | I suspect it's because the labelling is wrong on it | 15:12 |
| trandles | let me know if you want me to run anything else to add to the paste | 15:13 |
| trandles | I did _not_ do the actions under the lower confidence suggestion (the dac_read_search capability) | 15:15 |
| dtantsur | yeah. I have a gut feeling that `setsebool -P tftp_home_dir 1` is not exactly right, but I cannot prove it | 15:15 |
| dtantsur | okay, I can reproduce the problem | 15:16 |
| dtantsur | mmm, after turning the variable on and off, the problem no longer reproduces Oo | 15:18 |
| dtantsur | trandles: could you check if this solves you problem (with the home_dir var disabled)? sudo chcon -t tftpdir_rw_t /tftpboot/ | 15:22 |
| trandles | dtantsur, did you say `setsebool -P tftp_home_dir 0` didn't cause the problem to reappear? | 15:23 |
| trandles | I'm on a physical host, not a VM :( | 15:23 |
| trandles | so resetting isn't trivial | 15:23 |
| dtantsur | trandles: it didn't for me :( | 15:23 |
| dtantsur | yeah.. I can probably reboot myself to see | 15:23 |
| trandles | sorry | 15:23 |
| *** gyee has joined #openstack-ironic | 15:26 | |
| *** ijw has joined #openstack-ironic | 15:29 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Add correct SELinux context for /tftpboot and fix map-file ACL https://review.opendev.org/748258 | 15:31 |
| dtantsur | trandles: I presume ^^^ | 15:31 |
| openstackgerrit | Riccardo Pittau proposed openstack/sushy master: Use Sessions URL from root service if it's provided https://review.opendev.org/748256 | 15:33 |
| *** ijw has quit IRC | 15:33 | |
| trandles | +1 - I might do a redeploy of things later this week. I'll let you know if problems persist. :) | 15:34 |
| dtantsur | cool, yeah. lemme know if you have other problems with selinux | 15:35 |
| dtantsur | I hope not many - other code paths should be covered by the CI | 15:35 |
| * dtantsur ponders a sanity check that uses the tftp command | 15:36 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Support partition image efi contents https://review.opendev.org/748049 | 15:36 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Add fstab pointer to EFI partition https://review.opendev.org/748260 | 15:36 |
| *** belmoreira has quit IRC | 15:36 | |
| TheJulia | https://storyboard.openstack.org/#!/story/2008070 <-- dtantsur the issue I was speaking of. Which is largely more in combined bug territory since they are inter-twined issues. hjensas This is the story I mentioned downstream that I would tag you on. | 15:36 |
| trandles | I've successfully deployed hosts today so I don't _think_ I'll see any other problems related to selinux | 15:37 |
| dtantsur | \o/ | 15:37 |
| dtantsur | TheJulia: I'll read a bit later if you're not in a hurry. I need to dive into TLS again. | 15:37 |
| TheJulia | dtantsur: no worries, I told them this wouldn't be a quick turnaround on our end | 15:37 |
| TheJulia | largely because of complexity and the fact that we'll need to backport fixes and yeah | 15:37 |
| TheJulia | this sort of stuff is never fun to handle | 15:38 |
| * TheJulia remembers the last time iurygregory delved into this code | 15:38 | |
| iurygregory | it was funny | 15:42 |
| iurygregory | much better than grenade =D | 15:42 |
| * TheJulia wonders if iurygregory is going to share the uefi pain beverage that apparently makes it funny :) | 15:42 | |
| iurygregory | hahaha | 15:43 |
| iurygregory | uefi beer | 15:43 |
| TheJulia | i hope this is a thing | 15:43 |
| iurygregory | grenade is 5L of vodka | 15:43 |
| JayF | uefi beer makes you FAT | 15:44 |
| iurygregory | bios beer returns some interesting beers on google images | 15:45 |
| iurygregory | uefi/efi beer nope =( | 15:45 |
| JayF | bios beer comes in small servings and is always a bit stale | 15:45 |
| iurygregory | we need the ironic beer (please don't tell me that IPA counts lol) | 15:46 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 15:46 |
| TheJulia | we clearly need to become friends with a brewmaster in some city in some country that can export their fine products to all of us | 15:49 |
| JayF | Pixie Bruus | 15:50 |
| iurygregory | ++ | 15:51 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Let us finish the rabbit hunt https://review.opendev.org/748265 | 15:53 |
| dtantsur | here goes ^^^ | 15:53 |
| * dtantsur has resisted using commit summary "die, rabbit, die" | 15:53 | |
| *** vmud213 has quit IRC | 15:53 | |
| * TheJulia wonders if that could be in german instead. | 15:54 | |
| TheJulia | https://twitter.com/ashinclouds/status/1298649796559831040 | 15:54 |
| iurygregory | like + RT #done | 15:56 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 15:56 |
| dtantsur | this ^^ is very close for ironic and inspector, but I'm afraid to try it with keystone :D | 15:56 |
| JayF | I see, no credit for Jay :P not even a Co-Authored-By on the tweet! /s | 15:57 |
| *** ijw has joined #openstack-ironic | 16:00 | |
| *** lucasagomes has quit IRC | 16:01 | |
| openstackgerrit | Merged openstack/ironic master: Remove the unused coding style modules https://review.opendev.org/747578 | 16:02 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 16:03 |
| *** Lucas_Gray has quit IRC | 16:03 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy stable/ussuri: Less scary warning when GET /redfish/v1/SessionService fails https://review.opendev.org/748266 | 16:03 |
| JayF | OK so with no thread=False, os=False, and the threading _active fix, we still get a hang in IPA | 16:04 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy stable/train: Less scary warning when GET /redfish/v1/SessionService fails https://review.opendev.org/748267 | 16:04 |
| JayF | with those three combined, we get no deadlock but image downlaods fail | 16:04 |
| TheJulia | O.O | 16:04 |
| dtantsur | wonderful.. | 16:04 |
| JayF | the only other string I know to pull would be to look at changing how heartbeats work, like TheJulia and I talked about yesterday | 16:05 |
| JayF | but I'm starting to wonder if I should just circle back, find the minimum possible change to prevent ssl from causing deadlocks, and climb out of the rabbithole | 16:05 |
| TheJulia | Yeah | 16:05 |
| *** ijw has quit IRC | 16:05 | |
| JayF | which option do you all think I should pursue? | 16:06 |
| * TheJulia ponders | 16:06 | |
| TheJulia | the reason for using select was wallclock vs system time based sleeps right? | 16:07 |
| TheJulia | because time resets would cause wonkiness? | 16:07 |
| TheJulia | (which is like... now a feature of the ramdisk...) | 16:07 |
| JayF | yes | 16:07 |
| JayF | I wonder if now that it's an active feature of the ramdisk | 16:07 |
| JayF | as long as it's timed so that we do it before the heartbeater begins | 16:08 |
| JayF | that we can just completely punt that requirement | 16:08 |
| JayF | since we know time will be synced before heartbeating begins | 16:08 |
| JayF | the other piece though, is forcing heartbeats | 16:08 |
| JayF | using select() gave us a mechanism to force a heartbeat, and have that forced heartbeat "reset" the interval | 16:08 |
| * TheJulia looks | 16:09 | |
| TheJulia | line 440 | 16:10 |
| TheJulia | we try with what we have config wise before we start the hearbteater | 16:10 |
| *** ijw has joined #openstack-ironic | 16:10 | |
| TheJulia | heartbeater | 16:11 |
| TheJulia | and then afterwards before we tear things down we try again so the system hardware clock is synced | 16:11 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: [DNM] log du /opt https://review.opendev.org/748269 | 16:11 |
| *** Lucas_Gray has joined #openstack-ironic | 16:12 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: [WIP] Move ironic-standalone-ipa-src to Ubuntu Focal https://review.opendev.org/748213 | 16:12 |
| JayF | TheJulia: where is the "afterwards", can you link it? | 16:12 |
| rpittau | goodnight, see you on Friday o/ | 16:12 |
| JayF | o/ | 16:13 |
| TheJulia | JayF: one moment | 16:13 |
| JayF | fwiw I'm also available to do a short sync meeting to look at this together if you want, I'm also OK to keep going async on irc | 16:13 |
| TheJulia | https://github.com/openstack/ironic-python-agent/blob/00ad03b709275e185ccb5b8e80cb7db4b9071d56/ironic_python_agent/extensions/standby.py#L730 | 16:13 |
| JayF | calendar is clear for another ~45m | 16:13 |
| JayF | so it looks like, to me, we can completely punt on having to use non-clock-based means for heartbeating | 16:14 |
| JayF | and likely can solve the "force a heartbeat" problem using other means (like just resetting a "time_last_heartbeated" variable or similar) | 16:14 |
| JayF | honestly, even if it doesn't fix the eventlet code, it'd made that a lot less complex for something that doesn't need to be complex anymore | 16:14 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 16:15 |
| TheJulia | I'm not entirely sure why force a heartbeat would be needed? | 16:16 |
| JayF | it's a time-saving measure: when IPA completes a command, it forces a heartbeat | 16:16 |
| dtantsur | huge +1 to ^^^ | 16:16 |
| JayF | this prevents a cleaning step (such as erase_device_metadata) which might literally take 10 seconds, from effectively taking a multiple of heartbeat_interval seconds | 16:17 |
| dtantsur | it will speed up fast-track as well | 16:17 |
| TheJulia | oh yeah, that I think would be a great idea | 16:17 |
| dtantsur | because now we have to wait for the next heartbeat to get to write_image | 16:17 |
| JayF | we already do that, fwiw | 16:17 |
| JayF | forcing a heartbeat when a command is complete | 16:17 |
| dtantsur | even though the deploy.deploy step takes a second | 16:17 |
| JayF | fast track likely needs *two* heartbeats | 16:17 |
| JayF | one to complete the previous command, one to begin the next one | 16:17 |
| JayF | which is why you're seeing that behavior | 16:17 |
| dtantsur | it's the same heartbeat | 16:17 |
| dtantsur | no, not because of this | 16:18 |
| JayF | okay; I'm confused now then :D | 16:18 |
| dtantsur | because when fast-track deployment starts, we're somewhere between two heartbeats | 16:18 |
| *** naved001 has quit IRC | 16:18 | |
| dtantsur | but I honestly no longer remember details | 16:18 |
| dtantsur | I remember it takes surprisingly long to get to the first actual deploy step | 16:18 |
| JayF | Ah, so with fast track deployment, you have an agent sitting there, in available, you tell it to deploy | 16:19 |
| JayF | deploy doesn't start until next heartbeat, which could be up to max(heartbeat_interval) seconds later | 16:19 |
| JayF | there's no way to fix that unless you added a commmand to IPA to force a heartbeat, which is possible but sorta starts to make the Ironic<->IPA dance a little strange | 16:20 |
| *** naved001 has joined #openstack-ironic | 16:20 | |
| JayF | this is why when we did this (hackily) at Rackspace, we had very small heartbeat_intervals, to avoid this problem | 16:20 |
| dtantsur | maybe we need to make them smaller indeed | 16:21 |
| JayF | it's a direct tradeoff between speed and resources | 16:21 |
| TheJulia | perhaps we should jump on a call | 16:21 |
| JayF | I'm available until 10am pst | 16:21 |
| JayF | let me get my headset hooked up, you wanna start up a bluejeans call? I can do a meet if you prefer too | 16:21 |
| TheJulia | lets use meetpad | 16:22 |
| JayF | just link it and I'll click it | 16:22 |
| JayF | going to grab a soda first | 16:22 |
| TheJulia | https://meetpad.opendev.org/ironic | 16:22 |
| *** Lucas_Gray has quit IRC | 16:23 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Disable DHCP on the libvirt network rather than doing `killall dnsmasq` https://review.opendev.org/748272 | 16:30 |
| dtantsur | has been annoying me for some time ^^^ | 16:30 |
| *** mkrai has quit IRC | 16:38 | |
| *** k_mouza has quit IRC | 16:41 | |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible https://review.opendev.org/746774 | 16:43 |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Eventlet and heartbeat compatability fixes https://review.opendev.org/746774 | 16:43 |
| *** k_mouza has joined #openstack-ironic | 16:44 | |
| *** k_mouza has quit IRC | 16:48 | |
| *** mkrai has joined #openstack-ironic | 16:53 | |
| *** mkrai has quit IRC | 16:55 | |
| *** johnthetubaguy has joined #openstack-ironic | 16:56 | |
| *** dtantsur is now known as dtantsur|afk | 16:57 | |
| dtantsur|afk | o/ | 16:57 |
| *** dougsz has quit IRC | 17:00 | |
| *** mkrai has joined #openstack-ironic | 17:02 | |
| *** mkrai has quit IRC | 17:07 | |
| *** derekh has quit IRC | 17:08 | |
| openstackgerrit | vinay kumar muddu proposed openstack/ironic master: Suffix the ISO names with '.iso' https://review.opendev.org/748277 | 17:09 |
| trandles | in baremetal cli prompt, if I do something that generates a usage message it kicks me back to my shell prompt...maybe that's something to do with my terminal and not the baremetal cli | 17:11 |
| trandles | http://paste.openstack.org/show/797183/ | 17:11 |
| TheJulia | or we explicitly exit on error | 17:12 |
| TheJulia | I bet it is the calls that are made :\ | 17:12 |
| trandles | just a nag, maybe a feature ;) | 17:17 |
| TheJulia | I'd file a bug | 17:18 |
| *** bnemec has quit IRC | 17:18 | |
| TheJulia | it is likely something silly and easy | 17:18 |
| trandles | will do | 17:20 |
| openstackgerrit | Vishal Manchanda proposed openstack/ironic-ui master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744708 | 17:21 |
| *** k_mouza has joined #openstack-ironic | 17:22 | |
| *** k_mouza has quit IRC | 17:26 | |
| *** aedc_ has joined #openstack-ironic | 17:42 | |
| *** aedc has quit IRC | 17:45 | |
| *** k_mouza has joined #openstack-ironic | 17:56 | |
| *** belmoreira has joined #openstack-ironic | 17:57 | |
| *** ijw has quit IRC | 17:59 | |
| *** ijw has joined #openstack-ironic | 18:00 | |
| *** k_mouza has quit IRC | 18:01 | |
| *** ijw has quit IRC | 18:01 | |
| *** ijw has joined #openstack-ironic | 18:02 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:11 | |
| *** aedc_ has quit IRC | 18:15 | |
| *** belmoreira has quit IRC | 18:30 | |
| *** belmoreira has joined #openstack-ironic | 18:32 | |
| openstackgerrit | Vishal Manchanda proposed openstack/ironic-ui master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744708 | 18:33 |
| JayF | TheJulia: heartbeat change passed all of the integration tests ... except for metalsmith, which fails in the "frozen" manner as before | 18:36 |
| TheJulia | ugh | 18:38 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Add fstab pointer to EFI partition https://review.opendev.org/748260 | 18:39 |
| TheJulia | I need to lay down for a little bit and then I'll switch gears to talking a look at that directly :( | 18:40 |
| TheJulia | and CI can run on that | 18:40 |
| JayF | I'm going to dig thru the imagedownload code | 18:40 |
| JayF | and google around | 18:40 |
| JayF | see if maybe I can put an eventlet.sleep() in the right place | 18:40 |
| JayF | TBH the most helpful thing would be a reproducing unit or integration | 18:40 |
| JayF | so we can tighten up the testing cycle | 18:41 |
| JayF | https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/extensions/standby.py#L356 /me wonders if that needs an else: eventlet.sleep(0) | 18:41 |
| JayF | https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/extensions/standby.py#L408 or somewhere around here | 18:44 |
| *** antotala has joined #openstack-ironic | 18:47 | |
| guilhermesp | hey there! yeah i raised up the question here before but, we are struggling a bit to apply a plugin we have written to inspector to collect custom data | 18:50 |
| guilhermesp | ive head JayF have done something similar in CERN | 18:50 |
| guilhermesp | we have here antotala that could also provide some more details on his methods of writing the plugin | 18:51 |
| *** sradhakrishna has joined #openstack-ironic | 18:52 | |
| *** antotala94 has joined #openstack-ironic | 18:53 | |
| guilhermesp | JayF: sorry, verizon :P | 18:54 |
| JayF | CERN has adapted some of my code for $things, but I've not written anything that works with inspector; nor do I work for cern | 18:54 |
| JayF | I don't use anything inspector. | 18:54 |
| JayF | If you can summarize your problem, I can attempt to help, although I'm deep into a bugfix myself | 18:54 |
| *** antotala has quit IRC | 18:55 | |
| guilhermesp | ok sorry for the confusion JayF , basically: | 18:57 |
| guilhermesp | we have a downstream plugin for ironic-inspector, we need to add an extra element to the image path to properly collect our custom data | 18:57 |
| *** antotala has joined #openstack-ironic | 19:00 | |
| *** sradhakrishna has quit IRC | 19:01 | |
| guilhermesp | and also how do we add the custom element so I can have an image that will allow inspector to collect the new data. | 19:02 |
| antotala94 | I am using ironic-python-agent-builder to build the custom ramdisk image, | 19:03 |
| JayF | Assuimng you do that via a hardware manager, I'm not 100% sure for inspector, then you can use these exmaples for how to build a hardware manager https://opendev.org/openstack/ironic-python-agent/src/branch/master/examples | 19:04 |
| guilhermesp | btw that's antotala94 which is working with me on that matter. antotala94 mind provided which steps you did to build the images? | 19:04 |
| JayF | as far as getting it into the ramdisk, you can develop a diskimage-builder element to co-install the hardware manager package you create alongside IPA in the ramdisk | 19:05 |
| antotala94 | we need to have some binaries to be present in the custom image, I am putting those binaries here https://github.com/openstack/ironic-python-agent-builder/tree/master/dib/ironic-python-agent-ramdisk/static/usr/local/bin | 19:05 |
| JayF | https://docs.openstack.org/diskimage-builder/latest/developer/developing_elements.html | 19:05 |
| JayF | as far as I'm aware, we do not have an example of installing a separate pip package using IPA-builder | 19:05 |
| JayF | but it's all well documented and relatively simple | 19:05 |
| antotala94 | for installing a separate pip package I am putting a command in this script https://github.com/openstack/ironic-python-agent-builder/blob/master/dib/ironic-python-agent-ramdisk/install.d/ironic-python-agent-ramdisk-source-install/60-ironic-python-agent-ramdisk-install | 19:06 |
| JayF | those are likely reasonable places to put them, if you're willing to directly edit the repo | 19:07 |
| *** zzzeek has quit IRC | 19:07 | |
| *** zzzeek has joined #openstack-ironic | 19:09 | |
| antotala94 | JayF and guilhermesp is there any default passowrd for the rescue user ? To make sure the binaries are present and the package is installed I have created an iso and booted a vm, but couldn't login :( | 19:11 |
| *** belmoreira has quit IRC | 19:12 | |
| JayF | https://docs.openstack.org/ironic-python-agent/latest/admin/troubleshooting.html This page covers getting access to a ramdisk running IPA in detail, I don't know anything about that that's not documented here. | 19:12 |
| antotala94 | tried following this https://docs.openstack.org/ironic-python-agent/latest/admin/troubleshooting for custom user/password creation while creating the image | 19:12 |
| antotala94 | but it doesn't seems to work | 19:12 |
| *** zzzeek has quit IRC | 19:25 | |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: DNM Eventlet and heartbeat compatability fixes https://review.opendev.org/746774 | 19:26 |
| JayF | TheJulia: ^ added a couple of eventlet.sleep(0) calls, as well as some debugging prints | 19:27 |
| JayF | TheJulia: if it fixes, I can see which sleep fixes it. If it doesn't, we should at least have a better idea if it's freezing in the image download code | 19:27 |
| *** zzzeek has joined #openstack-ironic | 19:28 | |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: DNM Eventlet and heartbeat compatability fixes https://review.opendev.org/746774 | 19:29 |
| *** zzzeek has quit IRC | 19:35 | |
| *** zzzeek has joined #openstack-ironic | 19:36 | |
| *** stevebaker has joined #openstack-ironic | 19:37 | |
| *** tosky has quit IRC | 19:38 | |
| *** antotala94 has quit IRC | 19:43 | |
| *** antotala has quit IRC | 19:43 | |
| *** thiagop has quit IRC | 19:47 | |
| TheJulia | blarg, I don't know what was up with my sinuses today but at least I feel somewhat better | 19:47 |
| *** thiagop has joined #openstack-ironic | 19:48 | |
| TheJulia | anlooks like antolala94 left, oh well | 19:49 |
| *** hjensas has quit IRC | 19:55 | |
| *** k_mouza has joined #openstack-ironic | 19:57 | |
| *** hjensas has joined #openstack-ironic | 19:59 | |
| trandles | TheJulia, you have smoke from the fires out there? My allergies have been off the charts due to a couple wildfires near Santa Fe :( | 20:00 |
| TheJulia | we did a few days ago but clear skies today,but today I jsut feel like I've been run over | 20:00 |
| TheJulia | okay, maybe a little more haze than normal, but yeah | 20:01 |
| *** k_mouza has quit IRC | 20:01 | |
| trandles | Today is the first day everything doesn't smell like fire. I can breathe! | 20:02 |
| TheJulia | \o/ | 20:06 |
| thiagop | Hello folks! Long time no see, how is everyone doing around here? | 20:08 |
| thiagop | I'm looking for something to stretch my community contribution muscles after 4 years away, but so much have changed... :) | 20:08 |
| TheJulia | hey thiagop | 20:15 |
| TheJulia | thiagop: any specific problem that is interesting you? | 20:15 |
| TheJulia | or problem space? | 20:15 |
| thiagop | TheJulia: o/ | 20:15 |
| thiagop | Not exactly, maybe a low hanging fruit to start getting the hang of it. It's being a while I used gerrit | 20:16 |
| TheJulia | so trandles actualy ran into somethign earlier | 20:17 |
| trandles | https://storyboard.openstack.org/#!/story/2008071 | 20:17 |
| TheJulia | http://paste.openstack.org/show/797183/ | 20:17 |
| TheJulia | \o/ | 20:17 |
| thiagop | Cool! Let me take a look at it then... :) | 20:18 |
| TheJulia | \o/ | 20:18 |
| *** Lucas_Gray has joined #openstack-ironic | 20:21 | |
| TheJulia | trandles: of course, my wife says things are smokey outside, so maybe that is it | 20:35 |
| trandles | If I wanted to track the status of an ironic-provisioned node through it's `lifetime` would it be abusive to use traits for this? I'm thinking something like "BURNED_IN"...a way to denote "this node has passed some milepost in its lifecycle" | 20:37 |
| trandles | TheJulia, we've been lucky and not had any smokey summers for 5 or so years. Our luck ran out last weekend. | 20:37 |
| trandles | maybe setting a property would be more appropriate for what I'm thinking | 20:38 |
| *** jtomasek has quit IRC | 20:38 | |
| TheJulia | trandles: traits seem like a perfect case | 20:38 |
| TheJulia | because if your using nova could youd schedule on a trait | 20:39 |
| trandles | yeah although I'm not using nova that came to mind | 20:39 |
| TheJulia | CUSTOM_I_HAVE_MADE_THIS_MACHINE_SUFFER_EXTENSIVE_BENCHMARKS | 20:39 |
| trandles | it would ease acceptance workflow for our burn-in racks of spare nodes | 20:40 |
| *** bnemec has joined #openstack-ironic | 20:40 | |
| trandles | we could automate the process from first power-up to ready for deployment into production by tracking what steps have been performed successfully | 20:41 |
| TheJulia | stevebaker: have you thought about metalsmith firing up a uefi job? | 20:48 |
| TheJulia | trandles: ++ | 20:48 |
| stevebaker | TheJulia: I have not thought about that | 20:49 |
| TheJulia | so I'm thinking centos8+uefi | 20:49 |
| TheJulia | I'm also thinking diskimage-builder | 20:50 |
| TheJulia | looks like we _should_ have it | 20:52 |
| stevebaker | yeah, that seems reasonable | 21:10 |
| * TheJulia is hacking on a job definition | 21:11 | |
| JayF | TheJulia: it failed, but this is not at all what I expected to see: https://3293bb8ceefb436e626f-9f324d44a8b8b4bcaa74562bb2dc2591.ssl.cf1.rackcdn.com/746774/17/check/metalsmith-integration-ipa-src/c716eb4/controller/ironic-bm-logs/node-0_console_2020-08-26-20%3A43%3A27.log | 21:20 |
| JayF | (see the change for details on where I added those prints) | 21:20 |
| JayF | frankly, I'm surprised to see >1 of any of them print | 21:21 |
| trandles | I'm so full of fun today! So...I powered up a node, it went through inspection and landed in enroll state, all good. Then I added an introspection rule (to define deploy info) and did `baremetal introspection start <nodeid>`. The node rebooted and went back through inspection. Everything looked fine until I started paying attention to the IPA output and see that the heartbeats are producing a traceback, return code of 400 from the inspecto | 21:22 |
| trandles | r. Deleting the node, ipmi power cycle, inspection happy and heartbeats are fine. | 21:22 |
| openstackgerrit | Julia Kreger proposed openstack/metalsmith master: Add a centos8/uefi job https://review.opendev.org/748315 | 21:28 |
| TheJulia | wow :( | 21:29 |
| TheJulia | JayF: ^^ | 21:29 |
| JayF | yeah | 21:29 |
| TheJulia | with how it is spitting out entries, doesn't look like buffered io, so we can be fairly sure it died right there | 21:31 |
| JayF | so I wonder if it's something about sizing | 21:32 |
| JayF | e.g. if it gets a <max(chunk_size) | 21:32 |
| *** k_mouza has joined #openstack-ironic | 21:33 | |
| JayF | it's deadlocking or something | 21:33 |
| JayF | TheJulia: unless you have a direction to suggest, I'm going to revert to plan B: try to find the minimal possible fix for the ssl deadlock and punt the rest | 21:39 |
| TheJulia | Yeah, I'd revert to plan b at this point | 21:41 |
| TheJulia | I'm fairly sure it is mid-stream based upon the chunk count too :( | 21:41 |
| JayF | perhaps when a heartbeat triggers? | 21:42 |
| *** bnemec has quit IRC | 21:42 | |
| JayF | hmm | 21:42 |
| *** naved002 has joined #openstack-ironic | 21:43 | |
| *** naved001 has quit IRC | 21:43 | |
| *** zzzeek has quit IRC | 21:45 | |
| *** zzzeek has joined #openstack-ironic | 21:48 | |
| TheJulia | yeah, that could likely be it | 21:49 |
| TheJulia | that _would_ be the only thing occuring | 21:49 |
| TheJulia | the only other thing that could cause that is the conenction stalling | 21:49 |
| TheJulia | but we have a timer around that and under normal circumstances we would handle that | 21:49 |
| JayF | I wonder if we need an `eventlet.sleep(0)` after do_heartbeat? | 21:49 |
| JayF | we do that in serve_api | 21:50 |
| TheJulia | it is worth a try | 21:50 |
| TheJulia | that would enable rescheduling the thread | 21:50 |
| TheJulia | which would be what we need actually | 21:50 |
| TheJulia | because it is not going to do ti for us | 21:50 |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: DNM Eventlet and heartbeat compatability fixes https://review.opendev.org/746774 | 21:51 |
| TheJulia | ironic_python_agent/extensions/standby.py line 363 most likely as well | 21:51 |
| TheJulia | after yeild, when returning | 21:51 |
| JayF | after yield; it's going to return to the top of the loop | 21:52 |
| JayF | there will be no chunk and it'll hit the else | 21:52 |
| JayF | right? | 21:52 |
| JayF | yield should do the equivalent to the eventlet.sleep(0), it's explicitly yielding execution back to the caller | 21:52 |
| *** bnemec has joined #openstack-ironic | 21:52 | |
| TheJulia | yeah, it should | 21:52 |
| JayF | and I did see the else case being hit | 21:53 |
| * TheJulia wonders if it is not actually | 21:53 | |
| JayF | actaully that's wrong | 21:53 |
| JayF | 'no chunk' doesn't show at all in that log | 21:53 |
| JayF | so going to also drop that else; move an eventlet.sleep(0) after the yield | 21:54 |
| JayF | although tbh I think it's /probably/ not needed, if we toss them everywhere and make them work | 21:54 |
| JayF | we can remove them until it breaks again :D | 21:54 |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: DNM Eventlet and heartbeat compatability fixes https://review.opendev.org/746774 | 21:54 |
| TheJulia | k, the elif code is only if the thing times out completely and the interator returns | 21:55 |
| JayF | I am pretty convinced the heartbeater was the thing | 21:55 |
| JayF | because if that takes over execution, there's zero reason for it to ever give it back | 21:55 |
| TheJulia | yeah | 21:55 |
| TheJulia | thiagop: another thing that would be super helpful is reviews :) | 21:57 |
| thiagop | Sure, that would indeed get me up to speed with the changes. I'm kinda struggling to get bifrost going here (never tried it) | 21:58 |
| thiagop | But I gotta call it a day in 2min, tomorrow I'll check out some reviews | 21:59 |
| JayF | the best thing you can do is become an expert on greenthreading and deadlocks in IPA /s | 21:59 |
| JayF | TheJulia: that's 100% the problem. It's the only time in any test job we have a thing happening that takes longer than a heartbeat | 21:59 |
| TheJulia | thiagop: have a wonderful evening | 22:00 |
| TheJulia | JayF: greenthreading + deadlocks + socketio | 22:01 |
| TheJulia | since socket io is the root of all evil | 22:01 |
| JayF | sprinkle in DNS and networking | 22:01 |
| JayF | and you'll summon one of the old computer-gods | 22:01 |
| TheJulia | It is all socket IO | 22:01 |
| TheJulia | all the way down! | 22:01 |
| TheJulia | well, at least to the operating system | 22:02 |
| JayF | this will be the #2 most impressive bug I've helped fix if we figure this out | 22:02 |
| thiagop | TheJulia: you too! | 22:02 |
| * TheJulia gets out the god of something touching the thinnet cable | 22:02 | |
| JayF | #1 was also a threading issue :| | 22:02 |
| thiagop | JayF: HAHAHA, that would be a life achievement | 22:03 |
| JayF | $lastJob had a custom log handler, used multithreading.Queue to queue up log messages for http-upload to a log collector. There was a race on shutdown, where the thread that did the work emptying the queue would get shutdown BEFORE the queue was closed to new entries. and a multithreading.Queue by default will persist until empty | 22:03 |
| JayF | and so you'd end up with that multithreading.Queue handler thread stuck open forever | 22:03 |
| JayF | that was maybe even less complex than this, because at least for that I got a pretty tight reproduction loop going | 22:04 |
| TheJulia | Well, this takes much more to reproduce | 22:05 |
| *** k_mouza has quit IRC | 22:06 | |
| JayF | TheJulia: I'm going to suggest, if we end up finding this is the problem, that we may want to run a job with an obscenely-short heartbeat interval | 22:08 |
| JayF | as an additional testing dimension | 22:08 |
| *** bnemec has quit IRC | 22:09 | |
| openstackgerrit | Merged openstack/sushy stable/ussuri: Less scary warning when GET /redfish/v1/SessionService fails https://review.opendev.org/748266 | 22:13 |
| janders | good morning Ironic o/ | 22:13 |
| TheJulia | JayF: we could actually use it for some cases as a semi-standard operating mode | 22:16 |
| TheJulia | JayF: just I see it also tuning all the retry timeouts upward automagically | 22:16 |
| *** ftarasenko has quit IRC | 22:17 | |
| JayF | I mean, we just hit one: for testing fast-track deployment mode | 22:18 |
| JayF | but IDK if you all even have a test for that in CI right now | 22:18 |
| TheJulia | bifrost I believe | 22:19 |
| JayF | that should /probably/ be tested in IPA/Ironic as well then IMHO | 22:20 |
| openstackgerrit | Merged openstack/sushy stable/train: Less scary warning when GET /redfish/v1/SessionService fails https://review.opendev.org/748267 | 22:20 |
| TheJulia | I actually think the job is the one we cross gate | 22:20 |
| TheJulia | just we also use cirros in those | 22:20 |
| TheJulia | metalsmith is the only larger one we have afaik | 22:20 |
| JayF | ironic-standalone-ipa-src is bifrost? | 22:21 |
| TheJulia | no | 22:21 |
| * TheJulia has to go look | 22:21 | |
| JayF | then there is no bifrost test in ipa gate | 22:21 |
| *** bnemec has joined #openstack-ironic | 22:21 | |
| TheJulia | oh :\ | 22:21 |
| TheJulia | well we need to fix that then | 22:21 |
| JayF | especially if it's the only one doing fast track | 22:21 |
| TheJulia | I can fix that a little later | 22:21 |
| JayF | hell, might wanna hook that up before I figure this eventlet stuff out | 22:21 |
| JayF | or else I can sneaky a bug in :-| | 22:21 |
| TheJulia | the reason I did the metalsmith chagne earlier weas because I noticed we don't actually test uefi in the ipa gate :( | 22:21 |
| trandles | gonna quit on a high note: `| traits | ['CUSTOM_BURN_IN_REQUIRED']` | 22:29 |
| trandles | have a great day/night everyone | 22:29 |
| *** trandles is now known as trandles|AFK | 22:29 | |
| *** janders has quit IRC | 22:31 | |
| *** janders has joined #openstack-ironic | 22:31 | |
| *** thiagop has quit IRC | 22:32 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Add fast-track CI job to IPA https://review.opendev.org/748321 | 22:34 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Add bifrost integration CI job to IPA https://review.opendev.org/748322 | 22:34 |
| TheJulia | goodnight trandles|AFK | 22:34 |
| TheJulia | so I ahve no idea if both of those job changes will work, but it should cover it if they do | 22:34 |
| JayF | nice | 22:35 |
| JayF | TheJulia: hmm... unless those jobs already build a ramdisk from source, don't we need to specify to build a deploy ramdisk ? | 22:39 |
| *** rcernin has joined #openstack-ironic | 22:40 | |
| *** janders has quit IRC | 22:40 | |
| JayF | e.g. https://github.com/openstack/ironic-python-agent/blob/master/zuul.d/ironic-python-agent-jobs.yaml#L149 | 22:40 |
| TheJulia | so I _think_ the bifrost one does, I just don't remember off-hand | 22:40 |
| TheJulia | if it doesn't, I need to get way more creative | 22:40 |
| JayF | ack; I'll look real quick | 22:40 |
| JayF | I think I know which way is up for zuul stuff now | 22:40 |
| JayF | mayb.e | 22:40 |
| TheJulia | ok, I'm trying to polish this ipa patch I've been working on | 22:41 |
| TheJulia | :) | 22:41 |
| *** janders has joined #openstack-ironic | 22:42 | |
| *** trandles|AFK has quit IRC | 22:42 | |
| JayF | https://github.com/openstack/bifrost/blob/master/zuul.d/bifrost-jobs.yaml#L54 looks like not that particular one | 22:42 |
| TheJulia | joy | 22:43 |
| TheJulia | so that will take a new job definition then :\ | 22:43 |
| JayF | yeah, I can take a stab at it | 22:43 |
| *** bnemec has quit IRC | 22:44 | |
| openstackgerrit | Steve Baker proposed openstack/metalsmith master: metalsmith_deployment role switch to metalsmith_instances https://review.opendev.org/737888 | 22:46 |
| *** bnemec has joined #openstack-ironic | 22:48 | |
| *** rloo has quit IRC | 22:48 | |
| *** JamesBenson has quit IRC | 22:50 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Support partition image efi contents https://review.opendev.org/748049 | 22:52 |
| TheJulia | well, that still needs negative path testing, but not bad | 22:52 |
| *** bnemec has quit IRC | 22:54 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Add fstab pointer to EFI partition https://review.opendev.org/748260 | 22:58 |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Add fast-track CI job to IPA https://review.opendev.org/748321 | 22:59 |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Add bifrost integration CI job to IPA https://review.opendev.org/748322 | 22:59 |
| openstackgerrit | Julia Kreger proposed openstack/metalsmith master: Add a centos8/uefi job https://review.opendev.org/748315 | 22:59 |
| TheJulia | gah, that should d oit | 22:59 |
| JayF | TheJulia: those ^ assume that `vars: devstack_localrc: [list of vars]` is additive, which appears to be justified by the evidence | 22:59 |
| TheJulia | that _should_ do it I think | 23:00 |
| JayF | the hardest part was the git dance for two commits in the same changes lol | 23:00 |
| *** tkajinam has joined #openstack-ironic | 23:01 | |
| TheJulia | :) | 23:02 |
| * TheJulia goes and starts working on a pizza for dinner | 23:03 | |
| *** zzzeek has quit IRC | 23:10 | |
| *** zzzeek has joined #openstack-ironic | 23:11 | |
| JayF | TheJulia: instead of a busy loop for heartbeats, I wonder if using eventlet.spawn_after() is the better way. We could cancel it when forcing a heartbeat, then restart it. https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.spawn_after | 23:15 |
| JayF | just a thought, going to leave now though | 23:15 |
| *** Goneri has quit IRC | 23:17 | |
| *** Lucas_Gray has quit IRC | 23:18 | |
| *** bnemec has joined #openstack-ironic | 23:21 | |
| *** bnemec has quit IRC | 23:28 | |
| *** jamesdenton has quit IRC | 23:36 | |
| JayF | TheJulia: even with those changes, it died *in the middle of printing a log message*. So I think I'm going to go to a change that just monkey patches socket in api.app :( | 23:45 |
| * TheJulia blinks repeatedly | 23:46 | |
| TheJulia | JayF: got a link handy? | 23:46 |
| *** bnemec has joined #openstack-ironic | 23:50 | |
| *** stevebaker has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!