| opendevreview | Julia Kreger proposed openstack/ironic master: CI: set Neutron OVS of_inactivity_probe for CI load https://review.opendev.org/c/openstack/ironic/+/809426 | 00:10 |
|---|---|---|
| janders | good morning Ironic o/ | 00:21 |
| *** pmannidi is now known as pmannidi|AFK | 00:59 | |
| opendevreview | Merged openstack/ironic stable/train: Fix idrac-wsman set_power_state to wait on HW https://review.opendev.org/c/openstack/ironic/+/808982 | 02:12 |
| opendevreview | Merged openstack/ironic stable/train: Update the clear job id's constant https://review.opendev.org/c/openstack/ironic/+/798668 | 02:41 |
| stevebaker | TheJulia: you can ignore my request to look at those logs. It turns out the nodes are still running in bios mode. I'll probably need to pick your brains on Monday | 05:31 |
| arne_wiebalck | Good morning, Ironic! | 06:37 |
| janders | good morning arne_wiebalck | 06:53 |
| arne_wiebalck | hey janders o/ | 06:53 |
| iurygregory | good morning janders arne_wiebalck and Ironic o/ | 06:55 |
| iurygregory | TGIF | 06:55 |
| janders | hey iurygregory o/ | 06:55 |
| * janders had to google TGIF :) | 06:56 | |
| * iurygregory goes to grab more coffee | 06:56 | |
| iurygregory | janders, really? :O | 06:56 |
| janders | the IF part was pretty obvious but wasn't sure about the TG :) | 06:56 |
| iurygregory | got it =) | 06:57 |
| arne_wiebalck | heyh iurygregory o/ | 06:58 |
| opendevreview | Verification of a change to openstack/ironic bugfix/18.1 failed: Fix driver task pattern to reduce periodic db load https://review.opendev.org/c/openstack/ironic/+/809376 | 07:07 |
| opendevreview | Aija Jauntēva proposed openstack/ironic stable/wallaby: Add better error messages for invalid conf molds https://review.opendev.org/c/openstack/ironic/+/809508 | 07:54 |
| opendevreview | Aija Jauntēva proposed openstack/ironic stable/wallaby: Fix iDRAC import configuration missing task handling https://review.opendev.org/c/openstack/ironic/+/809509 | 07:54 |
| opendevreview | Aija Jauntēva proposed openstack/ironic stable/wallaby: Fix iDRAC import configuration job with errors https://review.opendev.org/c/openstack/ironic/+/809551 | 07:54 |
| opendevreview | Aija Jauntēva proposed openstack/ironic stable/wallaby: Redfish RAID: Use identity instead of durable_name https://review.opendev.org/c/openstack/ironic/+/809552 | 07:55 |
| opendevreview | Verification of a change to openstack/ironic master failed: Support HttpHeaders in create_subscription https://review.opendev.org/c/openstack/ironic/+/806859 | 08:01 |
| opendevreview | Verification of a change to openstack/ironic master failed: Support HttpHeaders in create_subscription https://review.opendev.org/c/openstack/ironic/+/806859 | 08:34 |
| opendevreview | Tadeas Kot proposed openstack/ironic-inspector master: Add support for state selector in the list introspection https://review.opendev.org/c/openstack/ironic-inspector/+/807578 | 09:15 |
| janders | see you next week Ironic o/ | 09:28 |
| janders | have a great weekend everyone | 09:28 |
| iurygregory | bye janders o/ | 09:35 |
| TheJulia | good morning | 13:08 |
| iurygregory | good morning TheJulia | 13:25 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-python-agent-builder stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/809671 | 13:28 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-python-agent-builder stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/809672 | 13:28 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-python-agent-builder master: Update master for stable/xena https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/809673 | 13:29 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-baremetal stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/networking-baremetal/+/809674 | 13:29 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-baremetal stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/networking-baremetal/+/809675 | 13:29 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-baremetal master: Update master for stable/xena https://review.opendev.org/c/openstack/networking-baremetal/+/809676 | 13:29 |
| TheJulia | stevebaker: ahh, yeah, they are still in bios mode (i looked anywa | 13:29 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-baremetal master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/networking-baremetal/+/809677 | 13:29 |
| TheJulia | anyway | 13:29 |
| TheJulia | ) | 13:29 |
| TheJulia | stevebaker: so yeah, if the xml files were never updated (since virtualbmc can't know how to swap wholesale configs back/forth for libvirt), then the would still be in bios mode and overriding everything would still cause things to boot. Coupled with libvirt binaries still speaking ipxe, the machine would have still booted. | 13:34 |
| TheJulia | There is likely an error in ipa in that it seems to not have caught the libvirt failure, but we were already down a funky rabbit hole too | 13:35 |
| TheJulia | Stderr: "Installing for i386-pc platform.\ngrub2-install: error: install device isn't specified.\n".: | 13:35 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-generic-switch stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/networking-generic-switch/+/809696 | 13:48 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-generic-switch stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/networking-generic-switch/+/809697 | 13:48 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-generic-switch master: Update master for stable/xena https://review.opendev.org/c/openstack/networking-generic-switch/+/809698 | 13:48 |
| opendevreview | OpenStack Release Bot proposed openstack/networking-generic-switch master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/networking-generic-switch/+/809699 | 13:48 |
| iurygregory | arne_wiebalck, https://github.com/metal3-io/ironic-hardware-inventory-recorder-image/pull/12 \o/ | 13:59 |
| iurygregory | it didn't add the label before because prow was down | 13:59 |
| arne_wiebalck | iurygregory: \o/ | 14:00 |
| arne_wiebalck | what is prow? | 14:00 |
| iurygregory | arne_wiebalck, its a k8s based CI/CD | 14:01 |
| iurygregory | https://github.com/kubernetes/test-infra/tree/master/prow | 14:01 |
| arne_wiebalck | Oh, ok ... thanks! | 14:03 |
| TheJulia | Any other reviews on https://review.opendev.org/c/openstack/ironic/+/803855 :) | 14:28 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix idrac-wsman having Completed with Errors jobs https://review.opendev.org/c/openstack/ironic/+/809747 | 14:31 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Fail when grub fails with current boot mode of BIOS https://review.opendev.org/c/openstack/ironic-python-agent/+/809754 | 14:39 |
| NobodyCam | Good Morning Ironic'ers... And OFC .... | 15:28 |
| NobodyCam | TGIF | 15:28 |
| TheJulia | good morning NobodyCam | 15:30 |
| NobodyCam | Good Morning | 15:30 |
| NobodyCam | TheJulia: | 15:31 |
| NobodyCam | hehehe | 15:31 |
| * TheJulia raises an eyebrow | 15:31 | |
| TheJulia | So I have this idea for something to superceed blendtec infomercials | 15:31 |
| TheJulia | "will it melt" | 15:31 |
| NobodyCam | diving in to the Joys of SNMP driver | 15:32 |
| TheJulia | oh my | 15:32 |
| NobodyCam | I like it, several levels heat, | 15:32 |
| NobodyCam | *of HEAT. | 15:32 |
| arne_wiebalck | Good morning TheJulia and NobodyCam o/ | 15:39 |
| NobodyCam | Good Morning arne_wiebalck | 15:39 |
| sam_z | mornin' ironic folks | 15:41 |
| * arne_wiebalck is dizzy from connecting to consoles for hours to fix nodes as part of the migration from virtual to physical instances ... older hardware can be challenging :) | 15:43 | |
| arne_wiebalck | Good morning sam_z o/ | 15:43 |
| NobodyCam | morning sam_z | 15:44 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: set Neutron OVS of_inactivity_probe for CI load https://review.opendev.org/c/openstack/ironic/+/809426 | 15:47 |
| * TheJulia crosses fingers.... | 15:47 | |
| TheJulia | only other option is to start hacking on devstcak | 15:50 |
| arne_wiebalck | bye everyone o/ | 15:57 |
| NobodyCam | night arne_wiebalck | 15:58 |
| sam_z | uh oh devstack | 16:20 |
| NobodyCam | so here is question; say someone has multiple resource-classes of hardware that use the SNMP driver, and ONE of those classes needs additional time between deployment SNMP power off (not IPA shutdown) and SNMP power on. | 16:33 |
| TheJulia | NobodyCam: a special trait woudl make sense and then a deploy step that I guess acted if there step was present | 16:44 |
| NobodyCam | oh I was thinking deploy steps wouldn't be able to effect the actions beyond IPA's | 16:45 |
| TheJulia | nope, they are logic code executed on the conductor | 16:45 |
| TheJulia | NobodyCam: dmitry put together a custom-agent recently which is entirely conductor side if memory serves | 16:48 |
| TheJulia | blarg | 17:08 |
| sam_z | heyo, i'm poking around in the API code, and i noticed that sometimes, controller methods are decorated with @pecan.expose and sometimes they're decorated with @ironic.api.methods.expose-- why is that? | 17:18 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: set Neutron OVS of_inactivity_probe for CI load https://review.opendev.org/c/openstack/ironic/+/809426 | 17:39 |
| TheJulia | sam_z: oh... umm... umm... they should all be @ironic.api.methods.expose. We went through an effort to nuke pecan a while back | 17:42 |
| TheJulia | but I guess... we missed something or it got re-introduced | 17:43 |
| sam_z | oh lol. it still lives on in some corners of the controller code | 17:46 |
| sam_z | what additional functionality does the @ironic.api.methods.expose decorator provide over the @pecan.expose decorator? | 17:48 |
| sam_z | "$ grep -R "@pecan.expose" . | wc -l" tells me there's 9 instances of @pecan.expose in the ironic repo btw | 17:50 |
| TheJulia | sam_z: got a link to an example real quick? | 18:14 |
| * TheJulia is currently deep in the neutron code trying to wrap her head around why it blows up on startup if there is a transient appearing failure | 18:14 | |
| sam_z | https://opendev.org/openstack/ironic/src/branch/master/ironic/api/controllers/v1/__init__.py#L254 | 18:15 |
| TheJulia | huh | 18:19 |
| sam_z | here's a list of where the 9 pecan.expose calls are (according to grep): https://paste.opendev.org/show/809416/ | 18:20 |
| sam_z | line numbers may be slightly off actually since i was messing with the code lol | 18:20 |
| TheJulia | hmm, there must be a reason | 18:21 |
| * TheJulia makes mental note to chat with stevebaker next week | 18:22 | |
| sam_z | TheJulia: all instances of pecan.expose are decorating either _lookup() or _route() methods apparently: https://paste.opendev.org/show/809417/ | 18:29 |
| TheJulia | that seems intentional | 18:30 |
| sam_z | there are 9 instances of @pecan.expose, 6 instances of _lookup(), and 3 instances of _route() in the codebase so there's probably a reason | 18:30 |
| sam_z | what that reason is, i have no idea because i'm cracking open the pecan documentation for the first time today B) | 18:31 |
| TheJulia | sam_z: I am fairly sure steve will know | 18:31 |
| sam_z | wow that's a lot of messages! | 18:53 |
| TheJulia | stevebaker: so... seems our neutron issues are all tied back to the initialization of the firewall. I don't grok the inner depths of that code very well, it seems that the code basically bails immediately upon failure because OVS reports an error about the openflow datapath not being valid... and I don't know where that is coming from but its a new transactional command to OVS so what might be happening is <200 ms of | 18:55 |
| TheJulia | cross thread lag *or* the integration bridge just not being fully fully up yet in the agent. | 18:55 |
| TheJulia | stevebaker: I suddenly wonder if ovs expects cross-cpu thread cache to actually be coherent | 18:55 |
| TheJulia | stevebaker: because it is explicitly impossible in VMs... | 18:56 |
| * TheJulia only knows this because she once had a database which *did* use it and when it got re-scheduled on to another CPU on a VM the cache was gone, so the VMs always got only one CPU which of course irked the DBA.... but all we could really tell the DBA was "complain to the vendor, their single threaded database!" | 18:59 | |
| opendevreview | Merged openstack/ironic stable/ussuri: Fix idrac-wsman set_power_state to wait on HW https://review.opendev.org/c/openstack/ironic/+/808981 | 19:05 |
| TheJulia | sweet | 19:32 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Disable Neutron firewall https://review.opendev.org/c/openstack/ironic/+/809895 | 19:37 |
| TheJulia | stevebaker: I'm going to try and discuss ^^^ with neutron folks next week, because I don't think neutron should die on startup. | 19:37 |
| NobodyCam | any interest in a patch that adds a configurable delay to snmp power on and off actions? | 21:47 |
| TheJulia | could be useful | 22:16 |
| TheJulia | she is hitting post event drop... hardcore | 22:18 |
| TheJulia | err, whoops | 22:18 |
| opendevreview | Verification of a change to openstack/ironic master failed: Support HttpHeaders in create_subscription https://review.opendev.org/c/openstack/ironic/+/806859 | 22:28 |
| opendevreview | Chris Krelle proposed openstack/ironic master: add snmp power action delay https://review.opendev.org/c/openstack/ironic/+/809902 | 22:32 |
| NobodyCam | there are no release notes or other things just 7 lines at this point. | 22:47 |
| opendevreview | Verification of a change to openstack/ironic master failed: Support HttpHeaders in create_subscription https://review.opendev.org/c/openstack/ironic/+/806859 | 22:50 |
| NobodyCam | Are ironic standalone test failing? | 23:29 |
| TheJulia | NobodyCam: not about an hour ago | 23:33 |
| NobodyCam | :) I saw two patches fail on the same test | 23:34 |
| TheJulia | O.o | 23:34 |
| TheJulia | link? | 23:34 |
| NobodyCam | https://zuul.opendev.org/t/openstack/build/1d2adc777778417bafcafa534d5187b0 | 23:35 |
| NobodyCam | looks like devstack | 23:35 |
| TheJulia | it is neutron most likely | 23:36 |
| TheJulia | why I'm chasing down the path of https://review.opendev.org/c/openstack/ironic/+/809895 | 23:36 |
| NobodyCam | https://zuul.opendev.org/t/openstack/build/2b270ae477d6454ca7df85d52e6be4e8 | 23:36 |
| NobodyCam | +++ | 23:37 |
| TheJulia | yup https://195efc623dcc108061a1-e3638d6f93b7f404b738b516d37f0f7d.ssl.cf5.rackcdn.com/806859/4/check/ironic-standalone/1d2adc7/controller/logs/screen-q-agt.txt | 23:37 |
| TheJulia | very end of the file | 23:37 |
| TheJulia | same | 23:39 |
| TheJulia | literally starts, gets down into the firewall startup and blows up | 23:39 |
| NobodyCam | :) | 23:39 |
| TheJulia | why in the world it doesn't keep retrying, I don't understand | 23:40 |
| TheJulia | I guess the model of "systemd will fix it" maybe, but... that is the wrong model to really take with a critical path item | 23:40 |
| NobodyCam | I've seen cases where systems didn't detect an app had crashed; or really mostly crashed | 23:45 |
| TheJulia | and systemd gives up eventually | 23:45 |
| NobodyCam | systems/systemd/ | 23:45 |
| TheJulia | what I don't really get is just how it fails, but I *suspect* if I start a day with no interruptions, I could likely trace it down | 23:46 |
| TheJulia | There is also just zero logging really around it all | 23:46 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!