| *** zzzeek has quit IRC | 00:31 | |
| *** zzzeek has joined #openstack-ironic | 00:39 | |
| *** tosky has quit IRC | 00:57 | |
| *** jawad_axd has joined #openstack-ironic | 01:01 | |
| *** zzzeek has quit IRC | 01:02 | |
| *** zzzeek has joined #openstack-ironic | 01:03 | |
| *** jawad_axd has quit IRC | 01:06 | |
| *** zzzeek has quit IRC | 01:13 | |
| *** zzzeek has joined #openstack-ironic | 01:14 | |
| *** zzzeek has quit IRC | 01:38 | |
| *** zzzeek has joined #openstack-ironic | 01:41 | |
| *** zzzeek has quit IRC | 01:50 | |
| *** zzzeek has joined #openstack-ironic | 01:52 | |
| *** ociuhandu has joined #openstack-ironic | 02:08 | |
| *** zzzeek has quit IRC | 02:09 | |
| *** ociuhandu has quit IRC | 02:13 | |
| *** zzzeek has joined #openstack-ironic | 02:13 | |
| *** rcernin has quit IRC | 02:45 | |
| *** uzumaki has quit IRC | 03:02 | |
| *** xinliang has joined #openstack-ironic | 03:03 | |
| *** rcernin has joined #openstack-ironic | 03:08 | |
| *** zzzeek has quit IRC | 03:20 | |
| *** zzzeek has joined #openstack-ironic | 03:21 | |
| *** rcernin has quit IRC | 03:27 | |
| *** rcernin has joined #openstack-ironic | 03:27 | |
| *** ociuhandu has joined #openstack-ironic | 03:56 | |
| *** ociuhandu has quit IRC | 04:01 | |
| *** ociuhandu has joined #openstack-ironic | 04:22 | |
| *** ociuhandu has quit IRC | 04:26 | |
| *** xinliang has quit IRC | 04:26 | |
| *** zzzeek has quit IRC | 04:34 | |
| *** zzzeek has joined #openstack-ironic | 04:35 | |
| *** zzzeek has quit IRC | 04:44 | |
| *** zzzeek has joined #openstack-ironic | 04:45 | |
| *** rcernin has quit IRC | 04:50 | |
| *** rcernin has joined #openstack-ironic | 04:54 | |
| *** zzzeek has quit IRC | 04:59 | |
| *** zzzeek has joined #openstack-ironic | 05:00 | |
| *** ociuhandu has joined #openstack-ironic | 05:05 | |
| *** mkrai has joined #openstack-ironic | 05:06 | |
| *** zzzeek has quit IRC | 05:06 | |
| *** zzzeek has joined #openstack-ironic | 05:08 | |
| *** ociuhandu has quit IRC | 05:10 | |
| *** bhagyashris|ruck is now known as bhagyashris | 05:33 | |
| *** zzzeek has quit IRC | 05:50 | |
| *** zzzeek has joined #openstack-ironic | 05:50 | |
| *** uzumaki has joined #openstack-ironic | 05:57 | |
| *** zzzeek has quit IRC | 06:05 | |
| *** jawad_axd has joined #openstack-ironic | 06:06 | |
| *** zzzeek has joined #openstack-ironic | 06:07 | |
| *** jawad_axd has quit IRC | 06:07 | |
| *** jawad_axd has joined #openstack-ironic | 06:07 | |
| *** mkrai has quit IRC | 06:13 | |
| *** zzzeek has quit IRC | 06:27 | |
| *** zzzeek has joined #openstack-ironic | 06:29 | |
| *** zzzeek has quit IRC | 06:50 | |
| *** zzzeek has joined #openstack-ironic | 06:50 | |
| *** ociuhandu has joined #openstack-ironic | 06:53 | |
| *** ociuhandu has quit IRC | 06:58 | |
| *** ociuhandu has joined #openstack-ironic | 07:00 | |
| *** zzzeek has quit IRC | 07:18 | |
| *** zzzeek has joined #openstack-ironic | 07:19 | |
| *** zzzeek has quit IRC | 07:24 | |
| *** zzzeek has joined #openstack-ironic | 07:25 | |
| *** ociuhandu has quit IRC | 07:27 | |
| *** ildikov has quit IRC | 07:45 | |
| *** pas-ha has quit IRC | 07:45 | |
| *** PrinzElvis has quit IRC | 07:46 | |
| *** pas-ha has joined #openstack-ironic | 07:47 | |
| *** ildikov has joined #openstack-ironic | 07:47 | |
| *** PrinzElvis has joined #openstack-ironic | 07:47 | |
| *** sshnaidm_ has joined #openstack-ironic | 08:02 | |
| *** sshnaidm has quit IRC | 08:05 | |
| *** rcernin has quit IRC | 08:07 | |
| *** ociuhandu has joined #openstack-ironic | 08:26 | |
| *** rcernin has joined #openstack-ironic | 08:26 | |
| *** rcernin has quit IRC | 08:31 | |
| *** uzumaki has quit IRC | 08:37 | |
| *** rcernin has joined #openstack-ironic | 08:43 | |
| *** rcernin has quit IRC | 08:47 | |
| *** ociuhandu has quit IRC | 08:55 | |
| *** zzzeek has quit IRC | 08:58 | |
| *** zzzeek has joined #openstack-ironic | 09:01 | |
| *** ociuhandu has joined #openstack-ironic | 09:08 | |
| *** ociuhandu has quit IRC | 09:23 | |
| *** ociuhandu has joined #openstack-ironic | 09:27 | |
| *** uzumaki has joined #openstack-ironic | 09:28 | |
| *** ociuhandu has quit IRC | 09:32 | |
| *** ociuhandu has joined #openstack-ironic | 09:54 | |
| *** ociuhandu has quit IRC | 10:02 | |
| *** ociuhandu has joined #openstack-ironic | 10:15 | |
| *** ociuhandu has quit IRC | 10:33 | |
| *** ociuhandu has joined #openstack-ironic | 10:42 | |
| *** sshnaidm_ is now known as sshnaidm|rover | 10:50 | |
| *** ociuhandu has quit IRC | 10:52 | |
| *** tosky has joined #openstack-ironic | 10:55 | |
| *** ociuhandu has joined #openstack-ironic | 11:18 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-specs master: Snapshot support https://review.opendev.org/c/openstack/ironic-specs/+/746935 | 11:22 |
|---|---|---|
| *** ociuhandu has quit IRC | 11:39 | |
| *** zzzeek has quit IRC | 11:43 | |
| *** zzzeek has joined #openstack-ironic | 11:44 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-specs master: Snapshot support https://review.opendev.org/c/openstack/ironic-specs/+/746935 | 11:49 |
| *** zzzeek has quit IRC | 11:51 | |
| *** zzzeek has joined #openstack-ironic | 11:55 | |
| *** uzumaki has quit IRC | 12:20 | |
| *** ociuhandu has joined #openstack-ironic | 12:26 | |
| *** ociuhandu has quit IRC | 12:45 | |
| *** iurygregory has joined #openstack-ironic | 13:18 | |
| iurygregory | o/ | 13:22 |
| *** mkrai has joined #openstack-ironic | 14:04 | |
| *** ociuhandu has joined #openstack-ironic | 14:18 | |
| *** ociuhandu has quit IRC | 14:23 | |
| *** ociuhandu has joined #openstack-ironic | 14:23 | |
| *** ociuhandu has quit IRC | 14:29 | |
| *** ociuhandu has joined #openstack-ironic | 14:30 | |
| *** sshnaidm|rover has quit IRC | 14:49 | |
| *** mkrai has quit IRC | 15:23 | |
| *** anuradha1904 has joined #openstack-ironic | 15:31 | |
| *** ociuhandu has quit IRC | 15:35 | |
| *** ociuhandu has joined #openstack-ironic | 15:36 | |
| *** ociuhandu has quit IRC | 15:40 | |
| *** ociuhandu has joined #openstack-ironic | 15:43 | |
| *** JasonF has quit IRC | 15:55 | |
| *** uzumaki has joined #openstack-ironic | 16:48 | |
| *** jawad_axd has quit IRC | 17:21 | |
| *** jawad_axd has joined #openstack-ironic | 17:22 | |
| *** ociuhandu has quit IRC | 18:07 | |
| *** ociuhandu has joined #openstack-ironic | 18:08 | |
| *** ociuhandu has quit IRC | 18:13 | |
| *** ociuhandu has joined #openstack-ironic | 18:18 | |
| *** mgoddard has quit IRC | 18:31 | |
| *** ociuhandu has quit IRC | 18:32 | |
| *** ociuhandu has joined #openstack-ironic | 18:33 | |
| *** ociuhandu has quit IRC | 18:37 | |
| *** ociuhandu has joined #openstack-ironic | 18:43 | |
| NobodyCam | happy holidays to all the Ironic family | 18:46 |
| JayF | \o/ same | 19:02 |
| *** ociuhandu has quit IRC | 19:11 | |
| *** sshnaidm has joined #openstack-ironic | 19:11 | |
| *** ociuhandu has joined #openstack-ironic | 19:11 | |
| *** sshnaidm is now known as sshnaidm|rover | 19:11 | |
| *** ociuhandu has quit IRC | 19:17 | |
| *** ociuhandu has joined #openstack-ironic | 19:54 | |
| *** ociuhandu has quit IRC | 19:58 | |
| *** ociuhandu has joined #openstack-ironic | 20:14 | |
| *** ociuhandu has quit IRC | 20:15 | |
| *** ociuhandu has joined #openstack-ironic | 20:16 | |
| *** ociuhandu has quit IRC | 20:16 | |
| *** ociuhandu has joined #openstack-ironic | 20:16 | |
| *** ociuhandu has quit IRC | 20:28 | |
| *** ociuhandu has joined #openstack-ironic | 20:42 | |
| *** zzzeek has quit IRC | 20:53 | |
| *** zzzeek has joined #openstack-ironic | 20:55 | |
| *** ociuhandu has quit IRC | 21:00 | |
| *** uzumaki has quit IRC | 21:05 | |
| *** ociuhandu has joined #openstack-ironic | 21:18 | |
| *** ociuhandu has quit IRC | 21:23 | |
| *** ociuhandu has joined #openstack-ironic | 21:36 | |
| *** ociuhandu has quit IRC | 21:43 | |
| *** ociuhandu has joined #openstack-ironic | 21:57 | |
| *** markguz_ has joined #openstack-ironic | 23:03 | |
| markguz_ | Hi, I;ve been scratching my head on a problem for a couple of weeks now where if i spawn a baremetal instance, it can take up to 10 minutes before the node powers on. vms deploy instantly but baremetal takes just so long. | 23:06 |
| markguz_ | i am running rocky. I was going through upgrading and hit this problem and have been debugging it for what seems like forever. i don't want to continue on with upgrading with ironic broken... | 23:07 |
| markguz_ | i've been checking rabbitmq and it seems ok. this is a relatively small stack with maybe 25 compute nodes and 50 baremetal nodes. and less that 100 users | 23:08 |
| markguz_ | i really don't kknow how to debug this any further. I'm trying to see what happens in nova when i make the request to spin up a baremetal server, but i'm kinda lost. | 23:09 |
| markguz_ | it seems to me that the request just seems to go into a void for a while then suddenly re-appears... | 23:11 |
| *** ociuhandu has quit IRC | 23:17 | |
| *** ociuhandu has joined #openstack-ironic | 23:17 | |
| *** ociuhandu has quit IRC | 23:21 | |
| ayoung | IPMI? | 23:33 |
| ayoung | btw markguz_ its break time and I doubt many people are around...I happen to be playiung a game on line and checked IRC.... | 23:34 |
| markguz_ | yeah. it's late in the day here, i'm in the US.. just frustrated with this. so... IPMI is working fine. manually controlling the nodes with ipmi is instant. | 23:35 |
| ayoung | And the nodes are all in the ready state, I take it | 23:35 |
| markguz_ | yup... | 23:35 |
| ayoung | available... | 23:35 |
| markguz_ | yeah | 23:36 |
| ayoung | its strange that it eventually works | 23:36 |
| JayF | sounds suspiciously like conductor overworked | 23:36 |
| JayF | or a long-held-lock on the node | 23:36 |
| ayoung | that says to me that maybe something is timing out...like it is having a schedular problem | 23:36 |
| ayoung | yeah | 23:37 |
| markguz_ | the conductor is doing nothing | 23:37 |
| markguz_ | this is not a busy environment | 23:37 |
| JayF | I'd poll `openstack baremetal node show $hostname` during the wait time, see when it flips to provision_state: wait call-back and power_state: on | 23:37 |
| JayF | that'll also tell you when nova schedules to it | 23:37 |
| JayF | that should help you isolate where in the path most of the time is being taken | 23:37 |
| JayF | or at least help bisect it :) | 23:38 |
| ayoung | is it possible that you have a non-existant something? | 23:38 |
| ayoung | Like, something was created and deleted, but the system still thinks it is there, and a request goes to it and dies in committee... | 23:38 |
| ayoung | I'm the worse person to try any help, BTW. I really only know Keystone | 23:38 |
| markguz_ | it's possible. not sure what that would be... this was working with pike.... then i went pike->queens->rocky and it no longer works | 23:39 |
| ayoung | But I make an OK rubber duck | 23:39 |
| ayoung | Time to upgrade | 23:39 |
| JayF | Heh | 23:39 |
| JayF | I mean, I can't ID any specific thing broken based on your description | 23:39 |
| JayF | most OpenStack cluster problems present that way :| | 23:39 |
| ayoung | Is this an upstream install, or from a distro? | 23:39 |
| JayF | the key is to figure out where it's taking the longest, or if it's just slow all the way thru | 23:39 |
| markguz_ | well.... eh.. no.. i stopped the process when this happened. i didn't wont to keep going with a broken component | 23:40 |
| ayoung | Do all baremetal nodes behave this way, or just one | 23:40 |
| JayF | I suspect ayoung may be right; and you'll discover the slowdown is "upstream" of Ironic, and once Ironic gets involved it goes pretty quick | 23:40 |
| ayoung | I am? Even a broken clock is right twice a day, I guess... | 23:40 |
| JayF | I mean, 'openstack cluster is slow' -> blame Nova scheduler | 23:41 |
| JayF | that's a tried and true troubleshooting flowchart :D | 23:41 |
| markguz_ | heh | 23:41 |
| JayF | lol | 23:41 |
| JayF | again I'll emphasize that we can't know annything without more info, but isolating what's taking a long time will help | 23:42 |
| ayoung | What process did you stop? Upgrading? | 23:42 |
| ayoung | I think he's look for help gathering that info | 23:42 |
| ayoung | where to look | 23:42 |
| JayF | if everything is slow, you're likely looking at rabbitmq issues, and I know diddly squat about troubleshooting that | 23:42 |
| markguz_ | ayoung: yeah. in my experience trying to upgrade something in openstack that's already not working just gets more not working. | 23:42 |
| JayF | > poll `openstack baremetal node show $hostname` during the wait time, see when it flips to provision_state: wait call-back and power_state: on | 23:42 |
| JayF | + reading conductor logs | 23:42 |
| ayoung | If it were Rabbit...I would think the message would just get dropped. But we do RPC, which means send and wait.... | 23:43 |
| ayoung | It could be that the message is getting dropped, but...that does not feel right | 23:43 |
| markguz_ | hmm... in the logs just after nova-conductor selects the node it starts a "block device mapping" for the instance, and that's the last reference for the instance uuid | 23:44 |
| ayoung | What are you using for storage? | 23:44 |
| markguz_ | noop | 23:44 |
| ayoung | noop? | 23:45 |
| markguz_ | it just uses the disk on the server. havene't changed that in for ever. | 23:45 |
| ayoung | Is it just this one server that is slow? Have you tried different Baremetal servers with the same problem? | 23:46 |
| markguz_ | i have tried many | 23:46 |
| markguz_ | all behave the same way | 23:46 |
| ayoung | What about disk images.... | 23:46 |
| ayoung | Could it be downloading the image each time? | 23:47 |
| ayoung | That sounds like the kind of mistake I would make. | 23:47 |
| markguz_ | storage_interface | noop | 23:47 |
| markguz_ | it streams direct from swift | 23:48 |
| ayoung | What is your overall set up? | 23:48 |
| JayF | I don't think storage interface matters at the nova-conductor mapping step | 23:48 |
| JayF | but IMBW | 23:48 |
| JayF | hmm | 23:48 |
| JayF | Are nova and Ironic running the same version? | 23:48 |
| ayoung | oooh | 23:49 |
| markguz_ | JayF: yep | 23:49 |
| markguz_ | hmm it just flipped and started the node | 23:50 |
| markguz_ | sat from 17:32 until 17:48 CST | 23:50 |
| ayoung | Well, if it was my son starting homework I would say that it finally got to a save point in his game. | 23:51 |
| ayoung | I know that is not helpful, but sometimes real life bleeds over. | 23:51 |
| JayF | I'd be checking logs for every service between nova-conductor and ironic | 23:51 |
| markguz_ | seems like a timeout at least | 23:52 |
| JayF | like nova-compute, ironic-conductor, as well as any services in the conductor log it talked to | 23:52 |
| JayF | who knows? I don't wanna speculate, the logs would have more info | 23:52 |
| JayF | if they don't seem to; enable debug and reproduce again | 23:52 |
| ayoung | What is the sequence after the message hits conductor | 23:52 |
| JayF | BTW; I'm going to be here for only ~8 more minutes | 23:52 |
| ayoung | I bet there is an error generated somewhere in there | 23:52 |
| JayF | Although these troubleshooting steps should get you more info to ask the channel again regardless of if I'm here or not :) | 23:52 |
| ayoung | Could it be placement? | 23:53 |
| ayoung | I would thinkg Placement would just fail outright | 23:54 |
| JayF | I don't wanna speculate too hard without real info | 23:54 |
| JayF | but yeah, that's what I suspect | 23:54 |
| JayF | I think as part of this upgrade, nova splits from placement | 23:54 |
| markguz_ | it could be anything. but regular VMs are more or less instant | 23:54 |
| JayF | and if placement isn't setup, I wouldn't be surprised to see this behavior | 23:54 |
| JayF | hmmm | 23:54 |
| markguz_ | actually that's stein | 23:54 |
| JayF | Ah | 23:54 |
| markguz_ | rocky is the last combined. | 23:54 |
| JayF | Yeah, you gotta find something in a log, or figure out where it's slowing down to get further help | 23:54 |
| JayF | we have lots of folks here who work with Queens clouds, so if you get more info it might be easy to nail down | 23:55 |
| markguz_ | yeah.. i'll dig around some more... and see. the logs seem to be nova-conductor triggering block-device mapping.. then nothing for 15mins then nova-compute -> neutron -> ironic | 23:56 |
| JayF | I'd reproduce with debug | 23:57 |
| markguz_ | handy just talking it out with folks has helped me clear the fog a bit. I'll do some more digging, now i have some info about what's happening. | 23:57 |
| markguz_ | thanks for your help ! | 23:57 |
| JayF | rubber ducking is what IRC troubleshooting is best for | 23:57 |
| JayF | good luck! quack quack | 23:57 |
| markguz_ | hehe | 23:57 |
| markguz_ | quack quack | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!