| cardoe | Interestingly I'm getting "SYS336An existing hash value is updated because some system configuration items are changed." under Maintenance -> Lifecycle Log for those boxes. | 02:36 |
|---|---|---|
| cardoe | I've even tried "baremetal node passthru call <node> eject_vmedia" | 02:48 |
| janders | cardoe hmm I can't say I understand what's happening. I would try to 1) reset iDRAC 2) InsertMedia manually and see what happens | 06:39 |
| janders | if it fails, it would be curious to see if 2) results in the same error as what Ironic is getting | 06:39 |
| janders | https://paste.opendev.org/show/biJ90eL92ZfMVJfU1fcf/ example InsertMedia query from my notes | 06:42 |
| janders | in case some details come in handy | 06:42 |
| rpittau | good morning ironic! o/ | 07:29 |
| janders | hey rpittau o/ | 07:43 |
| rpittau | hey janders :) | 07:43 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: pip: Use SETUPTOOLS_USE_STDLIB if python < 3.12 https://review.opendev.org/c/openstack/bifrost/+/924828 | 08:55 |
| mnasiadka | Good morning Ironic | 09:02 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: pip: Use SETUPTOOLS_USE_STDLIB if python < 3.12 https://review.opendev.org/c/openstack/bifrost/+/924828 | 09:14 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Log node UUID and not entire node instance https://review.opendev.org/c/openstack/ironic/+/924838 | 09:53 |
| iurygregory | good morning Ironic | 11:08 |
| masghar | Morning! | 11:31 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Log node UUID and not entire node instance to avoid logging its driver_info field https://review.opendev.org/c/openstack/ironic/+/924838 | 11:55 |
| opendevreview | Michal Nasiadka proposed openstack/networking-generic-switch master: Cumulus NVUE support https://review.opendev.org/c/openstack/networking-generic-switch/+/907551 | 12:16 |
| TheJulia | Good morning | 13:02 |
| iurygregory | good morning TheJulia =) | 13:13 |
| masghar | o/ | 13:51 |
| masghar | I am trying to understand the failures in https://review.opendev.org/c/openstack/ironic/+/924838, and I suspect the ironic-tempest-uefi-redfish-vmedia, ironic-tempest-uefi-redfish-https and ironic-tempest-bios-ipmi-direct-tinyipa jobs are broken? Or these are temporary failures maybe? | 14:48 |
| masghar | I havent been able to find a proper error in the logs | 14:48 |
| iurygregory | masghar, "openstack-cli-server: 'object store account set --property Temp-URL-Key=secretkey' is not an openstack-cli-server command. See 'openstack-cli-server --help'." | 14:50 |
| iurygregory | this is the problem | 14:50 |
| masghar | ...interesting | 14:51 |
| iurygregory | maybe we need to change our script <thinking> | 14:51 |
| masghar | As in the CI job script? or dev-scripts? | 14:51 |
| cid | Got the same failures, may be temporary. | 14:51 |
| masghar | Yeah I saw your patch too cid | 14:52 |
| iurygregory | dev-scripts is metal3/openshift | 14:52 |
| iurygregory | it's the devstack script | 14:52 |
| masghar | oh sorry, yes. devstack then? | 14:52 |
| iurygregory | yeah =) | 14:53 |
| iurygregory | I've asked in openstacksdk to see if they merged something.. | 14:56 |
| rpittau | it doesn't look like it | 14:58 |
| rpittau | we're using openstackclient 6.6.0 that was released 4 months ago | 14:58 |
| rpittau | puzzled... | 14:58 |
| JayF | Check devstack config. There was a recent change to enable the CLI server.. it makes me wonder if somehow we're using something not supported in that mode | 14:59 |
| rpittau | ah! | 14:59 |
| masghar | (That line is 3 years old in the devstack repo, its likely not the culprit by itself) | 15:00 |
| clarkb | melwitt was looking into this yesterday. I'm not sure ifa ny progress was made | 15:01 |
| rpittau | well next blame is aliens :) | 15:01 |
| iurygregory | yeah | 15:01 |
| iurygregory | JayF, do you have a link for that? | 15:01 |
| iurygregory | because it sounds like the culprit since it says "is not an openstack-cli-server command " | 15:02 |
| iurygregory | maybe we can go with : "openstack-cli-server: false" | 15:04 |
| masghar | perhaps, but where? | 15:05 |
| iurygregory | or spend time trying to understand if is the cli-server problem or our problem XD | 15:05 |
| masghar | yeah xD | 15:05 |
| iurygregory | masghar, basically we need to override the service in our jobs, I'm already on it if we think is the path forward | 15:06 |
| clarkb | the cli-server significantly cuts down on job runtime by reusing tokens and avoiding recurring python startup costs for openstack commands | 15:06 |
| clarkb | so basically yes you could probably disable it and your jobs would avoid the issue but then your runtimes go back up | 15:06 |
| clarkb | if you figure out why it isn't working and fix it then you can have the best of both worlds | 15:06 |
| melwitt | clarkb: I didn't figure out what is wrong with it unfortunately | 15:06 |
| melwitt | I don't think the commands are from a separate library or at least I didn't find one | 15:07 |
| rpittau | yeah, disabling it would be temporary anyway, but we can do it to unblock the CI | 15:07 |
| clarkb | melwitt: ya the next step may be to run the cli server and give it commands like that. Maybe it is a parsing thing with the osc shell? | 15:07 |
| masghar | I think I am in a relevant place? https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/server.html | 15:12 |
| masghar | aside from the ancient version | 15:13 |
| rpittau | masghar: except for the very old version, yes, for openstackclient | 15:13 |
| rpittau | although the openstack-cli-server code is here https://github.com/openstack/devstack/blob/master/files/openstack-cli-server/openstack-cli-server | 15:13 |
| masghar | thanks ^ | 15:14 |
| clarkb | right openstack-cli-server is a small proxy for one off command to a long running osc shell | 15:22 |
| clarkb | so the way to try and reproduce this would be to start an osc shell and see if `object store account set --property Temp-URL-Key=secretkey` works as a command in that shell. Then work from tehre | 15:22 |
| masghar | It...should as per the Pike docs | 15:23 |
| clarkb | masghar: those are server commands not object store commands but also pike is the wrong version | 15:24 |
| clarkb | I don't think the link you pointed to is related to this unless I'm missing something | 15:24 |
| rpittau | if you look at the logs it's working fine until we call object store https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_391/924838/2/check/ironic-tempest-uefi-redfish-vmedia/39105e3/controller/logs/screen-openstack-cli-server.txt | 15:25 |
| masghar | I saw this: https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/object-store-account.html | 15:25 |
| masghar | clarkb: I cant seem to find the latest docs | 15:25 |
| clarkb | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/object-store-account.html | 15:25 |
| masghar | clarkb: thanks rpittau: true! | 15:26 |
| clarkb | and yes I agree the docs indicate this command should be valid | 15:26 |
| rpittau | I wonder if the missing cloud name is messing with the parsing | 15:27 |
| rpittau | here https://github.com/openstack/devstack/blob/d84b874ef6fff88b93e9b0eab79dea6e6868f94f/lib/swift#L845 | 15:29 |
| masghar | maybe, because the other commands are providing the cloud name | 15:29 |
| rpittau | yep, exactly | 15:30 |
| rpittau | clarkb, melwitt, wdyt? | 15:30 |
| clarkb | maybe? dansmith got the proxy working (though I'm not sure if he was the original author) and may know if that is expected | 15:31 |
| melwitt | rpittau: I did think that looked wrong that the --os-cloud is empty. what error that would cause, I dunno off the top of my head | 15:32 |
| dansmith | is this an upload of something large to swift through stdin? | 15:33 |
| dansmith | (by chance) | 15:33 |
| clarkb | dansmith: no ist just trying to set a property | 15:33 |
| clarkb | on the account I think | 15:33 |
| melwitt | when I do that locally on a devstack it does | 15:33 |
| melwitt | $ openstack --os-cloud '' server list | 15:33 |
| melwitt | Missing value auth-url required for auth plugin password | 15:33 |
| dansmith | okay, I have literally not seen any failures other than the image upload from stdin thing I fixed | 15:33 |
| dansmith | but you can't use the proxy to connect to a different --os-cloud of course, | 15:33 |
| dansmith | so if that is what it's trying to do, that won't work | 15:34 |
| dansmith | perhaps the shim needs to not use the proxy if you try to do something with a different account? | 15:34 |
| dansmith | well, I guess I shouldn't say that for sure.. from memory, that won't work | 15:34 |
| clarkb | oh that makes sense. This is modifying a swift account directly and not using the existing keystone credetnials so ya need an opt out maybe | 15:34 |
| melwitt | the error from the job is "openstack-cli-server: 'object store account set --property Temp-URL-Key=secretkey' is not an openstack-cli-server command. See 'openstack-cli-server --help'." | 15:35 |
| melwitt | and the command was openstack --os-cloud '' --os-region-name RegionOne --os-auth-url https://10.176.194.170/identity --os-username=swift --os-password=secretservice --os-user-domain-name=Default --os-project-name=service --os-project-domain-name=Default object store account set --property Temp-URL-Key=secretkey | 15:36 |
| dansmith | actually, looking at it, I think maybe I'm wrong and it can handle different accounts in a single session | 15:37 |
| dansmith | so "--os-cloud ''" means it's trying to null out --os-cloud yeah? | 15:38 |
| melwitt | yeah it looks like empty string but I dunno if it's doing that on purpose | 15:38 |
| masghar | According to the devstack code, its on purpose | 15:38 |
| melwitt | ack thanks | 15:39 |
| dansmith | so I bet that it's getting lost and so it's calling --os-cloud=--os-region-name" | 15:39 |
| masghar | https://github.com/openstack/devstack/blob/d84b874ef6fff88b93e9b0eab79dea6e6868f94f/lib/swift#L847 | 15:39 |
| dansmith | however, I definitely don't understand what that's supposed to do | 15:40 |
| masghar | same | 15:43 |
| rpittau | ok so this is interesting https://github.com/openstack/devstack/commit/f56f7a557ac4941b5204852f4173db9cf82b4dae | 15:43 |
| rpittau | I think that is supposed to be an override as devstack-admin is supposed to be used | 15:44 |
| rpittau | but in this case it's not working? | 15:44 |
| rpittau | frickler: hey I think you committed that, if you have a moment to shine some light there :) | 15:45 |
| rpittau | maybe we can change that to something like ${OS_CLOUD:-""} ? | 15:46 |
| frickler | iiuc openstack-cli-server uses the authentication it was given when it was started, you cannot use other cloud credentials with it, I had a similar issue in my appcred patch | 15:53 |
| frickler | so you'd have to call /usr/local/bin/openstack instead | 15:53 |
| frickler | cf. https://review.opendev.org/c/openstack/devstack/+/923944/4/lib/keystone#447 | 15:54 |
| rpittau | oh wow so we need a bypass for that command in devstack | 15:54 |
| rpittau | thanks frickler | 15:55 |
| rpittau | just proposed this https://review.opendev.org/c/openstack/devstack/+/924867, if I got it right :) | 16:04 |
| JayF | it looks right to me | 16:05 |
| JayF | you might wanna do a depends-on ironic change just for science? | 16:05 |
| rpittau | yep! | 16:05 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [DNM] TEST fix for swift tempurls function https://review.opendev.org/c/openstack/ironic/+/924868 | 16:07 |
| dansmith | frickler: that's what I thought too, but note that it is passing those things through: https://github.com/openstack/devstack/blob/master/files/openstack-cli-server/openstack#L69 | 16:10 |
| rpittau | that was nice, been a while I haven't touched devstack :) | 16:10 |
| rpittau | good night! o/ | 16:10 |
| dansmith | and it's (at least trying to) set them for the new command: https://github.com/openstack/devstack/blob/master/files/openstack-cli-server/openstack-cli-server#L72 | 16:11 |
| masghar | Thanks everyone, see you tomorrow! | 16:42 |
| frickler | dansmith: yes, I saw that, but it didn't work in my use case, didn't dig deeper yet | 18:08 |
| dansmith | ack | 18:09 |
| JayF | Just got a demo of runbooks with cid; they are in very good shape, only one outstanding piece of feedback on the Ironic piece of the change, and one on the CLI piece of the change. Likely a good time for other reviewers to get a look. | 18:21 |
| cid | ^ ++ | 18:27 |
| TheJulia | cid: That sounds amazing! | 19:42 |
| cid | TheJulia: It does :D | 19:43 |
| cid | Currently testing and also awaiting as many more eyes on it | 19:44 |
| iurygregory | great job cid o/ | 19:50 |
| iurygregory | next week I will probably have some time for reviews, still catching up on downstream tasks since i was on pto last week | 19:50 |
| cid | tks | 19:51 |
| iurygregory | time to drop today, bye everyone o/ | 19:59 |
| JayF | \o | 20:01 |
| cid | I will join you. Bye people! | 20:07 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Support Automatic Lessee from instance metadata https://review.opendev.org/c/openstack/ironic/+/924887 | 22:47 |
| JayF | That+Nova side are mostly done, need to be tested a bit in devstack, but I have a couple of open questions on that Ironic change -- even if you don't have time for a full review, if you could weigh in on those it'd be helpful :) | 22:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!