Thursday, 2023-02-23

opendevreviewMerged openstack/ironic master: fix inspectwait logic  https://review.opendev.org/c/openstack/ironic/+/87265802:12
opendevreviewMerged openstack/ironic master: Relaxing console pid looking  https://review.opendev.org/c/openstack/ironic/+/87126204:40
rpittaugood morning ironic! o/07:48
rpittauJayF: I'll have a look today08:00
rpittauJayF: I went ahead and proposed releases for all the cycle-with-intermediary except bifrost, since we're still fixing CI there08:15
opendevreviewRiccardo Pittau proposed openstack/bifrost master: CI fixes and workarounds  https://review.opendev.org/c/openstack/bifrost/+/87465008:19
vanouJayF: If you have any feedback/suggestion on https://review.opendev.org/c/openstack/ironic/+/872750, please share with me. I'll refine it.08:55
opendevreviewEbbex proposed openstack/bifrost master: Finally fix jinja[spacing]  https://review.opendev.org/c/openstack/bifrost/+/87263409:19
opendevreviewEbbex proposed openstack/bifrost master: Fix no-free-form linter warnings  https://review.opendev.org/c/openstack/bifrost/+/87485209:19
opendevreviewEbbex proposed openstack/bifrost master: Fix yaml[octal-values] linter warnings  https://review.opendev.org/c/openstack/bifrost/+/87485309:19
opendevreviewEbbex proposed openstack/bifrost master: Fix key-order[task] linter warnings  https://review.opendev.org/c/openstack/bifrost/+/87485409:19
opendevreviewEbbex proposed openstack/bifrost master: Fix schema[meta] linter warnings  https://review.opendev.org/c/openstack/bifrost/+/87485509:19
opendevreviewEbbex proposed openstack/bifrost master: Fix schema[vars] linter warning  https://review.opendev.org/c/openstack/bifrost/+/87485609:19
opendevreviewEbbex proposed openstack/bifrost master: Simplify set_fact for ssh_public_key  https://review.opendev.org/c/openstack/bifrost/+/87485709:19
rpittauebbex: CI in bifrost is not fixed yet09:28
opendevreviewMerged openstack/ironic-inspector master: Use UTC for the timezone in functional tests  https://review.opendev.org/c/openstack/ironic-inspector/+/87466110:09
opendevreviewMark Goddard proposed openstack/networking-generic-switch master: Add ngs-stress test script  https://review.opendev.org/c/openstack/networking-generic-switch/+/87478911:07
iurygregorymorning Ironic11:13
opendevreviewMerged openstack/ironic master: Set lockutils default logging  https://review.opendev.org/c/openstack/ironic/+/87260811:23
opendevreviewIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: DNM - Testing CI for Antelope Release  https://review.opendev.org/c/openstack/networking-baremetal/+/87491311:50
iurygregoryjust a heads-up I've abandoned some ngs patches that were very old (2017/2018). 2019 till 2021 I'm  also planing to abandon if there is no updates (but not going to do this today)12:13
iurygregoryrpittau, I was planning to push the release patches today, but you were faster XD 12:23
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic-python-agent-builder master: DNM - Testing Antelope  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/87491812:34
opendevreviewMark Goddard proposed openstack/networking-generic-switch master: Support batching up commands  https://review.opendev.org/c/openstack/networking-generic-switch/+/74328312:36
opendevreviewMark Goddard proposed openstack/networking-generic-switch master: Support batching up commands  https://review.opendev.org/c/openstack/networking-generic-switch/+/74328312:44
kubajjTheJulia, JayF: could I also ask you some questions for the evaluation part of my dissertation (I am still finalising them with my supervisor, might have them next week)12:57
opendevreviewRiccardo Pittau proposed openstack/ironic master: [WIP] [PoC] A metal3 CI job  https://review.opendev.org/c/openstack/ironic/+/86387313:29
opendevreviewRiccardo Pittau proposed openstack/bifrost master: CI fixes and workarounds  https://review.opendev.org/c/openstack/bifrost/+/87465013:51
iurygregorynetworking-baremetal CI is broken .-.14:08
iurygregoryhttps://zuul.opendev.org/t/openstack/build/0353dc4b15844933a52728484d2bcfdc14:08
iurygregory"/bin/bash: coredumpctl: command not found"14:23
iurygregoryI'm wondering if something changed in ubuntu...14:23
rpittaumaybe missing command in jammy ?14:23
iurygregoryyeah14:23
iurygregorytrying to double check that14:23
*** dking is now known as Guest566114:45
*** Guest5661 is now known as dking15:00
zigoWhat's the ipa-trusted-cert.pem for?15:04
* zigo is trying to make sense of the mess in ironic-python-agent-builder ...15:05
dtantsurzigo: IPA makes HTTP(s) requests against Ironic, this can be used to validate them IIRC15:10
TheJuliagood morning15:12
dtantsurmorning TheJulia 15:12
TheJuliaI'm taking a low key day working on some presentation stuff unless the world expldoes15:13
TheJuliakubajj: sure!15:13
TheJulia... that test is running a bunch of jobs15:16
TheJuliait looks like dhcp might have failed15:16
TheJuliafailed to work which is a known neutron issue right now15:16
zigodtantsur: Ok, so that would be an internal PKI to Ironic?15:16
dtantsurzigo: likely yes15:16
zigook15:16
TheJuliaI believe it is so a end user certificate can be truested which they have configured on their APIs which may not be a signed CA15:17
zigoFYI, we (at infomaniak) are starting to build an Ironic env, so likely, I'll be doing some Debian work to clean-up the Ironic package and make them prod-ready forus.15:17
zigoI probably will have more questions as we go...15:17
TheJuliazigo: cool cool15:17
zigoThat's a good thing for the state of Ironic in Debian... :)15:18
TheJuliaexcellent15:22
iurygregorygood morning TheJulia 15:22
dkingDoes anybody know about using hardware-detect inspection? It seems that I previously had to add an element for it when building the IPA image, but now I don't see the element. Maybe it's already built in these days?15:23
dtantsurzigo: great news15:23
dtantsurdking: rpittau and I know something. https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/dib/extra-hardware is the element you're looking for.15:24
JayFkubajj: the best way to contact me for that is my email; jay at jvf dot cc 15:25
dkingOh, interesting. I was looking for it here: https://docs.openstack.org/diskimage-builder/latest/elements.html I suppose it's just missing there?15:25
dtantsurdking: we don't maintain elements specific to IPA in DIB any more15:25
dtantsurhttps://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html#ironic-python-agent-ipa-extra-hardware is the docs15:26
TheJuliakubajj: same, juliaashleykreger at gmail.com15:26
kubajjJayF, TheJulia: thanks15:26
dkingOhh! Now I remember. I knew that I saw it somewhere unusual, and that my brain was telling me that I didn't have to write my own element, but I completely forgot where. Thanks dtantsur!15:29
* TheJulia suggests lots of coffeee15:30
TheJuliaspeaking of!15:30
dtantsurtoo late for coffee here :)15:30
TheJuliaI need to put this to use! https://usercontent.irccloud-cdn.com/file/ckr6h80X/IMG_1269.JPG15:31
mgoddardHi, currently bumping into https://storyboard.openstack.org/#!/story/2010537. Looks like the fix has been released for all releases back to xena. Can I propose a wallaby release?15:31
TheJuliaAnd no, I don't do that particular exercise, although some stackers have encouraged me to take it up. Ugh long covid effects15:32
opendevreviewMerged openstack/networking-generic-switch master: Add support for Cisco Nexus devices (NX-OS)  https://review.opendev.org/c/openstack/networking-generic-switch/+/86848115:32
TheJuliamgoddard: sure!15:32
TheJulia... I thought we did backport that to wallaby though15:32
* TheJulia might be loosing her mind15:32
rpittaubye everyone, see you on monday! o/15:33
dtantsurTheJulia: this cup rocks :)15:33
mgoddardTheJulia:it's backported but not yet released15:34
JayFIs wallaby EM? 15:36
JayFWallaby is EM.15:36
JayFIt gets no more releases mgoddard, sorry 15:36
JayFthat is openstack policy; not ironic policy15:36
JayFyou'll have to pull it from git15:36
JayFhttps://releases.openstack.org/15:36
dtantsurJayF: that does suck though that we have a significant regression in the stable release that we cannot fix15:37
dtantsurI wonder if we can request an exception15:37
mgoddardJayF: ah, of course15:37
JayFI honestly don't know, lets ask in -releases 15:37
JayFer, -release15:37
dtantsuryup15:37
TheJuliaI think we've managed to pull it off at least once in the past, but it has been a long time15:39
JayFI'm asking; worse thing that can happen is a no; and we're in the same spot we're in now.15:40
TheJulia++15:43
dtantsurbtw we should probably release sushy yoga and zed15:44
TheJulia++15:45
dtantsurI guess it was a no, sorry, mgoddard 15:50
JayFdtantsur: part of me wonders if a non-wallaby-line sushy could be combined with wallaby ironic for mgoddard's case15:52
dtantsurJayF: we have done similar things in metal3. Just don't cross the major version boundary.15:52
dtantsursushy is reasonable semver-following15:53
TheJuliaI suspect it would...15:59
JayFsomeone in -release suggested we should consider sushy being independent15:59
JayFrather than cycle15:59
JayFwhich makes sense to me on the surface, but I haven't thought about it in depth15:59
TheJuliait.. does make more sense to be independent16:01
TheJulia(and maybe that case long ago in the past was independent or something16:01
dtantsurrebooting for updates, brb16:17
TheJuliaiurygregory: I think networking-baremetal is a test issue16:22
TheJuliaiurygregory: I think it is fine, otherwise16:22
TheJuliaIt looks like on tear down we're expecting to delete something but they are in the gray area of items which nova may delete, I think16:23
* dtantsur is back16:23
TheJuliaFeb 23 14:05:06.327929 np0033234847 neutron-server[84968]: WARNING neutron.pecan_wsgi.controllers.root [None req-a829a11c-bfa8-4654-8f98-4cc60a57673c admin admin] No controller found for: security-groups - returning response code 404: pecan.routing.PecanNotFound16:28
TheJuliaiurygregory: I think we're fine to release, but we need to figure out what changed. I bet this is fallout from the default plugin changes, oddly enough only the single tenant case hits it16:29
iurygregoryTheJulia, oh ack16:32
TheJuliathe underlying test passes, it is the cleanup from the test which detonates16:33
TheJulia*why* only that test, I have zero idea16:33
TheJuliaand it is because somewhere port security is getting turned on16:36
TheJuliaand the plugin is not by default.16:36
opendevreviewJulia Kreger proposed openstack/networking-baremetal master: Add port-security to devstack config  https://review.opendev.org/c/openstack/networking-baremetal/+/87493916:39
opendevreviewJulia Kreger proposed openstack/ironic master: add default conductor group capability  https://review.opendev.org/c/openstack/ironic/+/85570516:48
opendevreviewJulia Kreger proposed openstack/networking-baremetal master: [CI] Explicitly disable port security  https://review.opendev.org/c/openstack/networking-baremetal/+/87493917:11
TheJuliaokay, that should work17:11
sschmittThis is more of a meta question, but whats the logical/feature boundary between ngs and networking-baremetal. It seems like they both have switch configuration abilities now. If I wanted to add some functionality in this area which one would make sense?17:13
dtantsurkubajj: you haven't added API reference for inventory API, have you?17:16
dtantsurhmm, you have actually, why am I not seeing it..17:17
kubajjdtantsur: isn't it this? https://review.opendev.org/c/openstack/ironic/+/866876/12/api-ref/source/baremetal-api-v1-nodes-inventory.inc17:18
dtantsurkubajj: I see the problem, this file is not included anywhere. I'll fix it now, no worries17:18
* dtantsur working on adding the inventory API to gophercloud17:19
dtantsurkubajj: we definitely haven't updated the ironic client and openstacksdk, right?17:19
dtantsurfolks, could someone take a look at https://docs.openstack.org/api-ref/baremetal/ please? Am I the only who only sees very few headings?17:25
* dtantsur raises it to infra17:28
JayFdtantsur: feel free to @ me on gophercloud PRs, I'm generally interested, it's used downstream here, and I know some go17:29
dtantsurJayF: it will take me some time, but will do17:29
JayFdtantsur: you aren't the only one, it is on a very long list for me to look at it, if you tackle it you'd be doing me a favor17:29
kubajjdtantsur: I don't think we did (I don't really know what those are though)17:30
dtantsurheh, okay17:30
dtantsurJayF: fun fact, we have exactly one heading. I wonder what makes it special17:31
JayFWhich one works?17:31
dtantsurDeleting history entries for a node17:31
dtantsuralso the only one without any actual API reference. interesting.17:31
* dtantsur turns to firefox dev tools17:32
dtantsur.docs-body section h1 {17:33
dtantsur  display: none;17:33
dtantsur}17:33
dtantsurWTF\17:33
dtantsurif I disable this rule, the sections are back. W.T.F.17:33
JayFwhat css file is that in?17:33
dtantsurJayF: comes from combined.css, which is probably something rendered17:34
JayFwe should see if that occurs in other api refs17:34
dtantsurJayF: compute too17:34
JayFI wonder if it's a theme decision that our API ref dislikes for some reason17:34
JayFhttps://github.com/openstack/openstackdocstheme/blob/master/openstackdocstheme/theme/openstackdocs/static/css/combined.css#L92917:35
JayFdtantsur: ^17:35
dtantsurhttps://github.com/openstack/openstackdocstheme/commit/f81f3344076a09482545534e014318d7e961f825 has been around for a while..17:36
dtantsurJayF: let's move to openstack-infra? it affects not only us17:36
JayFI'm happy to drop from the troubleshooting17:36
JayFand go finish my client support for shards :D17:37
dtantsurdropping not allowed ;)17:37
JayFwhich probably should include gophercloud at some point if I'm honest :|17:37
opendevreviewDmitry Tantsur proposed openstack/ironic master: Add missing include for inventory API reference  https://review.opendev.org/c/openstack/ironic/+/87494617:39
dtantsurI won't be able to finish the workaround for api-ref today, sorry17:49
dtantsursee you on Monday o/17:49
jjyHi openstack-ironic community? Have a quick question on ironic image service. My user OS images is hosted with HTTP server that requires client basic authentication. Does the ironic image service supports HTTP basic authentication with the HTTP server. I had a quick look at the code. Looks like it only validate the server certificate.19:29
TheJuliajjy: well.... That is not a requirement I think anyone has ever articulated19:32
TheJuliajjy: you could try using a https://user:pass@url/file ... which is awful and might reveal an issue in API responses (so if you see that, let us know19:32
TheJuliaewwwwww19:33
TheJuliairccloud... bad irccloud19:33
JayFTheJulia: :| 19:33
JayFI'm pretty sure we block out image_url in api responses19:33
JayFI hope we do :)19:33
TheJuliahttps://paste.openstack.org/show/b3rYfSR2SfHfeUEMH23x/19:33
TheJuliajjy: ^^19:34
JayFit came across well here fwiw TheJulia 19:34
JayFyour "try using ... " message, originally19:34
TheJulianice19:34
TheJuliawell, irccloud itself changed it down to https://url/file and then added at thing about use of which username/password19:35
TheJuliaI guess as a helper...19:35
* TheJulia looks at the speed in which unit tests run and thinks "this desktop's days are numbered... substantially... because KSP2"19:42
opendevreviewJulia Kreger proposed openstack/ironic master: Get conductor metric data  https://review.opendev.org/c/openstack/ironic/+/86544719:45
TheJuliadtantsur: stevebaker[m] ^^ updated19:45
JayFif you get one, let me help pick out the parts, I <3 building custom desktops lol19:45
JayFor if there's a microcenter near SCALE, I should go on a pilgrimage there lol19:45
jjyThe background is we host a ironic service. The image url will be feeded by another component I believe. Probably I need to make our HTTP image server accept the url like https://user:pass@url/file?19:45
JayFjjy: https://user:pass@url/file is a standard url syntax for basic auth19:46
JayFjjy: So that should work anywhere a URL would work, theoretically19:46
TheJuliaiurygregory: I'll do the follow-up sometime next week. I would like to chat about the naming stuff since I think the comment could use a little more verbosity :)19:46
TheJuliayeah, the underlying HTTP client understands what to do with it19:46
TheJuliaat least it should19:47
* TheJulia tries19:47
jjyYep. I hope that. The image server is not developed by me. 19:48
TheJuliawell, python-requests didn't freak out when I asked it to use basic auth with my power meter19:49
jjyThe architecture is the ironic cache the image from the external image server and then IPA download the image from ironic and write to the local disk. Right?19:50
TheJulia>>> print(r.request.headers)19:50
TheJulia{'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Basic dXNlcjpwYXNz'}19:50
TheJuliait groks it19:50
TheJuliajjy: it *can*, there is an option for that19:50
TheJuliaif your using the direct deploy interface, it can also just go directly to the original image url supplied to ironic19:51
* TheJulia makes sad face at only 2.55kW being generated right now19:51
TheJulia(I have an iot webserver thing that montiors my power19:52
TheJulia)19:52
jjyThe option is in ironic or IPA? Could you point me to the option. Currently I am investigating where the authentication takes place. I was thinking the authentication is between ironic-conductor and the external HTTP image server. 19:54
TheJuliaone moment, I'll get it19:54
TheJuliain ironic.conf, you want the [agent] section option image_download_source19:56
TheJuliait can be set by conductor, and I believe the option you want is "local"19:56
jjyI see. We do not use that option. Do you suggest to add authentication between IPA and ironic-conductor? We do not have authentication between IPA and ironic-contuctor or ironic-inspector 19:58
TheJuliafor the image transfer?19:58
TheJuliaor in general?19:58
jjyin general19:59
TheJuliaahh19:59
TheJuliainspector has no concept of it, but it is only for reporting data in to discover/model machines19:59
TheJuliahttps://docs.openstack.org/ironic/latest/admin/agent-token.html is how the IPA agent is basically authenticated beyond certificates which can also be used19:59
jjyinspector only does out of band, no interaction with IPA?20:00
TheJuliaipa always tries to call inspector if so configured20:00
TheJuliabut it is not required20:00
TheJuliaif you want to just always authenticate to a remote url for the image file download, use the configuration option I noted with the value "http"20:01
TheJuliafwiw, the agent will do checksum verification as well20:02
iurygregoryTheJulia, sure20:02
jjy"if you want to just always authenticate to a remote url for the image file download, use the configuration option I noted with the value "http"20:45
jjyWas in lunch.  That requires to add image server CA into IPA ramdisk OS/20:46
jjyI plan to add authenticate between ironic-conductor and external HTTP image server. But it will not no authentication when IPA pull the image from irnoic-conductor. 20:53
JayFIf you're building your own ramdisk; including a custom CI is fairly trivial21:16
JayFI think we support it even if you don't build your own ramdisk, but I don't know how to configure that off the top of my head21:17
JayF**custom CA21:17
JayFhttps://github.com/openstack/ironic-python-agent-builder/tree/master/dib/ironic-python-agent-tls look at DIB_IPA_CA_FILE21:17
jjyWe build our own ramdisk, but we do not plant add the cert bundles during the build time. https://github.com/openstack/ironic-python-agent-builder/tree/master/dib/ironic-python-agent-tls The link is to add cert bundles while building.   21:23
TheJuliajjy: if you just define the credentials for basic auth in the url, the conductor will be able to use the credentials21:26
jjyThat is the authentication between conductor and external image server. Conductor cache the image? How about authentication between IPA and ironic-conductor, when IPA pull the image from ironic and write to the machine disk ? 21:40
JayFhttps://docs.openstack.org/kolla-ansible/latest/reference/deployment-and-bootstrapping/bifrost.html this looks pretty sweet21:41
JayFjjy: serach for image_download_source https://docs.openstack.org/ironic/latest/configuration/config.html21:42
JayFjjy: that behavior is configurable :D21:42
JayFjjy: by default, in master branch, it looks like IPA would fetch directly from the URL you provide21:42
jjyhmm we do not set up the image_download_source. The agent config is "[agent]22:00
jjydeploy_logs_collect = always22:00
jjydeploy_logs_local_path = /shared/log/ironic/deploy22:00
jjymax_command_attempts = 3022:00
JayFit defaults, per that doc, to `http`22:00
JayFwhich is the behavior I think you want, but I'm unsure22:00
jjyBut I did see the ironic (not IPA) was trying to validate the external HTTP image server certificate.22:02
JayFinteresting22:02
JayF>     IPA ramdisk retrieves instance image from HTTP service served at conductor nodes.22:02
JayFI'm not sure I can tell what that (http) means22:03
JayFvs local22:03
JayF>     Same as “http”, but HTTP images are also cached locally, converted and served from the conductor22:03
JayFhttp service /served at conductor nodes/ implies it's doing some caching, too22:03
JayFI don't have time to read the code and figure out exactly what this behavior is; I'll try to dedicate some time to you tomorrow afternoon if you'll be around jjy ?22:03
jjyThe validate happens there https://github.com/openstack/ironic/blob/master/ironic/common/image_service.py#L10022:04
jjyYes I will be around.22:04
JayFI'm in pacific time, I'd say probably between 11a-noonish I should become available22:05
JayFwe'll try and figure it out for you22:05
JayFBTW; can you let me know what your use case is? Just so we know who our users are?22:05
jjyWe deploy the ironic for Baremetal provisioning using the K8S operator. Check this https://github.com/metal3-io/ironic-image 22:07
JayFI'm very familiar with metal3.io :D 22:08
JayFare you a developer on that, or just a user?22:08
JayF"just" comes off more pajorative than I mean, just trying to determine if you're implementing new stuff for metal3 or trying to get your things working :)22:09
* JayF notes he's @jay.faulkner in the slack for metal322:09
jjyTo clarify the question, I want to confirm where the authentication takes place between ironic and external HTTP image server. We want to secure the communiation between ironic and external image server.22:10
JayFack; that makes sense. I believe as Julia suggested, an https url + username/password should work22:11
JayFbut we should nail down in what cases that connection would come from conductor, and what it comes from IPA22:11
JayFI think it's almost always going to be conductor, but we can figure it out for sure tomorrow22:12
JayFit's also possible one of the brilliant folks in the other time zones will just know the answer :D 22:12
jjyIf the connection come from dconductor, we would like to add the authentication between IPA and conductor.22:12
jjyCurrently I am a user of metal3:) Sent out my first trial pull quest to the upstream.22:13
JayFI'm fairly certain we have push-button support for IPA<>Conductor secure comms22:45
JayFwe use agent tokens as a form of auththentication to authenticate it's a valid agent22:45
JayFand use https certs to ensure data is secure in transit22:45
TheJuliaYes, it is a thing! I just remember the configuration knobs22:46
JayFis it okay that this exists? https://www.npmjs.com/package/bifrost-docker23:05
JayFit's 8 years old.23:05
JayF#3 on google for "bifrost container install" :| 23:06
TheJuliaSign, nothing we found when we did the name search. Likely perfectly fine albeit confusing23:45
JayF...does centos stream 9 just not operate like a normal operating system now?23:56
JayFthe cloud image ships with no enabled repos, and it appears you need to do a subscription manager dance to get them working23:56
TheJuliadunno.... but that is... concerning23:58
JayFwatch the last 5 minutes of my stream vod23:58
JayFI just know better than to try to use centos stream in the future lol23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!