| *** bdodd has quit IRC | 00:08 | |
| *** ociuhandu has joined #openstack-ironic | 00:24 | |
| *** ociuhandu has quit IRC | 00:29 | |
| *** k_mouza has joined #openstack-ironic | 00:40 | |
| *** k_mouza has quit IRC | 00:42 | |
| openstackgerrit | Merged openstack/ironic master: Add `get_node_network_data` to non-Neutron NetworkInterface https://review.opendev.org/688266 | 00:49 |
|---|---|---|
| openstackgerrit | Mohammed Naser proposed openstack/ironic-ui master: api: drop project_id from kwargs in client https://review.opendev.org/735028 | 01:22 |
| mnaser | dtantsur|afk: ^ fyi i saw you last worked on that same bit :) | 01:23 |
| mnaser | will need backporting to stable/ussuri too | 01:23 |
| *** Qianbiao has joined #openstack-ironic | 02:04 | |
| *** tzumainn has quit IRC | 02:07 | |
| mnaser | hmm -- anyone have experience with supermicro hardware reporting interfaces as disabled via redfish | 02:16 |
| mnaser | resulting in ironic obviously not adding them | 02:16 |
| mnaser | b/c of https://github.com/openstack/ironic/blob/72f36719a1d9579b8541f2c4a7eff1c794cc4ccd/ironic/drivers/modules/redfish/inspect.py#L227-L230 | 02:17 |
| mnaser | i'd like to stick to redfish introspection but i guess if i have to use ironic-inspector i could | 02:19 |
| *** k_mouza has joined #openstack-ironic | 02:31 | |
| *** k_mouza has quit IRC | 02:36 | |
| *** ricolin has joined #openstack-ironic | 02:39 | |
| TheJulia | mnaser: if you can curl one of the interfaces to show us what your seeing, it might help. I've got a supermicro machine with the ?sum? license that I could compare against | 02:47 |
| TheJulia | mnaser: from what I've seen in the docs, it looks like BMC visibility of nics is limited | 02:47 |
| mnaser | https://www.irccloud.com/pastebin/lLuAaxMc/ | 02:47 |
| mnaser | TheJulia: ^ but yeah, i think redfish as inspection needs a bit more time to come close to what inspector offers natively | 02:48 |
| TheJulia | well, it still has to contend with vast differences in implementations | 02:48 |
| * TheJulia wonders why it shows disabled | 02:49 | |
| mnaser | TheJulia: yeah.. i tried to do some google-ing if it was something other people ran into but not much info about it | 02:49 |
| * TheJulia seems to remember NICs were not part of the original spec | 02:50 | |
| * mnaser has a doc patch incoming.. | 02:51 | |
| mnaser | except i have to get off this silly vpn to push it | 02:51 |
| mnaser | https://github.com/openstack/ironic/blob/2637e59a849c52220ba0b05a4caac75c98e47e78/doc/source/admin/inspection.rst s/endpoint-override/endpoint_override/ | 02:52 |
| TheJulia | That entire "Status" section means the status of the component resource, so the BMC views it as disabled. I _wonder_ if it is one of the things that their integrated datacenter stuff license changes | 02:53 |
| mnaser | TheJulia: i _believe_ that this is an unlicensed box.. i think | 02:53 |
| TheJulia | mnaser: if completely unlicensed, you wouldn't have been able to curl the Redfish portion of the BMC | 02:54 |
| mnaser | oh | 02:54 |
| TheJulia | but there are 3 different licenses available at different price points for their BMCs | 02:54 |
| TheJulia | One of which I didn't find any info on at all :\ | 02:54 |
| mnaser | i'm not super intimately familiar with those bits yet | 02:55 |
| TheJulia | if you can curl the virtual media, that is a sign that it is got the 200+ dollar license | 02:55 |
| TheJulia | if you can't, you likely have the 30 dollar license | 02:56 |
| TheJulia | for "oob firmware upgrades" | 02:56 |
| mnaser | i'm starting to lean towards "use as much generic interfaces" | 02:56 |
| TheJulia | lol | 02:56 |
| mnaser | while maybe not as 'feature-ful' ... it'll always work | 02:57 |
| mnaser | inspector will always work, ipmitool will always work, pxe will always work, etc | 02:57 |
| TheJulia | not always. The next big headache will be not being able to send raw flags to BMCs over ipmi, but who knows how long that will be before people start to encounter that | 02:58 |
| mnaser | hopefully long enough for some other standard to emerge | 02:59 |
| mnaser | :P | 02:59 |
| TheJulia | well, it has... but... umm.... Yeeeaaahhhh | 02:59 |
| TheJulia | Perhaps that is a topic reserved for the next time we all have a chance to catch up in person | 02:59 |
| mnaser | soon, hopefully | 03:00 |
| mnaser | i'll have had a lot more ironic first-hand experience by then :) | 03:00 |
| *** Goneri has quit IRC | 03:00 | |
| TheJulia | \o/ | 03:01 |
| TheJulia | Anyway, I'm going to go back to not focusing on the computer | 03:01 |
| mnaser | yes! go do that, it's late :) | 03:01 |
| mnaser | take good care of yourself! | 03:01 |
| *** cdearborn has quit IRC | 03:03 | |
| *** rcernin has quit IRC | 03:08 | |
| Qianbiao | I thought redfish is opensource and free ... | 03:21 |
| Qianbiao | It's so expensive.. | 03:22 |
| TheJulia | "value add" :( | 03:48 |
| *** rcernin has joined #openstack-ironic | 03:54 | |
| *** rcernin has quit IRC | 03:55 | |
| *** rcernin has joined #openstack-ironic | 03:55 | |
| *** jdandrea has quit IRC | 04:07 | |
| *** yaawang has quit IRC | 04:11 | |
| *** yaawang has joined #openstack-ironic | 04:11 | |
| *** Qianbiao has quit IRC | 04:21 | |
| *** bdodd has joined #openstack-ironic | 05:08 | |
| *** gyee has quit IRC | 05:11 | |
| *** dsneddon has quit IRC | 05:24 | |
| *** arne_wiebalck has quit IRC | 05:38 | |
| *** arne_wiebalck has joined #openstack-ironic | 05:38 | |
| *** k_mouza has joined #openstack-ironic | 05:47 | |
| *** k_mouza has quit IRC | 05:52 | |
| *** yaawang has quit IRC | 05:55 | |
| *** yaawang has joined #openstack-ironic | 05:56 | |
| arne_wiebalck | Good morning, ironic! | 06:29 |
| *** Qianbiao has joined #openstack-ironic | 06:34 | |
| *** dustinc has quit IRC | 06:35 | |
| *** kaiokmo has joined #openstack-ironic | 06:51 | |
| *** yaawang has quit IRC | 06:55 | |
| *** yaawang has joined #openstack-ironic | 06:56 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic master: Switch Ironic to openstacksdk for Neutron https://review.opendev.org/734873 | 06:58 |
| *** Qianbiao has quit IRC | 07:07 | |
| *** yaawang has quit IRC | 07:08 | |
| *** yaawang has joined #openstack-ironic | 07:09 | |
| *** rpittau|afk is now known as rpittau | 07:18 | |
| rpittau | good morning ironic! o/ | 07:19 |
| *** jawad_axd has joined #openstack-ironic | 07:21 | |
| iurygregory | good morning arne_wiebalck rpittau and Ironic o/ | 07:22 |
| *** sshnaidm|afk is now known as sshnaidm | 07:23 | |
| rpittau | hey iurygregory :) | 07:36 |
| *** ociuhandu has joined #openstack-ironic | 07:41 | |
| *** Qianbiao has joined #openstack-ironic | 07:41 | |
| *** k_mouza has joined #openstack-ironic | 07:48 | |
| *** ociuhandu has quit IRC | 07:50 | |
| *** ociuhandu has joined #openstack-ironic | 07:52 | |
| *** k_mouza has quit IRC | 07:53 | |
| iurygregory | today gerrit is complaning more than normal to me -.-' Code Review - Error Server Unavailable 0 | 07:54 |
| *** rcernin has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack/ironic-specs master: Follow-up on iso boot spec https://review.opendev.org/728911 | 08:01 |
| *** yaawang has quit IRC | 08:11 | |
| *** yaawang has joined #openstack-ironic | 08:11 | |
| *** lucasagomes has joined #openstack-ironic | 08:14 | |
| iurygregory | rpittau, if you have a chance can you https://review.opendev.org/#/c/734585/ bifrost stable/queens | 08:15 |
| patchbot | patch 734585 - bifrost (stable/queens) - Fix multiple CI failures - 2 patch sets | 08:15 |
| iurygregory | a lot of files to look at | 08:15 |
| rpittau | a lot of failures on non-voting jobs also :) | 08:16 |
| iurygregory | well non-voting in stable/rocky is like that | 08:17 |
| rpittau | isn't that queens ? | 08:17 |
| iurygregory | yup | 08:17 |
| openstackgerrit | Merged openstack/ironic-specs master: Clean up specs https://review.opendev.org/733411 | 08:20 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Fix requirements check https://review.opendev.org/735076 | 08:23 |
| *** Lucas_Gray has joined #openstack-ironic | 08:24 | |
| openstackgerrit | Merged openstack/sushy master: Add RAIDType properties to storage resources https://review.opendev.org/731308 | 08:30 |
| arne_wiebalck | rpittau: I'm interested in the redfish interop profile as well and would join you on this, if that is ok with you? | 08:31 |
| *** ociuhandu has quit IRC | 08:31 | |
| rpittau | arne_wiebalck: I proposed myself as I saw noone else did and I think it's quite important, all considered. I'd be glad to work with you on that :) | 08:32 |
| arne_wiebalck | rpittau: I'm trying to get some of our hardware/procurement colleagues you met at the mid-cycle on board as well. | 08:32 |
| rpittau | nice :) | 08:32 |
| arne_wiebalck | rpittau: It's a long standing issue that we cannot describe what we need in terms of Redfish which results in asking for anything :) | 08:33 |
| *** Lucas_Gray has quit IRC | 08:34 | |
| *** ociuhandu has joined #openstack-ironic | 08:34 | |
| arne_wiebalck | *... for nothing. | 08:34 |
| rpittau | yeah, I understand the issue, and the frustration | 08:34 |
| *** ociuhandu has quit IRC | 08:36 | |
| arne_wiebalck | rpittau: let's see if we can mitigate his :) | 08:39 |
| arne_wiebalck | TheJulia: can you add rpittau and arne_wiebalck to the Redfish interop profile in the Victoria priorities? I left a comment on the review as well. | 08:40 |
| *** ociuhandu has joined #openstack-ironic | 08:42 | |
| openstackgerrit | Merged openstack/bifrost stable/queens: Fix multiple CI failures https://review.opendev.org/734585 | 08:43 |
| *** ociuhandu has quit IRC | 08:46 | |
| *** ociuhandu has joined #openstack-ironic | 08:47 | |
| *** uzumaki has joined #openstack-ironic | 08:48 | |
| *** rcernin has joined #openstack-ironic | 08:50 | |
| *** ociuhandu has quit IRC | 08:53 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic master: Add virtualbmc to lower-constraints https://review.opendev.org/735085 | 09:03 |
| Qianbiao | hello ironic. | 09:05 |
| Qianbiao | hello rpittau, do u got a minute giving a +2 and workflow for https://review.opendev.org/#/c/728123/ | 09:06 |
| patchbot | patch 728123 - ironic - Feature: Add raid configuration support for ibmc d... - 13 patch sets | 09:06 |
| Qianbiao | it should be ready for merging | 09:07 |
| rpittau | Qianbiao: I'll give it a look later, I'm a bit busy at the moment | 09:08 |
| rpittau | I promise :) | 09:08 |
| Qianbiao | sure, thanks in advance. | 09:08 |
| Qianbiao | :) | 09:08 |
| uzumaki | hey Qianbiao o/ | 09:10 |
| Qianbiao | morning uzumaki | 09:10 |
| Qianbiao | morning arne_wiebalck | 09:11 |
| uzumaki | what's up? | 09:11 |
| Qianbiao | it has been a while since last see you arne_wiebalck | 09:11 |
| arne_wiebalck | Hey Qianbiao o/ | 09:11 |
| Qianbiao | aha? | 09:11 |
| arne_wiebalck | I haven't moved :) | 09:11 |
| openstackgerrit | Merged openstack/bifrost master: bifrost-create-vm-nodes: generate a normal nodes.json https://review.opendev.org/733783 | 09:12 |
| openstackgerrit | Merged openstack/bifrost master: Add an option to use pre-built DIB images https://review.opendev.org/734812 | 09:12 |
| Qianbiao | <uzumaki> not a native english speaker, not sure what's up means in this scenes. :) | 09:12 |
| Qianbiao | uzumaki btw, i have test metal3 againest with ibmc hardware. If u need help, i may give some suggestion. | 09:14 |
| rpittau | arne_wiebalck, TheJulia, any other core, the requirements check in ironic is broken, this should fix it -> https://review.opendev.org/735076 | 09:14 |
| patchbot | patch 735076 - ironic - Fix requirements check - 1 patch set | 09:14 |
| *** rpittau is now known as rpittau|bbl | 09:15 | |
| rpittau|bbl | meeting + lunch break, ttyl | 09:15 |
| *** rcernin has quit IRC | 09:17 | |
| *** ociuhandu has joined #openstack-ironic | 09:20 | |
| openstackgerrit | Merged openstack/ironic-inspector master: Use unittest.mock instead of mock https://review.opendev.org/734351 | 09:21 |
| *** uzumaki has quit IRC | 09:26 | |
| *** ociuhandu has quit IRC | 09:28 | |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: Convert with_items to loop https://review.opendev.org/733847 | 09:29 |
| *** uzumaki has joined #openstack-ironic | 09:30 | |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: Convert with_items to loop https://review.opendev.org/733847 | 09:32 |
| *** ociuhandu has joined #openstack-ironic | 09:32 | |
| *** k_mouza has joined #openstack-ironic | 09:50 | |
| *** k_mouza has quit IRC | 09:55 | |
| openstackgerrit | Hervé Beraud proposed openstack/ironic-lib master: Use unittest.mock instead of mock https://review.opendev.org/734352 | 09:57 |
| uzumaki | arne_wiebalck, you haven't moved in space, but surely in time ;) | 10:11 |
| uzumaki | Qianbiao, oh, that's good news! I just realized because of misconfiguration in the ironic.conf, the conductor never started, and that's why my metal3 provisioning is stuck, rebuilding the image now to fix it | 10:11 |
| *** k_mouza has joined #openstack-ironic | 10:19 | |
| *** eliaswimmer_ has quit IRC | 10:20 | |
| *** ociuhandu has quit IRC | 10:25 | |
| *** eliaswimmer_ has joined #openstack-ironic | 10:29 | |
| openstackgerrit | Verification of a change to openstack/ironic failed: Use the new extension call for getting partition UUIDs https://review.opendev.org/731745 | 10:29 |
| uzumaki | if I enable the BIOS interface idrac-wsman, for instance, in the conf, and enable hardware types additional to idrac, should I see issues? | 10:30 |
| uzumaki | I mean, shouldn't the conductor only associate the enabled BIOS interface with the hardware type it is for? | 10:31 |
| Qianbiao | uzumaki ok. I use a standalone ironic env not image from official. | 10:34 |
| uzumaki | what ironic.conf do you have in that? specifically in the [DEFAULT] section? | 10:35 |
| Qianbiao | I use a full list of ironic.conf. but you can refer to https://github.com/metal3-io/ironic-image/blob/master/ironic.conf | 10:36 |
| uzumaki | I just want to see the [DEFAULT] section of ironic.conf that you setup in your environment, I'm just trying to see if enabling many hardware types at the same time causes issues if i only enabled 1 BIOS interface | 10:38 |
| iurygregory | you need to enable the interfaces that the hardware supports | 10:39 |
| Qianbiao | I only enable ibmc | 10:40 |
| iurygregory | uzumaki, I would say to only enable the driver you want to test | 10:40 |
| *** ociuhandu has joined #openstack-ironic | 10:42 | |
| *** derekh has joined #openstack-ironic | 10:43 | |
| uzumaki | that's what I"m going to do now! | 10:44 |
| *** alexmcleod has joined #openstack-ironic | 10:45 | |
| *** ricolin has quit IRC | 10:47 | |
| iurygregory | good luck uzumaki =) | 10:48 |
| *** ociuhandu has quit IRC | 10:52 | |
| *** ociuhandu has joined #openstack-ironic | 10:53 | |
| *** k_mouza has quit IRC | 10:55 | |
| *** ociuhandu has quit IRC | 10:58 | |
| *** killermoehre has joined #openstack-ironic | 10:59 | |
| uzumaki | thanks! I'll be needing that xD iurygregory | 11:08 |
| *** yaawang has quit IRC | 11:08 | |
| iurygregory | heheh | 11:09 |
| *** yaawang has joined #openstack-ironic | 11:09 | |
| *** threestrands has quit IRC | 11:10 | |
| killermoehre | hi. I build my own IPA distribution (just a fedora as base and adding ironic-python-agent from pypi) (https://github.com/killermoehre/tinyipa). the IPA in general works just fine, just the download of the image out of glance is super slow with only 1MB/s. | 11:11 |
| killermoehre | when the image is finally downloaded everything works as expected. I can't reproduce the issue with a base python interpreter downloading the image by hand | 11:12 |
| *** k_mouza has joined #openstack-ironic | 11:13 | |
| *** k_mouza has quit IRC | 11:23 | |
| *** k_mouza has joined #openstack-ironic | 11:24 | |
| *** ociuhandu has joined #openstack-ironic | 11:27 | |
| *** ociuhandu has quit IRC | 11:32 | |
| *** Qianbiao has quit IRC | 11:37 | |
| *** rcernin has joined #openstack-ironic | 11:38 | |
| *** rcernin has quit IRC | 11:43 | |
| uzumaki | it worked! \o/ iurygregory | 11:46 |
| iurygregory | uzumaki, \o/ | 11:47 |
| iurygregory | killermoehre, your issue seem to be related to your network setup I would say | 11:48 |
| killermoehre | iurygregory, it works very well if I do the same download at the same time from the same host by hand (curl or python requests.get()) where IPA is running | 11:50 |
| killermoehre | also I don't have the problem with IPA 2.0.X, only with 6.1.0 (other versions not yet tested) | 11:50 |
| iurygregory | hummm | 11:50 |
| iurygregory | we fixed an issue for high cpu usage not sure if it would help you | 11:52 |
| iurygregory | https://github.com/openstack/ironic-python-agent/commit/952489020ea36aced3737079fa17eff2e37b4da7 | 11:52 |
| killermoehre | well, one core is running at 140% | 11:53 |
| killermoehre | during download | 11:53 |
| *** rpittau|bbl is now known as rpittau | 11:54 | |
| rpittau | killermoehre: I suggest to try what iurygregory suggested, it seems related | 11:55 |
| rpittau | suggest-ception | 11:55 |
| iurygregory | lol | 11:55 |
| iurygregory | i liked | 11:55 |
| iurygregory | rpittau, one quick question since you are back | 11:55 |
| killermoehre | can I install from github directly via pip? | 11:55 |
| iurygregory | the permisions of all *.sh in scripts dir of bifrost need to be 755? | 11:56 |
| killermoehre | 0755 sounds general proper for shell scripts | 11:56 |
| rpittau | iurygregory: 755 should be ok to execute normally by anyone, probably 750 is better in terms of security, not super important though for this | 11:57 |
| iurygregory | install-deps.sh has 644 | 11:57 |
| rpittau | iurygregory: that depends on how the script is executed | 11:57 |
| *** rh-jelabarre has joined #openstack-ironic | 11:57 | |
| *** rh-jelabarre has quit IRC | 11:58 | |
| rpittau | if it's called directly, then it's not ok, if it's called by running 'bash' in front of it then it's fine | 11:58 |
| *** rh-jelabarre has joined #openstack-ironic | 11:58 | |
| rpittau | I usually use 640 or 750, just to prevent the execution from everyone, but 755 is generally ok | 11:59 |
| iurygregory | ack | 11:59 |
| iurygregory | using 755 seems to run test script without problems for me | 11:59 |
| rpittau | iurygregory: that's totally fine for an executable :) | 12:00 |
| *** rh-jelabarre has quit IRC | 12:02 | |
| *** rh-jelabarre has joined #openstack-ironic | 12:02 | |
| *** rh-jelabarre has quit IRC | 12:03 | |
| openstackgerrit | Verification of a change to openstack/ironic failed: devstack: increase concurrency and worker timeout for sushy-tools https://review.opendev.org/734595 | 12:04 |
| *** dougsz has joined #openstack-ironic | 12:06 | |
| *** rh-jelabarre has joined #openstack-ironic | 12:14 | |
| uzumaki | Why do I see SSL errors in the conductor logs? I'm using ironic with metal3 bare metal operator | 12:15 |
| openstackgerrit | Hervé Beraud proposed openstack/ironic master: Use unittest.mock instead of mock https://review.opendev.org/734350 | 12:16 |
| uzumaki | and when I disable TLS, I get this failure upon attempting BMC access | 12:18 |
| uzumaki | 2020-06-11 12:13:19.093 29 WARNING dracclient.wsman [req-e967852e-234b-41fe-9173-565b73f1997d - - - - -] A ConnectionError error occurred while communicating with 192.168.2.8, a | 12:18 |
| uzumaki | ttempt 1 of 3: requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) | 12:18 |
| uzumaki | I got a bunch of "DH_KEY_TOO_SMALL" errors in SSL that's why I disabled it. | 12:20 |
| *** tkajinam has quit IRC | 12:25 | |
| *** ociuhandu has joined #openstack-ironic | 12:32 | |
| *** phrobb has quit IRC | 12:32 | |
| *** vkmc has quit IRC | 12:32 | |
| *** mnaser has quit IRC | 12:32 | |
| *** jungleboyj has quit IRC | 12:33 | |
| *** gmann has quit IRC | 12:33 | |
| *** vkmc has joined #openstack-ironic | 12:33 | |
| *** jungleboyj has joined #openstack-ironic | 12:33 | |
| *** gmann has joined #openstack-ironic | 12:33 | |
| *** phrobb has joined #openstack-ironic | 12:33 | |
| *** mnaser has joined #openstack-ironic | 12:33 | |
| *** ociuhandu has quit IRC | 12:36 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/bifrost master: Add openstack collection https://review.opendev.org/734592 | 12:42 |
| uzumaki | what's the culprit when you see DH_KEY_TOO_SMALL SSL error in conductor log? iurygregory rpittau arne_wiebalck ? | 12:43 |
| uzumaki | ideas? | 12:44 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Use assertCountEqual instead of assertItemsEqual https://review.opendev.org/735127 | 12:44 |
| iurygregory | uzumaki, 0 ideas | 12:44 |
| iurygregory | =( | 12:44 |
| uzumaki | aww man, connection unexpectedly closed when trying without ssl =( | 12:45 |
| iurygregory | *boom* | 12:45 |
| uzumaki | <coughs sadly in smoke> | 12:45 |
| *** Qianbiao has joined #openstack-ironic | 12:47 | |
| rpittau | uzumaki: I put it there but it's a guess, I think it's because there's something wrong in the certificate used to communicate with that particular BMC | 12:48 |
| uzumaki | I see, well, I'm trying to see how I can troubleshoot this, I'll try without SSL | 12:49 |
| rpittau | uzumaki: do you have the full conductor logs whan you see the DH_KEY_TOO_SMALL error ? | 12:50 |
| uzumaki | yes | 12:50 |
| openstackgerrit | Harald Jensås proposed openstack/ironic master: Switch Ironic to openstacksdk for Neutron https://review.opendev.org/734873 | 12:51 |
| uzumaki | rpittau, https://pastebin.com/rfehLkYD | 12:54 |
| rpittau | uzumaki: to be honest it looks like a dracclient error to me | 12:56 |
| rpittau | still related to the certificate | 12:57 |
| iurygregory | rpittau, I think i did some magic and now we don't need the changes in test-bifrost.sh for the collections | 13:04 |
| killermoehre | iurygregory, the patch seems to work. master downloads the image fast | 13:05 |
| iurygregory | killermoehre, awesome! \o/ | 13:05 |
| rpittau | iurygregory: well that sounds great :) | 13:05 |
| rpittau | suggest-ception worked :) | 13:05 |
| iurygregory | ^ yup | 13:06 |
| rpittau | Qianbiao: I approved the raid support for ibmc, let's follow up to address the remaining comments when you have some time :) | 13:08 |
| Qianbiao | rpittau sure. | 13:08 |
| rpittau | Qianbiao: thanks! | 13:09 |
| Qianbiao | done local, just has not push. | 13:09 |
| *** rloo has joined #openstack-ironic | 13:17 | |
| killermoehre | btw, is there any list of external binaries needed? | 13:19 |
| iurygregory | for ipa? | 13:21 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Convert ironic-tempest-ipa-wholedisk-bios-agent_ipmitool to dib https://review.opendev.org/734759 | 13:21 |
| iurygregory | yay for timeouts in bifrost =( | 13:21 |
| killermoehre | iurygregory, yes, for IPA | 13:24 |
| iurygregory | killermoehre, bindep should take care of that | 13:26 |
| iurygregory | https://github.com/openstack/ironic-python-agent/blob/master/bindep.txt | 13:26 |
| *** kaiokmo has quit IRC | 13:33 | |
| killermoehre | this doesn't list stuff like `lshw` or `gdisk` | 13:34 |
| *** jamesdenton has joined #openstack-ironic | 13:34 | |
| rpittau | killermoehre: how are you building the iap ramdisk ? | 13:35 |
| rpittau | ipa* | 13:35 |
| killermoehre | basically I use dracut where I add a bunch of stuff to get network available | 13:35 |
| rpittau | killermoehre: the recommended way is to use ironic-python-agent-builder -> https://docs.openstack.org/ironic-python-agent-builder/latest/ | 13:36 |
| killermoehre | let me check. right now with a warm cache I get an image in 9 minutes | 13:42 |
| TheJulia | good morning | 13:44 |
| *** tzumainn has joined #openstack-ironic | 13:45 | |
| TheJulia | uzumaki: so.. what is the certificate signed as that the idrac has? it sounds like the client is going "oh, nope... bad ssl" and disconnecting based on OS library settings | 13:46 |
| iurygregory | good morning TheJulia =) | 13:48 |
| * TheJulia wonders why her heart rate is elevated | 13:50 | |
| killermoehre | rpittau, so the builder downloads from the interwebs. can I configure my own mirrors? | 13:50 |
| *** ricolin has joined #openstack-ironic | 13:51 | |
| *** jdandrea has joined #openstack-ironic | 13:51 | |
| killermoehre | and it fails because my build machine only has 3.8GiB RAM | 13:52 |
| TheJulia | ugh, there was a flag in diskimage-builder for lower memory machines a long time ago, but it is not in the docs pdf I just pulled up :( | 13:54 |
| uzumaki | TheJulia, is there a way for me to look at the iDRAC SSL cert? | 13:55 |
| uzumaki | and good morning! the elevated heart rate might be in anticipation of interesting time travel discussions this morning xD | 13:55 |
| TheJulia | uzumaki: sadly board meeting | 13:56 |
| uzumaki | TheJulia, <facepalm, sigh> ah, have a good one then! | 13:56 |
| uzumaki | and I'm pretty sure iDRAC SSL certs are always self-signed, no? | 13:57 |
| TheJulia | typically I use openssl s_client to evaluate data about certificates | 13:57 |
| ajya | uzumaki: when you had those SSL errors, did it cause any malfunction or was it just spamming the logs? | 13:57 |
| uzumaki | it failed provisioning | 13:57 |
| TheJulia | its a hard communications failure with the ssl client | 13:57 |
| uzumaki | TheJulia, not a bad idea, let's see if I can get into that | 13:58 |
| ajya | in any case i have SSL warnings all over the place as the cert is self signed and it does not break anything, but not sure how this affects metal3 | 13:59 |
| uzumaki | metal3 simply complains a WSMAN error (which I guess is basically the conductor log error bubbled to operator logs) and goes into registering error state | 14:00 |
| uzumaki | well, the s_client tells me the server key is "Server Temp Key: DH, 1024 bits", shouldn't it be 2048 and higher? | 14:02 |
| arne_wiebalck | In which situation does an Ironic controller call the Ironic API? We are seeing (what I believe to be) a way too high request rate on the APIs which comes to 70% from the other Ironic controllers. | 14:05 |
| TheJulia | uzumaki: I think so | 14:05 |
| *** ociuhandu has joined #openstack-ironic | 14:05 | |
| uzumaki | that's too bad, I was hoping to get my dev environment going with this server, it's the same hardware I used to develop the BIOS interface, should've worked =( | 14:06 |
| arne_wiebalck | If my math is right, each controller is requesting a full sorted list every 10 seconds ... | 14:06 |
| arne_wiebalck | ... which may explain why our DB is somewhat slow. | 14:06 |
| arne_wiebalck | 12 controllers, so 1 full sorted list per second. | 14:07 |
| TheJulia | uzumaki: looks like 2048 is the minimum for things like PCIDSS enforcement, so it may be your OS default | 14:09 |
| TheJulia | if your running in FIPS mode, it may also be that | 14:09 |
| TheJulia | arne_wiebalck: interesting because I'm not aware of anything that explicitly does that | 14:10 |
| iurygregory | FIPS XD | 14:11 |
| iurygregory | <3 | 14:11 |
| uzumaki | how does this OS level enforcement relates to provisioning failures with ironic? | 14:11 |
| arne_wiebalck | TheJulia: hmm | 14:11 |
| arne_wiebalck | TheJulia: I am puzzled, too. | 14:11 |
| uzumaki | iurygregory, has an evil laugh about puny infosec governmental standards.. xD | 14:12 |
| uzumaki | I mean, the issues is with the iDRAC certificate that I'm getting from the BMC, should it somehow depend on the OS as well? on the provisioner? | 14:12 |
| iurygregory | uzumaki, I did some reviews/changes in the ironic code to make sure that the code would work with FIPS | 14:12 |
| TheJulia | uzumaki: if the OS cannot connect to the BMC, then provisioning is going to fail | 14:12 |
| uzumaki | that is very interesting.. | 14:13 |
| uzumaki | iurygregory, oh, fancy, fancy! | 14:13 |
| arne_wiebalck | TheJulia: What happens if the endpoint_override in the [service_catalog] is set to the endpoint from the catalog? | 14:13 |
| uzumaki | so how can I setup my OS to connect with the BMC the 'right way'? given the default returned key if 1024bits? | 14:14 |
| openstackgerrit | Merged openstack/bifrost master: Stop supporting non-systemd systems https://review.opendev.org/733053 | 14:14 |
| uzumaki | stop supporting non-systemd systems? woahh \o/ | 14:15 |
| rpioso | Good morning, ironic! | 14:15 |
| uzumaki | morning! rpioso o/ | 14:16 |
| rpioso | uzumaki: o/ | 14:16 |
| uzumaki | how are you this fine morning? (assuming it's a fine morning wherever you are) | 14:16 |
| arne_wiebalck | TheJulia: Which is what we have ... not sure why tbh. | 14:17 |
| TheJulia | I am unsure | 14:18 |
| TheJulia | can't really context switch right now | 14:18 |
| uzumaki | that's alright, no need to overclock yourself | 14:18 |
| *** iurygregory_ has joined #openstack-ironic | 14:19 | |
| arne_wiebalck | TheJulia: np, I keep looking :) | 14:19 |
| *** iurygregory has quit IRC | 14:19 | |
| * iurygregory_ loves getting disconected =( | 14:19 | |
| *** iurygregory_ is now known as iurygregory | 14:19 | |
| * uzumaki finds the comment very ironic.. get it? ironic? but ironic? | 14:20 | |
| rpioso | arne_wiebalck, rpittau, TheJulia: We have cycles to work on Redfish Interoperability Profiles. We're going to use those to communicate Ironic's Redfish needs to our firmware development team. May we join your party? | 14:20 |
| *** jawad_axd has quit IRC | 14:20 | |
| rpioso | We're committing to delivering that to that team during this cycle. | 14:21 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: reno: update release versions for Ussuri and Train https://review.opendev.org/735145 | 14:21 |
| TheJulia | rpioso: party++ ? | 14:22 |
| arne_wiebalck | rpioso: Of course, you're more than welcome! | 14:22 |
| rpioso | arne_wiebalck: ty :-) | 14:23 |
| rpittau | rpioso: sure thing :) | 14:23 |
| rpioso | We've started working on it. | 14:23 |
| rpittau | the more the merrier :D | 14:23 |
| rpioso | rpittau: Cool! | 14:23 |
| rpioso | TheJulia: Would it be appropriate to add my name to the Victoria Redfish interop profile priority or are only core reviewers listed? | 14:24 |
| TheJulia | rpioso: anyone can be listed, we just generally ask that at least one core help be a POC but I don't think that is really a concern | 14:26 |
| rpioso | TheJulia: If you'd like, please feel free to add me. | 14:27 |
| rpittau | I'm happy to be the PoC and "supervise" the work :D | 14:27 |
| * rpioso is glad to work with his IRC nick twin :-) | 14:28 | |
| rpittau | ahah | 14:28 |
| iurygregory | hehe | 14:29 |
| *** ociuhandu has quit IRC | 14:36 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-inspector master: Merge jobs https://review.opendev.org/726121 | 14:36 |
| *** Goneri has joined #openstack-ironic | 14:46 | |
| openstackgerrit | Merged openstack/ironic master: Use the new extension call for getting partition UUIDs https://review.opendev.org/731745 | 14:48 |
| openstackgerrit | Mohammed Naser proposed openstack/ironic master: doc: fix endpoint_override for inspector https://review.opendev.org/735160 | 14:52 |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: Convert with_items to loop https://review.opendev.org/733847 | 14:54 |
| mnaser | i'm wondering if someone has done something like this | 14:56 |
| mnaser | oops, early enter, but.. | 14:56 |
| mnaser | deploying inspector in control plane, but dnsmasq's in 'edge' locations which give dhcp/tftp/enough to get the machine to pxe boot the inspector image | 14:56 |
| mnaser | and then the inspector talks back to the control plane when it boots up. so realy nothing but dnsmasq in those 'remote' sites | 14:57 |
| mnaser | goal is to have a control plane in one location and then conductors covering certain 'datacenters' which sit in fully split off L2 domains | 14:57 |
| TheJulia | very similar things, but not explicitly for inspection | 14:59 |
| TheJulia | static edge configuration dhcp is totally a thing though | 15:00 |
| TheJulia | which points to both ironic and inspector... or just allows mdns or hardcoded addresses | 15:00 |
| mnaser | ok, i'm going to be working on de-composing OSA as it makes a lot of assumption in that inspector == same server that runs dnsmasq | 15:00 |
| TheJulia | ++ | 15:01 |
| mnaser | i'll point to the patches if anyone is interested :> | 15:01 |
| mnaser | i wonder if there was ssomething before called 'ironic-server' | 15:04 |
| mnaser | it seems that it was listed as something that was 'deployed' | 15:04 |
| TheJulia | huh? | 15:06 |
| mnaser | TheJulia: our inventory inside OSA lists all the services and components where we install them, there was an 'infrastructure' component called 'ironic-server' | 15:07 |
| mnaser | it may just be something that was not properly copy pasted | 15:07 |
| TheJulia | k | 15:08 |
| *** sshnaidm_ has joined #openstack-ironic | 15:15 | |
| *** sshnaidm has quit IRC | 15:16 | |
| *** diurnalist has quit IRC | 15:20 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Update doc for multi-arch x86 https://review.opendev.org/735178 | 15:23 |
| TheJulia | Oo | 15:25 |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: Convert with_items to loop https://review.opendev.org/733847 | 15:31 |
| arne_wiebalck | TheJulia: I think I found what is requesting a full listing of all nodes every 10 seconds from the Ironic API ... | 15:35 |
| arne_wiebalck | TheJulia: ... it seems to be the Inspector. | 15:35 |
| arne_wiebalck | TheJulia: Would that make sense? | 15:36 |
| *** k_mouza has quit IRC | 15:37 | |
| TheJulia | arne_wiebalck: that does kind of make sense, but perhaps we can tune that | 15:38 |
| arne_wiebalck | TheJulia: it may be the sync_period which is 15 seconds by default. | 15:39 |
| arne_wiebalck | TheJulia: Not sure what it is syncing ... | 15:39 |
| *** k_mouza has joined #openstack-ironic | 15:42 | |
| openstackgerrit | Merged openstack/bifrost master: reno: update release versions for Ussuri and Train https://review.opendev.org/735145 | 15:42 |
| arne_wiebalck | TheJulia: The more sensible option is probably to not have 10 inspectors. So far I colocated it with the conductor and the api to have homogeneous controller nodes, may it is time to stop this practice. | 15:43 |
| arne_wiebalck | TheJulia: *12 inspectors | 15:43 |
| *** dougsz has quit IRC | 15:43 | |
| TheJulia | hmm | 15:43 |
| TheJulia | there really is no reason the entire node list should be called up | 15:43 |
| TheJulia | I could see a select of a portion of nodes though | 15:44 |
| arne_wiebalck | I switched of the inspectors now ... no more full list of all nodes requests it seems. | 15:45 |
| arne_wiebalck | *off | 15:45 |
| * arne_wiebalck gets quite some Ironic scaling lessons these days | 15:46 | |
| Qianbiao | TheJulia arne_wiebalck check ironic_inspector.conductor.manager.py#89 | 15:48 |
| Qianbiao | sync ironic nodes. | 15:48 |
| TheJulia | Qianbiao: thanks :) | 15:48 |
| * TheJulia is unfortunately paying attention to meetings at the moment | 15:48 | |
| openstackgerrit | Merged openstack/ironic master: Fix requirements check https://review.opendev.org/735076 | 15:49 |
| Qianbiao | np. I am just checking the inspector code. | 15:49 |
| arne_wiebalck | Qianbiao: yeah, could be this one | 15:50 |
| arne_wiebalck | Qianbiao: it is every 60 secs by default | 15:50 |
| Qianbiao | yeah. | 15:51 |
| arne_wiebalck | Qianbiao: this is it | 15:51 |
| arne_wiebalck | Qianbiao: we have 5000 nodes and a max_limit of 1000 | 15:51 |
| Qianbiao | :) good job. it's not pagination also. | 15:52 |
| arne_wiebalck | *5100 | 15:52 |
| arne_wiebalck | Qianbiao: so, the full list will create 5100/1000=6 requests | 15:52 |
| arne_wiebalck | 60/6 = 10 | 15:52 |
| arne_wiebalck | which is the rate I see | 15:53 |
| Qianbiao | you got it. | 15:54 |
| arne_wiebalck | with your help :) | 15:54 |
| * rpittau calls it a day! o/ | 15:58 | |
| *** rpittau is now known as rpittau|afk | 15:58 | |
| *** dougsz has joined #openstack-ironic | 15:58 | |
| *** lucasagomes has quit IRC | 16:02 | |
| *** gyee has joined #openstack-ironic | 16:03 | |
| *** Qianbiao has quit IRC | 16:06 | |
| *** ociuhandu has joined #openstack-ironic | 16:06 | |
| NobodyCam | Good morning Iro | 16:08 |
| NobodyCam | Good morning Ironic Folks | 16:08 |
| NobodyCam | ;p | 16:08 |
| iurygregory | morning NobodyCam | 16:09 |
| NobodyCam | morning iurygregory how goes the battle this day | 16:09 |
| iurygregory | the battle is over \o/ | 16:09 |
| NobodyCam | w00t w00t! | 16:09 |
| iurygregory | and no work tomorrow \o/ | 16:10 |
| *** dhellmann_ has joined #openstack-ironic | 16:10 | |
| NobodyCam | oh I'm jelly | 16:10 |
| NobodyCam | :p | 16:10 |
| iurygregory | now I just need to start my new routine for exercises XD | 16:10 |
| iurygregory | how are you? =) | 16:10 |
| *** dhellmann has quit IRC | 16:11 | |
| *** dhellmann_ is now known as dhellmann | 16:11 | |
| NobodyCam | doing good, just hanging out in the house ;p | 16:11 |
| iurygregory | hehe | 16:12 |
| NobodyCam | I seem to have. greater urge to type "all work and no play...something something" more often now a days | 16:12 |
| iurygregory | and today is the announcement of PS5 \o/ | 16:12 |
| NobodyCam | yea | 16:12 |
| *** iurygregory has quit IRC | 16:15 | |
| *** ociuhandu has quit IRC | 16:15 | |
| *** k_mouza has quit IRC | 16:23 | |
| *** k_mouza has joined #openstack-ironic | 16:26 | |
| openstackgerrit | Merged openstack/ironic master: Feature: Add raid configuration support for ibmc driver https://review.opendev.org/728123 | 16:35 |
| openstackgerrit | Merged openstack/ironic master: doc: fix endpoint_override for inspector https://review.opendev.org/735160 | 16:35 |
| openstackgerrit | Merged openstack/ironic-inspector master: Update doc for multi-arch x86 https://review.opendev.org/735178 | 16:35 |
| arne_wiebalck | https://usercontent.irccloud-cdn.com/file/X8nkCB2v/API_requests.png | 16:41 |
| arne_wiebalck | TheJulia: Qianbiao ^^ much better now :) | 16:42 |
| *** sshnaidm_ is now known as sshnaidm | 16:46 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:46 | |
| *** ricolin has quit IRC | 16:48 | |
| *** derekh has quit IRC | 17:01 | |
| TheJulia | arne_wiebalck: what did you change? | 17:06 |
| arne_wiebalck | TheJulia: switched off all inspectors | 17:07 |
| TheJulia | heh | 17:07 |
| TheJulia | wow | 17:07 |
| arne_wiebalck | TheJulia: need to see if I change the sync option, the number of inspectors, or both | 17:07 |
| arne_wiebalck | TheJulia: but the inspectors gets a full dump every minute | 17:07 |
| *** k_mouza has quit IRC | 17:07 | |
| arne_wiebalck | TheJulia: this can become heavy with many nodes and several inspectors | 17:08 |
| *** k_mouza has joined #openstack-ironic | 17:08 | |
| TheJulia | eek | 17:09 |
| TheJulia | yeah, lets fix that | 17:09 |
| * TheJulia puts on some music or something | 17:09 | |
| arne_wiebalck | TheJulia: do you think we should put sth on storyboard to at least keep track of this issue? | 17:09 |
| arne_wiebalck | TheJulia: maybe it is solved by config only, but maybe the sync can be made smarter as well | 17:10 |
| TheJulia | arne_wiebalck: if you don't mind, it would be appreciated | 17:10 |
| arne_wiebalck | TheJulia: will do | 17:10 |
| arne_wiebalck | have a nice evening, bye all o/ | 17:10 |
| TheJulia | arne_wiebalck: quick question, in your db is the uuid field indexed? | 17:11 |
| *** dougsz has quit IRC | 17:13 | |
| *** diurnalist has joined #openstack-ironic | 17:18 | |
| *** k_mouza has quit IRC | 17:18 | |
| TheJulia | hmm | 17:20 |
| *** k_mouza has joined #openstack-ironic | 17:23 | |
| TheJulia | wow, the sync code :( | 17:27 |
| *** k_mouza has quit IRC | 17:32 | |
| *** k_mouza has joined #openstack-ironic | 17:35 | |
| *** k_mouza has quit IRC | 17:39 | |
| TheJulia | arne_wiebalck: what is your cleanup period? | 17:40 |
| jamesdenton | QQ: When using iPXE, my system appears to load undionly.kpxe, but ends up in a loop where it seems to reinitialize and loads it again. What should be the next step once undionly.kpxe is loaded? | 17:50 |
| jamesdenton | this is w/ neutron integration, btw | 17:51 |
| TheJulia | jamesdenton: bios booting or are these uefi machines? | 17:51 |
| jamesdenton | they're uefi capable, but booting in legacy mode | 17:52 |
| TheJulia | so what _should_ be occuring is undionly.kpxe should be triggering dhcp again | 17:53 |
| TheJulia | the dhcp request should get a sparate response from neutron | 17:53 |
| TheJulia | we send the parameters such that this is the case with the flat or neutron network interface | 17:53 |
| jamesdenton | i'll see if i can capture what dhcp is sending | 17:56 |
| TheJulia | jamesdenton: ok | 17:56 |
| TheJulia | arne_wiebalck: what is your pxe filter? | 17:57 |
| arne_wiebalck | noop | 17:58 |
| TheJulia | so a big giant sync knob is what you need | 17:58 |
| arne_wiebalck | do you think it is the pxe filter sync? | 17:59 |
| TheJulia | looking at the code, yes | 17:59 |
| arne_wiebalck | I was thinking so initially, but the request rate does not macth | 17:59 |
| TheJulia | there is a cleanup worker as well | 17:59 |
| TheJulia | and they trigger independent clients | 17:59 |
| arne_wiebalck | did you see the math I did above? | 17:59 |
| TheJulia | I didn't | 18:00 |
| arne_wiebalck | the periodic task Qianbiao pointed to seems to match what I see | 18:00 |
| arne_wiebalck | the sync job runs every 60 secs | 18:01 |
| TheJulia | yup | 18:01 |
| arne_wiebalck | we have 5100 nodes which gives 6 requests (max-limit=1000) | 18:01 |
| arne_wiebalck | so a req every 10 secs | 18:01 |
| arne_wiebalck | this is what the APIs report | 18:01 |
| arne_wiebalck | you cannot see this in the graph, since I deducted the reqs from nova | 18:02 |
| arne_wiebalck | I can play with the interval parameter to confirm | 18:02 |
| TheJulia | well, I'm not really sure it matters if your using noop | 18:02 |
| TheJulia | but by all means | 18:03 |
| arne_wiebalck | aren't these different periodic tasks? | 18:03 |
| arne_wiebalck | I understood there are 2 | 18:03 |
| TheJulia | possibly 3 | 18:03 |
| arne_wiebalck | oh, ok | 18:03 |
| TheJulia | but trigger inspection should also populate the cache | 18:04 |
| TheJulia | I guess if you hit a different inspector, then that could be slightly problematic | 18:04 |
| arne_wiebalck | this is all without inspection | 18:04 |
| arne_wiebalck | this is just constant background noise | 18:04 |
| jamesdenton | TheJulia is the 2nd DHCP request from the PXE client supposed to look different from the 1st? | 18:04 |
| TheJulia | yes, it should | 18:05 |
| TheJulia | there should be an ipxe specific action | 18:05 |
| TheJulia | action is the wrong word | 18:06 |
| TheJulia | a ipxe specific flag in the reply | 18:06 |
| TheJulia | arne_wiebalck: yeah, inspector is just trying to keep its cache consistent with the state of ironic | 18:06 |
| TheJulia | which seems... problematic | 18:06 |
| arne_wiebalck | TheJulia: yes | 18:06 |
| TheJulia | Maybe we could populate as-needed | 18:06 |
| TheJulia | seems like it could be delay prone, but it wouldn't be as negatively impacting on load | 18:07 |
| arne_wiebalck | TheJulia: what data is it actually sync'ing? | 18:07 |
| TheJulia | the nodes cache and removing old nodes | 18:07 |
| arne_wiebalck | TheJulia: if you know from the top of your head | 18:07 |
| arne_wiebalck | TheJulia: the nodes cache is in the inspector's db, no? | 18:07 |
| TheJulia | I don't think so | 18:08 |
| arne_wiebalck | ok | 18:08 |
| arne_wiebalck | it keeps this is memory? | 18:08 |
| TheJulia | well it may be in db | 18:09 |
| TheJulia | eek | 18:09 |
| TheJulia | yeah, it is behind sqlalchemy | 18:09 |
| arne_wiebalck | ok, so the inspector gets all nodes from ironic and caches the ones it needs to inspect? | 18:10 |
| TheJulia | basically yes | 18:10 |
| arne_wiebalck | why does not only ask for the ones which have to be inspected? | 18:11 |
| arne_wiebalck | why does it ... | 18:11 |
| TheJulia | I don't know | 18:11 |
| arne_wiebalck | :) | 18:11 |
| TheJulia | which is why my comment "oh wow the sync code" | 18:11 |
| arne_wiebalck | ok, I see :-D | 18:11 |
| arne_wiebalck | the ironic db is usually quite small | 18:11 |
| arne_wiebalck | and if that is the cache and the reason to sync, the inspector drops almost all data it gets | 18:12 |
| arne_wiebalck | sorry the inspector db | 18:12 |
| arne_wiebalck | well it must drop most data since there is no way it store our 100MB DB every 10 seconds :) | 18:13 |
| arne_wiebalck | *stores | 18:13 |
| arne_wiebalck | but I am also not clear if it makes sense to have an inspector for every conductor | 18:14 |
| TheJulia | It can be a bottleneck but I think if we make it slightly smarter it should be fine or only cache as-needed or something | 18:16 |
| arne_wiebalck | yeah ... I guess increasing the frequency affects the responsiveness of the inspector (until it notices it has to inspect a node)? | 18:22 |
| bfournie | I'd like to request a new Ussuri release for ironic-lib 4.2 to pick up https://review.opendev.org/#/c/727677/. This is to help with a Triple "qemu-img convert" issue - https://bugs.launchpad.net/tripleo/+bug/1879472 | 18:27 |
| openstack | Launchpad bug 1879472 in tripleo "OVB overcloud deploy fails sporadically with "not enough free physical ports" error" [Critical,Triaged] | 18:27 |
| patchbot | patch 727677 - ironic-lib (stable/ussuri) - image_convert: retry resource unavailable and make... (MERGED) - 1 patch set | 18:27 |
| TheJulia | bfournie: have you posted a change for openstack/releases yet? | 18:38 |
| TheJulia | arne_wiebalck: well, when being told it should't matter, I guess it all depends on the actual running configuration | 18:39 |
| jamesdenton | TheJulia FWIW, I do see the difference in DHCP requests. 1st request is lacking Option 175, while the second request (from iPXE DHCP request) has it. Seems like it might be ignored, despite neutron's dnsmasq running with "--dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE" flags. Will continue to dig, thanks. | 18:40 |
| TheJulia | jamesdenton: check the actual neutron port and make sure the config is set for DHCP. You should see it in the port's properties in neutron.. Also, likely a good idea just to double check that you don't have any errors from dhcp or neutron in the ironic-conductor log output | 18:42 |
| arne_wiebalck | TheJulia: ok, I will think about what to do and file a story tomorrow | 18:42 |
| TheJulia | arne_wiebalck: yeah, I'm needing to think about it too | 18:42 |
| TheJulia | I think it is navigatable, but I'm also stimming which means I can't drive now and likely should just go take a nap :( | 18:43 |
| TheJulia | option b is go have some alcohol, which is not what I want to do | 18:43 |
| arne_wiebalck | make sure you take regular breaks! a nap sounds good :) | 18:44 |
| bfournie | TheJulia: not yet | 18:45 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-inspector master: WIP knob to disable peridics https://review.opendev.org/735212 | 18:47 |
| TheJulia | arne_wiebalck: it seems ^^^ that simple | 18:47 |
| TheJulia | at least to me | 18:47 |
| TheJulia | 3a01335d1e962e8cd59d597ab0ec795d3e5dfc8e as 4.2.next ? | 18:48 |
| arne_wiebalck | TheJulia: but then the inspector would not inspect anymore? | 18:49 |
| TheJulia | arne_wiebalck: I think it should still inspect then | 18:49 |
| arne_wiebalck | TheJulia: what is the sync for then? | 18:49 |
| * arne_wiebalck is confused | 18:50 | |
| TheJulia | bfournie: change pushed | 18:50 |
| TheJulia | arne_wiebalck: to reconcile everything outside of the inspection process itself | 18:50 |
| bfournie | TheJulia: Thank you! And yes, take a nap | 18:50 |
| arne_wiebalck | TheJulia: can you give an example? | 18:50 |
| TheJulia | arne_wiebalck: if you run an HA cluster and you can randomly hit an inspector and it is managing dnsmasq or specific tftp server access, then you need cluster wide consistency | 18:51 |
| arne_wiebalck | TheJulia: I see ... a knob sounds good then. I guess :) | 18:53 |
| arne_wiebalck | TheJulia: I will have a look tomorrow, have to go now. Thanks and take care! | 18:54 |
| TheJulia | arne_wiebalck: yeah, look tomorrow, it seems easy enough and if we need to tweek/improve the cache miss handling that seems "easy" | 19:14 |
| *** k_mouza has joined #openstack-ironic | 19:33 | |
| *** k_mouza has quit IRC | 19:37 | |
| mnaser | btw, ironic-ui in ussuri is broken and fix is -- https://review.opendev.org/#/c/735028/ | 20:00 |
| patchbot | patch 735028 - ironic-ui - api: drop project_id from kwargs in client - 1 patch set | 20:00 |
| TheJulia | mnaser: is it actually in the client library that the failure is happening, because it seems like project id is valid to pass through for non-admins if the rights are correct | 20:03 |
| TheJulia | then again I'm not sure people are actually setting that | 20:03 |
| mnaser | TheJulia: ok so based on my troubleshooting -- https://github.com/openstack/python-ironicclient/blob/master/ironicclient/client.py#L51-L60 | 20:03 |
| mnaser | so in our case, auth_type is not provided, endpoint and token is set, so auth_type is set to admin_token | 20:04 |
| mnaser | which takes us to the following plugin - https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/loading/_plugins/admin_token.py | 20:04 |
| mnaser | which only needs endpoint and a token to 'be loaded', hence i would get an error that project_id was provided but not needed (and really, if we have a token and an endpoint, we really dont need a project anyways) | 20:05 |
| mnaser | maybe the real bug is that the client shouldn't be using admin_token auth_type but that's up for debate/discussion | 20:05 |
| mnaser | git blame in the python ironicclient says that code has been there for 4 years so | 20:06 |
| * mnaser shrugs | 20:06 | |
| TheJulia | figures it would ahve been one of my changes | 20:06 |
| TheJulia | approved | 20:08 |
| * TheJulia pokes IPA | 20:08 | |
| jamesdenton | anyone know offhand if IPA is configured by default to pull internal or admin baremetal endpoint vs public endpoint? | 20:09 |
| TheJulia | jamesdenton: it uses what is provided via the [service_catalog] defaults in the conductor, the endpoint_override setting can be used to override it to an explicit url | 20:10 |
| jamesdenton | ahh ok, perfect. thank you! | 20:11 |
| openstackgerrit | Merged openstack/ironic-ui master: api: drop project_id from kwargs in client https://review.opendev.org/735028 | 20:22 |
| *** gyee has quit IRC | 20:36 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Add timeout operations to try and prevent hang on read() https://review.opendev.org/734709 | 20:57 |
| TheJulia | bfournie: ^^^ | 20:57 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-inspector master: WIP knob to disable periodic tasks https://review.opendev.org/735212 | 21:03 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-inspector master: DNM: Test change for disabling periodic sync https://review.opendev.org/735250 | 21:03 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Fix apidoc build https://review.opendev.org/734712 | 21:05 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-specs master: Victoria Cycle Priorities https://review.opendev.org/720100 | 21:22 |
| *** stevebaker has quit IRC | 21:35 | |
| *** rcernin has joined #openstack-ironic | 21:48 | |
| *** rcernin has quit IRC | 21:48 | |
| *** jtomasek has quit IRC | 21:48 | |
| *** rcernin has joined #openstack-ironic | 21:49 | |
| *** dustinc has joined #openstack-ironic | 21:55 | |
| *** eliaswimmer_ has quit IRC | 21:55 | |
| mnaser | hmm | 21:59 |
| mnaser | is it possible that it's needed for us to add a [inspector] section with credentials/etc to use ironic-inspector? | 21:59 |
| mnaser | i dont see it documented anywhere that you need to do this | 21:59 |
| mnaser | not here - https://docs.openstack.org/ironic/latest/admin/inspection.html#in-band-inspection or the inspector project doesnt talk too much about configuring ironic (makes sense) | 21:59 |
| mnaser | if something is missing, im happy to push up a doc patch | 22:00 |
| mnaser | i have the [service_catalog] section defined.. | 22:00 |
| mnaser | context: 2020-06-11 14:55:21.923 135762 ERROR ironic.conductor.task_manager [req-d7963a4e-5fba-4b78-b4e6-195ff290df3c f39b5bffe7bc6b7598f1635c720e120436b55623a742ef6b77ec2b0be18d89a0 0c3f4666fa794efb8b6ef4ab191f206f - 5ea99c872fbf48d3854b4fbf83652c87 default] Node 54f1832c-7365-4f35-ac3e-012ae3d95f17 moved to provision state "inspect failed" from state "inspect wait"; target provision state is "manageable": | 22:01 |
| mnaser | keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL | 22:01 |
| mnaser | unless it's because im missing auth_plugin in that section.. oops | 22:01 |
| mnaser | it sounds like we have to configure all of the sections, and service_catalog isn't really used? even devstack reflects that? | 22:11 |
| mnaser | https://opendev.org/openstack/ironic/src/branch/master/devstack/lib/ironic#L101 | 22:11 |
| mnaser | i'm working on cleaning up the OSA roles and i'd be happy to clean it up to rely strictly on service_catalog if that can be doe | 22:13 |
| *** threestrands has joined #openstack-ironic | 22:20 | |
| TheJulia | mnaser: you do need an [inspector] section to point an inspector deployment | 22:20 |
| TheJulia | hmm... I guess since there is no setting there it doesn't know | 22:20 |
| TheJulia | It may also be that the endpoint is incorrectly named or something fun like that | 22:21 |
| mnaser | TheJulia: yeah I thought ironic would hit up keystone to get the endpoint of inspector.. but yeah | 22:32 |
| TheJulia | it may also just be the config | 22:33 |
| *** tzumainn has quit IRC | 22:33 | |
| TheJulia | Oo | 22:35 |
| TheJulia | I think https://github.com/openstack/ironic/blob/15f0dc93df6f03303f3ab3813d48fbb7da8ac0a2/ironic/drivers/modules/inspector.py#L53 may be wrong | 22:35 |
| TheJulia | But I'd need a devstack up and running to check at this point and I'm basically done for the week | 22:36 |
| mnaser | TheJulia: you may be right on that | 22:38 |
| mnaser | neutron seems to be doing things differently -- https://github.com/openstack/ironic/blob/15f0dc93df6f03303f3ab3813d48fbb7da8ac0a2/ironic/common/neutron.py#L50-L56 | 22:38 |
| mnaser | i'm assuming this is maybe if someone is running inspector with noauth | 22:42 |
| TheJulia | what may also be the case is people generally afaik don't rely on keystone because they are pointing internally to a private ip anyway | 22:43 |
| TheJulia | but that could be a horribly wrong assumption | 22:44 |
| *** tkajinam has joined #openstack-ironic | 22:46 | |
| openstackgerrit | Merged openstack/ironic-python-agent master: Split and move logic for partition tables https://review.opendev.org/722654 | 22:53 |
| *** rloo has quit IRC | 22:59 | |
| *** rcernin has quit IRC | 23:48 | |
| *** rcernin has joined #openstack-ironic | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
