Tuesday, 2020-04-07

rpiosoTheJulia: Thank you!00:03
*** k_mouza has joined #openstack-ironic00:08
*** k_mouza has quit IRC00:10
*** rloo has quit IRC00:15
*** bfournie has joined #openstack-ironic00:16
*** gyee has quit IRC00:58
*** ociuhandu has joined #openstack-ironic01:04
*** ociuhandu has quit IRC01:09
*** cdearborn has quit IRC01:43
*** jdandrea has quit IRC01:51
*** Goneri has quit IRC02:00
*** k_mouza has joined #openstack-ironic02:10
*** k_mouza has quit IRC02:15
*** igordc has joined #openstack-ironic02:33
*** ociuhandu has joined #openstack-ironic02:34
*** ociuhandu has quit IRC02:39
*** hamzy has quit IRC02:46
*** hamzy has joined #openstack-ironic02:46
*** igordc has quit IRC03:16
openstackgerritMerged openstack/ironic master: Use auth values from neutron conf when managing Neutron ports  https://review.opendev.org/70867803:16
*** mkrai has joined #openstack-ironic03:23
*** mkrai has joined #openstack-ironic03:27
*** tzumainn has quit IRC03:53
openstackgerritMerged openstack/ironic-python-agent master: Use unittest mock from standard library  https://review.opendev.org/71770904:21
*** ricolin has joined #openstack-ironic05:30
iurygregorygood morning  Ironic o/06:23
arne_wiebalckGood morning, iurygregory and ironic!06:44
iurygregoryarne_wiebalck, o/06:51
*** rpittau|afk is now known as rpittau06:56
rpittaugood morning ironic! o/06:56
iurygregorymorning rpittau o/06:57
rpittauhey iurygregory :)06:57
*** uzumaki has joined #openstack-ironic07:07
*** hjensas has quit IRC07:07
uzumakigood morning ironic! o/07:07
iurygregorymorning uzumaki07:08
uzumakihow you doing iurygregory ?07:08
iurygregorydoing good and you?07:08
uzumakii'm great07:08
*** hjensas has joined #openstack-ironic07:21
*** uzumaki has quit IRC07:29
openstackgerritRiccardo Pittau proposed openstack/ironic-lib master: Convert jobs to dib  https://review.opendev.org/71699807:34
*** diga has joined #openstack-ironic07:40
*** ociuhandu has joined #openstack-ironic07:45
openstackgerritRiccardo Pittau proposed openstack/ironic-prometheus-exporter master: Use hacking for pep8 tests  https://review.opendev.org/71768907:50
openstackgerritRiccardo Pittau proposed openstack/ironic-prometheus-exporter master: Use hacking for pep8 tests  https://review.opendev.org/71768907:51
*** alexmcleod has joined #openstack-ironic08:00
*** mkrai has quit IRC08:01
*** hjensas is now known as hjensas|afk08:04
*** uzumaki has joined #openstack-ironic08:07
kaifengmorning everyone08:08
kaifenghave anyone paid attention to the memory consumption of ironic conductor? it seems consuming more than expected08:10
kaifengsystem shows ~277M mem usage even in the idle state08:10
arne_wiebalckkaifeng: yes08:10
arne_wiebalckkaifeng: I have raised this a year ago or so08:11
arne_wiebalckkaifeng: I see the exact same thing08:11
*** k_mouza has joined #openstack-ironic08:11
kaifengarne_wiebalck: i have removed some drivers but it seems doesn't help08:11
arne_wiebalckkaifeng: at the time I discussed this briefly with dtantsur|afk , but we never got to the bottom of this08:12
arne_wiebalckkaifeng: I am not sure there are any traces of our discussion from back then, let me check ...08:13
openstackgerritRiccardo Pittau proposed openstack/ironic master: Collect tftpd info from journald  https://review.opendev.org/71774508:14
rpittauhey kaifeng :)08:14
kaifengarne_wiebalck: wow, thanks!08:14
openstackgerritRiccardo Pittau proposed openstack/ironic master: Collect tftpd info from journald  https://review.opendev.org/71774508:14
kaifenghi rpittau08:14
*** k_mouza has quit IRC08:15
*** tkajinam has quit IRC08:17
*** mkrai has joined #openstack-ironic08:20
arne_wiebalckkaifeng: here's what I saw at the time when changing parameters:08:21
arne_wiebalck16 processes/1 thread each --> 300MB/process --> 4.8GB total08:21
arne_wiebalck4 processes/4 threads each --> 900MB/process --> 3.6GB total08:21
arne_wiebalck1 process / 16 threads --> 2.1GB/process --> 2.1GB total08:21
*** lucasagomes has joined #openstack-ironic08:22
arne_wiebalckkaifeng: the DB size was around 50MB08:22
kaifengarne_wiebalck: how do you get the consumption of each part, say, DB?08:23
kaifengwe are evaluating an edge case and running conductor in a VM which has only two vcpu08:24
openstackgerritMerged openstack/networking-baremetal master: Cleanup py27 support  https://review.opendev.org/71761508:25
arne_wiebalckkaifeng: for the DB I probably ran some command to sum up the contents; this was mostly to show that the one would need to DB to lay out in memory *many* times to explain the RSS foot print08:25
arne_wiebalckkaifeng: in our qa env, the DB was 25 times smaller, but the RAM consumption only a factor of 308:26
arne_wiebalckkaifeng: let me check what the consumption is right now in our prod nodes ...08:27
kaifengarne_wiebalck: in my case ironic doesn't manage any node at the time, just installed, configured and running idle08:27
arne_wiebalckkaifeng: yes ... we also saw it is jumping: it starts low, then all of a sudden it jumps to 300MB RSS and stays there08:28
arne_wiebalckkaifeng: probably when some periodic tasks kicks in and it loads some lib08:28
kaifengarne_wiebalck: i see this jumping after service restart, but it will drop after a while08:29
kaifengarne_wiebalck: periodic task consumes cpu (though it's also a bit higher than other services), i guess most of our periodic tasks are just doing db query08:31
* arne_wiebalck just realised it is almost 2 years ago we noticed this08:32
* kaifeng feels like a profiling is required :(08:33
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: Switch to unittest mock  https://review.opendev.org/71797908:34
*** uzumaki has quit IRC08:36
*** derekh has joined #openstack-ironic08:38
*** mkrai has quit IRC08:38
*** mkrai_ has joined #openstack-ironic08:38
*** ociuhandu has quit IRC08:50
*** ociuhandu has joined #openstack-ironic08:57
*** dtantsur|afk is now known as dtantsur09:01
*** priteau has joined #openstack-ironic09:01
dtantsurmorning ironic09:01
arne_wiebalckkaifeng: I made some attempts with pyrasite at the time but did not get very far09:03
arne_wiebalckkaifeng: in any case, let me know in case you find something: I just checked and our 16 worker API process use 3.4GB RSS09:05
kaifengarne_wiebalck: np, i will let you know if there is any finding09:07
iurygregorymorning dtantsur09:25
*** k_mouza has joined #openstack-ironic09:37
*** k_mouza has quit IRC09:49
*** k_mouza has joined #openstack-ironic09:57
*** k_mouza has quit IRC09:57
*** k_mouza has joined #openstack-ironic09:57
openstackgerritIlya Etingof proposed openstack/ironic master: Add `network_data` field to ironic node object  https://review.opendev.org/68791010:04
openstackgerritPavlo Shchelokovskyy proposed openstack/ironic master: Allow INSPECTWAIT state for lookup  https://review.opendev.org/71801710:12
*** ricolin has quit IRC10:16
openstackgerritVerification of a change to openstack/ironic failed: DRAC: Fix a failure to create virtual disk bug  https://review.opendev.org/71085210:25
*** rpittau is now known as rpittau|bbl10:25
*** k_mouza has quit IRC10:42
*** k_mouza has joined #openstack-ironic10:46
*** rachit7 has joined #openstack-ironic10:49
*** rachit7 has quit IRC11:03
openstackgerritIlya Etingof proposed openstack/ironic master: Validate node `network_data` field contents  https://review.opendev.org/71461411:12
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: Switch to unittest mock  https://review.opendev.org/71797911:15
openstackgerritIlya Etingof proposed openstack/ironic master: Add `get_node_network_data` method to NetworkInterface  https://review.opendev.org/68818411:16
openstackgerritIlya Etingof proposed openstack/ironic master: Add `get_node_network_data` to non-Neutron NetworkInterface  https://review.opendev.org/68826611:16
openstackgerritIlya Etingof proposed openstack/ironic master: Make redfish boot interface to use `network_data`  https://review.opendev.org/68827211:17
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: Standalone CLI: optional support for ironic-inspector  https://review.opendev.org/71806711:27
dtantsurTheJulia: ^^^11:33
*** ociuhandu has quit IRC11:41
*** ociuhandu has joined #openstack-ironic11:42
*** ociuhandu has quit IRC11:47
openstackgerritIlya Etingof proposed openstack/ironic master: Support burning configdrive into boot ISO  https://review.opendev.org/68725011:48
*** rh-jelabarre has joined #openstack-ironic11:48
openstackgerritIlya Etingof proposed openstack/ironic master: Add `network_data` field to ironic node object  https://review.opendev.org/68791011:48
openstackgerritIlya Etingof proposed openstack/ironic master: Validate node `network_data` field contents  https://review.opendev.org/71461411:48
openstackgerritIlya Etingof proposed openstack/ironic master: Add `get_node_network_data` method to NetworkInterface  https://review.opendev.org/68818411:49
openstackgerritIlya Etingof proposed openstack/ironic master: Add `get_node_network_data` to non-Neutron NetworkInterface  https://review.opendev.org/68826611:49
openstackgerritIlya Etingof proposed openstack/ironic master: Make redfish boot interface to use `network_data`  https://review.opendev.org/68827211:49
* etingof has been bitten by a tick - Spring!11:50
iurygregoryetingof, ouch =( I hope you are ok11:52
dtantsuretingof: oh. where did you manage to get it?11:53
* etingof wishes that the tick has died in a good health11:56
etingofI went to the woods the other day, might be from there11:57
*** dougsz has joined #openstack-ironic12:03
*** rpittau|bbl is now known as rpittau12:06
*** tkajinam has joined #openstack-ironic12:08
*** yaawang has quit IRC12:08
*** hjensas|afk is now known as hjensas12:10
* arne_wiebalck removed the first this year from his cat as well some days ago12:11
dtantsursigh12:11
openstackgerritRiccardo Pittau proposed openstack/ironic-inspector master: [WIP] Use mock from unittest  https://review.opendev.org/71808012:17
openstackgerritMerged openstack/ironic master: Do not autoescape all Jinja2 templates  https://review.opendev.org/71684512:23
openstackgerritVerification of a change to openstack/ironic-inspector failed: Move Babel to test-requirements  https://review.opendev.org/71774312:23
openstackgerritVerification of a change to openstack/ironic failed: Replace netaddr with ipaddress  https://review.opendev.org/71785212:24
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: [WIP] Get rid of oslo.config requirements  https://review.opendev.org/71772412:30
arne_wiebalckdtantsur: unless I am missing sth https://review.opendev.org/#/c/701926/ fell through the cracks I think12:33
patchbotpatch 701926 - ironic - [doc] Images need some metadata for software RAID - 2 patch sets12:33
*** igordc has joined #openstack-ironic12:34
dtantsurI'll check, thanks12:34
openstackgerritIlya Etingof proposed openstack/ironic master: Add `get_node_network_data` to Neutron NetworkInterface  https://review.opendev.org/68889612:34
arne_wiebalckdtantsur: would be good to have this (or something improved if needed) in before the release since it describes the requirements on images to be used with software RAID12:35
arne_wiebalckdtantsur: thanks12:35
openstackgerritIlya Etingof proposed openstack/ironic master: Add L3 boot section to the docs  https://review.opendev.org/68984412:35
dtantsurarne_wiebalck: first comment: I think we have a fallback to the old approach, no?12:35
dtantsursecond comment: does it work with standalone ironic?12:36
arne_wiebalckdtantsur: do partition images work with standalone ironic12:36
arne_wiebalck?12:36
dtantsurarne_wiebalck: perfectly fine12:37
arne_wiebalckdtantsur: then this should also work, I believe12:37
dtantsurarne_wiebalck: where will rootfs_uuid come from in this case?12:37
arne_wiebalckdtantsur: where does it come from for partition images>12:37
arne_wiebalck?12:37
openstackgerritRiccardo Pittau proposed openstack/ironic-prometheus-exporter master: Use hacking for pep8 tests  https://review.opendev.org/71768912:38
dtantsurarne_wiebalck: we know it because we create partitioning. Which makes me realize, do we even support partition images with sw RAID?12:38
arne_wiebalckdtantsur: I think we explicitly say, no support12:39
dtantsurokay, so rootfs_uuid is needed for whole disk images?12:39
arne_wiebalckdtantsur: but this mostly since noone ever tried12:39
dtantsurI don't think we require it for non-RAID case12:39
arne_wiebalckmind you, this is only the documentation :)12:40
dtantsurarne_wiebalck: yeah, but I'm trying to understand if rootfs_uuid has to be provided somehow for the standalone case (and document it)12:40
arne_wiebalckdtantsur: IIRC, the code will try to find the UUID first from glance, then ... let me check ...12:41
dtantsuryep, and if there's no glance, we only have the fallback?12:42
*** rloo has joined #openstack-ironic12:44
arne_wiebalckyes, we fall back to internal_info12:45
*** ociuhandu has joined #openstack-ironic12:45
dtantsurarne_wiebalck: well, internal_info cannot be modified, so we expect the first partition?12:45
arne_wiebalckdtantsur: on the conductor side, I think internal_info was where we looked before12:46
arne_wiebalckdtantsur: so, this is what the code does when glance fails12:46
arne_wiebalckdtantsur: on the IPA side I think we fall back to the first partition, let me check ...12:47
arne_wiebalckdtantsur: yes, if we do not manage to find the partition by uuid, we use the first one12:51
openstackgerritMerged openstack/networking-baremetal master: Use mock from unittest  https://review.opendev.org/71767412:51
dtantsurokay, let's make it clear in the docuemtnation12:51
*** ociuhandu has quit IRC12:52
openstackgerritRiccardo Pittau proposed openstack/ironic-inspector master: Use mock from unittest  https://review.opendev.org/71808012:53
arne_wiebalckdtantsur: so we will keep this fallback?12:54
arne_wiebalckdtantsur: I mean longterm :)12:54
dtantsurarne_wiebalck: we probably should. definitely until we have a solution for the standalone case.12:55
arne_wiebalckdtantsur: so, using the 1st partition will help with the standalone case12:55
arne_wiebalckdtantsur: probably we should also mark this in the code as such, to avoid it gets removed by some cleanup later12:56
arne_wiebalckdtantsur: did you have a chance to try standalone with s/w RAID?12:56
arne_wiebalckdtantsur: or know someone who did?12:56
*** ociuhandu has joined #openstack-ironic12:59
dtantsurno and no, unfortunately13:00
*** ociuhandu has quit IRC13:00
*** ociuhandu has joined #openstack-ironic13:01
*** jcoufal has joined #openstack-ironic13:03
*** mkrai_ has quit IRC13:06
TheJuliagood morning13:07
TheJuliaThis whole tick business is why I live in the desert....13:10
iurygregorygood morning TheJulia13:10
dtantsurmorning TheJulia13:11
rpittauhey TheJulia :)13:12
dtantsurI just cover myself in horribly-smelling repellents every time I venture out13:12
openstackgerritMerged openstack/ironic-python-agent master: Editing follow-up for UEFI Software RAID support  https://review.opendev.org/71778713:18
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: Get rid of the oslo.config requirement  https://review.opendev.org/71772413:19
dtantsuranother big dependency cut ^^13:19
*** jdandrea has joined #openstack-ironic13:20
*** Goneri has joined #openstack-ironic13:23
openstackgerritJulia Kreger proposed openstack/ironic master: Replace netaddr with ipaddress  https://review.opendev.org/71785213:25
dtantsurif anybody has a minute, this is probably an easy fix: https://storyboard.openstack.org/#!/story/200752613:30
dtantsurany opinions if https://review.opendev.org/#/c/717724/ needs a release note?13:31
patchbotpatch 717724 - python-ironicclient - Get rid of the oslo.config requirement - 3 patch sets13:31
dtantsurthere may be discrepancies in what the new approach supports..13:32
TheJuliathat bug looks easy13:32
*** ociuhandu has quit IRC13:32
*** ociuhandu has joined #openstack-ironic13:33
rpittaudtantsur: I was going to add a comment, I'm for a release note13:33
openstackgerritDmitry Tantsur proposed openstack/python-ironicclient master: Get rid of the oslo.config requirement  https://review.opendev.org/71772413:33
dtantsurrpittau: something alone these lines ^^^?13:33
TheJuliaI'm for, just a minor one should be fine13:33
dtantsur* along13:33
dtantsurwording suggestions welcome13:33
TheJulia"This version of ironic no longer requires oslo.config. Configuration loading logic was migrated to the already utilized openstacksdk library."13:35
dtantsurhjensas: can https://bugs.launchpad.net/tripleo/+bug/1871346 be an effect of your recent patch?13:35
openstackLaunchpad bug 1871346 in tripleo "Ironic nodes registration failing with error - ironicclient.common.apiclient.exceptions.InternalServerError: 'NoneType' object has no attribute 'keys'" [Critical,Triaged]13:35
dtantsurTheJulia: "configuration loading" may be a big vague13:35
dtantsurreviews of https://review.opendev.org/#/c/718067/ are also appreciated13:38
patchbotpatch 718067 - python-ironicclient - Standalone CLI: optional support for ironic-inspector - 1 patch set13:38
hjensasdtantsur: It looks like the local_link_connection changes I did is a suspect. yes.13:39
dtantsurmind taking a look?13:40
hjensasdtantsur: I will push the patch to drop netaddr, then look at it.13:40
TheJuliahjensas: already did13:44
openstackgerritHarald Jensås proposed openstack/ironic master: Drop netaddr - use netutils.is_valid_ipv6()  https://review.opendev.org/71811313:44
* TheJulia shrugs13:44
hjensasTheJulia: that one ^^, or the use in the dual-stack patch?13:45
TheJuliahjensas:  https://review.opendev.org/71785213:45
patchbotpatch 717852 - ironic - Replace netaddr with ipaddress - 2 patch sets13:45
dtantsurTheJulia: the ipaddress change is not backportable, unfortunately13:46
TheJuliaahh13:46
*** tzumainn has joined #openstack-ironic13:46
dtantsurnot without adding a dependency on the python 2 backport13:46
TheJuliatrue13:46
TheJuliayeah13:46
TheJuliaokay13:46
* TheJulia goes back to inspector bug13:46
dtantsurso we may need to make it two-staged13:46
hjensasI think we can take both, first min then TheJulia's. :)13:46
*** yaawang has joined #openstack-ironic13:47
dtantsurright13:48
* dtantsur will never stop dreaming of dropping oslo.utils :)13:48
* TheJulia points at code reviews13:49
TheJuliaI feel like the evil monkey from https://www.youtube.com/watch?v=fxynvzg03kM when I point out code reviews13:50
dtantsurlol13:54
iurygregorydtantsur, https://www.mememaker.net/api/bucket?path=static/img/memes/full/2020/Apr/7/13/drop-all-oslo-36622.png13:54
dtantsur:D13:54
*** jcoufal has quit IRC13:54
dtantsurto be clear, I don't suggest literally that13:54
iurygregoryI know13:54
iurygregoryXD13:54
dtantsurI'm trying to make people mindful about dependencies13:55
iurygregory++13:55
openstackgerritJulia Kreger proposed openstack/ironic-inspector master: Catch NotFound properly in the explicit inspect path  https://review.opendev.org/71811813:59
TheJuliadtantsur: ^^^ I think that is what you were hoping for? Looks like the testing is fine around that and searches the entire string in the stack trace, but since we log the exception and pass that the test still passes.14:00
dtantsurTheJulia: I don't think that's the right direction to take (although I only have gut feelings at this point)14:00
dtantsurNotFound inherits Error, so this change shouldn't change much14:01
dtantsurI suspect we don't process known exceptions correctly at the RPC boundary14:01
*** tkajinam has quit IRC14:03
TheJuliaso utils.Error is all that is expected for messaging14:03
*** tkajinam has joined #openstack-ironic14:03
TheJuliaso... I guess it should be good.. maybe?14:03
dtantsurTheJulia: NotFound inherits Error though?14:05
dtantsurI'm confused by the RemoteError bit in the traceback14:05
*** etingof has quit IRC14:06
*** yaawang_ has joined #openstack-ironic14:11
TheJuliadtantsur: NotFound is a bare declaration I think from Exception in the local ironic client14:11
TheJuliaso it would result in a nice error since that is not expected14:12
TheJuliaI thinik14:12
*** yaawang has quit IRC14:14
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Debug logging for failed disk folder mount with _install_grub2  https://review.opendev.org/71660314:15
openstackgerritArne Wiebalck proposed openstack/ironic master: [doc] Images need some metadata for software RAID  https://review.opendev.org/70192614:16
arne_wiebalckdtantsur: I tried to transform our discussion earlier into the doc patch ^.14:17
TheJuliayay rebasing patches14:22
dtantsurTheJulia: it's not a client side problem though. it comes as HTTP 500 to the client, I think because utils.Error gets translated into oslo's RemoteError.14:25
dtantsurbut I can only speculate based on the error message14:25
TheJuliagah, my brain is not functioning today14:26
TheJuliatzumainn's patch redid the dhcp unit tests quite a bit :(14:26
TheJuliadtantsur: but utils.Error is classified as remote expected error14:27
TheJuliaso something is not working right there :\14:27
*** etingof has joined #openstack-ironic14:30
tzumainnTheJulia, did my patch break something?14:31
openstackgerritHarald Jensås proposed openstack/ironic master: Fix AttributeError in check allowed port fields  https://review.opendev.org/71812714:31
hjensasdtantsur: I think ^^ should fix it. But I tried to write a test for it. create or a patch replacing llc with 'None' did'nt fail in unit test.14:33
TheJuliatzumainn: just the client mockings on unit testing for my make the pxe code friendly to dual stack configs14:36
tzumainnTheJulia, ah, let me know if I can help with anything!14:37
*** priteau has quit IRC14:37
TheJuliatzumainn: will do, I think I'll just have to have two mocks14:37
TheJuliabecause I actually have to look at the port's cconfiguration to determine what to send14:38
*** ricolin has joined #openstack-ironic14:39
*** tkajinam has quit IRC14:42
openstackgerritMark Goddard proposed openstack/tenks master: Revert "Cleanup py27 support"  https://review.opendev.org/71813614:46
*** yaawang has joined #openstack-ironic14:53
*** yaawang__ has joined #openstack-ironic14:54
*** yaawang_ has quit IRC14:56
*** mkrai has joined #openstack-ironic14:57
*** yaawang has quit IRC14:58
openstackgerritDmitry Tantsur proposed openstack/ironic master: Refactor AgentBase.heartbeat and process_next_step  https://review.opendev.org/71814215:07
* TheJulia sighs15:08
openstackgerritHarald Jensås proposed openstack/ironic master: Fix AttributeError in check allowed port fields  https://review.opendev.org/71812715:10
*** rcernin has quit IRC15:13
openstackgerritDmitry Tantsur proposed openstack/ironic master: WIP: In-band deploy steps  https://review.opendev.org/69876815:14
openstackgerritJulia Kreger proposed openstack/ironic master: "dual stack" support for PXE/iPXE  https://review.opendev.org/71421215:15
openstackgerritMerged openstack/ironic master: Nodes in maintenance didn't fail, when they should have  https://review.opendev.org/70292715:17
openstackgerritDmitry Tantsur proposed openstack/ironic master: WIP: Decompose core deploy step  https://review.opendev.org/69830915:18
*** dking_desktop has joined #openstack-ironic15:23
*** gyee has joined #openstack-ironic15:31
openstackgerritTzu-Mainn Chen proposed openstack/ironic master: Add node lessee field  https://review.opendev.org/70686415:34
openstackgerritArne Wiebalck proposed openstack/ironic master: Software RAID: Pass the boot mode to the IPA  https://review.opendev.org/69616915:39
arne_wiebalckTheJulia: ^^ is meant to address your point on the remaining UEFI RAID patch set15:40
openstackgerritDmitry Tantsur proposed openstack/ironic master: WIP: Support executing in-band deploy steps  https://review.opendev.org/69876815:54
openstackgerritDmitry Tantsur proposed openstack/ironic master: WIP: Decompose core deploy step  https://review.opendev.org/69830915:54
dtantsurthis deploy steps work is an infinite rabbit hole >_<15:55
rpittaurabbitmq hole ?15:56
dtantsur:D15:56
TheJuliaarne_wiebalck: thanks16:06
markguz_HI there, anyone here know anything about building ironic-python-agent with the ipa-builder tool? I'm having a 'mare of a time getting it to work16:08
rpittaugood night! o/16:08
*** rpittau is now known as rpittau|afk16:08
openstackgerritDmitry Tantsur proposed openstack/ironic master: WIP: Support executing in-band deploy steps  https://review.opendev.org/69876816:08
openstackgerritDmitry Tantsur proposed openstack/ironic master: WIP: Decompose core deploy step  https://review.opendev.org/69830916:08
*** lucasagomes has quit IRC16:08
*** dtantsur is now known as dtantsur|afk16:08
dtantsur|afko/16:08
TheJuliamarkguz_: what are you running into?16:08
markguz_it keeps bombing out with and error about "python -c "import requests; print requests.certs.where()"): ambiguous redirect"16:08
TheJuliamarkguz_: what are the arguments your passing and env vars your using for it if any?16:09
markguz_ironic-python-agent-builder -o my-ipa  ubuntu16:09
markguz_http://paste.openstack.org/show/791751/16:10
markguz_TheJulia: i tried the pip version of ipab, with no luck, so now i tried the git HEAD version.. same result.16:12
TheJuliamarkguz_: would you try centos8 and see if that works?16:12
markguz_already did16:12
markguz_nope16:12
markguz_centos or centos-minimal 816:13
TheJuliaI think centos defaults to centos7 unless you explicitly use centos816:13
* TheJulia installs the latest builder to see what is going on 16:14
markguz_yeah. i was supplying the --release 8 param16:14
TheJuliatry the centos8 element instead, just to be on the safe side16:14
markguz_k16:15
*** diga has quit IRC16:15
*** ociuhandu has quit IRC16:18
*** ociuhandu has joined #openstack-ironic16:19
markguz_TheJulia: "Element 'centos8' not found"16:19
markguz_it also failed with "--release 8 centos"16:19
TheJulia:(16:19
* TheJulia tries to figure out why she is running out of disk space trying to install it in ~16:20
markguz_http://paste.openstack.org/show/791753/16:20
*** mkrai has quit IRC16:20
markguz_i'm building this in a virtualenv on a CentOS 8 vm16:21
markguz_if that makes any difference16:21
markguz_python 3.616:21
TheJuliamarkguz_: it is using python2 internally :\16:37
markguz_yeah i was getting errors about python versions earlier, "different version required 2.17.17" something16:38
*** ociuhandu has quit IRC16:42
TheJuliahave you tried forcing it with the dib env var for python3?16:43
*** ociuhandu has joined #openstack-ironic16:43
markguz_hmmm. no16:45
arne_wiebalckwe use DIB_PYTHON_VERSION=316:45
rpittau|afkmarkguz_: that is really weird, I'm running it on virtualenv on vms (ubuntu bionic and centos8) and it works fine for me using: ironic-python-agent-builder -o my-ipa --release 8 centos16:45
markguz_rpittaulafk: TheJulia: i removed my other env vars and it worked for me, so one of my env vars seems to be breaking it16:46
markguz_i just tried that just now16:46
rpittau|afkgood :)16:46
TheJuliaI'm guessing the pem bit16:47
* rpittau|afk goes back in the shadows....16:47
* arne_wiebalck leaves as well16:47
arne_wiebalckbye everyone o/16:47
markguz_TheJulia: yep.. my guess too.16:48
*** ociuhandu has quit IRC16:48
markguz_hmmph. i wanted that in there... i wonder if it's because i have the root and intermediate in the same file16:48
markguz_yep. confirmed. b0rking on the pem file...16:55
TheJulia:(16:55
markguz_thanks for your help.. I can live without the pem for now.  I'll fiddle with it, see if it's a "bug" or user error16:57
openstackgerritIlya Etingof proposed openstack/ironic master: Fix gunicorn name on Py3@CentOS7 in devstack  https://review.opendev.org/71817316:57
openstackgerritMerged openstack/python-ironicclient master: Get rid of the oslo.config requirement  https://review.opendev.org/71772416:59
openstackgerritTzu-Mainn Chen proposed openstack/ironic master: Added node multitenancy doc  https://review.opendev.org/71817417:02
*** dougsz has quit IRC17:05
*** alexmcleod has quit IRC17:11
TheJuliaarne_wiebalck: thanks for the try/except catch add17:11
arne_wiebalckTheJulia: Thanks for the blueprint ;)17:13
openstackgerritHarald Jensås proposed openstack/ironic master: Fix AttributeError in check allowed port fields  https://review.opendev.org/71812717:14
TheJulia:)17:17
*** dougsz has joined #openstack-ironic17:21
*** dougsz has quit IRC17:22
*** k_mouza has quit IRC17:31
markguz_TheJulia: different question. Is it possible to set a tftp_server per network? i.e. one for the cleaning and one for the provisioning network?17:32
*** dking_desktop is now known as dking17:51
*** dking has quit IRC17:56
*** dking_desktop has joined #openstack-ironic17:57
*** dking_desktop has quit IRC17:58
*** ricolin has quit IRC18:02
*** k_mouza has joined #openstack-ironic18:13
TheJuliamarkguz_: not really, it is intended to be the conductor itself so firewalling rules . Is there a specific reason why that won't work?18:13
*** k_mouza has quit IRC18:14
markguz_if you have separate cleaning and provisioning networks then you have to allow traffic from one to flow to the other in order for the tftp server to be reachable18:14
markguz_which is not in and of itself a problem, but to my mind that kind of negates the two isolated networks thing...18:17
*** cdearborn has joined #openstack-ironic18:19
TheJuliamarkguz_: not between the networks if the conductor is on a third separate address external from the two18:36
TheJuliatwo networks that is18:36
markguz_hmm. then i need a third network that can route to both. ideally there should be no routing... at least IMHO.  anyway, i'll just stick with one clean/provide network and keep it simple18:38
TheJuliaetingof: so I'm looking at https://storyboard.openstack.org/#!/story/2007527 and wondering if we shouldn't put logic in to handle if the system is in UEFI mode, to not send a continious option at all?18:40
etingofTheJulia, does UEFI mode correlate with Continues value not being accepted?18:43
etingofwe can see if 'BootSourceOverrideEnabled@Redfish.AllowableValues' has Continues listed and apply it if it's there18:45
*** dougsz has joined #openstack-ironic18:45
etingofhowever this is confusing... at the sushy level we accept 'continues' flag, then at the redfish level we won't always pass it to the BMC...18:46
etingof*Continuous18:46
rpiosoetingof: I believe 'Continuous' can be used with UEFI. Should we ask mraineri?18:47
etingofContinuous and UEFI/Legacy boot modes - are they anyhow exclusive?18:48
etingofto a layman it appears that all the combinations make sense...18:49
* rpioso pinged mraineri out-of-band18:51
mraineriContinuous has some restrictions18:51
mraineriIt depends on the pairing of the boot override target18:52
mraineriLet me pull up the schema... It has the full restriction list...18:52
etingofconsidering all these complications, would it be easier in the end to operate on the boot order (as opposed to BootoverrideTarget)?18:53
mraineriIf you're trying to set up a permanent boot order, yes18:53
TheJuliaetingof: basically yes. Continious is not a possible option, and it seems the only non-override way to set it is via an oem-specific labeled attribute option under bios18:55
mraineriThis is one of the pieces of text in the schema: "The `Continuous` value is not supported for this property for UEFI boot source override because UEFI supports one-time boot only."18:55
rpiosoetingof, mraineri: Might the boot order require the use of  vendor-specific names for the boot sources?18:55
mraineriAnd the same rule applies to UefiBootNext18:55
TheJulialenovo's example code has 4 different attribute names it checks :(18:55
mraineriIt might, but there should also be a more well-defined enum in there to classify the type of boot option18:56
mraineriLet me look at that now...18:56
TheJuliaAnd the conundrum is we're trying to set a permanent override and it looks like the alternative options are all vendor specific without support for the field. :(18:56
mraineriThe "Alias" property will tell you if a given boot option is a CD, USB, drive, etc18:57
mraineriIt's the same enum used in the boot override property18:57
TheJuliaexcept that all seems one-time based on the 1.9 version document I've got on my other screen19:00
etingofif we go the boot-order way, my understanding is that if we tell sushy to boot from Hdd, under the hood sushy will reorder all boot devices so that those mapping to Hdd alias will appear first in the list19:03
etingofI am saying that we probably can't set the node to boot straight from the disk we have deployed the image19:04
etingofwe can only operate on "kinds" of boot media (Cd, Hdd), not concrete devices19:05
mraineriThen that might be a reason to use the AliasBootOrder property19:05
etingofyeah19:06
rpiosoetingof: How's that different from override?19:06
rpiosoAnd doesn't UEFI find the boot sources available on the drives?19:06
etingofrpioso, I think this imprecise boot media addressing problem is the same in both boot target override and boot order cases19:07
rpiosoetingof: Me, too. It's typically good enough.19:09
TheJuliaYeah, AlaisBootOrder seems to be the right way to go since we don't know the internal uefi names for the targets19:09
etingofthat's ^ what I mean19:09
TheJuliarpioso: one example from one vendor, with UefiTarget was just hexadecimle numbers, another vendor was the UEFI boot path explicitly set as if it was just queried from nvram19:10
TheJuliaso alias should provide us a nice mapping to keep... some semblance of sanity19:10
TheJuliaI joke, we can be insane all we want!19:10
rpiosoTheJulia: Not surprised :-)19:10
rpiosomraineri: Did the standard change or was there an interpretation issue?19:12
TheJuliarpioso: I think the answer is the standard evolved19:12
etingofcan we rely on the omnipresence of AliasBootOrder?19:13
* etingof hides19:13
*** Lucas_Gray has joined #openstack-ironic19:14
TheJuliaI suspect we need support read/grok/handle AliasBootOrder, and then maybe we could.... possibly19:14
etingofare you saying that we should have both mechanisms in sushy i.e. bootoverridetarget and bootorder?19:15
TheJuliayes19:15
TheJuliaand ironic will need to grok them19:16
etingofwhy can't it all be hidden in sushy?19:16
TheJuliaIt won't fix the ?somewhere between redfish 1.5 and 1.8? (TBH, I haven't figured it out yet) that the thinksystem in the bug seems to be, but something at a higher level is going to need to have enough operational context to grok and navigate it19:17
TheJuliaIt could potentially be hidable in sushy completely19:17
TheJuliapotentially19:17
etingofI do not see what kind of extra information sushy consumer (ironic) would possess that sushy does not...19:18
TheJuliaMoving to support the alias as a first class citizen and then falling back to the override should work19:19
TheJuliaThat would enable it to handle pre-1.6 and post-1.6+ for the most part19:20
TheJuliaI don't think the ThinkSystem in the storyboard link I pasted earlier will benefit from it though :(19:20
etingofI get the overall idea, I am just wondering why can't we have all these bits isolated inside sushy?19:21
jungleboyj:-(19:21
etingofcan't ThinkSystem be tackled via boot-order?19:21
*** Lucas_Gray has quit IRC19:22
TheJuliajungleboyj: maybe you can convince a bmc engineer to come hang out and chat with us one day? https://media1.giphy.com/media/Q94xQWspTUkShljj8P/giphy.gif?cid=ecf05e47a8afb1143f15bdfc2ea511ef61456beb2ec25985&rid=giphy.gif19:22
*** Lucas_Gray has joined #openstack-ironic19:22
TheJuliaetingof: you'd have to know internal names I think19:23
TheJuliathink...  looking at their examples19:23
rpiosomraineri: In the quote above from the schema, what does "UEFI boot source override" mean?19:23
etingofTheJulia, what about AliasBootOrder in that ThinkSystem?19:24
jungleboyjAHHHH The Cuteness!19:24
TheJuliaetingof: https://github.com/lenovo/python-redfish-lenovo/blob/master/examples/lenovo_set_bios_boot_order.py#L6819:24
TheJuliaetingof: no mention at least example wise of alias support19:24
jungleboyjThat would assume that I could figure out how to find the right developer.  :-)19:24
TheJuliajungleboyj: we promise more cuteness19:24
TheJuliajungleboyj: no memo-list like thing there? :)19:25
TheJuliahttps://media1.giphy.com/media/TkBoNth0Ps3Vm/giphy.gif?cid=ecf05e47a8afb1143f15bdfc2ea511ef61456beb2ec25985&rid=giphy.gif <-- similar gif of a corgi doing the same thing19:25
jungleboyjNOOOOOOO!  A Corgi....ahhh19:26
TheJuliais this a corgiphobia?19:26
larsksYou folks are incorgi-able.19:26
jungleboyjNot that ... They are just too cute!19:26
etingoflet's place these royal animals into github issue for python-redfish-lenovo19:26
jungleboyjI actually used to know who was leading up RedFish support for Lenovo but he left.  Let me check here.19:27
TheJuliagremlin seeks redfish happiness https://usercontent.irccloud-cdn.com/file/wXGctIut/IMG_20200407_122945.jpg19:31
jungleboyjYou are killing me.19:32
TheJuliaheh :)19:32
jungleboyjOur dog is a Longhaired Dachshund.  Similar body build, less log, more sausage like.19:33
jungleboyjWe have a couple cute corgis in the neighborhood.19:33
TheJuliaHe is just starting to do the independent ear tracking of sounds, I was kind of hoping for a "Corgi speaking with the Corgi mothership!" sort photo, but those will come soon enough19:34
*** derekh has quit IRC19:39
TheJuliaetingof: we also need to do the bios attribute BootMode too, not just the override19:41
etingofyeah, fun19:42
etingofbut how do we identify the right bios attribute?19:42
etingofis it always BootMode across the fleet?19:43
openstackgerritMerged openstack/ironic master: Add node lessee field  https://review.opendev.org/70686419:44
TheJuliayeah, looks like it is oem-definable19:45
TheJulialenovo's example has a few different various ones19:45
etingofso in that light, may be we should abandon all hope for uniform boot management and go the vendor-specific sushy extensions way?19:47
TheJuliaWell, maybe not because I think that might be too heavy wait and to heavy handed if it is uniformly in the attributes field19:47
TheJuliaIf someone else has stuff in Oem, there is nothing we can do19:48
jungleboyjTheJulia:  Does this help?19:48
jungleboyjfor reference, in our cluster solution we just use efibootmgr in linux local for persistent changes, and only ever use the 'Once' bootoverride for getting to that point19:48
TheJuliajungleboyj: maybe, that kind of ties in with the direction we're already heading. If UEFI boot mode, perhaps we just never try to set persistency with ironic ever19:49
jungleboyjSo, Jarrod is a font of knowledge and has already been down this path.19:50
TheJuliaOhhhh idrac9 supports secure boot key setting via redfish19:52
TheJuliaI'm not seeing an equivelent setting to the attribute value in Dell's docs.... still looking though19:53
*** ociuhandu has joined #openstack-ironic19:54
TheJuliayeah, it might be treated as an explicit bios setting without the override :\ Dell folks llikely know way better than my quick glance over their doc19:55
* TheJulia looks up ilo19:56
rpiosoTheJulia: There's a BIOS attribute for setting the boot mode.19:56
TheJuliarpioso: an explicit attribute under the attributes field?19:56
* rpioso looks19:57
openstackgerritTzu-Mainn Chen proposed openstack/ironic master: Added node multitenancy doc  https://review.opendev.org/71817419:57
rpiosoTheJulia: Yes, an explicit attribute. It's creatively named BootMode.19:59
TheJuliarpioso: \o/ for two vendors doing it the same way!19:59
* TheJulia dances19:59
jungleboyjMadness.20:00
TheJuliaIt _looks like_ HPE has a DefaultBootOrder array like the alias target setting20:00
*** ociuhandu has quit IRC20:00
TheJulia\o/ three vendors!20:01
TheJuliaBios attribute BootMode is in the ilo reference20:01
TheJulia\o/20:01
rpiosoTheJulia: How about removing the syncing of boot mode from preparing to boot the ramdisk and instance? Could it be a separate deploy step.20:01
jungleboyjTheJulia: I think you have talked to Jarrod before.  Going to send an introductory e-mail for you, him and the BMC developer.20:02
TheJuliajungleboyj: thanks20:02
jungleboyjAnytime.  I am going to have to get to meet the Corgi some time though.  :-)20:02
TheJuliarpioso: I think it is more an issue than just syncing boot mode, and eventual deploy step makes sense, but I'm wondering if there is a backportable way we can make this better20:03
TheJuliaAnd I think in part because we're seeing a whole blend of issues around the same area. Setting the boot mode, and setting what should be the preferred boot device because people seem to not want to be able to start with anything and end up in a consistent desirable end state, and that is kind of where these differences seem to start causing issues.20:05
*** derekh has joined #openstack-ironic20:13
*** derekh has quit IRC20:14
*** Lucas_Gray has quit IRC20:15
*** Lucas_Gray has joined #openstack-ironic20:21
openstackgerritOpenStack Release Bot proposed openstack/tenks stable/1.0: Update .gitreview for stable/1.0  https://review.opendev.org/71822020:37
openstackgerritOpenStack Release Bot proposed openstack/tenks stable/1.0: Update TOX/UPPER_CONSTRAINTS_FILE for stable/1.0  https://review.opendev.org/71822120:37
openstackgerritOpenStack Release Bot proposed openstack/tenks master: Update master for stable/1.0  https://review.opendev.org/71822220:37
*** Lucas_Gray has quit IRC20:42
*** Lucas_Gray has joined #openstack-ironic20:44
*** dougsz has quit IRC20:45
*** igordc has quit IRC20:47
*** igordc has joined #openstack-ironic20:48
*** stevebaker has quit IRC21:01
*** stevebaker has joined #openstack-ironic21:09
rpiosoTheJulia: The first thing we do is set the boot mode. That's before the server is given to ironic. The boot mode affects 1) which BIOS attributes are available, 2) how a NIC port is configured to PXE boot, and 3) how the boot order is expressed.21:11
rpiosoKinda key.21:12
openstackgerritMerged x/sushy-oem-idrac master: Bump hacking to 3.0.0  https://review.opendev.org/71590021:27
*** ociuhandu has joined #openstack-ironic21:34
TheJuliarpioso: yeah, I wish I had a nice chart detailing variations across vendors though.21:34
*** ociuhandu has quit IRC21:39
*** rcernin has joined #openstack-ironic22:11
*** Lucas_Gray has quit IRC22:36
openstackgerritJulia Kreger proposed openstack/sushy master: WIP: direct boot order setting support  https://review.opendev.org/71827522:45
openstackgerritJulia Kreger proposed openstack/sushy master: WIP: Add an option for setting the http boot uri  https://review.opendev.org/71827622:45
TheJuliaetingof: idea ^^22:45
TheJuliaerr, ideas22:46
*** tkajinam has joined #openstack-ironic22:56
*** yaawang__ has quit IRC23:16
*** yaawang__ has joined #openstack-ironic23:17
*** tzumainn has quit IRC23:37

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!