Wednesday, 2021-01-13

*** tosky has quit IRC00:01
*** tosin has quit IRC00:10
*** ianychoi has joined #openstack-ironic00:14
jandersguilhermesp I am indeed working on NVMe cleaning - please give me more info on your goals/requirements and I may be able to help00:40
TheJulialbragstad: to emulate the cache, with making the suggested change in https://review.opendev.org/c/openstack/ironic/+/768135/1/ironic/tests/unit/api/test_acl.py am I dumping the values out to assemble the cache at that point? Since I can't convert the object to json, and there is not a logical path there, I'm a bit blocked at this point, so any clarity would be super helpful..01:05
TheJuliaJayF: I think metal3 is running under uwsgi01:06
TheJuliabut... braaaaaiiinnssss01:06
TheJulialbragstad: I guess a good example would be useful if you happen to know of one at this point because I'm not sure how I'm sure your suggestion is actually possible to implement.01:07
TheJuliadtantsur|afk: I responded to your questions on https://review.opendev.org/c/openstack/ironic/+/768136/6 of note specifically is explicitly enforcing mocked specific values to be returned in a case where we know the conductor is not present to respond (i.e. assert 503 instead of !403) That is going to be problematic for QE folks trying to do external testing. If you can circle back to that tomorrow and at01:12
TheJulialeast provide comments in the review to that end, that would be helpful. I'll likely get to that either tomorrow or thursday at this rate01:12
*** tkajinam has quit IRC01:41
*** tkajinam has joined #openstack-ironic01:42
eanderssonHow do I get a password using the IPA?02:04
eanderssone.g. for firmware upgrade :D02:04
eanderssonTrying to figure out why we failed to login, but seems like the password is being replaced with stars02:04
eanderssonor nvm maybe we are looking at the wrong api :D02:05
eanderssonWas hitting 9999 manually for debugging but of course it obscures the password :D02:09
*** tkajinam has quit IRC02:09
*** tkajinam has joined #openstack-ironic02:10
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: Automaticaly set cipher suite  https://review.opendev.org/c/openstack/ironic/+/77052602:18
*** ccstone has quit IRC02:26
*** ccstone has joined #openstack-ironic02:26
eanderssondriver_info.get('test_password') does indeed just store ***** on my hardware_manager02:28
eanderssonI assume the obvious workaround is to not name it password?02:28
*** zzzeek has quit IRC02:30
*** zzzeek has joined #openstack-ironic02:31
eanderssonOr maybe it just isn't safe to store a password in driver_info for use of the IPA?02:33
*** zzzeek has quit IRC02:36
*** zzzeek has joined #openstack-ironic02:40
lbragstadTheJulia by cache do you mean test values?02:43
lbragstade.g., BASIC_CACHE_VALUES (it looks like)02:44
lbragstadTheJulia one of the better examples i can think of is what we did in nova and placement02:45
lbragstadwe simulated the functionality of keystone_authtoken by populating headers that model the various personas02:46
lbragstadthen - oslo.context translates those headers to a context object02:46
lbragstadwhich removes the dependency on keystone via keystone_authtoken (to validate the token)02:47
lbragstadbut it still exercises the functionality of oslo.context by using .from_environ() to create context objects02:50
lbragstadthe tests would still exercise this code - https://opendev.org/openstack/ironic/src/branch/master/ironic/api/hooks.py#L9702:51
lbragstadanother advantage is that the placement and nova test code doesn't carry around fake v2.0 or v3 token references02:52
lbragstadso - it's an implementation detail of keystone you don't have to care about02:52
*** Qianbiao has joined #openstack-ironic02:52
lbragstadi hope that helps answer your question - but i'll follow up tomorrow when we're both online and we can talk through it if i'm not making sense02:53
lbragstadi didn't get around to the ironic specification today either, but i plan to do that tomorrow02:53
*** rcernin has quit IRC02:57
*** mkrai has joined #openstack-ironic03:04
guilhermesphi janders thanks for the heads up!   I havent had much time today to test, but right now I have around 4 nodes with one nvme disk each. It is for a specific customer, those nodes wont be shared to different customers so it's not a priority right now ( as the cloud runs with no automated_cleaning ) but should be a requirement in a near future as the nodes can be shared across different tenants.03:08
guilhermespI though about replacing/adding support to nvme-cli package https://github.com/linux-nvme/nvme-cli  , I can totally find a slot to test securely erasing nvme devices so we can come up with the right commands. Do you have any wip to share ?03:10
*** rcernin has joined #openstack-ironic03:18
*** rcernin has quit IRC03:21
*** rcernin has joined #openstack-ironic03:21
TheJulialbragstad: not exactly answers it. I don't think I correctly conveyed my confusion with fake cache population03:39
*** ricolin_ has joined #openstack-ironic03:39
TheJuliaeandersson: Wait... you need IPA to have access to the secrets... or to somehow to be told the secrets to locally execute a firmware upgrade?03:40
TheJuliaeandersson: fwiw, anything with password or token in the field name will be obscured by the oslo policy settings that are by default03:40
*** ricolin_ has quit IRC03:54
jandersguilhermesp I should have something ready for you to look at in the coming days. Meanwhile, can you run these two commands and paste output somewhere (or email to janders@redhat.com)? http://paste.openstack.org/show/801580/03:55
jandersmy current design is along the lines of enabling the current storage cleaning code to figure out if a device is an NVMe or a SATA, and utilising type-specific cleaning (nvme-cli sanitise/format for NVMe or SATA secure erase for SATA) with optional fallback to shred should the "native" cleaning fail03:56
jandersso if I understand correctly that should do exactly what you need03:57
eanderssonTheJulia: Yea - the IPA plugin we use needs a username/plugin to perform a firmware upgrade locally. We tried to make everything generic enough and wanted all firmware upgrades to be handled in the IPA, but I think that maybe it's just a bad fit and the part that needs a username/pw should probably just be done in Ironic as a plugin.03:59
eanderssonSince it can be done remotely as well.04:05
eanderssonIt was really done as an attempt to stay consistent with how we do firmware upgrade for all hardware types.04:05
*** Qianbiao has quit IRC04:14
*** Qianbiao has joined #openstack-ironic04:19
TheJuliaeandersson: yeah, oob has the added benefit if likely being more secure in the grand scheme of the universe04:19
*** rcernin has quit IRC04:35
*** rcernin has joined #openstack-ironic04:35
*** pmannidi has quit IRC04:53
TheJulialbragstad: re-reading what your saying, I guess I'm feeling even more confused because I thought the semi-ideal path for a service such as ironic was to fake the cache  out, and then let the policy check drive all the way down into policy validation using cached token records, but it sounds like the testing model your saying that was actually went with was to mock out the middleware so it never even gets that04:57
TheJuliafar. Which I guess might make sense if only attempting to assemble one thing, return one aspect replacing the mock each time (fetch_token?)for every test as opposed to just repeatedly leveraging a common cache. Is any of that making any sense?04:57
TheJulialbragstad: maybe less confused at this point in the grand scheme, just feeling like mixed messaging has occured.05:03
*** gyee has quit IRC05:08
*** tzumainn has quit IRC05:14
*** Qianbiao has quit IRC05:24
*** rcernin_ has joined #openstack-ironic05:42
*** rcernin has quit IRC05:42
arne_wiebalckGood morning, ironic!06:47
jandersgood morning arne_wiebalck o/06:49
arne_wiebalckhey janders, good morning o/06:50
*** mkrai has quit IRC06:54
*** mkrai has joined #openstack-ironic06:54
*** Qianbiao has joined #openstack-ironic06:58
*** mkrai has quit IRC07:13
*** zzzeek has quit IRC07:28
*** rcernin_ has quit IRC07:28
*** zzzeek has joined #openstack-ironic07:31
*** openstackgerrit has quit IRC07:47
*** JayF has quit IRC07:52
*** mkrai has joined #openstack-ironic07:52
*** rcernin_ has joined #openstack-ironic08:06
*** mkrai has quit IRC08:13
*** akahat|rover is now known as akahat|lunch08:25
*** rpittau|afk is now known as rpittau08:25
rpittaugood morning ironic! o/08:25
*** rcernin_ has quit IRC08:26
rpittaujanders: for the nvme erase, I don't have anything to add to waht dtantsur said, I'll check the patches as they evolve though :)08:26
*** tosky has joined #openstack-ironic08:39
*** mkrai has joined #openstack-ironic08:44
jandersgood morning rpittau o/08:55
jandersrpittau thank you, sounds good! :)08:55
*** lucasagomes has joined #openstack-ironic09:05
*** derekh has joined #openstack-ironic09:35
*** openstackgerrit has joined #openstack-ironic09:36
openstackgerritAija Jauntēva proposed x/sushy-oem-idrac master: Update export system configuration  https://review.opendev.org/c/x/sushy-oem-idrac/+/75942509:36
*** hoonetorg has joined #openstack-ironic09:44
openstackgerritDerek Higgins proposed openstack/ironic master: Add a delay/retry is vmedia insert fails  https://review.opendev.org/c/openstack/ironic/+/77027009:51
*** quoth has joined #openstack-ironic09:57
*** akahat|lunch is now known as akahat|rover10:04
openstackgerritDerek Higgins proposed openstack/ironic master: Add a delay/retry is vmedia insert fails  https://review.opendev.org/c/openstack/ironic/+/77027010:17
*** dougsz has joined #openstack-ironic10:36
*** dougsz has quit IRC10:37
*** ociuhandu has joined #openstack-ironic10:40
*** Qianbiao has quit IRC10:40
*** dtantsur|afk is now known as dtantsur10:41
dtantsurmorning ironic10:41
rpittauhey dtantsur :)10:46
dtantsurTheJulia: what I strongly dislike is us inventing a testing framework in yaml, and we're half way through it.10:49
*** abdysn has joined #openstack-ironic11:01
iurygregorymorning arne_wiebalck dtantsur janders rpittau and Ironic o/11:03
rpittauHey iurygregory :)11:05
iurygregoryo/11:05
*** sshnaidm|afk is now known as sshnaidm|ruck11:19
*** mkrai has quit IRC11:26
*** ociuhandu has quit IRC11:36
jandershey iurygregory o/11:45
iurygregoryo/11:48
openstackgerritDmitry Tantsur proposed openstack/sushy master: [WIP] Secure boot support  https://review.opendev.org/c/openstack/sushy/+/77060712:04
*** tosin has joined #openstack-ironic12:39
*** ociuhandu has joined #openstack-ironic12:45
*** rh-jelabarre has joined #openstack-ironic12:53
*** ociuhandu has quit IRC12:55
*** ociuhandu has joined #openstack-ironic12:56
*** ociuhandu has quit IRC12:56
*** ociuhandu has joined #openstack-ironic13:01
lbragstadTheJulia i think what you're saying makes sense - but instead of using cached tokens directly, just set the request headers directly and simulate keystone_authtoken middleware13:05
lbragstadTheJulia we weren't prescriptive in telling people how to test their secure RBAC changes, because some service might have things that work for them13:06
guilhermespmornings!13:13
guilhermespnice janders ! yep that's what i was imagining how it could be implemented! I will reproduce the commands on your paste today and let you know the results13:14
openstackgerritAija Jauntēva proposed x/sushy-oem-idrac master: Add import system configuration  https://review.opendev.org/c/x/sushy-oem-idrac/+/77061413:17
*** paras333 has quit IRC13:26
openstackgerritAija Jauntēva proposed openstack/python-ironicclient master: Add 'deploy steps' for  provisioning API  https://review.opendev.org/c/openstack/python-ironicclient/+/76835413:29
guilhermespsome outputs janders http://paste.openstack.org/show/801590/13:29
openstackgerritDmitry Tantsur proposed openstack/sushy master: [WIP] Secure boot support  https://review.opendev.org/c/openstack/sushy/+/77060713:30
*** juanoterocas has joined #openstack-ironic14:18
openstackgerritAija Jauntēva proposed x/sushy-oem-idrac master: Add import system configuration  https://review.opendev.org/c/x/sushy-oem-idrac/+/77061414:21
*** rloo has joined #openstack-ironic14:27
*** mkrai has joined #openstack-ironic14:40
*** tzumainn has joined #openstack-ironic14:47
openstackgerritAija Jauntēva proposed x/sushy-oem-idrac master: Add import system configuration  https://review.opendev.org/c/x/sushy-oem-idrac/+/77061414:55
*** ociuhandu_ has joined #openstack-ironic14:58
*** ociuhandu has quit IRC15:02
*** MentalSiege has joined #openstack-ironic15:05
*** abdysn has quit IRC15:08
*** MentalSiege has quit IRC15:09
*** MentalSiege has joined #openstack-ironic15:10
openstackgerritDerek Higgins proposed openstack/ironic master: Add a delay/retry is vmedia insert fails  https://review.opendev.org/c/openstack/ironic/+/77027015:30
*** ociuhandu_ has quit IRC15:35
*** ociuhandu has joined #openstack-ironic15:35
*** juanoterocas has quit IRC15:37
*** ricolin has quit IRC15:38
openstackgerritDmitry Tantsur proposed openstack/sushy master: Secure boot support: enabling/disabling and resetting keys  https://review.opendev.org/c/openstack/sushy/+/77060715:42
*** ricolin has joined #openstack-ironic15:47
TheJulialbragstad: the initial internal stuff was at least suggestive, but it doesn't really matter. Anyway, back to banging my head against this.16:00
lbragstadTheJulia i'm on a call but i'll review the spec after16:01
TheJulialbragstad: reviewing spec has nothing to do with attempting to get pre-change testing into a workable/usable state to help preent breaking changes16:02
lbragstadtrue- but maybe i can help with the testing changes after that16:02
TheJuliaThat would actually be worse16:02
TheJuliabecause it is chicken/egg16:03
TheJuliathe spec doesn't cover the previous state testing16:03
lbragstadok - so you're saying you need help with testing more than you need a spec review?16:03
*** mkrai has quit IRC16:03
*** mkrai_ has joined #openstack-ironic16:03
TheJuliait would be greatly appreciated, although I may be on the right track at this point trying to incorporate review feedback. Maybe..16:04
lbragstadok16:04
*** rnoriega_ is now known as rnoriega12316:06
*** rnoriega123 is now known as rnoriega_16:06
*** rnoriega_ is now known as rnoriega16:09
*** ociuhandu_ has joined #openstack-ironic16:11
*** ociuhandu has quit IRC16:11
*** mkrai_ has quit IRC16:22
*** markguz_ has joined #openstack-ironic16:27
TheJuliaAll: Midcycle poll: https://doodle.com/poll/y9afrz6hhq7s23km?utm_source=poll&utm_medium=link16:27
TheJuliaTopic Etherpad: https://etherpad.opendev.org/p/ironic-wallaby-midcycle16:28
markguz_Hello I've been struggling with a weird problem for the last few weeks with ironic.  Essentially whenever I spawn a baremetal instance the process starts, scheduler schedules it, and the nova-compute on the ironic conductor node reports "Starting instance". Then nothing for 10mins or sometimes 20mins, then suddenly the ironic conductor will start the node16:30
markguz_No error is reported, no tracebacks no nothing. it just sits there and does nothing and suddenly springs into life.  This only happens with baremetal. VMs spawn as good as instantly16:31
markguz_I am beginning to lose my mind trying to debug this... . This is Rocky btw16:32
TheJuliamarkguz_: concurrent baremetal deployments?16:32
TheJuliamarkguz_: so, how big are the disk images your asking for to be deployed?16:32
markguz_TheJulia: There's no other activity on the baremetal. This is not a busy deployment.16:33
TheJuliathen how big are the disk images? I suspect they are being converted to raw format or something along those lines16:33
markguz_TheJulia: Images are already RAW. and the images are between 800M and 3G depending on which image is chosen16:35
markguz_And this worked ok until i upgraded from queens16:35
TheJuliawhat kind of hardware?16:36
markguz_I was planning on upgrading all the way to Stein at the time, but stalled out when Ironic stopped working properly16:36
TheJuliaare you using ipmi?16:36
markguz_TheJulia: yes16:36
TheJuliatry downgrading your ipmitool binary to an older version16:36
*** MentalSiege has quit IRC16:37
TheJuliathere were crypto changes in the last ?year? which doesn't align with HPE hardware16:37
TheJuliaat least, in some forks of it16:37
markguz_TheJulia: The thing is. I don't see any action from ironic conductor that would make me think there's a problem with ipmi.16:37
TheJuliaare there hanging ipmitool processes though?16:38
markguz_No16:38
TheJuliahmm16:38
markguz_"ipmitool version 1.8.18"16:38
TheJuliaokay, so you trigger a deployment on the nova side. Do we immediately go to deploy wait in ironic?16:39
markguz_no.16:39
markguz_that's what i'm confused about. nova-compute says "Starting" then literally NOTHING happens for at least 10mins16:39
TheJuliais nova even hitting the API quickly?16:39
TheJuliais there lots of VM activity on this cluster?16:39
markguz_no very little. we can go days with out spawning a vm16:40
TheJuliaanything in the nova-compute process log that lines up with this?16:40
*** adrianc has quit IRC16:41
markguz_nothing. zip, nada.  it's been driving my crazy.16:41
TheJuliahmm16:41
*** tosky has quit IRC16:41
TheJuliahave you asked the nova folks what could cause a delay in nova-compute getting the command to spawn an instance?16:42
*** adrianc has joined #openstack-ironic16:42
*** tosky has joined #openstack-ironic16:42
markguz_no. not yet. since nova is reporting "Starting instance... _do_build_and_run_instance" i really thought it was something to do with ironic16:43
* TheJulia cracks open the nova code base16:44
markguz_I was trying to decipher what nova compute manager does from there.  But i couldn't quite figure out what was happening and where it was called from16:44
TheJuliaif memory serves, it is a comamnd over rpc16:45
TheJuliait is *super* hard to follow16:45
markguz_that's what I thought. and yes it is super super hard to follow :-)16:46
markguz_i know rabbitmq is working ok. everything else in the deployment is working ok.16:46
TheJuliamarkguz_: does it say "building" for the vm_state?16:48
markguz_yep16:48
TheJuliawhat does task_state say?16:49
TheJuliahow quickly after triggering instance deployment are you seeing "Start spawning the instance on the hypervisor." inside the nova-compute logs?16:51
markguz_Task state is "None"16:52
markguz_and i see the start pretty much immediatly16:52
markguz_here's my nova.conf http://paste.openstack.org/show/801598/16:52
markguz_nova-compute for the ironic stuff i mean.16:52
*** lucasagomes has quit IRC16:53
TheJuliayeah16:53
TheJuliamarkguz_: so my theory, based upon the nova rocky code base, is what is happening is your stuck scheduling for some reason16:55
TheJuliabeause we're not actually hitting the driver spawn yet given the task state is also none. This is likely best for the nova experts in #openstack-nova, but honestly this is also first I've ever heard of this16:55
TheJuliawell, an issue quite like *this*16:56
markguz_here's the debug output http://paste.openstack.org/show/801599/16:58
*** gyee has joined #openstack-ironic17:00
*** MentalSiege has joined #openstack-ironic17:01
markguz_you see goes from 9.40 then 9.46 then 10.05 when the actual process finally starts17:01
*** ociuhandu_ has quit IRC17:06
*** ociuhandu has joined #openstack-ironic17:09
*** ociuhandu_ has joined #openstack-ironic17:13
*** MentalSiege has quit IRC17:15
TheJuliawow17:16
*** ociuhandu has quit IRC17:16
markguz_Wow?17:17
*** ociuhandu_ has quit IRC17:17
*** MentalSiege has joined #openstack-ironic17:18
TheJuliaI'm at a loss17:20
markguz_well. i don't feel so bad now :-)17:21
markguz_i'm usually pretty good at ferreting out these little problems but this is a stumper17:21
markguz_what i wish was that the debug was a bit more verbose.... log prints for every step of the way. like "doing this" "done with that" "now doing this" "now done with that"17:22
*** ociuhandu has joined #openstack-ironic17:22
markguz_but it seems to skip over a huge amount of steps17:22
TheJulia are you supplying a specific vif?17:24
TheJuliais this cinder integrated deployments?17:25
markguz_this is how i spin up a vm. http://paste.openstack.org/show/801600/17:25
markguz_no whole disk image17:25
markguz_so storage is noop and network is noop17:25
*** ociuhandu has quit IRC17:27
TheJuliamarkguz_: what does your ironic-api log look like?17:28
TheJuliaI suspect we're getting hung up in here https://github.com/openstack/nova/blob/34c5df7b2b8c7eebbfc8aa0bc3edba501dddcb61/nova/virt/ironic/driver.py17:29
*** sshnaidm|ruck is now known as sshnaidm|afk17:30
TheJuliacan you try with a specific vif, and not a network id?17:30
markguz_i used to do that back in the days of liberty. then something changed in pike or ocata and i had to stop doing that.17:32
tosinHi TheJulia, could you please check out my change https://review.opendev.org/c/openstack/ansible-collections-openstack/+/77003717:54
TheJuliatosin: might be good, do you have a change in bifrost to test that ?17:59
tosinTheJulia: yes it passed bifrost18:02
TheJulialink to that change?18:03
TheJuliaor log?18:03
rpittaugood night! o/18:05
*** rpittau is now known as rpittau|afk18:05
openstackgerritMonica Pardhi proposed openstack/ironic stable/victoria: DRAC: Fix a failure to create virtual disk  https://review.opendev.org/c/openstack/ironic/+/77065518:05
*** derekh has quit IRC18:07
TheJulialbragstad: okay, trying to create contexts and then trying to mock the keystonemiddleware token handling seems to be hitting roadblocks. Do you happen to have a link to where this is being done in nova or any other project? I might be trying to do things too simply or maybe I need to manually map everything, just not sure.18:09
lbragstadTheJulia sure - let me grab an example18:10
lbragstadnova mocked the context objects directly - then supplied a specific context object for the test18:10
TheJulialbragstad: much appreciated18:10
viks____TheJulia:  after i run `provide` command,  the node gets rebooted and i see dhcp request being sent from baremetal node via KVM console. I'm also able to see dhcp messages coming to eth1 i.e. second interface, and br-eth1.. but it seems baremetal node is not getting any dhcp reply back... any idea what i should be looking in to next?18:10
lbragstadplacement provided request environment headers and allowed oslo.context middleware to translate those headers into a context object (so no oslo.context mocking was necessary)18:11
TheJuliaand here I'm wondering why we even still have the enviroment header stuff wired in place18:12
* TheJulia sighs18:12
TheJuliaviks____: have you verified your mac addresses? Is the dhcp server in the namespace that should be getting the request, is it seeing the packets?18:12
TheJuliaviks____: i.e. tcpdump in the namespace past the br-eth118:12
tosinTheJulia: https://zuul.opendev.org/t/openstack/build/b1585da30a7248b19dc2fa168b04408d18:13
tosinhttps://zuul.opendev.org/t/openstack/build/fa2844d0458d411d9670ff86a4e1799f18:13
lbragstadTheJulia this is where that code lives in nova https://opendev.org/openstack/nova/src/branch/master/nova/tests/unit/policies and all those tests ultimately inherit from base.py18:13
lbragstadhttps://opendev.org/openstack/nova/src/branch/master/nova/tests/unit/policies/base.py#L4118:13
openstackgerritMonica Pardhi proposed openstack/ironic stable/ussuri: DRAC: Fix a failure to create virtual disk  https://review.opendev.org/c/openstack/ironic/+/77027718:13
lbragstadand the test implementation https://opendev.org/openstack/nova/src/branch/master/nova/tests/unit/policies/test_servers.py#L36818:14
TheJuliatosin: I was thinking something along the lines of https://review.opendev.org/c/openstack/bifrost/+/769495 where validation is toggled by the playbook so the logs can be observed18:14
lbragstadplacement is pursuing this approach https://review.opendev.org/c/openstack/placement/+/760240/13/placement/tests/functional/gabbits/resource-provider-secure-rbac.yaml18:15
viks____TheJulia: You mean to say the MAC id for `openstack baremetal port create` command... i think i have given it properly.. i gave the one with the PXE boot interface MAC id...18:15
openstackgerritMonica Pardhi proposed openstack/ironic stable/victoria: DRAC: Fix a failure to create virtual disk  https://review.opendev.org/c/openstack/ironic/+/77065518:15
viks____TheJulia: dhcp server in the namespace i see a tap interface, in which i do not see any dhcp requests coming in when checked with tcpdump18:16
lbragstadwhich replaces keystone_authtoken middleware with a "noauth" middleware https://review.opendev.org/c/openstack/placement/+/760240/13/placement/tests/functional/fixtures/gabbits.py#7418:17
lbragstadand that is implemented here - https://review.opendev.org/c/openstack/placement/+/760240/13/placement/auth.py#3518:17
TheJuliaviks____: ahh, then it is wiring in neutron. Neutron isn't attaching the network properly18:18
viks____TheJulia:  i see it only on  `eth1` which is second interface and `br-eth1`18:18
TheJuliaviks____: likely... your issue is the bridge mapping is wrong or missing18:19
TheJuliaand the network is not configured to use the bridge mapping properly for the barmeetal network. I'd go back to the examples and follow them very carefully in that area to see if something was missed18:19
openstackgerritMonica Pardhi proposed openstack/ironic stable/ussuri: DRAC: Fix a failure to create virtual disk  https://review.opendev.org/c/openstack/ironic/+/77027718:19
TheJuliaAlso along with the steps to create the network.18:19
tosinTheJulia: Oh I'll do that now18:19
TheJuliatosin: awesome18:19
viks____TheJulia:  bridge mapping in `/etc/neutron/plugins/ml2/openvswitch_agent.ini`?18:22
TheJuliaviks____: yes18:24
viks____TheJulia: it's mapped to `eth1`(second interface) which seems to be fine??18:25
viks____TheJulia: also step 12 in https://docs.openstack.org/ironic/train/install/configure-networking.html needs to be created in any specific NETWORK_CIDR?18:27
TheJuliaviks____: it is format physical_network_alias:bridge_mapping_interface(like br-eth1)18:34
TheJuliaviks____: not the raw interface18:34
TheJuliaviks____: so then the question is, how did you create the network for cleaning/deployment?18:34
TheJuliawith what parameters18:35
TheJuliawhat matters in that is the physical network provider18:35
openstackgerritTosin Farai proposed openstack/bifrost master: Test new features added to baremetal_node  https://review.opendev.org/c/openstack/bifrost/+/76949518:39
TheJulialbragstad: I guess the conundrum is use case evolution being so different at this point :\18:42
*** dtantsur is now known as dtantsur|afk18:43
dtantsur|afko/18:43
TheJuliagoodnight dtantsur|afk18:44
viks____TheJulia: sorry.. my mistake.. my mapping is set to `br-eth1`only..  i created single network as per step 12 in above link and using the same for provisioning and cleaning  as of nw.. but there seems to be some mismatch in `physical_network_alias` b/n `openvswitch_agent.ini` and `ml2_conf.ini`... i'll check that now18:45
TheJuliaviks____: ahh, those need to match if memory serves18:54
TheJulialbragstad: good news! only one test failing now :)18:54
lbragstadTheJulia what patch are you working from?18:55
TheJuliauploading18:55
openstackgerritJulia Kreger proposed openstack/ironic master: WIP Allow different token caches for policy tests  https://review.opendev.org/c/openstack/ironic/+/76813518:55
TheJuliait still needs a lot of cleanup unfortunately ^^^18:56
lbragstadTheJulia iiuc - your testing approach will hit issues when https://bugs.launchpad.net/keystonemiddleware/+bug/1777177 is fixed19:03
openstackLaunchpad bug 1777177 in keystonemiddleware "eliminate v2 keystone support" [Low,In progress] - Assigned to Gage Hugo (gagehugo)19:03
lbragstadif/when19:03
TheJuliahow?19:04
TheJuliaand I ask that as not being a keystone expert19:04
* TheJulia starts looking through the linked patches to gain context19:05
* lbragstad grabs some links19:06
viks____TheJulia: can you plz let me know if  step 12 in https://docs.openstack.org/ironic/train/install/configure-networking.html needs to be created in any specific NETWORK_CIDR?19:06
viks____also what step5 is for in https://docs.openstack.org/ironic/train/install/configure-pxe.html#dhcp-server-setup ?19:07
TheJulialbragstad: yeah, I'm not sure why that would break this, but maybe it is a patch I'm just not seeing?19:07
TheJuliaviks____: afaik, no specific network cidr is required as long as it is valid19:07
TheJuliaviks____: step 5 is so the kernel sends the reply on the bridge instead of the base ethernet interface19:08
TheJuliaa common setup issue19:08
viks____TheJulia: what that IP should be ?19:09
TheJuliait should match the addressing your using for your provisioning/cleaning networks19:10
TheJuliaor flat network upon which your provisioning the nodes19:10
TheJuliaall of which is essentially the same $thing19:11
viks____TheJulia: which means that it should be in same addressing as in the subnet created in the earlier step 12?19:13
openstackgerritJulia Kreger proposed openstack/ironic master: WIP Allow different token caches for policy tests  https://review.opendev.org/c/openstack/ironic/+/76813519:13
TheJulialbragstad: ^^^ so, had to change the test because we're not actualy doing full authentiation checks which voids out my resistance to dmitry's ask in a later patch for precise response codes19:14
TheJuliaviks____: which step 12?19:15
TheJuliaI don't actually see a step 1219:15
lbragstadTheJulia ok - checking again19:15
viks____TheJulia:   https://docs.openstack.org/ironic/train/install/configure-networking.html19:15
lbragstadi need to pull this down and test it locally19:16
TheJulialbragstad: no worries, more cleanup will be needed, since I fumbled through it. Like... I literally don't know why one of the methods is getting three arguments instead of two as defined.19:17
* TheJulia suspects it is the mystery of the century19:17
TheJuliaviks____: yes, same addressing because the conductor needs to try and know where to send traffic back to19:17
TheJuliathat is for tftp specifically, that shouldn't break dhcp since neutron if everything is configured right should enable the network plumbing such that the dhcp server will see the request and then respond to it19:18
viks____Thanks for your inputs.. will investigate further why tap interface in the network namespace(dhcp) not getting the dhcp traffic...19:19
* TheJulia goes and makes some lunch19:22
markguz_TheJulia: after shoehorning the patch in, things seem to be working as expected now.19:25
TheJuliais it a sign that I'm loosing my mind when someone says shoehorning a patch in I think of tinytoons?19:28
iurygregorynah =)19:29
markguz_I don't know what tinytoons is?  I'm a scotsman living in Texas, so things i say often cause folks to look confused19:29
iurygregorytinytoons is an animation if I do remember hehe19:30
iurygregoryor *cartoon*19:30
TheJuliaA cartoon which also had appearances of the Animaniacs and Pinky and the Brain19:30
markguz_somewhat after my time then....19:31
TheJulia1990-199319:31
TheJuliaerr, 199219:31
markguz_yeah i was in college then :-)19:31
markguz_i recall watching pinky and the brain, or was it ren and stimpy?, while nursing a hangover..19:32
TheJuliaRen and Stimpy seem like nursing a hangover material19:32
markguz_yeah that and rugrats19:33
TheJuliarugrats ++19:35
*** MentalSiege has quit IRC19:37
*** MentalSiege has joined #openstack-ironic19:40
*** browzing has joined #openstack-ironic19:42
erbarrHi, I'm getting an error status right after openstack server create, I did some digging and I'm thinking I might need to change some properties does this look bad?19:42
erbarrhttps://usercontent.irccloud-cdn.com/file/s10vxt9K/image.png19:42
TheJuliaerbarr: is the trait set?19:44
erbarrno idea, how do i check that?19:44
TheJuliaopenstack baremetal node trait list... I think19:45
erbarrlet me see19:45
browzinghello folks, running into an issue (and nothing's seemingly changed in our stack) where metal that boots into the ironic initramfs will not get a link on the main interfaces -- blowing up w/ a ENETUNREACH. Trying to bring up the link manually via IPMI results in error messages for the bnx2x drivers saying the firmware could not be found/loaded. Any ideas? https://imgur.com/a/7okDMhb19:45
* TheJulia wonders if everyone is firing up ironic today19:46
browzingWe were originally suspecting recent changes in ironic-ipa-downloader... but we're lost now.19:46
erbarrTheJulia, I got an empty return19:47
TheJuliaerbarr: ahh, well you need that trait or you need to remove the trait from the flavor19:47
TheJuliabecause the flavor says it is required19:47
erbarrcool, so to remove i guess openstack flavor batemetal trait remove or something along those lines, let me try19:48
erbarrTheJulia, thanks! Tiny toons I loved growing up, and i watched the animaniacs reboot over the holidays, had some moments19:49
TheJuliabrowzing: so we did start paring down the excess firmware in our public images19:49
TheJuliabrowzing: but I don't think we nuked network cards19:49
TheJuliabrowzing: is anything in /usr/lib/firmware ?19:51
TheJuliabrowzing: https://github.com/openstack/ironic-python-agent-builder/commit/6c9c505ca900dc14b1637175d311c3dc49e8536619:52
openstackgerritIury Gregory Melo Ferreira proposed openstack/networking-baremetal master: Add doc/requirements  https://review.opendev.org/c/openstack/networking-baremetal/+/76917219:54
browzing/lib/firmware is pretty bare19:57
browzingjust some microcode stuff19:57
TheJuliawhat is the ramdisk that is running?19:57
TheJuliacentos stream19:57
TheJulia?19:58
browzingI believe we were using https://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo/19:58
browzingtried to roll back to previous-current-tripleo19:58
TheJuliabrowzing: du -hcs /19:58
TheJuliawe can know quickly by size of the ramdisk19:58
browzingand tried the rdo variants too (unsure what differences those have, seemd to be the same size)19:58
TheJuliaor cat /etc/os-release19:58
browzingsec19:58
browzingSorry a bit hard to copy/paste via the IPMI for these dells :p19:59
browzinghttps://imgur.com/a/ejmwGSO19:59
browzing1004M for RD size20:00
TheJuliagot a link by chance to exactly what the downloader pulled down?20:00
browzinglet me k logs -f the statefulset ironic-ipa-downloader... sec20:00
TheJuliashould be in the container logs20:00
TheJuliano worries20:00
browzinghttps://pastebin.com/53nnzjmx20:01
TheJuliadownloading20:02
TheJuliaI'm going to rip the image apart and see what is going on in there20:02
TheJuliagoing to be a few minutes to download20:03
arne_wiebalckbye everyone o/20:03
browzingNo worries been debugging for about two days now :)20:04
browzing/waves farewell arne20:04
TheJuliagoodnight arne_wiebalck20:04
*** hoonetorg has quit IRC20:04
* TheJulia rips it apart20:06
TheJuliawell20:11
TheJuliait is there....20:11
browzingeeenteresting.... Dell r640's for reference20:12
browzingWhere should I be seeing the right firmware? This is p.much what /usr/lib/firmware looks like:20:14
TheJuliain /usr/lib/firmware20:14
browzinghttps://imgur.com/a/7AxRoch20:14
TheJuliahmmm https://images.rdoproject.org/centos8/master/rdo_trunk/previous-current-tripleo-rdo/20:16
TheJulianovember 18th20:16
TheJuliahttps://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo-rdo/20:17
browzingdownloading locally to dig around...20:18
TheJuliahttps://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo-rdo/ <-- smaller images20:19
TheJuliaerr20:19
TheJuliaripping apart the smaller one20:19
TheJuliawell20:20
TheJuliabingo20:20
TheJuliano firmware20:20
TheJuliacomparing to openstack's mirror20:23
TheJuliawell, our pipeline built image20:23
TheJulianot rdoproject's20:23
browzing:) :joy:20:23
browzinghrm yea I downloaded https://images.rdoproject.org/centos8/master/rdo_trunk/previous-current-tripleo-rdo/ironic-python-agent.tar and it appears to have the firmware in /usr/lib/firmware20:25
browzingwhats the difference between the rdo and non-rdo ipa tarballs?20:31
lbragstadTheJulia just checking something - but is this always the case?20:33
TheJuliabrowzing: no idea....20:33
TheJulialbragstad: what is this?20:34
lbragstadhttps://opendev.org/openstack/ironic/src/branch/master/ironic/api/controllers/v1/utils.py#L146220:34
TheJuliabrowzing: I've asked in #tripleo for them to trigger the images to rebuild20:34
lbragstadsorry - i can wait, too20:34
TheJulialbragstad: I think so, but if the middleware is different It could be we silently don't do anything20:38
lbragstadok20:38
TheJuliamiddleware different being the noauth middlware20:38
lbragstadhttp://paste.openstack.org/show/801609/20:39
lbragstadlooks like we only pass in the policy name20:39
lbragstadand i think that means the ironic fetches the policy values from the context object associated with the API request20:39
lbragstadand it passes that for the credentials and the target20:39
TheJuliathat is correct20:40
TheJuliamiddleware expands it if loaded and populates the data structures20:40
lbragstadthe middleware?20:41
lbragstadit looks like the controller utils do that?20:41
lbragstadhttps://opendev.org/openstack/ironic/src/branch/master/ironic/api/controllers/v1/utils.py#L146120:41
lbragstadoh.. https://opendev.org/openstack/ironic/src/branch/master/ironic/common/policy.py#L644-L64520:42
lbragstadthe policy module short circuits enforcement if you're using something other than keystone20:43
TheJuliawhich is our major other case20:43
lbragstadok - i might add a wrapper around check_policy?20:44
lbragstador just add kwargs20:44
lbragstadso that we can start passing in target data, right?20:45
TheJuliaWhy there?20:45
TheJuliawell20:45
lbragstadso - target data is going to change based on the API20:45
TheJuliamaybe20:45
TheJuliaIt is goign to change, largely we consolidated that down into utility methods, so most likely yes, eventually, not at this current patch though20:46
lbragstadtarget data for nodes is going to be different than target data for chassis20:46
lbragstadand right now i think we're using the same two values for target and credentials for all enforcement calls20:46
TheJuliathere are other calls furhter down in the utilities for the more specific methods20:47
TheJuliaor more specific checks20:47
lbragstadoh - i see it20:48
lbragstadcool20:49
TheJuliaI suspect a reference lookup table will be needed or something to keep it relatively clean20:55
lbragstadok - i have a bunch of changes on top of the patch you pushed20:57
lbragstadand i think i removed the requirement to stub keystonemiddleware and keep a cache of keystone tokens20:57
openstackgerritJulia Kreger proposed openstack/ironic master: Enable testing to dynamicly leverage ACL roles  https://review.opendev.org/c/openstack/ironic/+/76813521:05
TheJuliamaybe rebase since I cleaned it up?21:05
tosinTheJulia it failed bifrost :(21:08
tosinhttps://review.opendev.org/c/openstack/bifrost/+/76949521:08
TheJuliatosin21:12
TheJuliaerr21:12
TheJuliatosin: I think you changed the wrong module invocation21:12
TheJuliatosin: line 23 https://review.opendev.org/c/openstack/bifrost/+/769495/3/playbooks/roles/ironic-enroll-dynamic/tasks/main.yml21:13
TheJuliamaybe the ironic-deploy-dynamic/tasks/main.yaml file instead of ironic-enroll-dynamic/tasks/main.yml file?21:13
tosinSorry X| I'll fix that now21:14
lbragstadTheJulia ok - so i think i understand things a bit better now21:17
lbragstadi think we have at least two approaches for functional policy testing with ddt in ironic21:17
*** hoonetorg has joined #openstack-ironic21:18
lbragstad1.) we can stub out the keystonemiddleware implementation, like what is proposed in your patch21:18
lbragstad2.) implement a new auth_strategy that bypasses authentication completely (just like noauth)21:18
TheJuliaI really don't want to expose a second auth plugin/strategy21:19
lbragstadi tried #2 with auth_strategy = noauth and it broke all the existing functional API tests21:19
TheJuliayeah21:19
lbragstadwhich is a bummer21:19
lbragstad#1 is going to force us to know implementation details of ksm in ironic's functional testing21:19
TheJuliaI don't think there is any way to avoid that, and I'd actually suggest insulating against that knowledge will make it more difficult to move foward21:24
lbragstadi agree that i'm not a fan of ksm implementation details in ironic test code21:26
tosinTheJulia I don't think ironic-dynamic-deploy is in the playbooks21:26
TheJuliaI kind of feel like that is more minor, in the grand scheme of the universe21:26
TheJulialbragstad: at least, that is my feeling. I think insulation just makes this all harder to push forward because it further reduces the actual people with knowledge of the underlying thing we're still need to integrate/use21:27
lbragstadanother option would be to supply headers for all the existing functional tests so that they work with http://paste.openstack.org/show/801610/21:27
lbragstadbut i imagine that's a lot of work21:28
TheJuliatosin: sorry, had the name a little wrong https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-deploy-nodes-dynamic/tasks/main.yml#L5521:28
tosinthank you21:28
TheJulialbragstad: so funny enough, we could set it to http_basic21:30
lbragstadthe new TestACL strategy?21:30
TheJuliaironic has a basic auth middleware21:31
lbragstadhttps://github.com/openstack/ironic-lib/blob/master/ironic_lib/auth_basic.py21:33
browzingso just tested the initramfs image out of CI, no firmware... (this one: https://images.rdoproject.org/centos8/master/rdo_trunk/tripleo-ci-testing/ironic-python-agent.tar)21:36
TheJuliabrowzing: I suspect it has to do with the source images going into the proess21:39
openstackgerritTosin Farai proposed openstack/bifrost master: Test new features added to baremetal_node_action  https://review.opendev.org/c/openstack/bifrost/+/77066921:39
TheJuliathey are building form the old ironic-agent element deprecated in diskimage-builder21:40
browzinginteresting...21:40
*** hoonetorg has quit IRC21:43
TheJuliabrowzing: okay, it is Centos-Stream21:52
TheJuliano firmware is present in that21:52
*** rcernin has joined #openstack-ironic21:59
TheJuliabrowzing: https://review.opendev.org/c/openstack/tripleo-common/+/770671 should fix the issue your hitting22:02
browzingfantastic, been watching with popcorn whilst symlinking to older initramfs's :)22:07
browzingThanks a ton for wading through that TheJulia22:07
erbarrTheJulia, I tried both removing the trait from the flavor and adding it to the baremetal node and still getting the error "No valid host was found"22:33
browzingOne more question, what's the timeline for the PR approval, fix rollout, and image build being live?22:34
TheJuliadid you give things 3 minutes?22:34
openstackgerritLance Bragstad proposed openstack/ironic master: Attempt to slim down protection test base class  https://review.opendev.org/c/openstack/ironic/+/77067322:35
lbragstadTheJulia ok - i think i have things working,  but with less cache data and less implementation details of ksm22:35
lbragstadtox -e py3 -- ironic.tests.unit.api.test_acl.TestACLBasic passes for me22:35
TheJuliabrowzing: hjensas already has +2'ed it, so I expect it will be merged tomorrow morning22:35
TheJulialbragstad: yay!22:35
browzingcool thank you all - you're lovely :p22:36
TheJulialbragstad: I'm a few patches down the chain revising now so I can integrate it and evolve it22:36
lbragstadTheJulia ok - feel free to run with it22:36
lbragstadbut - i ripped most of the ksm stuff out and all the cache data, it looks like you already removed a bunch of it anyway22:37
lbragstadthe side-effect is that it pushes the request authorization headers closer to the tests, which isn't a bad thing IMO, but i'll defer to whatever ironic folks are comfortable with22:37
TheJuliaYeah, I don't know, dmitry really doesn't like the mock usage at all, but... I mean... the happy paths are extremely limited22:40
lbragstadyeah - i can understand that22:41
lbragstadthe good thing is the the mock is pretty simple22:41
TheJuliayeah22:41
lbragstadwe're really just mocking one method and it isn't doing anything22:42
TheJuliaand kind of backing away from the auth side checks, and focusing on roles, it keeps it more simple22:42
TheJuliatwo methods in a later patch22:42
TheJuliabecause we need to block it from trying to call the conductor22:42
lbragstadmmm22:42
TheJuliaall of the policy checks are in the api22:42
lbragstadok22:46
lbragstadthat's good22:46
openstackgerritTosin Farai proposed openstack/bifrost master: Test new features added to baremetal_node_action  https://review.opendev.org/c/openstack/bifrost/+/77066922:54
jandersgood morning Ironic o/23:01
*** rh-jelabarre has quit IRC23:17
jandersguilhermesp thank you for sending through the outputs. From this, it would seem that your devices don't support NVMe sanitize (and I am yet to find any that do! :). However different flavours of secure format will likely work. Have you tested any nvme-cli commands against those drives? If so I'd be interested to learn more and if not I can give you23:26
janderssome things to try and see what approaches work.23:26
*** pmannidi has joined #openstack-ironic23:33
openstackgerritTosin Farai proposed openstack/bifrost master: Test baremetal_node_action  https://review.opendev.org/c/openstack/bifrost/+/77067923:36
jandersTheJulia regarding the midcycle - would 7pm UTC be an option? That's 5am here. So far I marked 2pm UTC timeslots (midnight here) and see how popular those are. Thank you!23:38
TheJuliaI would be okay with it, but I don't know how many of the EU folks would be willing23:39
jandersI understand. Let's see what responses come in and I might bring the idea up again. Thanks TheJulia! :)23:45
TheJuliajanders: I added more time windows23:48
*** rloo has quit IRC23:52
jandersawesome, thank you TheJulia. I put in my times including the extra slots  (I don't know what happened to my first submission, it didn't save correctly - but that turned out good in the end :)23:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!