Thursday, 2020-09-03

*** k_mouza has quit IRC00:02
*** JamesBenson has quit IRC00:08
TheJuliaSo to the question if the larger proposed feature is acceptable. If the behavior contract of the raid configuration field is changed accordingly as discussed above then it does not have bearing on the overall larger piece of work's acceptability. Fair warning though, it is going to face greater scrutiny in the review process because the larger feature will be a larger piece of work and much more code outside00:08
TheJuliaof the driver's specific code and driver_internal_info field will need to be touched/manipulated. As mentioned on the call I highly suggest to submit patches in incremental steps to reach your desired end state and feature complete status. This having been completely stated, I still believe it would be good to have another call so that this is revisited by contributors since we did not discuss changing the00:08
TheJuliaraid_config field behavior contract and on the call and ultimately the contract offered by these interfaces needs to be explicitly stated in these terms such that it non-destructively updates it's relevant areas based on what it has available, not deleting any field value. This will likely require additional Restful API functionality in terms of CRUD operations on what I believe are presently read-only00:08
TheJuliafields/data. This also being said, IF this discussion is framed with generic version being available to at least _save_/_restore_ existing configuration data in the same base mechanism, then this becomes even easier for the community grok and see usefulness of a larger mechanism beyond the vendor specific steps/methods that would be part of a vendor specific interface.00:08
* rpioso puts on masks before entering grocery store zone.00:08
*** JamesBenson has joined #openstack-ironic00:09
TheJuliarpioso: +++++++00:11
TheJuliarpioso: I'm going to go assemble a shelf now, have a wonderful and safe evening!00:11
rpiosoTheJulia: Sound good. Let’s have that discussion. First though, I need to arrange an update on Redfish Interop Profiles :)00:12
TheJulia++00:13
TheJuliaawesome00:13
TheJuliaremind me tomorrow and I'll put a thing on the meeting for monday00:13
rpiosoTheJulia: Thank you! And you as well.00:13
rpiosoTheJulia: We may want to wait until ajya and I post a spec update. That’s on the back burner at the moment, as we approach feature freeze.00:15
rpiosoTheJulia: We could discuss RAID contract before revisiting the meta step proposal, though.00:17
*** k_mouza has joined #openstack-ironic00:29
*** hoonetorg has quit IRC00:29
*** k_mouza has quit IRC00:33
*** hoonetorg has joined #openstack-ironic00:41
erbarrthere hasn't been any changes that would cause PXE to not boot, correct? I haven't seen any so far00:42
*** lifeless has joined #openstack-ironic01:02
jandersI'm looking at testing redfish OOB introspection on bifrost/VMs. Does anyone here have any experience with this? I'm particularly after any reconfiguration of the bifrost-deployed vbmc that might be required01:07
*** zzzeek has quit IRC01:11
*** zzzeek has joined #openstack-ironic01:14
*** JamesBenson has quit IRC01:35
*** stevebaker has joined #openstack-ironic01:54
openstackgerritChristopher Dearborn proposed openstack/sushy master: WIP: Allow monitoring progress of a firmware update  https://review.opendev.org/74595002:04
openstackgerritChristopher Dearborn proposed openstack/ironic master: WIP: Redfish driver firmware update  https://review.opendev.org/74961902:12
jandersdtantsur|afk rpittau iurygregory regarding https://storyboard.openstack.org/#!/story/2008038 do I need a real-baremetal in order to be able to use OOB introspection? I tried setting up an environment using sushy-emulator and bifrost VMs but I hit this: http://paste.openstack.org/show/797405/ - looks like the response to02:27
jandershttp://localhost:9132/redfish/v1/Systems/<UUID>/Processors is not implemented (I don't see a template for it in sushy_tools/emulator/templates). I will try to progress other parts of the patch now  but it would be great to have your thoughts on this when your timezone comes online. Thanks! :)02:27
jandersI see https://storyboard.openstack.org/#!/story/2007454 is still in TODO which would explain it02:28
*** ijw has quit IRC02:47
*** ijw has joined #openstack-ironic02:48
*** tzumainn has quit IRC02:49
*** JamesBenson has joined #openstack-ironic03:05
*** JamesBen_ has joined #openstack-ironic03:09
*** JamesBenson has quit IRC03:10
*** hjensas has joined #openstack-ironic04:37
*** JamesBen_ has quit IRC04:39
*** m1p has quit IRC04:39
*** TheJulia has quit IRC04:39
*** m1p has joined #openstack-ironic04:40
*** TheJulia has joined #openstack-ironic04:40
*** hjensas has quit IRC04:58
*** JamesBenson has joined #openstack-ironic05:00
*** janders has quit IRC05:11
openstackgerritMerged openstack/ironic master: Support file:/// images for the direct deploy  https://review.opendev.org/74837905:26
*** hjensas has joined #openstack-ironic05:27
*** hjensas has quit IRC05:28
*** JamesBenson has quit IRC05:32
*** JamesBenson has joined #openstack-ironic05:33
*** hjensas has joined #openstack-ironic05:36
*** JamesBenson has quit IRC05:38
*** jtomasek has joined #openstack-ironic05:57
*** janders has joined #openstack-ironic06:01
openstackgerritlikui proposed openstack/ironic-python-agent master: Remove Babel lower-requirement  https://review.opendev.org/74964206:09
*** Qianbiao has joined #openstack-ironic06:30
*** belmoreira has joined #openstack-ironic06:36
*** JamesBenson has joined #openstack-ironic06:38
*** JamesBenson has quit IRC06:43
arne_wiebalckGood morning, ironic!06:44
*** sshnaidm|afk is now known as sshnaidm06:50
openstackgerritAnkit Kumar proposed openstack/ironic master: Enhance certificate verification for ilo harware type  https://review.opendev.org/74349006:52
*** Nisha_Agarwal has joined #openstack-ironic06:55
*** JamesBenson has joined #openstack-ironic07:06
*** JamesBenson has quit IRC07:24
*** tosky has joined #openstack-ironic07:40
*** Nisha_Agarwal has quit IRC07:42
openstackgerritVerification of a change to openstack/ironic failed: Increase memory of tinyipa vms  https://review.opendev.org/74763307:46
*** dougsz has joined #openstack-ironic07:52
rpittaugood morning ironic! o/07:52
iurygregorygood morning rpittau arne_wiebalck janders and Ironic o/07:56
rpittauhey iurygregory :)07:57
*** gyee has quit IRC07:57
*** dougsz has quit IRC07:57
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: Use tinycore 11 to build tinyipa  https://review.opendev.org/71164707:58
*** janders has quit IRC08:00
*** janders has joined #openstack-ironic08:07
*** lucasagomes has joined #openstack-ironic08:09
arne_wiebalckhey iurygregory o/08:19
*** derekh has joined #openstack-ironic08:47
openstackgerritRiccardo Pittau proposed openstack/ironic master: Replace retrying with tenacity  https://review.opendev.org/37657408:47
*** priteau has joined #openstack-ironic08:49
*** mkowalski has quit IRC09:00
*** dtantsur|afk is now known as dtantsur09:06
dtantsurgood morning ironic09:06
iurygregorygood morning dtantsur =)09:06
dtantsurjanders: maybe? I thought etingof implemented OOB introspection, but given the state of sushy-tools..09:06
dtantsuryou only need MAC addresses though and THAT part should have been implemented09:07
iurygregorylucasagomes, approved =) sorry forgot to look yesterday the result hehe09:08
lucasagomesiurygregory++ no rush!09:08
lucasagomesthanks for reviewing it09:08
iurygregoryyw =)09:08
*** dougsz has joined #openstack-ironic09:08
*** dougsz has quit IRC09:13
openstackgerritDmitry Tantsur proposed openstack/bifrost stable/stein: [Train and older] Fix several CI problems  https://review.opendev.org/74901409:18
jandersg'day Ironic! o/09:18
jandersdtantsur I think OOB introspection works, it's the sushy emulator that's at fault AFAIK09:19
dtantsurwhy am I not surprised?..09:19
janderswhen I work on redfish stuff, should I aim to work with a redfish-enabled real-baremetal?09:19
jandersor is sushy-emulator reasonably complete and worth a try?09:20
rpittaujanders: if you have a real baremetla server with redfish is the ideal scenario :)09:20
rpittausushy emulator can work, with its limitations09:20
rpittaulike as you saw the processors is not implemented at the moment09:21
rpittauin theory with the emulator you should have a close-to-real implementation of the redfish standard, on real baremetal, depending on the vendor, certain things can be "off"09:23
*** dougsz has joined #openstack-ironic09:27
*** ricolin has quit IRC09:29
*** janders9 has joined #openstack-ironic09:31
*** janders has quit IRC09:33
*** janders9 is now known as janders09:33
*** janders has quit IRC09:36
*** janders has joined #openstack-ironic09:39
openstackgerritMerged openstack/ironic-python-agent master: Update the cache if we don't have a root device hint  https://review.opendev.org/74707209:41
openstackgerritMerged openstack/ironic-python-agent bugfix/6.2: Make WSGI server respect listen_* directives  https://review.opendev.org/74959909:42
openstackgerritMerged openstack/ironic-python-agent bugfix/6.3: Make WSGI server respect listen_* directives  https://review.opendev.org/74959809:42
*** Nisha_Agarwal has joined #openstack-ironic09:50
Nisha_Agarwaldtantsur, Good morning...09:50
dtantsurhi Nisha_Agarwal09:50
Nisha_Agarwaldtantsur, after a long time working back in ironic :)09:51
dtantsurNisha_Agarwal: congrats :)09:51
Nisha_Agarwaldtantsur, a very small patch https://review.opendev.org/#/c/746416/ if u have time ....09:51
patchbotpatch 746416 - ironic - Adds few of the security dashboard parameters to c... - 10 patch sets09:51
Nisha_Agarwaldtantsur, thanks :)09:52
dtantsurwill try to get there, a bit busy day today though09:52
Nisha_Agarwaldtantsur, looking at more contribution now :)09:52
Nisha_Agarwaldtantsur, sure09:52
openstackgerritDmitry Tantsur proposed openstack/ironic master: documentation: follow-up to file:// support in direct deploy  https://review.opendev.org/74969509:56
*** rcernin has quit IRC10:01
*** rcernin has joined #openstack-ironic10:06
*** hjensas is now known as hjensas|afk10:11
*** rcernin has quit IRC10:11
openstackgerritNisha Agarwal proposed openstack/ironic master: Adds few of the security dashboard parameters to capabilities  https://review.opendev.org/74641610:18
*** rcernin has joined #openstack-ironic10:19
dtantsurWTF https://zuul.opendev.org/t/openstack/build/9d2905cea75e4297b82dcec95e1620da/log/logs/ironic-inspector.log10:21
openstackgerritNisha Agarwal proposed openstack/ironic master: Adds few of the security dashboard parameters to capabilities  https://review.opendev.org/74641610:21
iurygregoryinspector broken?10:21
dtantsuriurygregory: it's on Stein, fortunately10:21
dtantsurbut yeah, something is very wrong10:21
dtantsurah, of course it's eventlet, what else? https://bugs.launchpad.net/glance/+bug/171112310:22
openstackLaunchpad bug 1711123 in Glance "Failed to start openstack image service glance (openstack-glance-api.service)." [Undecided,Invalid]10:22
iurygregoryeventlet -.-'10:23
*** rcernin has quit IRC10:24
dtantsurmnasiadka: I'm seriously not sure if I can revive bifrost stein ^^10:27
* dtantsur has posted a call for help on the ML10:31
*** rcernin has joined #openstack-ironic10:31
*** priteau has quit IRC10:33
dtantsuriurygregory: I guess the privsep stuff on ironic-lib won't be ready for Victoria?10:34
*** rcernin has quit IRC10:36
*** Lucas_Gray has joined #openstack-ironic10:38
mnasiadkadtantsur: well, we just marked it as unbuildable :)10:40
dtantsurk10:43
dtantsurironic folks, please pay attention to any outstanding ironic-lib, metalsmith and sushy patches. the release deadline is, well, today.10:44
*** priteau has joined #openstack-ironic10:49
openstackgerritMerged openstack/metalsmith master: Explicitly set jobs to ML2/OVS  https://review.opendev.org/74934610:52
openstackgerritMerged openstack/metalsmith master: metalsmith_instances module cleanups  https://review.opendev.org/74854410:52
iurygregorydtantsur, yep, we decided last week in the weekly meeting10:53
dtantsurgot it10:54
dtantsurthen there are outstanding sushy and metalsmith patches that could be reviewed10:54
*** priteau has quit IRC10:54
iurygregoryI will look at them after the meeting o/10:54
dtantsurI think ironic-lib is fine, so anyone is welcome to propose a release10:54
iurygregoryack10:55
dtantsurI also think it may be the time to release IPA-builder10:55
iurygregoryI can check everything in about 45min hehe10:56
* iurygregory grabs something to eat before the meeting10:56
openstackgerritMerged openstack/metalsmith master: Document metalsmith_instances  https://review.opendev.org/73813110:59
*** JamesBenson has joined #openstack-ironic11:00
*** priteau has joined #openstack-ironic11:03
*** k_mouza has joined #openstack-ironic11:08
*** priteau has quit IRC11:11
openstackgerritMerged openstack/ironic master: Remove absolute path with iptables when L3 enabled  https://review.opendev.org/74893011:12
*** k_mouza has quit IRC11:14
*** hjensas|afk is now known as hjensas11:15
*** vmud213 has joined #openstack-ironic11:22
openstackgerritvinay kumar muddu proposed openstack/ironic master: Adds ilo-uefi-https boot interface to ilo5  https://review.opendev.org/74550111:22
*** vmud213 has quit IRC11:34
openstackgerritMerged openstack/sushy stable/ussuri: Use Sessions URL from root service if it's provided  https://review.opendev.org/74897211:46
mnasero/ has anyone played around with smartnics with ironic (i assume since a spec was implemented, someone was testing it on $vendor cards?)11:48
openstackgerritMerged openstack/sushy stable/train: Use Sessions URL from root service if it's provided  https://review.opendev.org/74897311:49
iurygregorydtantsur, for metalsmith the only patches I saw https://review.opendev.org/748315 https://review.opendev.org/73788811:51
patchbotpatch 748315 - metalsmith - Replace the Centos7 image job with a Centos8 UEFI ... - 6 patch sets11:51
patchbotpatch 737888 - metalsmith - metalsmith_deployment role switch to metalsmith_in... - 17 patch sets11:51
dtantsuriurygregory: ack. maybe we should released metalsmith (without branching) now and see if we can fit these in quickly enough11:52
dtantsurstevebaker: FYI ^^^11:52
openstackgerritRiccardo Pittau proposed openstack/bifrost master: Follow up on "Fix error 601"  https://review.opendev.org/74949911:54
iurygregorycan be, and we can do another release + branch after we merge 737888...11:54
iurygregorysushy https://review.opendev.org/#/q/project:openstack/sushy+status:open+branch:master WOW11:55
rpittauoh yes, tinyipa built on focal with tinycore 11 in ironic-standalone-ipa-src11:56
* rpittau more coffee AND ice-cream needed11:59
*** priteau has joined #openstack-ironic12:01
*** JamesBenson has quit IRC12:04
*** uzumaki has joined #openstack-ironic12:08
dtantsuractually, I may have a small fix for metalsmith, let me see12:09
*** ricolin has joined #openstack-ironic12:10
iurygregorydtantsur, ack I will go with ironic-lib12:14
iurygregoryand I can push metalsmith later today12:14
*** JamesBenson has joined #openstack-ironic12:19
*** JamesBenson has quit IRC12:23
openstackgerritDmitry Tantsur proposed openstack/metalsmith master: Stop requiring checksums with file images  https://review.opendev.org/74971912:24
dtantsuriurygregory, rpittau, please ^^12:26
*** Goneri has joined #openstack-ironic12:31
*** uzumaki has quit IRC12:32
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent master: Prepare to move ipa-builder jobs  https://review.opendev.org/74676612:32
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: Use converted ipa jobs  https://review.opendev.org/74972312:35
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: Use converted ipa jobs  https://review.opendev.org/74972312:36
Qianbiaohello Ironic~12:42
Qianbiaohi what share picture service is used usually.12:43
*** vmud213 has joined #openstack-ironic12:44
*** thiagop has joined #openstack-ironic12:44
Qianbiaoa baremetal server failed to boot from pxe ramdisk and display some Garbage characters12:45
Qianbiaoi want to share the screenshoot, and may be some of you know the issue12:46
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent master: Remove unused jobs  https://review.opendev.org/74972512:48
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent master: Remove unused jobs  https://review.opendev.org/74972512:49
rpittauQianbiao: this should work https://pasteboard.co/12:50
Qianbiaorpittau thanks.12:50
Qianbiaook this is it : https://pasteboard.co/Jpn13aR.png12:52
QianbiaoI can it dhcp finished, loading kernal file finished. and then show those garbage chars.12:52
Qianbiaoand after a while(10s maybe), server auto restart.12:52
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: Use tinycore 11 to build tinyipa  https://review.opendev.org/71164712:53
dtantsurQianbiao: never seen anything like that, but it does start booting the linux kernel, so whatever happens must have something to do with it.12:56
Qianbiao<dtantsur> may it be hardware compatible issue?12:57
QianbiaoI am using the default pxe options from ironic devstack.12:58
openstackgerritMerged openstack/ironic master: [trivial] remove emacs config from devstack script  https://review.opendev.org/74897812:58
openstackgerritMerged openstack/ironic master: Update nodes in notifications documentation  https://review.opendev.org/74842312:58
*** rcernin has joined #openstack-ironic12:59
dtantsurQianbiao: you mean want to tweak these options to make sure you actually see the kernel console output13:00
dtantsurit's probably a kernel fault13:01
openstackgerritDmitry Tantsur proposed openstack/sushy-tools master: DNM reproduce missing PowerState  https://review.opendev.org/74972713:01
dtantsurbfournie: trying to reproduce the dell issue ^^^13:01
dtantsurotherwise I cannot understand what is going on13:01
*** rloo has joined #openstack-ironic13:01
Qianbiaodtantsur yes, if i can see what is happening, maybe i can figure out how to fix it.13:03
bfourniedtantsur: good, btw wonder if we should handle the exception if try to set OFF and get back "already powered OFF"13:04
*** rcernin has quit IRC13:04
dtantsurbfournie: we should, I'm just not sure how.. especially since we probably cannot rely on the text to be in English.13:05
bfourniedtantsur: hmm, yeah13:05
dtantsurmaybe if we get HTTP 409 on power off, try the reboot command?13:06
bfourniedtantsur: yeah that would revert to previous behaviour13:06
dtantsurthe thing is, if we get no power state from the machine, I'd expect ironic to complain that it cannot get power state13:07
dtantsurI don't understand how we end up assuming it's powered on...13:07
dtantsurI think we should catch rpioso later today13:08
bfourniedtantsur: sure, does it depend on sync_power_state_interval?13:08
dtantsurbfournie: it's a possibility. maybe some action from our side removes PowerState from the System, but the power sync loop does not notice it quickly enough?13:08
dtantsurdoes not explain why we assume None==On13:09
* dtantsur is extremely confused13:09
bfourniedtantsur: yeah13:09
iurygregoryNone==On WHAT?13:09
iurygregoryhttps://media.giphy.com/media/SqmkZ5IdwzTP2/giphy.gif13:09
dtantsuriurygregory: https://bugzilla.redhat.com/show_bug.cgi?id=187330513:10
openstackbugzilla.redhat.com bug 1873305 in Bare Metal Hardware Provisioning "Failed to power on /inspect node when using Redfish protocol" [High,New] - Assigned to tsedovic13:10
* iurygregory looks13:10
* dtantsur sees that shadower is assigned and assumes he'll solve it! :-P13:10
iurygregorydtantsur, LOL13:10
shadowerHi! I'm your friendly load balancer for bugs assigned to the Ironic subcomponent of OCP13:12
* dtantsur likes friendly load balancers13:16
openstackgerritvinay kumar muddu proposed openstack/ironic master: Allow HttpImageService to accept custom certificate  https://review.opendev.org/74293613:17
TheJuliaheh13:19
openstackgerritDmitry Tantsur proposed openstack/ironic master: Update deploy steps documentation  https://review.opendev.org/74972913:21
dtantsurmorning TheJulia13:21
iurygregorygood morning TheJulia =)13:21
rpittaugood morning TheJulia :)13:23
Nisha_Agarwalmorning TheJulia13:24
*** JamesBenson has joined #openstack-ironic13:25
bfourniedtantsur: I see what you mean now, the response for /redfish/v1/Systems/System.Embedded.1 doesn't contain 'PowerState' when the power is Off, we only see it for the node which was manually powered On in the description (mgmt-e16-h12-b02-fc640) - 'PowerState': 'On'13:30
openstackgerritChristopher Dearborn proposed openstack/sushy master: Make message parsing more resilient  https://review.opendev.org/74892713:32
dtantsurbfournie: yeah, it's super weird13:34
*** tzumainn has joined #openstack-ironic13:35
openstackgerritRiccardo Pittau proposed openstack/ironic master: Replace retrying with tenacity  https://review.opendev.org/37657413:36
dtantsurlooking for reviews on https://review.opendev.org/#/c/749285/ it's probably another thing we'll want to backport13:39
patchbotpatch 749285 - bifrost - Create our own firewalld zone and use it on real b... - 2 patch sets13:39
iurygregorydtantsur, https://review.opendev.org/#/c/749734/ ironic-lib13:39
patchbotpatch 749734 - releases - Release ironic-lib and create stable/victoria - 1 patch set13:39
dtantsurthx!13:40
openstackgerritMerged openstack/ironic master: Support caching http:// images locally in the direct deploy  https://review.opendev.org/74949613:43
openstackgerritMerged openstack/ironic master: Allow setting image_download_source per node  https://review.opendev.org/74950413:50
openstackgerritMerged openstack/ironic master: documentation: follow-up to file:// support in direct deploy  https://review.opendev.org/74969513:50
openstackgerritMerged openstack/ironic master: Add L3 boot section to the docs  https://review.opendev.org/68984413:54
iurygregorylol release patch complains about stable/victoria o.o14:06
dtantsuriurygregory: mm, no, it complains about invalid hash14:07
iurygregoryo.o14:07
dtantsurhow did you end up with it? for me HEAD is different14:07
iurygregorytox -e venv -- list-unreleased-changes victoria openstack/ironic-lib14:08
dtantsuriurygregory: try using scripts for proposing releases, not editing yaml manually14:09
iurygregoryI did14:09
dtantsurah, hmmm. then I don't know, probably worth mentioning on #openstack-release14:09
iurygregorytox -e venv -- new-release victoria ironic-lib feature --stable-branch14:10
dtantsurwow14:10
iurygregorylol running again locally the same command I got the other commit14:13
iurygregory07d98520f09f65dc13c37150c91733527a105acf WTF14:13
rpittauiurygregory: yeah, I just tested as well and I got the correct hash14:13
iurygregorymagic14:13
iurygregoryO.o14:14
iurygregoryupdated14:16
*** cdearborn has joined #openstack-ironic14:16
cdearbornGood morning Ironic!14:17
iurygregorymorning cdearborn14:17
rpiosoGood morning, ironic14:17
iurygregorymorning rpioso14:17
dtantsurhi cdearborn, rpioso, may I use some of your time?14:18
rpiosobfournie, dtantsur: Could you try iDRAC firmware 4.10.10.10, assuming that version is available for FC640?14:18
dtantsuraha, lemme pass it on14:18
rpittauhey cdearborn rpioso :)14:19
cdearborndtantsur, sure14:19
dtantsurcdearborn: rpioso pretty much answered my question, but it was about https://bugzilla.redhat.com/show_bug.cgi?id=187330514:19
openstackbugzilla.redhat.com bug 1873305 in Bare Metal Hardware Provisioning "Failed to power on /inspect node when using Redfish protocol" [High,New] - Assigned to dtantsur14:19
dtantsurseeing something truly weird there14:20
rpiosodtantsur: Yeah, I examined the attached log. The System resource does not consistently contain the PowerState property.14:21
cdearborndtantsur: aok - let me know if you need me14:22
rpiosorpittau: o/14:22
rpiosodtantsur: Thank you for passing my ask along.14:23
dtantsurthank you for looking into it14:23
rpiosodtantsur: yw14:27
*** Nisha_Agarwal has quit IRC14:45
Qianbiaohi dtantsur, i enabled console redirect log and debug log.14:45
Qianbiaothis is the log: http://paste.openstack.org/show/797425/14:45
Qianbiaodo u have a minute look at it.14:46
*** Lucas_Gray has quit IRC14:46
JayFIf someone gets a chance, I'd love to get a +2A on https://review.opendev.org/#/c/747193/ (Thanks for the +2 dmitry)14:46
patchbotpatch 747193 - ironic-python-agent - If listen_tls is true, enable TLS on wsgi server - 10 patch sets14:46
dtantsurQianbiao: I'm not a kernel specialist, so I only have a random guess: if you're using tinyIPA as your ramdisk, you may want to try a DIB-built image14:48
Qianbiao<dtantsur> thanks, will try.14:48
Qianbiaoi guess it maybe hardware compatibility issue.14:49
QianbiaoBut not sure where.14:49
openstackgerritRiccardo Pittau proposed openstack/ironic master: Replace retrying with tenacity  https://review.opendev.org/37657414:49
dtantsurI'm not sure either, unfortunately14:49
Qianbiao:) You are still my hero14:49
openstackgerritRiccardo Pittau proposed openstack/ironic master: Replace retrying with tenacity  https://review.opendev.org/37657414:49
dtantsurQianbiao: if nothing helps, you may try building IPA images with different base OS and seeing if it makes any difference14:49
dtantsur:)14:50
Qianbiaook14:50
openstackgerritMerged openstack/bifrost master: Create our own firewalld zone and use it on real bare metal  https://review.opendev.org/74928514:50
openstackgerritJulia Kreger proposed openstack/ironic master: Trivial: fix minor typo on RFC number  https://review.opendev.org/74975414:50
*** Lucas_Gray has joined #openstack-ironic14:51
*** belmoreira has quit IRC15:00
*** rloo has quit IRC15:01
*** rloo has joined #openstack-ironic15:01
*** rloo has quit IRC15:02
*** rloo has joined #openstack-ironic15:02
*** Qianbiao has quit IRC15:04
openstackgerritJulia Kreger proposed openstack/metalsmith master: Rework jobs so failed devstack does not retry the job  https://review.opendev.org/74975815:04
openstackgerritDmitry Tantsur proposed openstack/bifrost bugfix/8.3: Create our own firewalld zone and use it on real bare metal  https://review.opendev.org/74975915:05
openstackgerritDmitry Tantsur proposed openstack/bifrost stable/ussuri: Create our own firewalld zone and use it on real bare metal  https://review.opendev.org/74976015:08
TheJuliahmmmm15:12
dtantsurmmmm?15:13
TheJuliatrying to figure out why cleaning fails in uefi mode on metalsmith15:13
*** vmud213 has quit IRC15:13
TheJuliaGot the VM to boot with the agent though \o/... It is just took a really long time because the job would be retried three times15:14
TheJuliaalso, https://review.opendev.org/#/c/749407/2 apparently I put W* in the config option, which I don't think was intentional15:14
patchbotpatch 749407 - ironic - Remove token-less agent support - 2 patch sets15:14
dtantsurfun15:14
*** iurygregory has quit IRC15:24
dtantsurgoing a bit earlier today, see you tomorrow15:31
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Accept and use a TLS certificate from the agent  https://review.opendev.org/74713615:31
*** dtantsur is now known as dtantsur|afk15:31
dtantsur|afko/15:31
rpiosocdearborn: Wondering if https://review.opendev.org/#/c/748927 should pass Dell Ironic CI before being merged? If so, please Workflow-1 it. iurygregory appears to be out.15:34
patchbotpatch 748927 - sushy - Make message parsing more resilient - 3 patch sets15:34
*** vmud213 has joined #openstack-ironic15:35
vmud213Hello Ironic!15:36
openstackgerritJulia Kreger proposed openstack/metalsmith master: Replace the Centos7 image job with a Centos8 UEFI based job  https://review.opendev.org/74831515:38
erbarrrpioso, looks like when that run it was hitting the setuptools issue, right now master is failing, I looked at nodes running and it looks like they're failing to pxe boot into ipxe, I'm looking to see what's going on15:38
TheJuliarpittau: I blocked workflow on https://review.opendev.org/#/c/748927 for the time being15:40
patchbotpatch 748927 - sushy - Make message parsing more resilient - 3 patch sets15:40
vmud213TheJulia: Mind taking a look into https://review.opendev.org/#/c/745501/ when time permits?15:40
patchbotpatch 745501 - ironic - Adds ilo-uefi-https boot interface to ilo5 - 10 patch sets15:40
TheJuliavmud213: sure, I'll try to look at it soon()15:41
vmud213Also Also this one Thanks.15:41
vmud213https://review.opendev.org/#/c/742936/15:42
patchbotpatch 742936 - ironic - Allow HttpImageService to accept custom certificate - 8 patch sets15:42
vmud213Thanks15:42
*** iurygregory has joined #openstack-ironic15:42
rpiosoerbarr: Thank you!15:47
rpiosoTheJulia: Thanks15:47
erbarranybody got any idea what could be causing it? started yesterday at around 8am15:48
TheJuliaugh zuul job names15:48
TheJuliaerbarr: causing what?15:48
openstackgerritMerged openstack/metalsmith master: Stop requiring checksums with file images  https://review.opendev.org/74971915:48
erbarrdell jobs stopped pxe booting on master15:49
TheJuliaOVN?15:50
TheJulialucasagomes: did the OVN default change merge to devstack?15:50
lucasagomesTheJulia, not yet no15:50
lucasagomesTheJulia, there are a few more things to fix, what we first did was to move the devstack code to deploy OVN to devstack itself15:51
TheJuliaerbarr: yesterday 8 am what timzone?15:51
erbarrcentral15:51
lucasagomesnow I am sending patches around with a depends-on which changes the default to OVN15:51
TheJuliaUTC-5 then15:51
lucasagomesmaking sure everything is ok before we go ahead and actually changing it15:52
lucasagomesfixing whatever I find in between as well15:52
openstackgerritMerged openstack/bifrost master: TLS support for API services  https://review.opendev.org/74792115:53
iurygregoryerbarr, do you have some logs? (sorry I was disconnected)15:55
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Change integration test for Metalsmith to use Centos8/UEFI  https://review.opendev.org/74977215:56
TheJuliaerbarr: can you confirm what the PATH variable devstack is being executed with?15:57
erbarriurygregory, I'm trying to dig through the bulk here https://stash.dellemc-community.org/logs/dellemc-poweredge-R730-R630-tempest-dsvm-ironic-idrac/2020-09-02:21:02/743868/5//logs/15:57
TheJuliahttps://review.opendev.org/#/c/748930/ may be the cause15:58
patchbotpatch 748930 - ironic - Remove absolute path with iptables when L3 enabled (MERGED) - 1 patch set15:58
TheJuliait is around the time window15:58
TheJuliahmm.. no ip netns in the logs16:01
*** Lucas_Gray has quit IRC16:02
erbarrTheJulia, it started yesterday morning, i looked at devstack and didn't see any merges around that time but maybe neutron16:02
iurygregoryWUT?!16:04
iurygregoryDevStack Version: victoria16:04
iurygregoryChange: ae21b3556ff215c0f73bc0ebbc90d72fa59a3459 move cleanup of pyc files 2020-09-02 04:48:31 +000016:04
iurygregoryOS Version: Ubuntu 16.04 xenial16:04
iurygregoryUbuntu Xenial in Victoria ?!16:04
iurygregorythis doesn't seem right..16:05
*** gyee has joined #openstack-ironic16:05
iurygregoryerbarr, any reason why it's not Bionic?16:06
erbarriurygregory, yea i know, I'm trying to update to zuul v3 and having trouble installing it16:07
iurygregoryouch =(16:07
iurygregorymaybe people on #zuul would be able to help16:07
erbarryea, any pointers as to why nodepool would launch and then instantly delete a node?16:07
erbarryea, i've talked to them a little, maybe i should more16:08
iurygregoryno idea =(16:08
iurygregoryany chance the jobs can run on bionic without moving to zuulv3? (just wondering)16:09
tosky(the move to zuul v3 is much needed at this point regardless)16:11
iurygregoryyeah16:11
erbarrtosky, agreed and I would love to do so, i'm trying16:11
iurygregoryhttps://developer.ibm.com/tutorials/migrating-openstack-from-zuul-2-to-3/ maybe would help16:12
iurygregorynot 100% sure16:12
erbarrfor just bionic, i understand that was tried as a simple nodepool change to dib bionic but that had some problems, i haven't tried it myself though16:13
*** lucasagomes has quit IRC16:14
TheJulialooking at the neutron log, dnsmasq is reporting no address available16:20
openstackgerritOpenStack Release Bot proposed openstack/ironic-lib stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/74977916:22
openstackgerritOpenStack Release Bot proposed openstack/ironic-lib stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/74978016:22
openstackgerritOpenStack Release Bot proposed openstack/ironic-lib master: Add Python3 wallaby unit tests  https://review.opendev.org/74978116:22
erbarrmmmnnnnn, ok, I'll look to see where those should come from and what would be different between ussuri and master16:22
erbarriurygregory: thanks, i'll take a look, I understand the overall picture and the flow, kind of just wish there was an easy install button lol, i liked the tutorial for zuul v3 with the containers, that was easy to follow16:24
*** priteau has quit IRC16:25
*** adrianc has quit IRC16:25
iurygregoryyeah install button would be good hehe16:26
*** adrianc has joined #openstack-ironic16:27
toskythe Software Factory project provides a prepackaged "zuul distribution"16:28
*** dougsz has quit IRC16:30
erbarrtosky: cool, i'll take a look thanks!16:31
*** rloo has quit IRC16:31
TheJuliaerbarr: is fa:16:3e:aa:52:43 a valid mac in your environment?16:32
*** rloo has joined #openstack-ironic16:33
*** rloo has quit IRC16:33
*** rloo has joined #openstack-ironic16:34
erbarrTheJulia, checking, with environment you mean baremetals?16:34
TheJuliaerbarr: yes16:34
erbarrok16:35
TheJuliaany hosts have that mac, because that is the mac that was recorded for use to dhcp with16:35
erbarrshould have been 44:A8:42:24:24:66 on that job, that's the target mac address for the nic16:38
TheJuliaerbarr: ignore that request, it looks like it is the dhcp address16:38
erbarrok16:38
* TheJulia looks for that16:38
TheJuliafound it16:39
*** ijw has quit IRC16:39
*** ijw_ has joined #openstack-ironic16:39
erbarrcool16:42
TheJuliait _looks_ like neutron never picked up that something had to change16:43
TheJuliafor the dhcp configuration16:44
TheJuliaand thus tftp failed16:44
TheJuliawe should see something the dhcp log when the port was last updated with all of the tftp data around 15:14:4116:44
TheJuliabut there is zilch16:44
erbarrouch16:46
openstackgerritRiccardo Pittau proposed openstack/ironic master: Replace retrying with tenacity  https://review.opendev.org/37657416:47
TheJuliathe primary neutron service logs indiacate it put the message on the message bus16:47
TheJuliawell, triggered the callback which it should16:48
TheJuliaerbarr: are you in #openstack-neutron?16:51
erbarrjoining16:51
erbarri'm in there now16:52
*** derekh has quit IRC16:55
erbarrTheJulia, thanks!17:01
*** gregwork has quit IRC17:29
*** vmud213 has quit IRC17:45
TheJuliaerbarr: are all jobs failing now on dell's CI?18:02
TheJuliaat least master branch ones?18:02
erbarryea18:03
TheJuliaIt does look like there was a change in notifications yesterday, but it looks completely unrelated area specifically with security groups... yet it is in update_port18:12
TheJuliahttps://review.opendev.org/#/c/508446/5/neutron/plugins/ml2/plugin.py18:12
patchbotpatch 508446 - neutron - Fix unnecessary security_groups_member_updated not... (MERGED) - 5 patch sets18:12
TheJuliaoh, no18:15
TheJuliathat was in 201818:15
TheJuliathey cherry picked that yesterday18:15
TheJuliaerbarr: when was the first failure exactly and when was the last success before that?18:16
*** priteau has joined #openstack-ironic18:24
erbarrthe first failure I saw started at this review https://review.opendev.org/749357 for patchset 218:32
patchbotpatch 749357 - ironic - Explicitely do not allocate initial space for virt... (MERGED) - 2 patch sets18:32
erbarrthis was the last pass https://review.opendev.org/749459 patchset 118:33
patchbotpatch 749459 - ironic - [DNM] Test multinode - 1 patch set18:33
erbarrpassing finished running at 7:19 am central time, failed job started running at 9:11 am, there was a job cancelled between the 2 due to another patchset18:35
erbarractually, found another job for that same failed trigger that started running at 8:1818:38
*** bdodd_ has joined #openstack-ironic18:44
*** bdodd has quit IRC18:46
*** priteau has quit IRC18:47
*** sshnaidm is now known as sshnaidm|off18:48
openstackgerritAija Jaunteva proposed openstack/sushy master: WIP: Allow monitoring progress of a firmware update  https://review.opendev.org/74595018:51
openstackgerritJulia Kreger proposed openstack/metalsmith master: Replace the Centos7 image job with a Centos8 UEFI based job  https://review.opendev.org/74831518:57
openstackgerritMerged openstack/ironic-python-agent master: If listen_tls is true, enable TLS on wsgi server  https://review.opendev.org/74719318:59
JayFhttps://review.opendev.org/#/c/747309/ should be ready for review now. Adding support in ipa-builder for enabling ^^ that feature.19:02
patchbotpatch 747309 - ironic-python-agent-builder - Add element to configure IPA with TLS, use configdir - 3 patch sets19:02
JayFI'll retest this tomorrow as well19:02
*** jtomasek has quit IRC19:05
*** jtomasek has joined #openstack-ironic19:17
*** jtomasek has quit IRC19:21
*** tosky has quit IRC19:45
openstackgerritChristopher Dearborn proposed openstack/ironic master: WIP: Redfish driver firmware update  https://review.opendev.org/74961919:47
*** thiagop has quit IRC19:59
*** zzzeek has quit IRC20:08
*** zzzeek has joined #openstack-ironic20:12
*** zaneb has quit IRC20:26
TheJuliaerbarr: unfortunately looking at neutron nothing seems to fall in that window :\20:34
erbarrTheJulia, thanks, I really appreciate the help20:39
openstackgerritJulia Kreger proposed openstack/metalsmith master: Replace the Centos7 image job with a Centos8 UEFI based job  https://review.opendev.org/74831520:40
*** Lucas_Gray has joined #openstack-ironic20:41
TheJuliaerbarr: the alternative idea may be to just start to revert neutron and related patches until we find the right one :\20:41
openstackgerritMerged openstack/ironic-python-agent stable/ussuri: Increase memory for tinyipa jobs  https://review.opendev.org/74763420:42
erbarrTheJulia, yea I was thinking on checking patch differences between master branch and ussuri and reverting one by one20:43
*** zaneb has joined #openstack-ironic20:44
*** zbitter has joined #openstack-ironic20:48
*** zaneb has quit IRC20:48
*** JamesBenson has quit IRC20:51
TheJuliaerbarr: you can likely just start reverting off neutron master, only 6 patches or so20:52
erbarrTheJulia, ok, is there an easy way via local.conf that you know of?20:54
TheJulianope20:54
TheJuliayou need to start nuking the patches on the neutron clone and restacking20:54
erbarrok, thanks!20:56
*** JamesBenson has joined #openstack-ironic21:08
stevebakermorning21:25
stevebakerdtantsur|afk: I'll refresh https://review.opendev.org/#/c/737888 but I don't think its a huge deal if it misses the release21:25
patchbotpatch 737888 - metalsmith - metalsmith_deployment role switch to metalsmith_in... - 17 patch sets21:25
openstackgerritMerged openstack/ironic-python-agent stable/ussuri: Make WSGI server respect listen_* directives  https://review.opendev.org/74959721:26
*** adrianc has quit IRC21:35
*** adrianc has joined #openstack-ironic21:35
*** k_mouza has joined #openstack-ironic21:38
openstackgerritChristopher Dearborn proposed openstack/ironic master: WIP: Redfish driver firmware update  https://review.opendev.org/74961921:41
*** bfournie has quit IRC21:54
*** k_mouza has quit IRC22:09
*** ijw_ has quit IRC22:25
*** ijw has joined #openstack-ironic22:25
openstackgerritJulia Kreger proposed openstack/ironic-inspector master: Power off before inspection  https://review.opendev.org/74984522:46
*** janders has quit IRC22:53
*** janders has joined #openstack-ironic22:55
jandersgood morning Ironic o/22:55
*** hjensas has quit IRC22:56
jandersmigrating to a new laptop. Managed to migrate my IRC setup and all the history is still there, woohoo :)22:56
TheJuliao/22:59
*** Lucas_Gray has quit IRC23:02
*** zzzeek has quit IRC23:05
*** zzzeek has joined #openstack-ironic23:06
*** Lucas_Gray has joined #openstack-ironic23:07
*** Goneri has quit IRC23:16
*** rcernin has joined #openstack-ironic23:24
*** Lucas_Gray has quit IRC23:25
*** ijw has quit IRC23:25
*** ijw has joined #openstack-ironic23:25
*** rcernin has quit IRC23:31
*** rcernin has joined #openstack-ironic23:31
*** zzzeek has quit IRC23:39
*** zzzeek has joined #openstack-ironic23:40
*** jamesden_ has joined #openstack-ironic23:53
*** jamesdenton has quit IRC23:56
*** zzzeek has quit IRC23:57
*** zzzeek has joined #openstack-ironic23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!