Tuesday, 2020-05-19

TheJuliahmmm00:08
TheJuliahmm00:08
TheJuliaI'm a bit stumped, but I feel like a clear brain in the morning may reveal truths of the caching code00:08
eanderssonNo worries! Thanks for the help!00:14
*** jhesketh has joined #openstack-ironic00:52
*** scott_c has joined #openstack-ironic01:09
*** scott__c has quit IRC01:11
*** scott_c has quit IRC01:15
*** yaawang has quit IRC01:22
*** yaawang has joined #openstack-ironic01:25
*** Qianbiao has joined #openstack-ironic01:34
openstackgerritlkk proposed openstack/ironic-tempest-plugin master: list port groups by node  https://review.opendev.org/72880701:35
eanderssonTheJulia upgraded to Train and same issue. It looks like is_image_available just does not think that I am the image owner01:50
*** yolanda has joined #openstack-ironic01:52
*** yolanda has quit IRC01:52
*** yolanda has joined #openstack-ironic01:53
*** rmcall has quit IRC01:53
TheJuliaFreaky02:01
eanderssonI changed that to just return True and it works 100%02:01
eanderssonI am just doing baremetal node provide <id> btw02:02
eanderssonand if I remove the reboot part of the firmware upgrade script it works 100%02:02
eanderssonWe are not actually planning on using glance for this in the final version, so technically this won't be an issue.02:04
eanderssonActually it kinda makes sense. node provide will use the ironic user to talk to glance, but the image was uploaded by my user.02:19
eanderssonSo they won't be in the same project?02:19
openstackgerritMerged openstack/ironic master: rm duplicate word 'that'  https://review.opendev.org/72881302:20
eanderssonYea that is it02:27
eanderssonDuring the startup cleaning stage the project id is an actual project id, but when it it hits restart required on my custom hardware handler it does not have a project id and fails02:29
eanderssonI reproduce it in my lab using this simple hardware manager02:31
eanderssonhttp://paste.openstack.org/show/793753/02:31
eanderssonfollowed by baremetal manage, baremetal provide02:31
eandersson> LOG.warning('context.project_id: %s image.owner: %s', context.project_id, getattr(image, 'owner', None))02:33
openstackgerritVerification of a change to openstack/networking-baremetal failed: Convert networking-baremetal job to dib  https://review.opendev.org/71869702:35
*** gyee has quit IRC03:23
*** uzumaki has joined #openstack-ironic03:52
*** cdearborn has quit IRC03:55
*** dhellmann has quit IRC04:09
*** threestrands has quit IRC04:15
*** ociuhandu has joined #openstack-ironic04:20
*** ociuhandu has quit IRC04:24
*** dhellmann has joined #openstack-ironic04:28
*** bdodd has quit IRC05:07
*** bdodd has joined #openstack-ironic05:27
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Use openstacksdk for ironic connection  https://review.opendev.org/68397105:47
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Use openstacksdk for ironic connection  https://review.opendev.org/68397105:49
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Use openstacksdk for ironic connection  https://review.opendev.org/68397105:51
arne_wiebalckGood morning, ironic!06:03
QianbiaoMorning  arne_wiebalck06:19
arne_wiebalckHey Qianbiao o/06:19
QianbiaoI am the earliest one :)06:19
Qianbiaoo/06:19
arne_wiebalckYou have a slight geographical advantage :)06:20
Qianbiao:)06:21
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Use openstacksdk for ironic connection  https://review.opendev.org/68397106:25
Qianbiaoarne_wiebalck is dtantsur on vacation. did not see him yestoday.06:45
Qianbiaodo u know that.06:45
arne_wiebalckyes, he is off this week06:45
Qianbiaook. thanks.06:45
QianbiaoHello, maelk. I find you are on this channel too.06:47
openstackgerritwangwei1 proposed openstack/ironic-prometheus-exporter master: Add coverage configuration for tox  https://review.opendev.org/72839206:49
openstackgerritlidong proposed openstack/ironic-python-agent-builder master: Add releasenotes link to README  https://review.opendev.org/72913706:49
QianbiaoFor the "NeedsMAC" issue, is it means that we need to run "openstack baremetal create node" explicitly before deploy?06:50
*** rpittau|afk is now known as rpittau06:51
rpittaugood morning ironic! o/06:51
Qianbiaogood morning rpittau06:54
rpittauhey Qianbiao :)06:55
Qianbiaohey :)06:55
iurygregorygood morning arne_wiebalck Qianbiao rpittau and Ironic o/07:04
rpittauhey iurygregory07:04
Qianbiaohi iurygregory07:04
uzumakiGood morning everybody!p/07:06
*** xinliang has joined #openstack-ironic07:06
iurygregoryhey uzumaki07:07
openstackgerritMasayuki Igawa proposed openstack/bifrost master: Remove .testr.conf  https://review.opendev.org/72914007:08
rpittauif anyone as a moment please review https://review.opendev.org/726119 it removes an annoying issue07:09
patchbotpatch 726119 - ironic-python-agent-builder - Change ownership of tce info dir - 3 patch sets07:09
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: WIP - Native zuulv3 grenade multinode multitenant  https://review.opendev.org/70503007:10
arne_wiebalckQianbiao: not sure what the "needs mac" issue is, but you will need a port which needs to be created via inspection or manually07:11
iurygregoryrpittau, I'm wondering if we want to backport and if a story is necessary07:12
rpittauiurygregory: backport what ?07:12
iurygregoryhttps://review.opendev.org/#/c/726119/07:12
patchbotpatch 726119 - ironic-python-agent-builder - Change ownership of tce info dir - 3 patch sets07:12
iurygregoryoh we have tags only07:13
iurygregorynot stable branches07:13
iurygregory=)07:13
rpittauyeah07:13
iurygregorynvm07:13
Qianbiaoarne_wiebalck check this https://github.com/metal3-io/baremetal-operator/pull/530#issuecomment-63058747607:13
*** uzumaki has quit IRC07:14
Qianbiaoit's a comment from dtantsur. I am not very clear about the logic.07:14
*** uzumaki has joined #openstack-ironic07:14
Qianbiaoaccording to his comment, i think "needmac" should be true. i will correct it first.07:17
rpittauQianbiao: the mac address is needed for pxeboot and inspection, so NeedsMAC should be true by default07:19
Qianbiaorpittau ok thanks.07:20
openstackgerritRiccardo Pittau proposed openstack/ironic master: Collect tftpd info from journald  https://review.opendev.org/71774507:23
*** TheJulia has quit IRC07:24
uzumakihey iurygregory ! how's it going?07:25
*** mnasiadka has quit IRC07:25
iurygregoryuzumaki, doing good and you?07:25
*** mnasiadka has joined #openstack-ironic07:27
uzumakiI'm well, thanks07:27
uzumakiI'm so bored and tired...07:27
iurygregoryouch =(07:28
uzumakiyeah :(07:29
*** TheJulia has joined #openstack-ironic07:29
openstackgerritRiccardo Pittau proposed openstack/python-ironic-inspector-client master: Reduce timeout for tempest job  https://review.opendev.org/72863507:35
*** ociuhandu has joined #openstack-ironic07:38
Qianbiaoquay.io is down? It seems i can not visit it.07:38
rpittauQianbiao: yes, looks like a 50007:39
rpittaualthough I don't see outages in the status page07:39
rpittauweird07:39
QianbiaoI almost doult our country network first in this case. :)07:40
Qianbiaos/doult/blame/07:40
rpittau:D07:40
rpittauyeah07:40
rpittauI tried with a private session to be sure was not cookies or cache, but it doesn't work07:40
*** uzumaki has quit IRC07:42
*** uzumaki has joined #openstack-ironic07:42
openstackgerritwangwei1 proposed openstack/ironic-prometheus-exporter master: Add coverage configuration for tox  https://review.opendev.org/72839207:45
rpittaumgoddard, TheJulia, we might be ready to release stable/train in ironic and inspector (just remembering what dtantsur|afk said last week)07:47
mgoddardrpittau: +107:47
mgoddardI'll take a look today07:47
rpittaumgoddard: thanks!07:47
rpittauI rechecked a patch in ironic train just now, hopefully it'll go through07:48
*** uzumaki has quit IRC07:52
openstackgerritMerged openstack/python-ironicclient master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/72901407:56
*** tkajinam has quit IRC08:02
*** ociuhandu has quit IRC08:03
*** ociuhandu has joined #openstack-ironic08:04
*** ociuhandu has quit IRC08:06
*** ociuhandu has joined #openstack-ironic08:06
*** yaawang has quit IRC08:08
*** yaawang has joined #openstack-ironic08:09
rpittauany good samaritan that has a minute to spare to review this https://review.opendev.org/727125 and save us some rechecks ?08:10
patchbotpatch 727125 - ironic - Increase timeout value for ironic-base job - 2 patch sets08:10
iurygregorywow 900 ftw08:11
*** lucasagomes has joined #openstack-ironic08:12
*** dougsz has joined #openstack-ironic08:13
openstackgerritRiccardo Pittau proposed openstack/python-ironicclient master: Update lower-constraints.txt  https://review.opendev.org/72724308:14
iurygregoryApproved since it was a trivial change to help our CI =)08:16
iurygregorylol I wrote and forgot to send XD sorry rpittau ^08:17
*** ociuhandu has quit IRC08:17
rpittauiurygregory: it's ok, thanks :)08:17
iurygregoryrpittau, arne_wiebalck just FYI we have III SPUC08:18
iurygregorythis friday08:18
openstackgerritMahendra Kamble proposed openstack/ironic master: DRAC : idrac-redfish inspect updates pxe port  https://review.opendev.org/69332808:18
*** ociuhandu has joined #openstack-ironic08:18
rpittauiurygregory: I saw it but I will ahve to skip again :/08:18
*** yaawang has quit IRC08:19
iurygregoryouch =(08:20
*** yaawang has joined #openstack-ironic08:21
*** ociuhandu has quit IRC08:23
*** alexmcleod has joined #openstack-ironic08:25
*** uzumaki has joined #openstack-ironic08:26
arne_wiebalckiurygregory: this may be a little late for me too ... although I like how this is an occasion to give "useful operator feedback" :D08:34
iurygregoryarne_wiebalck, hehehe08:35
iurygregoryarne_wiebalck, rpittau you can blame JayF for the time =)08:39
openstackgerritwangwei1 proposed openstack/ironic-prometheus-exporter master: Add coverage configuration for tox  https://review.opendev.org/72839208:40
openstackgerritMerged openstack/ironic-python-agent-builder master: Change ownership of tce info dir  https://review.opendev.org/72611908:49
*** lbotell has quit IRC08:57
*** ociuhandu has joined #openstack-ironic09:07
openstackgerritMerged openstack/ironic-inspector master: Stop running a periodic task for the noop PXE filter  https://review.opendev.org/72577009:10
openstackgerritMerged openstack/python-ironic-inspector-client master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/72901509:25
*** ociuhandu has quit IRC09:25
*** ociuhandu has joined #openstack-ironic09:26
openstackgerritRiccardo Pittau proposed openstack/python-ironic-inspector-client master: Reduce timeout for tempest job  https://review.opendev.org/72863509:26
openstackgerritVerification of a change to openstack/ironic-inspector failed: Update lower-constraints.txt  https://review.opendev.org/72723709:32
*** sshnaidm|afk is now known as sshnaidm09:40
*** k_mouza has joined #openstack-ironic09:48
*** xinliang has quit IRC10:20
*** derekh has joined #openstack-ironic10:26
openstackgerritRiccardo Pittau proposed openstack/python-ironic-inspector-client master: Update lower-constraints.txt  https://review.opendev.org/72724510:28
*** tkajinam has joined #openstack-ironic10:30
*** rpittau is now known as rpittau|bbl10:31
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: Build tinyipa on focal  https://review.opendev.org/72579910:31
*** k_mouza has quit IRC10:32
*** uzumaki has quit IRC10:36
*** xiaolin has quit IRC10:48
*** uzumaki has joined #openstack-ironic10:51
*** k_mouza has joined #openstack-ironic11:00
*** xiaolin has joined #openstack-ironic11:05
openstackgerritMerged openstack/ironic master: Increase timeout value for ironic-base job  https://review.opendev.org/72712511:12
openstackgerritMerged openstack/ironic-prometheus-exporter master: Add coverage configuration for tox  https://review.opendev.org/72839211:12
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Use openstacksdk for ironic connection  https://review.opendev.org/68397111:16
openstackgerritMerged openstack/ironic master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/72895711:26
*** dougsz has quit IRC11:34
*** aedc_ has joined #openstack-ironic11:39
*** uzumaki has quit IRC11:39
*** ociuhandu has quit IRC11:40
*** aedc has quit IRC11:41
*** ociuhandu has joined #openstack-ironic11:42
*** ociuhandu has quit IRC11:47
*** aedc_ has quit IRC11:53
*** sean-k-mooney has quit IRC11:53
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: WIP - Native zuulv3 grenade multinode multitenant  https://review.opendev.org/70503011:58
*** dougsz has joined #openstack-ironic11:59
openstackgerritMerged openstack/ironic-inspector master: Add Python3 victoria unit tests  https://review.opendev.org/72357112:01
*** rpittau|bbl is now known as rpittau12:04
*** jawad_axd has joined #openstack-ironic12:05
*** Qianbiao has quit IRC12:06
*** dougsz has quit IRC12:09
openstackgerritBob Fournier proposed openstack/ironic master: New configuration parameter to use ipmitool retries  https://review.opendev.org/72826112:17
*** v1k0d3n has joined #openstack-ironic12:23
*** Qianbiao has joined #openstack-ironic12:28
openstackgerritMerged openstack/bifrost master: Remove .testr.conf  https://review.opendev.org/72914012:49
*** rloo has joined #openstack-ironic12:52
openstackgerritAndreas Jaeger proposed openstack/bifrost master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/72924512:53
mgoddardrpittau: how is train looking?12:55
TheJuliarpittau: as long as backports have landed, I think we should be good12:56
TheJuliamgoddard: ^^^12:56
rpittaumgoddard, TheJulia, checking12:56
* TheJulia makes coffee12:56
rpittauone patch still needs to land in ironic12:56
*** ociuhandu has joined #openstack-ironic12:57
rpittauit's in recheck since a couple of hours12:57
rpittauinspector seems ok12:57
QianbiaoThe CI is really like a tortoise ;(12:58
Qianbiaoshould add more jenkins nodes.12:58
iurygregorygood morning TheJulia =)13:00
*** derekh has quit IRC13:01
*** derekh has joined #openstack-ironic13:01
openstackgerritVerification of a change to openstack/ironic-inspector failed: Update local.conf example file.  https://review.opendev.org/72644213:01
openstackgerritRiccardo Pittau proposed openstack/bifrost master: Install python3-pip system package  https://review.opendev.org/72924913:02
openstackgerritRiccardo Pittau proposed openstack/ironic-inspector stable/ussuri: Update TOX_CONSTRAINTS_FILE for stable/ussuri  https://review.opendev.org/72356913:03
TheJuliaQianbiao: too many integration test jobs :(13:06
*** ociuhandu has quit IRC13:06
Qianbiao<TheJulia> i find metal3 use libvirt to simulate bm. may it save resources and  speed up the test jobs13:07
TheJuliaBasically the same thing occurs for the integration jobs, about half the time is just installing software13:08
TheJuliafrom source13:08
Qianbiao:(13:08
*** Goneri has joined #openstack-ironic13:09
TheJuliawe also have two upgrade jobs, they perform a smoke test upgrade to latest, re-test13:09
iurygregoryno smoke test anymore =)13:10
rpittaumgoddard, TheJulia: inspector should be good to go13:11
mgoddardcool. I'll go through all the deliverables and see which have changes13:11
*** elemaill has joined #openstack-ironic13:13
*** tzumainn has joined #openstack-ironic13:19
openstackgerritRiccardo Pittau proposed openstack/ironic-inspector master: Increase timeout for introespection in ironic-inspector-tempest-managed  https://review.opendev.org/72925413:23
TheJuliavdrok: glanced at the linux kernel source for https://review.opendev.org/#/c/728529, changed my mind and my review13:23
patchbotpatch 728529 - ironic-python-agent - Add possibility to pass global request ID - 1 patch set13:23
openstackgerritRiccardo Pittau proposed openstack/ironic-inspector master: Increase timeout for introspection in ironic-inspector-tempest-managed  https://review.opendev.org/72925413:23
*** jdandrea has joined #openstack-ironic13:23
vdrokHey TheJulia , oh, good to know, thanks!13:24
TheJuliaI also commented on the ironic patch on the lengths13:24
iurygregoryrpittau, not sure if it will helpp increase .-. if I do remember this was the oslo problem that dtantsur|afk found...13:24
openstackgerritMerged openstack/bifrost master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/72924513:25
*** m1p has joined #openstack-ironic13:25
rpittauiurygregory: do you have a link handy for that ? I see the time increased there but no other issues13:25
vdrokAs for patches, I guess there is one important in client, https://review.opendev.org/72846413:25
patchbotpatch 728464 - python-ironicclient - Allow to pass additional arguments into adapter - 1 patch set13:25
*** ociuhandu has joined #openstack-ironic13:26
iurygregoryrpittau, let me see if I can find13:26
m1phello ironic. any known issues with node provisioning? "attributes" db tables remains empty during "node provide", so nodes are getting blacklisted and won't pxe boot. env is ussuri/centos8.13:26
iurygregoryrpittau, nvm it was in the ironic-inspector-non-standalone-tempest =)13:30
rpittauiurygregory: ok, thanks13:31
*** ociuhandu has quit IRC13:31
TheJuliam1p:  did you run ironic-dbsync ?13:35
openstackgerritRiccardo Pittau proposed openstack/python-ironic-inspector-client master: Reduce timeout for tempest job  https://review.opendev.org/72863513:35
TheJuliam1p: what does the last_error have populated?13:35
*** s8k has joined #openstack-ironic13:35
m1pTheJulia: undercloud deployment with tripleo, didn't do any manual steps. last_error is abort or timeout, depending. introspection works fine, i can see the attributes table getting populated, so the macs get whitelisted for pxe boot. during node provide, table stays empty13:39
TheJuliawhich table stays empty? I ask because we don't use an attributes table for cleaning/deployment13:39
openstackgerritRiccardo Pittau proposed openstack/ironic master: Increase timeout for introspection in ironic-inspector-tempest  https://review.opendev.org/72925913:40
m1pTheJulia: ironic-inspector.attributes13:40
TheJuliadifferent service13:40
TheJuliathe db you would be looking at is ironic13:40
*** cdearborn has joined #openstack-ironic13:40
TheJuliaironic-inspector has absolutely nothing to do with providing nodes to ironic13:40
m1pmy bad, thx13:41
m1pthen i need to find out why the nodes are blacklisted13:41
TheJuliam1p: time out generally means that we never heard from the agent13:41
TheJuliam1p: where are you percieving they are being blacklisted at?13:41
m1pthe macs are blacklisted in dnsmasq with ,ignore13:41
TheJuliayeah, for ironic-inspector13:41
TheJuliaNOT neutron as part of deployment13:41
m1pnodes hang in pxe boot waiting for response during node provide, that's why i assumed it's the same thing13:42
TheJuliaseparate dnsmasq running in a namespace13:42
TheJuliam1p: do you see anything on the console from the start of the pxe boot?13:42
m1pyes, it's not getting any reply13:43
TheJuliasounds like your machine is not actually connected to the same network used for cleaning/provisioning13:43
TheJuliaat least, as far as neutron is concerned13:43
m1pwith at least 4-5 redeployments per week and no relevant code changes in the repo? unlikely.13:44
m1pwhere can i check it13:44
TheJuliam1p: you need to backtrack from with-in the namespace (ip netns list) and test connectivity across interfaces (ip netns exec namespace-id ping some-address-on-the-physical-network13:44
TheJuliaso your saying this is actually a sporatic issue?13:45
TheJuliaor is it always happening?13:45
m1psame repo deployed just fine a month ago. i'll get back to you when i find something13:45
m1pi'm saying it's the first time this issue occured13:45
TheJuliaYeah, that is weird13:45
TheJuliaI'd check physical networking, maybe even try tcpdumping inside the namespace to see if it is seeing the dhcp packets13:46
m1pare you sure? as i said, ironic-inspector pxe boots the nodes without issues and introspects them.13:48
*** ociuhandu has joined #openstack-ironic13:48
*** ociuhandu has quit IRC13:54
TheJuliam1p: very sure, they are attached to networking differently14:02
m1pTheJulia: hmm, ok. i'll dive in tomorrow. it's really weird because it fails so early. the nodes won't even pxe boot, server response timeout. physical connectivity is 100% good. i'll get back when i have identified the issue.14:08
m1pthx for your time14:08
TheJuliam1p: are they in maintenance state?14:09
m1pno14:09
TheJuliaare they showing as available computes in nova?14:09
TheJuliacompute hypervisors that is14:09
m1pshould they this early?14:10
TheJuliawell, nova will fail hard early on14:10
TheJuliabut depends on exactly where errors are occuring14:10
TheJuliaIt sounds like time is not actually even passing for the timeout to uccur14:10
TheJuliaoccur14:10
*** tkajinam has quit IRC14:11
openstackgerritRiccardo Pittau proposed openstack/bifrost master: Update docs references for CentOS 8  https://review.opendev.org/72927314:11
m1pi think i didn't express myself clearly. fresh undercloud install, import nodes from instackenv, introspect nodes. until here everything is fine. then trying to move nodes from manageable to available state. here it happens, the nodes won't pxe boot for cleaning.14:11
*** tkajinam has joined #openstack-ironic14:11
m1pno nova14:11
TheJuliaahh14:11
TheJuliaand they power on I take it?14:11
m1pyes14:12
*** jawad_axd has quit IRC14:12
m1pthen server response timeout while waiting for pxe response14:12
TheJuliayeah, this sounds like networking, specificially with however neutron is attaching with the namespace14:12
*** jawad_axd has joined #openstack-ironic14:12
m1phow can i check it?14:12
m1pas i'm using tripleo, wouldn't an error here also affect ironic-inspector? or are they completly separated in regards to networking?14:14
TheJuliayou need to find the namespace neutron is using and start working from with-in that namespace to test networkign14:15
TheJuliaif you can see tcpdump packets, thats a good sign attachment wise14:15
TheJuliaand it is a completely separate dnsmasq instance than what ironic-inspector uses14:15
TheJuliait doesn't bind that from with-in the namespace14:15
openstackgerritRiccardo Pittau proposed openstack/networking-baremetal master: Convert networking-baremetal job to dib  https://review.opendev.org/71869714:15
openstackgerritRiccardo Pittau proposed openstack/networking-baremetal master: Convert networking-baremetal job to dib  https://review.opendev.org/71869714:16
*** jawad_axd has quit IRC14:18
iurygregoryrpittau, seems like manage doesn't help if we are trying to update the volume_connector =(14:19
iurygregorywondering if the node should be in manage before the volume connector is created14:21
*** riuzen has joined #openstack-ironic14:22
*** riuzen has quit IRC14:23
openstackgerritRiccardo Pittau proposed openstack/bifrost master: Update docs references for CentOS 8  https://review.opendev.org/72927314:28
TheJuliaiurygregory: what seems to be the issue?14:35
iurygregoryTheJulia, The requested action "volume connector update" can not be performed on node "20530ed7-232a-4b30-8a50-71d5406b9b17" while it is in state "None".14:36
iurygregoryhttp://paste.openstack.org/show/793770/ this is the code14:37
TheJuliaand this is a brand new error that has started to appear?14:37
TheJulialooking14:37
iurygregorythis is in the functional tests for openstacksdk =)14:37
TheJuliaI believe it does need to be manageable before you can update the target14:38
TheJuliaerr14:38
TheJuliaconnector14:38
iurygregoryyeah14:38
*** uzumaki has joined #openstack-ironic14:38
iurygregoryafter the connector is created it sets the node to manage14:39
iurygregoryor maybe it should be done before L6 where it adds .extra = {'answer': 42} ?14:40
TheJulialine 8 before line 214:41
iurygregoryok =)14:41
iurygregorytks!14:41
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Use openstacksdk for ironic connection  https://review.opendev.org/68397114:42
*** ociuhandu has joined #openstack-ironic14:46
*** jawad_axd has joined #openstack-ironic14:49
*** jawad_axd has quit IRC14:54
iurygregoryrpittau, maybe we should also increase the time for ironic-inspector-tempest in the same patch =)14:55
rpittauiurygregory: which patch ?14:56
elemaillHello, i am trying to deploy nodes with bifrost fresh install (ussuri/CentOS 8)  all seems to be fine, nodes are enrolled and pxe boot but the dhcp request from nodes are unanswered (i see request on the localhost/target machine) any idea what could be wrong ?14:57
iurygregoryrpittau, https://review.opendev.org/#/c/729254/ =)14:57
patchbotpatch 729254 - ironic-inspector - Increase timeout for introspection in ironic-inspe... - 2 patch sets14:57
rpittauiurygregory: ironic-inspector-tempest is defined in ironic, I put up another patch for that https://review.opendev.org/72925914:58
patchbotpatch 729259 - ironic - Increase timeout for introspection in ironic-inspe... - 1 patch set14:58
iurygregoryrpittau, ++14:58
TheJuliaeandersson: so what you described seems like a bug because we should be using the ironic user on clean to re-query... but yes they do need to be in the same project. I guess it makes sense to check if there is no context, then to fallback to the service level credentials14:58
iurygregoryI probably missed this one14:58
rpittauand probably needs to be merged first as it keep failing in ironic-inspector :/14:59
iurygregoryrpittau, yup =(14:59
gudrutis2Hi ironic 0/15:06
gudrutis2can I specify yum parameters when in `package-installs.yaml` ?15:06
*** hamzy_ is now known as hamzy15:09
* TheJulia looks at tests and could have sworn writing a number of tests15:09
*** Qianbiao has quit IRC15:13
erbarrare there explicit tests in ironic-tempest-plugin for bios interfaces like drac-wsman?15:15
openstackgerritMerged openstack/ironic stable/ussuri: Collect ramdisk logs also during cleaning  https://review.opendev.org/72863015:19
openstackgerritVerification of a change to openstack/ironic failed: Add `network_data` field to ironic node object  https://review.opendev.org/68791015:19
openstackgerritMerged openstack/ironic-inspector stable/ussuri: Update TOX_CONSTRAINTS_FILE for stable/ussuri  https://review.opendev.org/72356915:19
*** jawad_axd has joined #openstack-ironic15:20
TheJuliaerbarr: no, but your welcome to add them15:22
TheJuliajust need to explicitly navigate the drivers and all15:22
TheJuliaAnd also, maybe put a handle15:22
TheJulias/handle/knob15:22
*** dking has joined #openstack-ironic15:23
TheJuliaerbarr: mainly because changing them on random customers might be bad15:24
erbarrTheJulia: agreed, i'm just wondering how that's covered, I'm guessing it gets excercised as part of the settings used to run basic_ops or am I assuming too much?15:26
TheJuliano, explicit test case would be needed15:27
erbarrok, thanks, soo I need to learn a lot then lol15:27
iurygregoryerbarr, is not that hard I would say =)15:28
erbarriuerygregory, lol sounds like what a helpful person would say ;)15:30
*** riuzen has joined #openstack-ironic15:32
*** riuzen has quit IRC15:33
openstackgerritMark Goddard proposed openstack/bifrost stable/ussuri: Switch git_branch to stable/ussuri  https://review.opendev.org/72930215:37
openstackgerritRiccardo Pittau proposed openstack/ironic master: Add IPv6 ci Job  https://review.opendev.org/72408715:45
*** gyee has joined #openstack-ironic15:46
v1k0d3nhey everyone...i've been reading through ironic docs, and am having issues at the point where i'm starting an instance.15:46
v1k0d3nTheJulia said that if i bring up cars and whiskey, i might be able to get a little bit of help?15:47
v1k0d3nfine with folks sending me back to docs i might have missed as well. but i wanted to boot a bm instance off of a vlan...i'm assuming this is perfectly fine?15:48
TheJulialol15:50
TheJuliaso what seems to be the issue?15:50
v1k0d3ni seem to be going through browser tab-mania looking for something on this....15:50
openstackgerritRiccardo Pittau proposed openstack/ironic master: [WIP] Fix grub2 pxe job with native ubuntu ovmf package  https://review.opendev.org/71688915:50
TheJulia5 million tabs later... :)15:50
v1k0d3nwell, so i have a vlan101 that I would like to be my pxe network.15:50
erbarrahh, cars and whiskey, good to know lol15:50
v1k0d3nerbarr cars for sure...i love cars and motorcycles. so hopefully in good company? :)15:51
TheJuliayes, definitel15:52
TheJuliay15:52
TheJuliaI won't get on a motorcycle due to a crash a long time ago, but I have several car projects this weekend15:53
TheJuliawhich reminds me, I need to call the ford dealer.... One apparently has a safety recall :(15:53
v1k0d3nso i can share config...but i guess...when i create a bm node...i see that flat/noop are used?15:53
v1k0d3nfor the network_interface? i thought that it might be vlan? or am i understanding this incorrectly15:54
v1k0d3n?15:54
TheJuliav1k0d3n: so are you using a ml2 networking driver managing switches?15:54
TheJuliain other words, will machines be on tenant networks15:54
v1k0d3nuse...so i am providing a trunk port to the NIC being used...and then in openstack creating the VLAN ID's for each of the networks.15:55
v1k0d3ndoes that help?15:55
*** ociuhandu has quit IRC15:55
TheJuliakind of15:55
openstackgerritRiccardo Pittau proposed openstack/ironic master: [WIP] Fix uefi jobs with native ubuntu ovmf package  https://review.opendev.org/71688915:55
TheJuliaso noop is intended for standalone usage where you have static dhcp configured to signal/control machines15:55
*** ociuhandu has joined #openstack-ironic15:55
v1k0d3nwhat other details can i provide. there's a lot of terminology in ironic...i want to make sure i'm speaking the same lang/terms.15:55
TheJuliaflat is neutron with all the hosts on a pre-configured never changing vlan or otherwise "flat" network15:56
*** jawad_axd has quit IRC15:56
v1k0d3nah....ok, that makes sense.15:56
TheJuliaonly the neutron interface has all the smarts and logic to tell neutron to "change x port to y network for z host"15:56
v1k0d3nso in this case...there is no other devices controlling DHCP for that VLAN15:56
TheJuliaboth flat/neutron interfaces use neutron in the backend15:57
v1k0d3nthat explains flat/noop better.15:57
TheJuliajust flat is basically "dhcp only, no dynamic configuration otherwise"15:57
*** lucasagomes has quit IRC15:57
v1k0d3nwhich is best? should i do DHCP externally via an upstream device? because i can do that pretty easily.15:57
TheJuliait really depends on your use case and how you want to run the environment15:58
TheJuliaand also requirements15:58
v1k0d3nright now i'm still in testing phases. so requirements aren't very defined yet.15:58
TheJuliathe neutron interface basically requires a neutron ml2 driver for the switch with extra port structural information so basically the ml2 plugin can login to the switch and change the vlan tagging15:58
v1k0d3ni would say best case...or the case that is the most flexible/preferred.15:59
v1k0d3noh!15:59
v1k0d3nso that probably won't happen in this case then.15:59
TheJuliaokay15:59
v1k0d3ni'm so glad you mentioned that.15:59
*** ociuhandu has quit IRC15:59
TheJuliaare you using nova at all?15:59
TheJuliaor intending to use it15:59
*** ociuhandu has joined #openstack-ironic16:00
v1k0d3nbecause, since this is a home lab...i really only have Ubiquiti switches, and a pfSense box available.16:00
v1k0d3nreconfiguring is going to be difficult, i would imagine...or at least more complex.16:00
TheJuliamaybe, maybe not16:00
TheJuliabut for home labs, if your cool with managing your own dhcp, then you really don't need neutron at all16:00
TheJuliaand you can defintiely run with noop16:00
TheJuliawe somewhat document that in bifrost's documents16:01
v1k0d3ni feel like i just took the red pill....16:01
*** jawad_axd has joined #openstack-ironic16:02
TheJuliaThis can be good or bad16:02
v1k0d3nso that really explains a big part that i wasn't wrapping my head around...now i understand the intentions a lot better.16:02
iurygregorymatrix reference detected =)16:03
v1k0d3nit's a good thing. unleash flying mode.16:03
v1k0d3niurygregory ++16:03
erbarrglad is that one16:03
v1k0d3nso can you help me understand the "with nova" or "without nova" part?16:04
v1k0d3ni guess i just always assumed "with nova...like, duh". so i have options? and what does that buy me, really?16:05
TheJuliaso nova basically wants to create virtual ports in neutron for ip address management purposes.16:05
TheJuliadepends on how you wish to deploy your baremetal16:05
v1k0d3nok, that makes sense.16:05
v1k0d3nmy _thought_ was that I would be able to select which vlans a secondary interface would be able to attach to.16:06
TheJuliaso, generally in most cases people manage that kind of stuff with neutron16:07
TheJuliaBUT!16:07
v1k0d3nso my thinking (may be incorrect) is that I would have a vlan for pxe...which is dedicated for that (101), and then through openstack (neutron/nova) i would be able to attach that secondary interface to public (vlan 4).16:07
TheJuliayou said a home lab, do you just mean passing configuration to the nodes saying "use x vlan on y interface"?16:07
*** yolanda has quit IRC16:07
TheJuliayeah, that is basically a point at which you need all of the networking mechanics16:08
TheJuliaalso, pxe basically needs to be untagged16:09
TheJuliasome cards let you change the default config so the pxe is on a tagged network16:09
TheJuliabut yeah16:09
TheJuliaSorry, we're full of sad unfortunate truths16:09
TheJuliaUnfortunate Truths as a Service16:09
v1k0d3nhttps://usercontent.irccloud-cdn.com/file/dmcxAS5X/vlan-setup.png16:09
v1k0d3nso basically, that is my home lab setup today.16:10
v1k0d3nok, so the main interface coming in is actually untagged. i tag it on the neutron side...and let neutron manage who get's what vlan when bootrapping an instance.16:11
TheJuliaWow, way better diagram than my ascii list16:11
iurygregoryO.o omg16:12
v1k0d3noh boy...whenever i see "omg"...not sure if i've some something good or not.16:12
iurygregorygood =)16:13
v1k0d3nty :)16:13
v1k0d3nso these are mostly supermicro small servers for testing.16:13
v1k0d3ncurrently testing out deployments for 5G far edge and RT kernel testing16:14
v1k0d3nso now i can explain a bit...16:14
v1k0d3ne2 on the openstack environment AIO is an untagged interface coming in...that is configured to be tagged on the neutron side. so i think that's a good thing, right?16:15
v1k0d3nthe pfSense is the gateway at each of the VLANS, that is where I could enable DHCP if i have to...or if the ubnt switches can be configured for neutron autoconfiguration...i'd be happy to try that. haven't see anyone do that yet though, because I haven't been searching for it yet.16:18
openstackgerritVerification of a change to openstack/ironic failed: Fix image type reference  https://review.opendev.org/72888416:19
TheJuliasorry, on a call at the moment16:21
v1k0d3nno worries :)16:22
rpittaubye everyone, see you on monday! o/16:22
*** rpittau is now known as rpittau|afk16:22
iurygregorybye rpittau|afk16:24
TheJuliablarg conference calls16:39
TheJuliav1k0d3n: so if your not moving the ports around and your always okay with the machines being on vlan4... you could just try generating and injecting network configuration16:40
v1k0d3nso i guess...what if i do what to dynamically change to various vlans on the other interfaces for these two servers?16:41
TheJuliayou'd basically need neutron16:46
TheJuliaand an ml2 driver16:46
*** uzumaki has quit IRC16:46
TheJuliaOR your welcome to propose a new driver16:46
TheJuliaa new network interface driver that is16:46
v1k0d3nok, so I think that's ok16:53
v1k0d3nlet me try to look in docs for how i would set up neutron/nova for this. i think this is where i keep getting confused TheJulia16:56
openstackgerritMerged openstack/ironic master: Mark more configuration options as reloadable  https://review.opendev.org/72586716:56
openstackgerritMerged openstack/ironic-inspector stable/ussuri: Update local.conf example file.  https://review.opendev.org/72644216:56
*** derekh has quit IRC16:59
*** k_mouza has quit IRC16:59
*** ociuhandu has quit IRC17:01
*** ociuhandu has joined #openstack-ironic17:02
*** ociuhandu has quit IRC17:04
*** ociuhandu has joined #openstack-ironic17:04
*** scott_c has joined #openstack-ironic17:05
openstackgerritMerged openstack/ironic-tempest-plugin master: Add basic IPv6 support to BaremetalSingleTenant  https://review.opendev.org/72266317:05
v1k0d3ni'm not sure i understand yet if i can do this in my setup or not.17:06
TheJuliayou can use neutron without nova, fwiw17:06
*** scott_c has quit IRC17:14
*** jawad_ax_ has joined #openstack-ironic17:23
*** jawad_axd has quit IRC17:27
*** ociuhandu has quit IRC17:28
*** ociuhandu has joined #openstack-ironic17:29
eanderssonTheJulia, I had arne_wiebalck help med reproduce it as well17:32
eanderssonI was slightly wrong, basically there is no project_id and that is why it does not match the image_id.17:33
eandersson*image project_id17:33
eanderssonLooking at the code you can do reboot_requested during the cleaning phase if the image is public or if you don't use glance.17:34
*** ociuhandu has quit IRC17:41
*** jawad_ax_ has quit IRC17:45
*** jawad_axd has joined #openstack-ironic17:45
eanderssonarne_wiebalck https://storyboard.openstack.org/#!/story/200769517:48
TheJuliahmm, might be a good case to just use deploy kernel/ramdisk from a webserver instead of glance17:51
eanderssonYea - we are not actually going to use Glance17:51
eanderssonbut this was very confusing :D17:51
TheJuliav1k0d3n: You can, you'll likely need some glue someplace. The key is kind of where do you want glue or not17:51
arne_wiebalckeandersson: thanks, I added the error message I saw this morning.17:51
TheJuliaeandersson: yeah, I imagine so :(17:52
eanderssonAt least if there is a bug people have some reference (and workarounds)17:52
arne_wiebalckbye everyone o/17:55
eanderssono//17:55
*** ociuhandu has joined #openstack-ironic18:31
*** rcernin has quit IRC18:34
*** jawad_axd has quit IRC18:53
openstackgerritZane Bitter proposed openstack/ironic master: Revert "Deprecate ibmc"  https://review.opendev.org/72935518:57
TheJuliawtf18:59
*** k_mouza has joined #openstack-ironic18:59
openstackgerritZane Bitter proposed openstack/ironic master: Revert "Deprecate ibmc"  https://review.opendev.org/72935519:00
TheJuliazaneb: approved19:01
TheJuliagreat catch19:01
*** k_mouza has quit IRC19:04
*** ociuhandu has quit IRC19:09
*** ociuhandu has joined #openstack-ironic19:09
openstackgerritMerged openstack/ironic master: Add `network_data` field to ironic node object  https://review.opendev.org/68791019:10
*** ociuhandu has quit IRC19:15
iurygregorywoot merged \o/19:16
*** scott_c has joined #openstack-ironic19:24
*** dougsz has joined #openstack-ironic19:27
TheJulia\o/19:32
TheJulianext!19:32
v1k0d3nyeah i'm not sure what to do TheJulia...19:33
iurygregorythe next in chain already has your +2 TheJulia =)19:33
TheJuliaiurygregory: queue mr. burns doing his "excellent" response19:34
iurygregoryhehehe19:34
TheJuliav1k0d3n: Hardware is always a bit of a choose your own adventure so I suspect a good starting point is see if you can just deploy on the locally attached network, then add in the external network, and if you then want to go further, just take a logical step at a time19:34
v1k0d3ni guess, not even sure where to start, i guess. whatever is the easiest path?19:34
TheJuliawe'll be here, to at least provide crazy ideas19:35
TheJuliav1k0d3n: bifrost is typically an easy starting path19:35
v1k0d3ni don't know if i really understand the difference between the two.19:35
v1k0d3ni can't really have ironic configure vlans, so i can give Kolla another raw interface to use (which i just configured), and let pfsense do the dhcp allocation.19:36
v1k0d3nmaybe at some point i can work on something w/unifi...but have to see it working and then solve that need after i understand a bit more.19:37
TheJuliaI feel like you've kind of got like a matrix of constraints and requirements that you need to sort through19:37
TheJulialike... you can do manual dhcp management, but then your using the noop network interface19:37
TheJuliano neutron, no networking integration really19:38
TheJuliayou CAN create and inject configuration into deployed nodes that turn on things like other network interfaces, but to manage that at a switch level is a different service19:38
v1k0d3ni think that's the easiest fro now...and i can go with the manual networking for now.19:38
v1k0d3nbut things are failing for noop as well.19:39
TheJuliahow so?19:39
v1k0d3nso really it's the troubleshooting part. sort of like when i was trying to explore octavia...didn't really know _which_ logs to look for, and then some of the logs where a bit cryptic or standard "ignore that one..." sort of stuff.19:39
v1k0d3nif you know what i mean :)19:40
TheJuliaokay, well, you've started down a path to deployment, what do you see around yourself. To re-use our most used question "what do you see on the console of the machine"19:40
v1k0d3nlet me try noop again, and see. wish i had something working and could work backwards.19:40
TheJuliathe node last_error field returned by ironic's api can help us help you quite a bit19:40
* TheJulia forgot how convoluted the pxe code was19:41
v1k0d3nwell, i see ipmitool running and getting a poweron status back from the host19:44
v1k0d3nError response 0xc1 from Get PICMG Properities19:44
v1k0d3nthat's the last error message19:44
TheJuliayou can safely ignore that19:44
v1k0d3nhahaha!19:44
TheJuliait is coming from the ipmitool command in debug output mode19:44
TheJuliaif you turn off debug logging, it will go away19:44
v1k0d3nsame here, i guess?19:45
v1k0d3nhttps://www.irccloud.com/pastebin/lMpb9n7r/19:45
v1k0d3nwith all of that...that's all debug i guess19:45
v1k0d3nyeah19:46
TheJuliayeah19:50
TheJuliadoes ironic report the power state of the node?19:50
v1k0d3nit does. that says that the power is on.19:50
*** jawad_axd has joined #openstack-ironic19:50
v1k0d3ni did get past that early on by testing a non-working port...making sure it's different when working vs not working.19:50
v1k0d3nso i know that ipmitool is at least communicating with the bmc port correctly.19:51
TheJuliaokay19:55
TheJuliaso what provision_state is the machine in? is the maintenance flag set?19:56
v1k0d3nlooking at some other items in my setup to make sure ports are configured correctly, etc.19:56
*** xiaolin has quit IRC19:56
*** uzumaki has joined #openstack-ironic20:08
*** lbotell has joined #openstack-ironic20:23
lbotellHi all, has anybody worked with laptops as baremetal nodes?20:24
*** dougsz has quit IRC20:27
*** lbotell37 has joined #openstack-ironic20:28
*** scott__c has joined #openstack-ironic20:28
*** scott_c has quit IRC20:30
eanderssonlbotell it's difficult because laptops etc wont have impi / redfish support20:32
lbotellSince laptop will not auto start after a power reset I was wondering if there is a way to mix drivers, in this case wake-on-lan and snmp PDUS so that I can power off the laptop with the snmp PDU and power it on using wake-on-lan20:33
TheJulialbotell: afaik no20:34
TheJulialbotell: you could make a hybrid hardware type20:34
* TheJulia needs to go lay down20:35
TheJuliabbl20:35
lbotellOhh I see, where can I start?20:35
TheJulialbotell: well, let me get you a link20:35
TheJulialbotell: sometihing kind of like https://opendev.org/x/ironic-staging-drivers, however if you look at setup.cfg you'll see entry_points references to drivers. At the same time you will see a ironic.hardware.type field with various hardware types20:36
TheJuliahttps://opendev.org/x/ironic-staging-drivers/src/branch/master/ironic_staging_drivers/wol/__init__.py#L23 <-- wake on lan20:37
TheJuliahttps://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/generic.py#L37 <-- is the generic hardware class that is based upon20:38
*** elemaill has quit IRC20:38
lbotellOk, let me take a look at those20:40
lbotellI appreciate your help, thanks TheJulia , eandersson20:41
v1k0d3nso TheJulia looks like the networking (outside of neutron, to be clear) is working correctly....the two hosts i have will boot with MaaS on a separate vlan when I tag it properly for the older MaaS setup I have.20:44
*** jawad_axd has quit IRC20:44
*** Goneri has quit IRC21:03
*** jdandrea has quit IRC21:05
TheJuliav1k0d3n: okay, so if you do simillarly with ironic, what do you encounter?21:09
v1k0d3nTheJulia i see ipmitool talk with the management port...and nothing else. it looks like it deploys to an extent...but can't find relevant logs or output.21:10
v1k0d3ni don't really want to give up until i can get a bit further.21:11
TheJuliaso what is the very last action your performing with ironic?21:12
v1k0d3nnode create > node manage > node provide (available) > openstack server create > fail21:12
TheJuliaokay!21:12
TheJuliaso, without neutron, your not going to be using openstack server create21:13
v1k0d3noh well snap. ok then!21:13
TheJuliaThis is likely where things start to go sideways in the nova-compute log21:13
v1k0d3nso then...i guess what should i expect next?21:13
v1k0d3nok, that's what i thought...but couldn't find a lot in the logs really.21:14
TheJuliaits okay21:14
TheJuliaI'm looking for some specific docs, give me a few minutes21:14
v1k0d3nawesome! thank you!!21:14
TheJuliahttps://docs.openstack.org/ironic/latest/user/ <-- super high level docs21:17
TheJuliaBut what you really want is ---> https://docs.openstack.org/ironic/latest/install/standalone.html21:17
*** Goneri has joined #openstack-ironic21:20
TheJuliav1k0d3n: that standalone doc... you likely could just start around line 521:23
eanderssonIs there an inbuilt function in the IPA for executing and capturing the error code? e.g. if I want a clean step to fail if the bash script returns a non zero code?21:41
eanderssonCurrently using ironic_python_agent.utils.execute but don't think it allows to fail on a exit code21:42
*** Goneri has quit IRC21:48
*** elemaill has joined #openstack-ironic21:50
*** uzumaki has quit IRC21:51
*** uzumaki has joined #openstack-ironic21:57
TheJuliaI'd catch the exception, tbh21:57
*** alexmcleod has quit IRC22:12
*** lbotell37 has quit IRC22:15
*** uzumaki has quit IRC22:16
*** lbotell73 has joined #openstack-ironic22:18
*** lbotell73 has quit IRC22:22
*** scott__c has quit IRC22:22
*** scott_c has joined #openstack-ironic22:23
*** scott_c has quit IRC22:24
*** rloo has quit IRC22:34
*** elemaill has quit IRC22:34
*** uzumaki has joined #openstack-ironic22:36
*** lbotell has quit IRC22:53
*** rcernin has joined #openstack-ironic22:56
*** elemaill has joined #openstack-ironic23:02
*** elemaill has quit IRC23:33

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!