Thursday, 2019-03-14

openstackgerritTakashi NATSUME proposed openstack/ironic master: Fix misuse of assertTrue  https://review.openstack.org/64320800:37
*** andrein has quit IRC00:38
*** gyee has quit IRC00:39
openstackgerritTakashi NATSUME proposed openstack/ironic master: Fix misuse of assertTrue  https://review.openstack.org/64320800:39
openstackgerritLars Kellogg-Stedman proposed openstack/ironic master: prevent serial console lockout  https://review.openstack.org/64273401:39
*** whoami-rajat has joined #openstack-ironic02:02
*** hoonetorg has quit IRC02:59
openstackgerritQianBiao Ng proposed openstack/ironic master: Add Huawei iBMC driver support  https://review.openstack.org/63928803:06
*** hoonetorg has joined #openstack-ironic03:12
*** MattMan_ has quit IRC03:24
*** MattMan_ has joined #openstack-ironic03:25
openstackgerritNguyen Van Trung proposed openstack/ironic master: Support firmware upgrade for irmc driver  https://review.openstack.org/59999204:04
*** zhangfei has joined #openstack-ironic04:20
openstackgerritMerged openstack/sushy stable/stein: Fix wrong default JsonDataReader() argument  https://review.openstack.org/64283705:35
openstackgerritRachit Kapadia proposed openstack/ironic master: Set boot_mode in node properties during OOB Introspection  https://review.openstack.org/63969805:43
openstackgerritKaifeng Wang proposed openstack/ironic master: TEST lower-constraint job  https://review.openstack.org/64323106:01
*** jtomasek has joined #openstack-ironic06:15
*** w14161_1_ has joined #openstack-ironic07:02
*** w14161_1 has quit IRC07:05
openstackgerritNguyen Van Trung proposed openstack/ironic master: Support firmware upgrade for irmc driver  https://review.openstack.org/59999207:07
openstackgerritVarsha Verma proposed openstack/ironic master: Deprecates `hash_distribution_replicas` config option  https://review.openstack.org/64300207:14
*** Qianbiao has joined #openstack-ironic07:17
QianbiaoHello, everyone.07:17
*** mkrai has joined #openstack-ironic07:22
*** verma-varsha has joined #openstack-ironic07:22
QianbiaoHello, may someone help to identify why failed to active node07:28
Qianbiaologs: http://paste.openstack.org/show/747767/07:28
mkraiHi, is the cleaning API exposed via Nova?07:28
mkraiI mean the "openstack baremetal node clean" API?07:29
QianbiaoI did not use clean API07:29
Qianbiaoopenstack baremetal maintenance set $NODE && openstack baremetal delete $NODE07:30
Qianbiaothen recreate a node.07:30
mkraiQianbiao: From your log, it seems node is busy with some task07:30
mkraiTry the steps after the current task is completed07:30
QianbiaoWhat kind of task? where i can get the task info.07:31
openstackgerritKaifeng Wang proposed openstack/ironic-inspector master: Follow up to api-ref revision  https://review.openstack.org/64323807:31
Qianbiaobecause it's a fresh node.07:31
Qianbiaoi delete it and then recreate07:31
mkraiI would suggest to wait for the completion of one API before running the second one07:32
QianbiaoHmmm.07:32
Qianbiaothe cmd i run has an option wait07:32
mkraiYour node is deleted?07:32
Qianbiaoopenstack baremetal node manage $NODE && openstack baremetal node provide $Node07:33
Qianbiao  --wait [<time-out>]  Wait for a node to reach the desired state, manageable.07:33
Qianbiao                       Optionally takes a timeout value (in seconds). The07:33
Qianbiao                       default value is 0, meaning it will wait indefinitely.07:33
Qianbiaoby default it will wait indefinitely.07:33
QianbiaoYes, I delete the node first, and then create it again.07:34
mkraiso you faced this error at which API?07:34
QianbiaoI thought it's node provide API07:34
mkraiWhat is the state of your node now?07:36
Qianbiaomanageable07:36
mkraican you try to run the provide API now?07:36
Qianbiaosure07:37
Qianbiaoweird, it's ok now.07:37
Qianbiaoavailable07:37
Qianbiaoit turns to available07:37
mkraiYes so the issue was you ran the two apis one after another before the manageable API could complete07:38
Qianbiao<mkrai> check the help info of "openstack baremetal node manage"07:39
Qianbiaoit should wait indefinitely.07:39
QianbiaoI use the cmd before, it never failed.07:39
mkraiIt waits indefinitely until it completes and during that time the node is locked by Ironic07:39
Qianbiaoyes, than it should be ok, right?07:40
mkraiYes it's ok07:40
Qianbiaoit will wait manage finished, and then run provide.07:40
Qianbiaoso, what should i do to fix this. because i need to use them in ansible playbook. i donot know how long it will take.07:41
QianbiaoIf the API works as it desc, it should wait and no error occurs.07:42
arne_wiebalckgood morning, ironic!07:42
mkraiarne_wiebalck: GM07:43
arne_wiebalckmkrai: o/07:43
*** e0ne has joined #openstack-ironic07:45
*** yaawang has quit IRC07:45
*** yaawang has joined #openstack-ironic07:47
Qianbiaomkrai should i upgrade openstack client07:47
Qianbiaoopenstack baremetal --version07:47
Qianbiaoopenstack 3.16.107:47
QianbiaoIt's weird, the script has been used for half a year, and this is the first time it happens.07:47
*** gkadam_ has joined #openstack-ironic07:50
*** pcaruana has joined #openstack-ironic08:07
*** pcaruana has quit IRC08:11
*** tssurya has joined #openstack-ironic08:12
*** pcaruana has joined #openstack-ironic08:23
openstackgerritMark Goddard proposed openstack/ironic master: Fix idrac Job.state renamed to Job.status  https://review.openstack.org/64314208:24
*** trungnv has quit IRC08:25
*** hoonetorg has quit IRC08:32
openstackgerritKaifeng Wang proposed openstack/ironic master: TEST lower-constraints job  https://review.openstack.org/64323108:34
*** hjensas has quit IRC08:35
*** hoonetorg has joined #openstack-ironic08:45
iurygregorygood morning everyone o/08:52
*** sburette has joined #openstack-ironic08:53
*** dtantsur|afk is now known as dtantsur08:59
*** dougsz has joined #openstack-ironic08:59
dtantsurmorning ironic08:59
iurygregorymorning dtantsur08:59
openstackgerritKaifeng Wang proposed openstack/ironic master: TEST lower-constraints job  https://review.openstack.org/64323109:00
mkraiHi, is the cleaning API exposed via Nova?09:02
mkraiI mean the "openstack baremetal node clean" API?09:02
dtantsurmkrai: hi, no.09:05
dtantsurfolks, is the lower-constraints job broken? is anybody already looking into it?09:05
* dtantsur asks infra09:08
openstackgerritDmitry Tantsur proposed openstack/ironic master: Add libpq-dev to bindep  https://review.openstack.org/64325109:12
dtantsurmaybe this will help ^^09:13
openstackgerritKaifeng Wang proposed openstack/ironic master: TEST lower-constraints job  https://review.openstack.org/64323109:26
*** andrein has joined #openstack-ironic09:27
*** mkrai has quit IRC09:30
openstackgerritKaifeng Wang proposed openstack/ironic master: Fix lower-constraints job  https://review.openstack.org/64323109:33
dtantsurrpittau|afk: morning! do you think that pinning the postgres job to xenial can unbreak us for now?09:41
iurygregoryif CI seems broken is probably because is on Bionic (for legacy jobs) =)09:42
*** e0ne has quit IRC09:43
dtantsuriurygregory: well, the lower-constraints job is fixed with the patch above. the postgres job is non-voting, but it fails because of bionic, and we need it green before the final release09:44
dtantsurwhich may mean sticking with xenial for stable/stein09:44
iurygregorydtantsur, gotcha09:44
*** e0ne has joined #openstack-ironic09:46
iurygregorywow =O09:46
iurygregorythats sad =(09:46
*** pcaruana has quit IRC09:48
*** hjensas has joined #openstack-ironic09:57
openstackgerritDmitry Tantsur proposed openstack/ironic-tempest-plugin master: discovery: optional support for testing changing driver  https://review.openstack.org/64326409:58
openstackgerritDmitry Tantsur proposed openstack/ironic-inspector master: Pass reset_interfaces when updating a driver from the rules  https://review.openstack.org/64134010:03
rpittau|afkdtantsur: we can give it a try, I can convert the job in the test I have10:05
rpittau|afkComing back from Paris, had to do an errand this morning and trains were late10:06
rpittau|afkInstead of 1:30 took me more than 2 hours :/10:06
dtantsurtrains..10:06
dtantsurokay, let's try reverting to xenial, then propose a patch with moving it to bionic + fixes10:07
rpittau|afkdtantsur: ack10:07
*** rpittau|afk is now known as rpittau10:13
rpittauI made it back \o/10:13
rpittaugood morning ironic! o/10:13
rpittaudtantsur, I think we're good on the lower constraints fix patch, going to +2 that asap10:18
dtantsuryep, it's good, feel free to approve it10:21
openstackgerritRiccardo Pittau proposed openstack/ironic master: [WIP] changes to fix postgres job  https://review.openstack.org/64243010:25
mgoddardI have to say, recent experience of using the idrac drivers rather than ipmitool has not been too pleasant10:26
mgoddardthings keep getting wedged10:27
iurygregorymorning rpittau o/10:32
rpittauhi iurygregory :)10:32
*** w14161_1_ has quit IRC10:32
*** w14161_1 has joined #openstack-ironic10:39
hjensasgood morning.11:00
mgoddardso close to deploy time RAID. After 30 minutes of watching various reboots, and even applying the RAID config, the idrac comes back with 'Configuration job already created, cannot create another config job on specified target until existing job is completed or is cancelled'11:00
hjensasrpittau: the pxelinux.0 load issue still relevant?11:00
rpittauhey hjensas11:00
rpittauhjensas, it's relevant for bionic11:00
rpittauthe update to the dist filter is necessary or bionic won't find pxelinux.011:01
*** yolanda has quit IRC11:02
*** yolanda has joined #openstack-ironic11:03
hjensasrpittau: this one right? https://review.openstack.org/64250711:07
patchbotpatch 642507 - ironic - Update dist filter for devstack ubuntu - 1 patch set11:07
rpittauhjensas, correct11:07
openstackgerritRiccardo Pittau proposed openstack/ironic master: [WIP] pinning postgres job to xenial  https://review.openstack.org/64243011:12
rpittauoh the irony, it rains like hell but there's no water in my building11:15
*** zhangfei has quit IRC11:18
dtantsurheh11:22
*** derekh has joined #openstack-ironic11:29
*** pcaruana has joined #openstack-ironic11:32
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add libvirt-based virtual media emulation  https://review.openstack.org/64045211:35
etingofraining cats and dogs on rpittau o/11:35
*** sburette has quit IRC11:37
rpittauhi etingof :D11:37
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Render System->Managers linkage  https://review.openstack.org/64309411:38
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add docs on libvirt virtual media emulation  https://review.openstack.org/64302711:38
*** bfournie has quit IRC11:42
rpittaudtantsur, it seems like the issue with postgres is indeed related to bionic, pinning to xenial makes it pass https://review.openstack.org/64243012:06
patchbotpatch 642430 - ironic - [WIP] pinning postgres job to xenial - 7 patch sets12:06
dtantsurrpittau: cool, let's pin it for now and prepare a proper patch with migration12:06
rpittaudtantsur, ack12:07
openstackgerritRiccardo Pittau proposed openstack/ironic master: Pinning postgres job to xenial  https://review.openstack.org/64243012:13
*** verma-varsha has quit IRC12:17
*** bfournie has joined #openstack-ironic12:18
*** trown|outtypewww is now known as trown12:19
*** e0ne has quit IRC12:31
Qianbiao<rpittau> what is ack stand for?12:38
Qianbiaothe same as TCP ack ?12:38
iurygregoryQianbiao, ack = acknowledged12:44
rpittauQianbiao: kind of :)12:44
Qianbiaoi see.12:45
iurygregoryQianbiao, did you have lucky with your env to see the tests with standalone queens? =)12:45
Qianbiaokind like "hi, received"12:45
QianbiaoIt works in my ENV.12:46
iurygregorythe tempests tests run fine?12:46
QianbiaoI didnot use tempest :)12:46
iurygregoryyou have a conf that i could check what is different in the job?12:46
QianbiaoI code a ansible playbook myself.12:46
openstackgerritDao Cong Tien proposed openstack/ironic master: Update doc for UEFI first  https://review.openstack.org/63122512:47
QianbiaoI checked the log, but no lucky.12:47
iurygregorywell the devstack works the problem is run the tempest tests that get many different errors XD12:48
*** Qianbiao has quit IRC12:49
openstackgerritMerged openstack/ironic master: Fix lower-constraints job  https://review.openstack.org/64323112:50
*** Qianbiao has joined #openstack-ironic12:50
*** hwoarang has quit IRC12:56
*** hwoarang has joined #openstack-ironic12:57
dtantsurokay, the CI should be back to norm. recheck the hell out of it!12:59
iurygregoryrecheck all the things!13:01
*** andrein has quit IRC13:11
*** sburette has joined #openstack-ironic13:21
*** e0ne has joined #openstack-ironic13:32
*** baha has joined #openstack-ironic13:34
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add HTTP image download for virtual media emulation  https://review.openstack.org/64333013:34
*** mjturek has joined #openstack-ironic13:37
*** mjturek has quit IRC13:38
*** efried has joined #openstack-ironic13:38
*** Wryhder has joined #openstack-ironic13:40
*** Lucas_Gray has joined #openstack-ironic13:44
*** mjturek has joined #openstack-ironic13:46
*** hwoarang has quit IRC13:53
*** hwoarang has joined #openstack-ironic13:55
openstackgerritNikolay Fedotov proposed openstack/ironic master: Kg key for IPMIv2 authentication  https://review.openstack.org/64165814:06
*** hjensas has quit IRC14:06
*** Wryhder has quit IRC14:12
*** rh-jelabarre has joined #openstack-ironic14:13
*** andrein has joined #openstack-ironic14:14
*** rcernin has quit IRC14:17
*** hjensas has joined #openstack-ironic14:20
Qianbiaodtantsur TheJulia huawei ironic ci is ready now14:20
Qianbiaocheck the newest comment https://review.openstack.org/#/c/639288/14:20
patchbotpatch 639288 - ironic - Add Huawei iBMC driver support - 16 patch sets14:20
dtantsurthanks, will check when I finish my meetings14:20
Qianbiaook thanks14:20
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add HTTP image download for virtual media emulation  https://review.openstack.org/64333014:24
openstackgerritRiccardo Pittau proposed openstack/ironic-inspector master: Removing two deprecated values from json responses  https://review.openstack.org/64335014:24
dtantsurQianbiao: I'm getting "The server at 112.93.129.99 is taking too long to respond" :(14:24
dtantsurcan there be some kind of firewall or IP filtering?14:24
dtantsurQianbiao: honestly, I cannot even ping that IP14:28
Qianbiaowhy14:34
Qianbiaohttp://112.93.129.99/builds/Ironic-3rd-party-CI/146/14:34
Qianbiaolog is here14:34
*** hwoarang has quit IRC14:35
*** hwoarang has joined #openstack-ironic14:36
rpittaudtantsur, I can access the server14:36
dtantsurinteresting14:38
dtantsurmay be some global routes from my place..14:38
iurygregoryi cant access too =(14:39
*** dougsz has quit IRC14:45
*** hwoarang has quit IRC14:46
*** yaawang has quit IRC14:47
Qianbiao:( it's a normal server in china zone.14:47
*** yaawang has joined #openstack-ironic14:48
*** hwoarang has joined #openstack-ironic14:52
iurygregoryi will try to access from home since I'm at the office and I'll see if it works14:56
*** yolanda has quit IRC14:57
Qianbiaook thanks.14:57
Qianbiaoi will re-trigger CI14:57
Qianbiaoagain. because the log contains some sensitive info... i need to hide them14:57
*** baha has quit IRC14:58
dtantsurrpittau: do you think we can split out the jobs that do work from https://review.openstack.org/#/c/636169/ ?14:58
patchbotpatch 636169 - ironic-tempest-plugin - Run all defined jobs defined in check and gate - 15 patch sets14:58
iurygregorydtantsur, i was thinking about this14:59
iurygregoryif you want i can push the patch without wholedisk and ironic-standalone queens15:00
rpittaudtantsur, yeah15:01
dtantsuriurygregory: oh, it's your patch, sorry :)15:01
dtantsuryeah, let's do small steps.15:01
iurygregorysure =)15:02
iurygregoryafter meeting im going to update15:02
rpittauI wonder if we should split by branch also ?15:02
iurygregoryrpittau, for the ones that arent working you mean?15:04
*** Qianbiao has quit IRC15:04
rpittauiurygregory, yes :)15:09
iurygregoryrpittau, sure makes sense =) to not be blocking if we get another working15:10
*** hjensas has quit IRC15:12
*** yajushiSri has joined #openstack-ironic15:18
*** dougsz has joined #openstack-ironic15:31
*** baha has joined #openstack-ironic15:34
openstackgerritNikolay Fedotov proposed openstack/ironic master: Kg key for IPMIv2 authentication  https://review.openstack.org/64165815:35
*** rachit7_ has joined #openstack-ironic15:38
openstackgerritMerged openstack/ironic-inspector stable/rocky: Use getaddrinfo instead of gethostbyname while resolving BMC address  https://review.openstack.org/64256015:39
*** hamzy has quit IRC15:43
openstackgerritMerged openstack/ironic-inspector stable/rocky: Fix lookup when ipmi_address is a hostname  https://review.openstack.org/64276015:44
openstackgerritMerged openstack/ironic master: Fix misuse of assertTrue  https://review.openstack.org/64320815:44
*** yolanda has joined #openstack-ironic15:45
openstackgerritMerged openstack/ironic master: Adding ansible python interpreter as driver_info  https://review.openstack.org/64240216:00
*** Lucas_Gray has quit IRC16:04
rpiosoGood morning, ironicers.16:16
rpiosomgoddard: Sorry to hear about those challenges.16:16
mgoddardrpioso: not your fault16:16
rpiosomgoddard: Have you been able to get beyond it?16:16
mgoddardI ended up modifying the idrac driver to use ipmitool for power & management16:16
rpiosomgoddard: Ugh!16:17
mgoddardrpioso: we always use ipmitool drivers16:17
mgoddardrpioso: can I recommend that you add them to the supported power & management interfaces?16:18
* rpioso hasn't tried idrac RAID on master; however, our downstream solution based on queens, the last LTS version, uses it, and it works.16:18
mgoddardI kept getting various errors from the idrac16:20
rpiosomgoddard: We've batted that idea around recently, in the context of the story I'm working on to add Redfish interfaces to it. I'll discuss it some more with my team.16:20
mgoddardthis one in particular:16:20
mgoddardConfiguration job already created, cannot create another config job on specified target until existing job16:20
mgoddard is completed or is cancelled16:20
mgoddardredfish might help. Is it supported on idrac 8?16:20
rpiosomgoddard: Was the job queue clear before you started?16:20
mgoddardrpioso: job queue was clear16:21
rpiosomgoddard: 13G, right?16:21
*** rachit7_ has quit IRC16:21
mgoddardR63016:21
rpiosomgoddard: That's a yes :-)16:21
mgoddardidrac firmware is just over a year old16:22
rpiosomgoddard: It is, however, I believe the Redfish support on 14G is ahead of 13G.16:22
rpiosomgoddard: Redfish power and management should be fine on either.16:22
mgoddardrpioso: can it set the power state and boot mode?16:22
mgoddardok16:23
rpiosomgoddard: Did the modification fix the issue?16:26
mgoddardrpioso: the RAID one?16:26
mgoddardYes, it's looking better now16:26
mgoddardManaged to get a RAID deploy step working, I think. Need to validate the disk is as expected16:27
rpiosomgoddard: This one -- "Configuration job already created, cannot create another config job on specified target until existing job is completed or is cancelled"16:27
mgoddardrpioso: oh, no - switched to ipmitool at that point - couldn't get past that error16:28
* rpioso is lost16:28
rpiosomgoddard: I thought you modified the idrac h/w type.16:28
mgoddardrpioso: I did. The error was when setting the boot device, ipmitool management interface avoided it16:29
rpiosomgoddard: So it was a BIOS configuration job that was already created, right?16:29
mgoddardrpioso: I couldn't see any job in the queue. Hadn't created any BIOS jobs. I think it was a job to set the boot device16:30
rpiosomgoddard: Which boot mode is the server configured to use?16:30
mgoddardpreece: seems it couldn't set the boot mode or power state: http://paste.openstack.org/show/747797/16:31
mgoddardsorry preece, not you.16:32
mgoddardrpioso: ^16:32
rpiosomgoddard: You may have run into a known race condition. Let me dig up the artifacts.16:35
rpiosomgoddard: Which boot mode is the server configured to use?16:35
mgoddardrpioso: local16:35
*** jtomasek has quit IRC16:35
rpiosomgoddard legacy BIOS or UEFI?16:36
mgoddardrpioso: oops, BIOS16:36
openstackgerritMerged openstack/ironic master: Deploy templates: conductor and API nits  https://review.openstack.org/64044616:36
rpiosomgodddard: We're working on a fix to it -- https://review.openstack.org/#/c/633833/16:37
patchbotpatch 633833 - ironic - Fix potential race condition on node power on and ... - 4 patch sets16:37
rpiosomgoddard: Btw, boot device configuration is a BIOS thing.16:38
*** e0ne has quit IRC16:38
rpiosomgoddard: Internally, I'm reviewing an alternative approach mbeierl and I are batting around.16:40
rpiosombeierl: ^^^^^16:40
rpiosomgoddard: Are you using master?16:40
mgoddardrpioso: yes16:41
iurygregorydtantsur, rpittau just fyi im going to remove the following jobs from the patch just to unblock http://paste.openstack.org/show/747796/ my decision was based on the last patch i saw the job running and  the results are differents between the patches (except for wholedisk-pike that was broken in both) make sense?16:43
rpiosomgoddard: For the time being, if you're game for trying idrac power and management again, you could add a short sleep immediately after https://github.com/openstack/ironic/blob/9625b8aa71896bbf626971f8bbabe48d0d34b39e/ironic/drivers/modules/drac/management.py#L24916:44
rpiosomgoddard: Five seconds should do the trick.16:44
rpittauiurygregory, looks good, also I don't know how much we want to invest in pike tests :/16:44
mgoddardrpioso: thanks, I'll stick with ipmitool so I can focus on other things16:44
iurygregoryrpittau, yeah XD16:45
*** tssurya has quit IRC16:45
*** emine__ has quit IRC16:46
dtantsuriurygregory: let's make them non-voting rather than removing? otherwise yes16:46
rpiosomgoddard: Understood. Once we settle on the fix, I'd be glad to let you know.16:46
mgoddardrpioso: thanks :)16:46
iurygregorydtantsur, ack o/16:46
iurygregoryalso I will add comments about the jobs that we want to get back voting16:47
rpiosomgoddard: It's a timing issue, so it's difficult to reliably repro. It depends on the generation of the iDRAC and its firmware version.16:47
rpiosomgoddard: Since you've encountered it, it would be much appreciated if you could try to reproduce it when time permits and try the one line hack. And then subsequently pick up the real fix to confirm it resolves the issue. We've had trouble reproducing it in house.16:49
*** gkadam_ has quit IRC16:50
mbeierlthanks, rpiosoand mgoddard.  I just came back from lunch16:51
iurygregorydtantsur, fow the wholedisk in the gate i should let non-voting or remove and add a note?16:52
dtantsuriurygregory: you mean, pike multinode? I guess nobody is going to recover it..16:53
mgoddardrpioso: I'll give it a try at some point16:53
iurygregorydtantsur, nope  ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode16:53
rpiosomgoddard: Awesome :-)16:53
dtantsuriurygregory: well, master must be recovered16:54
iurygregoryi know but since its broken and the job is on gate i should keep on gat with non-voting?16:54
dtantsurah16:56
dtantsuriurygregory: non-voting jobs must not be in the "gate" pipeline, only in check16:56
iurygregoryack so i will keep a note that we want to re add to the gate16:57
rpittauiurygregory, weird, that job should be working just fine16:59
*** arne_wiebalck has quit IRC16:59
*** arne_wiebalck has joined #openstack-ironic17:00
* rpioso goes to lunch17:00
*** rpioso is now known as rpioso|afk17:01
iurygregoryrpittau, yeah17:01
*** sziviani has joined #openstack-ironic17:04
rpittaugood night o/17:05
*** rpittau is now known as rpittau|afk17:05
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Run all defined jobs defined in check and gate  https://review.openstack.org/63616917:06
*** trown is now known as trown|lunch17:06
*** gyee has joined #openstack-ironic17:07
*** hamzy has joined #openstack-ironic17:09
*** anupn has joined #openstack-ironic17:11
anupnHi Folks, any clue when Bifrost provisions the user image does it not work on the drive /dev/sda (by default) if it already had partitions on it?17:13
anupnMy baremetal got provisioned correctly, and I see the state as active, but I don't see the OS in the BIOS although I see two partitions are created on /dev/sda17:14
anupnany idea if it might be OS problem or boot loader issue17:15
*** sburette has quit IRC17:15
TheJuliadoes it boot?17:15
TheJuliathe bios seeing it may be it knowing how to read/grok the disk which milage may vary based on vendor and vendor focus17:16
*** dtantsur is now known as dtantsur|afk17:18
dtantsur|afk\o17:18
anupnThe IP address that is handed out is active. So I have 4 different drives, and each one has OS in it. This OS is getting provisioned in sda. So if I boot into another OS on drive sdb I see the IP sent by DHCP being assigned to the NIC17:26
anupnTheJulia17:26
anupnIt says it writes image correctly to sda17:26
anupnBut I can't boot into sda as the OS option is not coming in BIOS17:27
TheJuliaanupn: Ahh, so a couple different possibilities exist. Largely the first one is that you need to actually clean the drives17:30
TheJuliabetween deployments that is, we're not really going to support troubleshooting issues when cleaning is not performed because the bios nvram might have an update pointing to that other disk and there is no way for the bios to know if it is not valid if you've been writing oses across various drives17:31
anupnTheJulia: Here is my node show http://paste.openstack.org/show/747806/. Ah so I should erase the sda drive or whatever I chose before this deployment?17:32
anupnTheJulia: Also does it matter if there are multiple drives or not?17:33
TheJuliaanupn: you should have cleaning enabled...17:33
anupnI am checking if cleaning was enabled by default in Bifrost17:34
anupnor Ironic17:34
TheJulianvram, if using uefi boot, is going to choose based upon the records it has and depending on bios behavior and ultimately what your writing, it may not even be compatible. These are things that you'll need to investigate and resolve17:34
openstackgerritJulia Kreger proposed openstack/ironic master: fast tracked deployment support  https://review.openstack.org/63599617:34
anupnTheJulia: Mine is BIOS boot. And I see cleaning is disabled by default https://github.com/openstack/bifrost/blob/734f63cdbb5458aac05a68f5fea3a3146aafec69/playbooks/roles/bifrost-ironic-install/defaults/main.yml#L417:37
*** derekh has quit IRC17:38
openstackgerritJulia Kreger proposed openstack/ironic-tempest-plugin master: Add fast-track testing capability  https://review.openstack.org/63703917:39
openstackgerritJulia Kreger proposed openstack/ironic master: Add fast-track testing  https://review.openstack.org/64110417:39
anupnTheJulia: I may be wrongly understanding but it didn't complained about unable to clean earlier partitions. I see earlier there was one partition and now deployment created two partitions17:42
*** priteau has joined #openstack-ironic17:45
*** e0ne has joined #openstack-ironic17:46
TheJuliaanupn: so that sounds like it could be the configuration drive17:48
anupnTheJulia: Oh so you mean one partition is configuration drive, yes that could be17:50
openstackgerritBob Fournier proposed openstack/ironic-inspector stable/queens: Use getaddrinfo instead of gethostbyname while resolving BMC address  https://review.openstack.org/64340917:51
anupnas I saw the contents in config drive getting attached by IPA. So what should I be trying now? A complete empty drive + only single empty drive in the system with no other drives?17:52
anupnTheJulia: Also will cleaning just clean the drive where image is getting written or all the drives that I have in my system? I doubt it will erase other drives where image is not getting written17:54
TheJuliaanupn: I really don't quite understand17:55
TheJuliaanupn: cleaning cleans all drives17:55
anupnTheJulia: Ah okie. Then I would probably not use cleaning for now as I want the contents on the other drives. What I meant is should I then wipe the sda drive on my own and let that drive (sda) be alone in the system without having other drives installed?18:01
*** trown|lunch is now known as trown18:01
openstackgerritBob Fournier proposed openstack/ironic-inspector stable/queens: Use getaddrinfo instead of gethostbyname while resolving BMC address  https://review.openstack.org/64340918:23
openstackgerritBob Fournier proposed openstack/ironic-inspector stable/queens: Fix lookup when ipmi_address is a hostname  https://review.openstack.org/64341918:23
*** e0ne has quit IRC18:33
TheJuliaanupn: worth a try18:36
*** patchbot has quit IRC18:37
*** patchbot has joined #openstack-ironic18:39
*** e0ne has joined #openstack-ironic18:40
anupnTheJulia: yup, trying that now18:47
*** gmann is now known as gmann_afk18:51
*** priteau has quit IRC18:56
openstackgerritJulia Kreger proposed openstack/ironic master: Make metrics usable  https://review.openstack.org/64343619:10
TheJuliadtantsur|afk: iurygregory ^^^19:10
*** priteau has joined #openstack-ironic19:16
*** priteau has quit IRC19:21
*** anupn has quit IRC19:25
*** rpioso|afk is now known as rpioso19:25
*** dougsz has quit IRC19:27
*** absubram has joined #openstack-ironic19:35
openstackgerritJulia Kreger proposed openstack/ironic master: Kg key for IPMIv2 authentication  https://review.openstack.org/64165819:41
*** yolanda has quit IRC19:46
efriedo/20:02
* efried is excited20:02
efriedCheck this out, y'all20:02
efriedhttps://review.openstack.org/#/c/642899/20:02
patchbotpatch 642899 - nova - WIP/PoC: Bypass ironicclient for node.get - 7 patch sets20:02
efriedI swapped out 'node.get' with a direct API call through ksa and the job passes.20:03
efried(after 7 patch sets worth of debugging :)20:03
efriedTheJulia, dtantsur|afk: ^20:03
dustincnice!20:09
*** yajushiSri has quit IRC20:15
*** Lucas_Gray has joined #openstack-ironic20:17
mbeierlanyone here worked with L3 routing, dhcprelay and Ironic?20:19
mbeierlthe dnsmasq process on the ironic node is telling me: DHCPDISCOVER(br-ctlplane) 24:6e:96:22:3c:a4 ignored and I don't know why20:21
larsksmbeierl: there are usually a couple of dnsmasq processes. Are you sure you're looking at the right one?20:22
mbeierlnobody   16685  0.0  0.0  53888  1168 ?        S    11:17   0:00 dnsmasq --no-hosts --no-resolv --except-interface=lo --pid-file=/var/lib/neutron/dhcp/daae77ea-f861-491d-ba4e-be44a386a9b3/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/daae77ea-f861-491d-ba4e-be44a386a9b3/host --addn-hosts=/var/lib/neutron/dhcp/daae77ea-f861-491d-ba4e-be44a386a9b3/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/daae77ea-f861-491d-ba4e-be44a386a9b3/20:23
mbeierlopts --dhcp-leasefile=/var/lib/neutron/dhcp/daae77ea-f861-491d-ba4e-be44a386a9b3/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=tapf5dffafc-f1 --dhcp-range=set:tag0,192.168.120.0,static,255.255.255.0,86400s --dhcp-range=set:tag1,192.168.121.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=512 --conf-file=/etc/dnsmasq-ironic.conf --domain=localdomain20:23
larsksYeah, that's the right one.20:23
mbeierllarsks: thanks :)20:23
mbeierllarsks: oh - hang on - the log message is from the inspector PID20:24
mbeierlso I have no log message20:24
larsksYeah, that's what I was wondering about :)20:24
larsksYou expect to see those messages from the inspector dnsmasq instance.20:24
mbeierlok20:25
mbeierlany way to turn up logging for ironic's dnsmasq?20:25
mbeierlput it into the config file?20:25
*** priteau has joined #openstack-ironic20:25
larsksI'm not sure.  You can inspect the files in /var/lib/neutron/dhcp/daae77ea-f861-491d-ba4e-be44a386a9b3/ to make sure the config looks sane.20:26
*** dsneddon has quit IRC20:27
larsksIs CI healthy?  I see a failure in openstack-tox-lower-constraints because some dependencies were missing (http://logs.openstack.org/34/642734/5/check/openstack-tox-lower-constraints/46a5e93/job-output.txt.gz#_2019-03-14_01_42_41_059548).20:29
larsksI'm going to recheck and cross my fingers.20:29
efriedIronic API not yet set up to handle X-OpenStack-API-Version ??20:36
efriednm, must have been spelling it wrong.20:37
mbeierllarsks: the configs in neutron look good.  I can see the entry for my MAC addr in host20:38
mbeierland the set and ip addr for it all match up in the other neutron files20:38
larsksmbeierl: and the problem is your host is just not getting a response?20:39
mbeierlcorrect.  I have wireshark running on the ctlplane network and I see the request coming in with the correct MAC addr, via the relay20:39
*** anupn has joined #openstack-ironic20:40
mbeierlif I'm looking at br-ctlplane (which is the IP address where Ironic and dnsmasq are running), what does this line in the dnsmasq ps entry mean? --interface=tapf5dffafc-f120:40
mbeierloh, and I should mention, PXE booting from the same L2 subnet works.  I get DHCP responses no problem20:41
openstackgerritMerged openstack/ironic master: fast tracked deployment support  https://review.openstack.org/63599620:43
*** priteau has quit IRC20:43
*** hamzy has quit IRC20:45
larsksmbeierl: That seems odd.  I don't have any clever ideas, sorry.20:46
mbeierllarsks: thanks.  Just to clarify - the Ironic dnsmasq listens to the TAP interface only?20:47
larsksmbeierl: that should be what the --interface=tapf5dffafc-f1 option means.20:49
mbeierlinteresting.20:49
mbeierllarsks: the tap5f... in on bridge br-int inside of OVS20:50
mbeierlthere is another br-ctlplane, but it has no tap interface20:50
larsksThat's probably okay. That matches what my local setup looks like, at least.20:52
mbeierlok20:52
dustincefried: I am new to Openstack and was wondering what the benefit of using ksa over ironicclient is, are there any benefits/helpers in ironicclient that would be lost or have to be reimplemented? Is there a performance benefit to calling the API directly with ksa?20:52
*** mjturek has quit IRC20:53
efrieddustinc: okay, there's a lot of questions in there...20:53
dustincsorry :)20:54
efrieddustinc: Generally speaking, OpenStack is trying to move away from having these *client layers in between service A and service B it wants to talk to.20:54
efried(no, no worries at all, just it's going to take some explaining)20:54
mbeierllarsks: ovs-tcpdump of the interface does not show the packets coming in :(20:55
efriedThe idea is that the API ought to be pretty accessible (usability-wise) and not need a client layer to massage inputs and outputs.20:55
efriedNow what we would *really* like is to use openstacksdk - but in the case of nova-talks-to-ironic, there's some framework needed that isn't in place yet.20:55
efriedSo using direct ksa is the next best thing.20:55
larsksWhat is "ksa"?20:56
efriedNote that ironicclient uses ksa internally anyway. So the only "benefit" is massaging inputs and outputs.20:56
efriedkeystoneauth...20:56
larsksAh, thanks.20:56
efriedhttps://github.com/openstack/keystoneauth20:56
dustincI see. So would the intention to eventually be to completely deprecate ironicclient, or would it still exist for some other consumer?20:56
efried...which is a shim layer that presents some convenient mechanisms for service A to talk to service B over a keystone-authenticated session.20:56
*** anupn has quit IRC20:57
efrieddustinc: Not sure, perhaps TheJulia wants to answer that; but certainly the intention is to remove use of ironicclient completely from nova.20:57
mbeierllarsks: one more question - is your environment set up for L3 routed Ironic?  (ie: for spine/leaf)?20:58
efriedactually, I think ironicclient is also the CLI...20:58
larsksmbeierl: nope, it is not.20:58
efriedBUT we're also trying to move all CLIs to osc with plugins, so...20:58
dustincI was wondering if it was but wasn't sure20:58
mbeierllarsks: ok. so I might have missed something else.  Thanks.  I'll go digging some more.  Gotta run for today!20:58
efriedWhat I don't know is whether ironicclient has any other consumers.20:58
dustincit seems it would make sense to have a single CLI that uses same interfaces/methods as components use internally so any massaging between is only implemented once20:59
efriedyou would think so20:59
efriedbut the client layers turn out to be more trouble than they're worth.20:59
dustinc(that is just a general though, I don't know enough about the project as a whole to know how it applies specifically)20:59
dustincthought*20:59
dustincI see20:59
*** rh-jelabarre has quit IRC21:00
efriedIf you dig into the ironicclient code, for example, you'll see it's like 50% overblown abstraction layers and 40% legacy code that we have to keep around for backward compatibility.21:00
efriedThe other 10% is translating input args into querystrings and calling .json() on the responses21:00
dustincthat sounds fun21:00
efriedright, so the point is that all of that is very little code, and should really be done in a consumer-specific way by the consumer.21:01
efriedso21:01
*** baha has quit IRC21:01
dustincthanks for the info, it makes a lot more sense now. I also commented on your code with a small detail/specific question21:01
efriedIn https://review.openstack.org/#/c/642899/ I have done that, created a direct-through-ksa method for one of the otherwise-ironicclient methods, and the CI job proves that it's working, as far as it goes.21:01
patchbotpatch 642899 - nova - WIP/PoC: Bypass ironicclient for node.get - 8 patch sets21:01
*** anupn has joined #openstack-ironic21:02
dustincI think I am going to go dig into ksa now, thanks efried21:05
efrieddustinc: Subsequent patches (which ima go work on now) should be smaller because the framework is already in place.21:05
efrieddustinc: Okay, good luck with that. #openstack-keystone is your home for questions on ksa.21:05
dustincas I try to ramp up on one project, my learning backlog is growing exponentially...21:08
efriedyes, that happens21:12
efrieddustinc: However, if you want an entry that lets you start slinging code fairly quickly without having to grok a huge amount of background, you could work on this thing...21:12
efriedYou would have both nova and ironic people chipping in to buy you a cookie21:13
* TheJulia tries to wake up21:14
efriedTheJulia: Here's what you missed: That patch to get nova talking through ksa instead of ironicclient for node.get - works!21:15
TheJuliambeierl: sounds like your trying to do a spine and leaf config?21:15
TheJulialarsks: ci was healthy at least two hours ago, haven't looked since21:15
efriedTheJulia: so I'm going to start working on the other APIs (unless I can talk dustinc into doing it :)21:15
TheJuliaefried: we don't handle the generic tag afaik, it has not been a priority for anyone21:16
efriedgeneric tag?21:16
*** trown is now known as trown|outtypewww21:16
dustincedfried: I can def. work on it, just going to be a bit slow to get started21:17
TheJuliadustinc: basically ironicclient has a lot of code that doesn't need to be carried and the overall idea that efried is heading along is to make things simpler by removing the library which should ultimately fix a lot of the headaches because the client automatically does json to object conversion magic and realistically I think everyone would be happier with just json and less hand-holding that the client library21:18
TheJuliaprovides. Also, an overall community desire exists to remove a lot of the heavy cross-app library dependencies for just rest APIs to reduce the overall footprint because client libraries start to pull in many dependencies21:18
TheJuliadustinc: since they are also geared for end user on command lines21:18
efriedphew, well said, that's what I meant :)21:18
efriedTheJulia: Does ironicclient have any consumers other than nova and ironic CLI?21:19
TheJuliaefried: beyond nova, tons of consumers and the osc plugin21:19
efriedokay, so we're not going to get rid of ironicclient after this :)21:19
dustincmakes sense...is the intention to continue to use them for cli, or is there a goal to replace it with something else for that as well?21:19
*** andrein has quit IRC21:20
dustinc(sorry that question is already out of date)21:20
TheJuliaThe other motivation was consumer level groking of the data, which turns out to be kind of important21:20
TheJuliaefried: the generic api microversion request21:20
TheJuliaoh, and dustinc... if you like beer... we buy that too21:21
efriedTheJulia: Okay, not sure how, but sending microversion=x.y through adapter.get seems to actually work. Maybe ksa has a special case built in. (I seem to maybe kinda remember something about that.)21:22
TheJuliaheh, yeah, irccloud is kind of laggy and I'm a bit distracted today21:22
TheJuliaefried: yeah, there is code in ksa to handle our silliness21:22
dustincwaiting to find out if I go to Denver, will be nice to meet everyone I haven't yet21:22
efriedso dustinc if you're interested in pursuing this, I can help start you off.21:23
dustincTheJulia: I was just running LimeChat locally when I saw irccloud in your host string and checked it out...sold..21:23
TheJuliadustinc: toatally worth the $5 a month21:24
dustincefried: definitely interested21:24
* TheJulia blames monty21:24
openstackgerritMerged openstack/ironic master: Fix idrac Job.state renamed to Job.status  https://review.openstack.org/64314221:24
openstackgerritMerged openstack/ironic master: Deploy Templates: API reference  https://review.openstack.org/64249821:24
openstackgerritMerged openstack/ironic master: Deploy Templates: documentation  https://review.openstack.org/64253121:24
dustincTheJulia: does the mobile client stay connected reliably enough to always be within the 2-hour free window?21:24
TheJuliaI don't know, I've paid for it from about 30 minutes after I started using it21:25
TheJuliaI was totally sold21:25
TheJulialargely because I get the notifications and all21:25
dustincI am still trying to figure out what is free and what is not since i am on the initial trial21:25
TheJuliaYeah, I've been using it for years21:26
efrieddustinc: The pattern is going to be pretty simple: in nova/virt/ironic/driver.py search for self.ironicclient.call. The first arg is going to be like 'foo.bar'. Write a method in nova/virt/ironic/client_wrapper.py _IronicClient called foo_bar and within it duplicate the logic of the respective method in python-ironicclient/ironicclient/v1/foo/bar.py21:26
dustinc(I think they warned me about this in 5th grade DARE class)21:26
*** bfournie has quit IRC21:26
efrieddustinc: Start off with another get-ish one to keep it simple21:26
efriedwhen you need a put/post/delete you'll have to add another primitive to _IronicWrapper.21:27
TheJuliaefried: I haven't looked recently at your patches, how much of the unit test headaches did you have to touch21:27
TheJuliasounds liek you kind of routed around the object craziness21:27
efrieddustinc: Do you know how to work changes in a series?21:27
dustincnot exactly..was just trying to find out21:27
efriedTheJulia: I haven't touched unit/functional test yet. I commented out all the CI except the job that does real things.21:27
efrieddustinc: Someone recently authored an excellent article in the contributor docs on this subject. Let me go dig it up...21:28
dustincI have tons of experience with Github on a single team with feature branch model but pretty mcuh nothing else21:28
efrieddustinc: https://docs.openstack.org/contributors/code-and-documentation/patch-series-tutorial.html21:29
dustincexcellent..thanks21:30
dustinchey look who that someone was...21:30
*** whoami-rajat has quit IRC21:31
efriedI guess I could take my handle out of those screenshots. Lazy copy/paste.21:32
hogepodgehey everybody, wanted to make you all aware of this http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003857.html22:20
efrieddustinc: Ima revert the zuul stuff in my patch so I can start working on the unit test. If you already pulled, you'll have to pull again...22:21
*** pcaruana has quit IRC22:21
dustincok thanks22:24
*** e0ne has quit IRC22:28
openstackgerritJulia Kreger proposed openstack/ironic master: Make metrics usable  https://review.openstack.org/64343622:29
TheJuliahogepodge: would it help to add to the meeting agenda?22:30
*** dteselkin has quit IRC22:33
*** dteselkin has joined #openstack-ironic22:35
*** absubram has quit IRC22:36
*** gmann_afk is now known as gmann22:40
rpiosoTheJulia: o/22:50
TheJuliao/22:50
rpiosoTheJulia: Do you have the info you need?22:50
*** rcernin has joined #openstack-ironic23:00
TheJuliarpioso: Thanks!23:02
*** anupn has quit IRC23:13
rpiosoTheJulia: yw23:13
*** gyee has quit IRC23:34
*** Lucas_Gray has quit IRC23:39
*** dsneddon has joined #openstack-ironic23:55
*** sthussey has quit IRC23:57
*** dsneddon has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!