Wednesday, 2018-06-13

TheJuliaor what exactly it does or does not do00:00
mgagneor maybe for check_versions to not crash if the version field is missing00:00
NobodyCamLOL nope just popped in at the exact wrong time :p00:00
mgagneok, I will see what I can report tomorrow on that00:00
TheJuliaNobodyCam: note taken... so we have a way to summon you now... muahahahahaha!00:00
NobodyCamlol00:00
TheJuliamgagne: okay :)00:00
*** trungnv has quit IRC00:01
TheJuliai'm going to get some water and crash. I need to be up at like 4 AM local to jump on a train to DC00:01
mgagnegood luck ;)00:01
*** trungnv has joined #openstack-ironic00:02
NobodyCamhave a good night all!00:02
TheJuliagoodnight NobodyCam00:02
TheJuliagoodnight world00:09
*** masber has joined #openstack-ironic00:10
*** MattMan has quit IRC00:12
*** MattMan has joined #openstack-ironic00:12
*** masuberu has quit IRC00:13
*** masuberu has joined #openstack-ironic00:26
openstackgerritAlessandro Nesta proposed openstack/networking-baremetal master: fix tox python3 overrides  https://review.openstack.org/57425600:26
*** masber has quit IRC00:29
*** zhangfei has joined #openstack-ironic00:31
*** arundhati has quit IRC00:32
*** gyee has quit IRC00:34
*** mkrai has joined #openstack-ironic00:35
*** retr0h has joined #openstack-ironic00:35
*** csmart has joined #openstack-ironic00:35
*** hshiina has joined #openstack-ironic00:49
*** cjloader has quit IRC00:58
*** tiendc has joined #openstack-ironic01:08
rloomgagne: fwiw, i just tried doing db migration from juno to ocata, using ocata release. it worked fine as long as you set some network-interface-related config to the desired value first. after ocata, we put in rolling upgrades. i think it was in pike. so it is possible you can go directly to pike, but after that (so i'm not sure if after ocata or after pike), you have to do it via each release.01:22
rloomgagne: i can give you the exact config info tomorrow (or thurs) if you remind me. it is mentioned in the release notes.01:23
*** rloo has quit IRC01:23
*** liuzz_ has quit IRC01:53
*** liuzz has joined #openstack-ironic01:54
*** arundhati has joined #openstack-ironic02:15
openstackgerritwangdequn proposed openstack/ironic master: change 'a optional CA' to 'an optional CA'  https://review.openstack.org/57493902:24
openstackgerritwangdequn proposed openstack/ironic master: change 'a ordinary file' to 'an ordinary file'  https://review.openstack.org/57494002:30
*** zshi_ has quit IRC02:33
openstackgerritwangdequn proposed openstack/ironic master: change 'a ordinary file ' to 'an ordinary file'  https://review.openstack.org/57494302:35
openstackgerritwangdequn proposed openstack/ironic master: modify 'a user' to 'an user'  https://review.openstack.org/57494402:40
openstackgerritwangdequn proposed openstack/ironic master: modify 'a user' to 'an user'  https://review.openstack.org/57494502:44
*** zshi_ has joined #openstack-ironic02:47
*** arundhati has quit IRC02:50
*** tiendc has quit IRC02:55
*** tiendc has joined #openstack-ironic02:55
openstackgerritwangdequn proposed openstack/ironic master: modify 'a user from' to 'an user from'  https://review.openstack.org/57494802:56
openstackgerritwangdequn proposed openstack/ironic master: rectify 'a URL' to 'an URL'  https://review.openstack.org/57495003:00
openstackgerritwangdequn proposed openstack/ironic master: rectify 'from a unit test' to 'from an unit test'  https://review.openstack.org/57495103:06
openstackgerritshuaili.wang proposed openstack/ironic master: Modify the Ironic api-ref's parameters in parameters.yaml  https://review.openstack.org/51788303:08
openstackgerritwangdequn proposed openstack/ironic master: rectify 'generate a URL' to 'generate an URL'  https://review.openstack.org/57495503:14
openstackgerritwangdequn proposed openstack/ironic master: rectify 'on a UEFI' to 'on an UEFI'  https://review.openstack.org/57495603:17
openstackgerritwangdequn proposed openstack/ironic master: rectify 'by a unique human-readable' to 'by an unique human-readable'  https://review.openstack.org/57495903:24
openstackgerritwangdequn proposed openstack/ironic master: rectify 'When a UUID' to 'When an UUID'  https://review.openstack.org/57496103:28
*** hoonetorg has joined #openstack-ironic03:28
*** boris_42_ has quit IRC03:29
*** rajinir has joined #openstack-ironic03:31
openstackgerritwangdequn proposed openstack/ironic master: rectify 'is a unique identifier' to 'is an unique identifier'  https://review.openstack.org/57496303:32
*** zhangfei has quit IRC03:50
*** zhangfei has joined #openstack-ironic04:01
*** links has joined #openstack-ironic04:24
*** arundhati has joined #openstack-ironic04:40
*** pcaruana has joined #openstack-ironic04:50
openstackgerritMerged openstack/ironic master: Validating fault value when querying with fault field  https://review.openstack.org/57471805:03
*** pcaruana has quit IRC05:18
*** adreznec has quit IRC05:27
*** adreznec has joined #openstack-ironic05:32
*** rajinir has quit IRC05:39
*** hjensas has quit IRC05:44
*** liuzz_ has joined #openstack-ironic05:45
*** liuzz has quit IRC05:48
*** mjura has joined #openstack-ironic05:55
*** zshi_ has quit IRC05:59
*** pcaruana has joined #openstack-ironic06:06
*** zshi_ has joined #openstack-ironic06:12
*** lifeless has quit IRC06:26
*** lifeless has joined #openstack-ironic06:26
*** liuzz has joined #openstack-ironic06:30
*** liuzz_ has quit IRC06:30
*** zshi_ has quit IRC06:32
*** zshi_ has joined #openstack-ironic06:35
*** masber has joined #openstack-ironic06:35
*** masuberu has quit IRC06:38
*** masuberu has joined #openstack-ironic06:41
*** masber has quit IRC06:45
*** lifeless_ has joined #openstack-ironic06:48
*** lifeless has quit IRC06:48
*** serlex has joined #openstack-ironic06:50
*** masuberu has quit IRC06:54
*** masuberu has joined #openstack-ironic06:54
*** AlexeyAbashkin has joined #openstack-ironic06:54
*** masuberu has quit IRC06:55
*** masuberu has joined #openstack-ironic06:55
*** masuberu has quit IRC06:57
*** masuberu has joined #openstack-ironic06:57
*** masuberu has quit IRC06:58
*** masuberu has joined #openstack-ironic06:59
*** milan has joined #openstack-ironic06:59
*** rcernin has quit IRC07:00
*** masuberu has quit IRC07:00
*** masuberu has joined #openstack-ironic07:00
*** masuberu has quit IRC07:01
*** masuberu has joined #openstack-ironic07:01
*** masuberu has quit IRC07:02
*** masuberu has joined #openstack-ironic07:03
*** tesseract has joined #openstack-ironic07:07
*** skazi has quit IRC07:17
*** threestrands has quit IRC07:21
*** skazi has joined #openstack-ironic07:26
*** arundhati_ has joined #openstack-ironic07:31
*** arundhati has quit IRC07:31
openstackgerritzenghui.shi proposed openstack/ironic master: BIOS Settings: add sync_node_setting  https://review.openstack.org/57172207:45
*** hshiina has quit IRC07:49
*** ajya has joined #openstack-ironic07:51
*** lifeless has joined #openstack-ironic07:52
*** lifeless_ has quit IRC07:52
*** arxcruz|brb is now known as arxcruz|ruck07:59
*** jaganathan has quit IRC07:59
*** jcoufal has joined #openstack-ironic08:00
*** e0ne has joined #openstack-ironic08:06
*** dougsz has joined #openstack-ironic08:10
*** skazi has quit IRC08:14
*** skazi has joined #openstack-ironic08:20
*** skazi has quit IRC08:22
*** skazi has joined #openstack-ironic08:22
*** AlexeyAbashkin has quit IRC08:31
*** derekh has joined #openstack-ironic08:33
*** arundhati__ has joined #openstack-ironic08:34
*** arundhati_ has quit IRC08:37
*** AlexeyAbashkin has joined #openstack-ironic08:40
*** trungnv has quit IRC08:43
*** nizam037 has joined #openstack-ironic08:44
*** rbartal has joined #openstack-ironic08:48
*** dtantsur|afk is now known as dtantsur08:58
dtantsurmorning ironic08:58
*** links has quit IRC09:08
dtantsurvdrok, hi! what can be better than removing drivers in the morning? :D https://review.openstack.org/#/c/574647/09:13
patchbotpatch 574647 - ironic - Remove the fake_ipmitool, fake_ipmitool_socat and ...09:13
etingofmorning everyone o/09:17
dtantsurhey etingof09:18
nizam037hi all. can we provision 2 nodes into deleted simultaneously?... is there any option for that..09:20
*** links has joined #openstack-ironic09:25
dtantsurnizam037: what's the problem with issuing two deletion requests in a raw? they're asynchronous09:27
* dtantsur brb09:27
*** lifeless_ has joined #openstack-ironic09:34
*** lifeless has quit IRC09:34
*** pbourke has quit IRC09:43
*** pbourke has joined #openstack-ironic09:44
*** lifeless has joined #openstack-ironic09:44
*** lifeless_ has quit IRC09:46
openstackgerritsunqingliang proposed openstack/bifrost master: fix typos  https://review.openstack.org/57505809:51
yolandahi, can i get an extra review for https://review.openstack.org/#/c/568616/ ?09:53
patchbotpatch 568616 - ironic-tempest-plugin - Add manual clean step ironic standalone test09:53
yolandagood morning09:53
openstackgerritsunqingliang proposed openstack/bifrost master: add release note  https://review.openstack.org/57506209:59
openstackgerrityolanda.robla proposed openstack/ironic-tempest-plugin master: WIP: Do not boot nodes if not set in standalone tests  https://review.openstack.org/57422810:00
vdrokmorning all, dtantsur , etingof , yolanda10:02
etingofvdrok, o/10:03
* etingof just mass-tagged triaged stories with the 'ironic-triaged' tag -- https://storyboard.openstack.org/#!/board/8310:04
*** AlexeyAbashkin has quit IRC10:06
*** AlexeyAbashkin has joined #openstack-ironic10:06
etingofdtantsur's script can tell how much work is glooming -- http://paste.openstack.org/show/723373/10:07
openstackgerritChuck Short proposed openstack/networking-baremetal master: Switch to using stestr  https://review.openstack.org/57472310:08
*** masuberu has quit IRC10:10
*** AlexeyAbashkin has quit IRC10:11
*** links has quit IRC10:12
*** d0ugal has quit IRC10:17
sambetts|afketingof: shame it can't tell you how many of them are duplicates :-P10:17
dtantsurwell, once SB supports duplicates...10:18
*** arundhati__ has quit IRC10:18
*** lifeless has quit IRC10:23
*** lifeless has joined #openstack-ironic10:24
*** arundhati__ has joined #openstack-ironic10:24
*** links has joined #openstack-ironic10:25
openstackgerritDerek Higgins proposed openstack/ironic master: WIP: Testing a basic zuulv3 job  https://review.openstack.org/57324410:27
*** nizam037 has quit IRC10:29
*** lifeless_ has joined #openstack-ironic10:32
*** lifeless has quit IRC10:32
*** MrBerns2 has joined #openstack-ironic10:33
MrBerns2Hi guys, can anyone tell me how I can enable 'shred' for node cleaning?10:35
*** d0ugal has joined #openstack-ironic10:36
*** jaganathan has joined #openstack-ironic10:40
etingofI am wondering if we can use some other properties of the story to tag it ironic-triaged...? Like if the task(s) are 'Merged' already, can we consider the story as triaged?10:40
TheJuliaMrBerns2: is cleaning enabled on the conductor?10:40
TheJuliaMrBerns2: if secure erase is supported, we will use that instead of shred, we fall back to shred if secure erase is not available10:41
MrBerns2TheJulia, Yes secure erase is enabled. I can see that it fails though and the IPA does not fall back to shred10:43
TheJuliaMrBerns2: it fails to unlock the disk?10:43
MrBerns2Well no, it seems that hdparam has some a problem with the running kernel of the IPA10:44
MrBerns2 Stderr: u'The running kernel lacks CONFIG_IDE_TASK_IOCTL support for this device.\nSECURITY_ERASE: Invalid argument\n': BlockDeviceEraseError: Error erasing block device: Erase failed for device /dev/sda: Unexpected error while running command.10:44
TheJuliawow.... that is a new one10:44
MrBerns2Then IPA does not fall back to shred because it says that its not enabled10:45
MrBerns2Failed to invoke ata_erase, fallback to shred is not enabled: Error erasing block device: Erase failed for device /dev/sda: Unexpected error while running command.10:45
TheJuliawhat version of the software are you using?10:45
MrBerns2And it seems according to the code IPA checks the driver_internal_info field of ironic to see if this is enabled10:46
TheJuliayeah, if we think it is enabled but it blows up then we fail hard and don't fall back to allow operators to attempt to fix the issue10:46
MrBerns2Oh I see10:46
MrBerns2TheJulia, Which software is that, hdparam or IPA?10:47
TheJuliathe reason for that is shred on SSDs can be super costly10:47
MrBerns2FYI after this failure it seems that the disk is nuked, I.E the controller seems to show it as failed10:47
TheJuliaipa, also any information you can provide about the ipa image your using10:47
TheJuliaMrBerns2: And what is the make/model or at least disk controller and configuration your using?10:48
MrBerns2I built the ipa image using D.I.B #, centos7 and the only element is the devuser10:48
TheJuliaso master branch code....10:48
MrBerns2Controller is the PERC H310 Mini10:48
TheJuliarpioso|afk: ^^^ any thoughts for when your awake10:48
MrBerns2diskimage-builder (2.14.1)10:49
MrBerns2FYI the drives that were erased were HDDs10:50
dtantsuretingof: hmm. I wonder if we still want to triage the merged stories to make sure they have received an adequate fix..10:51
*** hjensas has joined #openstack-ironic10:52
*** hjensas has quit IRC10:52
*** hjensas has joined #openstack-ironic10:52
MrBerns2TheJulia, I can provide any more information if its required,not sure about the master branch code of IPA but it should be quite recent this image was build this week. If possible is there any way to recover these disks after the failed hdparam erase?10:54
TheJuliaMrBerns2: how long does the entire process take? I guess hdparm would blow up on something being presented as SAS/SCSI10:55
TheJuliadtantsur: I like the idea, but time/brain power wise, I'm fearful whoever goes down that road will end up burnt out before they hit the 30% mark10:56
openstackgerritMerged openstack/ironic master: change 'a ordinary file ' to 'an ordinary file'  https://review.openstack.org/57494310:56
MrBerns2After booting the error only took a few minutes to appear10:56
TheJuliaMrBerns2:  and cleaning errored I take it?10:56
etingofdtantsur, I suspect the story flips from Active to Merged once all its tasks get merged, by this point the story disappears from our triage board effectively untriaged10:57
dtantsurTheJulia: agreed :(10:57
TheJuliaMrBerns2: by chance, have you tried with a debian based ramdisk?10:57
dtantsuretingof: right, maybe it's fine, at least for now10:57
MrBerns2TheJulia, yep the cleaning errored. I so far I've tried with a debian, centos, coreos and ubuntu disk10:58
MrBerns2same result10:58
TheJuliadtantsur: mentally I think it kind of goes along with the whole discussion that I proposed to update the principles... perhaps going that far is trying to strive to perfection... and there is the saying that the enemy of good is perfection.10:58
etingofdtantsur, what we can do is to tag the story as ironic-triaged once all its *ironic* tasks are merged10:58
TheJuliaMrBerns2: Interesting...10:59
openstackgerritMerged openstack/ironic master: Remove dulicate uses for zuul-cloner  https://review.openstack.org/57317710:59
*** serlex has quit IRC11:00
dtantsuretingof: I don't really feel like automatically triaging anything.. I'm fine with ignoring the merged stories, but the "triaged" tag should have a touch of a human (cat, owl) attention.11:00
TheJuliaMrBerns2: can you get us a dmesg output put into a paste.openstack.org link? I'm wondering if it is however the raid controller is presenting itself, the fact we're getting weird ioctl errors... makes me think we're going to have to patch IPA to become more knowledgeable... but of what I'm not exactly sure11:00
MrBerns2i am concerned though since I cant even see the physical disks in the BIOS now11:00
TheJuliawut?11:01
TheJuliaoh wow11:01
MrBerns2Yeah11:01
TheJuliain the raid controller?11:01
MrBerns2They are behind a raid controller but are not configured for raid, so should just be disks11:01
MrBerns2I used racadm to check them and 'racadm raid get pdisks -o -p state' => State                            = Failed11:02
TheJuliathe dell raid controllers still tend to pass-thru the disks... oh wow11:02
etingofdtantsur, alright then, just keep in mind that the 'triaged' tag may have been set on the story by folks from other projects e.g. triaging *their* stuff, not ours11:02
dtantsuryeah, well, ironic-triaged11:02
dtantsurit's a fun situation to untangle..11:03
etingofdtantsur, that's the shiny new world, but I am mentioning the inheritance we deal with11:03
TheJuliaMrBerns2: Yeah, we're going to need a dmesg output at a minimum, and we'll need rpioso|afk and friends. I'm sure they will want bios version details11:03
dtantsuretingof: life is hard, I'll go better remove a few more drivers :D11:04
* etingof hopes that will cheer dtantsur up ;) 11:04
MrBerns2TheJulia, ok cool, will get the BIOS output in a moment11:04
*** AlexeyAbashkin has joined #openstack-ironic11:05
* TheJulia slides dtantsur some Japanese whisky to accompany the driver removal11:05
dtantsur\o/11:05
dtantsurI certainly need to get some11:05
* TheJulia should seriously like open a bar called the developers bar11:05
etingofThe Foobar perhaps11:07
TheJuliawe'll need to graph balmer peaks though.... somehow.11:08
TheJuliafoobar might be good because coffee/tea would also need to be a thing11:08
TheJuliaand dive bar food.11:09
* TheJulia needs some VC11:09
*** arundhati__ has quit IRC11:10
*** jaganathan has quit IRC11:11
etingofit can be tricky with programmers, consider: Three Haskell programmers walk into a bar. The bartender asks "do you all want a beer?" The first one says "no." The other two turn out to never have walked into the bar at all.11:11
TheJuliaMrBerns2: also, if you have an identical server, the hdparm -I output would be extremely helpful11:13
TheJuliaor even nearly, same model/raid controller11:13
openstackgerritMerged openstack/ironic master: Consolidate the setting of ironic-extra-vars  https://review.openstack.org/57317811:13
TheJuliaMrBerns2: additionally, make/model of the disks11:17
MrBerns2TheJulia http://paste.openstack.org/show/GIJq66wrVUU0EGCV6wW1/11:19
*** arundhati has joined #openstack-ironic11:19
MrBerns2Thats from a similar server11:19
TheJuliaMrBerns2: from your failed cleaning, the conductor should have logs that have been uploaded to it from IPA. /var/log/ironic/deploy most likely11:20
MrBerns2TheJulia, there are a few there, but there are none from today, when I performed the cleaning11:24
TheJuliaokay11:25
TheJulia:(11:25
TheJuliaso those disks look like hybrid sas/sata disks with the WWN present11:25
TheJuliawell, that disk I should say11:26
MrBerns2I was logged into the IPA though when it was doing its thing and the secure erase failed and have those logs11:26
MrBerns2But the most relevant thing from those was the kernel error I pasted earlier, then there is a more generic error about HW manager not being found11:26
TheJuliaMrBerns2: the more data we have the better at this point. was that the ioctl missing in the kernel error?11:28
MrBerns2he running kernel lacks CONFIG_IDE_TASK_IOCTL support for this device.\nSECURITY_ERASE: Invalid argument\n': BlockDeviceEraseError: Error erasing block device: Erase failed for device /dev/sda11:30
MrBerns2TheJulia, here is some more of the log for context: http://paste.openstack.org/show/153QFdHa3YbtRpdlDlCk/11:32
*** zhangfei has quit IRC11:33
*** liuzz has quit IRC11:41
*** lifeless_ has quit IRC11:42
*** lifeless has joined #openstack-ironic11:42
TheJuliaMrBerns2: fyi, I'm reading on a train, so I'm intermittently loosing connectivity11:43
MrBerns2No problem TheJulia11:43
MrBerns2FYI I booted a livecd for that node so will provide a kern.log paste shortly11:44
TheJuliaSo I think what has occured is the information returned via the perc controller says "hey, I'm an ata disk, I'm supporting security", but it won't actually pass the ata-7 standard secure erase message over to the drive11:44
jrolletingof: ++ lol11:44
jrollmorning11:44
TheJuliaMrBerns2: awesome!11:44
* TheJulia opens up the hdparm source code11:45
*** serlex has joined #openstack-ironic11:47
openstackgerrityolanda.robla proposed openstack/ironic-tempest-plugin master: Add manual clean step ironic standalone test  https://review.openstack.org/56861611:51
*** tiendc has quit IRC11:56
openstackgerrityolanda.robla proposed openstack/ironic-tempest-plugin master: Add manual clean step ironic standalone test  https://review.openstack.org/56861611:57
openstackgerritDerek Higgins proposed openstack/ironic master: WIP: Testing a basic zuulv3 job  https://review.openstack.org/57324412:00
*** arundhati has quit IRC12:03
*** trown|outtypewww is now known as trown12:04
MrBerns2TheJulia, http://paste.openstack.org/show/91sKqwvxNe50HwwmRxr2/12:09
MrBerns2TheJulia, That is the kern log from the server with the nuked disks/hdparam problem12:09
TheJuliaso it looks like it doesn't even see anything disk wise12:12
MrBerns2Yeah looks like it12:12
openstackgerritMerged openstack/ironic master: Remove the fake_ipmitool, fake_ipmitool_socat and fake_snmp drivers  https://review.openstack.org/57464712:14
TheJuliato my train station, will be back in a little later, need to jump on the metro12:14
MrBerns2This doesn't look promising => DISCLAIMER: If you hit kernel or firmware bugs (which are plenty with not widely-tested features such as ATA Secure Erase) this procedure might render the drive unusable or crash the computer it's running on. (https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase)12:14
*** dprince has joined #openstack-ironic12:16
sambetts|afksecure erase scares me for that ^ reason ...12:17
*** marios has quit IRC12:20
dtantsurugh12:20
*** marios has joined #openstack-ironic12:20
MrBerns2Perhaps secure erase should not be done automatically when doing automated cleaning, unless explicitly enabled by the operator12:21
openstackgerritPavlo Shchelokovskyy proposed openstack/ironic master: Create a base ironic job from devstack-tempest  https://review.openstack.org/52016712:22
*** Alexey_Abashkin has joined #openstack-ironic12:24
*** AlexeyAbashkin has quit IRC12:26
*** AlexeyAbashkin has joined #openstack-ironic12:27
*** Alexey_Abashkin has quit IRC12:29
*** hoonetorg has quit IRC12:37
*** jcoufal has quit IRC12:45
*** dougsz has quit IRC12:47
*** hoonetorg has joined #openstack-ironic12:47
*** ajya has quit IRC12:53
*** ajya has joined #openstack-ironic12:54
*** AlexeyAbashkin has quit IRC12:54
openstackgerritIlya Etingof proposed openstack/ironic master: Adds boot mode support to ManagementInterface  https://review.openstack.org/52677312:55
openstackgerritIlya Etingof proposed openstack/ironic master: Added Redfish boot mode management  https://review.openstack.org/52831412:57
openstackgerritIlya Etingof proposed openstack/ironic master: Move boot-related code to boot_mode_utils.py  https://review.openstack.org/56938212:59
*** baha has joined #openstack-ironic13:01
*** dougsz has joined #openstack-ironic13:03
*** AlexeyAbashkin has joined #openstack-ironic13:04
*** Goneri has joined #openstack-ironic13:07
*** jcoufal has joined #openstack-ironic13:13
MrBerns2Is there any way for ironic to print firmware information about a node? (BIOS, iDRAC etc) I can find mention of upgrading firmware during node cleaning but I cant find any documentation on how to set it up13:16
openstackgerritPavlo Shchelokovskyy proposed openstack/ironic master: Create a base ironic job from devstack-tempest  https://review.openstack.org/52016713:17
*** markguz_ has joined #openstack-ironic13:18
*** markguz_ has quit IRC13:23
*** mjturek has joined #openstack-ironic13:28
* TheJulia is back at the visa office \o/13:29
*** AlexeyAbashkin has quit IRC13:33
TheJuliaMrBerns2: that is what im afraid of... I've only had that happen to me when doing secure erase and the power is turned off mid way. I suspect it is in a weird security state, and that the command was lost in the raid controller passthrough13:36
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Remove the remaining fake drivers  https://review.openstack.org/57511313:40
*** dougsz has quit IRC13:44
arnewiebalckHi Ironic: how do operators deal with nova enforcing the instance's power state to what nova has in its DB? Concrete case: If users bring up a node via IPMI, nova will shut it down. Seems one can completely disable the sync, but this will make the nova DB and reality go out of sync. It'd be nice to convince nova to use the instance as the source of truth (similar to Ironic's force_power_state_during_sync option). Any suggestion13:45
*** ajya has quit IRC13:51
*** ajya has joined #openstack-ironic13:51
MrBerns2TheJulia, were you able to recover the OSD when that happened?13:53
*** ElCoyote_ has joined #openstack-ironic13:54
NobodyCamGood Morning Ironic'ers, Happy Hump Day!13:54
TheJuliaThe specific model, it was mid-erase which afaik is unrecoverable. In your case the drive should be in SEC1 which i think is recoverable... Just not sure why rhe raid controller is not showing anything. Hopefully dell folks can provide insight13:55
TheJuliaNobodyCam: also known as visa pickup day.13:55
TheJuliaNobodyCam: Good morning!13:56
NobodyCamthe How long will you be traveling?13:56
NobodyCams/the//13:57
*** dougsz has joined #openstack-ironic13:59
TheJuliaS/rhe/the/14:00
NobodyCam:P14:00
openstackgerritDerek Higgins proposed openstack/ironic master: WIP: Testing a basic zuulv3 job  https://review.openstack.org/57324414:00
TheJuliaNobodyCam: 30 hours outbound... I think 22 return.14:00
TheJuliaDepending on flights and all14:00
NobodyCam:( oh wow14:01
dtantsurO___o14:01
TheJuliaYeah, it will be brutal... :(    but for a good cause (the community)14:04
NobodyCamoh very cool!14:06
NobodyCamHere's everything going well on the trip! when do you actually depart?14:06
MrBerns2Thanks TheJulia, Ill see what the Dell folks have to say14:07
*** ajya has quit IRC14:12
*** ajya has joined #openstack-ironic14:12
*** parlos has quit IRC14:13
openstackgerritPavlo Shchelokovskyy proposed openstack/ironic master: Remove deprecated [keystone] config section  https://review.openstack.org/57281714:15
openstackgerritPavlo Shchelokovskyy proposed openstack/ironic master: Remove deprecated [conductor]api_url option  https://review.openstack.org/57281814:15
openstackgerritDmitry Tantsur proposed openstack/ironic master: Remove the remaining fake drivers  https://review.openstack.org/57511314:20
dtantsurokay, here it goes ^^^14:20
TheJulia*boom*14:23
* dtantsur cannot do it any more.. at least not this week14:26
NobodyCamoh I have two days scheduled off but not until next week14:30
*** mjura has quit IRC14:32
openstackgerrityolanda.robla proposed openstack/ironic-tempest-plugin master: WIP: Do not boot nodes if not set in standalone tests  https://review.openstack.org/57422814:34
*** hoonetorg has quit IRC14:35
*** rbartal has quit IRC14:52
*** cjloader has joined #openstack-ironic14:53
*** markguz_ has joined #openstack-ironic15:00
*** r-daneel has quit IRC15:05
*** jcoufal has quit IRC15:09
*** jcoufal has joined #openstack-ironic15:10
*** AlexeyAbashkin has joined #openstack-ironic15:18
*** rpioso|afk is now known as rpioso15:21
rpiosoGood morning, ironic :-)15:22
*** rajinir has joined #openstack-ironic15:22
etingofmorning, rpioso ;)15:23
rpiosoetingof: Hey!15:23
*** AlexeyAbashkin has quit IRC15:30
*** lifeless has quit IRC15:31
*** lifeless has joined #openstack-ironic15:31
*** links has quit IRC15:36
TheJuliadtantsur: no driving yourself crazy... well... more crazy than the rest of us ;)15:38
TheJuliarpioso: hey, did you see the chat from earlier this morning?15:38
NobodyCamGood morning rpioso15:38
TheJuliaalso, Good morning!15:38
rpiosoTheJulia: Good morning! Yes, I read the novel ;-)15:39
TheJuliaBut it is not even a good novel! the characters are... well... yeah ;)15:39
* TheJulia notes the novel lacks space ships...15:40
*** AlexeyAbashkin has joined #openstack-ironic15:42
*** dprince has quit IRC15:42
rpiosoTheJulia: I'm going to reach out to an internal resource. Perhaps they'll have more insight than I do. I've never used secure erase. Also, I have no experience with IPA's cleaning. I use the idrac h/w type and its OOB RAID cleaning, which doesn't support secure erase.15:43
rpiosoNobodyCam: Happy camel day!15:44
NobodyCamhehe15:44
dtantsurgoing to a meetup today (and tomorrow), so leaving a bit earlier15:46
dtantsursee you15:46
*** dtantsur is now known as dtantsur|afk15:46
*** r-daneel has joined #openstack-ironic15:46
TheJuliarpioso: Okay, it seems like if there is any way we can get their drives out of SEC5, that might be good but it blocks all IO until unlocked or the secure erase completes. I think otherwise we're going to need to figure out a way to detect a pass-thru controller that might block secure erase and prevent us from even going down the path to begin with. My big worry is their disk(s) are inoperative as a result15:46
rpiosoTheJulia: LOL15:46
etingofdtantsur|afk, good luck!15:47
rpiosodtantsur|afk: What's your handicap? ;-)15:47
TheJuliarpioso: I _suspect_ that if the drives are removed from MrBerns2 server, and directly (without a raid controller in the mix) unlocked, they might be visible/usable again. I suspect the raid controller is showing them as failed and blocking access because it is in a security state.15:48
TheJuliaMrBerns2: out of curiosity, what hardware type did you use?15:48
*** r-daneel_ has joined #openstack-ironic15:49
TheJuliaHey, while we're at it, can I get some volunteers with SSDs and HDDs to run hdparm -I /dev/device and supply the output?   I'm wondering if one of the fields is being stripped and resulting as no value with a pass-through which might be a solid indicator.... sadly the cases of this being encountered are super limited on the internet.15:50
*** r-daneel has quit IRC15:51
*** r-daneel_ is now known as r-daneel15:51
*** gyee has joined #openstack-ironic15:51
MrBerns2TheJulia, pardon, what do you mean by the hardware type?15:52
*** jaganathan has joined #openstack-ironic15:52
TheJuliaMrBerns2: driver15:52
MrBerns2TheJulia, The driver is ipmi15:53
MrBerns2I will try mounting those disks on a node with no controller and see if I can unlock them15:53
rpiosoMrBerns2, TheJulia: I found http://downloads.dell.com/manuals/common/rc_h310_h710_h710p_h810_ug_en-us.pdf15:54
MrBerns2If I want to disable secure erase (so just keep the erase_devices_metadata step), should I just add 'erase_devices_priority = 0' to ironic.conf?15:55
openstackgerritIlya Etingof proposed openstack/ironic master: Move boot-related code to boot_mode_utils.py  https://review.openstack.org/56938215:56
rpiosoMrBerns2, TheJulia: Take a look at Chapter 7 Security Key And RAID Management. It contains some troubleshooting guidance.15:57
TheJuliaMrBerns2: you can, it won't shred, seems like we're going to need to add some more logic asap15:58
*** weshay is now known as weshay|ruck16:00
*** arxcruz|ruck is now known as arxcruz16:01
MrBerns2My aim for the time being is just to keep the metadata_erase step (so remove partitions & data etc) and not run the sercure_erase step which seems to be causing all these problems16:01
MrBerns2Thanks rpioso I will have a look at that chapter16:02
rpiosoMrBerns2: np16:03
openstackgerritIlya Etingof proposed openstack/ironic master: Adds boot mode support to ManagementInterface  https://review.openstack.org/52677316:07
openstackgerritIlya Etingof proposed openstack/ironic master: Move boot-related code to boot_mode_utils.py  https://review.openstack.org/56938216:11
openstackgerritIlya Etingof proposed openstack/ironic master: Added Redfish boot mode management  https://review.openstack.org/52831416:12
*** baha has quit IRC16:14
*** r-daneel_ has joined #openstack-ironic16:17
*** r-daneel has quit IRC16:18
*** r-daneel_ is now known as r-daneel16:18
*** ajya3 has joined #openstack-ironic16:19
*** ajya has quit IRC16:19
*** jcoufal has quit IRC16:28
*** r-daneel has quit IRC16:31
*** r-daneel has joined #openstack-ironic16:31
*** etingof is now known as etingof|brb16:32
*** tesseract has quit IRC16:33
*** ianychoi has quit IRC16:34
*** ianychoi has joined #openstack-ironic16:35
*** zul has quit IRC16:39
*** zul_ has joined #openstack-ironic16:39
*** zul has joined #openstack-ironic16:40
*** etingof|brb has quit IRC16:43
*** zul_ has quit IRC16:44
openstackgerritDerek Higgins proposed openstack/ironic master: WIP: Testing a basic zuulv3 job  https://review.openstack.org/57324416:51
openstackgerritAija Jaunteva proposed openstack/sushy-tools master: Document how to run emulators as systemd services  https://review.openstack.org/55198416:51
*** dougsz has quit IRC16:52
*** baha has joined #openstack-ironic16:55
*** etingof has joined #openstack-ironic16:58
*** Alexey_Abashkin has joined #openstack-ironic16:58
*** AlexeyAbashkin has quit IRC16:58
*** Alexey_Abashkin is now known as AlexeyAbashkin16:58
*** e0ne has quit IRC17:03
*** derekh has quit IRC17:03
*** AlexeyAbashkin has quit IRC17:10
*** ajya3 has quit IRC17:13
openstackgerritHarald JensÃ¥s proposed openstack/ironic-inspector master: Revert: Gate fix: Cap hacking to avoid gate failure  https://review.openstack.org/57517717:13
*** trown is now known as trown|lunch17:24
*** AlexeyAbashkin has joined #openstack-ironic17:28
openstackgerritMerged openstack/virtualbmc master: multiprocess server, ZMQ-based management cli tool  https://review.openstack.org/48887417:35
*** AlexeyAbashkin has quit IRC17:36
*** Goneri has quit IRC17:43
*** etingof has quit IRC17:46
*** yolanda has quit IRC17:50
*** akhilaki has joined #openstack-ironic17:58
*** akhilaki has left #openstack-ironic18:03
*** openstackgerrit has quit IRC18:04
* sambetts|afk --> ome 18:05
sambetts|afknight all18:05
*** yolanda has joined #openstack-ironic18:05
rpiosoTheJulia: I wonder if IPA contains the device driver for the PERC H310.18:10
*** harlowja has joined #openstack-ironic18:10
*** r-daneel_ has joined #openstack-ironic18:11
*** r-daneel has quit IRC18:11
*** r-daneel_ is now known as r-daneel18:11
TheJuliarpioso: MrBerns2 indicated it saw it the first go-around, and afterwards the idrac is saying that the perc card is reporting the drive as failed.18:11
TheJuliaand that the controller was just passing through the disk :\18:11
rpiosoTheJulia: What saw it the first go-around?18:11
rpiosoTheJulia: IPA or the iDRAC?18:11
TheJuliarpioso: IPA when it first tried to secure erase the disks18:11
*** jaganathan has quit IRC18:17
rpiosoTheJulia: I ask because of the message MrBerns2 sent at 6:22 AM EDT about the CONFIG_IDE_TASK_IOCTL command with argument SECURITY_ERASE failing.18:18
rpiosoTheJulia: That argument was invalid.18:18
rpiosoMrBerns2: Is the drive Instant Scramble Erase (ISE) capable?18:20
*** e0ne has joined #openstack-ironic18:20
rpiosoMrBerns2: I'm presently chatting with my colleague. Are you available?18:25
*** Goneri has joined #openstack-ironic18:28
rpiosoTheJulia: Which command does IPA use to perform the secure erase?18:29
*** trown|lunch is now known as trown18:30
*** openstackgerrit has joined #openstack-ironic18:31
openstackgerritMerged openstack/ironic-tempest-plugin master: Power fault recovery: tempest tests  https://review.openstack.org/55817018:31
TheJuliaI've created the secure erase issue as https://storyboard.openstack.org/#!/story/200254618:32
TheJuliarpioso: one moment18:33
TheJuliaMrBerns2: I don't think the password change only option is available in many cases, so it wouldn't have been an instant erase...18:33
TheJuliaerr18:33
TheJuliathat was for rpioso18:33
TheJuliahttps://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/hardware.py#L954-L95918:34
TheJuliarpioso: ^^^18:34
TheJuliarpioso: I did some googling around and it looks like when there is a pass-through, that ioctl is not available with the SECURITY_ERASE flag18:35
rpiosoTheJulia: Interesting ...18:36
TheJuliabut hdparm shows the security flags becuase the other bits make it through it seems....18:36
TheJuliaI think I have an idea of how we can at least... possibly... spot a pass-through18:36
TheJuliaI suspect the master password count is bogus18:37
*** etingof has joined #openstack-ironic18:40
*** boris_42_ has joined #openstack-ironic18:44
*** etingof has quit IRC18:48
*** etingof has joined #openstack-ironic18:48
*** cjloader has quit IRC18:57
*** cjloader_ has joined #openstack-ironic18:57
*** cjloader_ has quit IRC18:58
*** cjloader has joined #openstack-ironic18:58
*** markguz__ has joined #openstack-ironic18:59
*** cjloader has quit IRC18:59
*** cjloader has joined #openstack-ironic19:00
*** markguz_ has quit IRC19:03
*** hoonetorg has joined #openstack-ironic19:04
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Try to unlock failed device before proceeding  https://review.openstack.org/57520519:23
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Refuse secure erase if password count is invalid  https://review.openstack.org/57520619:23
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Provide knob to disable ata secure erase  https://review.openstack.org/57520719:23
*** markguz__ has quit IRC19:26
*** markguz_ has joined #openstack-ironic19:27
*** e0ne has quit IRC19:34
*** e0ne has joined #openstack-ironic19:40
*** e0ne has quit IRC19:43
*** lifeless_ has joined #openstack-ironic19:49
*** lifeless has quit IRC19:49
*** cjloader has quit IRC19:52
*** e0ne has joined #openstack-ironic19:56
*** cjloader_ has joined #openstack-ironic19:57
*** cjloader_ has quit IRC19:57
*** cjloader has joined #openstack-ironic19:57
*** e0ne has quit IRC19:58
*** cjloader has quit IRC19:58
*** cjloader has joined #openstack-ironic19:58
*** baha has quit IRC20:11
*** AlexeyAbashkin has joined #openstack-ironic20:11
openstackgerritChristopher Dearborn proposed openstack/ironic master: Fix for Unable to create RAID1 on Dell BOSS card  https://review.openstack.org/57487920:11
*** cdearborn has joined #openstack-ironic20:14
*** AlexeyAbashkin has quit IRC20:16
TheJuliacdearborn: replied to your storyboard/launchpad question on the patch20:53
*** trown is now known as trown|outtypewww21:01
*** openstackgerrit has quit IRC21:04
*** cjloader_ has joined #openstack-ironic21:07
*** cjloader has quit IRC21:07
*** cjloader_ has quit IRC21:11
*** lifeless_ has quit IRC21:23
*** lifeless has joined #openstack-ironic21:24
*** openstackgerrit has joined #openstack-ironic21:24
openstackgerritguang-yee proposed openstack/networking-generic-switch master: support hpe device type for the HPE 5900 series switches  https://review.openstack.org/47105121:24
openstackgerritJulia Kreger proposed openstack/ironic master: Adds enable_ata_secure_erase option  https://review.openstack.org/57523821:26
openstackgerritDerek Higgins proposed openstack/ironic master: WIP: Testing a basic zuulv3 job  https://review.openstack.org/57324421:26
*** exodusftw has quit IRC21:28
openstackgerritChristopher Dearborn proposed openstack/ironic master: Fix for Unable to create RAID1 on Dell BOSS card  https://review.openstack.org/57487921:32
*** exodusftw has joined #openstack-ironic21:33
*** milan has quit IRC21:35
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Provide knob to disable ata secure erase  https://review.openstack.org/57520721:36
*** lifeless_ has joined #openstack-ironic21:36
*** lifeless has quit IRC21:37
openstackgerritguang-yee proposed openstack/networking-generic-switch master: support hpe device type for the HPE 5900 series switches  https://review.openstack.org/47105121:47
*** cdearborn has quit IRC21:56
*** srart has joined #openstack-ironic22:02
*** Goneri has quit IRC22:06
*** harlowja has quit IRC22:11
*** harlowja has joined #openstack-ironic22:12
*** harlowja has quit IRC22:21
*** serlex has quit IRC22:24
*** rcernin has joined #openstack-ironic22:26
*** rcernin is now known as rcernin|brekkie22:27
*** rcernin|brekkie is now known as rcernin22:27
*** threestrands has joined #openstack-ironic22:47
*** athomas has quit IRC22:47
*** markguz_ has quit IRC22:49
*** anupn has quit IRC23:00
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Try to unlock failed device before proceeding  https://review.openstack.org/57520523:30
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Refuse secure erase if password count is invalid  https://review.openstack.org/57520623:30
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Provide knob to disable ata secure erase  https://review.openstack.org/57520723:30
TheJuliaMrBerns2: ^^^ I hope those will prevent your issue from ever occurring again... and giving you the tunable you need. Please take a look those when you have some time. :)23:31
openstackgerritJulia Kreger proposed openstack/ironic master: Adds enable_ata_secure_erase option  https://review.openstack.org/57523823:32
*** mjturek has quit IRC23:33
*** cjloader has joined #openstack-ironic23:52
*** cjloader has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!