Monday, 2015-04-13

*** zhenguo has joined #openstack-ironic00:01
*** achanda has quit IRC00:05
*** jamielennox|away is now known as jamielennox00:06
*** davideagnello has joined #openstack-ironic00:06
*** davideagnello has quit IRC00:11
*** gabriel-bezerra has joined #openstack-ironic00:14
Haomengmrda: :)00:20
openstackgerritHaomeng,Wang proposed openstack/python-ironicclient: (WIP)validate input uuid from client side  https://review.openstack.org/17261800:20
*** naohirot has joined #openstack-ironic00:41
*** kkoski has joined #openstack-ironic01:02
*** igordcard_ has quit IRC01:04
*** kkoski has quit IRC01:06
*** achanda has joined #openstack-ironic01:21
*** Marga_ has quit IRC01:50
*** chenglch has joined #openstack-ironic02:22
*** chenglch has quit IRC02:36
*** chenglch has joined #openstack-ironic02:37
*** ricolin has quit IRC02:39
*** ramineni has joined #openstack-ironic02:40
*** Haomeng|2 has joined #openstack-ironic02:43
*** Haomeng has quit IRC02:43
*** achanda has quit IRC02:44
*** achanda has joined #openstack-ironic02:48
*** oomichi has quit IRC02:55
*** achanda has quit IRC02:59
*** lintan_ has joined #openstack-ironic03:05
*** Marga_ has joined #openstack-ironic03:11
*** Marga_ has quit IRC03:12
*** subscope_ has joined #openstack-ironic03:12
*** Marga_ has joined #openstack-ironic03:12
openstackgerritTan Lin proposed openstack/ironic: Improve AMT driver doc  https://review.openstack.org/16960603:20
*** naohirot has quit IRC03:28
Haomeng|2mrda: are you around?03:40
Haomeng|2mrda: do you know if we can change the exception string now, I understand it is in freezon status, right?03:41
*** achanda has joined #openstack-ironic03:59
*** mdbooth has quit IRC04:01
*** naohirot has joined #openstack-ironic04:01
*** achanda has quit IRC04:05
*** lintan_ has quit IRC04:07
*** mdbooth has joined #openstack-ironic04:08
*** saripurigopi has joined #openstack-ironic04:17
mrdaHaomeng|2: Sorry for delay in getting back to you.  I don't think we want to make any string changes now.04:17
mrdaonly bug fixes, and even then, no string changes.  I think otherwise our i18n folk are too overworked.04:18
*** lintan_ has joined #openstack-ironic04:23
Haomeng|2mrda: ok, thk04:27
*** subscope_ has quit IRC04:43
*** chenglch has quit IRC05:07
openstackgerritAnusha Ramineni proposed stackforge/proliantutils: UPDATE_PERSISTENT_BOOT to 'NETWORK' fails on Gen9  https://review.openstack.org/17278505:09
saripurigopiIs there a way to enroll ironic nodes from horizon ?05:10
openstackgerritAnusha Ramineni proposed stackforge/proliantutils: UPDATE_PERSISTENT_BOOT to 'NETWORK' fails on Gen9  https://review.openstack.org/17278505:15
openstackgerritAnusha Ramineni proposed stackforge/proliantutils: UPDATE_PERSISTENT_BOOT to 'NETWORK' fails on Gen9  https://review.openstack.org/17278505:22
Haomeng|2saripurigopi: tripleo UI have such function, can you try with tuskar-ui?05:29
saripurigopiHaomengl2, sure will try with tuskar-ui.05:29
*** pradipta has joined #openstack-ironic05:35
*** chenglch has joined #openstack-ironic05:35
saripurigopiLooking at the ironic specs, kilo-archive is created for postponed BPs. Is it open to submit the review changes?05:49
*** achanda has joined #openstack-ironic05:53
Haomeng|2saripurigopi: yes, we can create bp for next release now, welcome05:54
*** yuanying has quit IRC06:04
*** yuanying has joined #openstack-ironic06:08
*** jcoufal has joined #openstack-ironic06:26
*** rwsu has joined #openstack-ironic06:34
saripurigopiHaomengl2, do I need to submit the BP again or will it be automatically moved to Liberty?06:35
Haomeng|2saripurigopi: no need to submit again, will not move to *L* release, so dont care06:41
Haomeng|2saripurigopi: which is your bp patch? can you share the link?06:41
saripurigopiHaomengl2, okay. pxe_ucs driver - https://review.openstack.org/#/c/15973406:43
*** ukalifon1 has joined #openstack-ironic06:46
Haomeng|2saripurigopi: I understand this is the code patch, do you have blueprint patch/06:49
Haomeng|2saripurigopi: that shoud be blueprint's spec06:49
saripurigopiHaomengl2 , sorry , here is BP review - https://review.openstack.org/#/c/139517/06:50
Haomeng|2saripurigopi: oh, your bp is approved, so dont worry06:51
Haomeng|2saripurigopi: once main branch is *L*, we can go ahead to do these patches06:52
saripurigopiHaomengl2: okay.06:52
Haomeng|2saripurigopi: so no need to submit bp again:)06:52
Haomeng|2saripurigopi: thanks for your bp:)06:52
saripurigopiHaomengl2: can I submit the code changes for review, or should I wait till main branch is "L"06:53
*** yuanying has quit IRC06:54
Haomeng|2saripurigopi: I see deva's comments, so it is better we start to new patch during the second or third week of April:)07:07
saripurigopiHaomengl2: okay. :-)07:08
Haomeng|2saripurigopi: and we can fix some Jenkins issue now, and run ut with latest code in our local dev env I think:)07:08
Haomeng|2saripurigopi: good luck:)07:08
Haomeng|2saripurigopi: :)07:08
saripurigopiHaomengl2: sure, thank you.(y)07:08
*** jamielennox is now known as jamielennox|away07:11
*** yuanying has joined #openstack-ironic07:21
*** achanda has quit IRC07:21
*** achanda has joined #openstack-ironic07:22
*** jistr has joined #openstack-ironic07:24
*** chlong has quit IRC07:25
*** yuanying has quit IRC07:28
*** yuanying has joined #openstack-ironic07:31
*** mrda is now known as mrda-away07:32
*** Marga_ has quit IRC07:32
*** yuanying has quit IRC07:33
*** ifarkas has joined #openstack-ironic07:41
*** rsalevsky has joined #openstack-ironic07:43
*** yuanying has joined #openstack-ironic07:48
*** yuanying has quit IRC07:51
*** yuanying has joined #openstack-ironic07:52
*** yuanying has quit IRC07:54
*** yuanying has joined #openstack-ironic07:59
*** yuanying has quit IRC08:02
*** chenglch has quit IRC08:03
*** ijw has quit IRC08:03
*** yuanying has joined #openstack-ironic08:07
*** lucasagomes has joined #openstack-ironic08:09
*** andreykurilin has joined #openstack-ironic08:11
*** a1exhughe5 has joined #openstack-ironic08:12
*** dtantsur|afk is now known as dtantsur08:14
dtantsurMorning everyone :)08:14
pshigedtantsur: morning :)08:14
*** edcragg has joined #openstack-ironic08:15
*** stendulker has joined #openstack-ironic08:16
*** lsmola_ has joined #openstack-ironic08:16
*** MattMan has joined #openstack-ironic08:21
*** achanda has quit IRC08:22
lucasagomesdtantsur, morning08:22
dtantsuro/08:22
pshigelucasagomes: morning :)08:23
lucasagomespshige, morning there08:23
*** romcheg has joined #openstack-ironic08:23
*** ndipanov has joined #openstack-ironic08:28
*** tiagogomes_ has joined #openstack-ironic08:29
*** derekh has joined #openstack-ironic08:32
openstackgerritSHIGEMATSU Mitsuhiro proposed openstack/ironic: Add Ceilometer to Ironic's Conceptual Architecture  https://review.openstack.org/17238408:46
pshigedtantsur: thank you for your comment. I fixed it.08:48
dtantsurok08:48
dtantsurifarkas, hey-hey, how was your PTO? ready for some reviews? ;)08:50
ifarkasdtantsur, hi. thanks, it was excellent!08:51
ifarkasdtantsur, I will check the open patches for discoverd08:51
dtantsurifarkas, yep, there should be 3 of my patches there. And you were lucky enough to avoid a huge documentation patch :D08:52
dtantsurthanks08:52
ifarkashehe08:52
*** pelix has joined #openstack-ironic08:53
*** athomas has joined #openstack-ironic09:01
*** jamielennox|away is now known as jamielennox09:02
dtantsurlucasagomes, ifarkas, btw have you seen our folks proposing an openstackclient plugin for ironicclient?09:06
dtantsurit's starting with https://review.openstack.org/17167209:06
ifarkasdtantsur, I haven't yet, but I will take a look09:07
lucasagomesdtantsur, yeah i think you posted it here on friday09:10
lucasagomeslemme look at the patch see if there's comments/opnions about it09:10
dtantsurdeva wants to discuss it a bit later, otherwise seems like no comments09:11
lucasagomesyeah09:12
lucasagomesbut looks good :-)09:12
naohirotlucasagomes: good morning, I have a question about Local boot with partition images09:14
lucasagomesnaohirot, hi there, sure09:14
naohirotlucasagomes: https://github.com/openstack/ironic/commit/9a1544b76a0bb75bcf5c28647ede9079a10e097e#diff-fe0ee0bce65182e98f6966120e008656R90809:14
naohirotlucasagomes: Hi, here "deployed with Ironic **must** contain ``grub2`` installed within it."09:14
pshigenaohirot: good evening :)09:15
lucasagomesnaohirot, yeah, just the packaged installed into it09:15
lucasagomesbecause the ramdisk uses the grub2 version from the image09:15
lucasagomesand isntall it on the disk09:15
naohirotlucasagomes: I'm not sure how to create ubuntu image with grub2.09:15
naohirotpshige: Hi good evening :)09:15
lucasagomesnaohirot, with dib? yeah I had a patch up but apparently it didn't work for some people (ramesh87 said it worked for him tho)09:16
lucasagomeslemme try to find it09:16
lucasagomesnaohirot, you can install it manually if needed too09:16
lucasagomeshttps://review.openstack.org/#/c/167929/09:16
naohirotlucasagomes: I see, you are working on it. I couldn't find how to create the image using dib.09:17
naohirotlucasagomes: without dib, is there any way to create the image?09:18
naohirotlucasagomes: will the image be a whole disk image?09:19
lucasagomesnaohirot, theoretically you could use other image buildings projects. Or even just download the images directly from fedora or ubuntu09:19
lucasagomesDIB just customize them09:19
lucasagomesnaohirot, oh no, if u install grub2 from that package it will only install the grub2 package and that's it09:20
lucasagomesit won't create a full disk image (I just replied greghaynes comment there)09:20
naohirotlucasagomes: Okay, I think you are saying I can do it manually by refereeing the  https://review.openstack.org/#/c/167929/09:20
naohirotlucasagomes: right?09:20
lucasagomesyeah please try that patch09:21
lucasagomesapparently it didn't work for ubuntu with lintan_09:21
lucasagomes:-/ I need to find a way to fix it09:21
naohirotlucasagomes: It seems readhat element has some code which invokes grub2-install, here https://github.com/openstack/diskimage-builder/blob/master/elements/redhat-common/pre-install.d/15-remove-grub09:23
lucasagomesyeah09:23
lucasagomesthis element will install it anyway09:23
lucasagomesI believe they do remove grub by default to save space09:23
lucasagomes(since grub can add ~30mb in the image that won't be used if not local boot)09:24
naohirotnaohirot: If I try redhat, can I create the image using current version of dib?09:25
lucasagomesnaohirot, yeah you can, but for local boot you'll still need #167929 to install the bootloader09:26
lucasagomesor you can install it after the image generation by hand (by mouting the image and installing it in the chroot)09:27
lucasagomesdtantsur, can we close this https://bugs.launchpad.net/ironic/+bug/1397988 ?09:27
openstackLaunchpad bug 1397988 in Ironic "Ironic should allow choosing the root device" [Medium,In progress] - Assigned to Lucas Alvares Gomes (lucasagomes)09:27
naohirotlucasagomes: Aha, 15-remove-grub09:27
naohirot removes grub2. So apparently I need to change it so that it doesn't remove it.09:27
lucasagomesnaohirot, yeah but not all images come with grub209:27
lucasagomesafaict fedora cloud images doesn't09:27
openstackgerritMerged openstack/ironic: Use mock patch decorator for eventlet.greenthread.sleep  https://review.openstack.org/17204709:28
lucasagomesthe best way I would say it to use #167929 which will make sure grub2 is installed as part of the image generation09:28
dtantsurlucasagomes, I guess we can09:29
lucasagomescool yeah because the spec code is already merged and all09:29
naohirotlucasagomes: I'd like to make sure additionally one thing, do we need grub2 both deploy and boot images? or just deploy image?09:33
lucasagomesnaohirot, just for the image being deployed09:33
lucasagomesthe deploy ramdisk doesn't need any change (it needs to be updated, since this feature is a K feature)09:33
naohirotlucasagomes: Okay grub2 element needs to be specified in the disk-image-create command, but not ramdisk-image-create09:35
lucasagomesnaohirot, yes09:35
naohirotlucasagomes: dtantsur: okay I'll try disk-image-create with the grub2 element, thanks :)09:36
lucasagomesnaohirot, yw :-) lemme know if it works for u09:36
naohirotlucasagomes: sure!09:36
raminenidtantsur: hi09:38
*** lintan_ has quit IRC09:39
raminenidtantsur: this patch https://review.openstack.org/#/c/169162/ , not getting merged since Friday, not sure what is the issue09:40
openstackgerritMerged openstack/ironic: Improve AMT driver doc  https://review.openstack.org/16960609:43
dtantsurthere was some problem with Zuul, let's recheck once more09:44
dtantsurdone09:44
dtantsurramineni, ^^09:44
raminenidtantsur: thanks09:45
*** lintan_ has joined #openstack-ironic09:52
openstackgerritNaohiro Tamura proposed openstack/ironic-specs: iRMC Virtual Media Deploy Driver for Ironic  https://review.openstack.org/13486509:56
*** naohirot has quit IRC09:57
*** igordcard_ has joined #openstack-ironic09:58
*** romcheg has quit IRC10:03
*** romcheg has joined #openstack-ironic10:07
openstackgerritMerged openstack/ironic: Add Ceilometer to Ironic's Conceptual Architecture  https://review.openstack.org/17238410:10
*** igordcard_ has quit IRC10:10
*** igordcard_ has joined #openstack-ironic10:11
*** athomas has quit IRC10:11
*** amitpp has joined #openstack-ironic10:14
*** athomas has joined #openstack-ironic10:18
*** amitpp has quit IRC10:22
kejiahugood morning all, I am trying to use devstack deploying openstack with Ironic on HP moonshot, does anybody have any clue on the configs? thanks.10:25
tiagogomes_the ssh_pxe driver passing by a ssh password doesn't work for me, I had to pass the private key10:27
*** romcheg has quit IRC10:28
*** gillllliard is now known as gilliard10:29
*** romcheg has joined #openstack-ironic10:30
dtantsurlucasagomes, what about writing a spec for https://bugs.launchpad.net/ironic/+bug/1443327 ?10:31
openstackLaunchpad bug 1443327 in Ironic "Custom PXE parameters per node" [Wishlist,New] - Assigned to Lucas Alvares Gomes (lucasagomes)10:31
dtantsurlucasagomes, I suspect some people will want a pet-vs-cattle discussion again :)10:32
lucasagomeshmm10:32
lucasagomesyeah well I can do that10:32
lucasagomesI thought it could be very simple and not need a spec10:32
lucasagomesit won't change any data model or api10:32
*** igordcard_ has quit IRC10:32
lucasagomesjust an extra parameter at driver_info10:32
*** igordcard_ has joined #openstack-ironic10:33
kejiahutiagogomes_, cheers, I haven't got to the step, but I will make a node of that issue, thanks10:33
kejiahus/node/note/10:33
dtantsurlucasagomes, personally I don't mind it, not sure what folks will say. though as we already have a per-node deploy ramdisk, this makes sense to me10:33
dtantsurok, let's go with just a bug for now10:34
lucasagomesack10:34
lucasagomesyeah we can discuss it on the patch10:34
lucasagomesif it turns out we need a spec I will put one up10:34
dtantsurcool10:34
tiagogomes_kejiahu, that was not particularly addressed to you :)10:35
kejiahu:) still worth to make an attention.10:36
*** takadayuiko has joined #openstack-ironic10:37
*** chenglch has joined #openstack-ironic10:37
*** kozhukalov has joined #openstack-ironic10:39
*** andreykurilin has quit IRC10:43
*** stendulker has quit IRC10:50
*** ramineni has quit IRC10:53
tiagogomes_the dnsmasq process for providing an IP to baremetal seems not being able to reolve my controller hostname, as it runs with --no-resolv and --no-hosts10:56
tiagogomes_any way of fixing that?10:56
*** yuanying has quit IRC10:58
lucasagomestiagogomes_, neutron is running the dnsmasq process for u?11:00
tiagogomes_lucasagomes, yes11:00
lucasagomesif so, I think it's possible to solve that by having a custom dnsmasq.conf11:00
lucasagomesyou can set it in the dhcp_agent.ini11:00
lucasagomes/etc/neutron/dhcp_agent.ini11:01
lucasagomesunder [DEFAULT] there's an option called "dnsmasq_config_file"11:01
lucasagomesI believe you can overwrite the options by setting it11:01
* lucasagomes is not a neutron expert tho11:01
tiagogomes_lucasagomes, ok, I'll try11:02
tiagogomes_thanks11:02
lucasagomesnp11:02
*** Haomeng has joined #openstack-ironic11:08
saripurigopiany of the drivers support out-of-band h/w introspection now?11:10
dtantsurilo11:11
*** Haomeng|2 has quit IRC11:11
saripurigopidtantsur: okay. will have a look at it.11:17
*** romcheg has quit IRC11:19
openstackgerritYuiko Takada proposed stackforge/ironic-discoverd: Use keystonemiddleware auth credentials  https://review.openstack.org/17078211:21
*** chenglch has quit IRC11:23
*** ukalifon3 has joined #openstack-ironic11:34
*** ukalifon3 has quit IRC11:34
*** ukalifon2 has joined #openstack-ironic11:35
*** ukalifon5 has joined #openstack-ironic11:37
*** ukalifon1 has quit IRC11:38
*** ukalifon2 has quit IRC11:40
*** mgoddard has joined #openstack-ironic11:42
*** romcheg has joined #openstack-ironic11:42
*** lucasagomes is now known as lucas-hungry11:47
*** andreykurilin has joined #openstack-ironic11:58
*** andreykurilin has quit IRC11:59
*** andreykurilin has joined #openstack-ironic12:00
*** romcheg has quit IRC12:00
*** Haomeng|2 has joined #openstack-ironic12:00
*** Haomeng has quit IRC12:03
*** ukalifon5 has quit IRC12:03
*** ukalifon1 has joined #openstack-ironic12:09
*** jamielennox is now known as jamielennox|away12:12
*** pradipta has quit IRC12:13
*** maurosr is now known as maurosr_12:16
*** takadayuiko has quit IRC12:18
*** EmilienM|afk is now known as EmilienM12:19
*** zz_jgrimm has quit IRC12:22
*** maurosr_ has quit IRC12:22
*** zz_jgrimm has joined #openstack-ironic12:24
*** dtantsur is now known as dtantsur|brb12:25
*** maurosr_ has joined #openstack-ironic12:25
*** maurosr_ is now known as maurosr12:26
*** dprince has joined #openstack-ironic12:26
*** saripurigopi has quit IRC12:33
*** andreykurilin has quit IRC12:39
*** lucas-hungry is now known as lucasagomes12:41
*** jroll has quit IRC12:50
*** jroll has joined #openstack-ironic12:50
jrollmorning12:58
lintan_morning jroll12:58
jrollhiya lintan_ :)12:59
*** romcheg has joined #openstack-ironic13:00
*** openstackgerrit has quit IRC13:00
pshigejroll: morning :)13:03
lucasagomesjroll, lintan_ morning13:03
*** openstackgerrit has joined #openstack-ironic13:03
jrollhey pshige, lucasagomes :)13:03
pshigelintan_: morning :)13:03
*** krtaylor has quit IRC13:03
*** lintan_ has quit IRC13:04
*** trown|outttypeww is now known as trown13:04
*** Marga_ has joined #openstack-ironic13:05
*** rloo has joined #openstack-ironic13:08
*** kkoski has joined #openstack-ironic13:13
openstackgerritBrad P. Crochet proposed openstack/python-ironicclient: Implementation of baremetal reboot command  https://review.openstack.org/17251713:30
openstackgerritBrad P. Crochet proposed openstack/python-ironicclient: Implementation of baremetal port create  https://review.openstack.org/17246113:30
*** andreykurilin has joined #openstack-ironic13:30
NobodyCamgood morning Ironic13:40
TheJuliaGood morning13:43
rloomorning TheJulia, NobodyCam13:44
*** absubram has joined #openstack-ironic13:45
NobodyCammorning TheJulia and rloo :)13:45
NobodyCamhow was the weekend13:45
rlooNobodyCam: always too short :-). How was yours?13:46
*** mjturek1 has joined #openstack-ironic13:46
NobodyCamoh the same. and the "kids" made sleep ruff lastnight :-p13:46
NobodyCambut other wise I'm doing good13:47
NobodyCam:)13:47
TheJuliaBusy, next weekend is house cleaning and then listing13:47
rloooh, TheJulia. You're moving?!13:47
NobodyCamoh!13:48
TheJuliarloo: I've mostly been living with my partner for the last 6 months, so it was time to dispose of my house :)13:49
TheJulias/was/is/13:49
rlooTheJulia: gotcha. Makes sense! :)13:50
NobodyCamlol13:51
*** mjturek1 has quit IRC13:51
*** mjturek1 has joined #openstack-ironic13:51
*** dtantsur|brb is now known as dtantsur13:56
dtantsurmorning jroll, NobodyCam, TheJulia, rloo!13:57
NobodyCammornign dtantsur :)13:57
NobodyCambrb13:57
*** a1exhughe5 has quit IRC14:01
*** tiagogomes_ has quit IRC14:01
*** edcragg has quit IRC14:01
*** tiagogomes_ has joined #openstack-ironic14:01
*** a1exhughe5 has joined #openstack-ironic14:01
*** edcragg has joined #openstack-ironic14:02
rloohi dtantsur, jrist14:02
rloooops, jrist. hi to you too, but I meant jroll :)14:02
*** stendulker has joined #openstack-ironic14:04
lucasagomesrloo, NobodyCam TheJulia morning14:04
lucasagomesjroll, you there14:04
lucasagomes?14:04
NobodyCammorning lucasagomes14:05
NobodyCamrloo: dtantsur: got a free second to look at https://review.openstack.org/#/c/172582 ?14:06
* dtantsur is looking14:07
NobodyCam:) awesome TY14:07
*** jerryz has joined #openstack-ironic14:10
openstackgerritLucas Alvares Gomes proposed openstack/ironic: Agent: Do not power off node if deployment fail  https://review.openstack.org/17293214:11
NobodyCamis jlvillal about?14:12
pshigeNobodyCam, TheJulia, rloo: morning :)14:12
NobodyCammorning pshige :)14:13
rlooevening? pshige, aft lucasagomes14:14
lucasagomes:-)14:15
pshigerloo: actually just before midnight in Japan :)14:18
rloopshige: good NIGHT then!14:18
NobodyCampshige: omg thats late14:20
*** krtaylor has joined #openstack-ironic14:20
*** BadCub_Hiding is now known as BadCub14:20
BadCubmornin folks14:21
NobodyCamMorning BadCub14:21
openstackgerritZhenguo Niu proposed openstack/ironic: Fix typos in vendor-passthru.rst  https://review.openstack.org/17293414:21
pshigeBadCub: morning14:21
BadCubmhiya pshige, NobodyCam :)14:22
BadCubhiya even14:22
* BadCub obviously needs more coffee14:22
BadCubAnyone available to look at https://review.openstack.org/#/c/171334/ ?14:27
NobodyCamlol: http://techcrunch.com/2015/04/09/this-custom-apple-ii-watch-is-my-new-favorite-thing/?ncid=rss&cps=gravity_1462_652061677749533598#.sw5hky:Mbwo14:27
dtantsurBadCub, I will14:28
BadCubdtantsur: awesome :)14:28
dtantsurBadCub, I guess devananda will want to, if he's around today14:29
*** jistr is now known as jistr|mtg14:29
BadCubdtantsur: He will most likely take a look, but we do need to land it anyway too :)14:30
*** saripurigopi has joined #openstack-ironic14:30
*** achanda has joined #openstack-ironic14:31
NobodyCamdevananda: should be back some time today. I expect he's recovering from his PTO14:31
jlvillalBadCub, I'm awake now :)14:32
NobodyCammorning jlvillal14:32
BadCubjlvillal: I am working on being awake! lol14:32
jlvillalSorry I meant NobodyCam  :)14:32
NobodyCamthank you for all the work on 17133414:33
BadCubLOL14:33
NobodyCamlol14:33
jlvillalNobodyCam, You're welcome.14:33
jlvillalNobodyCam, Hopefully some other people can try it out and see if there are any mistakes14:33
NobodyCam++14:33
BadCubbrb14:33
openstackgerritMerged stackforge/ironic-discoverd: Check hooks availability on start-up  https://review.openstack.org/17239414:34
openstackgerritMerged stackforge/ironic-discoverd: Delay failure from pre-processing hooks  https://review.openstack.org/17205114:39
*** achanda has quit IRC14:41
devanandamorning, all14:47
BadCubdevananda: morning14:47
NobodyCamgood morning devananda14:48
devanandawhat's the status of things?14:48
* devananda is just making first cup of coffee14:48
BadCubdevananda: hope you had a good long weekend! :-)14:48
NobodyCamdevananda: take a look at 17133414:48
devanandaBadCub: not long enough :p14:48
openstackgerritMerged stackforge/ironic-discoverd: Doublecheck node provision state on receiving data from ramdisk  https://review.openstack.org/17209414:48
BadCubdevananda: we still have two major patches to land before we can cut. Also two patches outside Ironic have not been landed.14:49
devanandaBadCub: I see a new bug: cleaning results in deadlocks14:50
BadCubdevananda: yes, ^ was discovered late on Friday14:50
*** MattMan has quit IRC14:51
openstackgerritBrad P. Crochet proposed openstack/python-ironicclient: Implementation of baremetal power on and off  https://review.openstack.org/17294614:52
pshigedevananda: morning :)14:54
*** rwsu has quit IRC14:54
devanandaBadCub: everything on https://etherpad.openstack.org/p/IronicReviewDay I presume?14:54
*** rwsu has joined #openstack-ironic14:54
dtantsurdevananda, morning14:54
lucasagomesdevananda, morning14:57
lucasagomesdevananda, back to US already?14:57
BadCubdevananda: correct ^ is up_to_date14:57
devanandaBadCub: cheers15:01
devanandalucasagomes: yup! already getting pinged by ttx about our RC status too15:01
lucasagomesdevananda, awesome!15:01
BadCubdevananda: :)15:01
devanandaanyone working on the client negotiation patch now? it looks like it needs a tiny bit more work this morning15:02
NobodyCamBadCub: https://review.openstack.org/#/c/168844 +a'd15:04
NobodyCamdevananda: jlvillal was working on it.15:04
NobodyCamjlvillal: are you still working on it?15:04
BadCubNobodyCam: awesomeness! TY :)15:04
BadCubbrb15:06
jlvillalNobodyCam, devananda: I can stop if someone else wants to work on it.15:06
jlvillalI'm getting ready for work at the moment15:06
devanandajlvillal: nope. just checking to see that someone is :)15:06
devanandajlvillal: I just posted two more comments15:07
*** jistr|mtg is now known as jistr15:07
jlvillaldevananda, Thanks.  I will work on it.  Hopefully have an updated patch in a bit.15:07
rloojlvillal: I, too, have just posted some comments. Morning jlvillal :-)15:07
jlvillalrloo, Good morning and thanks.15:07
*** jerryz has quit IRC15:07
rloohi lucasagomes, if you have a minute, question about your comment: https://review.openstack.org/#/c/171334/16//COMMIT_MSG15:09
lucasagomesrloo, sure15:09
rloolucasagomes: what did you mean? by "fall forward"?15:09
jlvillaldevananda, I think I can only do self._parse_version_headers() if I rename the function.  Because the functions have the same name, unless I'm missing something.15:09
rloolucasagomes: I think you mean there's a bug in ironic. nothing wrong with the patch, right?15:09
jlvillaldevananda, But dtantsur suggested I rename the function :)15:09
devanandajlvillal: exactly15:10
lucasagomesrloo, yeah, that's something we need to fix in Ironic to not return 40615:10
lucasagomesand be backward compat15:10
devanandajlvillal: also, you can use super(Class, self).function15:10
rloolucasagomes: something we should fix before rc1 i think?15:10
dtantsurjlvillal, yeah, because these are 2 different functions, not one overloaded15:10
jlvillaldevananda, Thanks15:10
devanandai can't type - that's the wrong method def. but the right idea15:10
lucasagomesrloo, that was what I was trying to point out when we were discussing that logical name patches remember?15:10
* devananda sips is coffee15:10
rloolucasagomes: yeah, I remember that. And don't know why it wasn't fixed. Didn't we agree it should be fixed?15:10
lucasagomesrloo, hmm for the logical name itself I think we cna fix it in rc1, that should be simple I suppose15:11
lucasagomeslemme look15:11
lucasagomesrloo, yeah, not sure what was the agreement. But yeah it def should be fixed15:11
*** stendulker has quit IRC15:12
devanandalucasagomes: i dont understand your comment there15:12
rloolucasagomes: can't find the etherpad from mrda, looking...15:12
lucasagomesdevananda, it's because 406 is not only returned when the version requested is > than what is supported15:13
lucasagomesit could be the other way around too15:13
devanandaI think I'm just not parsing "fall forward" yet15:13
lucasagomesso falling back to the minimum version will not help15:13
lucasagomesdevananda, so we maybe should not fall foward15:13
devanandaoh. you mean that an older client talking to a newer server may see new error codes?15:13
lucasagomesdevananda, yes, that's what I was pointing in the logical name patch before15:14
devanandaeg, if I dont upgrade my client, whereas I used to get a 400, I may start getting a 406 error?15:14
lucasagomesbasically before the microversion if you tried to update a node with a atrribute that didnt exist it would return 40015:14
lucasagomesnow with microversioning and name15:14
devanandai think that's just fine -- it informs the client of the change15:14
lucasagomesif you try to add name to an old version it returns 40615:14
lucasagomesimo it should return 400, as before15:14
lucasagomesto be backward compat15:14
devanandai disagree :)15:14
lucasagomesright15:15
lucasagomesthe idea of microversion is not make the api looks and feel like an old api>15:15
lucasagomes?*15:15
*** zz_jgrimm is now known as jgrimm15:15
devanandait is to allow the API to change while maintaining backwards compatibility with older clients, by allowing both client and server to negotiate the semantics of their exchange15:16
devanandawhereas a client previously got a 400 error for a particular request (which it then should not attempt to repeat)15:16
rloolucasagomes, devananda: found the etherpad: https://etherpad.openstack.org/p/ironic-microversion-handling15:17
lucasagomesright, so it's in our plan to implement a "roll forward"15:17
devanandaif that client should continue making the request with no version header, it will start working when the server is upgraded15:17
rloolucasagomes, devananda: see 1B. I think this is what you're discussing.15:17
devanandaif that client should contoinue making hte request with a version header, it will get a different error message, indicating the server is capable of a different exchange15:17
devanandain either case, we haven't broken the old client15:17
devanandayes, i've changed my mind on this as a result of spending more time implementing the client side of this and using microversions15:20
devanandathat said, I'd like to suggest we focus today on finalizing the RC and addressing those things which actually negatively affect a lot of users15:21
rloodevananda: ok. so to be clear (maybe I've lost track). with an old client + new server, no header is sent to the server. for lucas' example, the server defaults to 1.1, and will 406 cuz logical names aren't supported. whereas it would have 400'd with an old server.15:22
devanandarloo: right15:22
rloodevananda: ok. I"m going to update that etherpad and add a comment to lucas' so it is 'recorded'. then i'll focus on the rc1 stuff :-)15:23
lucasagomesack15:23
lucasagomesthanks rloo15:23
*** MattMan has joined #openstack-ironic15:23
devanandaI don't think this will affect our users nearly as much as, say, cleaning causing deadlocks, or the new client refusing to connect to any pre-1.6 server15:23
devanandabut I agree that it could, potentially, affect someone who was relying on the server failing with an HTTp 400 error15:24
lucasagomesyea15:24
lucasagomesagree with that, is not serious, but is a breakage15:24
devanandawhen they tried, repeatedly, to update a property which did not exist :)15:24
dtantsurwell, we used microversion to not break people who expect 400 from node-set-provision-state inspect15:25
devanandalucasagomes: but who does it affect? only someone who expected to attempt to update a specific non-existent property15:25
dtantsurwhile it's weird to me, we could continue gating literally everything15:25
lucasagomesdevananda, yeah, that's true... but that's one example... dtantsur is has another15:26
devanandadtantsur: I agree - we could - but the more we dig into this, the more I think that's overkill15:26
lucasagomestrying to find a node by it's name instead of uuid with an old server the same15:26
lucasagomesor old client*15:26
lucasagomesthe 406 is returned in other places which wasn't returned before15:27
lucasagomesbut I agree I don't see anything major here15:27
devananda:)15:28
lucasagomesand my idea of microversioning maybe was wrong, I thought it was be a layer to make the API feels like an old API15:28
lucasagomeswhich is not15:28
*** mtanino has joined #openstack-ironic15:32
devanandaJoshNang: i'm trying to understand https://bugs.launchpad.net/ironic/+bug/144281015:36
openstackLaunchpad bug 1442810 in Ironic "Cleaning results in deadlocks" [High,In progress] - Assigned to Josh Gachnang (joshnang)15:36
JoshNangdevananda: yeah, i'm frankly not clear on the root cause here.15:37
NobodyCamHe was hitting that whrn attempting to put 20+ node in to cleaning at one time15:37
NobodyCamoh morning JoshNang15:37
JoshNanggood morning :)15:38
openstackgerritBrad P. Crochet proposed openstack/python-ironicclient: Implementation of set and unset  https://review.openstack.org/17296315:38
devanandaJoshNang: you say "deadlock" - this has a very specific meaning in DB terms. do you, in fact, mean database deadlock? or something else?15:39
NobodyCamI understood it more as Hung or Stuck nodes?15:40
JoshNangyeah, i think that was a poor word choice. it was a stuck lock15:41
devanandaJoshNang: stuck waiting on what?15:41
devanandaJoshNang: converting from RPC call -> cast shouldn't change how or when a TaskManager lock is reelased15:42
JoshNangwhen it tried to acquire the lock in the conductor's continue_node_clean, it would almost always hit a nodelocked exception15:42
devanandanor should it change how or when the lock is acquired ...15:42
JoshNangright, it didn't seem sensible looking at the code15:42
lucasagomesJoshNang, after the error. The "resevation" field of the node was still set?15:43
lucasagomesdevananda, I think it doesn't... It just call cast and return. If it fails to acquire the lock it will try again on the next heartbeat15:43
JoshNanglucasagomes: yeah. later i manually cleared them but they came back the next time it got to that method15:43
lucasagomesso it will eventually work15:43
lucasagomesthat's why I think that the Closes-Bug tag there is wrong, cause it just work it around15:44
devanandalucasagomes: ah, i see15:44
lucasagomesit doesn't fix the problem with the locks15:44
lucasagomeswhich can be a race in between there15:44
devanandaidea!15:44
lucasagomesJoshNang, gotcha15:44
devanandaJoshNang: the continue-clean is passed by the agent on every heartbeat, right?15:44
JoshNangyeah, i think it has to be a race. some nodes finished cleaning. i got a handful of the 20 to finish15:44
JoshNangdevananda: yes15:44
devanandaJoshNang: so what if the conductor is in the middle of one cleaning operation when the agent sends a heartbeat?15:44
devanandaJoshNang: it's going to error. because it can't continue cleaning while it's already cleaning15:45
*** jistr has quit IRC15:45
JoshNanglet me look. i think it only continues when it sees the current command is complete, or no command has started (when the agent first boots)15:45
BadCubbrb15:47
JoshNangyeah, the rpc only gets called on success or clean_version_mismatch (which restarts it) https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/agent_base_vendor.py#L13915:47
lucasagomesyeah it will just return if the command still running and do nothing15:47
lucasagomeshttps://github.com/openstack/ironic/blob/master/ironic/drivers/modules/agent_base_vendor.py#L322-L32315:48
lucasagomeswhich goes here https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/agent_base_vendor.py#L155-L15715:48
devanandaconductorManager._do_next_clean_step starts off by changing node.clean_step and calling node.save()15:49
devanandaso it's going to mismatch on the next heartbeat, won't it?15:49
* lucasagomes diggs15:49
JoshNangright15:50
*** romcheg has quit IRC15:51
devanandayea, on any heartbeat, if node.provision_state == states.CLEANING and node.clean_step: it's going to try to start cleaning again15:51
devanandaeven if cleaning is already in progress15:52
lucasagomeshmm so on heartbeat, if the node is cleaning we should perhaps test the progress15:54
JoshNangno, it'll check the agent, see the clean_step is different than the last command on the agent, and stop15:54
lucasagomesif it's running just do nothing15:54
* lucasagomes still needs to read more the code15:54
lucasagomesoh yeah...15:54
JoshNangthe first thing it does if its CLEANING and has a clean step is run through checks for failed, running, not the right step, etc here: https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/agent_base_vendor.py#L30815:55
lucasagomeswe never call continue_cleaning from the api, it's only called from the agent_base_vendor15:55
devanandaJoshNang: shouldn't all this just check that the node isn't already locked by another thread and, if so, stop?15:56
JoshNangdevananda: doesn't vendor passthru grab the lock before starting any of this code?15:56
devanandaJoshNang: the heartbeat -> continue_cleaning() -> _notify_conductor_resume_clean path does not use an exclusive lock15:56
openstackgerritMerged openstack/ironic: iLO driver documentation for UEFI secure boot  https://review.openstack.org/16884415:56
devanandaJoshNang: /me double cehcks15:57
kbs1Anyone have a good install/config/setup guide for IPA somewhere? agent accesses to temp_urls are throwing authorization errors so I'd like to see if something basic in my setup is busted.15:57
*** igordcard_ has quit IRC15:57
*** mgoddard has quit IRC15:57
*** mgoddard1 has joined #openstack-ironic15:57
jlvillalrloo, Instead of doing an assert for the value not being one of the valid states, do you have a suggestion?  I could raise a ValueError?15:59
NobodyCamkbs1: are you using devstack?15:59
rloojlvillal: wondering what will be useful for the user.16:00
kbs1No. Using a pure bare metal deployment.16:00
JoshNangkbs1: our docs for that are definitely lacking there. maybe a mismatched swift user or temp url key? https://github.com/openstack/ironic/blob/master/etc/ironic/ironic.conf.sample#L75416:00
rloojlvillal: need to look at the code again. sec.16:00
jlvillalrloo, I'm not sure.  It would indicate a bug in the code.16:00
devanandaJoshNang: hm, yea. vendor_passthru should take an exclusive lock. driver_vendor_passthru does not, which is what I was thinking of16:00
rloojlvillal: internal error, go on irc and ask? :-)16:01
kbs1NobodyCam: Using a devstack setup as a cross-reference, though.16:01
jlvillalrloo, I'm just trying to be a little defensive to make sure we don't run along and have a huge error message if it goes recursive until the stack fills up16:01
rloojlvillal: can we default to something instead of asserting?16:02
NobodyCamkbs1: so you've seem http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html#deploying-ironic-with-devstack16:02
NobodyCamseen even16:02
rloojlvillal: or just leave the assert in there :-)16:02
jlvillalrloo, So try to cover up a bug in the code?16:02
jlvillalrloo, I vote for leave the assert there as it should only trigger if it is a bug in the code.16:03
*** dttocs has joined #openstack-ironic16:03
rloojlvillal: so in general, if code is being used by others, it just seems odd to have asserts. developers always say, "it won't trigger in production", but what if it does?...16:03
rloojlvillal: but it isn't a big deal cuz it won't trigger when the user is using it.16:04
jlvillalrloo, I would assume if it does trigger in product developers should be told because it is a bug.  I can change it to a an if statement and raise a ValueError() if that makes it more appealing.16:05
jlvillalrloo, I would prefer to not ignore the case though.16:05
rloojlvillal: it also seems to me that self.os_ironic_api_version should have a value too? and we aren't asserting on that?16:05
jlvillalrloo, Agreed.16:06
*** ifarkas has quit IRC16:07
rloojlvillal: this is the first assert being added to the client. it opens the door for more asserts. just want to make sure we think it through. it is a personal thing for me, that i don't like asserts in 'production' code.16:09
jlvillalrloo, Okay.  I will just change it to an if statement and raise an Exception.  Works out to the same thing :)16:10
kbs1NobodyCam: Yes. I've seen this.16:10
rloojlvillal: but with hopefully a more meaningful msg :-)16:10
jlvillalrloo, A rose by a different name.... :)16:10
rloojlvillal: think of the poor user or developer:-)16:11
jrollkbs1: might be worth checking for clock skew between ironic-conductor and swift16:11
jrolllucasagomes: I'm back now, you pinged?16:11
lucasagomesjroll, welcome back :-) yeah, trying to remember what was it16:11
lucasagomesah16:11
kbs1jroll: will do.16:12
lucasagomesjroll, do you guys have some downstream code that keeps the node powered on in case the deployment fail?16:12
kbs1JoshNang: will check that out as well.16:12
jrolllucasagomes: yeah, we do16:12
jrolllucasagomes: mostly just hacks, nothing interesting, but I can share if that's helpful16:12
lucasagomesjroll, right I put some code up for that so I can access the logs from IPA when it fails16:12
lucasagomesjroll, no it's ok16:12
JayFI thought it behaved that way upstream :(16:13
lucasagomesI was more wondering... what's the action in nova16:13
JayFcleaning/cleanfail works that way16:13
JoshNangyeah that's on my list of "need to upstream soonish"16:13
lucasagomesJayF, not really :-( I added a config option that so we can do it16:13
jrollJayF: I don't think anything like that is upstream16:13
jrolllucasagomes: what do you mean the action in nova16:13
lucasagomesjroll, JayF https://review.openstack.org/#/c/172932/16:13
JoshNangcleanfail definitely doesn't power off the node16:13
lucasagomesjroll, so see the Depends-On tag in that patch, it links to a patch in nova16:14
lucasagomeswhich I prevent the destroy() if the node is in maintenance16:14
lucasagomesI wonder if you do same, or you have a smarter way of doing it16:14
JayFoooh16:14
jrolllucasagomes: aha, I see, now that I think about it we do let nova kill it16:14
JayFour downstream doesn't behave that way16:14
JayF==jroll16:15
jrollso we don't leave it on (though we power on after destroy is done)16:15
lucasagomesright, /me is glad I asked16:15
lucasagomesbut if you power off you lose the logs16:15
lucasagomescause when the deploy fails, nova automatically calls destroy()16:15
jrollright, so in general we don't have issues that the ramdisk logs will help with16:15
*** saripurigopi has quit IRC16:15
jrollalmost never16:15
NobodyCammorning jroll JayF :)16:15
lucasagomesright16:16
JayFmorning16:16
* JayF not around long, will be on a plane in an hour16:16
lucasagomesjroll, I find it specially useful the logs to debug IPA itself when implementing a new extension or something16:16
NobodyCamJayF: where you off to this fine monday16:16
jrolllucasagomes: the error passed back to ironic is usually enough to tell, e.g. we had a bad load balancer in front of swift one day, was fairly obvious what the issue is16:16
lucasagomesI worked it around ismilar to what I put in the patch when doing the iscsi for ipa16:16
jrolllucasagomes: indeed, for dev it's useful, we haven't found it useful in prod16:16
lucasagomesso I could see what went wrong in IPA16:16
JayFNobodyCam: training at our home office in Texas. Part of our "Emerging Leaders" program16:17
jrolllucasagomes: I'm +1 on the idea of these patches fwiw16:17
lucasagomesjroll, ack JayF16:17
lucasagomesyeah I want it to also be useful for you guys downstream16:17
devanandaJoshNang: your change from call -> cast assumes that no other driver will need a response from continue_node_clean()16:17
lucasagomesperhaps we should do differently on the nova side16:17
jrolllucasagomes: yeah, dunno if we'd use it in prod, but we might16:17
lucasagomesbut maybe we have different use cases for that16:17
lucasagomesok16:18
lucasagomescheers for the infos/ideas16:18
NobodyCamJayF: awesome! enjoy!16:18
jrolllucasagomes: quickly looking, both patches seem sane :)16:18
JoshNangdevananda: correct, but that conductor function isn't returning anything ATM16:18
lucasagomesyeah I've tested it here locally16:18
lucasagomesit's simple, I default it to continue doing what it does today16:19
devanandaJoshNang: except exceptions :)16:19
lucasagomesso it won't break anything16:19
JoshNangdevananda: :) true16:19
lucasagomesit's more to help me and others with debugging IPA when needed16:19
JayFI bet it'd be interesting to run production that way16:19
JayFwhat I'd love to see is a way to say "was the falure ironic or external" and decide to leave on / destroy accordingly16:20
devanandaJoshNang: anyway, this change is just ignoring the errors that continue_node_clean() may o rmay not raise, but I agree that there is absolutely no need for the calling thread to wait on the receiving thread in the same conductor in this case16:20
JayFfor instnace, we have failures to neutron occassionally. There's no value in leaving that Agent up because Neutron did everything right, but it's a failure all the same16:20
lucasagomesJayF, oh right, yeah right now it just do for any failures. It would be interesting indeed to maybe classify how it failed and act upon that16:21
JoshNangdevananda: right. for errors you get a vague 'expected exception' log message, and because the heartbeats keep coming, it'll try again later. certainly not ideal16:21
devanandaJoshNang: so I'm good with the change. I'm still not sold on the whole driver-sends-rcp-to-the-conductor approach, though16:21
devanandaJoshNang: mind if I clean it up a bit?16:22
JoshNangdevananda: go ahead!16:22
lucasagomesJayF, I can try to take a look at it. The idea here now was just a "hey, deployment fails. You still have all logs so go and figure out"16:22
JayFlucasagomes: just a thought; you doing what you're doing is better than what we have now :)16:23
JoshNangagreed, but it was the least bad way i had come up with for getting back to the conductor from the driver in an async fashion. i'd love for a better solution16:23
*** a1exhughe5 has quit IRC16:23
JayFlucasagomes: just specifically finding agent failures and keeping the agent  on would be something I'd be +1 to running in my prod16:23
lucasagomesnice, thanks for the feedback. Yeah I will take a look at it16:23
*** Sukhdev has joined #openstack-ironic16:24
JoshNangJayF: there's a spec up to send the agent logs back on heartbeats and after deployments (presumably failures as well).16:25
*** lsmola_ has quit IRC16:25
openstackgerritJohn L. Villalovos proposed openstack/python-ironicclient: Client should fall back to the lower versions if necessary  https://review.openstack.org/17133416:32
jlvillalrloo, devananda ^^^^16:32
jlvillalAnd anyone else who would like to review :)16:32
JayFJoshNang: I saw that, and am currently pondering it16:32
devanandaJoshNang: yanno what, nvm. leaving comments on the review. I think this needs one more pass16:34
*** ukalifon1 has quit IRC16:34
JoshNangdevananda: kk16:34
openstackgerritChris Krelle proposed openstack/ironic: Update ilo drivers documentation for inspection  https://review.openstack.org/17006516:37
devanandaJoshNang: posted16:40
JoshNangdevananda: thanks16:40
devanandaJoshNang: the change i just suggested breaks 2 unit tests16:41
openstackgerritJohn L. Villalovos proposed openstack/python-ironicclient: Client should fall back to the lower versions if necessary  https://review.openstack.org/17133416:41
*** subscope_ has joined #openstack-ironic16:41
jlvillal^^^^ very minor change to make sure test both 'user' and 'negotiated' failure paths.16:41
JoshNangdevananda: is partial-bug the better tag here? not sure if there's a 'workaround' tag16:42
JoshNangor related-bug16:43
* NobodyCam thinks partial-bug16:43
JoshNangwfm16:43
jlvillalJoshNang, https://wiki.openstack.org/wiki/GitCommitMessages   There is 'Related-Bug'16:43
jlvillalJoshNang, I think you listed all the options they talk about16:44
JoshNangjlvillal: thanks16:44
*** MattMan has left #openstack-ironic16:47
openstackgerritDevananda van der Veen proposed openstack/ironic: Open Liberty development  https://review.openstack.org/17127416:48
devanandaboth related and partial will leave the bug open16:48
devanandaif we're saying "this bug is fixed in kilo" we should actually use closes-bug16:48
devananda(this is me thinking through it) the bug is "cleaning results in deadlocks". assuming this patch results in no more deadlocks, then it closes it.16:49
devanandaJoshNang: ^16:50
lucasagomesdevananda, oh good catch on the RPC_VERSION bump16:51
*** ramineni has joined #openstack-ironic16:51
* lucasagomes should have catched it16:51
JoshNangalright, i'll stick with closes.16:52
devanandaJoshNang: also, would the bug description be better as "cleaning may leave nodes locked and require manual intervention to unlock" ?16:53
JoshNangyes, definitely16:54
* devananda edits16:54
*** stendulker has joined #openstack-ironic16:55
devanandaanyone tested https://review.openstack.org/#/c/171743/ ?16:56
devanandathat's the DIB patch for https://bugs.launchpad.net/ironic/+bug/144155616:56
openstackLaunchpad bug 1441556 in diskimage-builder "Deploying whole disk image with local boot fails" [Medium,Triaged] - Assigned to Lucas Alvares Gomes (lucasagomes)16:56
* lucasagomes looks16:57
*** derekh has quit IRC16:57
lucasagomesoh perhaps greg should be assigned to that bug16:57
lucasagomeshe did the workd16:57
*** dtantsur is now known as dtantsur|afk16:57
devanandaI see a comment from NobodyCam that it needs to be tested, but no furhter comments from any of us that it *was* tested16:58
devanandawhich seems to be why the DIB team hasn't approved it yet (and reasonably so)16:58
lucasagomesI can try give it a go16:59
lucasagomeslemme see if I have an env handy16:59
NobodyCamoh the dib patch. I was going to see if I could fix my broken env today to test that16:59
NobodyCamlucasagomes: that would be awesome@16:59
NobodyCams/@/!/16:59
*** stendulker has quit IRC16:59
rloojlvillal: thx, see comments.17:03
*** Marga_ has quit IRC17:04
*** Marga_ has joined #openstack-ironic17:05
jlvillalrloo, Thanks.  Yeah I'm not exactly sure what to do for the wording.  I will think.... :)17:05
*** Marga_ has quit IRC17:05
*** Marga_ has joined #openstack-ironic17:06
jlvillalrloo, I like your suggestion17:07
*** subscope_ has quit IRC17:08
NobodyCambrb17:09
lucasagomesNobodyCam, devananda testing that local boot patch for DIB17:11
openstackgerritAnusha Ramineni proposed openstack/ironic: iLO driver documentation for node cleaning  https://review.openstack.org/16916217:11
openstackgerritJosh Gachnang proposed openstack/ironic: Convert internal RPC continue_node_cleaning to a "cast"  https://review.openstack.org/17258217:13
rloojlvillal: the 'negotiated version' stuff may not be clear. I wonder if we should mention something like 'no API version was specified and the version used by the client didn't work.'17:14
devanandalucasagomes: cheers17:18
*** achanda has joined #openstack-ironic17:18
devanandajlvillal: yea, the wording is really close but could be a bit better17:18
*** davideagnello has joined #openstack-ironic17:18
devanandaother than that, this looks good. /me tests locally17:19
*** ramineni has quit IRC17:20
devanandajlvillal: also, the message is much longer than my command line ... perhaps wrapping it would help :)17:20
BadCubanyone have a moment to look at: https://review.openstack.org/#/c/170065/ Needs +2/+A17:21
jlvillaldevananda, Okay :)17:21
*** edcragg has quit IRC17:21
BadCubalso - https://review.openstack.org/#/c/169162/ - ILO Driver doc for node cleaning Need to have +2s/+A reapplied. Was rebased17:21
* BadCub kicks etherpad17:24
NobodyCamBadCub: https://review.openstack.org/#/c/169162 re +a'd17:25
BadCubNobodyCam: ty :-)17:25
*** jcoufal has quit IRC17:26
*** zhenguo has quit IRC17:27
NobodyCambrb17:28
openstackgerritJohn L. Villalovos proposed openstack/python-ironicclient: Client should fall back to the lower versions if necessary  https://review.openstack.org/17133417:32
jlvillalrloo, devananda: Feed back on the messages appreciated.  Thanks.17:32
rlooNobodyCam: wrt 170065, are we going to change things so that python-ironicclient has a kilo version?17:33
rlooNobodyCam: vs version 0.4 or whatever it is.17:33
*** Marga_ has quit IRC17:33
*** meghal has joined #openstack-ironic17:34
*** harlowja_away is now known as harlowja17:37
lucasagomesdevananda, http://paste.openstack.org/show/203666/17:40
*** Marga_ has joined #openstack-ironic17:40
lucasagomesdoens't seem to work, node is stuck in the deploywait even when it's finished the bootloader installation17:40
lucasagomesand the pass_bootloader_install_info is present at methods17:40
*** trown is now known as trown|lunch17:41
* lucasagomes will test without the change see if it works fine (could be his env)17:42
*** Marga_ has quit IRC17:44
lucasagomesdevananda, apparently that patch is not passing any credentials to the curl command17:45
NobodyCamrloo: let me take a look. I just addressed comments in the last rev of that review17:45
devanandalucasagomes: huh. good catch17:45
rlooNobodyCam: sorry, I didn't check to see who added what in that patch. Might not be you.17:48
NobodyCam:)17:49
*** r-daneel has joined #openstack-ironic17:51
*** athomas has quit IRC17:55
devanandajlvillal: LGTM17:56
devanandalucasagomes: you working on the DIB patch?17:57
lucasagomesdevananda, I'm just testing it w/o to see if it's my env (but I doubt)17:57
lucasagomesit's late here :/17:57
lucasagomesunless nobody else can do it I will17:57
devanandalucasagomes: ah, of course17:57
BadCubanyone want to look at https://review.openstack.org/#/c/172582/  & https://review.openstack.org/#/c/171334/ ?17:58
lucasagomesdo DIB has a cut?17:58
*** mgagne_ is now known as mgagne17:58
lucasagomescause since it's not in the Ironic code databse should we wait on it ?17:58
devanandaDIB isn't going to do a stable branch, as far as I know17:58
lucasagomesright, so we could cut our RC-1 without relying on that patch right?17:58
devanandabut if we can land a fix in DIB, at least we can document in our release notes that this bug is fixed with DIB after some point in time17:59
jlvillaldevananda, I like your plan :)17:59
lucasagomesright17:59
devanandacorrect me if i'm wrong, but without fixing this in DIB, we have to say that it is currently not supported in Kilo17:59
lucasagomesyeah that would be nice :/17:59
devanandaat this point, i'm really tempted to just do that17:59
lucasagomes| f30027a7-4e1a-4d70-b4f8-9b5619714a80 | testing | ACTIVE | -          | Running     | private=10.1.0.13 |18:00
lucasagomesit works without the patch18:00
lucasagomesdevananda, well it's only supported on Kilo18:00
*** pelix has quit IRC18:00
jlvillalrloo, I'm used to two spaces after a period.  But I have a co-worker of mine who hates that ;)18:00
*** mgoddard1 has quit IRC18:00
lucasagomesdevananda, there's 2 problems here, if we use an old ramdisk with new server18:00
lucasagomesit won't call the pass_bootloader_info etc... so Ironic will get stuck on DEPLOYWAIT waiting for it18:01
jrolljlvillal: the never-ending battle18:01
rloojlvillal: not having met you, my guess is that you're older than the younger folks ;)18:01
lucasagomesthat patch is trying to fix the old server (which doesn't contain the pass_bootloader_info method) with a new ramdisk18:01
rloojlvillal: so it shouldn't matter, if something is pretty-fying your text. I don't really care, just thought I'd point it out for consistency blah.18:02
*** romcheg has joined #openstack-ironic18:03
lucasagomesdevananda, for the problem #1 I think we talked about passing a flag as part of the pass_deploy_info, and if not set Ironic would not wait on pass_bootloader_info18:03
devanandalucasagomes: right. we've introduced a new feature which breaks deployments if you try to use the new feature after upgrading, without ALSO updating the dib ramdisk.18:03
rloojlvillal: why, back in the good old days when I used a typewriter...18:03
lucasagomesbut apparently nobody did that18:03
lucasagomesdevananda, exactly18:03
devanandalucasagomes: there was not enough thought put into the upgrade path for people, and as such, we're trying (still) to work around that with some very late fixes18:03
lucasagomesso we can put in our docs, that for local boot it also requires updating the deploy ramdisk18:03
jrolluh18:04
jrolllucasagomes: devananda: I thought we fixed problem #118:04
jrollold ramdisk with new server18:04
* lucasagomes looks at the code18:04
jrollnever mind me, I'm thinking of whole dism image18:04
jlvillalrloo, I am older than the younger folks :)18:04
devanandajroll: right. whole disk image + netboot landed18:04
jrollyeah18:04
rloojlvillal: :D18:05
jrollignore me, too many things on my mind :)18:05
NobodyCamrloo: looking at your comemnt in ilo.rst on 170065, are you looking for a break down of essential vs extra there?18:05
jlvillalrloo, And I actually did take a typewriter class in high school :)18:05
devanandajroll: this is the ramdisk waiting or not waiting on an extra callback from ironic18:05
jrollright.18:05
jlvillalWhich helped me type faster on my computer...18:05
devanandawe found a way to detect that in newer ramdisks18:05
jrolldevananda: I typed before I thought, really, ignore me18:05
devanandajroll: :)18:05
jroll:)18:05
lucasagomesyeah what devananda said :-)18:05
lucasagomesthe whole_disk_image is fixed yeah18:06
jlvillalrloo, I took the typewriter class while I owned a computer.  TRS-8018:06
rlooNobodyCam: from the user's point of view, what is the difference, what does it matter?18:06
lucasagomesdevananda, so, maybe document that to use local boot we require also to update the deploy ramdisk with a new version18:06
lucasagomesif used with DIB18:06
NobodyCamrloo: ack... let me see what I can comeup with18:06
lucasagomeswith DIB ramdisk*18:06
rloojlvillal: TRS-80 rings a bell ;)18:06
devanandalucasagomes: does that obviate the need for this patch?18:07
NobodyCammodel i ii or III18:07
jlvillalrloo, NobodyCam: I started with the Model I18:07
rlooNobodyCam: not a big deal. It can be added later. but I thought there was some diff, that eg essential were/had to be all discovered.18:07
NobodyCam:) nice18:07
lucasagomesdevananda, well that patch only solves the old server with new ramdisk18:07
devanandalucasagomes: no ... i think this patch is making the newer DIB ramdisk compatible with older ironic services18:07
devanandaright18:07
devanandaso we still need that18:07
devanandabut it doesn' tneed to block our RC118:08
lucasagomesyeah, exactly18:08
NobodyCamjlvillal: remember https://www.youtube.com/watch?v=FQizYzw27FY18:08
lucasagomesit would be nice to have the other bits old ramdisk with new server in place... but seems late18:08
lucasagomesthat will also require another change in DIB to set that flag and so on :/18:09
*** Marga_ has joined #openstack-ironic18:09
jlvillalNobodyCam, I remember LDOS :)18:09
NobodyCamkeep watching18:09
lucasagomesbut overall I think it's ok... local boot is a Kilo feature18:09
lucasagomesso requiring a new DIB to work with it doesn't seems crazy18:09
lucasagomesit's not awesome, but not bad either18:09
jlvillalNobodyCam, Oh yeah I remember the dancing demon :D18:10
* lucasagomes we should try to be awesome...18:10
NobodyCamlol18:10
jrolllucasagomes: ++18:10
jrollif you're using juno-era ramdisk, you need to update18:10
jrollso it doesn't seem all that terrible18:11
lucasagomesyes18:11
lucasagomesto use that specific feature18:11
jrollright18:11
lucasagomesit's not that we are breaking deployments or anything18:11
lucasagomesonly if you want to use local boot18:11
jrollyeah18:11
lucasagomesright, so I will relax a bit if people agree on that18:12
lucasagomesand get a break18:12
devanandalucasagomes: requiring a new DIB to use the new feature is fine. requiring a new DIB just because I upgraded to Kilo is not fine, and breaking deploys if I choose t ouse a newer ramdisk and NOT update my service -- also not fine.18:12
*** ijw has joined #openstack-ironic18:12
lucasagomesdevananda, right yeah we don't do that18:12
devanandalucasagomes: well, without this DIB patch, we have broken deployments of stable/juno, right?18:12
*** Marga_ has quit IRC18:12
devanandabecause the new ramdisk is going to expect a second callback tht will never come18:13
lucasagomesdevananda, no we don't, because we check if we passed the boot_option kernel cmdline in the ramdisk18:13
*** Marga_ has joined #openstack-ironic18:13
devanandaoh18:13
lucasagomesand old servers won't do that18:13
devanandayea18:13
lucasagomesand for new servers we check if boot_option = local before waiting for the call back18:13
*** Marga_ has quit IRC18:13
devanandaright. too many different race conditions in my brain right now :-/18:13
lucasagomesno problem18:13
lucasagomesso it's not terrible18:13
*** Marga_ has joined #openstack-ironic18:13
lucasagomesthere's room for making it awesome, but we failed on that :-(18:14
devanandaok - so lets recap - what are the release blockers at this point18:15
openstackgerritMerged openstack/python-ironicclient: Upgrade hacking to latest release  https://review.openstack.org/17207218:15
devanandahttps://bugs.launchpad.net/ironic/+bug/144281018:15
openstackLaunchpad bug 1442810 in Ironic "Cleaning may leave nodes locked and require manual intervention to unlock" [High,In progress] - Assigned to Josh Gachnang (joshnang)18:15
BadCubhttps://review.openstack.org/#/c/171334/18:16
devanandahttps://review.openstack.org/#/c/172582/  << fix cleaning18:16
devanandaI think ^ is the only server-side patch that's blocking the RC18:16
openstackgerritChris Krelle proposed openstack/ironic: Update ilo drivers documentation for inspection  https://review.openstack.org/17006518:16
NobodyCamrloo: take a look and let me know what you think18:17
rlooBadCub: 171334 doesn't block rc1, does it? that's the client.18:17
devananda171334 is a client side change which we need to unbreak stable/juno, and I'll handle getting that into stable/kilo after RC118:17
rlooNobodyCam: looking...18:17
devanandabut that doesn't block our RC118:17
devanandarloo: you're correct18:17
BadCubdevananda: that works18:17
BadCubso: https://review.openstack.org/#/c/172582/ should be our only blocker for rc1 release18:18
devanandayah18:19
BadCubunless the Tempest patch is an issue as well?18:19
BadCubbut shouldn;t be18:19
openstackgerritDevananda van der Veen proposed openstack/ironic: Open Liberty development  https://review.openstack.org/17127418:19
devanandaBadCub: does not block our RC, but again, something we need to (work with tempest team to) land18:20
BadCubdevananda: pinged them up several times last week :(18:21
devanandai dont see any feedback on the review from them18:21
BadCubnope18:21
openstackgerritLucas Alvares Gomes proposed openstack/ironic: Local boot note about updated deploy ramdisk  https://review.openstack.org/17302918:22
lucasagomesdevananda, jroll ^18:22
* lucasagomes not sure if that's the best way to word it18:22
jrollwoot18:22
lucasagomesfeel free to update the patch18:22
*** Sukhdev has quit IRC18:23
jrolllucasagomes: maybe "since 2015/04/01" or something like that?18:23
lucasagomesjroll, "... in the kilo-era (since 2015/04/01)"  ?18:24
BadCubbrb18:24
jrolllucasagomes: yeah, something like that, don't know the actual date18:25
devanandadates are not that useful. how about pointing at a commit SHA ?18:25
devanandaor a release #18:25
lucasagomesthat was the patch https://review.openstack.org/#/c/169419/18:25
devanandadib does releases to PIP18:25
*** coolsvap|afk is now known as coolsvap18:26
*** trown|lunch is now known as trown18:26
lucasagomesthat's in ironic18:26
lucasagomeshmm in dib is18:26
devanandaso I think  it's reasonable to say "built with dib >= x.y.z" or some such18:26
lucasagomeshttps://review.openstack.org/#/c/169354/18:26
* lucasagomes checks last dib release18:27
lucasagomesok the last release seems to incoorporate this changes18:27
lucasagomeshttps://pypi.python.org/pypi/diskimage-builder18:27
openstackgerritAlex Weeks proposed openstack/ironic-specs: Add implement-rescue-mode spec  https://review.openstack.org/17187818:27
lucasagomes2015-04-0918:27
* lucasagomes updates the patch18:27
jrollnice18:28
devanandaJoshNang: only issue with https://review.openstack.org/#/c/172582/2/ironic/tests/conductor/test_manager.py,cm is that these are going to be logged as exceptions18:29
devanandaJoshNang: given thta you seemed to be hitting those locks quite often when this was call(), I would imagine you'll start getting a lot of exceptions logged now18:29
aweeksjroll: JoshNang: I think I've addressed the comments on this spec https://review.openstack.org/#/c/171878/18:29
openstackgerritLucas Alvares Gomes proposed openstack/ironic: Local boot note about updated deploy ramdisk  https://review.openstack.org/17302918:29
lucasagomeshere we go18:29
devanandaJoshNang: even though they're being ignored and mostly benign, becaues the agent just retries18:29
lucasagomesI will take a break, have a good night everyone!18:29
devanandalucasagomes: cheers, gnight!18:30
lucasagomesdevananda, jroll feel free to update that patch if needed18:30
JoshNangdevananda: actually, i saw almost no lock conflicts with that change18:30
*** lucasagomes is now known as lucas-dinner18:30
devanandaJoshNang: interesting18:30
jrollaweeks: lucas-dinner awesome, thanks18:30
devanandaJoshNang: i'm really surprised. i wouldn't expect that to change the lock behavior *at all*18:31
JoshNangdevananda: right. i think there's something deeper wrong here that i couldn't find18:32
*** ijw_ has joined #openstack-ironic18:32
devanandaJoshNang: yea.... :-/18:32
* devananda +2's the change18:32
* NobodyCam +2 but hold off on +A so others can also look18:33
rloodoes anyone else want to look at 172582? Otherwise I'll +2.18:33
rlooah, NobodyCam beat me to it. I'm going to +A then.18:34
NobodyCamrloo: ++18:34
BadCubawesome18:34
rlooso we good now, for RC1? (After that merges of course.)18:34
BadCubrloo: we should be :-)18:34
*** ijw has quit IRC18:34
NobodyCamlol: http://www.merriam-webster.com/top-ten-lists/top-10-rare-and-amusing-insults-vol-2/loblolly.html18:35
devanandarloo: I believe so, yes18:35
devanandawe still need to land the client fix ASAP18:35
*** dttocs has quit IRC18:36
devanandaand I need to sort out how we're supposed to handle stable branches of the client in the next day or two18:36
*** jxiaobin has joined #openstack-ironic18:36
devananda(that fix needs to be on it)18:36
*** coolsvap is now known as coolsvap|afk18:36
rloodevananda: great. I've already +2'd the client. Need one more...18:36
devanandai held off on +2'ing my own code ... though at this point jlvillal has rewritten most of it :)18:36
* NobodyCam I'll look now18:37
rloodevananda: I dare you to +2 it!18:37
BadCublol18:37
jlvillalI enjoyed working on it :)18:37
* BadCub waves the shiny button at devananda18:38
*** Marga_ has quit IRC18:38
*** Marga_ has joined #openstack-ironic18:38
devanandarloo: done18:38
rloodevananda: ha ha. awesome. thx!18:39
* NobodyCam stops reviewing18:39
BadCubawesome18:39
*** dprince has quit IRC18:43
*** dttocs has joined #openstack-ironic18:44
*** meghal has quit IRC18:44
*** dttocs has quit IRC18:52
*** dttocs has joined #openstack-ironic18:53
NobodyCambrb18:55
*** dprince has joined #openstack-ironic18:59
openstackgerritChris Krelle proposed openstack/ironic: Update ilo drivers documentation for inspection  https://review.openstack.org/17006519:05
*** ijw has joined #openstack-ironic19:07
*** meghal has joined #openstack-ironic19:07
*** ijw_ has quit IRC19:09
openstackgerritJim Mankovich proposed openstack/ironic-specs: Send iLO health metrics to ceilometer  https://review.openstack.org/12737819:13
*** jmank has joined #openstack-ironic19:13
openstackgerritVictor Lowther proposed openstack/ironic: Fix drac implementetation of set_boot_device.  https://review.openstack.org/17304719:19
openstackgerritMerged openstack/ironic: Fix type of value in error middleware response header  https://review.openstack.org/17154419:21
*** coolsvap|afk is now known as coolsvap19:21
devanandaodd - https://review.openstack.org/#/c/17203619:25
NobodyCamwhats the error there?19:27
devanandaahhh shit ...19:28
*** Sukhdev has joined #openstack-ironic19:28
* devananda digs more19:28
NobodyCamhttp://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/logs/screen-n-cpu.txt.gz#_2015-04-13_18_37_10_125 ???19:29
NobodyCamthis looks strange ... thou I do not beleieve it is releated to the failure: http://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/logs/ironic-bm-logs/baremetalbrbm_0_console.txt.gz19:33
*** kkoski has quit IRC19:34
devanandait looks like tempest failed to delete after successfully deleting a nova instance19:37
devanandabut perhaps i'm reading something wrong19:37
NobodyCamdevananda: here : http://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/logs/screen-n-api.txt.gz#_2015-04-13_18_56_20_40519:38
devanandathere *does* appear to be an odd interaction around cleaning. or perhaps just one i haven't seen before19:38
devanandayea, that19:38
NobodyCamya19:38
NobodyCamwas looking at that19:38
devanandathat delete was being retried internally by Nova at the time19:39
devanandaactually nvm, not at that time19:39
jrollwell, doesn't tempest try to delete, then poll the instance to watch for it disappearing?19:39
devanandahttp://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/logs/screen-n-cpu.txt.gz?level=INFO#_2015-04-13_18_51_15_16019:39
jroll404 at the end seems fine19:39
*** bradjones has quit IRC19:40
devanandahttp://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/console.html#_2015-04-13_18_56_21_74319:41
rloois it possible it timed out cuz cleaning took too long? although there wasn't anything to clean, right?19:42
NobodyCamthe patch to turn off disk eraseing landed no?19:43
JoshNangit did19:43
JoshNangit took 5 mins for the agent to boot and heartbeat19:43
JoshNanghttp://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/logs/screen-ir-cond.txt.gz#_2015-04-13_18_56_22_77919:43
JoshNangpowered on at http://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/logs/screen-ir-cond.txt.gz#_2015-04-13_18_51_39_23019:43
devanandayea -- http://logs.openstack.org/36/172036/2/gate/gate-tempest-dsvm-ironic-agent_ssh/14006ac/logs/screen-ir-cond.txt.gz#_2015-04-13_18_56_22_77919:43
devanandawow :(19:43
devanandaduring that time tempest is still waiting for the node to free up, as it should19:44
BadCubick19:44
NobodyCamso is that just more of: https://bugs.launchpad.net/openstack-ci/+bug/131106619:46
openstackLaunchpad bug 1311066 in OpenStack-Gate "Some nodes allocated in node pool are very very slow" [High,Confirmed]19:46
openstackgerritMerged openstack/python-ironicclient: Client should fall back to the lower versions if necessary  https://review.openstack.org/17133419:48
NobodyCamwoo hoo19:48
*** coolsvap is now known as coolsvap|afk19:48
BadCubyeah19:49
BadCubthe last one approved, first one merged lol19:49
devanandahttp://paste.openstack.org/show/203712/19:49
NobodyCamwow lines #6 and #9 show the same amount of time19:51
NobodyCamalmost19:52
devanandaso our current tempest test goes through 3 PXE boot cycles, each of which takes ~ 5 minutes19:53
devanandabefore cleaning, it only did 2 cycles19:53
NobodyCamyep19:53
devanandadid anyone bump the timeout in tempest test for the delete?19:54
NobodyCamI though JoshNang put up a patch to change the time for state transations in genrenal19:54
JoshNangyeah, sec, looking for the patch19:54
devanandait's still 6019:55
JoshNanghttps://review.openstack.org/#/c/168175/19:55
JoshNangah, this is the unprovision timeout19:55
devanandayes19:55
*** Marga_ has quit IRC19:56
devanandahttp://git.openstack.org/cgit/openstack/tempest/tree/tempest/config.py#n104719:56
*** achanda has quit IRC19:57
*** coolsvap has joined #openstack-ironic19:57
NobodyCamJoshNang: are you tossing up a patch for that?19:57
devanandaI got it19:57
NobodyCamTY devananda19:57
NobodyCam:)19:58
JoshNangis that not what 168175 is doing?19:58
openstackgerritMerged openstack/ironic: iLO driver documentation for node cleaning  https://review.openstack.org/16916219:58
NobodyCamJoshNang: humm: https://review.openstack.org/#/c/168175/5/lib/tempest20:00
NobodyCammaybe 300 is not enough? rev four had it at 60020:01
NobodyCamdevananda: JoshNang: https://review.openstack.org/#/c/168175/4..5/lib/tempest20:02
devanandaJoshNang: ah, you're right - that is raising the value20:02
JoshNangi put it at 600 because i had no idea how long it would take20:02
openstackgerritayoung proposed openstack/ironic: Fix drac implementation of set_boot_device.  https://review.openstack.org/17304720:02
JoshNangi thought 10 mins seemed ridiculous...but maybe not :(20:02
* NobodyCam has worked on some hardware that took > 10 minutes to post :-p20:04
devanandaapparently not. in this gate job, cleaning didn't start until 5:03 after the DELETE call was finished20:04
devananda2015-04-13 18:56:21.743 |     Details: Timed out waiting for node 03794d6d-47af-4f3e-8932-67be02bce9ed to reach provision_state state(s) [None]20:04
devananda2015-04-13 18:56:22.779 16346 INFO ironic.conductor.manager [-] Executing cleaning on node 03794d6d-47af-4f3e-8932-67be02bce9ed, remaining steps: []20:05
devanandashould we raise all the timeouts from 5 to 10 minutes?20:07
devanandathis one just happened to take a tiny bit longer than most ...20:07
NobodyCamthats prob the safe thing to do.20:08
JoshNang++, it was definitely quite close to finishing.20:09
rloothat test didn't just start timing out today though?20:09
devanandarloo: i have not checked logstash20:09
JoshNangits 3-4ish minutes usually, and the timeout is 520:09
NobodyCamcould it be load on the infa vm's? didn't most projects open L today20:09
devanandaNobodyCam: those are just rax / hpcloud instances ...20:10
rlooi've seen a few 'recheck' comments in some patches, but I didn't look to see why.20:10
*** lucas-dinner has quit IRC20:10
NobodyCamahh20:10
* devananda needs to step away soon and get lunch20:11
NobodyCammmmm food20:11
openstackgerritMerged openstack/ironic: Convert internal RPC continue_node_cleaning to a "cast"  https://review.openstack.org/17258220:11
BadCubfood sounds yummy20:11
* NobodyCam should go pick up his glasses20:12
BadCublooks like we are good for rc1 :)20:12
NobodyCamnice!20:12
devanandarloo: looks like only one hit in the last 7d for timed out waiting on delete20:13
NobodyCamawesome woke EVERYONE!20:13
BadCub^ was the last pending20:13
NobodyCamwork even :-p20:13
JoshNang\o/20:13
devanandahowever, we've got 16 hits for ironic jobs failing due to timeout waiting to reach some state or other, in the last 7 days20:13
rloodevananda: good to know. thx.20:13
rloodevananda: so reaching some state that isn't due to delete :-(20:14
rlooI thought there was a bug about timeouts.20:15
devanandarloo: however, some of those logstash entries are probably other errors masquerading as timeouts20:15
devanandai picked one at random -- it's actually a failure in a change in DIB. ironic never got the callback, sothe deploy timed out20:15
rloohttps://bugs.launchpad.net/ironic/+bug/140806720:16
openstackLaunchpad bug 1408067 in Ironic "ironic jobs sometimes fail with node callback timeout" [Medium,Triaged]20:16
devanandawell ... that actually could the the same thing, reallly20:16
devanandayea20:16
devanandathings like this -- http://logs.openstack.org/15/172715/2/check/check-tempest-dsvm-ironic-pxe_ssh/efef2e0/logs/ironic-bm-logs/baremetalbrbm_0_console.txt.gz20:16
devanandaanyone create an e-r query for that?20:18
rloohmm, i wonder if this is also related: https://bugs.launchpad.net/ironic/+bug/144100720:19
openstackLaunchpad bug 1441007 in devstack "Gate failure: Timed out waiting for Nova hypervisor-stats count >= 1" [Undecided,In progress]20:19
devanandarloo: no, i think that's related to the order that services are started in devstack20:20
*** kkoski has joined #openstack-ironic20:20
devanandaadam_g was working on something there last week IIRC20:20
devanandaif ironic-api is started too late, nova-compute will bail. has nothing to do with deploys timing out, if it's the bug I'm thinking of20:21
* devananda goes in search of lunch20:21
*** Marga_ has joined #openstack-ironic20:24
openstackgerritChris Krelle proposed openstack/ironic: Update ilo drivers documentation for inspection  https://review.openstack.org/17006520:24
*** achanda has joined #openstack-ironic20:25
NobodyCamrloo: and that one will render correctlly :-p20:26
openstackgerritVictor Lowther proposed openstack/ironic: Fix drac implementation of set_boot_device.  https://review.openstack.org/17304720:27
devanandacommented on https://bugs.launchpad.net/ironic/+bug/1408067 and changed to High20:27
openstackLaunchpad bug 1408067 in Ironic "ironic jobs sometimes fail with node callback timeout" [High,Triaged]20:27
devanandaBadCub: ^ something to start tracking, given that I've seen 16 failures matching this in 7 days ....20:28
* devananda really lunches now20:30
NobodyCam:)20:30
NobodyCamgo eat!20:30
*** kozhukalov has quit IRC20:30
*** jgrimm is now known as zz_jgrimm20:33
*** HenryG has quit IRC20:33
rlooNobodyCam: ok, i'll wait for it to render first, before looking ;)20:35
NobodyCamrloo: ++ :) Thank you20:35
BadCubdevananda: got it20:36
*** HenryG has joined #openstack-ironic20:37
*** kkoski has quit IRC20:38
*** kkoski has joined #openstack-ironic20:39
*** mjturek1 has quit IRC20:52
*** andreykurilin has quit IRC20:55
trowndo LOG.debug messages need internationalization?20:56
rlootrown: don't need20:58
rlootrown: this might help: http://docs.openstack.org/developer/oslo.i18n/guidelines.html21:00
trownthanks rloo21:03
*** trown is now known as trown|outttypeww21:04
*** Marga_ has quit IRC21:09
*** Marga_ has joined #openstack-ironic21:10
*** andreykurilin has joined #openstack-ironic21:20
*** Sukhdev has quit IRC21:28
*** mrda-away is now known as mrda21:33
mrdaMorning21:33
*** mjturek1 has joined #openstack-ironic21:34
jlvillalmrda, Good morning21:35
mrdao/21:36
devanandavictor_lowther: looks like you proposed 173047 without a bug tagged. mind filing one so we can track that?21:45
devanandavictor_lowther: also, how serious is that?21:45
devanandaI'm about to cut Kilo, so if there's some critical bug in the DRAC driver ... it'd be nice to know that now :)21:46
NobodyCammorning mrda21:47
BadCubhiya mrda21:47
victor_lowtherdevananda: It will cause Ironic to error out if set_boot_device is called against the pxe_drac driver with the device that the system is already set to.21:48
devanandavictor_lowther: I don't know off hand how to guage the impact of that. Does it happen during every deploy? or only when the user manually invokes set-boot-device?21:49
victor_lowtherI know that tripleo does that.21:49
victor_lowtherIt happens in every deploy that the scripts do not check the boot device to see if it needs changing before trying to change the boot device. :)21:49
devanandaI don't believe the Nova driver does that at all21:50
victor_lowtherThe non-randisk parts of idonic-discoverd do.21:51
pshigemrda: morning21:51
victor_lowtherer, ironic21:51
victor_lowtherif I remember the sequence of events that led me to find the patch properly.21:51
devanandathat would make sense21:52
*** Marga_ has quit IRC21:52
devanandavictor_lowther: seems like a reasonable backport candidate to me. quite small and contained within the driver. also needs unit tests :)21:53
victor_lowtherI wouldn't call it critical, but i knew exactly what was failing in the drac driver, and I am less familiar with the higher-level codepaths.21:53
victor_lowtherThe IPMI and other drivers probably just don't care, but the peculiarities of the drac and the lifecycle controller make it a little more brittle about accepting non-changes like that.21:54
devanandavictor_lowther: as you can imagine, every driver seems to have some unique quirks in this area ;)21:55
devanandai think AMT only accepts the command the first time you send it21:55
NobodyCamdevananda: yep21:55
devanandaany request to change boot device after the first one, and before the next reboot, is ignored21:55
victor_lowtherAMT uses sorta the same WSMAN braindamange that drac does, so that makes sense.21:55
victor_lowtherIIRC.21:56
*** Marga_ has joined #openstack-ironic21:56
*** Marga_ has quit IRC21:57
*** kkoski has quit IRC22:03
*** andreykurilin has quit IRC22:05
rlooNobodyCam: wrt 170065. Sorry. Is this something we wanted to get into rc1? (I don't think it is necessary.)22:07
NobodyCamrloo: it a doc, not a blocker, but nice to have I would think22:08
rlooNobodyCam: well, your or someone else's call. use what is in the patch now even though it isn't great, or fix, or defer til after rc1.22:09
devanandathe patch to open liberty is in the gate, btw22:09
BadCubnice22:09
devanandaonce 171274 merges, RC is official and lierty is open22:10
NobodyCamdevananda: w00t22:10
NobodyCamrloo: humm let me take a look22:10
devanandatime for us to start writing some real release notes22:14
openstackgerritChris Krelle proposed openstack/ironic: Update ilo drivers documentation for inspection  https://review.openstack.org/17006522:35
*** rfchapman has joined #openstack-ironic22:47
*** Marga_ has joined #openstack-ironic22:49
*** mtanino has quit IRC22:51
*** coolsvap has quit IRC22:51
*** dprince has quit IRC22:56
*** mtanino has joined #openstack-ironic23:04
*** chlong has joined #openstack-ironic23:15
openstackgerritPierre Riteau proposed openstack/python-ironicclient: Add support for filtering nodes by chassis UUID  https://review.openstack.org/16796423:21
openstackgerritPierre Riteau proposed openstack/python-ironicclient: Add support for filtering nodes by chassis UUID  https://review.openstack.org/16796423:21
*** jamielennox|away is now known as jamielennox23:22
openstackgerritMerged openstack/ironic: Cleanup DHCPFactory._dhcp_provider after tests  https://review.openstack.org/17203623:25
openstackgerritMerged openstack/ironic: Local boot note about updated deploy ramdisk  https://review.openstack.org/17302923:25
openstackgerritChris Krelle proposed openstack/ironic: Update ilo drivers documentation for inspection  https://review.openstack.org/17006523:31
*** yuanying has joined #openstack-ironic23:39
*** r-daneel has quit IRC23:43
*** saripurigopi has joined #openstack-ironic23:51
*** saripurigopi has quit IRC23:53
*** Haomeng has joined #openstack-ironic23:56
openstackgerritChris Krelle proposed openstack/ironic: Check temp dir is usable for ipmitool driver  https://review.openstack.org/16038323:58
*** Haomeng|2 has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!