Friday, 2014-08-22

NobodyCamadam_g: it sure could :)00:02
*** penick has joined #openstack-ironic00:02
*** bensonmiller has quit IRC00:04
jrollNobodyCam: heh, np00:17
NobodyCamadam_g: like this: http://paste.openstack.org/show/tOVdkOaaVu2sD49utrla/00:22
NobodyCamhey jroll00:22
* jroll ducks00:22
jrollwhat's up?00:22
NobodyCamhuh00:23
NobodyCam:-p00:23
jrolloh, thought you needed something :P00:23
NobodyCamits after 500:23
NobodyCamnope00:23
jrollright, beer time00:24
* jroll goes home00:24
jrollhave a good night :)00:24
NobodyCamhave a good night jroll :)00:24
*** dlaube has quit IRC00:30
*** aswadr has joined #openstack-ironic00:31
*** tatyana has joined #openstack-ironic00:32
*** mikedillion has quit IRC00:33
*** ChuckC has quit IRC00:41
*** nosnos has joined #openstack-ironic00:59
*** harlowja is now known as harlowja_away01:00
*** penick has quit IRC01:05
openstackgerritMotohiro Otsuka proposed a change to openstack/ironic: Set ssh_virt_type to vmware  https://review.openstack.org/11615101:19
openstackgerritMotohiro Otsuka proposed a change to openstack/ironic: Add boot_device support for vbox  https://review.openstack.org/11615201:19
*** harlowja_away is now known as harlowja01:40
*** penick has joined #openstack-ironic01:44
*** rloo_afk has quit IRC01:52
*** penick has quit IRC01:54
*** nosnos has quit IRC02:03
*** nosnos has joined #openstack-ironic02:04
*** nosnos_ has joined #openstack-ironic02:08
*** nosnos has quit IRC02:08
*** Poornima has joined #openstack-ironic02:30
*** rushiagr_away is now known as rushiagr02:37
*** harlowja is now known as harlowja_away02:49
*** nosnos_ has quit IRC02:56
*** yuanying has quit IRC03:08
*** krtaylor has joined #openstack-ironic03:21
*** yuanying has joined #openstack-ironic03:24
openstackgerritRuby Loo proposed a change to openstack/ironic: properly format user guide in RST  https://review.openstack.org/11616803:31
*** Poornima has quit IRC03:38
*** nosnos has joined #openstack-ironic03:40
openstackgerritMotohiro Otsuka proposed a change to openstack/ironic: Add boot_device support for vbox  https://review.openstack.org/11615203:40
openstackgerritMotohiro Otsuka proposed a change to openstack/ironic: Set ssh_virt_type to vmware  https://review.openstack.org/11615103:40
*** tatyana has quit IRC03:45
openstackgerritRuby Loo proposed a change to openstack/ironic: properly format user guide in RST  https://review.openstack.org/11616803:48
*** nikunj2512 has joined #openstack-ironic04:04
*** Poornima has joined #openstack-ironic04:04
*** pcrews has quit IRC04:14
*** tatyana has joined #openstack-ironic04:40
openstackgerritGregory Haynes proposed a change to openstack/ironic: Add conductor_hostname property to nodes  https://review.openstack.org/10968804:43
greghaynesok, that patch ended up much larger than expected :/04:43
openstackgerritRamakrishnan G proposed a change to openstack/ironic: Take iSCSI deploy out of pxe driver  https://review.openstack.org/11123204:49
*** rakesh_hs has joined #openstack-ironic04:53
*** pradipta_away is now known as pradipta04:54
*** shakamunyi has quit IRC05:03
*** rakesh_hs has quit IRC05:03
*** rakesh_hs has joined #openstack-ironic05:05
*** rakesh_hs has quit IRC05:09
*** rakesh_hs has joined #openstack-ironic05:10
*** rushiagr is now known as rushiagr_away05:21
*** Poornima has quit IRC05:26
*** tatyana has quit IRC05:27
*** rushiagr_away is now known as rushiagr05:29
openstackgerritRamakrishnan G proposed a change to openstack/ironic: Add support for creating vfat and iso images  https://review.openstack.org/10541305:33
*** rushiagr is now known as rushiagr_away05:40
*** harlowja_away has quit IRC05:44
*** tatyana has joined #openstack-ironic05:52
*** killer_prince is now known as lazy_prince06:00
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/11587806:12
*** lazy_prince is now known as killer_prince06:12
*** killer_prince is now known as lazy_prince06:14
*** shakamunyi has joined #openstack-ironic06:29
*** shakamunyi has quit IRC06:34
openstackgerritRamakrishnan G proposed a change to openstack/ironic: Add support for interacting with swift  https://review.openstack.org/10579506:34
*** rushiagr_away is now known as rushiagr06:34
*** k4n0 has joined #openstack-ironic06:35
openstackgerritRamakrishnan G proposed a change to openstack/ironic: IloVirtualMediaIscsi deploy driver  https://review.openstack.org/11386507:04
*** Poornima has joined #openstack-ironic07:06
*** viktors|afk is now known as viktors07:31
*** derekh has joined #openstack-ironic08:14
*** dtantsur|afk is now known as dtantsur08:15
dtantsurMorning Ironic, TGIF!08:16
*** lucasagomes has joined #openstack-ironic08:32
yuriyzmorning Ironic dtantsur08:45
lucasagomesmorning yuriyz  dtantsur08:50
*** zz_naotok has quit IRC08:50
lucasagomesgate is broken?08:50
lucasagomesI see a couple of failures, some random ones08:50
dtantsurlucasagomes, yuriyz, morning08:50
dtantsurseems like yes08:50
*** zz_naotok has joined #openstack-ironic08:51
*** rameshg87 has joined #openstack-ironic08:52
*** igordcard has joined #openstack-ironic08:57
*** lazy_prince is now known as killer_prince09:15
*** killer_prince is now known as lazy_prince09:19
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Add iPXE Installation Guide documentation  https://review.openstack.org/10680909:21
*** ndipanov has joined #openstack-ironic09:22
openstackgerritRamakrishnan G proposed a change to openstack/ironic: IloVirtualMediaAgent deploy driver  https://review.openstack.org/11588509:22
*** pradipta is now known as pradipta_away09:23
*** pelix has joined #openstack-ironic09:28
*** yuanying has quit IRC09:30
*** lazy_prince is now known as killer_prince09:45
*** killer_prince is now known as lazy_prince09:49
rameshg87dtantsur, hi10:44
dtantsurrameshg87, hi!10:45
rameshg87dtantsur, is there some problem with jenkins recently, i have been facing jenkins failures in all my submittals :(10:46
rameshg87dtantsur, http://logs.openstack.org/32/111232/16/check/gate-ironic-docs/20806a9/console.html says build_sphinx failed10:46
dtantsurrameshg87, yeah, seems like that. I didn't investigate.10:46
rameshg87dtantsur, okay, seemed not related to my change.  it's failing from yesterday.i thought it might get fixed in a day.  but no submittals have gone in yesterday :)10:47
*** yuanying has joined #openstack-ironic10:50
*** yuanying has quit IRC11:06
openstackgerritVladyslav Drok proposed a change to openstack/ironic: Add driver name on driver load exception  https://review.openstack.org/11204911:08
*** nosnos has quit IRC11:17
*** nosnos has joined #openstack-ironic11:17
*** nosnos has quit IRC11:21
*** dtantsur is now known as dtantsur|lunch11:25
lucasagomesyeah this docs problem seems to be something in the wsme extension for sphinx11:26
lucasagomesjobs in ceilometer are failing for the same reason11:26
rameshg87lucasagomes, okay11:28
*** rakesh_hs has quit IRC11:29
rameshg87lucasagomes, i have one question in agent driver11:30
rameshg87lucasagomes, https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/agent.py#L221 - agent seems to use management interface to set the boot device11:31
lucasagomesrameshg87, ack, (I know very little about it)11:31
rameshg87lucasagomes, but agent driver does have management interface - https://github.com/openstack/ironic/blob/master/ironic/drivers/agent.py#L34-L4111:31
rameshg87lucasagomes, or am i missing something ??11:31
lucasagomesrameshg87, oh... looks like a bug11:31
lucasagomesjroll, ^11:31
lucasagomesrameshg87, so that utils method will check if the mgmt interface is present11:31
lucasagomesbefore setting the boot device11:32
lucasagomesthat's why it didn't fail11:32
rameshg87lucasagomes, yeah seemed a bug to me11:32
lucasagomes+111:32
rameshg87lucasagomes, i see that for agent ssh driver - https://github.com/openstack/ironic/blob/master/ironic/drivers/agent.py#L8211:32
rameshg87lucasagomes, so must have missed it :)11:32
lucasagomesyup but not for ipmitool/ipminative ones11:33
*** nikunj2512 has quit IRC11:34
*** yuanying has joined #openstack-ironic11:37
*** lucasagomes is now known as lucas-hungry11:39
*** yuanying has quit IRC11:41
jrollmorning ironic, happy friday! :)11:57
jrolllucas-hungry: turns out I was right :P11:57
jrollrameshg87: want to add a bug, fix it, both, neither?11:58
rameshg87jroll, i can :)11:58
rameshg87jroll, do both11:58
rameshg87jroll, wanted to confirm with you11:58
jrollrameshg87: yeah I agree :)11:58
jrollthanks!11:58
rameshg87jroll, okay, i will take care of it11:58
* jroll is going to work harder on getting check jobs in11:58
dtantsur|lunchjroll, morning!12:10
rameshg87jroll, i have a question12:13
rameshg87jroll, why doesn't agent driver validate whether image_source has been specified or not: https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/agent.py#L190-L20512:13
rameshg87jroll, like pxe driver12:13
openstackgerritRamakrishnan G proposed a change to openstack/ironic: Add management interface for agent drivers  https://review.openstack.org/11624112:26
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic: Updated from global requirements  https://review.openstack.org/11560012:28
*** rameshg87 has quit IRC12:28
*** linggao has joined #openstack-ironic12:29
openstackgerritVladyslav Drok proposed a change to openstack/ironic: Remove gettextutils _ injection  https://review.openstack.org/11063412:30
*** Poornima has quit IRC12:32
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-ironicclient: Updated from global requirements  https://review.openstack.org/11625412:34
jrollram<tab> :(12:34
jrollmorning dtantsur|lunch :)12:35
dtantsur|lunchoh, I'm not on lunch any more :)12:35
*** dtantsur|lunch is now known as dtantsur12:35
jroll:P12:35
dtantsurhmmm... though I can have some tea! brb12:35
*** yuanying has joined #openstack-ironic12:37
*** yuanying has quit IRC12:42
*** rushiagr is now known as rushiagr_away12:51
*** lucas-hungry is now known as lucasagomes12:55
*** lazy_prince has quit IRC12:58
jrollvdrok: thanks for splitting that up :)12:59
jrolloh, right, our gate is broken13:00
jrollwhee.13:00
jrollfix is +A'd though13:00
jrollbbiab13:00
openstackgerritSyed Ismail Faizan Barmawer proposed a change to openstack/ironic: Add UEFI based deployment support in Ironic  https://review.openstack.org/11435713:00
openstackgerritDmitry Tantsur proposed a change to openstack/ironic: EXPERIMENTAL Endpoint for accepting discovered data  https://review.openstack.org/11600313:17
*** Poornima has joined #openstack-ironic13:21
*** kincl has joined #openstack-ironic13:22
vdrokthanks jroll!13:28
*** aswadr has quit IRC13:32
*** pcrews has joined #openstack-ironic13:33
NobodyCamgood morning Ironic, TGIF :)13:34
lucasagomesmorning NobodyCam13:34
NobodyCammorning lucasagomes13:35
linggaoGood morning NobodyCam lucasagomes jroll dtantsur vdrok13:35
dtantsurNobodyCam, linggao, morning!13:35
lucasagomesmorning linggao :)13:35
NobodyCammorning linggao :)13:35
NobodyCammorning dtantsur13:36
vdrokmorning linggao NobodyCam13:36
vdrokafternoon dtantsur lucasagomes13:37
lucasagomesvdrok, afternoon/evening13:37
dtantsurvdrok, g'afternoon13:37
*** jasondotstar has joined #openstack-ironic13:37
NobodyCamgood morning vdrok :)13:38
*** yuanying has joined #openstack-ironic13:38
NobodyCamlucasagomes: question ? did you see my paste's from last night?13:39
lucasagomesNobodyCam, the ones about the nova commands proxy?13:39
NobodyCamya13:40
lucasagomesyup13:40
NobodyCamthat look ok?13:40
lucasagomesNobodyCam, I remember you were getting an error13:40
lucasagomesso I commented about it13:40
lucasagomesbecause afair it was doing a client.node.list()13:41
*** Alexei_987 has joined #openstack-ironic13:41
lucasagomesand appending the returned list to another list13:41
lucasagomesafaici, we should get the ironic node list (client.node.list()) and covert each element to a "nova bm" object/dict13:42
lucasagomesand then return the list of converted objects13:42
NobodyCamlucasagomes: http://paste.openstack.org/show/Em861wMwFvrFlsWkugfX/13:42
lucasagomesNobodyCam, w00t!13:43
*** yuanying has quit IRC13:43
NobodyCamplease look at the fields.13:43
NobodyCamif we have to convert username/address/it gets mucky13:44
lucasagomesoh because of the different drivers right?13:44
NobodyCamyep13:44
lucasagomeswell nova bm only supported ipmitool + pxe13:44
lucasagomeswe can constraint it13:44
lucasagomesonly fill the username/password if the driver is == pxe_ipmitool13:45
lucasagomesbecause that's what is supported by nova bm13:45
lucasagomesso you can look at ipmi_username/ipmi_password if driver == pxe_ipmitool13:45
NobodyCamya if we limit to one we're good13:45
lucasagomesI think it's fair13:45
NobodyCamwhat about nova's interface field13:47
NobodyCamnote show vs list in the output13:47
lucasagomesnova only supported 1 NIC per node?13:47
NobodyCam??13:48
NobodyCambeen so long13:48
NobodyCamlol13:48
lucasagomesI see like MAC Address13:48
lucasagomeson the baremetal-node-list13:48
lucasagomesand for baremetal-node-show idk what "interfaces" looked like... hmm maybe we should start a devtest with nova bm and check that out13:49
lucasagomesor look at the code, may be faster13:49
NobodyCamlucasagomes: http://docs.openstack.org/user-guide/content/cli_nova_baremetal-details.html13:50
lucasagomesoh13:51
NobodyCambrb13:54
*** russellb is now known as rustlebee13:54
*** k4n0 has quit IRC13:55
*** teju has joined #openstack-ironic13:56
NobodyCamok had to run out for a quick smoke :-p13:59
lucasagomesack14:00
NobodyCamoh lucasagomes dtantsur did you see https://review.openstack.org/#/c/68942/3214:02
lucasagomesnop14:03
lucasagomeslemme see14:03
dtantsurNobodyCam, yeah, we have a patch somewhere I guess14:03
dtantsurNobodyCam, https://review.openstack.org/#/c/116093/14:03
dtantsurshould we approve it?14:04
NobodyCamdtantsur: what failed in the test14:04
NobodyCamand we will need mrda to approve14:05
dtantsurtrying to figure out...14:05
dtantsurseems like still tempest problem14:06
*** rameshg87 has joined #openstack-ironic14:07
*** teju has quit IRC14:08
rameshg87jroll, hi14:14
jrollhey rameshg8714:17
jrollrameshg87: the image_source validation (or lack thereof) was probably just an oversight14:17
rameshg87hi jroll14:18
NobodyCammorning jroll, rameshg8714:18
jrollmorning NobodyCam :)14:18
NobodyCam:)14:18
rameshg87good morning NobodyCam :-)14:18
rameshg87jroll, so i think it makes sense to add image_source validation since agent driver can't deploy without it14:19
rameshg87jroll, and regarding the tests for https://review.openstack.org/#/c/116241/  . do you have any particular things in mind that we can assert14:19
jrollrameshg87: sure, it would be good14:20
*** rameshg87 has quit IRC14:21
jroll.14:22
jrollwhyyyyyy14:22
jrolladam_g: I'm tempted to recheck 116093 with no bug unless you think that's a real failure14:31
*** yuanying has joined #openstack-ironic14:39
NobodyCamjroll: see error here:http://logs.openstack.org/93/116093/3/check/check-tempest-dsvm-virtual-ironic/acaf2f9/logs/testr_results.html.gz14:40
jrollNobodyCam: yeah, saw the error, looks like a tempest issue but idk14:40
jrollI like to recheck for funsies14:40
NobodyCamjroll: ++ on recheck14:41
jrollok14:41
* jroll does it14:41
jrollalso14:41
jrollother than failing tests, why would we need mrda to approve it?14:41
jrollit... doesn't work without that change14:41
jrollafaik14:41
jrolllike, just do it and poke mrda14:42
*** yuanying has quit IRC14:43
NobodyCam:) only because he's doing the nova patches14:45
NobodyCambrb14:46
lucasagomesjroll, yeah if that's required (a must) we should then just approve and let him know14:46
jrollyep, +114:46
* jroll bbl14:49
*** rushiagr_away is now known as rushiagr14:49
openstackgerritDmitry Tantsur proposed a change to openstack/ironic: EXPERIMENTAL Endpoint for accepting discovered data  https://review.openstack.org/11600315:00
openstackgerritVladyslav Drok proposed a change to openstack/ironic: Cleanup _LW, _LC, _LI translations  https://review.openstack.org/11630315:02
* NobodyCam is back15:05
openstackgerritSteve Martinelli proposed a change to openstack/ironic-specs: Update spec theme to show incubation status  https://review.openstack.org/11630615:07
*** vdrok is now known as vdrok_away15:10
*** chuckC has joined #openstack-ironic15:10
*** killer_prince has joined #openstack-ironic15:14
*** killer_prince is now known as lazy_prince15:15
*** amitpp has joined #openstack-ironic15:19
*** jgrimm has joined #openstack-ironic15:22
*** shakamunyi has joined #openstack-ironic15:24
*** igordcard has quit IRC15:26
*** amitpp has quit IRC15:29
openstackgerritChris Krelle proposed a change to openstack/ironic: Proxy Nova baremetal commands to Ironic  https://review.openstack.org/11631615:31
NobodyCamI'm not sure that is the correct place to submit that patch15:33
NobodyCamalso it needs cleanup and such :-p15:33
*** amitpp has joined #openstack-ironic15:38
*** yuanying has joined #openstack-ironic15:40
*** shakamunyi has quit IRC15:41
*** yuanying has quit IRC15:44
dtantsurTGIF and I'm leaving now :)16:00
dtantsurhave a nice weekend :)16:00
*** dtantsur is now known as dtantsur|afk16:00
adam_gjroll, we're blocked until https://review.openstack.org/#/c/116103/ merges16:01
adam_gits been queued for gate checks for 6 hrs now16:03
NobodyCamnight dtantsur|afk16:03
NobodyCamadam_g: gah :(16:03
*** amitpp has quit IRC16:08
*** amitpp has joined #openstack-ironic16:11
*** bandicot has joined #openstack-ironic16:17
*** bandicot is now known as bensonmiller16:20
*** tatyana has quit IRC16:20
jrolladam_g: gotcha16:23
JayFadam_g: should we ask -infra to bump it?16:24
JayFadam_g: I know they actively encourage folks to have them bump prio on gate-fixing bugs16:24
*** dlaube has joined #openstack-ironic16:29
adam_gJayF, ha, it just started running as i was typing a ping to -infra to get it bumped16:30
JayF:)16:30
*** rushiagr is now known as rushiagr_away16:31
*** openstackgerrit has quit IRC16:34
NobodyCam:-p16:34
*** derekh has quit IRC16:34
*** yuanying has joined #openstack-ironic16:41
*** yuanying has quit IRC16:45
*** Poornima has quit IRC16:54
ryanpetrellothis ironic stable failure look familiar to anybody?16:57
ryanpetrellohttp://logs.openstack.org/98/116198/1/check/gate-pecan-tox-ironic-stable/56b7b42/console.html16:57
ryanpetrello2014-08-22 08:37:36.666 | Traceback (most recent call last):16:57
ryanpetrello2014-08-22 08:37:36.666 |   File "ironic/tests/api/v1/test_drivers.py", line 49, in test_drivers16:57
ryanpetrello…16:58
ryanpetrello2014-08-22 08:37:43.665 |     raise mismatch_error16:58
ryanpetrello2014-08-22 08:37:43.665 | MismatchError: ['fake-host1', 'fake-host2'] != [u'fake-host2', u'fake-host1']16:58
adam_gyou may need to run with PYTHONHASHSEED='0'16:58
adam_ghmm. maybe zuul was laying, 116103 is queued again at 7 hrs16:58
adam_glying16:59
JayFthen I'd totes ping infra16:59
adam_gryanpetrello, that looks like a side effect of the recent tox upgrade on the slaves17:00
NobodyCamadam_g: it resets with each review that land ahead of it17:00
NobodyCamlands even17:00
ryanpetrellogotcha17:00
adam_gNobodyCam, oh, right17:01
*** comstud is now known as bearhands17:02
*** Alexei_987 has quit IRC17:05
*** viktors is now known as viktors|afk17:09
adam_ghas anyone seen this failure http://logs.openstack.org/03/116103/2/check/check-tempest-dsvm-virtual-ironic-nv/2fcf03b/logs/testr_results.html.gz ? the ironic job still failed in check on 11610317:11
*** amitpp has quit IRC17:11
*** lucasagomes is now known as lucas-dinner17:15
NobodyCamhave a good weekend lucas-dinner17:23
adam_gugh. theres actually two gate breakers ATM.17:28
NobodyCamtwo?17:30
adam_ghttps://bugs.launchpad.net/tempest/+bug/136036717:30
*** openstackgerrit has joined #openstack-ironic17:31
*** pelix has quit IRC17:31
NobodyCam:(17:34
jrollbtw, I won't be working on ironic today, but if y'all need something from me feel free to poke me17:34
jrollI'll be online17:35
adam_gthe hash ordering thing is just a PITA from the tox upgrade breaking the world, but that second bug is another case of 'we really need to be voting on other projects'17:35
*** harlowja has joined #openstack-ironic17:36
NobodyCamyes, or cores need to look at what there is a failed job.17:40
NobodyCamwe could non-voting-stable and non-voting-unstable17:41
NobodyCamto indecate if the just is infact stable or not17:41
NobodyCam:-p17:41
*** yuanying has joined #openstack-ironic17:41
NobodyCamindicate even17:42
jrollno, we just need to be able to vote17:43
jrollit's absurd that we can't17:43
adam_gyeah, unless our tests are actually voting nobody is going to pay attention17:44
JayFI've seen people proudly declare how they ignore nonvoting tests on the ML and otherwise :(17:44
NobodyCamuggh doc jobs are failing17:44
NobodyCamhttps://review.openstack.org/#/c/11609317:44
JayFnot sure why it's OK to be so blaze' about breaking other folks stuff17:44
*** yuanying has quit IRC17:46
NobodyCami'd even be happy with notice that anohter porject is landing something that is going to break us :-p17:46
NobodyCampatch ####### failed your test but was +a'd17:47
SpamapSQuestion about concurrency in Ironic17:48
NobodyCamshoot17:48
jrolladam_g: the best are when they change lib/ironic, our tests fail, and they merge it anyway17:48
SpamapSSuppose I tell Ironic to rebuild 100 machines17:48
jrollJayF: because f the not blessed projects17:48
SpamapSCan you walk me through what you would expect to happen, if I have one conductor.17:49
jrollSpamapS: uh huhhhhh17:49
SpamapSassume the machines have a 5 minute POST17:49
jrollSpamapS: it should work, just slowly, I think.17:49
NobodyCamSpamapS: it rebuilds them one by one?17:49
jrollSpamapS: if you use the agent driver, it is alllllll goooooood.17:49
jrolloh, rebuild, ha17:49
jrollI guess it would still be fine17:50
SpamapSso, power off... power on.. deploy ramdisk.. deploy root .. power off.. power on. ?17:50
NobodyCamyep17:50
jrollNobodyCam: one by one? really?17:50
SpamapSso 16 hours to rebuild 100 machines17:50
SpamapSassuming image transfer is instant. ;)17:50
NobodyCamI've actually never tested that,17:51
jrolllolol17:51
jrollit should be parallel17:51
SpamapSWhat I'd like to see is they are all powered off, powered on, and in the deploy ramdisk.17:51
SpamapSif we only do one dd at a time.. thats fine17:51
jrollespecially with the agent model17:51
*** igordcard has joined #openstack-ironic17:51
NobodyCamSpamapS: have you actually tested at that scale?17:52
SpamapSjroll: can you explain the difference the agent driver would introduce? I'm only familiar with IPMI+PXE17:52
SpamapSNobodyCam: no, but we're about to. :)17:52
SpamapSNobodyCam: working out a few more kinks and somebody asked how thats likely to work.17:52
jrollSpamapS: the image is deployed by:17:53
jrollan http call to the agent from conductor17:53
jrollagent downloads image from swift and writes it out17:53
NobodyCamSpamapS: how are the rebuild commands being issued? 100 nova rebuild <balh> commands?17:53
jrollagent calls back to ironic and ironic reboots17:53
jrollthat's the fundamental difference, anyway17:53
*** amitpp has joined #openstack-ironic17:54
SpamapSjroll: so how do we know the root partition is o-k to destroy? run agent from ramdisk?17:56
SpamapSNobodyCam: correct17:56
SpamapSNobodyCam: one after the other17:56
NobodyCamI expect that is how the rebuild are going to run17:56
NobodyCamone after the other17:56
NobodyCambut would love to see the test results17:57
SpamapSNobodyCam: they are reported all as in progress by Nova17:58
*** amitpp has quit IRC17:58
SpamapSNobodyCam: but I haven't paid attention to what state each node is in while they're in that state.17:58
NobodyCam:)17:59
dlaubeinteresting stuff.. can't wait to hear the results SpamapS18:00
NobodyCammorning dlaube :)18:00
SpamapSIf they all end up sitting in the deploy ramdisk and are just rebooted as they finish, then thats good, because then 16 hours shrunks to 10 minutes + nnodes*image_transfer_time.18:02
SpamapSshrinks18:02
dlaubeg'morning guys18:03
NobodyCamSpamapS: it may get https://github.com/openstack/ironic/blob/master/etc/ironic/ironic.conf.sample#L544 threads18:05
NobodyCamminus any periodic task workers18:06
JayFjroll: Does agent driver support rebuild yet?18:06
JayFSpamapS: fwiw right now agent only supports full disk images, and doesn't honor --preserve-ephemeral (the worst named flag ever)18:07
SpamapSJayF: it's amazing isn't it?18:07
JayFSpamapS: but I can tell you we've done lots of simultaneous builds before, not with one conductor, but like >500 nodes + 4 conductors, we had someone build around 100 machines near-simultaneously without incident18:08
SpamapSJayF: I think we should just call it --http://beyondbasicplay.files.wordpress.com/2013/09/catching-bubbles.jpg18:08
JayFwe should call it18:08
JayF--preserve-snowflake18:08
JayFbecause as an option it completely goes against the cloud-style of deployment and disposable servers :(18:08
jrollJayF: I believe rebuild is a thing, but we don't support --preserve-ephemeral18:09
jrollJayF: actually we don't support ephemeral at all18:09
jrollbecause whole disk images18:09
jrollthat said, any disks that are not the boot disk will be preserved in the upstream agent18:09
SpamapSJayF: 2TB of swift data isn't really ever going to fit in that model, and it also isn't cost effective to run that over iSCSI. DAS is not going away.18:09
jrollmorning dlaube :)18:10
JayFSpamapS: A man can dream, right?18:11
SpamapSI mean we can just dispose of a swift server. But that is going to waste ridiculous amounts of CPU and bandwidth to resync.18:11
JayFI think we should just accept that it's a workaround because some things don't fit in that model yet18:11
jrollNobodyCam: did your soft power patch land?18:11
JayFthat being said, like jroll said, the agent will not image any extra disks18:11
JayFthe logic for which disk to image is a little weird18:11
JayFI think it's something like smallest disk >8GB18:11
jroll4GB18:11
SpamapSI'd like to have the disks and the partitions of the box addressible using a set of UUID's.18:12
JayFpatches welcome :)18:12
SpamapSThats something I think we're wanting to do soon-ish.18:12
JayFit's not perfect but it works right now for many cases18:12
JayFwe have a stack of things for k we want to do soonish18:12
SpamapSA few of our engineers at HP are a bit nervous about the fact that preserve_ephemeral is not defaulted to True.18:13
jrollwhat18:13
JayFand by that I mean we're doing it now in our production but just couldn't get it through "The Process(tm)" in time18:13
* jroll flips tables18:13
jrollsorry18:13
jroll18:12:24         SpamapS | I'd like to have the disks and the partitions of the box addressible using a set of UUID's. <- in ironic?18:14
SpamapSjroll: yes in Ironic.18:14
SpamapSJust basically as it is claiming the disks, feed back the UUID's it has claimed.18:14
jrollironic shouldn't know about those things, this is not an inventory system :|18:14
jrollwell18:15
jrolls/know about/keep track of18:15
SpamapSAnd then I can say "deploy image X to wherever the disk with uuid==xxxxx is"18:15
SpamapSIt knows where MACs are.18:16
NobodyCamjroll: nope, but adam_g put this up: https://review.openstack.org/#/c/11609318:16
jrollNobodyCam: where's your soft power patch, then?18:16
SpamapSjroll: It's not inventory. It's just a bread crumb.18:16
NobodyCammine or Ng's?18:16
jrollehhhh18:16
jrollNobodyCam: whichever we want to land18:16
jrollSpamapS: I don't believe in preserve_ephemeral, and I barely believe in rebuild, so I'm likely the wrong person to talk to about this stuff18:17
jrollI can see ironic keeping track of disks18:17
jrollbut not letting you say "I want to use disk x"18:17
jrollidk18:17
jrollI would want to hear more, I guess18:18
SpamapSjroll: well oddly enough, I'm talking about how to get out of the rebuild business. :)18:18
jrollright, but you're still in the snowflake business at that point :)18:18
jrolland I get the use case, but18:18
jrollthis is the cloud, folks should use it like a cloud :/18:18
SpamapSjroll: I want to express "it would be most efficient if you deployed to the box with the data I left behind"18:18
NobodyCamjroll: I thought we came up with neither: but https://review.openstack.org/#/c/107864/ <-Ng's https://review.openstack.org/#/c/107778 <- Nobodycam's18:19
JayFSpamapS: IMO the concept of 'I left data behind' scares the crap out of me, as a multi-tenant operator of Ironic ;)18:19
SpamapSjroll: so would you put massive databases on iscsi/ceph/SAN and just take the complexity and CAPEX hit?18:19
JayFI would upgrade massive databases in-band18:20
JayFinstead of using provisioning tooling to do it18:20
jrollNobodyCam: hmm, I thought we had one in the right direction, idk18:20
jrollright, I wouldn't reinstall the OS to upgrade a thing18:20
SpamapSJayF: we both know multi-tenant hardware sharing is not a good idea. But anyway, the data is there on the platters so you have to remove it if you don't want it left behind anyway. :)18:20
JayFSpamapS: which we do.18:20
NobodyCamjroll: I liked Ng version,18:20
SpamapSJayF: right so you'd not have any UUID's for the user.18:21
NobodyCambut what I had may fit better with what landed in nova, if thats what your thinking?18:21
JayFSpamapS: one of the stack of things for K is decom, implemented and working in our deployment of Ironic, with a hardware manager that does all the wiping :)18:21
SpamapSJayF: I'd treat these like volumes. I want to boot from the volume that has UUID=XXXX, which I still own because I asked you not to delete it.18:21
JayFSpamapS: fwiw code is up as well ):18:21
JayFs/):/:)/18:21
JayF^ aka the 'turn that frown upside-down' regexp18:21
jrollNobodyCam: yeah, maybe, idk. see SpamapS' email to the list18:21
jrolllol18:21
SpamapSjroll: regarding snowflakes btw, I don't see it as snowflakes. It is a cost savings to use DAS and to be able to use image based deployments on real servers with DAS.18:23
jrollSpamapS: I'm curious what the use case is for using 'rebuild' to upgrade things18:24
SpamapSThe reality is that we're not suggesting using MySQL with a single node and saying if you don't preserve my data, our business dies.18:24
NobodyCamSpamapS: why not ssh switch to run level 1 and shurdown / unmount everything then issue halt?18:24
jrollor more, if "that's how tripleo works" is the use case18:24
jrollNobodyCam: ironic powers it back on :P18:25
JayFSpamapS: my thoughts are kinda different than jroll's on this. Mostly just as an operational guy, the idea of swapping out my OS and leaving my data and hoping that it works when it comes back up frightens me18:25
JayFSpamapS: I much prefer to replicate data, stand up parallel clusters and migrate, or do upgrades in a rolling fashion18:25
SpamapSNobodyCam: runlevel 1 is no SSH, but even if I leave SSH up, at some point the OS shuts everything down so it can unmount all remaining disks.18:25
JayFbut I don't particularly want to restrict your use case :) I just don't understand why you'd want to go down that path18:25
NobodyCamjroll: ok ssh in switch to run level 1 and shurdown / unmount everything then hard power off with ironic18:25
SpamapSjroll: we do image based deployment. The idea is simple: put a single stream of bits through QA, and then put the exact same stream of bits into production.18:25
NobodyCamhumm18:26
JayFsimilar, I'm guessing, to the folks who don't understand why we do what we do18:26
jrollhrm.18:26
JayFSpamapS: I'd implement that pattern with Docker containers ;)18:26
SpamapSAnd that is a valid, possible next step.18:26
SpamapSThat leaves us with kernel updates, which might be isolated enough to not want an image based update pattern.18:27
SpamapSI believe this is precisely the CoreOS model.18:27
SpamapSThat also puts us in a bad spot for cinder-volume, which cannot run inside a docker container18:28
SpamapS(because iscsi is not namespaced properly)18:29
JayFThat's unfortunate about iscsi not being namespaced :(18:29
JayFalthough you can still use containers, even if not docker.18:29
SpamapSYeah I kind of hoped a year ago when we figured that out that we would just wave our hands until they fixed it.18:29
JayFI think we are the 'they'18:29
JayFpeople working for big companies on open source enabling what we need to do :)18:29
JayFthere is no they anymore, we are they :P18:30
SpamapSMy version of "they" was realy Serge Hallyn18:30
SpamapSBut then he got busy. :-P18:31
SpamapSIt's realy not iSCSI itself, but the netlink socket control mechanism it relies on.18:31
SpamapSIf they'd just used ioctls like everybody else, it would be done.18:31
SpamapSAnyway, another version of what we do doesn't even bother with a docker container.18:32
SpamapSIt works like your agent sort of. Download the image, force it over top of the root partition, and restart everything.18:32
SpamapSBut that still can't update kernels, and pid1 would be haphazard. :-P18:33
SpamapSAnyway your time has been much appreciated. Thanks. :)18:35
NobodyCamjroll: SpamapS: looking at https://review.openstack.org/#/c/107778 and https://review.openstack.org/#/c/116093 we could make use of the nova timeout value and attempt a soft power off.18:37
*** yuanying has joined #openstack-ironic18:42
NgI sort of want to think that maintenance mode or something like it, could be a way out here18:42
Ngcan you rebuild() a node that's in maintenance mode?18:43
Ngi.e. if we could just tell Ironic to stop enforcing power state on a per-node basis, that would be sufficient18:43
SpamapS@Ng18:44
SpamapSgah18:44
* SpamapS stabs hipchat18:44
NobodyCamNg: oh18:44
Ngmwaha18:44
Ngso like, we say "stop asserting power", ssh in and do "shutdown -h now", poll Ironic for power state == OFF, call rebuild, flip power assertion back on (or maybe have that be implicit in the rebuild)18:45
SpamapSNg: thats a workaround for sure, but it stands to reason that most of the time if I'm doing a 'nova stop' I don't mean "and do nothing to avoid corrupting the root filesystem"18:45
SpamapSkvm issues a soft stop18:45
NobodyCamthats true a node in maintenance mode skips the power stuff18:45
NgSpamapS: what does nova-kvm do if the guest OS ignores the ACPI signal?18:46
*** yuanying has quit IRC18:46
NgI think my proposed patch added another power state, so you could distinguish between sending a node a hard or a soft power off18:47
Ngbut I was literally asleep when I wrote it, so I'm not really sure what it did. probably implemented Counting The Moon as a Service18:47
JayFNg: def count_the_moon()18:48
JayFNg:   return 118:48
JayFNg: ## todo(jayf) support for other planets18:48
SpamapS#winning18:48
Nghehe18:48
NgI would be so stunned if anyone got that reference :)18:49
SpamapSNg: I don't really know. Easy enough to test, just kill -9 pid 1 and then issue a stop..18:49
NgI don't often test things, but when I do, it involves sending -9 to init18:49
NobodyCamlol18:51
NobodyCambrb18:53
*** ndipanov has quit IRC18:57
dlaubehas anyone tried playing with ipv6 in ironic?19:26
dlaubein terms of provisioning baremetal servers with v6 addresses I mean19:26
jrollisn't that more about the switch than ironic itself?19:27
jrollnot sure what you mean19:27
JayFI sadly do not have experience with it :-(19:28
jroll^^19:28
dlaubewell, I'm mostly interested in hearing if anyone is using v6 addresses with IPMI as it relates to ironic…. and any ipv6 support/limitations in neutron19:29
dlaubebeen reading up over here -> https://wiki.openstack.org/wiki/Neutron/IPv619:30
jrollahh19:30
jrollI think neutron supports it ok19:30
jrollI wouldn't trust ipmi to do so but idk19:30
dlaubegotcha19:31
dlaubeyeah, I've been looking into it and a bunch of vendors let you use v6 for IPMI but usually not by default and not easily :P19:31
dlaubehttp://ipvsix.me/?p=15919:32
jrollyeah19:32
jrollI mean19:32
jrollthose should be on an isolated network anyhow19:32
dlaubeyeah19:32
*** Hefeweizen has quit IRC19:34
NobodyCamdlaube: only enough to file this: https://bugs.launchpad.net/ironic/+bug/132594219:39
dlaubety19:41
*** yuanying has joined #openstack-ironic19:43
*** yuanying has quit IRC19:47
*** penick has joined #openstack-ironic19:55
*** igordcard has quit IRC20:04
*** penick has quit IRC20:11
*** rwsu has quit IRC20:13
*** Hefeweizen has joined #openstack-ironic20:21
*** rushiagr_away is now known as rushiagr20:27
NobodyCamconf calls done20:35
NobodyCam:-p20:35
NobodyCamnow brb20:35
*** yuanying has joined #openstack-ironic20:44
*** yuanying has quit IRC20:48
adam_gim gonna be disappearing soon but our gate should be good again once https://review.openstack.org/#/c/116353/ + https://review.openstack.org/#/c/116103/ merge20:57
adam_g116103 looks like its about to fail its 10 hr gate job :( so it will need a recheck20:58
adam_g116353 will hopefully be approved and sent to gate before EOD20:58
NobodyCamadam_g: I'll keep an eye on them and recheck / reverify if needed20:58
adam_gNobodyCam, thanks20:58
JayFwhen a gate is broken in two ways21:05
JayFhow do you get the patches merged?21:05
JayFlike obviously unless both are applied the checks still fail, right?21:05
JayFlike this case, that might not be what's happening, but generally what's the approach?21:06
NobodyCamping infra21:06
NobodyCam:-p21:06
NobodyCambeg fungi or clarkb for help21:06
NobodyCamlol21:06
JayFthat's a horrible answer :(21:06
NobodyCamhehehehe21:06
NobodyCamin this case both foxes are to tempest so you could make one dependent on the other21:09
NobodyCams/foxes/fixes/21:09
*** linggao has quit IRC21:17
adam_gJayF, in this case the fact that ironic job is non-voting makes it easier21:18
JayFaah21:18
JayFbecause it's not 2x ironic patches to fix21:18
jrolllol21:18
JayFit's 2x tempest patches21:19
JayFthe fact the irnoic job is non-voting is also a direct root cause21:19
jrollright21:19
jrollwe wouldn't have this problem in the first place if we voted21:19
*** rushiagr is now known as rushiagr_away21:20
adam_gdoh21:23
adam_gjroll, yeah, we should propose making the job voting again in the tempest check pipeline, on the basis we are almost a week from FF and can't afford to have things blocked like this21:25
NobodyCamadam_g: ++ I would support that21:25
NobodyCamis reverify not working anymore?21:26
NobodyCamor Mr J just that backed up21:26
NobodyCammrda: been on today?21:30
adam_gNobodyCam, i think its backed up on its post-back. i see 116103 running in the check queue now21:31
NobodyCamshould be just about that time. :)21:31
NobodyCamadam_g: sweet :)21:31
adam_g i need to run for a while. ill check back later. thanks for kicking 116103, NobodyCam21:32
*** rushiagr_away is now known as rushiagr21:33
NobodyCam:)21:34
*** rushiagr is now known as rushiagr_away21:36
jrolladam_g: don't they want us integrated or whatever to do that21:40
NobodyCamjroll: we'll never get integrated if they don't stop breaking us LOL :(21:43
jrollikr21:43
*** yuanying has joined #openstack-ironic21:45
*** igordcard has joined #openstack-ironic21:46
*** jasondotstar has quit IRC21:48
*** yuanying has quit IRC21:49
NobodyCamuggh 116103 going to need a recheck :-p22:11
*** penick has joined #openstack-ironic22:15
*** penick_ has joined #openstack-ironic22:19
*** penick has quit IRC22:21
*** penick_ is now known as penick22:21
*** lucas-dinner has quit IRC22:21
*** yuanying has joined #openstack-ironic22:45
*** yuanying has quit IRC22:50
NobodyCam116103 recheck started :-p22:50
*** igordcard has quit IRC23:22
*** chuckC has quit IRC23:31
*** penick has quit IRC23:36
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic: Updated from global requirements  https://review.openstack.org/11560023:38
jrollmuch gate23:43
jrollsuch broken23:44
jrollwow23:44
* jroll looks for a bug # to recheck23:45
JayFI don't think we should recheck yet23:45
JayFuntil those tempest things land, right?23:45
greghaynesrecheck such bug23:45
*** yuanying has joined #openstack-ironic23:46
jrollI rechecked23:47
jrollJayF: we need that to land23:47
jrollJayF: that is the tempest thing23:47
jroll(one of the)23:47
JayFoh.23:47
JayFblargh.23:47
jrollhttps://review.openstack.org/#/c/116103/23:47
jrollyeah23:47
jrollI'm going to get up early tomorrow and just hammer rechecks on that until it goes through23:48
jrollsaturday morning means quick gate :P23:48
jrollthe heck, are "reverify"s not working?23:49
*** yuanying has quit IRC23:51
NobodyCamjenkins way backed up23:53
NobodyCampostback take a long while23:53
jrollyeah, I didn't realize it didn't tell you when jobs starts23:53
NobodyCambut watch zuul the job will restart in a few23:53
jrollyep23:54
NobodyCam:)23:54
NobodyCamand Ty23:54
jrollNobodyCam: jay says it already failed neutron-icehouse though :P23:54
NobodyCamya :(23:54
jrolls/:P/:(23:54
NobodyCam:'(23:54
NobodyCamlol23:54
jroll:(((((((((((((((((((((((23:54
* jroll has had a long week and now it ends with a broken gate23:55
*** aswadr has joined #openstack-ironic23:55
NobodyCam:( /me knows what he'll be doing this weekend ... typing recheck bug <RamdomInt^4>23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!