| *** zdin0bot has quit IRC | 00:13 | |
| *** matsuhashi has joined #openstack-ironic | 00:20 | |
| *** zdin0bot has joined #openstack-ironic | 00:30 | |
| *** zdin0bot has quit IRC | 00:35 | |
| *** zdin0bot has joined #openstack-ironic | 00:44 | |
| *** shakamunyi has joined #openstack-ironic | 00:50 | |
| *** shakamunyi has quit IRC | 00:54 | |
| *** datajerk has joined #openstack-ironic | 01:25 | |
| *** nosnos has joined #openstack-ironic | 01:50 | |
| *** zdin0bot has quit IRC | 01:56 | |
| *** zdin0bot has joined #openstack-ironic | 01:56 | |
| *** killer_prince has quit IRC | 02:22 | |
| *** Haomeng has joined #openstack-ironic | 02:25 | |
| *** killer_prince has joined #openstack-ironic | 02:29 | |
| *** datajerk has quit IRC | 02:50 | |
| *** shakamunyi has joined #openstack-ironic | 02:51 | |
| *** shakamunyi has quit IRC | 02:56 | |
| *** coolsvap|afk has joined #openstack-ironic | 03:05 | |
| *** coolsvap|afk is now known as coolsvap | 03:05 | |
| *** zdin0bot1 has joined #openstack-ironic | 03:08 | |
| *** zdin0bot has quit IRC | 03:09 | |
| *** radsy has quit IRC | 03:37 | |
| *** Mikhail_D_ltp has joined #openstack-ironic | 03:46 | |
| *** Mikhail_D_ltp has quit IRC | 03:53 | |
| *** matsuhashi has quit IRC | 03:54 | |
| *** eghobo has joined #openstack-ironic | 04:02 | |
| *** zdin0bot1 has quit IRC | 04:07 | |
| *** nosnos has quit IRC | 04:10 | |
| *** eghobo has quit IRC | 04:12 | |
| *** matsuhashi has joined #openstack-ironic | 05:04 | |
| *** matsuhashi has quit IRC | 05:10 | |
| *** matsuhashi has joined #openstack-ironic | 05:11 | |
| *** nosnos has joined #openstack-ironic | 05:13 | |
| *** matsuhashi has quit IRC | 05:17 | |
| *** matsuhashi has joined #openstack-ironic | 05:17 | |
| *** zdin0bot has joined #openstack-ironic | 05:24 | |
| *** eghobo has joined #openstack-ironic | 05:28 | |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions https://review.openstack.org/90233 | 05:51 |
|---|---|---|
| *** lazy_prince has joined #openstack-ironic | 05:51 | |
| *** zdin0bot has quit IRC | 05:53 | |
| *** shakamunyi has joined #openstack-ironic | 05:53 | |
| *** zdin0bot has joined #openstack-ironic | 05:54 | |
| *** shakamunyi has quit IRC | 05:58 | |
| *** zdin0bot has quit IRC | 06:05 | |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions https://review.openstack.org/90233 | 06:05 |
| *** rameshg87 has joined #openstack-ironic | 06:06 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/ironic: Imported Translations from Transifex https://review.openstack.org/88508 | 06:08 |
| *** zdin0bot has joined #openstack-ironic | 06:08 | |
| *** vkozhukalov has joined #openstack-ironic | 06:18 | |
| *** ifarkas has joined #openstack-ironic | 06:31 | |
| *** viktors has joined #openstack-ironic | 06:33 | |
| *** zdin0bot has quit IRC | 06:36 | |
| *** killer_prince has quit IRC | 06:42 | |
| *** killer_prince has joined #openstack-ironic | 06:43 | |
| *** shakamunyi has joined #openstack-ironic | 06:54 | |
| *** vkozhukalov has quit IRC | 06:56 | |
| *** shakamunyi has quit IRC | 06:58 | |
| *** matsuhashi has quit IRC | 07:01 | |
| *** matsuhashi has joined #openstack-ironic | 07:03 | |
| *** matsuhashi has quit IRC | 07:23 | |
| *** eghobo has quit IRC | 07:24 | |
| *** matsuhashi has joined #openstack-ironic | 07:25 | |
| *** ifarkas has quit IRC | 07:45 | |
| *** ifarkas has joined #openstack-ironic | 07:45 | |
| openstackgerrit | Dmitry Tantsur proposed a change to openstack/ironic: Consider Glance checksum when caching master images https://review.openstack.org/90390 | 07:49 |
| openstackgerrit | Dmitry Tantsur proposed a change to openstack/ironic: Implement more robust caching for master images https://review.openstack.org/85387 | 07:49 |
| *** jistr has joined #openstack-ironic | 07:51 | |
| *** yuriyz has joined #openstack-ironic | 07:51 | |
| dtantsur | Morning Ironic | 07:51 |
| *** ndipanov has joined #openstack-ironic | 07:51 | |
| *** shakamunyi has joined #openstack-ironic | 07:55 | |
| *** mrda is now known as mrda_away | 07:57 | |
| Mikhail_D_wk | Morning dtantsur :) | 07:57 |
| Mikhail_D_wk | Morning all! :) | 07:58 |
| *** shakamunyi has quit IRC | 08:00 | |
| *** derekh has joined #openstack-ironic | 08:03 | |
| Haomeng | morning dtantsur, Mikhail_D_wk :) | 08:08 |
| *** vkozhukalov has joined #openstack-ironic | 08:19 | |
| *** viktors has quit IRC | 08:23 | |
| *** viktors has joined #openstack-ironic | 08:24 | |
| *** athomas has joined #openstack-ironic | 08:26 | |
| *** martyntaylor has joined #openstack-ironic | 08:27 | |
| *** lucasagomes has joined #openstack-ironic | 08:31 | |
| romcheg | Morning folks | 08:37 |
| yuriyz | morning Ironic | 08:40 |
| dtantsur | morning romcheg, yuriyz | 08:45 |
| *** shakamunyi has joined #openstack-ironic | 08:56 | |
| *** shakamunyi has quit IRC | 09:01 | |
| romcheg | lucasagomes: Hi, around? | 09:08 |
| openstackgerrit | Ghe Rivero proposed a change to openstack/ironic: Clean oslo dependencies files https://review.openstack.org/90670 | 09:09 |
| agordeev | morning Ironic! | 09:20 |
| GheRivero | morning | 09:20 |
| dtantsur | morning, agordeev, GheRivero | 09:21 |
| dtantsur | GheRivero, could you have a look at https://review.openstack.org/#/c/85387 ? I think it seriously touches code you originally wrote | 09:22 |
| GheRivero | sure. On my way | 09:23 |
| agordeev | mornong GheRivero dtantsur ! | 09:30 |
| openstackgerrit | Mikhail Durnosvistov proposed a change to openstack/ironic: Cleanup mock patch without `with` part 2 https://review.openstack.org/73256 | 09:32 |
| openstackgerrit | Mikhail Durnosvistov proposed a change to openstack/ironic: Cleanup mock patch without `with` part 3 https://review.openstack.org/86536 | 09:32 |
| openstackgerrit | Mikhail Durnosvistov proposed a change to openstack/ironic: Cleanup mock patch without `with` part 1 https://review.openstack.org/73223 | 09:32 |
| openstackgerrit | Mikhail Durnosvistov proposed a change to openstack/ironic: Get rid of the newline "\" https://review.openstack.org/66793 | 09:32 |
| openstackgerrit | A change was merged to openstack/ironic: Remove hardcoded node id value https://review.openstack.org/88433 | 09:35 |
| lucasagomes | romcheg, hey :) | 09:40 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Get rid of the swap partition https://review.openstack.org/83726 | 09:45 |
| *** mkerrin has quit IRC | 09:45 | |
| agordeev | lucasagomes: morning :) | 09:48 |
| lucasagomes | agordeev, good morning :) | 09:50 |
| lucasagomes | good morning GheRivero dtantsur agordeev yuriyz romcheg :) | 09:51 |
| dtantsur | morning, lucasagomes :) | 09:51 |
| *** stephenpearson has joined #openstack-ironic | 09:51 | |
| openstackgerrit | A change was merged to openstack/python-ironicclient: Updated from global requirements https://review.openstack.org/89244 | 09:55 |
| dtantsur | GheRivero, also, could you have a look at blueprint, explaining all these changes to caching: https://blueprints.launchpad.net/ironic/+spec/pxe-master-images-caching | 09:55 |
| romcheg | lucasagomes: https://review.openstack.org/#/c/88448/ Since we don't support XML API, do we need this? | 09:59 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Consistent partitioning layout https://review.openstack.org/90675 | 10:01 |
| lucasagomes | romcheg, xml still enabled in the api no? | 10:02 |
| lucasagomes | romcheg, I don't think we disabled it in wsme (cuz they didn't have a straight forward way to disable it) | 10:03 |
| romcheg | lucasagomes: I think it is, but I remember we agreed that we cannot support it | 10:03 |
| lucasagomes | romcheg, yeah, that's correct | 10:03 |
| lucasagomes | romcheg, :/ idk, I still think that if we do not support we should disable it... | 10:03 |
| romcheg | I think it's reasonable to disable it too | 10:05 |
| *** matsuhashi has quit IRC | 10:05 | |
| *** nosnos has quit IRC | 10:05 | |
| lucasagomes | yeah, so while it's enabled, I don't think that patch is complete useless... I mean, it's still there to people try/use so we should fix the problems associated with it | 10:06 |
| romcheg | Let's check how to do that. If we cannot do that in wsme, we can easyly do that by a separate hook | 10:06 |
| lucasagomes | yeah, or just send a patch to wsme to allow disabling it | 10:06 |
| lucasagomes | romcheg, https://review.openstack.org/#/c/78108/ | 10:06 |
| romcheg | lucasagomes: wow | 10:07 |
| romcheg | thanks! | 10:07 |
| lucasagomes | romcheg, np... heh I think people don't review much things in ironic | 10:08 |
| lucasagomes | lemme review that patch | 10:08 |
| lucasagomes | in wsme | 10:08 |
| lucasagomes | ** | 10:08 |
| romcheg | yup | 10:08 |
| romcheg | For the first look I can see that it touches Content-Type but doesn't touch URI extensions | 10:09 |
| lucasagomes | yeah it makes the content type configurable | 10:11 |
| lucasagomes | romcheg, hmmm yeah true... so ".xml" would still return an xml? lemme test | 10:11 |
| romcheg | If Accept is not configured, it checks for the URI extension (.xml or .json) | 10:12 |
| lucasagomes | ic | 10:16 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Use GB instead of MB for swap https://review.openstack.org/83788 | 10:17 |
| *** coolsvap is now known as coolsvap|afk | 10:38 | |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Return error immediately if set_console_mode is not supported https://review.openstack.org/90376 | 10:43 |
| *** mkerrin has joined #openstack-ironic | 11:01 | |
| *** andreykurilin has joined #openstack-ironic | 11:26 | |
| andreykurilin | lucasagomes, hi | 11:27 |
| andreykurilin | lucasagomes, can you approve my patch?:) https://review.openstack.org/#/c/72418/ | 11:28 |
| dtantsur | FYI: got a new page for tracking testing on Fedora: https://etherpad.openstack.org/p/IronicTestingOnFedora | 11:36 |
| dtantsur | ifarkas, lucasagomes, adam_g ^^^ | 11:41 |
| dtantsur | I got tempest/scenario/test_baremetal_basic_ops.py pass on F20 \o/ | 11:48 |
| agordeev | dtantsur: congrats! | 11:53 |
| openstackgerrit | Ghe Rivero proposed a change to openstack/ironic: Clean oslo dependencies files https://review.openstack.org/90670 | 12:00 |
| ifarkas | dtantsur, nice, that's pretty cool! | 12:08 |
| *** killer_prince has quit IRC | 12:13 | |
| *** rameshg87 has left #openstack-ironic | 12:19 | |
| dtantsur | ifarkas, lucasagomes could you have a look at https://review.openstack.org/#/c/85387/ again? | 12:19 |
| ifarkas | dtantsur, sure, I will take a look on that | 12:20 |
| dtantsur | also, if there any cores around, could you land https://review.openstack.org/#/c/73223/ ? This is testing refactoring and it's here since Feb 13. | 12:21 |
| dtantsur | lucasagomes, romcheg ^^^ ? | 12:21 |
| *** jdob has joined #openstack-ironic | 12:26 | |
| *** lazy_prince2 has joined #openstack-ironic | 12:36 | |
| *** lazy_prince has quit IRC | 12:36 | |
| *** lazy_prince2 is now known as lazy_prince | 12:37 | |
| *** martyntaylor has left #openstack-ironic | 12:40 | |
| *** killer_prince has joined #openstack-ironic | 12:42 | |
| *** rloo has joined #openstack-ironic | 13:04 | |
| lucasagomes | andreykurilin, hey, I just tested that patch, there's some problems with it :) | 13:06 |
| lucasagomes | :( | 13:06 |
| lucasagomes | andreykurilin, [stack@localhost python-ironicclient]$ ironic node-list | 13:07 |
| lucasagomes | print_list() got an unexpected keyword argument 'sortby' | 13:07 |
| *** jbjohnso has joined #openstack-ironic | 13:07 | |
| andreykurilin | lucasagomes, hm...how could I miss that.. i will fix it soon. thanks:) | 13:13 |
| lucasagomes | andreykurilin, np... I will review again after the fix -- apart from that the patch lgtm | 13:14 |
| openstackgerrit | Andrey Kurilin proposed a change to openstack/python-ironicclient: Reuse module `cliutils` from common code https://review.openstack.org/72418 | 13:17 |
| *** matty_dubs|gone is now known as matty_dubs | 13:18 | |
| andreykurilin | lucasagomes: already fixed:) | 13:18 |
| andreykurilin | lucasgomes: `grep` did not find more "sortby" argument, except two places that you have been found | 13:21 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Add ManagementInterface https://review.openstack.org/86063 | 13:27 |
| lucasagomes | andreykurilin, awesome :D thank u! | 13:27 |
| lucasagomes | we really need to add tests to the shell commands in the client | 13:28 |
| lucasagomes | to capture those errors :/ | 13:28 |
| lucasagomes | (not as part of ur patch tho) | 13:28 |
| NobodyCam | good morning Ironic | 13:32 |
| matty_dubs | Morning NobodyCam | 13:32 |
| NobodyCam | morning matty_dubs :) | 13:33 |
| lucasagomes | morning NobodyCam matty_dubs :) | 13:33 |
| lucasagomes | how was the weekend? | 13:33 |
| NobodyCam | morning lucasagomes ... was good | 13:33 |
| NobodyCam | :) | 13:33 |
| *** rloo has quit IRC | 13:33 | |
| NobodyCam | thou I may not be able to be fully here for the jam this morning | 13:33 |
| matty_dubs | lucasagomes: Short! | 13:33 |
| *** rloo has joined #openstack-ironic | 13:34 | |
| NobodyCam | There is very bad weather around here | 13:34 |
| romcheg | Morning matty_dubs, NobodyCam | 13:38 |
| NobodyCam | morning romcheg | 13:39 |
| *** rloo has quit IRC | 13:41 | |
| *** rloo has joined #openstack-ironic | 13:41 | |
| lucasagomes | heh matty_dubs +1 | 13:42 |
| lucasagomes | NobodyCam, oh :( saturday was pretty bad here as well, but sunday was incredible sunny! | 13:42 |
| lucasagomes | andreykurilin, the missing tests is pretty much all the shell modules tests heh | 13:43 |
| *** lazy_prince has quit IRC | 13:44 | |
| andreykurilin | lucasagomes: :-D I will not promise to increase coverage to 100%, but i'll add some new tests:) | 13:45 |
| NobodyCam | lucasagomes: http://weather.weatherbug.com/GA/Atlanta-weather/severe-weather/local-alerts.html | 13:45 |
| lucasagomes | andreykurilin, heh sure! Thank you | 13:46 |
| lucasagomes | NobodyCam, oh :( | 13:46 |
| NobodyCam | been watching the weather channel all weekend :-p | 13:49 |
| lucasagomes | heh, hope things get better over there before the summit | 13:50 |
| openstackgerrit | David Shrewsbury proposed a change to openstack/ironic: Implement instance rebuild in nova.virt.driver https://review.openstack.org/90429 | 13:50 |
| NobodyCam | we may have a window to get into Loyisiana this morning ... if we do we're going to take it | 13:51 |
| NobodyCam | lol Rv's and Tornado's don't mix well .. so we are doing our best to avoid them | 13:52 |
| NobodyCam | Louisiana even :-p... /me needs more coffeee | 13:52 |
| matty_dubs | LOL I thought that was an attempt to immitate a southern accent or something | 13:54 |
| NobodyCam | :) | 13:56 |
| NobodyCam | nope just lack of coffee | 13:56 |
| Shrews | FYI, the "Baremetal" filter topic for openstack-dev is now "Baremetal-Ironic" and will now filter email subjects with [baremetal] or [ironic]. | 13:59 |
| NobodyCam | nice | 13:59 |
| matty_dubs | Silly question -- how do those filters work? | 14:04 |
| matty_dubs | I always just filtered in my mail client based on subject | 14:04 |
| Shrews | matty_dubs: you have to log in to http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev and you can select which filters you want | 14:05 |
| matty_dubs | Oh, nice! I had no idea that was a thing | 14:06 |
| Shrews | that link is at the bottom of every email sent to the ML | 14:06 |
| Shrews | :) | 14:06 |
| matty_dubs | I've always just received *all* of openstack-dev | 14:06 |
| matty_dubs | Ha, I knew about the mailman page, but not that you could filter there | 14:06 |
| Shrews | it's handy | 14:06 |
| NobodyCam | http://vortex.accuweather.com/adc2004/pub/includes/columns/newsstory/2014/650x366_04280447_ap685400658243.jpg :-p | 14:06 |
| Shrews | NobodyCam: hope that's not you! | 14:07 |
| NobodyCam | its not | 14:07 |
| * Shrews imagines bagels strewn everywhere | 14:07 | |
| NobodyCam | just flashed that on the weather channel | 14:07 |
| NobodyCam | lol | 14:07 |
| NobodyCam | not the Bagels | 14:07 |
| *** rwsu has joined #openstack-ironic | 14:08 | |
| *** jgrimm has joined #openstack-ironic | 14:13 | |
| *** linggao has joined #openstack-ironic | 14:16 | |
| *** shakamunyi has joined #openstack-ironic | 14:29 | |
| -openstackstatus- NOTICE: Gerrit downtime for upgrade begins in 90 minutes. See: https://wiki.openstack.org/wiki/GerritUpgrade | 14:30 | |
| NobodyCam | oh that is goin to impact the review jam | 14:32 |
| lucasagomes | NobodyCam, ping re onva resource tracker... if I have two nodes, both of them have 512 ram, 10 gb disk, 1 cpu, the resource tracker should AUDIT 1024 ram, 20gb disk, 2 cpus, am I right? | 14:32 |
| NobodyCam | *CORES* ^^^^^^^^^^^ | 14:32 |
| lucasagomes | ouch :/ | 14:32 |
| NobodyCam | lucasagomes: yes I think so .. but that is kinda not true | 14:33 |
| *** mdenny has joined #openstack-ironic | 14:33 | |
| lucasagomes | NobodyCam, right... I'm taking a look at it, cause it doesn't | 14:33 |
| lucasagomes | it seems to pick only one node to report right now | 14:33 |
| * lucasagomes is taking a look | 14:34 | |
| openstackgerrit | Dmitry Tantsur proposed a change to openstack/ironic: Consider Glance checksum when caching master images https://review.openstack.org/90390 | 14:35 |
| openstackgerrit | Dmitry Tantsur proposed a change to openstack/ironic: Implement more robust caching for master images https://review.openstack.org/85387 | 14:35 |
| *** zdin0bot has joined #openstack-ironic | 14:40 | |
| openstackgerrit | Aleksandr Gordeev proposed a change to openstack/ironic-python-agent: Fix unexpected stevedore traceback in tests https://review.openstack.org/90756 | 14:40 |
| *** killer_prince is now known as lazy_prince | 14:43 | |
| *** shakamunyi has quit IRC | 14:44 | |
| *** lazy_prince is now known as killer_prince | 14:46 | |
| NobodyCam | brb | 14:46 |
| openstackgerrit | Dmitry Tantsur proposed a change to openstack/ironic: Consider Glance checksum when caching master images https://review.openstack.org/90390 | 14:46 |
| devananda | g'morning, all | 14:50 |
| dtantsur | devananda, morning | 14:50 |
| matty_dubs | Howdy devananda | 14:51 |
| lucasagomes | devananda, morning | 14:53 |
| devananda | NobodyCam: hmm, gerrit downtime is an hour from now. | 14:54 |
| devananda | lucasagomes: no -- rt will audit TWO compute nodes, each with 512mb | 14:54 |
| lucasagomes | devananda, right... because when I see the AUDIT logs | 14:55 |
| lucasagomes | devananda, it seems to be reporting 1 node only | 14:55 |
| lucasagomes | 2014-04-28 15:55:17.690 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 512 | 14:56 |
| lucasagomes | once that node get's deploying, not AUDIT is shown anymore | 14:56 |
| NobodyCam | devananda: good morning | 14:56 |
| lucasagomes | devananda, or, if that node has no properties, the rt is reporting: | 14:57 |
| lucasagomes | 2014-04-28 15:57:02.248 AUDIT nova.compute.resource_tracker [req-5e51f002-7eb0-4e80-a413-977909693ef5 None None] Free ram (MB): 0 | 14:57 |
| lucasagomes | 2014-04-28 15:57:02.248 AUDIT nova.compute.resource_tracker [req-5e51f002-7eb0-4e80-a413-977909693ef5 None None] Free disk (GB): 0 | 14:57 |
| lucasagomes | 2014-04-28 15:57:02.248 AUDIT nova.compute.resource_tracker [req-5e51f002-7eb0-4e80-a413-977909693ef5 None None] Free VCPU information unavailable | 14:57 |
| devananda | lucasagomes: right. we've a bug there. try adam_g's patch | 14:58 |
| devananda | lucasagomes: it should report 0 resources for nodes taht are deployed and errored | 14:58 |
| devananda | lucasagomes: not hide them (which is the current broken behavior) | 14:58 |
| lucasagomes | devananda, right lemme try with his patch (I think i did but it's the same) | 14:59 |
| lucasagomes | 1 sec | 14:59 |
| lucasagomes | devananda, yup same | 15:00 |
| lucasagomes | devananda, I will open a bug about it | 15:00 |
| devananda | :( thanks | 15:02 |
| lucasagomes | devananda, https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py | 15:02 |
| lucasagomes | ops https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L455 | 15:02 |
| lucasagomes | https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/driver.py#L324 | 15:02 |
| lucasagomes | they seems to be using get_host_status, which is the sum of the resources of all nodes avaiable | 15:03 |
| lucasagomes | perhaps we should do the same in Ironic | 15:03 |
| rloo | hi, is there a bug bash? | 15:03 |
| *** shakamunyi has joined #openstack-ironic | 15:04 | |
| rloo | err, a review jam? | 15:04 |
| devananda | rloo: hi! I'm getting the etherpad cleaned up | 15:04 |
| devananda | https://etherpad.openstack.org/p/IronicReviewDay | 15:04 |
| lucasagomes | rloo, we didn't start yet | 15:04 |
| rloo | ok, let me know when you are starting :-) | 15:04 |
| *** killer_prince is now known as lazy_prince | 15:05 | |
| devananda | lucasagomes: yes, but are those returning stats for the instances on the current (local)host? | 15:06 |
| devananda | lucasagomes: or for the whole (federated) cluster? | 15:06 |
| Shrews | hrm, with latest master branch in devstack, i'm now getting: "oslo.config.cfg.DuplicateOptError: duplicate option: rpc_backend" | 15:06 |
| lucasagomes | Shrews, delete the .pyc from the openstack/common/rpc/ | 15:07 |
| devananda | Shrews: i was getting that a bit last week, but then updated everything and it's fine | 15:07 |
| *** pradipta has joined #openstack-ironic | 15:07 | |
| devananda | that -- and git pull in every dir in /opt/stack/ | 15:07 |
| lucasagomes | it's because of the oslo.message that now deletend all the openstack/common/rpc/* stuff, but you need to clean ur git repo to get rid of the *pyc | 15:07 |
| lucasagomes | devananda, I think on the localhost... I'll investigate | 15:08 |
| lucasagomes | devananda, the problem is that ironic doesn't have a way to filter the nodes that is being managed by a specific host (from outside, internally we do) | 15:09 |
| Shrews | lucasagomes: thx | 15:09 |
| lucasagomes | Shrews, np | 15:09 |
| openstackgerrit | A change was merged to openstack/ironic: Fix bypassed reference to node state values https://review.openstack.org/88403 | 15:10 |
| devananda | lucasagomes: well, and nova-compute is managing *all* ironic nodes, in a sense | 15:12 |
| devananda | lucasagomes: and with >1 n-cpu, they are all managing all nodes | 15:12 |
| devananda | so - folks who want to do reviews -- i'm adding things here: https://etherpad.openstack.org/p/IronicReviewDay | 15:13 |
| devananda | feel free to, you know, review, discuss, etc :) | 15:13 |
| lucasagomes | devananda, hmm, right | 15:14 |
| lucasagomes | devananda, I opened the bug (https://bugs.launchpad.net/ironic/+bug/1313779) will do some investigation | 15:15 |
| lucasagomes | at least it doesn't affect the deployment, I deployed both nodes and the scheduler pick them and all | 15:15 |
| lucasagomes | it just the audit that looks broke | 15:15 |
| openstackgerrit | David Shrewsbury proposed a change to openstack/ironic: Implement instance rebuild in nova.virt.driver https://review.openstack.org/90429 | 15:19 |
| rloo | devananda: https://review.openstack.org/#/c/88711/3/ironic/common/driver_factory.py. Did you see these comments? | 15:19 |
| devananda | rloo: thanks. no :) | 15:23 |
| rloo | lucasagomes: https://review.openstack.org/#/c/86063/12/ironic/drivers/base.py. That whole issue about the node being avail in task. | 15:24 |
| devananda | hm, I dont know if tripleo is enabling pxe_ssh yet | 15:24 |
| *** viktors is now known as viktors|afk | 15:24 | |
| rloo | lucasagomes: there was a patch awhile ago to remove the node. but did we ever decide/clean up task so that it had exactly one node? | 15:25 |
| *** yuriyz has quit IRC | 15:25 | |
| lucasagomes | rloo, it was something that me devananda and comstud were discussing last week | 15:27 |
| lucasagomes | rloo, right now we don't use multiple-node locks and will be refactored out of the taskmanager | 15:27 |
| lucasagomes | rloo, I also opened a but about the redundant 'node' parameter in the driver interfaces to track the work of removing it from the interface | 15:27 |
| rloo | lucasagomes: is there a bug open about refactoring taskmanager? | 15:28 |
| lucasagomes | rloo, https://bugs.launchpad.net/ironic/+bug/1312632 | 15:28 |
| lucasagomes | rloo, hmm not that I know | 15:28 |
| openstackgerrit | Jarrod Johnson proposed a change to stackforge/pyghmi: Correct sensor offset for byte 5 state values https://review.openstack.org/90778 | 15:28 |
| lucasagomes | devananda, comstud ^ there's one? | 15:28 |
| *** rloo has quit IRC | 15:28 | |
| *** rloo has joined #openstack-ironic | 15:29 | |
| rloo | lucasagomes: https://review.openstack.org/#/c/61859/. | 15:29 |
| lucasagomes | rloo, I see, will link that bug with this one | 15:30 |
| -openstackstatus- NOTICE: Gerrit downtime for upgrade begins in 30 minutes. See: https://wiki.openstack.org/wiki/GerritUpgrade | 15:30 | |
| *** rloo has quit IRC | 15:31 | |
| *** rloo has joined #openstack-ironic | 15:31 | |
| rloo | lucasagomes: thx. i think there should be a bug to refactor taskmanager too | 15:31 |
| lucasagomes | rloo, +1 yeah, idk if comstud is work on that... if not I can give it a go | 15:32 |
| *** dwalleck has joined #openstack-ironic | 15:32 | |
| rloo | lucasagomes: thx. otherwise if the refactoring isn't done, i think removing the node arg is somewhat wrong. | 15:33 |
| lucasagomes | rloo, :( I can put it back, devananda any opnion here ^ ? | 15:33 |
| * lucasagomes really wants that base interface to get merged to unlock the rest of the work | 15:33 | |
| rloo | lucasagomes: i am fine with the change as long as there is a note about other change/bug. | 15:34 |
| lucasagomes | rloo, I left a comment on that patch, as a TODO, opening a bug and referencing there would be fine? | 15:34 |
| lucasagomes | rloo, ack | 15:34 |
| rloo | lucasagomes: yeah, fine with me anyway. I seem to recall where we had such a change in something else ;) | 15:34 |
| lucasagomes | :D | 15:35 |
| *** zdin0bot has quit IRC | 15:35 | |
| *** zdin0bot has joined #openstack-ironic | 15:35 | |
| devananda | a) comstud's refactoring of taskmanager is great. b) we should make taskmanager only take one node, since that's all it actually ever does today. c) we can remove the node param from most driver interfaces. but yes, it should be tracked somewhere | 15:36 |
| devananda | oh, and b') any multi-node operations we might need to eventually do won't be do-able with the originally planned approach because that was not written with the hash ring in mind | 15:37 |
| devananda | so we'd have to refactor that code anyway, and basically want to pass multiple locks around (rather than multiple nodes in a single lock) | 15:38 |
| *** coolsvap|afk is now known as coolsvap | 15:38 | |
| devananda | also, wouldnt' it be neat if taskmanager used taskflow lib at some point? :) | 15:39 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/ironic: Remove 'fake' and 'ssh' drivers from default enabled list https://review.openstack.org/88711 | 15:39 |
| lucasagomes | devananda, hmm I dunno much taskflow, but sounds a bit overkill | 15:39 |
| rloo | thx devananda. so we need to make sure there is a bug to track the refactoring of taskmanager. the other stuff is being tracked. | 15:41 |
| rloo | ha ha, one day, taskflow will be in everything ;) | 15:41 |
| lucasagomes | :) | 15:42 |
| *** zdin0bot has quit IRC | 15:53 | |
| *** eghobo has joined #openstack-ironic | 15:57 | |
| Shrews | So, why isn't the full disk space being reported for 'df -k' here? http://pastebin.com/dfB454QC | 16:00 |
| Shrews | for vda3, that is | 16:00 |
| Shrews | that should be with a 1G ephemeral and 9G root | 16:02 |
| * devananda goes afk for a bit | 16:04 | |
| lucasagomes | Shrews, ? /dev/vda1 1032088 34052 945608 3% /mnt | 16:07 |
| lucasagomes | ^ that's ur ephemeral being reported by df -k | 16:07 |
| Shrews | lucasagomes: yeah, that's correct. but vda3? | 16:07 |
| lucasagomes | /dev/vda3 174536 10132 155405 6% / | 16:07 |
| lucasagomes | ^ | 16:07 |
| lucasagomes | ohhh | 16:07 |
| Shrews | that's no where near 9G | 16:07 |
| *** rloo has quit IRC | 16:08 | |
| *** rloo has joined #openstack-ironic | 16:08 | |
| lucasagomes | looking | 16:08 |
| Shrews | this is my flavor def: http://paste.openstack.org/show/77470/ | 16:08 |
| lucasagomes | Shrews, can u run, df -Th | 16:09 |
| lucasagomes | for amore human readble unit sizes | 16:09 |
| *** Mikhail_D_ltp has joined #openstack-ironic | 16:10 | |
| Shrews | lucasagomes: -T not supported on cirros, but: http://paste.openstack.org/show/77471/ | 16:10 |
| lucasagomes | ah | 16:10 |
| lucasagomes | cirros :( | 16:10 |
| lucasagomes | yeah | 16:10 |
| lucasagomes | Shrews, so the fs is that size because the image is that size | 16:12 |
| lucasagomes | we are not resizing the filesystem | 16:12 |
| lucasagomes | we only dd that image to the partition and that's it | 16:12 |
| Shrews | lucasagomes: so this is intentional then? | 16:13 |
| *** matty_dubs is now known as matty_dubs|lunch | 16:13 | |
| lucasagomes | Shrews, thinking... | 16:13 |
| lucasagomes | Shrews, tools like cloud-init have a growroot tools to do it | 16:14 |
| lucasagomes | Shrews, idk whether we should use those tools, or make ironic resize it | 16:14 |
| Shrews | i mean, that's fine, afaic. i just wanted to know the reason :) | 16:14 |
| lucasagomes | Shrews, right, yeah the reason is that we don't resize the fs to occupy the whole disk after copying the image to the disk... now I dunno whether ironic should do it or not | 16:15 |
| lucasagomes | resizing fs's is not the most neat thing to do :D | 16:15 |
| lucasagomes | although growing the fs should be fine | 16:15 |
| lucasagomes | the whole partition* | 16:16 |
| lucasagomes | devananda, do u think we should resize the root partition fs after copying the image to the disk? ^ | 16:16 |
| Shrews | imo, i would think a user would expect the full space to be available (for log growth and what not). | 16:18 |
| lucasagomes | I think as well... but dunno whether we should use ironic to do it or rely on tools like cloud-init to do so | 16:19 |
| Shrews | at the very least, it's confusing for simple minds such as myself :) | 16:19 |
| lucasagomes | imo I don't see problem in having ironic to do it | 16:19 |
| *** vkozhukalov has quit IRC | 16:19 | |
| Shrews | ah, yeah. i don't know which one should handle it. | 16:20 |
| lucasagomes | Shrews, yeah, because, if we want to resize we probably need to introspect in the image to see what fs it's use | 16:21 |
| lucasagomes | and then use ntfsresize for ntfs | 16:21 |
| lucasagomes | resize2fs for linux fs's | 16:21 |
| lucasagomes | etc... | 16:21 |
| * Shrews lunches | 16:27 | |
| devananda | ironic shouldn't do that, in part because init does it, and in part because the user may not want that | 16:32 |
| devananda | Shrews: if you build your image from dib (rather than using cirros) you should get one that auto-resizes | 16:33 |
| devananda | for tempest testing, i think cirros is probably sufficient (after all, that's what the rest of devstack's tests use) | 16:33 |
| JayF | I've seen cloud-init resize filesystem/partitions to fill disks properly on hardware (with the needed patches to read configdrive off a partition, which are upstream now iirc) | 16:33 |
| devananda | JayF: exactly | 16:33 |
| JayF | and I agree that we should let tools like cloud-init do that work for us on first boot | 16:33 |
| lucasagomes | +1 | 16:35 |
| lucasagomes | so let cloud-init do it for us | 16:35 |
| -openstackstatus- NOTICE: Gerrit is unavailable until further notice for a major upgrade. See: https://wiki.openstack.org/wiki/GerritUpgrade | 16:36 | |
| *** ChanServ changes topic to "Gerrit is unavailable until further notice for a major upgrade. See: https://wiki.openstack.org/wiki/GerritUpgrade" | 16:36 | |
| *** newell_ has joined #openstack-ironic | 16:40 | |
| *** ifarkas has quit IRC | 16:41 | |
| *** shakamunyi has quit IRC | 16:46 | |
| *** ndipanov is now known as ndipanov_gone | 16:47 | |
| *** jistr has quit IRC | 16:54 | |
| *** derekh has quit IRC | 16:54 | |
| *** lazy_prince is now known as killer_prince | 16:54 | |
| *** harlowja_away is now known as harlowja | 17:00 | |
| *** lucasagomes is now known as lucas-afk | 17:09 | |
| *** matty_dubs|lunch is now known as matty_dubs | 17:09 | |
| *** dwalleck has quit IRC | 17:13 | |
| *** shakamunyi has joined #openstack-ironic | 17:13 | |
| *** hemna has joined #openstack-ironic | 17:19 | |
| *** dwalleck has joined #openstack-ironic | 17:24 | |
| *** stephenpearson has quit IRC | 17:32 | |
| *** shakamunyi has quit IRC | 17:35 | |
| *** killer_prince is now known as lazy_prince | 17:38 | |
| *** lazy_prince is now known as killer_prince | 17:38 | |
| *** lazy_prince2 has joined #openstack-ironic | 17:38 | |
| *** eghobo has quit IRC | 17:44 | |
| hemna | j #openstack-infra | 17:44 |
| hemna | bah | 17:44 |
| *** eghobo has joined #openstack-ironic | 17:44 | |
| *** john-n-seattle1 has quit IRC | 17:44 | |
| *** john-n-seattle has joined #openstack-ironic | 17:49 | |
| matty_dubs | dtantsur: Hey, have you run into "AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED" on F20 + devstack? | 17:51 |
| matty_dubs | It seems to have fallen over and now I can't even find the service | 17:51 |
| dtantsur | matty_dubs, is it Fedora? | 17:52 |
| matty_dubs | Yeah, 20 | 17:52 |
| *** datajerk has joined #openstack-ironic | 17:52 | |
| *** athomas has quit IRC | 17:52 | |
| dtantsur | $ systemctl status rabbitmq-server.service ? | 17:52 |
| matty_dubs | Bingo! | 17:53 |
| matty_dubs | For some reason I wasn't finding that | 17:53 |
| dtantsur | ok, and what is there? | 17:53 |
| matty_dubs | Apr 17 16:32:03 dhcp-57-107.bos.redhat.com rabbitmqctl[18699]: Stopping and halting node 'rabbit@dhcp-57-107' ... | 17:53 |
| matty_dubs | For who-knows-why | 17:53 |
| matty_dubs | Lemme try to restart | 17:54 |
| matty_dubs | I think what threw me off is that bash-completion wasn't finding it, but was showing other r* stuff | 17:54 |
| dtantsur | 1. try to restart; 2. I experiences troubles with some lock file out of nothing, so I had to: yum remove rabbitmq-server && ./unstack.sh && ./stack.sh | 17:54 |
| dtantsur | * experiences = experienced | 17:54 |
| matty_dubs | Oh, wow | 17:55 |
| dtantsur | I just rebuild my devstack today, so it should work, but this problem does occur sometimes | 17:56 |
| matty_dubs | Seems set now, though now I need to restart some other services that appear to have just stopped trying when AMQP went down | 17:56 |
| matty_dubs | Kind of concerning that it randomly stopped/ | 17:56 |
| jrist | disable_service rabbit | 17:56 |
| jrist | enable_service qpid | 17:56 |
| jrist | in my localrc | 17:56 |
| dtantsur | jrist, that may work, but not necessary out-of-box with Ironic | 17:57 |
| *** coolsvap is now known as coolsvap|afk | 17:57 | |
| dtantsur | default for openstack is still rabbit, I think | 17:57 |
| jrist | confirm | 17:57 |
| *** tatyana has joined #openstack-ironic | 17:58 | |
| dtantsur | but I actually tried both and both worked for me (qpid in instack, rabbit in devstack) | 17:58 |
| matty_dubs | dtantsur: BTW, did you say you were looking at beating selinux into submission? I still have to "setenforce 0" to get anything working. | 17:59 |
| matty_dubs | which makes dwalsh weep: http://stopdisablingselinux.com/ | 17:59 |
| dtantsur | matty_dubs, what's you problems? Can you create a separate section in https://etherpad.openstack.org/p/IronicTestingOnFedora and fill in your experience? | 18:00 |
| dtantsur | matty_dubs, I'm trying to have a single source of information on what work/does not work on Fedora | 18:00 |
| matty_dubs | To use Horizon you have to disable SELinux: $ sudo setenforce 0 | 18:00 |
| matty_dubs | ^ also my experience | 18:00 |
| matty_dubs | I forget if there was more that didn't work | 18:01 |
| dtantsur | matty_dubs, ah, yeah, and that's hard to fix | 18:01 |
| matty_dubs | I tried to hit the dashboard early-on to test | 18:01 |
| *** lazy_prince2 has quit IRC | 18:01 | |
| dtantsur | problem is horizon tries to access directories that httpd should not access according to SEL | 18:01 |
| *** killer_prince is now known as lazy_prince | 18:01 | |
| dtantsur | I do want to fix it eventually, and any ideas are much appreciated | 18:02 |
| *** vkozhukalov has joined #openstack-ironic | 18:05 | |
| *** epim has joined #openstack-ironic | 18:07 | |
| *** dwalleck has quit IRC | 18:11 | |
| *** datajerk has quit IRC | 18:12 | |
| *** Mikhail_D_ltp has quit IRC | 18:13 | |
| *** Mikhail_D_ltp has joined #openstack-ironic | 18:15 | |
| *** gmatefi has joined #openstack-ironic | 18:16 | |
| *** datajerk has joined #openstack-ironic | 18:17 | |
| Shrews | devananda: ping | 18:19 |
| devananda | Shrews: pong | 18:20 |
| Shrews | devananda: so, i'm pretty happy with the new rebuild() in 90429. seems to be working well. should i be attempting to validate the type of image requested in the rebuild? e.g., not a whole disk image or whatever | 18:21 |
| *** jbjohnso has quit IRC | 18:21 | |
| Shrews | i think there was a conversation about that recently | 18:22 |
| devananda | Shrews: awesome. good question. well, we don't have full-disk image support in ironic yet, but please raise that on that bp/review | 18:22 |
| devananda | Shrews: because full disk obviously won't support --preserve | 18:22 |
| devananda | but it could be rebuilt | 18:22 |
| devananda | and eg a SAN volume re-attached | 18:23 |
| *** pradipta is now known as pradipta_away | 18:35 | |
| lucas-afk | wow cool now it will be possible to edit the commit message directly on gerrit? | 18:37 |
| Shrews | lucas-afk: i was *just* noticing that. neat | 18:38 |
| lucas-afk | yeah super handy | 18:38 |
| Shrews | and thank goodness that horrific WIP button that mordred wanted so much has been eradicated! that developer should have been eradicated with it | 18:41 |
| mordred | Shrews: :) | 18:42 |
| mordred | Shrews: the feature is now on the change voting page, fwiw | 18:42 |
| jroll | we're still meeting today, yes? | 18:42 |
| Shrews | mordred: i know ;) | 18:43 |
| rloo | jroll: yes | 18:44 |
| jroll | cool :) | 18:44 |
| *** agordeev2 has joined #openstack-ironic | 18:47 | |
| *** martyntaylor has joined #openstack-ironic | 18:48 | |
| *** lucas-afk is now known as lucasagomes | 18:54 | |
| *** romcheg1 has joined #openstack-ironic | 18:54 | |
| *** romcheg1 has quit IRC | 18:55 | |
| *** romcheg1 has joined #openstack-ironic | 18:55 | |
| *** martyntaylor has left #openstack-ironic | 18:56 | |
| *** mrda_away is now known as mrda | 18:57 | |
| mrda | Good morning Ironic | 18:58 |
| *** max_lobur has joined #openstack-ironic | 18:58 | |
| *** dkehn__ is now known as dkehnx | 19:04 | |
| *** openstackgerrit has quit IRC | 19:04 | |
| *** epim has quit IRC | 19:21 | |
| *** epim has joined #openstack-ironic | 19:23 | |
| *** dwalleck has joined #openstack-ironic | 19:24 | |
| *** ChanServ changes topic to "OpenStack Bare Metal Provisioning | Docs: http://docs.openstack.org/developer/ironic/ | Bugs: https://bugs.launchpad.net/ironic | Status: https://etherpad.openstack.org/p/IronicWhiteBoard" | 19:31 | |
| -openstackstatus- NOTICE: Gerrit upgrade to 2.8 complete. See: https://wiki.openstack.org/wiki/GerritUpgrade Some cleanup tasks still ongoing; join #openstack-infra if you have any questions. | 19:31 | |
| *** zdiN0bot has joined #openstack-ironic | 19:57 | |
| *** dwalleck has quit IRC | 19:59 | |
| *** jbjohnso has joined #openstack-ironic | 20:00 | |
| *** dwalleck has joined #openstack-ironic | 20:00 | |
| jroll | 13:00:49 yuriyz | python-agent does not have any authorization for commands, is this good? <- we assume the agent will run on a trusted network | 20:01 |
| *** yuriyz has joined #openstack-ironic | 20:01 | |
| jroll | 13:00:49 yuriyz | python-agent does not have any authorization for commands, is this good? <- we assume the agent will run on a trusted network | 20:01 |
| JayF | I personally wouldn't use an agent whose job is to wipe disks and rewrite firmwares on an untrusted network -- even if there was some authorization built in :) | 20:01 |
| yuriyz | on separate vlan? | 20:02 |
| *** iron1 has joined #openstack-ironic | 20:02 | |
| devananda | jroll: to be clear, IPA (will) use(s) a signed URL when POSTing to ironic-api service | 20:02 |
| devananda | jroll: but doesn't auth commands it receives | 20:02 |
| devananda | jroll: is that the question / point? | 20:02 |
| *** lucasagomes is now known as lucas-dinner | 20:02 | |
| jroll | devananda: I think so? | 20:03 |
| yuriyz | and agent should be authorized on Ironic API side | 20:03 |
| jroll | devananda: IMO this should be deployed on a secure network and so I'm not worried about auth | 20:03 |
| devananda | gmatefi: morgabra: so... IIUC, neutron has a concept of PIF and VIF | 20:04 |
| devananda | they're a bit mangled for ironic because one PIF has 0 or 1 VIF | 20:04 |
| jbjohnso | afternoon | 20:04 |
| JayF | ipa_api_url is being passed in via kernel cli params in our test environment right now, and honestly, if someone manages to steal the IP of our API node on the trusted network or override the values I'm sending over PXE, I have bigger problems than agent auth :) | 20:05 |
| devananda | *and that VIF has an externally determiend MAC | 20:05 |
| devananda | whereas for usual (virtualized) case, one PIF may have many VIF, all with fake mac addresses | 20:05 |
| jbjohnso | devananda, I hear I should describe confluent | 20:05 |
| devananda | JayF: how is ipa talking to ir-api without auth? | 20:05 |
| morgabra | devananda: I don't think neutron has a pif/vif model anywhere I've seen | 20:05 |
| gmatefi | normally, neutron ML2 plugins perform a lookup using binding:host_id, this works for vSwitch attached ports. So, here the point is that the Neutron port needs to be directly bound to TOR ports, rather than to vSwitch ports | 20:05 |
| jroll | devananda: for testing we're running with noauth | 20:06 |
| morgabra | maybe it's specified on the port models? | 20:06 |
| *** datajerk has quit IRC | 20:06 | |
| yuriyz | should agent send query to Ironic API w/o keystone token if keystone auth enabled? | 20:06 |
| jroll | devananda: in the future I was planning on adding those urls to the public urls thing | 20:06 |
| jroll | that makes them un-authd | 20:06 |
| *** jgrimm has quit IRC | 20:06 | |
| *** tatyana has quit IRC | 20:06 | |
| yuriyz | jroll, yes we need this | 20:07 |
| jroll | yuriyz: need what, specifically? | 20:07 |
| *** dwalleck_ has joined #openstack-ironic | 20:08 | |
| yuriyz | add nodeless passthru to public api | 20:08 |
| jroll | oh, right | 20:08 |
| jroll | I think that's a pecan config | 20:08 |
| devananda | morgabra: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L599 | 20:08 |
| jroll | which brings up another thing - is that supposed to be editable by operators? | 20:08 |
| devananda | jroll: yikes. please dont do that. | 20:09 |
| jroll | yuriyz: my plan is to add the /lookup and /heartbeat endpoints here. does that work for you? https://github.com/openstack/ironic/blob/master/ironic/api/config.py#L32 | 20:09 |
| jroll | devananda: right, that's what I thought | 20:09 |
| devananda | jroll: what the agent POSTs back to ironic-api needs to be auth'd | 20:09 |
| yuriyz | I dont like idea place keystone token on tftp or http server for agent | 20:09 |
| jroll | devananda: I'm not sure I like that | 20:10 |
| devananda | jroll: we pass keystone token via tftp to the agent today -- it's a bad hack and insecure. | 20:10 |
| yuriyz | I think this is ok | 20:10 |
| jroll | yuriyz: I agree | 20:10 |
| jroll | right | 20:10 |
| devananda | jroll: IMO we should be passing a signed temp url to the agent | 20:10 |
| jroll | and by 'not sure I like that' I mean 'I don't see a good way to do that' | 20:10 |
| jroll | if there is an agent imposter on the network | 20:10 |
| jroll | how does that stop them? | 20:10 |
| jroll | assuming that is the problem we're trying to solve here | 20:10 |
| gmatefi | IMHO the correct solution would be involvement of the TPM module, i.e building trust relationship at node discovery/registration, and rely on TPM signature after | 20:11 |
| jbjohnso | devananda, so anyway, confluent is the console/hardware management project I've been doing incorporating pyghmi's support | 20:11 |
| devananda | jbjohnso: gotcha. new project | 20:11 |
| morgabra | devananda: I'm pretty sure the vif/pif is a nova thing, I'll have to look into how that maps to neutron calls | 20:11 |
| morgabra | part of what I'm figuring out this week | 20:11 |
| jbjohnso | devananda, https://www.youtube.com/watch?v=G_lDaktYnsQ | 20:12 |
| *** dwalleck has quit IRC | 20:12 | |
| jbjohnso | that's the service in action | 20:12 |
| JayF | gmatefi: I'd be very -1 to anything requiring TPM chips in nodes ipa is running on | 20:12 |
| *** yuriyz has quit IRC | 20:12 | |
| jbjohnso | gmatefi, jroll fyi I had thoughts on that matter | 20:14 |
| jbjohnso | basically, pluggable hard authentication schemes | 20:14 |
| gmatefi | JayF: definetly not a generally required solution due to extra TPM costs, only works in cases when the operator requires enhanced security | 20:14 |
| jbjohnso | SOL and bridge filtered protocols are two schemes I can think of easy enough to enable | 20:14 |
| jbjohnso | well SOL or SEL actually.. | 20:15 |
| jbjohnso | SOL or SEL: authenticate by way of BMC | 20:15 |
| jbjohnso | Switch: authenticate by way of switch | 20:15 |
| jbjohnso | SOL/SEL requires BMC and trust relationship established that way | 20:15 |
| jbjohnso | Switch method requires a managed switch supporting the right mibs and would break in an architecture that let virtual machines get bridge filter packets out | 20:16 |
| jroll | I simply don't understand the point of authenticating in either direction between ironic and IPA. Anyone running this should have VLANs set up and switched between in such a way that users and ironic will never be on the same network. | 20:16 |
| jbjohnso | I know off hand that the linux bridge does not let such packets good | 20:16 |
| jroll | if a user can connect to the management network, it's game over, no matter the authentication scheme | 20:16 |
| JayF | It's hard for me to imagine any situation where running the agent on anything but a private, secured, and trusted network seems like a good idea. | 20:16 |
| jroll | unless the auth scheme is some sort of manual process | 20:16 |
| jbjohnso | well for me, if a scehme is 'free' to implement (cfg-wise) I'd be all up for further risk mitigation | 20:17 |
| *** shakamunyi has joined #openstack-ironic | 20:17 | |
| devananda | JayF: will you -1 UEFI support too? | 20:17 |
| lifeless | devananda: morning | 20:17 |
| JayF | devananda: UEFI support as an option is good. Requirement for specific hardware (like a TPM chip, or a *requirement* for UEFI boot), seems like a bad idea | 20:18 |
| jroll | jbjohnso: I just don't see how that would not be broken. I'd be thrilled to be proved wrong. | 20:18 |
| devananda | JayF: you can't totally trust the firmware after a tenant has been on there without TPM/UEFI means you can't trust the whole network any more | 20:18 |
| devananda | lifeless: hi there! | 20:18 |
| jbjohnso | devananda, UEFI support should be easy. Shouldn't require so much as a finger lifted by user to enable.. | 20:18 |
| lifeless | devananda: openflow can mitigate that | 20:18 |
| JayF | devananda: I don't believe that's true, if you have the proper protections in place on the hardware. | 20:18 |
| devananda | lifeless: funny thing, we were talking about that | 20:19 |
| lifeless | :) | 20:19 |
| devananda | lifeless: integration between netron and ToRs's | 20:19 |
| jbjohnso | devananda, even with that you can't trust after a tenant.. | 20:19 |
| gmatefi | JayF: having a separate ironic VLAN per node sounds interesting to control boot server access and so the token, but it is not really a well-scalable solution | 20:19 |
| iron1 | iron1: fyi- my team is workingon uefi support for ironic | 20:19 |
| * mrda thinks the summit session on multi-tennancy should be interesting :) | 20:19 | |
| devananda | lifeless: and then JayF pointed out that the agent should only be run on a fully open trusted network, and therefor doesn't need any auth | 20:19 |
| jroll | mrda: :) | 20:19 |
| jbjohnso | devananda, TPM or other secure firmware protection scheme still doesn't in practice extend to the entirety of firmware in a system | 20:20 |
| JayF | gmatefi: I was thinking about that being awesome as well, but impractical in most deployment schemes. | 20:20 |
| devananda | gmatefi: vlan-per-node destroys any scalable deployment model | 20:20 |
| devananda | s/destroys/prevents/ | 20:20 |
| lifeless | devananda: uh, I'd want a lot stronger statement than that to discard defense in depth. | 20:20 |
| jbjohnso | jroll, well, I think I'd shy away from it being declared 'untrusted tenant ready' (that's a whole other bunch of problems), but having someone untrusted on the VLAN is something to be avoided, but should not be game over.. in theary | 20:21 |
| devananda | lifeless: same here | 20:21 |
| lifeless | devananda: our security folk would be rolling in the aisles laughing at the idea of no-auth | 20:21 |
| JayF | jbjohnso: devananda: The only way to trust firmware after a untrusted tenant has been on there is to require signatures on all user servicable hardware or remove the capability to modify the firmware | 20:21 |
| devananda | uh huh | 20:21 |
| devananda | JayF: precisely | 20:21 |
| devananda | JayF: and i'd like the agent to support that | 20:21 |
| devananda | doesn't need to require it -- but it does need t osupport it | 20:21 |
| JayF | devananda: That sounds more like something the hardware would support than having IPA support it? | 20:22 |
| jbjohnso | is theer any intent of supporting things without a BMC/SEL? | 20:22 |
| devananda | jbjohnso: without a BMC ? | 20:22 |
| jbjohnso | devananda, yeah, just thinking about auth mechanisms | 20:22 |
| lifeless | note that 'modifythe firmware' may be undocumented. E.g. the hard drive attack. | 20:22 |
| jbjohnso | devananda, e.g. push public key via SEL entries | 20:22 |
| lifeless | I don't believe there is *any* hardware available to day which can be trusted after an untrusted tenant has been on it | 20:23 |
| devananda | JayF: ironic and IPA needs to support the hardware aspects of a trusted boot chain, eventually, IMNSHO | 20:23 |
| jbjohnso | lifeless, correct. Though practically speaking, that won't stop some people sadly | 20:23 |
| devananda | lifeless: i agree. and yet there are customers who buy jsut that from some vendors | 20:23 |
| lifeless | devananda: I know | 20:24 |
| lifeless | devananda: for some workloads it may not matter | 20:24 |
| jbjohnso | so anyway, in the case of untrusted admins on your vlan | 20:24 |
| lifeless | jbjohnso: huh, assume other machines may get compromised | 20:24 |
| lifeless | jbjohnso: untrusted admins is a different case again | 20:24 |
| jbjohnso | lifeless, well, generically, anyone with admin capbilities for a NIC attached to a vlan | 20:25 |
| lifeless | ok | 20:25 |
| lifeless | so every trunked port | 20:25 |
| lifeless | and when VMs are allowed to attach to provider networks, possibly arbitrary VMs too | 20:25 |
| jbjohnso | it would be wise to at least try to mitigate that risk | 20:25 |
| devananda | 20:16:17 < jroll> I simply don't understand the point of authenticating in either direction between ironic and IPA. | 20:26 |
| jbjohnso | so, PXE is of course a blatant risk in that chain, however you slice it | 20:26 |
| morgabra | devananda: so, I'd like to make a high-level document to write down goals and talk about design for neutron concerns. Etherpad? Wiki? | 20:26 |
| JayF | lifeless: jbjohnso: I disagree with the assertion that it doesn't exist today. It however, does not exist on most off the shelf hardware. That's why I'd say most of the security risk and mitigation has to be done at the hardware engineering / supply chain level, not in Ironic (other than integrations with firmware flashing, etc) | 20:26 |
| devananda | jroll: i'd like to go back to that (i think we got into the weeds talkign about firmware) | 20:26 |
| morgabra | I'm having a hard time keeping it straight in my head | 20:26 |
| devananda | morgabra: ++ | 20:26 |
| jroll | right | 20:26 |
| lifeless | JayF: oh? We could use that for CI :) | 20:26 |
| devananda | morgabra: feel free to start one and then add a link here so we can track it https://etherpad.openstack.org/p/IronicWhiteBoard | 20:27 |
| jbjohnso | JayF, I'd be srurpsied at any hardware that really has total 100% protection. E.g. no ability to write to any non-volatile chip. No HDD firmware, no nic firmware... | 20:27 |
| JayF | lifeless: I think russell_h has said that Rackspace might be willing to provide some 3rd party CI for some agent integrations in the future. :) Not quite there yet though. | 20:27 |
| jbjohnso | anyhow, authentication will be non-conventional, but I was planning to do certificate grants based on exactly such non-conventional paths | 20:27 |
| devananda | jroll: leaving a service wide open because you _assume_ the network is trusted does not make any sense to me | 20:27 |
| jroll | devananda: sure | 20:28 |
| jroll | devananda: let's back up | 20:28 |
| jroll | devananda: how do you see this auth working? | 20:28 |
| devananda | lifeless: you'll be interested in http://openstacksummitmay2014atlanta.sched.org/event/754c3678d31f9f74e020b9a1e6f4dece | 20:28 |
| JayF | devananda: I guess it's hard to imagine how passing credentials of any type -- including a signed URL -- through a pxe or ipxe config could be secure, if anyone on that network could spoof the mac and get that configuration value as well. | 20:28 |
| jroll | in such a way that if someone is on the provisioning vlan they cannot break it? | 20:28 |
| jroll | what JayF said | 20:28 |
| matty_dubs | I guess it could at least break replay attacks | 20:29 |
| matty_dubs | BUt that doesn't help if an attacker gets it first | 20:29 |
| lifeless | jroll: I think you're asking the wrong question | 20:29 |
| jroll | lifeless: go on | 20:29 |
| devananda | JayF: for one, it makes it more difficult for a potential attacker (but, sure, not impossible) | 20:29 |
| morgabra | https://etherpad.openstack.org/p/Ironic_Neutron_Integration | 20:29 |
| jbjohnso | jroll, well, with *PXE* and assuming no good filtering, it will be breakable | 20:29 |
| lifeless | jroll: you seem to be asking 'is it possible [today] to be entirely safe' and then using that to say 'don't secure at all' | 20:29 |
| lifeless | jroll: the one doesn't imply the other | 20:29 |
| morgabra | I'm going to try to condense today's discussion and add links to everything | 20:30 |
| JayF | jbjohnso: there is no 'good filtering' for pxe, except that in the ToR switch or simialr | 20:30 |
| jbjohnso | jroll, but the break point is good to be very nicely well defined | 20:30 |
| lifeless | jroll: *clearly* PXE is spoofable | 20:30 |
| JayF | jbjohnso: you can lock down by MAC, but MACs are trivial to spoof | 20:30 |
| morgabra | and try to guess on some next steps :P | 20:30 |
| jroll | lifeless: if it's trivial to break the auth, then why bother | 20:30 |
| jbjohnso | JayF, right, I was specifically thinking in the switch, but not trusting | 20:30 |
| lifeless | jroll: but other initial boot mechanisms like iLO are out of band and securable | 20:30 |
| devananda | morgabra: thanks much! | 20:30 |
| *** agordeev2 has quit IRC | 20:30 | |
| jbjohnso | JayF, jroll I was picturing more https based stuff injected in a secure, authenticated manner | 20:30 |
| lifeless | jroll: making the rest of the architecture weak because the only generic standard we have so far is weak is a poor idea IMO | 20:30 |
| jroll | lifeless: just like I'd just as soon write plain text passwords to a DB as write MD5'd passwords | 20:31 |
| jbjohnso | jroll, so the idea would be, have an authentication scheme with a common chokepoint | 20:31 |
| jroll | lifeless: I'm not putting a dependency on 3rd party propietary boot mechanism | 20:31 |
| lifeless | jroll: not a helpful analogy sorry | 20:31 |
| lifeless | jroll: not asking you to | 20:31 |
| devananda | jroll: pass a secure token via OOB channels, then verify it in the POST back to ironic-api | 20:31 |
| devananda | jroll: there's no passing ofkeys along an unsecured channel necessary | 20:31 |
| lifeless | jroll: just saying I'm not going to support putting end to end insecure code into Ironic Its indefensible | 20:31 |
| jbjohnso | devananda, you can even avoid passing private keys at all | 20:31 |
| jroll | devananda: what sort of oob channels? | 20:31 |
| devananda | jroll: and no need to trust the whole network | 20:31 |
| devananda | jroll: via the BMC | 20:32 |
| *** dwalleck_ has quit IRC | 20:32 | |
| *** Mikhail_D_ltp has quit IRC | 20:32 | |
| jbjohnso | jroll, e.g. virtual usb boot device | 20:32 |
| jroll | devananda: then you're just putting the security on the OOB network | 20:32 |
| jroll | devananda: same idea no? | 20:32 |
| devananda | jroll: most vendors have a means to do this. iLO driver does it via virtual-floppy | 20:32 |
| devananda | jroll: right! no, not at all | 20:32 |
| *** zdiN0bot has quit IRC | 20:32 | |
| jbjohnso | jroll, and you have a stateful, mutually authenticated scheme | 20:32 |
| devananda | OOB network *must* be trusted. and it *must* be physically isloated | 20:32 |
| jbjohnso | jroll, the client can auth the BMC and the BMC can auth the client. PXE lacks that ability | 20:32 |
| jbjohnso | devananda, well, it doesn't *have* to be, though the documentation should pretend it is | 20:33 |
| lifeless | jroll: what we need is something that is secure *after* the PXE boot step has happened | 20:33 |
| jbjohnso | devananda, the BMCs with properly strong keys are resilient to unath | 20:33 |
| lifeless | jroll: and then we can work on secureing the PXE boot step as another action item | 20:33 |
| devananda | jbjohnso: all of them, from every vendor, even if they haven't been updated recently? .... | 20:34 |
| jbjohnso | devananda, I said 'can be' not 'always' | 20:34 |
| lifeless | jroll: for instance, using openflow to prevent arp and ip spoofing / observing from ports on the deployment lan | 20:34 |
| jbjohnso | devananda, in other words, OOB network should be secured as if you have untrusted guys on it, but probably should keep untrusted guys off of it | 20:35 |
| *** zdiN0bot has joined #openstack-ironic | 20:35 | |
| jbjohnso | lifeless, right, which is one reason I think public key exchange rather than private key injection would be nicer | 20:35 |
| jroll | IMO, even with a good authentication scheme, if your provisioning network is owned, you're owned. | 20:35 |
| devananda | jroll: that's still not an argument to make it _insecure_ by design | 20:36 |
| jroll | I see the point of using that authentication | 20:36 |
| jbjohnso | lifeless, e.g. if you blessed only certain ports for udp ports from ports 4011 or 69, that would suffice even if ip spoofing were possible | 20:36 |
| jroll | using some authentication * | 20:36 |
| devananda | right. on that, i'm goign to break for lunch | 20:36 |
| jbjohnso | jroll, well, the point being the authentication at worst doesn't have to do harm | 20:36 |
| devananda | haven't eaten in ~6hrs | 20:36 |
| jroll | right | 20:36 |
| jbjohnso | jroll, even if you don't trust it, it's mitigating risk | 20:36 |
| jroll | what I'm saying is | 20:36 |
| jroll | putting that authentication in, is not a high priority in the short term | 20:37 |
| jroll | for us. | 20:37 |
| jbjohnso | well, it might be nice even if the bootstrap for the auth is currently weak to go through the motions | 20:37 |
| jbjohnso | so that when you go to really mean it, you don't have to rework it as much, but I dunno | 20:38 |
| jroll | that's fair | 20:38 |
| JayF | I think we have a strong preference for getting things working first, then iterating on things. We're still in the 'get things working' stage, and I think most of us over here viewed the auth pieces as one of the things we'd all iterate on | 20:38 |
| jroll | ^ | 20:39 |
| JayF | we're still talking about an ironic driver that's so far under development it still only exists in a gerrit review request :) | 20:39 |
| jroll | heh | 20:40 |
| jbjohnso | right, I don't know if you have other accesses, but for our rework of HPC provisioning, we are taking a number of analagously weak directives and requiring them to be strongly authenticated with a single 'weaken if you want' point of entry | 20:40 |
| jbjohnso | it may be that in this architecture you only have one to start with anyway, so having two phase may not make sense | 20:41 |
| *** jgrimm has joined #openstack-ironic | 20:41 | |
| *** epim has quit IRC | 20:44 | |
| lifeless | jbjohnso: I don't understand the blessing of ports thing | 20:45 |
| jroll | jbjohnso: I hear ya | 20:46 |
| *** epim has joined #openstack-ironic | 20:49 | |
| *** dwalleck_ has joined #openstack-ironic | 20:49 | |
| jbjohnso | for us the first step is getting a normal looking authentication credential from something weird | 20:49 |
| jbjohnso | lifeless, belssing of ports? | 20:49 |
| jbjohnso | lifeless, well, for the xCAT team, the schemes were going to be remote media injected, BMC attested (serial or SEL, haven't decided, SEL if possible probably), and switch attested | 20:50 |
| jbjohnso | e.g. the FSM os deployment does remote media injected boot payload and keys | 20:51 |
| jbjohnso | and uses https for all OS payload and callbacks | 20:51 |
| jbjohnso | boot payload being ~50 megabytes today, next product will have ~700 KB payloads over remote media | 20:52 |
| jbjohnso | oh, the PXE thing | 20:52 |
| jbjohnso | so if the ToR switch forbids DHCPOFFERs/ProxyDHCPOFFERs from anything but 'blessed' ports | 20:53 |
| jbjohnso | and tftp port... | 20:53 |
| jbjohnso | and if it done correctly (in practice I'm skeptical that one should trust any blacklist scheme if possible), then the integrity of the payload should be fine | 20:54 |
| jbjohnso | though confidentiality might not be | 20:54 |
| lifeless | jbjohnso: well you're saying that even if ip spoofing is possible | 20:54 |
| lifeless | jbjohnso: and with PXE the spoofer would MITM | 20:54 |
| jbjohnso | so in that sort of scenario, a PXE scheme where a node generates it's own private key and then advertises said public key in a nice way | 20:54 |
| lifeless | jbjohnso: so if you have that, I think you can just get end to end confidentiality on your ethernet :) | 20:55 |
| jbjohnso | lifeless, well, if they sucsesfully spoofed the deployment server ip but could not serve tftp or dhcp or proxydhcp, they couldn't molest the integrity of stuff (though they could DoS) | 20:55 |
| jbjohnso | lifeless, no, not confidentiality | 20:55 |
| jbjohnso | lifeless, integrity | 20:55 |
| *** dwalleck__ has joined #openstack-ironic | 20:55 | |
| lifeless | jbjohnso: yes, I saw what you said. I'm saying I think we can go further. | 20:56 |
| lifeless | jbjohnso: but your point seems to be that a small bit of network support gets us integrity | 20:56 |
| jbjohnso | lifeless, well, I'm just trying to imagine the least secure setup that *could* theoretically be secure | 20:56 |
| lifeless | and if we have integrity we can indeed do DH + SSL from the machines | 20:56 |
| jbjohnso | lifeless, so ideally, we don't ship shared secret, passphrase, or private keys | 20:56 |
| jbjohnso | and node generates its own private key and then publishes it in a way that can't be tampered with | 20:57 |
| lifeless | jbjohnso: thats DH :) | 20:57 |
| jbjohnso | e.g. via BMC or bridge filtered protocol. | 20:57 |
| lifeless | jbjohnso: once you have integrity a lot of things have simple existing answers | 20:57 |
| lifeless | simple-to-use, not simple-to-invent :) | 20:57 |
| *** zdiN0bot has quit IRC | 20:58 | |
| jbjohnso | lifeless, well, integrity within a very limited protocol set | 20:58 |
| *** gmatefi has quit IRC | 20:58 | |
| *** dwalleck_ has quit IRC | 20:59 | |
| jbjohnso | lifeless, which is why I go to things like pushing CSR over SEL, serial, or switch | 20:59 |
| jbjohnso | lifeless, to get the point where you could do client certificate TLS, for example. | 20:59 |
| *** jdob has quit IRC | 20:59 | |
| jbjohnso | I'd shy away from putting an auth token in somewhere like SEL, that was not necessarily assumed by the designers to be a secret storage | 21:00 |
| jbjohnso | but public key, who cares | 21:01 |
| lifeless | so I'd like to be able to set ingress filters on ports | 21:03 |
| lifeless | even trunked ones | 21:03 |
| lifeless | to just say 'unicast frames to X,Y,Z macs only' | 21:03 |
| lifeless | and egress filters likewise, for the source mac | 21:03 |
| jbjohnso | well, that's not good either | 21:03 |
| jbjohnso | you'd have to do port based | 21:03 |
| lifeless | that gets you integrity and visibility, no ? | 21:04 |
| jbjohnso | macs can be faked | 21:04 |
| lifeless | right, which is what neutron is all about :) | 21:04 |
| lifeless | jbjohnso: doesn't matter if they fake the mac. thats what 'filters on portsts' is for. | 21:04 |
| lifeless | bah, 'filters on ports' | 21:04 |
| jbjohnso | yeah, but I just like simpler filtering rules | 21:04 |
| jbjohnso | if the rule can be 'thou shalt only be able to send source udp port 4011, 69, or 57 if thou art port 3 on switch1' | 21:05 |
| jbjohnso | I think that's the simplest rule I can imagine in 'vmap' syntax that would facilitate the bare minimum payload delivery integrity | 21:06 |
| jbjohnso | before switching to more conventional stuff like TLS | 21:06 |
| jbjohnso | err 67 not 57 | 21:06 |
| jbjohnso | then you could put a public key in and then trust that payload to perform a trick without some untrusted puppetmaster mucking the bits | 21:07 |
| jbjohnso | all switches would have a rule blanket blocking that, except the switch with blessed port | 21:08 |
| lifeless | jbjohnso: so you'd limit yourself to one deployment at a time ? | 21:09 |
| lifeless | jbjohnso: or you're saying you're limiting only the Ironic conductor ? | 21:09 |
| jbjohnso | only the deployment server(s) | 21:10 |
| jbjohnso | you could extend rules to cover more servers, but deployment targets can go hog wild because they don't use those source ports | 21:10 |
| jbjohnso | when it comes to things like security, my trust of more layers is low | 21:11 |
| *** matty_dubs is now known as matty_dubs|gone | 21:11 | |
| lifeless | jbjohnso: me too - but I have many birds in this area I want to kill | 21:12 |
| jbjohnso | anyway, so if a scheme abstractly doesn't ever send a private key, then I think it has the highest chance of working in the most constrained environments | 21:12 |
| lifeless | jbjohnso: I want to increase isolation beween bm tenants | 21:12 |
| jbjohnso | with the lowest chance of being completely screwed over | 21:12 |
| lifeless | jbjohnso: which I think, done well, has the room to secure PXE quite thoroughly too | 21:12 |
| lifeless | jbjohnso: a rogue port in *either* scheme will completely compromise the thing | 21:13 |
| lifeless | jbjohnso: I am not arguing against your scheme btw | 21:13 |
| lifeless | jbjohnso: its compatible with what I want to do at the switch plumbing layer | 21:13 |
| lifeless | jbjohnso: tripleo @ HP has someone starting soon on networking work, so we may see some traction soon | 21:14 |
| jbjohnso | right, and practically speaking, even if the deployment does all the right things, the deployed OS usage is likely to be lax on security within a vlan | 21:14 |
| lifeless | another reason for port level security on the switches :) | 21:15 |
| jbjohnso | but anyway, I was just trying to describe what I think is the worst case that isn't a complete lost cause | 21:15 |
| lifeless | I think its a good design principle | 21:15 |
| jbjohnso | e.g. it's not absolutely mandatory that a remote media is the only way, but remote media would do things dramatically different | 21:15 |
| *** dwalleck__ has quit IRC | 21:15 | |
| jbjohnso | remote media is more one way, so deploying a private key into the payload is the logical step. But with PXE, that would not be kosher | 21:16 |
| jbjohnso | so if possible, any context should generate their own private key | 21:16 |
| jbjohnso | the trick being how to deliver the public key unto the authentication framework for blessing | 21:17 |
| jbjohnso | which could be serial, SEL, or LLDP | 21:17 |
| jbjohnso | at least that's all I can think of that wouldn't be crazy odd | 21:17 |
| JayF | how would you deliver public key via lldp? | 21:17 |
| JayF | that's really strange | 21:17 |
| *** max_lobur has quit IRC | 21:17 | |
| jbjohnso | JayF, I have to go, but you can send any ascii data over LLDP | 21:18 |
| lifeless | I'd like tohave one lowest common denominator way | 21:18 |
| lifeless | thats secure and efficient | 21:18 |
| jbjohnso | JayF, system description you can put base64 encode stuff | 21:18 |
| lifeless | bootstrap a SSL session somehow | 21:18 |
| lifeless | and then we're good | 21:18 |
| jbjohnso | serial might be tricky for esxi and windows, don't know | 21:18 |
| JayF | jbjohnso: that's certainly interesting, but not sure how it's more secure than trusting the vlan configured by the switch :) | 21:18 |
| jbjohnso | SEL is easy | 21:18 |
| jbjohnso | JayF, well, the risk would be whether the guests can get LLDP through | 21:18 |
| jbjohnso | the BMC is probably the closest thing to a ubiquitous solution | 21:18 |
| JayF | wait, you'd have the agent itself send the lldp? | 21:19 |
| lifeless | we could encrypt to something only the target machine knows | 21:19 |
| lifeless | e.g. machine serial # | 21:19 |
| jbjohnso | but have to establish trust with BMC (set the password in a trusted context) | 21:19 |
| devananda | lifeless: OOB delivery seems the LCD, even though it lacks a generic ipmlementation, every vendor has one | 21:19 |
| jbjohnso | lldp can do anything | 21:19 |
| jbjohnso | but the switch has to support it and do lldp-mib for that to be useless | 21:19 |
| JayF | jbjohnso: except transmit data across layer 2 boundries, which would render it less useful for some deployments | 21:19 |
| * devananda returns from lunch and tries to catch up | 21:19 | |
| jbjohnso | devananda, what LCD? | 21:19 |
| *** vkozhukalov has quit IRC | 21:19 | |
| devananda | lowest common denominator | 21:19 |
| devananda | sorry | 21:19 |
| lifeless | devananda: what about encrypting with a passphrase that the agent can pickup from the machine; | 21:19 |
| lifeless | devananda: for the OOB case the passphrase can be passed OOB | 21:19 |
| devananda | lifeless: if the agent is not used for enrollment, that would be OK. | 21:20 |
| jbjohnso | devananda, switch filtering on PXE with SEL or Serial I think could be a contender for LCD | 21:20 |
| lifeless | devananda: for the in-band case where there's no such thing on the machine, put it in the kernel line | 21:20 |
| jbjohnso | anyway, I'm out for the day | 21:20 |
| lifeless | devananda: we'd need to be able to enroll e.g. the serial # in Ironic | 21:20 |
| lifeless | jbjohnso: ciao | 21:20 |
| jbjohnso | devananda, not all have remote media, but almost all switches can do filtering *if* configured right | 21:21 |
| devananda | lifeless: stashing the SN is easy. discovering it means trusting the initial (discovery) agent | 21:21 |
| lifeless | devananda: or the user's CMDB supplies it | 21:22 |
| devananda | jbjohnso: right, but most server vendors do. | 21:22 |
| devananda | lifeless: yep. if present :) | 21:22 |
| jroll | is it safe to assume that untrusted code running on a box can access any data that the agent running on the same box could access? | 21:25 |
| *** jbjohnso has quit IRC | 21:25 | |
| jroll | e.g. data stored in the BMC | 21:25 |
| lifeless | its accurate, yes. | 21:26 |
| lifeless | except again with some non-commodity hardware | 21:26 |
| jroll | of course | 21:27 |
| lifeless | where you can sign into a chip and you need that secret to ask it questions | 21:27 |
| *** epim has quit IRC | 21:27 | |
| lifeless | I wouldn't depend on that myself | 21:27 |
| jroll | well, what I was getting at is that you can't store an authentication secret in the BMC | 21:27 |
| *** linggao has quit IRC | 21:28 | |
| *** iron1 has quit IRC | 21:29 | |
| devananda | jroll: the purpose of passing an auth secret into the host via the BMC is to set up a trusted connection thereafter, and prevent MITM attacks from subsequent communication with that host | 21:30 |
| devananda | jroll: if the host is already compromized (eg, poisoned firmware) there's nothing you can do besides unrack it | 21:30 |
| jroll | devananda: sure | 21:31 |
| jroll | devananda: you would have to remove that secret from the bmc at deploy time, yeah? | 21:31 |
| devananda | jroll: saying, essentially, "you can't trust the BMC if the host is poisoned" is orthogonal to securing the network traffic on the IB networks | 21:31 |
| jroll | no, I understand | 21:32 |
| devananda | jroll: right. which is the plan for passing keys via the OOB channel | 21:32 |
| jroll | ok | 21:32 |
| devananda | jroll: and one reason why I think using the SN isn't great... but it's better than nothing | 21:32 |
| JayF | Would there be a method for passing keys around via the OOB channel if the OOB channel doesn't support virtual media? | 21:32 |
| jroll | devananda: yeah, I don't like the SN thing at all, too predictable | 21:32 |
| JayF | Like there's nothing in the DCMI or IPMI standards that allow arbitrary storage of data in the BMC to be read out from the OS? | 21:33 |
| JayF | (I don't think there is, but imbw) | 21:33 |
| devananda | JayF: possibly... i haven't looked at it because most server-grade hardware has virtual media support | 21:33 |
| devananda | JayF: i'm not aware of that in the IPMI spec, but jbjohsno would be the one to ask | 21:33 |
| JayF | Yeah I don't think it is in the DCMI spec, which is usually the least common denominator for that stuff. | 21:35 |
| * JayF has definately worked with hardware that doesn't support virtual media via bmc | 21:36 | |
| devananda | hemna: hi! around? | 21:36 |
| hemna | heyas | 21:44 |
| *** romcheg1 has quit IRC | 21:46 | |
| devananda | hemna: question on cinder that google hasn't been able to answer for me | 21:50 |
| hemna | ok I can try and answer :) | 21:51 |
| devananda | hemna: can i specify a raid topology for my cinder volumes when booting an instance? | 21:51 |
| hemna | The best you can do is, depending on the driver, specify the RAID topology in the volume type | 21:51 |
| hemna | which then applies that volume type at volume creation time. | 21:51 |
| hemna | the 3PAR drivers support specifying a CPG (which has the raid configuration) during volume creation time via a volume type. | 21:52 |
| devananda | CPG? | 21:52 |
| hemna | it's a 3PAR thing | 21:52 |
| devananda | i see | 21:52 |
| devananda | so cinder is exposing driver implementation via the API ?:( | 21:52 |
| hemna | the raid config is on a CPG on the 3PAR. Think of a CPG as a pool | 21:53 |
| hemna | so you can specify the CPG name in the volume type | 21:53 |
| devananda | ah, gotcha | 21:53 |
| hemna | and that is done by the cinder admin | 21:53 |
| hemna | not by a user. | 21:53 |
| *** openstackgerrit has joined #openstack-ironic | 21:54 | |
| devananda | so I can pick a volume from a set of available storage pools, some of which may have been build ahead of time at different raid levels | 21:54 |
| hemna | a user can specify which volume type they want to use at volume creation time. At the user will not know what's in the volume type. | 21:54 |
| hemna | at attach time yah | 21:54 |
| hemna | the admin creates the volume type, specifies which CPG (Raid level they want) | 21:54 |
| devananda | so in a previous life, i knew many folks who would attach multiple EBS volumes and build a software raid across them | 21:54 |
| hemna | user creates a volume and says, use volume type X. | 21:54 |
| hemna | then attaches that volume. | 21:54 |
| devananda | cause EBS is slow, and striping makes it somewhat faster | 21:55 |
| hemna | no reason why you couldn't do that I suppose | 21:55 |
| hemna | but the arrays are really good at doing RAID internally | 21:55 |
| devananda | gotcha | 21:56 |
| devananda | but cinder doesn't expose any API for that | 21:56 |
| devananda | ? | 21:56 |
| hemna | no | 21:57 |
| devananda | hemna: k, thx. the background here is that it would be great if we could leverage the cinder API to specify | 21:57 |
| devananda | RAID layout of local storage when Ironic is provisioning a node | 21:57 |
| hemna | hrmm | 21:57 |
| devananda | it sounds like we could fake that | 21:57 |
| hemna | so yah, the best you get is passing the volume type | 21:58 |
| hemna | at volume creation time. | 21:58 |
| devananda | create an ironic driver for cinder, define some common storage pools | 21:58 |
| hemna | and however the admin setup and configured that volume type means whatever raid level. | 21:58 |
| devananda | which are really just stubs that pass a RAID spec to the provisioning mechanism | 21:58 |
| devananda | which then builds the raid locally on that node | 21:58 |
| hemna | it gets you want you want basically. | 21:58 |
| devananda | JayF: ^ | 21:59 |
| JayF | I've been reading along. Sadly I have exactly no knowledge of cinder, so I'd have to do a bit of research to know what exactly you mean :) | 21:59 |
| JayF | so you're saying the 'driver' would be in the agent, and we'd ship a json file describing the raid array to the agent to make it so on disk? | 22:00 |
| JayF | er, I mean, the driver would interface with the agent | 22:00 |
| devananda | JayF: the cinder "driver" would create the vol spec that gets passed (via nova) (via ironic) to the agent | 22:00 |
| JayF | that sounds reasonable to me, and fits with vk's provisioning api bp | 22:01 |
| devananda | yep | 22:01 |
| devananda | my point is that we should leverage cinder and the existing user APIs for vol creation and attachment | 22:01 |
| devananda | rather than add anything to Ironic's API for that | 22:01 |
| hemna | the trick with volume types in cinder is, the key/value pairs in the volume types are specific to the volume driver it's assigned to. | 22:01 |
| JayF | although I still think we should get Ironic master + IPA booting a server before putting too much effort/design into features like raid | 22:01 |
| JayF | does cinder talk partitioning, or just raid? | 22:01 |
| devananda | JayF: ++ | 22:01 |
| hemna | cinder and it's API doesn't really have a concept of RAID | 22:02 |
| hemna | just volume types | 22:02 |
| hemna | which are key/value pairs | 22:02 |
| hemna | that get passed to the volume drivers | 22:02 |
| devananda | JayF: the arbitrary partition stuff in that BP disturbes me. It's going way beyond the scope of Ironic | 22:02 |
| JayF | devananda: If I had my wish, all images would come with partitions baked in and no part of ironic would be in the business of partitioning | 22:03 |
| JayF | but if the line isn't there, I'm not sure where it should be drawn | 22:04 |
| devananda | hemna: I keep hoping we can avoid implementing, in ironic, a descriptive language for storage layout | 22:04 |
| devananda | hemna: that really seems like it belongs in cinder | 22:04 |
| hemna | yah | 22:04 |
| jroll | devananda: but would that mean importing from cinder to parse the layout? | 22:04 |
| jroll | also, where do you see the line with partitioning? | 22:04 |
| hemna | so if Ironic is going to ask cinder to create volumes, then ironic just needs to be able to pass a volume type along with the creation request | 22:05 |
| hemna | the layout (raid levels, qos levels) is up to the volume driver itself, at creation time. | 22:06 |
| devananda | JayF: simle. ironic only supports the partition layout described by nova (root, swap, ephemeral). both swap and ephemeral are optional. the guest can utilize any extra space as they see fit, ironic won't muck with the GPT until the instance is deleted. | 22:07 |
| devananda | hemna: i think we're talking about slightly different things | 22:07 |
| hemna | ok | 22:07 |
| devananda | hemna: and both are interesting :) | 22:07 |
| JayF | devananda: ephemeral meaning what in the context of nova? | 22:07 |
| devananda | but i thinik we should only do one | 22:07 |
| devananda | a) ironic asks cinder to create a new volume and then attaches it to the instance | 22:07 |
| devananda | b) user wants their instance to be deployed by ironic with a local RAID | 22:08 |
| JayF | devananda: I actually can agree with that, and that means LVM is out of scope... -- unless a more complex paritioning scheme and/or LVM are baked into a raw image written directly to the disk | 22:08 |
| devananda | i think (a) is backwards -- the user should have already created the cinder vol, then boot an instance in ironic, and then attach the cinder vol themselves | 22:08 |
| devananda | (b) is what i've been talking about | 22:08 |
| devananda | JayF: in Nova (and EC2) context, ephemeral volume == volume outside of the instance's virtual disk, comprised of another local virtual disk, which will be lost if the instance is rebooted or migrated to antoehr compute host | 22:09 |
| hemna | for the most part cinder has no idea what is in the volumes it creates, and most of the time are simply empty unpartitioned block devices. | 22:09 |
| hemna | you could partition that volume any way you like and do local RAID at that point. | 22:10 |
| JayF | devananda: okay, that's what I thought it might mean, but it seems backward-ish to even have that as an option, but if that's the way it is that's the way it is :) | 22:10 |
| devananda | hemna: by "you", do you mean a user within the instance to which the vol gets attached? | 22:10 |
| hemna | but that's terribly inefficient for smart arrays that do RAID themselves. | 22:10 |
| hemna | yes | 22:10 |
| devananda | right. inefficient for any dedicated hardware. | 22:11 |
| devananda | and completely irrelevant for ironic's case | 22:11 |
| devananda | since the guest can see any unprovisioned local storage and use mdadm/lvm/etc themselves | 22:11 |
| hemna | yup | 22:11 |
| devananda | so | 22:11 |
| devananda | they _alraedy_ have the same functionality | 22:11 |
| devananda | heh | 22:11 |
| devananda | JayF: this all backs up my point that any _software_ RAID, or partitioning beyond root/swap/ephemeral on the first local disk, is outside of the scope of ironic | 22:14 |
| devananda | JayF: when, in the past, I've spoken about RAID support in Ironic, I was specifically referring to _hardware_ RAID config | 22:14 |
| devananda | which would present all raid'd disks as a single block device to the host OS | 22:15 |
| JayF | devananda: the only reason I'd maybe suggest software RAID should be considered into the scope is that a RAID'd root, without hardware assistance, is impossible without ironic integration | 22:15 |
| JayF | all the other things have some method of workaround; bake LVM or a weird partitioning scheme into your image, etc | 22:15 |
| JayF | but tbh for my use case, I don't think we'd have ironic setup any raid at all, so I don't have a dog in the fight really :) | 22:15 |
| devananda | JayF: if you need software raid, and you bake it into the image, and ironic only writes it to the first disk -- couldn't you just rebuild the second disk after boot? | 22:17 |
| JayF | Hmm. | 22:17 |
| JayF | So the image would contain a disassembled raid array superblock... | 22:17 |
| JayF | that would only work for RAID 1 though | 22:17 |
| devananda | right | 22:18 |
| JayF | not RAID 0,5,6,10 | 22:18 |
| devananda | which is all you want for a raid'd root | 22:18 |
| JayF | I've run more than a few production boxes with all the disks in one partiton, and running RAID 5 (years ago), 6, and 10 | 22:18 |
| JayF | but I think that's a reasonable workaround for most cases | 22:18 |
| devananda | (yes, there is a middle ground where the overhead of slicing off 2 disks is greater than the overhead of sharing root and data traffic on the same raid) | 22:18 |
| JayF | Yeah I've run more than a few 4 disk, raid 10 servers :) | 22:19 |
| JayF | (although most of them had hardware raid, after working with systems for a while, if I were building them today I'd use MD) | 22:19 |
| devananda | anything between 4 and 8 disks, that's usually the case | 22:19 |
| devananda | anything that has a HW BBU and a decent write cache -- use that | 22:20 |
| devananda | and any time you care about performance of your RAID, you buy that | 22:20 |
| devananda | so I think there's really no reason for mdadm here, aside from a 2-disk raid'd root + JBOD and no HW BBU | 22:20 |
| JayF | I don't agree with that philosophy wholly, but it's not very relevant to the Ironic conversation :) | 22:20 |
| devananda | well, it kinda is. you brought it up :) | 22:21 |
| JayF | I'd say it this way, we shouldn't make a value judgement of Hardware RAID > Software RAID, and allow that to impact scoping | 22:21 |
| JayF | that's what I mean when I say it's not relevant | 22:21 |
| devananda | ah | 22:21 |
| devananda | i'm perfectly happy to make that value judgement, fwiw ;) | 22:22 |
| JayF | if it's in scope it's in scope, and the operator should get to choose HW vs MD raid :) | 22:22 |
| JayF | if you're such a fan of h/w raid I'd suspect HP raid implementations must be better than $other_vendor stuff I've used in the past :P | 22:22 |
| * devananda doesn't take the bait | 22:23 | |
| JayF | Oh I'm not trying to bait you, I'm being somewhat serious :) | 22:23 |
| devananda | JayF: you presume i'm making a value judgement based on HP hardware. | 22:24 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions https://review.openstack.org/90233 | 22:24 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Adding a reference driver for the agent https://review.openstack.org/84795 | 22:24 |
| JayF | Hm. I guess I sorta was, although it wasn't intentional | 22:25 |
| devananda | JayF: i'm making a value judgement that hw raid > softwre raid based on performance tuning databases for a living | 22:25 |
| devananda | (in a former life) | 22:25 |
| JayF | my history is much more in monitoring and maintenance than performance tuning, and monitoring statuses across a hetrogenous environment is painful, which is why I've learned to prefer MD in most cases | 22:26 |
| devananda | also a fair point :) | 22:26 |
| *** jgrimm has quit IRC | 22:27 | |
| JayF | Yeah, that's why I'd just say we should determine scope separately from sw vs hw being better, because both sides are valid to have a preference about | 22:27 |
| *** zdin0bot has joined #openstack-ironic | 22:28 | |
| JayF | although I think someone being able to provide a superblock'd raw image, then reassembling on first boot for RAID 1 is a good enough workaround for that use case that I'm very OK tabling the rest until we are a lot further down the road than we are now :) | 22:28 |
| devananda | :) | 22:28 |
| devananda | i agree | 22:29 |
| devananda | and just to help clarify the scope, ironic should prepare the hw and provision the image on it. it should be as independent as possible from the content of that image. | 22:30 |
| devananda | i would be much happier with whole-disk-images | 22:30 |
| devananda | and let them autoresize on first boot | 22:30 |
| JayF | You wouldn't get any argument from me to always require whole-disk-images :) | 22:31 |
| JayF | that's the case the agent was originally written for, and we're using right now | 22:31 |
| devananda | i dont recall at the moment why we moved away from it back in the early days of nova-bm | 22:31 |
| *** dwalleck__ has joined #openstack-ironic | 22:31 | |
| devananda | lifeless: do you? (why nova-bm used partition images rather than whole disk) | 22:31 |
| *** zdin0bot has quit IRC | 22:41 | |
| devananda | ugh, yea, i can't wait to move to a gerrit-based BP review process | 22:45 |
| devananda | JayF: fwiw, there's no way in LP for me to reject a blueprint, so i've commented on the BP and etherpad | 22:46 |
| JayF | devananda: https://review.openstack.org/#/c/86163/ that's the beginnings of the implementation | 22:47 |
| JayF | if you wanted to -1 -2 it with comments there | 22:48 |
| *** zdin0bot has joined #openstack-ironic | 22:48 | |
| devananda | JayF: in that that API allows arbitrary partition creation, yea, it's a step in the wrong direction | 22:51 |
| devananda | JayF: otoh, for partition-images, we do need /some/ support | 22:51 |
| devananda | it's the next patch which scares me | 22:53 |
| *** rloo has quit IRC | 23:01 | |
| *** rloo has joined #openstack-ironic | 23:02 | |
| *** rloo has quit IRC | 23:03 | |
| *** rloo has joined #openstack-ironic | 23:03 | |
| *** lucas-dinner has quit IRC | 23:06 | |
| openstackgerrit | A change was merged to openstack/ironic-python-agent: Check configdrive size before writing to partition https://review.openstack.org/89390 | 23:09 |
| *** eguz has joined #openstack-ironic | 23:10 | |
| *** eghobo has quit IRC | 23:14 | |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Make docker image smaller by using Precise https://review.openstack.org/90845 | 23:18 |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Make docker image smaller by using Precise https://review.openstack.org/90845 | 23:19 |
| *** hemna is now known as hemna__ | 23:22 | |
| Shrews | devananda: Hey. So, lifeless pointed out something with rebuild() re: my requirement to first power down a node before redeploy. I think my initial reasons for doing this have been eliminated since I no longer call spawn() so it technically is no longer required (and I verified rebuild works without power down). However, perhaps it's still a good idea to do so to shutdown anything that might be writing to the ephemeral partition? Thoughts from either | 23:52 |
| Shrews | you or lifeless? | 23:52 |
| *** radsy has joined #openstack-ironic | 23:54 | |
| *** radsy has joined #openstack-ironic | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
