Tuesday, 2014-03-11

comstudthat's probably not my ideal, but I'm not unhappy with it00:00
comstudthe image builder code could be moved into oslo so it can be shared00:01
comstudeven if its small enough to copy for now00:01
russell_h+1 to that too00:01
comstudthis should allow you to avoid any nova changes, really, and get something working more quickly00:03
comstudwhich I think should be one of the goals:)00:03
NobodyCamcomstud: ++ for getting things working quickly.. I like todo's / fixme: pointing out why we diverted from what would apper the "standard" method..00:07
comstudnod00:08
NobodyCam:)00:08
comstudi'm dropping off (i'm technically off today :)00:08
NobodyCam:) after 5 here :)00:08
NobodyCamnight comstud :)00:08
comstudsame here00:09
comstudlaters00:09
russell_hits definitely time to booze00:09
comstudgood idea00:09
* NobodyCam has no beer or cider... looks like i'm down (or up) to Rum & (diet) coke00:10
comstudhorrible problem to have00:10
comstud:)00:10
* comstud & ok out00:11
* jroll throws a beer at NobodyCam00:11
* jroll walks away as well00:11
NobodyCamthank you and night jroll00:11
NobodyCam:)00:11
*** blamar has quit IRC00:17
*** eghobo has quit IRC00:18
*** matsuhashi has joined #openstack-ironic00:21
devanandajroll: same bug tracker in LP is fine00:28
*** devananda changes topic to "OpenStack Bare Metal Provisioning | Docs: http://docs.openstack.org/developer/ironic/ | Bugs: https://bugs.launchpad.net/ironic | Status: https://etherpad.openstack.org/p/IronicWhiteBoard"00:34
*** blamar has joined #openstack-ironic00:34
NobodyCamw00h00 Core 0:         +98.0°C  (high = +86.0°C, crit = +100.0°C)00:37
devanandaNobodyCam: yikes00:42
devanandaNobodyCam: white board cleaned up now00:42
NobodyCamdevananda: ye brick after removing .cache dir00:43
NobodyCamdevananda: awesome00:43
*** lucasagomes has quit IRC00:53
*** jbjohnso has joined #openstack-ironic01:03
*** dwalleck has joined #openstack-ironic01:07
*** blamar has quit IRC01:23
openstackgerritMichael Davies proposed a change to openstack/ironic: Improve Ironic Conductor threading & locks  https://review.openstack.org/7851501:27
*** dwalleck_ has joined #openstack-ironic01:29
*** dwalleck has quit IRC01:33
NobodyCamfyi: removing snd recreating ~/.cache/image-create solved my build woes01:37
*** nosnos has joined #openstack-ironic01:40
*** jbjohnso has quit IRC01:55
*** antonym has quit IRC02:03
*** antonym has joined #openstack-ironic02:03
*** dwalleck_ has quit IRC02:09
devanandaNobodyCam: are you working on / pushing up another rev of 72969?02:09
NobodyCamthe under cloud bits yes02:16
NobodyCamas soon as I can test02:16
NobodyCam:)02:17
rloodevananda: do you have a few minutes to discuss https://bugs.launchpad.net/ironic/+bug/129064502:17
*** lazy_prince is now known as killer_prince02:18
devanandarloo: that's disconcerting02:20
rloodevananda. so the oslo code moved the configs from ../session.py to a new options.py, and in the move, they removed the default setting.02:20
rloodevananda: but they have a method that can be used to set the defaults. Assuming the method works, we can add code to call it.02:21
rloowe actually have code that calls it, in common/config.py02:22
rlooparse_args()02:22
rloobut I am wondering why common/config.py exists. I couldn't find any code that uses it.02:22
rloodevananda: and service.py, prepare_service() has similar code to config.parse_args(). So it seems like prepare_service() should call config.parse_args(), or get rid of config.parse_args() and move code to prepare_service()?02:24
rloodevananda: and/or I can just put up a review, and people can -X it. ha ha.02:24
devanandarloo: sounds like two issues02:25
rloodevananda. yes. Ok, I'll copy the default value from config.parse_args() for now, to service.py file.02:25
devanandarloo: thanks!02:27
devanandarloo: as for common.config vs service.prepare_service, i'm not sure. but they both seem to be in use02:27
devanandaironic/tests/conf_fixture.py:from ironic.common import config02:27
rloodevananda.  ah, I thought I did a grep. must have missed that. thx.02:28
*** derekh has quit IRC02:33
jroll17:28:44      @devananda | jroll: same bug tracker in LP is fine <- cool, thanks. /cc JayF02:40
*** killer_prince is now known as lazy_prince02:44
*** lazy_prince is now known as killer_prince02:45
devanandarloo: i've confirmeed the bug but lowered it to medium -- there's a simple work around (set the connection param)02:49
rloodevananda: yes, that's right. thx.02:50
*** harlowja is now known as harlowja_away02:54
*** harlowja_away is now known as harlowja03:06
*** vkozhukalov has quit IRC03:09
*** dwalleck_ has joined #openstack-ironic03:14
*** dwalleck has joined #openstack-ironic03:16
*** dwalleck_ has quit IRC03:19
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Conductor logs better error if seamicroclient missing  https://review.openstack.org/7952003:24
*** matsuhashi has quit IRC03:41
rloorussell_h: sorry, forgot to mention. wrt database.connection, use connection=sqlite:///ironic/openstack/common/db/ironic.sqlite or set to something else in your config file. (bug 1290645)03:49
*** rloo has quit IRC03:52
*** dwalleck has quit IRC04:19
*** killer_prince is now known as lazy_prince04:24
*** rameshg87 has joined #openstack-ironic04:27
*** matsuhashi has joined #openstack-ironic04:40
*** eghobo has joined #openstack-ironic04:56
*** matsuhashi has quit IRC06:05
*** eghobo has quit IRC06:07
openstackgerritJenkins proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/7886206:07
*** eghobo has joined #openstack-ironic06:08
*** matsuhas_ has joined #openstack-ironic06:08
*** harlowja is now known as harlowja_away06:08
*** eguz has joined #openstack-ironic06:09
*** eghobo has quit IRC06:09
*** eguz has quit IRC06:11
*** romcheg has joined #openstack-ironic06:20
*** bashok has joined #openstack-ironic06:23
*** romcheg has quit IRC06:24
*** vkozhukalov has joined #openstack-ironic06:27
*** killer_prince has joined #openstack-ironic06:40
*** eghobo has joined #openstack-ironic06:41
*** rsacharya has joined #openstack-ironic07:14
*** max_lobur has joined #openstack-ironic07:35
*** romcheg has joined #openstack-ironic07:37
*** max_lobur has quit IRC07:50
*** vkozhukalov has quit IRC07:50
*** dshulyak has joined #openstack-ironic07:54
*** rameshg87 has left #openstack-ironic07:55
*** mrda is now known as mrda_away08:01
*** ifarkas has quit IRC08:04
*** Manishanker has joined #openstack-ironic08:09
*** eghobo has quit IRC08:18
*** jistr has joined #openstack-ironic08:33
openstackgerritYuriy Zveryanskyy proposed a change to openstack/ironic: Add TaskManager lock on change port data  https://review.openstack.org/7762308:33
*** ifarkas has joined #openstack-ironic08:34
agordeevmorning Ironic :)08:39
GheRiveromorning08:41
agordeevGheRivero: morning !08:42
yuriyzmorning GheRivero, agordeev08:44
agordeevyuriyz: morning08:46
*** max_lobur has joined #openstack-ironic08:49
*** max_lobur1 has joined #openstack-ironic09:00
*** derekh has joined #openstack-ironic09:01
*** rsacharya has quit IRC09:02
*** max_lobur has quit IRC09:03
*** martyntaylor has joined #openstack-ironic09:06
*** rsacharya has joined #openstack-ironic09:06
*** killer_prince has quit IRC09:07
*** jrist has quit IRC09:07
*** matsuhas_ has quit IRC09:08
*** tatyana has joined #openstack-ironic09:08
*** matsuhashi has joined #openstack-ironic09:09
*** martyntaylor has quit IRC09:10
agordeevironic.nova.driver still requires 'v2.0' at the of admin_url. Is it really necessary? Can be a bug?09:21
*** jrist has joined #openstack-ironic09:21
*** lucasagomes has joined #openstack-ironic09:24
*** martyntaylor has joined #openstack-ironic09:28
*** romcheg has quit IRC09:31
*** athomas has joined #openstack-ironic09:35
*** martyntaylor has quit IRC09:49
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Move duplicated _get_node(s)_mac_addresses()  https://review.openstack.org/7878709:50
*** martyntaylor has joined #openstack-ironic09:53
*** martyntaylor has quit IRC09:56
*** martyntaylor has joined #openstack-ironic09:57
*** romcheg has joined #openstack-ironic10:04
*** rsacharya has quit IRC10:05
*** martyntaylor has quit IRC10:05
*** saju_m has joined #openstack-ironic10:05
*** killer_prince has joined #openstack-ironic10:07
*** saju_m has quit IRC10:18
*** martyntaylor has joined #openstack-ironic10:19
*** ndipanov_gone is now known as ndipanov10:26
openstackgerritYuriy Zveryanskyy proposed a change to openstack/ironic: Add TaskManager lock on change port data  https://review.openstack.org/7762310:28
*** matsuhashi has quit IRC10:29
*** saju_m has joined #openstack-ironic10:30
mrda_awayHi devananda, Just wondering whether you can take a look at https://review.openstack.org/#/c/78515/ as its ready for review and it's a bug marked High.  It's the do_node_tear_down() pair for do_node_deploy().10:33
*** saju_m has quit IRC10:54
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: PXE validate() to fail if no Ironic API URL  https://review.openstack.org/7957911:00
*** jrist has quit IRC11:58
*** linggao has joined #openstack-ironic12:03
*** killer_prince has quit IRC12:05
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: PXE validate() to fail if no Ironic API URL  https://review.openstack.org/7957912:19
*** romcheg has quit IRC12:28
*** lucasagomes is now known as lucas-hungry12:31
*** jdob has joined #openstack-ironic12:31
*** romcheg has joined #openstack-ironic12:42
*** nosnos has quit IRC12:44
*** rloo has joined #openstack-ironic12:46
*** lazy_prince is now known as killer_prince12:56
*** jbjohnso has joined #openstack-ironic13:11
*** jrist has joined #openstack-ironic13:26
*** romcheg has quit IRC13:32
*** romcheg has joined #openstack-ironic13:32
*** lucas-hungry is now known as lucasagomes13:38
NobodyCamgood morning says the man making coffee13:54
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: PXE validate() to fail if no Ironic API URL  https://review.openstack.org/7957913:54
rloomornin' NobodyCam13:55
NobodyCammorning rloo :)13:56
lucasagomesmorning rloo NobodyCam13:59
lucasagomes:)13:59
NobodyCammorning lucasagomes :)13:59
rlooafternoon lucasagomes!13:59
romchegMorning NobodyCam lucasagomes rloo!13:59
jbjohnsoafter digging so far into obscure terminal stuff, I can do some fun practical jokes13:59
jbjohnsolike putting echo -e '\x1b)0\x0e' into someone's bashrc14:00
rloohi romcheg!14:00
rloojbjohnso: that scares me ;)14:00
NobodyCamhey hey romcheg :) morning morning jbjohnso14:00
jbjohnsotry it out, it has a neat effect14:00
jbjohnsoecho -e '\x0f' to undo14:01
agordeevjbjohnso: "reset" is also to undo14:02
*** jbjohnso_ has joined #openstack-ironic14:03
jbjohnso_the things you learn when trying to figure out why ESXi's serial console does not really work well at all unless your terminal was watching throughout the whole boot process14:03
*** jbjohnso has quit IRC14:06
*** dwalleck has joined #openstack-ironic14:09
*** dwalleck_ has joined #openstack-ironic14:11
*** dwalleck has quit IRC14:14
NobodyCambrb14:20
*** matty_dubs|gone is now known as matty_dubs14:28
GheRiveromorning all14:29
NobodyCamgood morning GheRivero :)14:29
*** dwalleck_ has quit IRC14:31
*** dwalleck_ has joined #openstack-ironic14:35
*** jrist has quit IRC14:43
*** jbjohnso_ has quit IRC14:55
*** blamar has joined #openstack-ironic15:02
*** dwalleck_ has quit IRC15:05
*** bashok has quit IRC15:05
devanandagood morning, all15:07
NobodyCamgood orning devananda :)15:08
*** jbjohnso_ has joined #openstack-ironic15:08
NobodyCammorning even15:08
NobodyCam:-p15:08
*** digambar has joined #openstack-ironic15:09
devanandalucasagomes: i18n in a unit test?15:11
*** jistr has quit IRC15:11
lucasagomesdevananda, morning15:12
romchegMorning devananda15:12
lucasagomesdevananda, yeah i was a bit unsure as well so I put the '?'15:12
lucasagomesi18n?15:12
lucasagomesbut we can leave without it15:12
devanandaaiui, i18n is meant for user-facing strings15:12
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Conductor logs better error if seamicroclient missing  https://review.openstack.org/7952015:13
lucasagomesdevananda, right, but that skipTest will print the string for the user running the tests no?15:14
devanandalucasagomes: true ...15:14
devanandalucasagomes: though I would point out that's a developer, not a user of the product15:15
lucasagomesdevananda, yeah, its kinda trivial, but I was unsure as well if in that case we should i18n it, so I left the comment15:15
lucasagomesdevananda, yeah15:15
lucasagomesok, let's leave it without it then15:15
devanandathanks for pointing it out15:16
NobodyCamspeacking of SM did we file a bug for not including SMclient in requirments?15:18
lucasagomesdevananda, np, thank u for the patch15:19
lucasagomesdevananda, btw15:19
devanandaNobodyCam: no. but that patch removes it15:19
lucasagomesdevananda, we might want to do something similar wiht pyghmi as well no?15:19
lucasagomesI mean we haven't decided yet whether native ipmi will be the default driver for ironic, but right now I think it's better to keep ipmitool because people are working on other stuff for it like console access15:20
devanandalucasagomes: ++15:20
*** jistr has joined #openstack-ironic15:35
*** jistr is now known as jistr|afk15:36
*** harlowja_away is now known as harlowja15:38
*** harlowja has quit IRC15:43
*** harlowja has joined #openstack-ironic15:48
*** eghobo has joined #openstack-ironic15:53
*** hemna has joined #openstack-ironic16:03
*** dwalleck_ has joined #openstack-ironic16:08
NobodyCambrb16:10
*** ifarkas has quit IRC16:11
*** yuriyz has quit IRC16:13
*** vkozhukalov has joined #openstack-ironic16:16
*** jrist has joined #openstack-ironic16:19
openstackgerritJim Rollenhagen proposed a change to openstack/ironic: Add Node.instance_info field  https://review.openstack.org/7946616:21
*** matty_dubs is now known as matty_dubs|lunch16:22
*** jgrimm has joined #openstack-ironic16:26
*** zigo has joined #openstack-ironic16:29
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Make sure auth_url exists and is not versionless  https://review.openstack.org/7966616:30
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Make sure auth_url exists and is not versionless  https://review.openstack.org/7966616:32
jrolldevananda: if we decide to upload the configdrive to swift and use a pointer, do you see the nova driver doing the upload, or ironic special-casing that key and doing it? (I'd prefer the former, I think)16:33
NobodyCamgah quick walkies brb16:34
*** jistr|afk is now known as jistr16:36
*** killer_prince is now known as lazy_prince16:42
lucasagomesdevananda, ping re hash ring replicas16:46
lucasagomesdevananda, I saw ur patch to make it experimental16:46
lucasagomeswhat's missing for that?16:46
lucasagomesI mean, what is missing to make it not experimental? more tests?16:47
*** ifarkas has joined #openstack-ironic16:58
*** rsacharya has joined #openstack-ironic16:59
JoshNangre: the seamicro patch, we talked about 3rd party testing for 3rd party drivers at the meetup. where do you guys see the line between included and 3rd party drivers?17:00
devanandalucasagomes: functional testing17:01
lucasagomesdevananda, ack17:01
lucasagomesdevananda, thanks17:01
devanandalucasagomes: that patch series should go a long way towards rebalancing when the ring changes17:06
devanandalucasagomes: but it doesn't cover any of the prepare-early-so-rebalance-is-faster ideas17:06
lucasagomesJoshNang, included in the requirements u mean? hmm I think that the libraries for drivers that only works for a specific driver shouldn't be included in the requirements.txt17:06
*** romcheg1 has joined #openstack-ironic17:07
devanandalucasagomes: we also need a way to trigger a rebalance, and i prefer swift's approach of making it manual17:07
lucasagomesdevananda, ahh right yeah, that needs some work to cache images before and all17:07
lucasagomesdevananda, idk the swift's approach17:07
lucasagomesdevananda, but by ur comment, ur saying that in case a conductor join/leave the cluster17:08
devanandalucasagomes: operator runs something to tell the ring to rebalance, since that is a pretty heavy-weight operation17:08
lucasagomesthe ring won't be automatically rebalanced?17:08
devanandaso17:08
devanandaright now two somewhat orthogonal things happen17:08
devanandaRPC routing and who-owns-what changes as soon as the ironic.conductors table either gets udpated, or the record's updated_at is older than the timeout17:09
devanandathe API service is actually SELECTing from that table upon every API request -- I consider this a bug17:09
devanandait should be cached and refreshed periodically17:09
lucasagomesright17:10
devanandaso we have the RPC routing based on the current hash ring17:10
devanandaand17:10
devanandawe have the currently prepared deployment environments, console state (not yet implemented), etc17:11
devanandai added an RPC fanout-cast to tell all the conductors "go check the ring and update your local state (files, console, etc) based on it"17:11
lucasagomesright, yeah API always gets it from the DB, it might be easy to make a cache using a dict mapping + periodic task to update it. But in case we have many API services running we also want to make sure they all would redirect a request to one node to the specific conductor which is managing it17:12
lucasagomesso we have to make sure things r sync'd between the API service17:13
lucasagomesservices*17:13
lucasagomesdevananda, ah awesome!17:13
devanandalucasagomes: right. which gets tricky if we're caching it in memory in each API17:13
lucasagomesdevananda, yes17:13
devanandathis is where a distributed in-memory cache is compelling17:14
devanandaor a static mapping and a "button" to rebalance it17:14
devanandaeg. what swift does17:14
lucasagomes+1 yeah I would vote to use some dist mem cach mechnism for this type of situations17:14
lucasagomesmemcacache idk17:14
lucasagomesok17:15
russell_hcould the conductor respond with some known error if it doesn't own the node the API thinks it does17:15
russell_hand that triggers a refresh in the API?17:15
russell_hthe thing with a distributed cache is you need to be careful with how you write to it, you want a guarantee of strong consistency for hash rings / master election17:16
lucasagomesrussell_h, hmm it's a good idea, but I don't think conductors explicity knows which node it "owns"17:17
lucasagomesall conductors with the drivers to manage that node is a potential owner17:17
russell_hI volunteer to make this modular, so you can choose between a zookeeper backend or a dbapi (or sqlalchemy? not sure how best to model that) backend17:17
lucasagomesalso, to make it trigger the api to refresh is hard cause if there's many APIs under a haproxy idk hw to trigger that one specifically17:18
lucasagomesso that's why I think that having all APIs to use a dist. in-mem cache mechanism would be a better implementation17:18
russell_hwell17:18
russell_hso there are two cases you care about17:18
russell_ha conductor shuts down cleanly, or a conductor dies17:19
devanandarussell_h: also, a conductor is only shut down temporarily17:19
russell_hif it dies, it won't be removed from the hashring until its database entry is past the TTL17:19
devanandais a third case17:19
russell_hah, yes17:19
russell_hgood point17:19
russell_his there a mechanism for that today?17:19
devanandano17:20
russell_hits almost like you want to make the case where it removes itself from the database a special case17:20
russell_hfor example, Cassandra has some tool you can run to remove a node from the ring17:21
russell_hotherwise it is presumed to be down, but eventually to return17:21
devanandaexactly17:21
devanandathis is akin to swift's tool that informs the cluster a rebalance is needed17:22
*** matty_dubs|lunch is now known as matty_dubs17:24
lucasagomesdevananda, [off-topic] I know u and lifeless have spoken a good bit about using bittorrent or multicast for transfering the image to the nodes17:25
*** martyntaylor has quit IRC17:25
lucasagomesdo you guys have any etherpad or wiki page or idk... with details of that?17:26
devanandalifeless: did ya'll sketch something out for the glance multicast stuff?17:26
devanandalucasagomes: not that I have on hand17:26
lucasagomesdevananda, I see, ok, mind if I pick ur brain some time this week to talk about it?17:27
*** ndipanov is now known as ndipanov_gone17:28
openstackgerritJenkins proposed a change to openstack/ironic: Updated from global requirements  https://review.openstack.org/7933417:30
devanandalucasagomes: don't mind at all17:30
lucasagomesdevananda, ack thanks, I will give it some thoughts and raise some questions17:30
devanandarussell_h: we have a mapping impelemented in ironic.common.hash_ring. right now, it's sync'd on the API side by always polling the DB (abstracting that so it could use zookeeper would be great)17:33
devanandarussell_h: where I think we might have an issue today is that the API will respond to changes in the ring faster than the conductors, and the tear_down() method may not be idempotent17:34
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Replace fixtures with mock in test_keystone.py  https://review.openstack.org/7970217:38
*** rupsky has joined #openstack-ironic17:38
devanandaiow, if conductor A did a deploy, then is stopped, and a user requests the deploy be torn down, the API will route it to another conductor that doesn't have the deploy environment on disk yet17:38
devanandaand I suspect that will raise exceptions17:38
devanandawhich it should not17:39
russell_hah, gotcha17:39
russell_hits also going to crush the database17:39
russell_hwhen it lists out every conductor on every API request17:39
russell_hI started a fix for that the other day but never finished it17:40
devanandayea17:41
devanandaI had thought the API cached it17:41
*** rupsky has quit IRC17:41
devanandabut it doesn't17:41
*** rupsky has joined #openstack-ironic17:41
devanandathe conductor is17:41
russell_hI can submit a fix for that today17:41
devanandahttp://git.openstack.org/cgit/openstack/ironic/tree/ironic/api/hooks.py#n9717:43
russell_hwhat if we attached a hashring timestamp to every message17:43
devanandathat's the obj we need to cache17:43
*** tatyana has quit IRC17:43
russell_hso if a conductor has an older hashring than the request, and the request appears to have been routed incorrectly (according to its view of the world), it would refresh17:44
*** romcheg1 has quit IRC17:44
*** rsacharya has quit IRC17:44
russell_hdevananda: right, we also need the ConductorAPI to periodically refresh its hash rings17:44
*** athomas has quit IRC17:45
russell_hanyway, I'll fix the caching thing, we can address the expiration issues separately17:45
lucasagomesrussell_h, devananda maybe open a bug for it and tag as rc ?17:48
devanandarussell_h: i would prefer the refresh/resync be either continuous (eg, zookeeper), event-based (something wrouted wrong), or only-manual17:48
*** rupsky has quit IRC17:49
devanandaperiodic refresh alone will result in everything having different views for some time, when ever there's a change17:50
*** max_lobur1 has quit IRC17:51
*** eghobo has quit IRC17:53
*** rupsky has joined #openstack-ironic17:53
devanandarussell_h: I think refresh when there's an unexpected message would be the easiest to implement on the conductor side17:53
*** eghobo has joined #openstack-ironic17:55
*** derekh has quit IRC18:00
Shrewsadam_g: have you tried the latest devstack deploy? conductor mgr is giving me "Timeout reached when waiting callback for node" for a deploy.18:04
devanandaShrews: is the deploy starting, then timing out? or just not starting?18:05
Shrewsdevananda: starting, then timing out18:05
devanandaShrews: increase the timeout :)18:05
NobodyCamok expanse report done and registered for atlanata too :)18:06
Shrewsdevananda: which setting is that?18:06
devanandaShrews: though, the default seems to be 30 minutes ... which is crazy if that's timing out18:06
devanandaShrews: deploy_callback_timeout18:06
Shrewsdevananda: yeah, it seemed crazy long already18:06
devanandawhen i ran through it last week, it was ~ 10 minutes18:07
*** romcheg1 has joined #openstack-ironic18:07
devanandagiven hpcs performance, i wasn't terribly surprised. 30 is too long though18:07
Shrewsdevananda: is there any reason to peg nova with NOVA_BRANCH=refs/changes/26/71026/15 anymore?18:08
devanandanope18:08
devanandashould be nova and ironic, both from trunk now18:08
Shrewsyeah, that's what I'm doing. just checking18:09
devanandaShrews: https://etherpad.openstack.org/p/IronicAndDevstackAgain seems quite out of date18:12
devanandas/quite/slightly/18:12
devanandai'm giving it a run now18:12
devanandabut if you have local notes and can toss them up, it'd be great18:12
Shrewsdevananda: change the patchset number from 20 to 2118:13
Shrewsotherwise, pretty up-to-date18:13
Shrewsadam_g's 2nd bullet point is addressed by pulling down the latest devstack18:14
devanandaah18:14
devanandaShrews: i've updated the localrc. how's this look: http://paste.openstack.org/show/73169/18:14
Shrewsdevananda: add HOST_IP18:16
devanandastatic value?18:16
devanandai thought that was interpreted18:16
Shrewsthat actually may only needed for us using HP cloud instances18:17
Shrewsbe* needed18:17
Shrewspretty much what i have, though18:18
adam_gShrews, devstack just completed. will let you know18:26
Shrewsi just bumped the timeout. going to try again18:27
Shrewsbiab18:29
lucasagomesI'm done for today18:30
lucasagomeshave a good night everybody18:30
NobodyCamnight lucasagomes18:30
adam_gnova boot gets me a running libvirt domain and an instance that is ACTIVE (though power state is Shutdown)18:30
adam_gShrews, ^18:30
lucasagomesadam_g, it takes a while to sync the power state from ironic to nova18:31
adam_gah18:31
lucasagomesI think the nova default's periodic task is 180 sec18:31
adam_glooks like it.. running now18:33
lucasagomesadam_g, actually more 600 secs, see CONF.sync_power_state_interval in nova18:33
lucasagomesanyhoo, gotta run have a good night18:33
Shrewsadam_g: you need to periodically check the provisioning state with 'ironic node-show'18:33
Shrewsneed a walk, back soon18:33
devanandaadam_g: Shrews: i copied some of my old notes to that etherpad. not sure if they're still necessary or if you guys have other work arounds in place18:34
devanandag'night, lucasagomes18:35
*** jbjohnso_ has quit IRC18:37
*** jbjohnso_ has joined #openstack-ironic18:42
adam_gis tftp expected to be served directly from the ironic node (via the infrastructure network) or from within the neutron tenant fixed network?18:46
devanandaadam_g: mgmt network. though i'm not sure that's well separated today18:52
NobodyCamwalkies ...brb18:52
*** digambar has quit IRC18:53
*** max_lobur has joined #openstack-ironic18:54
lifelessdevananda: lucasagomes https://etherpad.openstack.org/p/glance-multicast-image-transfer19:03
*** harlowja has quit IRC19:04
*** harlowja has joined #openstack-ironic19:08
*** lazy_prince has quit IRC19:14
*** killer_prince has joined #openstack-ironic19:24
*** SpamapS has quit IRC19:26
*** SpamapS_ has joined #openstack-ironic19:29
*** SpamapS_ is now known as SpamapS19:30
*** ifarkas has quit IRC19:33
*** jgrimm has quit IRC19:34
devanandaadam_g: have you seen something overwrite /etc/resolv.conf with the host's IP ?19:36
devanandathis is causing the git clone for diskiage-builder to fail19:37
devanandaso it's happening before that19:37
devanandaperhaps when neutron starts?19:37
adam_gdevananda, something being devstack related or a general 'something'?19:37
adam_gresolvconf will do that19:37
devanandaright, so i'm explicitly mangling resolv.conf before i start19:38
NobodyCamwoo hoo : undercloud | CREATE_IN_PROGRESS19:38
NobodyCambrb19:38
devanandaadam_g: and resolvconf is mangling it in a different way that actually breaks it, because 10.0.0.3 can't resolv github19:39
adam_gdevananda, ive been doing the same without issue.19:40
* devananda tries agai19:40
devanandan19:40
adam_gdevananda, you should be able to turn it off 'resolvconf --disable-updates'19:41
adam_gShrews, what are you suing for FIXED_RANGE and NETWORK_GATEWAY on HPC?  having trouble with instances reaching the tftp server, and my neutron L3 agents are all 'DOWN'19:50
Shrewsadam_g: i was using FIXED_RANGE=172.16.0.0/24 and NETWORK_GATEWAY=172.16.0.1, but i suspect issues with the gateway i've chosen. trying something else, atm19:52
NobodyCamahh new error InstanceDeployFailure: PXE deploy failed for instance :)19:56
openstackgerritAdam Gandelman proposed a change to openstack/python-ironicclient: Avoid traceback with insufficient auth credentials  https://review.openstack.org/7949419:57
*** jbjohnso_ has quit IRC19:58
*** romcheg1 has quit IRC20:02
*** jistr has quit IRC20:02
*** romcheg1 has joined #openstack-ironic20:02
ShrewsI don't understand enough about neutron, but I'm getting a floating IP assigned to the external network bridge (br-ex). I thought the NETWORK_GATEWAY ip should be assigned to it?20:11
*** vkozhukalov has quit IRC20:13
*** _LXXIII has joined #openstack-ironic20:18
*** _LXXIII has quit IRC20:20
*** _LXXIII has joined #openstack-ironic20:21
*** _LXXIII has quit IRC20:23
*** _LXXIII has joined #openstack-ironic20:23
*** rupsky has quit IRC20:27
*** rupsky has joined #openstack-ironic20:30
devanandaurgh... lost connectivity to my devstack VM -again-20:33
lifelessShrews: on a nova compute host? or Ironic test setup/ or ... ?20:33
Shrewslifeless: devstack setup20:34
devanandaironic-client should not be spitting out the whole token when in debug mode ...20:35
devanandabut - yay, deploy started. waiting for callback20:37
* Shrews very interested in devananda's results20:38
devanandatook 3 false starts to get the environment prepared properly20:38
devanandaresolvconf broke once, so git clone dib failed20:39
devanandasomething borked the network and i had to "nova floating-ip-disassociate" to be able to reconnect it this last time20:39
adam_gdevananda, do VMs successfully tftpboot?20:39
devanandaoh. that happened again20:40
devanandayay20:40
devanandaadam_g: waiting20:40
*** mrda_away is now known as mrda20:41
devanandaNobodyCam: nova lists the vm state as ACTIVE while it's still waiting for callback20:41
devanandaNobodyCam: i consider that a bug. the nova driver should keep the state as SPAWNING until it's actually available20:41
NobodyCamdevananda: ya20:41
Shrewsdevananda++20:42
Shrewsthat's confused me more than once20:42
adam_galso--is the special network configuration for baremetal in lib/neutron relevant to ironic? or is that special to nova-baremetal? it looks like the patch should be special casing for ironic as well, but its currently not20:42
mrdamorning all20:42
*** jgrimm has joined #openstack-ironic20:42
devanandaadam_g: good point. the lib/baremetal stuff is legacy cruft. it should be ripped out IMO.20:42
devanandaadam_g: it depended on bm_poseur which asn't been maintained for ~1yr20:43
adam_gdevananda, ahh--im just wondering if neutron is setting up its networks appropriately for the ironic case--i imagine the same stuff its doing in the presence of n-baremetal needs to happen for ironic as well?20:44
*** ceoqia has joined #openstack-ironic20:44
NobodyCamahh the driver just calls set_provision_state and then moves on20:45
NobodyCamhttps://github.com/openstack/ironic/blob/master/ironic/nova/virt/ironic/driver.py#L42520:45
devanandaadam_g: dont make assumptions based on lib/baremetal20:45
devanandait may be totally wrong20:45
adam_gdevananda, oh no, im not :)  i'm looking at lib/neutron20:45
devanandaoh20:46
adam_git does a 'is_baremetal && do_some_specific_ovs_setup'20:46
devanandahmm20:46
adam_gwondering if it needs to do a is_ironic as well--testing that now20:47
devanandais that part of the patch ya'll are working on? if so, it's probably reasonable20:47
*** ceoqia has quit IRC20:47
adam_gsomething is screwy for me with gateways and routers in neutron once everything is up20:47
adam_gand the instances cannot reach outside to the tftp server20:47
adam_gim curious if your nodes are doing the same now that you've gotten as far as me :)20:48
Shrewsadam_g: i think you and i are hunting the same dragon.20:48
devanandai'm barely able to maintain a stable network conncetion to the HOST :(20:48
adam_gShrews, does 'sudo ip netns list' show you a namespace for dhcp + router?20:49
devanandaadam_g: so far i haven't seen any POST come back from the VM20:50
devanandait hasnt even started to copy the iamge20:50
NobodyCamdevananda: looks like driver is waiting for nodes to destroy but not waiting on deploy :-p20:50
devanandathere should be a POST to the API to /vendor_passthru/, then disk is mounted and image copied20:50
Shrewsadam_g: no, but i'm rerunning stack.sh atm20:50
Shrewsfunny thing is devananda and I watched this process work last week20:51
openstackgerritA change was merged to openstack/ironic: Conductor logs better error if seamicroclient missing  https://review.openstack.org/7952020:51
Shrewsadam_g: fyi, as a quick experiment, i changed the 'if is_baremetal' check in lib/neutron to add '|| is_service_enabled ironic', but it bails out with the "Neutron settings for baremetal not set" error.20:58
Shrewsi think we need to figure out what that code is actually doing and decide if we need it20:58
*** jdob has quit IRC20:58
*** linggao has quit IRC21:02
devanandawe also need better error handling around delete21:02
devanandaand to allow deletes during wait-callback21:03
devanandai thought there ws a patch for that?21:03
lifelessdevananda: did you see the glance etherpad I linked?21:04
devanandalifeless: didn't read it yet. E_MEETINGS21:05
NobodyCamya I thought so to? wasn't it looked at / landed on a review jam21:05
devanandaNobodyCam: and yet -- 2014-03-11 21:01:25.394 21781 WARNING wsme.api [-] Client-side error: Node c5394778-5d81-4b56-b7b2-1e2164eb92ec is already being provisioned.21:05
devanandawhen i try to delet it21:05
devanandaNobodyCam: the bigger problem is that nova won't retry a delete21:08
devanandaNobodyCam: once it gets an exception during the delete attempt, it marks the node state and backs off, and prevcents any more user attempts to delete it21:09
NobodyCamoh21:10
devanandaNobodyCam: there's got to be a sync deleted states periodic task in nova. either it's not running, very long, or our driver isn't tied in yet21:10
devanandahttp://paste.openstack.org/show/73191/21:12
*** romcheg1 has quit IRC21:28
*** harlowja has quit IRC21:28
*** rwsu has quit IRC21:28
*** _LXXIII has quit IRC21:28
*** max_lobur has quit IRC21:28
*** rloo has quit IRC21:28
*** dshulyak has quit IRC21:28
*** dtantsur has quit IRC21:28
*** rainya has quit IRC21:28
*** jgrimm has quit IRC21:28
*** dwalleck_ has quit IRC21:28
*** antonym has quit IRC21:28
*** ndipanov_gone has quit IRC21:28
*** mrda has quit IRC21:28
*** JoshNang has quit IRC21:28
*** bigjools has quit IRC21:28
*** Madasi has quit IRC21:28
*** shortstop has quit IRC21:28
*** eghobo has quit IRC21:28
*** hemna has quit IRC21:28
*** lynxman has quit IRC21:28
*** sbadia has quit IRC21:28
*** dhellmann has quit IRC21:28
*** pradipta has quit IRC21:28
*** lsmola has quit IRC21:28
*** morgabra has quit IRC21:28
*** ewindisch has quit IRC21:28
*** Shrews has quit IRC21:28
*** blamar has quit IRC21:28
*** davidlenwell has quit IRC21:28
*** Slower_ has quit IRC21:28
*** mdurnosvistov has quit IRC21:28
*** toure has quit IRC21:28
*** aignatov has quit IRC21:28
*** romcheg has quit IRC21:28
*** Hefeweizen has quit IRC21:28
*** datajerk1 has quit IRC21:28
*** openstackgerrit has quit IRC21:28
*** early has quit IRC21:28
*** pleia2 has quit IRC21:28
*** comstud has quit IRC21:28
*** matty_dubs has quit IRC21:28
*** lifeless has quit IRC21:28
*** mordred has quit IRC21:28
*** Haomeng has quit IRC21:28
*** lucasagomes has quit IRC21:28
*** greghaynes has quit IRC21:28
*** JayF has quit IRC21:28
*** EmilienM has quit IRC21:28
*** jroll has quit IRC21:29
*** wendar has quit IRC21:29
*** russellb has quit IRC21:29
*** adam_g has quit IRC21:29
*** mikal has quit IRC21:29
*** Manishanker has quit IRC21:29
*** ekarlso has quit IRC21:29
*** pquerna has quit IRC21:29
*** anteaya has quit IRC21:29
*** NobodyCam has quit IRC21:29
*** devananda has quit IRC21:29
*** wendar has joined #openstack-ironic21:41
*** russellb has joined #openstack-ironic21:41
*** _LXXIII has joined #openstack-ironic21:41
*** jroll has joined #openstack-ironic21:41
*** rloo has joined #openstack-ironic21:41
*** max_lobur has joined #openstack-ironic21:41
*** dtantsur has joined #openstack-ironic21:41
*** adam_g has joined #openstack-ironic21:41
*** dshulyak has joined #openstack-ironic21:41
*** greghaynes has joined #openstack-ironic21:41
*** lucasagomes has joined #openstack-ironic21:41
*** tatyana has joined #openstack-ironic21:41
*** EmilienM has joined #openstack-ironic21:41
*** derekh has joined #openstack-ironic21:41
*** mikal has joined #openstack-ironic21:41
*** JayF has joined #openstack-ironic21:41
*** mdurnosvistov_ has joined #openstack-ironic21:41
*** jgrimm has joined #openstack-ironic21:41
*** eghobo has joined #openstack-ironic21:41
*** dwalleck_ has joined #openstack-ironic21:41
*** hemna has joined #openstack-ironic21:41
*** blamar has joined #openstack-ironic21:41
*** romcheg has joined #openstack-ironic21:41
*** Manishanker has joined #openstack-ironic21:41
*** antonym has joined #openstack-ironic21:41
*** Hefeweizen has joined #openstack-ironic21:41
*** datajerk1 has joined #openstack-ironic21:41
*** ndipanov_gone has joined #openstack-ironic21:41
*** davidlenwell has joined #openstack-ironic21:41
*** mrda has joined #openstack-ironic21:41
*** JoshNang has joined #openstack-ironic21:41
*** Haomeng has joined #openstack-ironic21:41
*** Slower_ has joined #openstack-ironic21:41
*** openstackgerrit has joined #openstack-ironic21:41
*** toure has joined #openstack-ironic21:41
*** mdurnosvistov has joined #openstack-ironic21:41
*** shortstop has joined #openstack-ironic21:41
*** Madasi has joined #openstack-ironic21:41
*** bigjools has joined #openstack-ironic21:41
*** lynxman has joined #openstack-ironic21:41
*** sbadia has joined #openstack-ironic21:41
*** dhellmann has joined #openstack-ironic21:41
*** pradipta has joined #openstack-ironic21:41
*** lsmola has joined #openstack-ironic21:41
*** early has joined #openstack-ironic21:41
*** morgabra has joined #openstack-ironic21:41
*** ewindisch has joined #openstack-ironic21:41
*** Shrews has joined #openstack-ironic21:41
*** aignatov has joined #openstack-ironic21:41
*** ekarlso has joined #openstack-ironic21:41
*** pleia2 has joined #openstack-ironic21:41
*** comstud has joined #openstack-ironic21:41
*** pquerna has joined #openstack-ironic21:41
*** anteaya has joined #openstack-ironic21:41
*** matty_dubs has joined #openstack-ironic21:41
*** lifeless has joined #openstack-ironic21:41
*** mordred has joined #openstack-ironic21:41
*** NobodyCam has joined #openstack-ironic21:41
*** devananda has joined #openstack-ironic21:41
*** dickson.freenode.net sets mode: +o devananda21:41
*** rainya has joined #openstack-ironic21:41
devanandaadam_g: how have you not lost network access to the HPCS instance you're testing in?21:43
devanandaadam_g: or are you not using it any more?21:43
* devananda takes a break from meetings and gets some fresh air21:44
NobodyCam:)21:44
*** harlowja has joined #openstack-ironic21:45
adam_gdevananda, networking has been stable since last week21:46
*** rwsu has joined #openstack-ironic21:46
adam_gdevananda, at least in east21:46
devanandaadam_g: urgh. my instance is in the same az as the one i gave you, but keeps losing net21:47
adam_gregion-b.geo-1 is what ive been using21:49
devanandaadam_g: did you get access to spin up your own?21:49
adam_gdevananda, yup.21:49
devanandaadam_g: cool21:50
devanandai'll delete the one i made last week then21:50
*** dwalleck_ has quit IRC22:00
*** dwalleck has joined #openstack-ironic22:02
*** max_lobur1 has joined #openstack-ironic22:04
*** max_lobur has quit IRC22:04
*** dwalleck_ has joined #openstack-ironic22:05
*** mdurnosvistov_ has quit IRC22:06
NobodyCamjust me or is anyone seeing oddness with https://github.com/openstack/nova22:06
openstackgerritRuby Loo proposed a change to openstack/ironic: Default value for database.connection config option  https://review.openstack.org/7978722:07
NobodyCami'm getting page not found... other sites seem to be working, thou could just me TimeWarnner22:07
rlooNobodyCam: works for me22:08
NobodyCamrloo: thank you :)22:08
*** dwalleck has quit IRC22:09
NobodyCamhehe now its working for me :_p22:09
NobodyCamrloo: has the magic today :)22:10
rlooNobodyCam: just need to be patient ;)22:10
*** _LXXIII has quit IRC22:11
*** rupsky_ has joined #openstack-ironic22:13
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Generic MixinVendorInterface using static mapping  https://review.openstack.org/7838422:13
*** rupsky has quit IRC22:14
devanandaGheRivero: i'm landing ^ now. should https://review.openstack.org/#/c/71332/ be reopened on top of that?22:17
*** matty_dubs is now known as matty_dubs|gone22:19
NobodyCambrb22:23
*** max_lobur1 has quit IRC22:25
*** tatyana has quit IRC22:27
*** dwalleck_ has quit IRC22:28
adam_gShrews, matching what the gate uses in localrc got me a functional neutron setup22:41
adam_gNETWORK_GATEWAY=10.1.0.122:41
adam_gFIXED_RANGE=10.1.0.0/2422:41
adam_gFIXED_NETWORK_SIZE=2522:41
adam_gpaste fail.. FIXED_NETWORK_SIZE=25622:42
*** openstackstatus has joined #openstack-ironic22:44
NobodyCamodd Warning: given size (11264) exceeds max allowable size (0)22:50
NobodyCamok my deploys are failing at batemetal deploy helper with ^^ error from sfdisk23:01
NobodyCamhttps://review.openstack.org/#/c/79344 ahh23:12
*** lucasagomes has quit IRC23:16
openstackgerritA change was merged to openstack/ironic: Generic MixinVendorInterface using static mapping  https://review.openstack.org/7838423:29
openstackgerritZhongyue Luo proposed a change to openstack/ironic: Remove unused method from timeutils  https://review.openstack.org/7884823:33
*** rupsky_ has quit IRC23:41
adam_gthoughts on an optional DIB element to backdoor deploy ramdisk with an ssh server for easier debugging?23:44
NobodyCamI'd love to at least get some logging from the deploy ramdisk23:45
NobodyCam:-p23:45
NobodyCamthere use to break points in the ramdisk where it would drop to a prompt23:46
NobodyCambut I haven't looked at it in a long while23:46
jrollfwiw we ship an ssh server in our ramdisk image, it's super helpful23:46
jrollwe also plan to ship logs back from our deploy agent. not sure how yet.23:47
NobodyCam:)23:47
JoshNangjroll: maybe ship them back with the heartbeat?23:48
russell_hI'd say we just ship them in a separate API call23:48
russell_hthen we can make it pluggable what to do with them or something23:48
russell_hI haven't seen much in the way of logging infrastructure in openstack, but maybe I missed something23:49
jrollyeah, I was thinking just set up a log handler to something outside of the code, and configure that separately23:49
jrolle.g. logstash or whatever23:50
NobodyCamssh root@$SEED_IP23:50
NobodyCamdoh ww23:50
*** dividehex has joined #openstack-ironic23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!