Monday, 2014-01-20

*** romcheg1 has joined #openstack-ironic00:04
*** romcheg has joined #openstack-ironic00:07
*** romcheg1 has quit IRC00:09
*** romcheg has left #openstack-ironic00:51
*** jbjohnso has joined #openstack-ironic00:57
*** jbjohnso has quit IRC01:08
*** romcheg has joined #openstack-ironic01:23
*** romcheg has left #openstack-ironic01:23
*** nosnos has joined #openstack-ironic01:24
*** Haomeng has quit IRC01:51
*** jbjohnso has joined #openstack-ironic02:15
*** mrda has quit IRC02:39
*** mrda has joined #openstack-ironic02:42
*** d4n_ has joined #openstack-ironic03:14
*** d4n_ has left #openstack-ironic03:14
*** vkozhukalov has joined #openstack-ironic03:28
*** jbjohnso has quit IRC04:06
*** coolsvap has joined #openstack-ironic04:19
*** yfujioka has quit IRC04:26
*** aignatov has joined #openstack-ironic05:12
*** aignatov has quit IRC05:17
*** michchap has quit IRC05:37
*** michchap has joined #openstack-ironic05:37
*** nosnos has quit IRC05:52
*** nosnos_ has joined #openstack-ironic05:52
openstackgerritJenkins proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/6503206:04
*** bashok has joined #openstack-ironic06:26
*** tzumainn has quit IRC06:26
*** mrda has quit IRC06:41
*** vkozhukalov has quit IRC06:44
*** nosnos_ has quit IRC06:52
*** nosnos has joined #openstack-ironic06:53
*** mrda has joined #openstack-ironic07:09
*** mrda has quit IRC07:20
*** Haomeng has joined #openstack-ironic07:34
*** Haomeng has quit IRC07:40
*** Haomeng has joined #openstack-ironic07:40
*** ifarkas has joined #openstack-ironic07:43
*** mdurnosvistov has joined #openstack-ironic07:54
*** mrda has joined #openstack-ironic08:02
*** jistr has joined #openstack-ironic08:06
*** mrda has quit IRC08:09
agordeevmorning Ironic :)08:20
Haomengagordeev: morning:)08:23
GheRiveromorning all08:24
HaomengGheRivero: morning:)08:25
agordeevHaomeng, GheRivero morning :)08:27
Haomengagordeev: :)08:27
*** vkozhukalov has joined #openstack-ironic08:28
*** vkozhukalov has quit IRC08:34
*** Alexei_987 has joined #openstack-ironic08:36
*** Haomeng has quit IRC08:39
*** kpavel has joined #openstack-ironic08:40
*** kpavel_ has joined #openstack-ironic08:40
*** Haomeng has joined #openstack-ironic08:43
*** kpavel has quit IRC08:44
*** kpavel_ is now known as kpavel08:44
*** mrda has joined #openstack-ironic08:44
*** vkozhukalov has joined #openstack-ironic08:51
openstackgerritDmitry Shulyak proposed a change to openstack/ironic: alembic with initial migration and tests  https://review.openstack.org/6741508:53
*** romcheg has joined #openstack-ironic09:01
*** dshulyak has joined #openstack-ironic09:04
*** derekh has joined #openstack-ironic09:15
*** aignatov_ has joined #openstack-ironic09:24
*** jistr has quit IRC09:31
*** jistr has joined #openstack-ironic09:32
*** lsmola_ has joined #openstack-ironic09:38
*** derekh is now known as derekh_afk09:59
*** bashok has quit IRC10:02
*** bashok has joined #openstack-ironic10:02
*** rwsu has joined #openstack-ironic10:03
*** vkozhukalov has quit IRC10:04
*** vetalll has joined #openstack-ironic10:11
*** vkozhukalov has joined #openstack-ironic10:16
openstackgerritDmitry Shulyak proposed a change to openstack/ironic: alembic with initial migration and tests  https://review.openstack.org/6741510:16
*** max_lobur_afk is now known as max_lobur10:18
*** ifarkas has quit IRC10:29
*** martyntaylor has joined #openstack-ironic10:31
*** ifarkas has joined #openstack-ironic10:32
*** mrda has quit IRC10:39
*** mrda has joined #openstack-ironic10:46
*** bashok has quit IRC11:01
*** martyntaylor has quit IRC11:07
*** martyntaylor1 has joined #openstack-ironic11:08
*** lucasagomes has joined #openstack-ironic11:09
*** jistr has quit IRC11:11
*** jistr has joined #openstack-ironic11:14
*** martyntaylor has joined #openstack-ironic11:15
*** martyntaylor1 has quit IRC11:15
*** bashok has joined #openstack-ironic11:20
*** athomas has joined #openstack-ironic11:33
*** derekh_afk is now known as derekh11:38
max_loburping GheRivero11:50
GheRiveromax_lobur: pong11:50
max_loburHi Ghe11:50
GheRiveromorning11:50
max_loburThank you for review of https://review.openstack.org/#/c/66349/311:51
GheRiverolet-s hope it makes its way on time11:51
max_loburdo we need some other reviews to make it merged / do I need ping someone or just wait11:52
GheRiveroI don't know. Actually, I never know how I ended in the core group for requirements in the first place :)11:54
max_loburhehe :)11:54
*** mrda has quit IRC11:56
max_loburI'll try to ask someone in infra channel11:57
openstackgerritA change was merged to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/6503212:03
*** coolsvap has quit IRC12:07
*** jbjohnso has joined #openstack-ironic13:02
*** max_lobur is now known as max_lobur_afk13:26
*** martyntaylor has quit IRC13:33
*** jdob has joined #openstack-ironic13:33
*** martyntaylor has joined #openstack-ironic13:36
*** linggao has joined #openstack-ironic13:45
*** rloo has joined #openstack-ironic13:47
HaomengGheRivero: ping14:07
HaomengGheRivero: I have a global requirement patch to be reviewed https://review.openstack.org/#/c/64408/ , can you help? Thanks:)14:10
HaomengGheRivero: :)14:10
*** bashok has quit IRC14:14
*** tzumainn has joined #openstack-ironic14:21
*** tzumainn has quit IRC14:21
*** vetalll has left #openstack-ironic14:21
lucasagomesare the tests in ironic failing for u guys as well? DBError: (IntegrityError) UNIQUE constraint failed:14:23
lucasagomes(wondering if it's the sqlite version)14:23
Haomenglucasagomes: what db are you using? MySQL?14:24
lucasagomesHaomeng, it's for the tests so sqlite I think it is14:24
lucasagomes[lucasagomes@lucasagomes ironic]$ sudo rpm -q sqlite14:25
lucasagomessqlite-3.8.2-1.fc20.x86_6414:25
*** matty_dubs|gone is now known as matty_dubs14:25
Haomenglucasagomes: I found we have some code branch for sqllite only, and I use MySQL by default:)14:25
Haomenglucasagomes: this code - https://github.com/openstack/ironic/blob/master/ironic/db/sqlalchemy/migrate_repo/versions/014_add_address_uc_sqlite.py#L2114:27
lucasagomesHaomeng, lemme take a look14:27
lucasagomesthanks :D14:27
*** jrist has quit IRC14:28
Haomenglucasagomes: not sure if this is root cause:)14:28
Haomenglucasagomes: another root cause, mysql is case non-sensitive, but sqlite is case sensitive, our Jenkins  will use MySQL as default database14:30
Haomenglucasagomes: so for some code, it working with MySQL, but not working with  sqlite:)14:31
lucasagomesyea I will try to change the engine14:31
Haomenglucasagomes: good luck:)14:31
lucasagomesand run the tests with an older version of sqlite14:31
lucasagomesHaomeng, cheers14:31
Haomenglucasagomes: :)14:31
lucasagomesthanks for the inputs as well :)14:31
Haomenglucasagomes: :)14:31
Haomenglucasagomes: nice day, I will go to sleep:)14:32
lucasagomesHaomeng, have a g'night buddy14:33
Haomenglucasagomes: :)14:33
*** max_lobur_afk is now known as max_lobur14:33
*** yuriyz has joined #openstack-ironic14:34
*** rloo has quit IRC14:36
*** rloo has joined #openstack-ironic14:36
GheRiveroI found this about sqlite/mysql the other day https://bugs.launchpad.net/ironic/+bug/126551814:39
GheRiveroHaomeng: taking a look to your patch14:39
*** coolsvap has joined #openstack-ironic14:41
*** rloo has quit IRC14:44
*** rloo has joined #openstack-ironic14:44
*** rloo has quit IRC14:46
*** rloo has joined #openstack-ironic14:46
openstackgerritDmitry Shulyak proposed a change to openstack/ironic: alembic with initial migration and tests  https://review.openstack.org/6741514:55
*** nosnos has quit IRC15:06
*** pradipta has joined #openstack-ironic15:19
*** rloo has quit IRC15:22
*** rloo has joined #openstack-ironic15:22
*** martyntaylor has quit IRC15:23
*** rloo has quit IRC15:23
*** martyntaylor has joined #openstack-ironic15:23
*** rloo has joined #openstack-ironic15:24
*** lucasagomes is now known as lucas-hungry15:33
*** vkozhukalov has quit IRC15:34
*** kpavel has quit IRC15:38
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Removes use of timeutils.set_time_override  https://review.openstack.org/6743215:45
openstackgerritdekehn proposed a change to openstack/ironic: Adds Neutron support to Ironic  https://review.openstack.org/6607115:59
*** ifarkas has quit IRC16:08
*** lucas-hungry is now known as lucasagomes16:21
*** romcheg1 has joined #openstack-ironic16:21
*** romcheg has quit IRC16:22
openstackgerritYuriy Zveryanskyy proposed a change to openstack/ironic: Add parameter for filtering nodes by maintenance mode  https://review.openstack.org/6393716:24
*** mdurnosvistov has quit IRC16:30
*** dshulyak has quit IRC16:36
*** rloo has quit IRC16:47
*** jistr has quit IRC16:48
*** rloo has joined #openstack-ironic16:48
*** rloo has quit IRC16:49
*** rloo has joined #openstack-ironic16:49
*** aignatov_ has quit IRC16:50
*** GheRiver1 has joined #openstack-ironic16:53
*** GheRiver1 has quit IRC16:53
*** zheliabina has joined #openstack-ironic17:02
*** rloo has quit IRC17:07
*** ifarkas has joined #openstack-ironic17:08
*** rloo has joined #openstack-ironic17:08
*** vkozhukalov has joined #openstack-ironic17:08
*** zheliabina has quit IRC17:08
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Delete the iscsi target the deploy  https://review.openstack.org/6787717:11
*** rloo has quit IRC17:13
*** rloo has joined #openstack-ironic17:13
*** rloo has quit IRC17:14
*** rloo has joined #openstack-ironic17:14
*** rloo has quit IRC17:14
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Delete the iscsi target  https://review.openstack.org/6787717:15
*** rloo has joined #openstack-ironic17:15
*** rloo has quit IRC17:16
*** rloo has joined #openstack-ironic17:16
*** agordeev2 has joined #openstack-ironic17:16
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Delete the iscsi target  https://review.openstack.org/6787717:18
*** rloo has quit IRC17:18
lucasagomespep8 :P17:18
*** rloo has joined #openstack-ironic17:18
*** romcheg has joined #openstack-ironic17:20
*** romcheg1 has quit IRC17:21
*** mdurnosvistov has joined #openstack-ironic17:30
*** Alexei_987 has quit IRC17:33
*** rloo has quit IRC17:36
*** rloo has joined #openstack-ironic17:36
*** rloo_ has joined #openstack-ironic17:36
*** rloo has quit IRC17:36
*** martyntaylor has quit IRC17:38
*** matty_dubs is now known as matty_dubs|lunch17:38
*** rloo_ has quit IRC17:40
*** rloo has joined #openstack-ironic17:40
*** aignatov_ has joined #openstack-ironic17:41
*** matty_dubs|lunch has quit IRC17:43
*** rloo has quit IRC17:43
*** rloo has joined #openstack-ironic17:44
*** rloo has quit IRC17:45
*** rloo has joined #openstack-ironic17:46
*** aignatov_ has quit IRC17:47
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Add support for retrieving SDR data  https://review.openstack.org/6729917:48
*** rloo has quit IRC17:51
*** rloo has joined #openstack-ironic17:51
jbjohnsoif anyone is interested, that's probably where I'm going to land concretely in SDR support for IPMI17:52
jbjohnsofor numeric linear/linearizable sensors17:53
*** rloo has quit IRC17:53
*** rloo has joined #openstack-ironic17:54
NobodyCamgood morning Ironic says the man lazy-a-lee poking his head in and waving17:55
GheRiveromorning NobodyCam17:56
openstackgerritdekehn proposed a change to openstack/ironic: Adds Neutron support to Ironic  https://review.openstack.org/6607117:56
rloomorning NobodyCam. Happy Martin Luther King Jr. Day17:57
jbjohnsoon that neutron stuff, looks like the way to neutron is very simplistic, is the idea just pxelinux options, or supporting adaptive responses to support non-BIOS stuff?17:58
NobodyCammorning GheRivero rloo :)17:58
NobodyCamso anyone have any thoughts on something like node-reset in the cli?18:01
NobodyCamlucasagomes: devananda ^^^18:01
rlooNobodyCam: what would a node-reset do?18:02
romchegNobodyCam: what is it intended to do?18:03
NobodyCamreset everything to default...18:03
romchegwhoops, rloo was first :)18:03
NobodyCammake my testing easier :-p18:03
*** derekh has quit IRC18:03
devanandamorning, all!18:03
rlooNobodyCam: sure then!18:03
romchegNobodyCam: reset ramdisk? :D18:03
devanandaI'm running late today... gonna grab breakfast and be back for the meeting18:03
romchegMorning devananda, NobodyCam rloo18:03
NobodyCamsay you get a node stucking provision state deploying18:04
NobodyCam:) devananda we too18:04
NobodyCamromcheg: nope just  the nodes data base18:04
lucasagomesNobodyCam, morning18:04
lucasagomesdevananda, romcheg rloo morning18:04
NobodyCammorning lucasagomes18:04
rlooafternoon lucasagomes, evening romcheg.18:05
lucasagomesNobodyCam, you mean node tear-down?18:05
lucasagomesto clean up the enviroment etc18:05
romchegWell, we can definitely just power down the node but the data which already are on the node must be cleaned up18:05
lucasagomesyou can PUT {'target': 'deleted'} to /states/provision18:05
rlooNobodyCam: you just want the db updated to reflect a different state?18:05
NobodyCamlucasagomes: not if stuck in deploying18:06
NobodyCamand oh say the instance uuid disapired18:06
NobodyCam:-p18:06
lucasagomesNobodyCam, so it's deploying, but didn't fail yet?18:07
NobodyCamcorrect ... inface would never fail18:07
lucasagomesyea, cause there's no timeout18:07
NobodyCam*infact18:07
lucasagomesfor pxe, :P18:07
*** matty_dubs|lunch has joined #openstack-ironic18:07
lucasagomes:/*18:07
NobodyCamso I thought about node-reset18:08
romchegWe need lock breaking to implement node-reset18:08
NobodyCamjust in case nodes ever got out of sync18:08
lucasagomeshmm /me thinking18:08
lucasagomescause idk about node-reset seems to powerful might leave things in an inconsistent state18:09
lucasagomesI see the benefits for our tests18:09
lucasagomesthat would help us18:09
lucasagomesbut then we can just go and edit the database18:09
lucasagomesI think the timeout would be a better final approach18:09
NobodyCamlucasagomes: that use it helpful for us in testing but real world... could be misused...18:09
lucasagomesdeploy would fail cause it timeout, so you can do a node tear-down18:09
lucasagomesNobodyCam, yea18:10
*** rloo has quit IRC18:10
NobodyCam:-p18:10
romchegBut we never know what time should pass before a timeout18:10
lucasagomesromcheg, ther'es a configuration for that already18:10
lucasagomeswe just don't use it18:10
*** rloo has joined #openstack-ironic18:10
*** athomas has quit IRC18:10
NobodyCamyea it up to the network to set a realistic value for thier network18:11
NobodyCamgah /me needs more coffee18:11
romchegYes, but as far as I remember we agreed to not perform operations that might brake anything automatically18:11
NobodyCamits up to the network admin to ....18:11
*** rloo has quit IRC18:13
*** pradipta has quit IRC18:13
*** rloo has joined #openstack-ironic18:13
max_loburmorning / evening Ironic18:14
NobodyCammorning max_lobur18:15
lucasagomesmorning max_lobur18:15
lucasagomesromcheg, https://bugs.launchpad.net/nova/+bug/119507318:16
*** vkozhukalov has quit IRC18:19
*** matty_dubs|lunch is now known as matty_dubs18:26
* NobodyCam makes coffee for the meeting 18:39
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Add support for retrieving SDR data  https://review.openstack.org/6729918:43
*** linggao has quit IRC18:55
*** yuriyz has left #openstack-ironic18:56
*** linggao has joined #openstack-ironic18:57
* devananda is back18:57
NobodyCamWB18:57
romchegGuys, why we set os=False when doing monkeypatch?18:58
*** aignatov_ has joined #openstack-ironic19:03
openstackgerritMatt Wagner proposed a change to openstack/ironic: API: Add sample() method on Node  https://review.openstack.org/6553619:35
*** pokrov has joined #openstack-ironic19:41
devanandamax_lobur, lucasagomes if you guys can't reproduce bug 1244747, we can reclose it and chalk it up to wierdness in my env (which needed to be wiped anyway)19:42
lucasagomesdevananda, hmm I have to try again and see if I can reproduce that19:43
lucasagomesI tested it before it got reopened19:43
lucasagomesI will do it tomorrow19:43
max_loburlucasagomes, thanks!19:43
max_loburI tested just today19:44
max_loburworked fine19:44
lucasagomesright I'll write it down here and test it tomorrow then19:45
lucasagomesmax_lobur, thank you for fixing :D19:45
*** pokrov has quit IRC19:45
*** k4n01 has joined #openstack-ironic19:51
romchegI and max_lobur tested thread pool executor for futures which might be useful for performing some tasks in Ironic19:57
romchegWe discovered that we need to monkey patch os by eventlet for that19:57
romchegHowever, currently os if not patched.19:57
romchegCan someone please tell me why?19:58
romchegI ran tests with eventlet.monkey_patch(os=True) and everything seems to work19:58
NobodyCamgreat meeting eveyone20:01
max_loburlet's finish with tempest and then go to race20:01
k4n01same here20:01
devanandaawesome meeting,e veryone!20:01
max_loburhttps://review.openstack.org/#/c/67854/1 this is the first patch expanding tempest20:01
devanandaok, temest API tests20:01
romchegregarding to tempest test coverage: there might be problems in changing the API specification: to update Ironic's code we need to update tempest. But that update will fail until patch to Ironic if merged20:01
devanandaso20:02
devanandathat's a good point, BUT the wrong way to look at it20:02
devanandawe shouldn't be breaking API compat after icehouse release20:02
devanandaright now we're in the wierd situation of having no prior release taht we need to maintain compat with20:02
max_lobur:)20:02
devanandaso we have been breaking things between patches without regard20:02
devanandawe need to stop that20:03
devanandaand the tempest API stuff will force us to :)20:03
devanandait's good, IMNSHO20:03
NobodyCamdevananda: +120:03
lucasagomesyea I tend to agree with that20:03
devanandawhen we update our API, we need to be compatible at a minimum between one patchset and the next20:03
lucasagomesapi should ne stable20:03
lucasagomesbe*20:03
devanandaand when we release icehouse, we'l need to remain compatible with THAT until the J release20:03
NobodyCamif we need to break it then we'll to talk aboutv2 of the api20:03
devanandafwiw, once Icehouse is released, it will be added to the stable checks20:04
max_loburok, then I'm going to port all the existing tests from the API to tempest + create new ones for uncovered pathways, right?20:04
devanandaand we should contine to test compat of the tip of python-irnoicclient against the last stable release, too20:05
devanandamax_lobur: ++20:05
max_loburok :)20:05
devanandamax_lobur: in an ideal world, i think we should have the same API unit and functional tests, but i may be overzealous in testing some times ...20:05
max_loburhaha20:05
rlooare we happy with our api then? lucasagomes: any TODOs or 'i'd like to change' things?20:06
max_loburhow much changes do we expect in the api till icehouse?20:06
devanandadeploy is done, afaik20:06
lucasagomesrloo, yea, well there's only one inconsistency i'd like to remove20:07
devanandawe're missing an endpoint for interrupt-type things, though, right?20:07
lucasagomesbetween the /provision and /power calls20:07
devananda(gah, i need to step AFK again for a few minutes... sorry. brb)20:07
lucasagomes /power does return something while /provision doesn't20:07
lucasagomesthere's a patch for that20:07
lucasagomesalso there's some FIXMEs on that part as well20:08
*** ifarkas has quit IRC20:08
rloolucasagomes: ok, so we should try to get those in soon.20:08
lucasagomesbecause we need to return the Location Header pointing to /states so that clients knows that they need to GET /states in order to track the states changes20:08
lucasagomesrloo, unfortunately it needs to be fixed in wsme first20:08
rloolucasagomes: :-(20:09
lucasagomeswsme doesn't currently supports changing the location header20:09
lucasagomesthere's a bug for it, we can try to implement that in wsme20:09
lucasagomesit's open source :)20:09
rloolucasagomes: true!20:09
max_lobur:)20:09
lucasagomesrloo, https://bugs.launchpad.net/wsme/+bug/123368720:10
rloothx lucasagomes. Guess it affects me too ;)20:11
lucasagomeshah20:11
lucasagomesyea, well it's assigned to him for a long time20:12
rloolucasagomes: it is 'wishlist' so may not be important to fix soon.20:12
lucasagomesso it might worth to talk to him to see if he's making a patch for it first20:12
lucasagomesrloo, yea20:12
lucasagomesalthough it's very important20:12
rloois it worth adding a comment that it is important for us? or i guess contacting julien.20:13
max_loburdevananda, ok, so I'll start from what we have in the API, and then will increase that once new API's are merged20:13
max_loburwe can move to race20:13
lucasagomesrloo, +120:14
max_loburbrb in 2 mins20:14
devanandaback20:14
lucasagomesrloo, I think it's important for the REST thing in general20:14
lucasagomesmany people would use the location header for async tasks20:14
lucasagomesmaybe we can even talk about changing that priority20:14
lucasagomesto medium at least20:14
rloolucasagomes: yeah, at a minimum, medium!20:15
devanandafwiw, other folks are spinning up projects based on pecan/wsme now, too20:15
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Add support for retrieving SDR data  https://review.openstack.org/6729920:15
max_loburback20:15
devanandaso there'll be mroe exposure for missing capabilities like that20:16
lucasagomesindeed20:16
devanandamax_lobur: ok,s o, race conditions ...20:16
max_loburyes20:16
max_loburI investigate greenthreads world, also asked a few guys around20:16
*** aignatov_ has quit IRC20:16
max_loburincluding romcheg :)20:16
max_lobur1. Currently we patch threads, so each our RPC request is already handled within greenthread20:17
max_loburIt's already can hang the whole process as you mentioned20:17
max_loburso moving it to another greenthread won't add the problems20:18
devanandamax_lobur: sounds about right20:18
max_lobur2. python futures thread pool executor has handy add_done_callback feature20:18
max_loburhttp://pythonhosted.org/futures/#concurrent.futures.Future.add_done_callback20:19
max_loburso we can reuse futures instead of implementing our own threads20:19
max_lobur3. What our guys suggested to me is to use processes instead of greenthreads20:19
*** aignatov_ has joined #openstack-ironic20:19
max_loburto protect ourselves from hanging conductor20:20
max_loburIt may be possible, since I think we won't have a lot of deployments/reboots handled in the same time by one conductor20:20
devanandamax_lobur: that would require IPC to coordinate any effect of the callable20:20
max_loburprocesses won't hit the performance20:20
devanandamax_lobur: ah. that is definitely not true20:20
max_loburam I right?20:21
devanandamax_lobur: we should not plan according to "we wont do a lot of X"20:21
lucasagomeswe might have many deployments at the same time20:21
max_loburok :)20:21
openstackgerritA change was merged to stackforge/pyghmi: Add support for retrieving SDR data  https://review.openstack.org/6729920:21
devanandaright now, we dont know what scalability issues we'll hit20:21
devanandaand im sure we'll hit some20:21
max_loburthen we should go with greenthreads20:21
devanandabut as a goal, we shouldn't set ourselves low20:22
devanandathink thousands or tens of thousands :)20:22
max_loburfuthermore there is an issue with processes + eventlet :)20:22
max_loburhttp://stackoverflow.com/questions/21242022/eventlet-hangs-when-using-futures-processpoolexecutor20:22
devanandayes20:22
max_loburI hit it today :)20:22
devanandaheh20:22
max_loburso I'm glad we're choosing threads :D20:22
devananda++ stick with threads20:23
lucasagomesright, there's also the two steps lock approach (while I haven't thought a lot about it), I thought about implementing it like: a flag on acquire to not release the node once it's finished...20:23
max_loburyes20:23
devanandayep20:23
max_loburan alternate solution - intent lock20:23
devanandamy initial thoughts here (some months abck) were about an intent lock / two-phase lock20:23
lucasagomesnow that we have the routing algorithm in place we are sure that the rpc call will land in the same conductor20:23
max_lobure.g. to maintain lock between two rpc requests20:23
lucasagomesso on we can acquire the lock with another flag, and we would successfully acquire the node on the second time if conductor is the same and the flag is true20:24
max_loburyep. this means our WITH block won't actually release the lock if the flag was passed20:25
max_loburright?20:25
lucasagomesmax_lobur, yes20:25
lucasagomesmax_lobur, we would have a flag for that,20:25
*** rloo has quit IRC20:25
lucasagomesI mean, a parameter on the acquire() method20:25
max_loburyea, I see20:26
devanandalucasagomes: not just "true" - i think we would need a token20:26
*** rloo has joined #openstack-ironic20:26
max_loburlucasagomes, + your current patch should be merged20:26
devanandaso that other API requests remain blocked during the interval20:26
devanandahowever20:26
lucasagomesmax_lobur, yes, actually we need that intent lock patch before, and then my current implementation depending on the intent lock patch20:26
devanandalet's think about this in another situation20:26
devanandawhat happens when the RPC layer is choked and slow?20:27
lucasagomesdevananda, right, so you mean like... you acquire and it returns a key to you20:27
*** rloo has quit IRC20:27
devanandawhat happens when RPC messages get lost?20:27
lucasagomesthat is passed as an argument for the next acquire() so that releases the lock20:27
devanandalucasagomes: exactly20:27
lucasagomesdevananda, right that also works20:27
devanandalucasagomes: otherwise there's still a race20:27
lucasagomesbut needs to add a db field20:27
devanandalucasagomes: nope20:27
lucasagomeswhere the key will live20:27
*** rloo has joined #openstack-ironic20:27
devanandalucasagomes: it'd be in memory in the ResourceManager20:28
devanandai think20:28
lucasagomesdevananda, hmm right20:28
lucasagomesyea that might work as well20:28
devanandaif that conductor fails or the RPC mapping gets changed, another conductor wouldn't need to break the lock20:28
devanandaif a real TaskManager lock hasn't been taken yet20:28
devanandabut20:29
devanandalet's think about what happens with threads and with two-phase lock if RPC fails or is slow20:29
*** k4n01 has quit IRC20:30
devanandathe point of the API tier making two RPC requests20:30
devanandais to give the user some reasonably-quick feedback20:30
devanandaif their request is likely to succeed or not, and thus whether itw as accepted20:30
max_loburyes20:30
devananda- user request20:30
devananda-- api receives it20:30
devananda-- api makes synchronous RPC call20:30
devananda-- conductor does something, returns a value across RPC20:30
devananda-- api makes async RPC call20:31
devananda- user gets response20:31
devananda-- conductor does something else20:31
max_lobur+20:31
lucasagomeslooks correct20:31
max_loburbut the second async call may lost20:31
devanandawe're talking about adding $something at the fourth step (condcutor does something)20:32
max_loburand the user already got 200 response20:32
devanandawhich sets up a lock until the last step20:32
max_loburdevananda, +20:32
devanandain teh threading model, IIRC, this is a separate greenthread which will effectively sleep20:32
devanandaand get woken up // reattached at the last step20:32
devanandamax_lobur: what prevents >1 API request triggering >1 greenthread?20:33
max_loburthe first API will acquire the lock20:33
max_lobursubsequent ones will fail to acquire it20:33
devanandamax_lobur: ah. teh TaskManager lock20:33
devanandamax_lobur: and taht is then held until the last step?20:34
max_loburgreenthread is started only if lock has been acquired20:34
*** rloo has quit IRC20:34
max_loburgreenthread has a callback20:34
max_loburin the end20:34
max_loburadd_done_callback()20:34
max_loburthis callback will release the lock20:34
max_loburso it will be held untill greenthread is done20:34
devanandawhen is greenthread done?20:35
*** rloo has joined #openstack-ironic20:35
devanandawhat "ends" it?20:35
max_loburlet's see20:35
*** rloo has quit IRC20:35
devanandathe second RPC call will create a new greenthread20:35
*** rloo has joined #openstack-ironic20:35
devanandaso how will that cause the first one to finish, triggering the add_done_callback hook?20:35
*** rloo has quit IRC20:35
*** rloo has joined #openstack-ironic20:36
max_loburgreenthread is done when utils.node_power_action(task, task.node, new_state) returned20:36
max_loburor the part of utils.node_power_action(task, task.node, new_state) responsible for taking power action20:36
max_loburthe async part20:36
devanandamax_lobur: that isn't called by the first RPC call though20:36
devanandahow does the second RPC call attach to the greenthread created in the first one?20:37
*** agordeev2 has quit IRC20:37
devanandamaybe i'm missing something obvious...20:37
max_loburin threading model we have one rpc call20:37
lucasagomesI think that in the thread approach20:37
lucasagomesthere's no second rpc call20:37
lucasagomesit's only a sync call20:37
max_loburlucasagomes, ++20:37
lucasagomesthat will acquire the lock manually (not using with)20:37
lucasagomesand that will spawn a greenthread (if data is correct and validated)20:38
lucasagomesand return to the client20:38
devanandaahh20:38
max_loburit will acquire lock, do validation, tell greenthread to release lock at the end, start greenthread, return 200 to API20:38
lucasagomesso it's a sync call, but at the same time it's async20:38
max_loburlucasagomes, yea :D20:38
max_lobursync call which lefts something running after it20:39
lucasagomesyup20:40
devanandagotcha20:40
devanandathat makes sense20:40
devanandathanks for explaining!20:41
max_lobursure :)20:41
devanandaso the greenthread resumes and does the actual work of power_on right after the RETURN 200 is done20:41
lucasagomesyea20:41
max_loburno, right befor 200 is returned20:41
devanandaso there is tiny risk if the API tier is too slow for some reason20:41
devanandaright before?20:42
max_loburwe spawn greenthread in the step before return 20020:42
max_loburit starts running in parralel20:42
max_loburbecause we can't do anything after return :)20:42
max_loburwe'll need yield in this case, but that's another story :)20:43
lucasagomesmax_lobur, hmm well it needs to return to the client 20020:43
max_loburit does20:43
lucasagomesso clients knows that the request was accepted and is being processed (in the background by the greenthread)20:43
lucasagomesright ok20:43
lucasagomesso yea it does the actual work after the return (I mean it starts before but will finish after)20:44
max_loburhttps://review.openstack.org/#/c/66368/1/ironic/conductor/manager.py20:44
max_loburlucasagomes, ++20:44
max_loburutils.async_node_power_action20:44
max_loburspawns a greenthread20:44
max_loburand after this start_change_node_power_state returns control -> sends RPC response20:45
devanandahttps://review.openstack.org/#/c/66368/1/ironic/conductor/utils.py L 10120:45
devanandavalidates, then spawns a ThreadWithCallback20:45
max_loburahh20:46
devanandawhen taht returns, it'll trickle up and trigger the API return, yes?20:46
max_loburthat not a mistake20:46
max_loburdevananda, +20:46
devanandaright20:47
max_loburasync_node_power_action will either just return (which meands everything validated and the thread is running)20:47
max_loburor throw an exception20:47
max_loburfrom validators20:47
max_loburin both cases it will go back to the API20:47
devanandayep. or anode locked exception20:47
devanandaand for either exception, it won't start teh ThreadWithCallback20:48
max_loburor exception from task.driver.power.validate(node)20:48
lucasagomesbtw, node locked exception should be converted to conflict in the api layer20:48
max_loburyes20:48
devanandaactualy20:48
lucasagomesbut that's another thing, just a note here20:48
devanandaNodeLocked would come from https://review.openstack.org/#/c/66368/1/ironic/conductor/manager.py : 18920:48
*** derekh has joined #openstack-ironic20:48
max_loburdevananda, ++20:49
max_loburwhich is also before the spawning thread20:49
devanandaright20:49
devanandamax_lobur: stylistic question20:49
devanandamax_lobur: could we use a context manager at that layer (in conductor/manager.py) to achieve a similar ting with threads20:49
devanandamax_lobur: instead of adding manual_acquire, task.next(), etc20:50
max_loburI thought of it20:50
max_loburIt's can be done but20:50
max_lobur1. It will be more complex20:51
max_lobur2. It may not be obvious that this with won't release the lock20:51
max_lobur2 seems critical to me20:51
max_loburIt will be fake WITH20:51
devanandaah20:52
max_loburthat releases nothing when control goes out from it20:52
devanandastylistically, i'm not fond of what's there now, either20:52
devanandait's not clear when the lock is released20:52
devanandaall we see is endcallback = task.next()20:52
max_loburend_callback = lambda: task.next()20:52
devanandayea20:52
max_loburyes not obvious20:53
*** mrda has joined #openstack-ironic20:53
max_loburI wanted to rewrite this20:53
max_loburreplace generator with usual class20:53
max_loburwith meaningful methods20:53
devananda++20:53
devanandatask.start, task.end20:53
max_loburand then wrap it into generator for our automatic with20:53
max_loburyep20:53
devanandasounds good20:53
lucasagomeshmm I tend to agree, I thought that the two-steps lock (intent lock) would look a bit complicated to understand, but the thread approach is actually very complicated as well (when the lock is realeased, having a sync call which is not sync)20:54
lucasagomesboth will need some documentation20:54
devanandai think this could be much cleaner20:54
devanandai like the approach, though20:54
max_loburcool20:54
max_loburthen I'll polish the code nearest time20:54
max_loburand create tests20:54
devanandait'll be much easier to understand and debug it than a two-phase lock, in my experience20:55
devananda*based on my ..20:55
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Add 'get_sensor_data' to Command class  https://review.openstack.org/6793520:55
max_loburIt will shine as a new ferrari :)20:55
devanandait's very clear that API makes a call to Conductor to start a process. that requiers a lock. if the lock is granted, process starts in teh background20:55
lucasagomes:D20:55
devanandaif lock is not granted, it fails immediately20:55
devanandathere's a single RPC message + answer, regardless20:56
devanandaand so debugging it is much easier -- we always know that there should be RPC request + response20:56
devanandawhat needs to be clearer is when the Manager is starting a background greenthread, what that does, and when it will end20:56
devanandavs when the Manager is just doing something synchronously and returning it20:57
devanandaeg, I think it could be something like ....20:57
max_loburI changed change_node_power_state20:57
max_loburto start_change_node_power_state20:57
max_loburin my pathc20:57
devanandayea. i mean in the manager code20:58
max_loburwe can change to start_async_change_node_power_state20:58
devanandathe RPC method names are important, too20:58
max_loburyes, I'm talking about manager and the rpc api20:59
max_loburwe had change_node_power_state20:59
*** rloo has quit IRC20:59
devanandawhat about something like this20:59
max_loburand will have start_change_node_power_state20:59
devanandamethod = utils.node_power_action20:59
*** rloo has joined #openstack-ironic20:59
devanandaargs = [new_state]20:59
*** rloo has quit IRC20:59
devanandatask_manager.background(context, node_id, args)21:00
*** rloo has joined #openstack-ironic21:00
devanandavs, eg. for something that is synchronous21:00
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Add 'get_sensor_data' to Command class  https://review.openstack.org/6793521:00
devanandamethod = update_node21:00
devanandaargs = [values...]21:00
*** rloo has quit IRC21:01
devanandatask_manager.do(context, node_id, method, args)21:01
*** rloo has joined #openstack-ironic21:01
devananda... (woops, first example should have had "method, args" too)21:01
*** rloo has quit IRC21:02
devanandataht wasn't very clear. i can expand on that in a pastebin21:03
*** rloo has joined #openstack-ironic21:03
max_loburyep, would be great21:03
devanandateh gist is, i'd like to see the ConductorManager code be more consistent between the existing "with lock: do something synchronous" and the new "with thread: do something async"21:03
devanandak. i'll go sketch something out21:04
max_loburI see21:04
max_loburthanks!21:04
lucasagomesnice21:04
lucasagomesso right, should I abandon my patch and stop thinking about the two phases lock ?21:05
lucasagomesas I said to max_lobur I'm ok with both approachs21:05
max_loburah, wanted to ask21:05
max_loburdevananda, do you think we can have some other features requiring two phase lock?21:05
max_loburothers than avoiding race conditions21:06
max_loburI meant intent lock21:06
devanandaprobably many things will need to use this mechanism21:06
devanandapower, deploy, rescue, firmware,e tc21:06
devanandaanything that needs to make a physical change to a machine21:06
devanandashould probably validate that before it starts21:07
devanandaand return to user "OK, i'm going to start now"21:07
max_loburno, it's all that can be fixed with threads21:07
devanandaah, sorry21:07
max_loburI meant some others21:07
devanandano21:07
lucasagomesyea I can't think about any other use case as well21:07
max_loburthat will definitely require two subsequent RPC calls or even more21:07
lucasagomesapart from fixing a lock between two rpc calls21:07
devanandawhat comes to mind is taht, if we need to return a handle to the two-phase lock *to the user*21:08
devanandathen we can't do taht with a background thread21:08
max_loburtrue21:08
devanandabut that seems like a terrible idea :)21:08
max_loburso the user will gain the control over the node right?21:08
devanandai mean, let's NOT do taht21:08
lucasagomesheh21:08
lucasagomes+121:08
max_loburok :)21:08
devanandai'm just saying, everything else should be doable with bgthread21:08
max_loburok, thanks Guys for a discussion, I gotta hurry to the bus stop, otherwise I'll need to walk through the forest with a half meter snow :D21:09
devanandamax_lobur: lol. dont miss the bus :)21:10
devanandamax_lobur: and thanks for the patches and discussion, too!21:10
max_loburbye All!21:10
lucasagomesmax_lobur, hah see ya21:10
lucasagomesgnight21:10
*** max_lobur is now known as max_lobur_afk21:10
* devananda should walk home soon21:10
lucasagomesyea I'm overtime as well21:10
lucasagomesdevananda, thanks for ur input!21:11
devanandanp!21:11
lucasagomesI'm done for the day21:11
lucasagomesdevananda, have a g'night :D21:11
devanandag'night!21:12
*** lucasagomes has quit IRC21:12
devanandaok, noisy kids in cafe, i'm walking home... bbiafm21:27
mrdahey jh21:33
*** aignatov_ has quit IRC21:34
*** mdurnosvistov has quit IRC21:38
openstackgerritA change was merged to openstack/ironic: Fix non-unique pxe driver 'instance_name'  https://review.openstack.org/6565721:41
openstackgerritA change was merged to openstack/ironic: Fix non-unique tftp dir instance_uuid  https://review.openstack.org/6685821:42
devanandaback21:48
openstackgerritMatt Wagner proposed a change to openstack/ironic: API: Add sample() method on Node  https://review.openstack.org/6553621:55
*** datajerk has quit IRC21:55
*** martyntaylor has joined #openstack-ironic22:00
*** matty_dubs is now known as matty_dubs|gone22:01
*** linggao has quit IRC22:06
*** jdob has quit IRC22:10
openstackgerritGhe Rivero proposed a change to openstack/ironic: PXE instance_name is no longer mandatory  https://review.openstack.org/6797422:20
*** michchap has quit IRC22:30
*** michchap has joined #openstack-ironic22:31
openstackgerritA change was merged to openstack/ironic: Replace assertTrue with explicit assertIsInstance  https://review.openstack.org/6742022:49
*** datajerk has joined #openstack-ironic23:02
*** martyntaylor has quit IRC23:07
*** romcheg has quit IRC23:08
openstackgerritA change was merged to stackforge/pyghmi: Add 'get_sensor_data' to Command class  https://review.openstack.org/6793523:17
*** datajerk has quit IRC23:26
*** derekh has quit IRC23:30
devanandaHaomeng: ping23:32
devanandaHaomeng: how likely do you think it is that https://blueprints.launchpad.net/ironic/+spec/send-data-to-ceilometer will be done by i3?23:32
mrdahey devananda23:37
devanandahiya!23:38
mrdaso I was wondering if you have any low-hanging ironic fruit that I could look at?23:39
devanandai'm doing bug/bp cleanup today anyway23:39
devanandaso i'll start tossing things your way as I spot them. take what you like :)23:40
mrdacool - just looking to learn the code base and help wherever I can23:40
devanandahttps://bugs.launchpad.net/nova/+bug/1195073 should be closed, and another one opened and fixed23:40
devanandato remove teh pxe_deploy_timeout variable from drivers/modules/pxe.py23:40
devanandasince we don't have any mechanism today to enforce that23:40
mrdaok23:40
devanandathe variable is unused23:40
* devananda notes taht we will need to introduce a generic "tiemout" mechanism at a higher level, so a bug should be filed for that23:41
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Add 'get_health' to Command class  https://review.openstack.org/6798723:41
devanandamrda: this would be a slightly meatier bug: https://bugs.launchpad.net/ironic/+bug/126459623:42
mrdadevananda: I'll look at both.  First I'll close down +bug/1195073, open a new one, and then look at +bug/126459623:43
devanandamrda: https://bugs.launchpad.net/ironic/+bug/1255648 would be another good one, but may require real hardware with IPMI support to test your fix23:47
devanandamrda: let me ask, as there's a few more like taht -- do you have access to hardware to test against?23:49
devanandawith access to the IPMI network23:49
mrdadevananda: Bottom line, is that I don't right this moment.  But I need to get the lab back up again.  I might need to work up to that.23:49
devanandamrda: ack23:50
mrdadevananda: so with regards to https://bugs.launchpad.net/nova/+bug/1195073 , I should remove the Ironic tag as I've raised and linked https://bugs.launchpad.net/ironic/+bug/1270981 for the Ironic portion, but the nova half of this still needs action, so leave it open and unassigned?23:58
devanandamrda: yep23:58
devanandaand i just triaged the new bug23:58
mrdata23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!