Tuesday, 2014-02-04

*** ekarlso- has joined #openstack-ironic00:04
*** rloo has quit IRC00:11
*** rloo has joined #openstack-ironic00:11
*** matsuhashi has joined #openstack-ironic00:12
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Add futures to requirements  https://review.openstack.org/7017700:24
devanandaneeded a rebase00:24
*** mikal_ is now known as mikal00:29
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Implement a multiplexed VendorPassthru example  https://review.openstack.org/7086300:32
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Bump required version of stevedore  https://review.openstack.org/7086200:32
devanandamore rebasing coming up00:32
*** nosnos has joined #openstack-ironic00:33
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Move test__get_nodes_mac_addresses  https://review.openstack.org/7048100:33
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Implement _update_neutron in PXE driver  https://review.openstack.org/7046800:33
devanandalifeless: re your comment on https://review.openstack.org/#/c/55549/21 -- I believe we should avoid automatically breaking locks when a conductor has gone away.00:36
devanandalifeless: or, as a compromise, expose a CONF option that toggles between automaticaly breaking them and requiring manual release00:37
devanandaautomatic failover is the bane of many a sysadmin's existence ...00:37
lifelessdevananda: bad automatic failover is ...00:38
lifelessdevananda: put it this way, tripleo is aiming at scale, where we expect conductors to be failing every day or two just due to volume...00:38
devanandalifeless: failing over a stateful $thing in a potentially destructive way is always bad :)00:39
devanandanow, whether or not ironic supports destructive things in a particular deployment ma ybe the deciding factor00:39
devanandaeg, if you're not doing any firmware updates, then automatic failover is probably fine00:39
*** nosnos_ has joined #openstack-ironic00:44
lifelessdevananda: so I think thats a good point. We could say 'Ironic will failover automatically when a conductor goes offline: ensure that operations are resilient to a second conductor starting the same operation00:44
lifelessdevananda: ; you can be notified in your driver by method XYZ that a failover has occured, if you need to abort operations)00:44
lifelessdevananda: if we don't put this in now, it will be a lot harder/riskier later.00:44
lifelessdevananda: Note that no admin can ever be more sure about the safety of releasing a lock than Ironic00:45
lifelessdevananda: because e.g. a firmware update running on a remote machine is invisible to the admin if the conductor in question has died00:45
lifelessdevananda: and also other admins might be using the API concurrently...00:45
*** nosnos has quit IRC00:47
devanandalifeless: the admin can see, via Ironic's API, that a firmware update was in progress, and presumably can tell via some other means taht said conductor is or isn't dead00:47
lifelessdevananda: you have great faith in admins :)00:48
devanandaheh00:48
lifelessdevananda: IME if you give an admin a hammer, they will use it.00:48
devananda"ensure operations are idempotent and non-fatal when resumed on another conductor"00:48
lifelessdevananda: yup00:48
devanandais an interesting requirement00:48
devanandalifeless: right - so in the existing proposal's history, i tried to have as many safeguards put in place as possible, while still giving the admin a tool00:49
lifelessso to put it another way00:49
devanandalifeless: but i have the same concern. this is a dangerous tool to be handing out00:49
lifelessassume conductors will crash, both software and machine, at the worst possible time, eventually.00:49
lifelessI think thats a reasonable defensive assumption to make00:50
devanandaindeed00:50
*** ekarlso has quit IRC00:50
lifelessand having made it, resuming elsewhere doesn't seem like such a risk.00:50
devanandaright00:50
lifelessbut I still don't like giving *admins* lock breaking, because fumble-fingers...00:51
devanandalet me tease out an assumption from "resume elsewhere"00:51
devanandaevery operation which uses the locking mechanism to ensure other ops aren't conflicting00:52
devanandaneeds to be resumable00:52
devanandaand need to be tracked in such a way hat another conductor is guaranteed to resume it00:52
devanandaif the conductor holding said lock dies00:53
devanandamore and more, i'm seeing that we need a third service for task management00:53
devanandaor at minimum, a library and a datastore for task flows / states / something00:54
*** nosnos_ has quit IRC00:54
*** nosnos has joined #openstack-ironic00:54
devanandathat'll take some work, more than we can do for Icehouse00:56
NobodyCambrb00:57
NobodyCamlol wow quick on the "a devananda :-p01:09
NobodyCamlol01:09
NobodyCams/"/+/01:10
devanandayea... i saw it pass pep8, which is all that the change would have affected01:10
devanandaand it needs to land to unblock /everything/ else01:10
NobodyCamlol01:10
NobodyCam++01:11
devanandawe should see in a few minutes if the devstack change has fixed our gate problem01:11
NobodyCam:)01:11
* NobodyCam X fingers (but does not hold breath)01:12
devanandaI'm going to head out for dinner plans in a few minutes01:12
devanandai'll look when i get home...01:13
NobodyCam:)01:13
NobodyCamI wont text you the answer01:13
NobodyCamhave a good night01:13
*** rloo has quit IRC01:46
*** r-mibu has joined #openstack-ironic01:51
*** r-mibu has left #openstack-ironic01:53
NobodyCamuggg01:53
lifelessNobodyCam: devananda: devstack change was in the wrong place (russell's was too) - new fix proposed, waiting reviewers.01:56
*** epim has quit IRC02:17
NobodyCamack TY lifeless :)02:26
*** dkehn_ has joined #openstack-ironic03:03
*** Haomeng has joined #openstack-ironic03:07
openstackgerritJoe Gordon proposed a change to openstack/ironic: Update nova.conf.sample for python-keystoneclient 0.5.0  https://review.openstack.org/7089603:13
*** Haomeng has quit IRC03:56
*** harlowja is now known as harlowja_away04:13
*** harlowja_away is now known as harlowja04:15
*** coolsvap has joined #openstack-ironic05:21
*** rwsu has quit IRC05:41
*** killer_prince has joined #openstack-ironic05:47
*** nosnos_ has joined #openstack-ironic06:00
*** nosnos has quit IRC06:03
openstackgerritJenkins proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/6802406:05
*** jcooley_ has joined #openstack-ironic06:26
*** datajerk1 has joined #openstack-ironic06:31
*** mdurnosvistov has joined #openstack-ironic06:31
*** coolsvap_away has joined #openstack-ironic06:32
*** matsuhashi has quit IRC06:33
*** matsuhashi has joined #openstack-ironic06:33
*** SpamapS_ has joined #openstack-ironic06:33
*** coolsvap has quit IRC06:38
*** SpamapS has quit IRC06:38
*** mdurnosvistov__ has quit IRC06:38
*** datajerk has quit IRC06:38
*** zul has quit IRC06:38
*** med_ has quit IRC06:38
*** mordred has quit IRC06:38
*** mrda is now known as mrda_away06:43
*** harlowja is now known as harlowja_away06:44
*** zul has joined #openstack-ironic06:46
*** mordred has joined #openstack-ironic06:54
*** coolsvap_away has quit IRC07:04
*** aignatov_ is now known as aignatov07:08
*** jcooley_ has quit IRC07:12
*** saju_m has joined #openstack-ironic07:12
*** coolsvap has joined #openstack-ironic07:26
GheRiveromorning Ironic07:38
*** rwsu has joined #openstack-ironic07:42
*** aignatov is now known as aignatov_07:44
*** jcooley_ has joined #openstack-ironic07:48
*** rwsu has quit IRC07:51
*** mdurnosvistov_ has joined #openstack-ironic07:52
*** rwsu has joined #openstack-ironic07:53
*** jcooley_ has quit IRC07:56
*** killer_prince is now known as lazy_prince08:08
*** jistr has joined #openstack-ironic08:11
*** mdurnosvistov_ has quit IRC08:16
*** yuriyz has joined #openstack-ironic08:30
*** coolsvap has quit IRC08:36
*** ifarkas has joined #openstack-ironic08:41
*** pradipta has joined #openstack-ironic08:43
*** romcheg has joined #openstack-ironic08:45
*** aignatov_ is now known as aignatov08:45
*** lsmola has joined #openstack-ironic08:56
*** athomas has joined #openstack-ironic09:00
*** ndipanov has joined #openstack-ironic09:02
*** pradipta has quit IRC09:03
*** coolsvap has joined #openstack-ironic09:23
mdurnosvistovMorning all! :)09:27
*** sanek11 is now known as agordeev09:28
yuriyzmorning mdurnosvistov, how are you?09:33
*** mdurnosvistov has quit IRC09:34
*** lucasagomes has joined #openstack-ironic09:34
*** mdurnosvistov has joined #openstack-ironic09:37
mdurnosvistovyuriyz: fine thx =) and you?09:44
yuriyzok09:44
*** tatyana has joined #openstack-ironic09:47
*** romcheg has left #openstack-ironic09:52
*** aignatov is now known as aignatov_10:13
*** martyntaylor has joined #openstack-ironic10:14
*** jcooley_ has joined #openstack-ironic10:17
*** jcooley_ has quit IRC10:23
*** jistr has quit IRC10:27
*** coolsvap has quit IRC10:30
*** lazy_prince has quit IRC10:46
*** jistr has joined #openstack-ironic10:50
openstackgerritMax Lobur proposed a change to openstack/ironic: Remove unnecessary json dumps/loads from tests  https://review.openstack.org/7041310:57
*** russellb has joined #openstack-ironic11:00
*** aignatov_ is now known as aignatov11:01
*** pradipta has joined #openstack-ironic11:02
*** jcooley_ has joined #openstack-ironic11:13
*** jcooley_ has quit IRC11:19
*** nosnos_ has quit IRC11:30
*** Haomeng has joined #openstack-ironic11:39
*** aignatov is now known as aignatov_12:00
*** aignatov_ is now known as aignatov12:12
*** aignatov is now known as aignatov_12:22
*** aignatov_ is now known as aignatov12:24
openstackgerritJenkins proposed a change to openstack/ironic: Updated from global requirements  https://review.openstack.org/6882312:34
*** matsuhashi has quit IRC12:36
*** pradipta has quit IRC12:36
*** matsuhashi has joined #openstack-ironic12:36
*** matsuhashi has quit IRC12:41
*** jroll has joined #openstack-ironic13:18
*** rainya has joined #openstack-ironic13:27
openstackgerritLucas Alvares Gomes proposed a change to openstack/python-ironicclient: Add node-states command  https://review.openstack.org/7097913:28
lucasagomesNobodyCam, ^, will be used in the driver as well, right now we are fetching the whole node resource in order to know the states13:29
*** jdob has joined #openstack-ironic13:30
*** athomas has quit IRC13:33
*** athomas has joined #openstack-ironic13:41
*** saju_m has quit IRC13:41
*** rloo has joined #openstack-ironic13:41
*** ekarlso- has quit IRC13:45
*** ekarlso has joined #openstack-ironic13:45
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Handle multiple exceptions raised by jsonpatch  https://review.openstack.org/6845713:46
*** jbjohnso has joined #openstack-ironic13:50
*** ifarkas has quit IRC13:51
*** ifarkas has joined #openstack-ironic13:52
*** lsmola has quit IRC14:05
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Handle multiple exceptions raised by jsonpatch  https://review.openstack.org/6845714:20
*** jbjohnso has quit IRC14:21
*** lsmola has joined #openstack-ironic14:22
*** jbjohnso has joined #openstack-ironic14:24
*** matty_dubs|gone is now known as matty_dubs14:29
*** medberry has joined #openstack-ironic14:40
*** medberry has joined #openstack-ironic14:40
*** lsmola has quit IRC14:41
*** coolsvap has joined #openstack-ironic14:42
*** jdob_ has joined #openstack-ironic14:49
*** jdob has quit IRC14:49
*** jcooley_ has joined #openstack-ironic14:51
*** lsmola has joined #openstack-ironic14:53
*** jcooley_ has quit IRC14:57
*** jbjohnso has quit IRC15:03
*** lsmola has quit IRC15:05
*** jdob has joined #openstack-ironic15:13
NobodyCamgood morning Ironic15:16
*** jbjohnso has joined #openstack-ironic15:17
GheRiveromorning NobodyCam15:18
*** lsmola has joined #openstack-ironic15:18
NobodyCam:)15:18
NobodyCammaken coffee15:18
lucasagomesmorning NobodyCam GheRivero15:19
*** jdob_ has quit IRC15:19
NobodyCammorning lucasagomes :)15:19
rloomorning NobodyCam. Hi GheRivero, lucasagomes ;)15:19
lucasagomesmorning rloo15:19
lucasagomes:)15:19
yuriyzmorning all :)15:19
GheRiveromorning *15:19
NobodyCammorning rloo15:19
NobodyCammorning yuriyz15:19
NobodyCamlol15:19
lucasagomesheh morning all (easier)15:19
lucasagomesheh15:19
NobodyCamwho on first15:19
rloohi yuriyz!15:19
NobodyCams/who/who's/15:19
lucasagomesNobodyCam, r u working on a new patch-set for the driver?15:20
lucasagomesI was thinking about start adding some tests to it later15:20
lucasagomes(usually things get a long time to land in nova so we might start doing some cleaning up on that code)15:20
NobodyCam:) i always accect help with tests15:20
lucasagomes(and adding the missing parts)15:20
lucasagomesheh cool15:21
NobodyCam:-p15:21
lucasagomesNobodyCam, I found a problem today, couldn't investigate how that happened15:21
lucasagomesbut in a nova delete15:21
lucasagomesit deleted the instance from nova but it wasn't undeployed by ironic15:21
NobodyCamhumm15:21
max_loburmorning Ironic15:22
NobodyCammorning max_lobur :)15:22
lucasagomeslooking at the logs I only found some exceptions raised by the n-cpu15:22
lucasagomesNobodyCam, http://paste.openstack.org/show/62444/15:22
lucasagomesmorning max_lobur15:22
* NobodyCam looks15:22
lucasagomesNobodyCam, idk looks like a nova bug15:23
lucasagomesbecause the destroy() method doesn't seems to be called15:23
*** jistr has quit IRC15:23
lucasagomesand looking at the ironic api log no /states/provision was called as well15:23
*** jistr has joined #openstack-ironic15:23
*** kobier has joined #openstack-ironic15:24
NobodyCamdid the node power off?15:24
lucasagomesnop15:24
lucasagomesit might be my enviroemnt as well (testing env, with many manual tweaks)15:24
NobodyCamlooks like it tried to power off15:25
lucasagomesI gotta clean it up for a more valuable testing to be honest15:25
NobodyCamhumm at least Get the power state15:25
lucasagomesyea15:26
lucasagomesgot it but didn't issue any power off15:26
NobodyCamhummm15:26
lucasagomesthat GET could be part of the periodic task syncing the states as well15:26
lucasagomesheh anyhoo, no worries I will investigate that15:26
NobodyCam:)15:26
NobodyCamlucasagomes: nothing in the ironic logs?15:28
lucasagomesNobodyCam, nothing apparently15:28
lucasagomeslike no PUT request for the states15:28
NobodyCam:(15:28
lucasagomeswhich would be the first thing to do in a destroy()15:28
NobodyCamya15:29
lucasagomesthat's why I think that nova called destroy() but it failed to be routed (those exceptions)15:29
lucasagomescalled on the api, but the message didn't go to the conductor I mean15:29
NobodyCamya that would be my first guess with that log15:30
NobodyCamany word on devstack and the gate?15:31
lucasagomesI put up a guide on how to setup ironic + nova with devstack15:32
lucasagomesbut to make it more automatic devstack would need some fixes in place15:32
NobodyCamlucasagomes: gate was broken lastnight15:32
lucasagomesNobodyCam, yea15:32
lucasagomeswell that's idk about15:32
lucasagomes:(15:32
NobodyCam:)15:32
lucasagomeshoping to get it fixed, still broke15:32
NobodyCamwounder if it would be worth making the tempest tests non voting until things are working15:45
*** jcooley_ has joined #openstack-ironic15:46
lucasagomesNobodyCam, +115:47
devanandamorning, all15:48
NobodyCamgood morning devananda15:49
lucasagomesdevananda, morning15:49
NobodyCamI've got a recheck running on 70869 atm15:50
lucasagomesdevananda, NobodyCam, that volume_driver part of the Ironic driver can we split it out in another review (/me trying to see what can be splitted out of that driver to make small patch-sets)15:50
devanandaNobodyCam: great, i was just goign to kick that off15:50
lucasagomes>15:50
lucasagomes?*15:50
*** aignatov is now known as aignatov_15:50
devanandalucasagomes: yes15:50
lucasagomesdevananda, ack15:50
NobodyCamdevananda: yes, I am splitting the schudler out too15:50
devanandagreat15:51
lucasagomesNobodyCam, ok15:51
lucasagomesNobodyCam, r u going to submit any code soon?15:51
*** kobier has quit IRC15:51
*** jcooley_ has quit IRC15:51
NobodyCamlucasagomes: yes, yesterday got away from me15:51
lucasagomesNobodyCam, ack15:52
devanandalucasagomes: for sunjing's console patch (which is presently abandoned) we'll need to expose another state in the API15:52
devanandalucasagomes: some way for client to request turning on/off the console15:52
devanandai think we can do this the same way, eg, node/xxx/state/console15:53
NobodyCamdevananda: tempest still failing, any thought on making the tespest tests non voting until things are back in shape?15:54
devanandaNobodyCam: let's see if they still fail15:55
NobodyCamthey did on 7086915:55
devanandaoh15:55
lucasagomesdevananda, I see, I will take a look15:56
lucasagomesalso about states: https://review.openstack.org/#/c/70979/15:56
lucasagomesit's not critical because we can get it from fetching the whole node resource (as we do now in the driver)15:56
lucasagomesbut ^ would be handy15:56
lucasagomesget it = get the states15:57
devanandaNobodyCam: some feedback from internal folks on the nova "ironic" driver -- string matching on the ironic driver name is not ideal.15:59
devanandaeg, "if 'pxe' in node.driver"15:59
devanandaNobodyCam: the Nova driver needs to be able to support different Ironic drivers.15:59
NobodyCamdevananda: ack16:00
NobodyCamwill take a look16:00
devanandaNobodyCam: like we did for the SSH driver, I suspect we'll need to have a module with some abstraction of different properties16:00
NobodyCamam calling into the cloudos talk now16:01
devanandak k16:01
NobodyCamyes16:01
NobodyCamits on tempest today16:02
*** jcooley_ has joined #openstack-ironic16:09
NobodyCamdevananda: see the comments on the review :)16:09
NobodyCamlucasagomes: are you goung to push up today?16:09
lucasagomesNobodyCam, I will try16:10
NobodyCam:)16:10
lucasagomesNobodyCam, do u wanna to push up another review removing the scheduler part before?16:10
NobodyCamI can16:10
lucasagomesNobodyCam, ack, so I can split the volume part16:11
NobodyCamok16:11
NobodyCamlet me see where I left things16:11
lucasagomesheh ok16:11
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Clean up duplicated code in objects  https://review.openstack.org/7101816:15
devanandaanyone want to dig through logs with me to find where https://review.openstack.org/#/c/70869/ failed?16:16
NobodyCamhttp://logs.openstack.org/69/70869/1/check/check-tempest-dsvm-ironic/cb9f171/console.html#_2014-02-04_12_39_27_83916:18
NobodyCamwait wrong paste16:18
NobodyCamhttp://logs.openstack.org/69/70869/1/check/check-tempest-dsvm-ironic/cb9f171/console.html#_2014-02-04_12_39_26_87016:20
devanandaNobodyCam: right. that's the tempest failure. i need the underlying failure in some project, program, kernel, etc16:21
NobodyCamahh16:22
GheRiveroguestfs?16:22
*** hemna has joined #openstack-ironic16:23
devanandaGheRivero: guestfs issue should have been fixed in https://review.openstack.org/#/c/70885/16:29
devanandahowever that patch seems to have broken nova in the tempest-dsvm-ironic tests16:30
devanandalook at n-cpu log file... there are no errors, but also no work done. Only one instance ever claimed, instead of dozens16:30
lucasagomeshttp://paste.openstack.org/show/62456/ these r all the erros I can see on that log16:31
devanandalucasagomes: yes, in the tempest log. but why did those tempest actions fail? normally, there aer corresponding errors in the service logs16:32
*** max_lobur has quit IRC16:33
lucasagomesdevananda, I see16:33
lucasagomesalthough there's a bunch of timeouterror16:33
lucasagomeseven the builderrorexception there are being raised in a function called "wait_for_server_status"16:34
lucasagomeswhich might be due a timeout as well16:34
devanandaright, but what's failing?16:34
*** hemna has quit IRC16:34
devanandai see a lot of timeouts in the n-scheduler log file16:34
devanandabut none in n-cpu16:35
*** max_lobur has joined #openstack-ironic16:35
lucasagomeslemme digg more into the services16:36
GheRiverohttp://logs.openstack.org/69/70869/1/check/check-tempest-dsvm-ironic/cb9f171/logs/libvirtd.txt.gz16:36
lucasagomesNobodyCam, btw, I got a review ready to remove that volume_driver, and another review adding it back (another patch-set)16:36
lucasagomesNobodyCam, can I go ahead with it?16:37
NobodyCamlucasagomes: yes16:38
lucasagomesNobodyCam, ack16:38
devanandaGheRivero: interesting. do those errors exist in previous successful runs of check-tempest-dsvm-ironic?16:38
GheRiverolooking for that16:39
*** lsmola has quit IRC16:39
GheRiveronot16:39
GheRiverohttp://logs.openstack.org/71/66071/19/gate/gate-tempest-dsvm-ironic/8df0bf6/logs/libvirtd.txt.gz16:39
devanandaGheRivero: yes -- http://logs.openstack.org/96/66996/3/gate/gate-tempest-dsvm-ironic/4b78840/logs/libvirtd.txt.gz16:39
*** jcooley_ has quit IRC16:40
GheRiveronot those related to /opt/stack/data/nova/instances/UUID16:41
devanandaah, right16:41
*** shausy has joined #openstack-ironic16:41
ifarkasdevananda, hey, I cc-ed you to an email regarding https://bugs.launchpad.net/ironic/+bug/1264596, for which you asked about its status on the meeting yesterday.16:44
lucasagomesNobodyCam, https://review.openstack.org/#/c/71026/16:44
* NobodyCam clicks16:44
NobodyCam:) lucasagomes are you going to set WIP?16:45
ifarkasdevananda, I wasn't working on it because Michael Davies asked me if he can jump on it a couple weeks ago, so I was surprised that it's still assigned to me. But I unassigned myself from the bug earlier today.16:45
lucasagomesNobodyCam, hah lemme do that (or people will start complanting about the libvirt thing again)16:45
devanandaifarkas: ack, thanks16:45
ifarkasdevananda, np, sorry for the confusion16:47
*** jistr has quit IRC16:47
devanandaGheRivero: that's not the problem, it seems. search logstash for virSecurityDACRestoreSecurityFileLabel  AND build_status:"SUCCESS"16:49
*** shausy has quit IRC16:49
devanandaGheRivero: more builds with that message succeed than fail16:49
GheRivero:?16:50
devanandaGheRivero: http://logstash.openstack.org/#eyJzZWFyY2giOiJ2aXJTZWN1cml0eURBQ1Jlc3RvcmVTZWN1cml0eUZpbGVMYWJlbCAgQU5EIGJ1aWxkX3N0YXR1czpcIlNVQ0NFU1NcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL2xpYnZpcnRkLnR4dFwiIiwiZmllbGRzIjpbImZpbGVuYW1lIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMzYwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTE1MzI2NjAwMTIsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=16:51
devanandathat libvirt error is not the culprit16:51
*** lsmola has joined #openstack-ironic16:51
GheRiverogood for me :)16:52
*** thedodd has joined #openstack-ironic16:54
devanandaok, closer16:55
devanandalook in nova scheduler log16:55
devananda"Filter ComputeFilter returned 0 hosts"16:55
devanandaum16:57
devanandan-cpu isn't running16:57
NobodyCambrb17:02
*** matty_dubs is now known as matty_dubs|lunch17:03
* devananda steps afk17:05
ekarlsolifeless: you around boss ?17:06
*** digambar has joined #openstack-ironic17:11
*** ifarkas has quit IRC17:12
*** pradipta has joined #openstack-ironic17:28
*** mdurnosvistov_ has joined #openstack-ironic17:32
*** jbjohnso has quit IRC17:33
*** jcooley_ has joined #openstack-ironic17:34
devananda70869 failed again17:34
*** jrist has quit IRC17:35
digambarHey17:36
lifelessekarlso: hi17:36
digambarI have installed the ironic with devstack17:37
NobodyCamlifeless: do you have a link for the devstack review you mentioned last night?17:37
*** athomas has quit IRC17:37
digambarbut I have installed it on VM17:37
digambarwill be ok for development ?17:38
NobodyCamdigambar: awesome: did you follow https://etherpad.openstack.org/p/IronicDeployDevstack17:38
lifelessNobodyCam: which one?17:38
digambarnops17:38
digambarlet me go through it'17:38
lifelessNobodyCam: possibly you're thinking of Ib618da1bd21da09f8855ec4691bff79c4c3b3d9c17:38
NobodyCamlifeless: last night you said 7:56 | lifeless > NobodyCam: devananda: devstack change was in the wrong place (russell's was too) - new fix proposed, waiting reviewers.17:39
devanandalifeless: that fix changed what is broken in our gate, but did not unbreak it17:39
lifelessdevananda: yeah17:39
lifelessdevananda: turns out there is a honking great race condition in nova; diagnosis proceeding in #openstack-nova now17:39
lifelesssame thing that was breaking https://review.openstack.org/#/c/70239/17:40
NobodyCamTy lifeless :)17:42
*** hemna has joined #openstack-ironic17:43
*** SpamapS_ is now known as SpamapS17:43
*** jcooley_ has quit IRC17:44
digambarHey17:45
digambar./tripleo-incubator/scripts/setup-network17:45
digambarafter firing above command got the error17:45
digambar./tripleo-incubator/scripts/setup-network17:45
digambarerror: command 'net-list' doesn't support option --persistent17:45
lucasagomesdigambar, can you paste the error please?17:45
lucasagomesdigambar, it's the version of ur virsh command17:45
lucasagomesshould be 1.1.*+17:46
lucasagomes[lucasagomes@lucasagomes ironic]$ virsh --version17:46
lucasagomes1.1.3.317:46
digambarok17:46
*** creiht has quit IRC17:46
*** jbjohnso has joined #openstack-ironic17:46
digambarvirsh --version == 0.9.817:46
digambarNeed to update then17:47
*** creiht has joined #openstack-ironic17:47
lucasagomes:) yup17:47
digambardo you have any link for ubuntu to update the virsh17:47
*** jrist has joined #openstack-ironic17:48
digambarI mean libvert version17:49
lucasagomesdigambar, hmm in fedora I think it's libvirt-client17:49
lucasagomes.17:49
digambarhmm17:49
* lucasagomes don't know the debian equivalent package name17:49
digambarok17:49
digambarlet me check17:49
devanandaso folks, our present gate breakage is being looked into by nova17:51
devanandaapparently a nasty thread race condition if anyone's interested17:51
lucasagomesdigambar, dpkg -S /usr/bin/virsh ?17:51
digambarok17:52
lucasagomesdigambar, if that's the path to the virsh command on ubuntu17:52
digambaryes17:52
digambarlibvirt-bin: /usr/bin/virsh17:52
lucasagomes:)17:52
lucasagomesso libvirt-bin is the package :)17:53
digambares17:54
digambaryes17:54
lucasagomesok I gotta run (came to the office today)17:55
lucasagomesnight devananda NobodyCam digambar17:55
NobodyCamnight lucasagomes17:55
devanandalucasagomes: g'night17:55
digambarNight Luca17:55
*** lucasagomes has quit IRC17:55
*** jrist has quit IRC17:56
*** harlowja_away is now known as harlowja17:59
NobodyCambrb18:00
*** matty_dubs|lunch is now known as matty_dubs18:01
*** max_lobur is now known as max_lobur_afk18:05
*** jrist has joined #openstack-ironic18:09
*** coolsvap is now known as coolsvap_away18:16
*** tatyana has quit IRC18:21
* NobodyCam makes another cup of coffee18:22
*** jcooley_ has joined #openstack-ironic18:24
*** shausy has joined #openstack-ironic18:25
shausyHi18:25
NobodyCamshausy: hello, welcome to Ironic18:25
shausyin baremetal provsioning, how the tftboot is communicating to compute and provsioning instance ?18:26
*** jcooley_ has quit IRC18:26
*** dkehn has quit IRC18:27
*** martyntaylor has left #openstack-ironic18:28
*** dkehn has joined #openstack-ironic18:28
NobodyCamshausy: in Ironic or nova bare metal, both use iscsi and a deployment ramdisk for pxe18:30
shausyI have an issue while booting instance, its keeping for tftp and gets timedout18:30
NobodyCamshausy: are you using nova baremetal or ironic18:30
shausynova-baremetal18:30
NobodyCamdo you see any errors on the odes console?18:31
*** wendar_ is now known as wendar18:31
NobodyCam*nodes18:31
shausyi dont see any errors, once its gets the dhcp lease its acquire the IP and getting strucked in tftp process and getting timeout18:32
NobodyCamdo you see anything in from baremetal deploy helper18:33
shausygive me sometime, let me connect to my server and have a look18:35
NobodyCamsure18:35
shausythanks18:35
*** jistr has joined #openstack-ironic18:42
* NobodyCam makes a bagel18:44
shausy@NobodyCam  this is the only log i could see as of now in console "PXE-E32: TFTP open timeout18:50
shausy"18:50
devanandamikal: iirc, you had some tools for reviewing reviewers? was there anything that aggregated someone's review feedback (like i want to see all the comments left by user 1234)18:50
devanandawoops, wrong channel :)18:51
*** thedodd has quit IRC18:51
NobodyCamhumm18:52
NobodyCamshausy: has this EVER worked or is this a new install?18:52
shausywas working for couple of instances launch18:53
shausynow only im facing this issue18:53
NobodyCam:) real HW or vm;s18:54
shausyreal hw18:54
*** jcooley_ has joined #openstack-ironic18:55
shausyonce the server gets the dhcp ip , how the pxe images pushed to the hw ?18:55
NobodyCamvia bmaremetal deploy helper18:56
shausyi dont see any errors in deploy helper too :(18:57
NobodyCamanything change with firewalls or switch config?18:57
*** tatyana has joined #openstack-ironic18:58
shausyjust i have rebooted, nothing else18:59
shausyi feel so service is gone out, that could be an issue19:00
shausydo we need any service to help baremetal-delpoy-helper ?19:00
openstackgerritA change was merged to openstack/ironic: Fix missing keystone option in ironic.conf.sample  https://review.openstack.org/7086919:00
NobodyCamdevananda:  A change was merged to openstack/ironic W)) HOO19:01
NobodyCamshausy: how did you setup the env ? did you use DiskImage Builder19:02
NobodyCamsudo service nova-baremetal-deploy-helper restart19:02
shausyyes images was build using disk image builder19:03
shausyi have restarted the compute n baremetal in conpute node19:03
shausydo we need to have dnsmasq up and running ?19:04
NobodyCamare you using neutron?19:04
shausyno we are still in quantum19:05
devanandawooot!19:05
NobodyCamdeva with quantun (nova network) we need dnsmasq correct?19:06
shausyusing quantum dhcp for pxe booting the instances19:06
devanandaNobodyCam: yes19:07
NobodyCamshausy: ^^19:07
devanandawell19:07
devanandaat one point, nova "baremetal" driver worked with nova-network by using file injection and static IPs assigned by dnsmasq19:08
NobodyCamshausy: anything in the compute nodes syslog for / about  tft attempts or errors?19:08
devanandathis was definitely removed prior to Havana. I think it was only in Grizzly and fixed very early in H- cycle19:08
devanandaI don't think this worked with quantum19:09
devanandaquantum != nova network19:09
NobodyCamya thats what kinda thought19:09
NobodyCamack19:09
shausyi dont see any errors in compute node syslog19:11
shausyin controller   sys Feb  4 13:09:04 d00-50-56-9c-4f-7a dnsmasq-dhcp[2793]: DHCPOFFER(tap0819589e-ca) 172.16.1.5 38:ea:a7:0f:3f:5e19:11
shausyFeb  4 13:09:08 d00-50-56-9c-4f-7a dnsmasq-dhcp[2793]: DHCPREQUEST(tap0819589e-ca) 172.16.1.5 38:ea:a7:0f:3f:5e19:11
shausyFeb  4 13:09:08 d00-50-56-9c-4f-7a dnsmasq-dhcp[2793]: DHCPACK(tap0819589e-ca) 172.16.1.5 38:ea:a7:0f:3f:5e host-172-16-1-519:11
NobodyCamdevananda: you re kicked the checks19:12
shausywhen instance was launched successfully i got this messages in log19:12
shausyFeb  4 10:37:26 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/pxelinux.0 to 172.16.1.419:12
shausyFeb  4 10:37:38 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: file /tftpboot/pxelinux.cfg/4cfcae2e-0f51-0650-9515-7bc83c2eed0b not found19:12
shausyFeb  4 10:37:42 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/pxelinux.cfg/01-38-ea-a7-0f-42-02 to 172.16.1.419:12
shausyFeb  4 10:37:44 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/pxelinux.0 to 172.16.1.319:12
shausyFeb  4 10:37:47 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/2ba19b48-7a10-4562-b687-31a71e71780d/deploy_kernel to 172.16.1.419:13
shausyFeb  4 10:37:56 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: file /tftpboot/pxelinux.cfg/184987f1-ebae-9257-935a-846cd1af8961 not found19:13
shausyFeb  4 10:38:00 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/pxelinux.cfg/01-38-ea-a7-0f-3f-5e to 172.16.1.319:13
shausyFeb  4 10:38:05 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/94b97407-44b0-46ed-9063-e309824f2ea1/deploy_kernel to 172.16.1.319:13
shausyFeb  4 10:38:17 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/2ba19b48-7a10-4562-b687-31a71e71780d/deploy_ramdisk to 172.16.1.419:13
shausyFeb  4 10:38:34 d00-50-56-9c-74-0c dnsmasq-tftp[23439]: sent /tftpboot/94b97407-44b0-46ed-9063-e309824f2ea1/deploy_ramdisk to 172.16.1.319:13
shausyFeb  4 10:39:00 d00-50-56-9c-74-0c kernel: [12129.891272] scsi3 : iSCSI Initiator over TCP/IP19:13
NobodyCamshausy: yes19:13
shausybut now i feel some service is down, thats why /tftpboot is getting timedout19:13
shausyafter restarting the dnsmasq its starts working19:14
NobodyCamya19:14
NobodyCamlooks like some thing was up with dnsmasq19:15
shausyi have one qns, so tftp is linked in dnsmasq ?19:15
*** tatyana has quit IRC19:15
NobodyCamshausy: yes19:15
shausywhy its needed, dnsmasq is s kind of dhco server right19:15
shausywhat role of dnsmasq inn tftp ?19:16
NobodyCamyes, setting the pxe boot options19:16
*** comstud has joined #openstack-ironic19:17
shausywe use quantum to take care of dhcp boot options19:17
NobodyCamI dont believe quantum will set pxe options19:17
*** jcooley_ has quit IRC19:17
shausythis is coming from quantum dnsmasq-dhcp[2793]: DHCPOFFER(tap0819589e-ca) 172.16.1.5 38:ea:a7:0f:3f:5e19:17
shausyoh ok19:18
NobodyCamlooks to me like it is comming form dnsmasq19:18
shausythanks alot19:18
NobodyCamAnytime19:19
shausydo we have any docs which talks abt internals of nova-baremetal and deploy helper ?19:19
NobodyCamyou've seen https://wiki.openstack.org/wiki/Baremetal ?19:19
shausyya i have19:20
shausyjust want to know  internals of it19:21
openstackgerritA change was merged to openstack/ironic: Add futures to requirements  https://review.openstack.org/7017719:22
shausy@NobodyCam & Devaanda thanks a lot19:22
NobodyCamshausy: take a lok at the code.19:23
shausySure19:23
NobodyCamhttps://github.com/openstack/nova/blob/master/nova/cmd/baremetal_deploy_helper.py19:23
*** hstimer has joined #openstack-ironic19:30
*** jcooley_ has joined #openstack-ironic19:36
*** jcooley_ has quit IRC19:44
openstackgerritA change was merged to openstack/ironic: Add lockfile>=0.8 to requirements.txt  https://review.openstack.org/6935819:47
openstackgerritA change was merged to openstack/ironic: Sync db.sqlalchemy code from Oslo  https://review.openstack.org/6900419:47
*** jcooley_ has joined #openstack-ironic19:49
*** aignatov_ is now known as aignatov19:52
*** mdurnosvistov_ has quit IRC19:59
*** shausy has quit IRC19:59
mikaldevananda: no, sorry. I don't track comment text like that.20:00
devanandamikal: ack20:01
*** jistr has quit IRC20:03
*** jcooley_ has quit IRC20:03
*** mdurnosvistov_ has joined #openstack-ironic20:04
*** jcooley_ has joined #openstack-ironic20:07
NobodyCambrb20:10
*** thedodd has joined #openstack-ironic20:14
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Clean up duplicated code in objects  https://review.openstack.org/7101820:18
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Fix 'run_as_root' parameter check in utils  https://review.openstack.org/7032420:19
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Remove unnecessary json dumps/loads from tests  https://review.openstack.org/7041320:20
* devananda rebases all the things20:20
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Removed duplicated function to create a swap fs  https://review.openstack.org/6872220:21
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Move test__get_nodes_mac_addresses  https://review.openstack.org/7048120:22
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Process public API list as regular expressions  https://review.openstack.org/6662120:26
NobodyCamdevananda: got a second for https://review.openstack.org/#/c/70979/20:38
* devananda looks20:38
*** max_lobur has joined #openstack-ironic20:38
*** kobier has joined #openstack-ironic20:39
openstackgerritA change was merged to openstack/ironic: Replace CONF.set_default with self.config  https://review.openstack.org/7039720:41
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Fix log and test for NeutronAPI.update_port_dhcp_opts  https://review.openstack.org/7109420:45
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Implement _update_neutron in PXE driver  https://review.openstack.org/7046820:45
devanandaNobodyCam: this just gets the states, ya? nothing more?20:46
NobodyCamya so you can see what's in what state20:47
devanandaisn't that part of node-show already?20:48
devanandaNobodyCam: so i dont think that's useful on the CLI20:51
devanandabut it is usedul in the python lib20:51
*** max_lobur has quit IRC20:53
NobodyCam:) we can exclude from cli20:53
*** max_lobur has joined #openstack-ironic20:53
devanandacommented20:53
NobodyCam:)20:54
devanandaNobodyCam: two easy ones for you: https://review.openstack.org/#/c/71094/1 and https://review.openstack.org/#/c/70481/320:54
* NobodyCam looks20:55
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/6802420:55
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Allow concurrent image downloads in pxe logic  https://review.openstack.org/6390420:56
*** sgrasley has joined #openstack-ironic20:57
*** jrist has quit IRC20:58
openstackgerritA change was merged to openstack/ironic: Removed duplicated function to create a swap fs  https://review.openstack.org/6872221:01
devanandaNobodyCam: also https://review.openstack.org/#/c/70862/221:01
NobodyCamdevananda: lin #91 of https://review.openstack.org/#/c/71094/1/ironic/common/neutron.py21:02
devanandaya21:02
NobodyCamdon't you want a %s or %{port_id}s21:03
NobodyCamor a :21:03
NobodyCamlol21:03
devanandanope21:03
devanandasee lifeless' earlier comment about that, and mien21:04
devanandamine21:04
NobodyCamcomments on 71094?21:05
devanandaNobodyCam: so, one min... while I dig into this21:06
*** hstimer has quit IRC21:07
devanandai'm testing it out to see if, in fact, LOG.exception is workign the same fo rus as for nova21:07
jbjohnsodevananda, fyi, I might be making a big change to the underpinnings of session soon.  I think I figured out how it can be monkey patched with impunity21:08
jbjohnsoof pyghmi that is21:08
devanandainteresting21:08
*** mrda_away is now known as mrda21:09
mrdaMorning all21:09
NobodyCamgood morning mrda21:09
mrdahi NobodyCam21:10
*** jrist has joined #openstack-ironic21:11
devanandaNobodyCam: you're right - missing one %s21:11
NobodyCam:)21:11
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Implement _update_neutron in PXE driver  https://review.openstack.org/7046821:12
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Fix log and test for NeutronAPI.update_port_dhcp_opts  https://review.openstack.org/7109421:12
NobodyCamthat looks better :-P21:12
NobodyCamwill +2 after mr J is done with them21:13
devanandaack, ty21:14
*** aignatov is now known as aignatov_21:15
devanandaNobodyCam: one more - global req update from jenkins :)https://review.openstack.org/#/c/68823/421:21
NobodyCamlooks like it bumps stevedore up to .14 will we need a patch to undo the work around we did21:24
devanandai have taht up already21:25
devanandathough it's not essential21:25
devanandathe work-around should have added support for both pre-.14 and >=0.14 stevedore21:25
NobodyCamI do recall that21:27
*** jcooley_ has quit IRC21:31
*** jdob has quit IRC21:53
openstackgerritA change was merged to openstack/ironic: Updated from global requirements  https://review.openstack.org/6882321:57
openstackgerritA change was merged to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/6802422:02
*** jcooley_ has joined #openstack-ironic22:10
openstackgerritDevananda van der Veen proposed a change to openstack/ironic: Implement a multiplexed VendorPassthru example  https://review.openstack.org/7086322:12
devanandanow with better examples!22:13
*** jcooley_ has quit IRC22:16
*** jcooley_ has joined #openstack-ironic22:17
max_lobur:)22:35
*** matty_dubs is now known as matty_dubs|gone22:36
*** tatyana has joined #openstack-ironic22:37
*** jbjohnso has quit IRC22:39
NobodyCamcongratz max_lobur :)22:41
max_loburNobodyCam: TY!22:45
NobodyCam:)22:46
* max_lobur woo-hoo22:46
NobodyCamyou and Roman deserve it22:47
*** mdurnosvistov_ has quit IRC22:48
comstudwin 5522:51
comstudwhoooops22:51
NobodyCam:-p22:51
devanandahey look - another vendor driver submission :)22:55
devanandahttps://blueprints.launchpad.net/ironic/+spec/ironic-ilo-power-driver22:55
devanandahttps://blueprints.launchpad.net/ironic/+spec/ironic-ilo-virtualmedia-driver22:55
*** harlowja is now known as harlowja_away23:01
NobodyCamWoo Hoo23:02
NobodyCam!!!!!23:02
openstackNobodyCam: Error: "!!!!" is not a valid command.23:02
max_loburnice23:09
max_loburit's going to be very popular23:09
NobodyCamyes WE are23:10
NobodyCamhehe23:10
*** max_lobur1 has joined #openstack-ironic23:11
*** tatyana has quit IRC23:13
*** harlowja_away is now known as harlowja23:13
devanandaok, i need to get some food and fresh air ... bbiaw23:14
*** max_lobur has quit IRC23:15
devanandamax_lobur_afk: oh hi! one of your patches need to be rebased, btw23:15
*** max_lobur1 has quit IRC23:15
*** max_lobur has joined #openstack-ironic23:15
NobodyCamenjoy food devananda :)23:15
*** max_lobur has quit IRC23:18
*** max_lobur has joined #openstack-ironic23:18
*** max_lobur1 has joined #openstack-ironic23:21
*** max_lobur has quit IRC23:23
NobodyCambrb quick walkies23:35
*** max_lobur1 has quit IRC23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!