Friday, 2014-08-15

*** mkerrin has quit IRC00:04
*** dlaube has quit IRC00:17
*** dlaube has joined #openstack-ironic00:21
*** penick has quit IRC00:23
*** mkerrin has joined #openstack-ironic00:44
*** dlaube has quit IRC00:54
*** killer_prince has quit IRC01:03
*** ellenh has quit IRC01:12
*** nosnos has joined #openstack-ironic01:15
*** killer_prince has joined #openstack-ironic01:15
*** killer_prince is now known as lazy_prince01:15
*** nosnos has quit IRC01:22
*** chenglch has joined #openstack-ironic01:22
*** nosnos has joined #openstack-ironic01:23
*** lazy_prince has quit IRC01:30
openstackgerritYuiko Takada proposed a change to openstack/ironic: Update API document with BootDevice  https://review.openstack.org/11441501:49
*** krtaylor has joined #openstack-ironic01:54
*** killer_prince has joined #openstack-ironic01:56
*** killer_prince is now known as lazy_prince01:56
*** Krast has joined #openstack-ironic02:33
*** rloo_afk has quit IRC02:39
*** annegentle_ has quit IRC03:02
*** Krast has quit IRC03:10
*** Krast has joined #openstack-ironic03:10
*** nosnos has quit IRC03:28
*** Haomeng has quit IRC03:38
*** Haomeng has joined #openstack-ironic03:38
*** nosnos has joined #openstack-ironic03:41
*** harlowja is now known as harlowja_away03:42
*** rushiagr_away is now known as rushiagr03:56
*** pcrews has quit IRC04:04
*** rushiagr is now known as rushiagr_away04:56
*** rushiagr_away is now known as rushiagr05:03
openstackgerritHaomeng,Wang proposed a change to openstack/ironic: WIP: Add send-data-to-ceilometer support for pxe_ipminative driver  https://review.openstack.org/11248605:05
chenglchHello, may i ask a question about ironic driver. If I setup the two nova compute node with ironic driver ? They get the node info from ironic client, ironic client get info from ironic database, so the compute node will return the same resource to nova conductor?05:16
*** Krast has quit IRC05:31
Haomengchenglch: sure, welcome05:34
Haomengchenglch: what is your concern?05:36
Haomengchenglch: sorry, can not get your question05:37
chenglchhello haomeng05:39
chenglchso only one compute node with ironic driver is needed , is that right?05:39
Haomengchenglch: Ironic support multiple compute nodes, they will call ironic client to connect single api and single database05:40
Haomengchenglch: and our ironic conductor support multi-work05:41
Haomengchenglch: can handle multi-request from multi-clients05:41
openstackgerritHaomeng,Wang proposed a change to openstack/ironic: WIP: Add send-data-to-ceilometer support for pxe_ipminative driver  https://review.openstack.org/11248605:43
chenglchwhen the node.set_provision_state is called, ironic api will let ironic-conductor to do the provision work, may i setup the ironic-conductor on multiple node?05:43
Haomengchenglch: no need, we have logic to balance the work for multiple conductors05:44
chenglchiornic conductor and neutron network node is on the same machine?05:45
Haomengchenglch: you can get concept via this document I think05:46
Haomengchenglch: http://docs.openstack.org/developer/ironic/deploy/user-guide.html05:46
Haomengchenglch: no, Ironic depends on neutron client directly05:46
Haomengchenglch: the neutron client will find the neutron url from keystone I think05:47
Haomengchenglch: so, ironic conductor and neutron can be different machine05:47
Haomengchenglch: this is following openstack deployment model, all service can be deployed defferent machine/nodes05:48
Haomengchenglch: and register themself into keystone with api url05:48
chenglchI meet a problem when I use the pxe_driver. pxe_driver will prepare the kernel and ramdisk in /tftpboot, and pxe_driver will update the dnsmasq opttion. This work is done by ironic-conductor05:49
Haomengchenglch: yes, what problem it is?05:49
Haomengchenglch: ironic conductor will call neutron client I think05:50
Haomengchenglch: let me find the code05:50
chenglchbut the the client machine is power on, dnsmasq will do the dhcp work, and tell the client baremetal machine the tftp-server05:50
chenglchbut the ip address of the tftp-server that  client machine get is always the neutron network node05:51
Haomengchenglch: one question, why do you setup two ironic compute nodes? I think one single ironic compute nodes is enough, because it is not *real* hypervisor, the real one is the physical machine05:54
chenglchI have check the dnsmasq on neutron network node, the option  server-ip-address, and tftp-server is set to the ip address I want, but client baremetal machine don't care that ip in  --dhcp-optsfile05:54
Haomengby default, the tftp server is from conductor node05:55
chenglchNow i only setup up one compute node with one ironic-conductor, but the ironic-conductor and neutron-netowrk node is not the same machine05:55
Haomenggot05:56
Haomengyou mean the dhcp's tftp address is not conductor's ip?05:57
chenglchyes05:57
Haomengchenglch: can you share the log which debug flag enabled?05:57
Haomengchenglch: let me check the log, maybe it is bug05:57
Haomengand let me go through the cod logic about how to set tftp's ip05:58
chenglchthe tftp-server address in dnsmasq is the conductor's ip05:58
Haomengchenglch: that should be correct05:58
Haomengchenglch: what is wrong?05:58
Haomengchenglch: because, by defaut, tftp server is in ironic conductor side05:59
chenglchbut when pxe started ,the client baremetal machine get from dnsmasq  is not the conductor's ip05:59
Haomengchenglch: you mean get ip is the one same with neutron server's ip?06:00
chenglchyes06:00
Haomengchenglch: pxe is trying to connect to neutron's ip to run tftp to download the kernel+ram disk?06:01
chenglchyes06:01
Haomengchenglch: ok, got, can you share the conductor's log to http://paste.openstack.org/?06:02
Haomengchenglch: let me trace back the log with code, maybe it is defect06:02
Haomengdid you set tftp_server option in ironic.conf?06:04
Haomengchenglch: in [pxe] section06:05
chenglchI set the tftp_server to the ironic conductor ip in [pxe]06:05
Haomengchenglch: should be sure tftp_server ip is correct, which ip is runing tftp service06:05
Haomengchenglch: ok06:05
chenglchDoes dnsmasq need some special config?06:06
Haomengchenglch: no, dnsmasq is controled by neutron06:08
Haomengand our ironic conductor will call neutron api to update dhcp port06:08
Haomengso it is better you can provide the conductor log file06:08
*** Krast has joined #openstack-ironic06:09
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/11413506:10
chenglchDo we need start tftp on ironic conductor manually ?06:14
Haomengchenglch: if you deploy ironic with devstack, tftp will be started by devstack stack scripts, else, have to make sure it is in service by manually06:15
Haomengchenglch: another question, do you have more dhcp server in your lan?06:15
Haomengchenglch: can you run tcpdump to make sure our dnsmasq receive the dhcp request from your physical pxe?06:16
chenglchNo , i can see the pxe from clinet get the ip from neutron network node06:16
Haomengwhat code version are you using? is it latest code? or icehouse release?06:17
chenglchI get the code from devstack, and  kill the conductor service on the network node, Then I start the ironic-conducor on another machie, the code version is the latest06:19
Haomengchenglch: ok, I got06:19
Haomengchenglch: the tftp ip is registered to neutron's dnsmasq from your firt conductor node06:19
chenglchYes,06:21
chenglchThe question i met from dnsmasq is like this one   http://ubuntuforums.org/showthread.php?t=160571506:22
chenglchSo i'm not sure is there any config needed in dnsmasq to support remote tftp06:23
*** pcrews has joined #openstack-ironic06:40
Haomengchenglch: no need to config dnsmasq, it is controlled by neutron06:40
Haomengyou make sure the ip in such file /opt/stack/data/neutron/dhcp/e04ad51a-e591-4164-b75f-8f7a3ab472ba/opts , it is conductor's ip? but pxe get neutron's ip?06:42
Haomengchenglch: can you share your dnsmasq option file content, such as - /opt/stack/data/neutron/dhcp/e04ad51a-e591-4164-b75f-8f7a3ab472ba/opts06:42
Haomengthis is my option file - http://paste.openstack.org/show/95401/06:43
*** jcoufal has joined #openstack-ironic06:44
mrdaHave a good weekend Ironic!06:46
*** zz_naotok is now known as naotok06:46
*** mrda is now known as mrda-weekend06:46
Haomengmrda: good night, and nice weekend:)06:46
*** dtantsur|afk is now known as dtantsur06:47
dtantsurMorning, Ironic06:47
Haomengdtantsur: morning:)06:47
dtantsurbrb07:24
*** yuriyz has quit IRC07:47
chenglchHello Haomeng , this is my option file  http://paste.openstack.org/show/95424/07:50
Haomengchenglch: server-ip-address - 10.1.0.72/10.1.0.24107:59
Haomengchenglch: I assume one is from your first conductor's ip, another one is 2nd conductor ip right?07:59
Haomengchenglch: so you manually kill first conductor which is same neutron node with conductor in a machine, right?08:00
chenglchyes , I kill the first conductor on 10.1.0.7208:02
Haomengchenglch: can you check the first conductor's ironic.conf tftp_server08:02
Haomengchenglch: I guess the first conductor 's tftp_server = 10.1.0.7208:03
chenglchI think i find the cause in my environment, when the ironic-conductor and neutron network node is on the same machine, I add " enable-tftp tftp-root=/opt/stack/data/ironic/tftpboot" in a config file I create called dnsmasq.conf, and I enable this file in the dhcp-agent.ini , now clinet pxe can get the tftp file08:03
Haomengchenglch: cool:)08:03
openstackgerritHaomeng,Wang proposed a change to openstack/ironic: WIP: Add send-data-to-ceilometer support for pxe_ipminative driver  https://review.openstack.org/11248608:04
chenglchbut without  enable-tftp in dnsmasq,conf  when the ironic-conductor and neutron network node on the same machine , the client pxe can't get the right file08:05
Haomengchenglch: I understand by default, devstack will enable tftp in dnsmasq right?08:08
Haomengchenglch: if it is not enabled, will not work, make sense08:08
chenglchso , if we want to use remote tftp, must disable tftp in dnsmasq?08:11
*** jistr has joined #openstack-ironic08:12
*** derekh has joined #openstack-ironic08:13
Haomengchenglch: I have devstack env, let me check the settings08:21
Haomengchenglch: checked, enable-tftp in dnsmasq,conf is false by default, means that we will not use dnsmasq built-in tftp-server08:22
*** lucasagomes has joined #openstack-ironic08:25
chenglchHaomeng , then you start tftp-server manually?08:26
Haomengchenglch: devstack will setup tftp server - https://github.com/openstack-dev/devstack/blob/a3ecab6618cb7c9ca8089d8b24ac503ed5a719e7/lib/ironic#L40708:26
Haomengchenglch: so no need to start by manually, if you did not run stack.sh to install openstack, have to setup tftp server by manually I think08:27
chenglchMaybe my environment is very strange, when the network node and ironic-conductor is on the same machine, i must enable-tftp in dnsmasq to make the pxe work correct. I will reinstall my environment next time08:33
chenglchThanks Haomeng08:33
Haomengchenglch: welcome:)08:34
Haomengchenglch: nice weekend:)08:34
*** ndipanov_gone is now known as ndipanov08:35
openstackgerritImre Farkas proposed a change to openstack/ironic: Add DracDriver and its DracPower module  https://review.openstack.org/10485008:47
openstackgerritHaomeng,Wang proposed a change to openstack/ironic: WIP: Add send-data-to-ceilometer support for pxe_ipminative driver  https://review.openstack.org/11248608:57
openstackgerritRakesh H S proposed a change to openstack/ironic: IPMI double bridging functionality  https://review.openstack.org/9577508:59
*** pelix has joined #openstack-ironic09:07
*** yuriyz has joined #openstack-ironic09:12
dtantsurFolks, have you seen these errors recently: http://logs.openstack.org/46/113346/3/check/check-tempest-dsvm-virtual-ironic/8c2f0d7/ ?09:14
dtantsurIs it something with the gate?09:15
*** chenglch has quit IRC09:15
*** dtantsur is now known as dtantsur|lunch09:24
*** Krast has quit IRC09:56
lucasagomesdtantsur|lunch, yup yeah i've seem that in some patches10:22
lucasagomeslemme take a look10:22
*** naotok is now known as zz_naotok10:32
*** Alexei_987 has joined #openstack-ironic10:51
Haomenglucasagomes: morning10:53
lucasagomesHaomeng, morning :)10:53
Haomenglucasagomes: do you know how to debug our unittest code, I have a unittest passed in my venv but failed in Jenkins10:54
Haomenglucasagomes: I try with import pdb;pdb.set_trace(), it does not work in unittest code10:55
lucasagomesHaomeng, yeah :/ I usually debug it primarily with strings heh I think that there should be a better way of doing that10:55
lucasagomesHaomeng, can I see the error on jenkins?10:55
Haomenglucasagomes: sure10:55
Haomenglucasagomes: this is my patch,10:56
Haomenglucasagomes: https://review.openstack.org/#/c/112486/10:56
Haomenglucasagomes: I worked on this unittest for some time, it is working as well in my venv, but not working in jekins10:57
* lucasagomes clicks10:57
Haomenglucasagomes: I think my code works, I test it outof our ironic code tree also10:58
Haomenglucasagomes: I will be away and back for a while10:58
Haomenglucasagomes: welcome your comments10:58
lucasagomesHaomeng, hmm just for reference in my local env it's failing like the gate11:01
lucasagomeshttp://paste.openstack.org/show/95485/11:01
Haomenglucasagomes: same with jenkins, ok let me recreate my env, thank you Lucas11:02
Haomenglucasagomes: :)11:04
Haomenglucasagomes: nice day:)11:04
lucasagomesHaomeng, I will see if I can spot something else11:04
lucasagomesbut yeah looks like it ur env11:04
lucasagomesHaomeng, you too buddy! have a nice day11:05
ifarkasHaomeng, you should be able to use pdb with unittests if you are not using test runner11:07
ifarkasHaomeng, you can do eg. python -m testtools.run ironic.tests.test_disk_partitioner.DiskPartitionerTestCase11:08
openstackgerritRakesh H S proposed a change to openstack/ironic: IPMI double bridging functionality  https://review.openstack.org/9577511:09
*** lucasagomes is now known as lucas-hungry11:18
*** rushiagr is now known as rushiagr_away11:30
*** rushiagr_away is now known as rushiagr11:35
*** dtantsur|lunch is now known as dtantsur12:03
*** rushiagr is now known as rushiagr_away12:08
*** nosnos has quit IRC12:37
*** rakesh_hs has joined #openstack-ironic12:44
*** lazy_prince has quit IRC12:49
*** lucas-hungry is now known as lucasagomes12:49
*** rakesh_hs_ has joined #openstack-ironic12:52
*** rakesh_hs has quit IRC12:52
*** rakesh_hs_ has quit IRC13:02
*** russellb is now known as rustlebee13:03
*** rushiagr_away has quit IRC13:09
*** lucasagomes_ has joined #openstack-ironic13:15
*** pcrews_ has joined #openstack-ironic13:15
*** ndipanov has quit IRC13:15
*** yuriyz has quit IRC13:20
*** lucasagomes has quit IRC13:20
*** pcrews has quit IRC13:20
*** rushiagr_away has joined #openstack-ironic13:27
*** yuriyz has joined #openstack-ironic13:37
*** lucasagomes_ is now known as lucasagomes13:38
*** yuriyz has left #openstack-ironic13:40
*** yuriyz has joined #openstack-ironic13:41
*** killer_prince has joined #openstack-ironic13:45
*** killer_prince is now known as lazy_prince13:46
lucasagomesgate seems to be broken due some tempest stuff :(13:51
* lucasagomes dig13:51
Shrewslucasagomes: example?13:57
lucasagomesShrews, morning http://logs.openstack.org/46/114046/4/gate/gate-tempest-dsvm-virtual-ironic/7faaf8f/console.html13:57
* Shrews helps lucasagomes dig13:57
lucasagomeshttps://bugs.launchpad.net/openstack-ci/+bug/131106613:58
lucasagomesmaybe... acording to the elastic check...13:58
lucasagomesaccording*13:58
lucasagomesmaybe that's it cause I can't find a proper error, I looked into what was merged in tempest/ironic none seems to be causing that13:59
Shrewsyeah, i see the 15m warning in that console log14:01
*** jgrimm has joined #openstack-ironic14:01
*** jistr is now known as jistr|afk14:04
*** lazy_prince has quit IRC14:12
NobodyCamgood morning Ironic, TGIF :)14:14
Shrewsmorning NobodyCam14:14
*** jistr|afk is now known as jistr14:15
NobodyCammorning Shrews14:15
NobodyCam:)14:15
* Shrews needs to reboot. brb14:16
Shrewsoh lovely. after updating my system, FF no longer wants to work *at all*14:20
* Shrews thinks he'll start his vacation a day early14:20
*** killer_prince has joined #openstack-ironic14:21
*** killer_prince is now known as lazy_prince14:21
NobodyCam:-p14:21
*** mitz has quit IRC14:27
lucasagomesmorning NobodyCam14:35
NobodyCammorning lucasagomes :)14:39
openstackgerritDmitry Tantsur proposed a change to openstack/python-ironicclient: Add sort_key and sort_dir parameters to *-list  https://review.openstack.org/11455514:40
*** pelix has quit IRC14:41
dtantsurNobodyCam, morning14:45
dtantsurand TGIF :)14:45
jrollmorning ironic :)14:48
Shrewsjroll: morning. how goes the begging?14:48
Shrews:)14:48
jrollhehehe14:50
jrollI got 2/3 approved14:50
jrollone landed, one had conflicts14:50
jrollhad to rebase, should go through when people are around :)14:50
jrollwhat happened was, I put up a new patch, saw ianw -1 it, and jumped on him :P14:51
dtantsurjroll, morning14:52
jrollmorning dtantsur :)14:52
NobodyCammorning dtantsur and jroll :)14:53
*** ellenh has joined #openstack-ironic14:59
jrollheya NobodyCam :)14:59
Shrewsanyone here use gerrymander for reviewing?15:09
jrollShrews: huh, haven't seen that before15:11
jrollI've been using gertty lately, it's nice15:11
*** ellenh has quit IRC15:13
*** Haomeng|2 has joined #openstack-ironic15:17
*** Haomeng has quit IRC15:18
NobodyCambrb15:27
*** ifarkas has quit IRC15:30
dtantsurCalling it a day, have a nice weekend15:31
jrollyou too, dtantsur :)15:32
dtantsurlucasagomes, note that I'm not PTO on Mon, so no bug team report or just use the dashboard to get some stats :)15:32
*** dtantsur is now known as dtantsur|afk15:32
lucasagomesdtantsur|afk, ack, cheers for the heads up15:33
lucasagomesdtantsur|afk, have a great night, enjoy the weekend!15:33
lucasagomesjroll, morning15:33
jrollmorning lucasagomes :)15:33
ShrewsALL: Interesting python oddity that QA guys just pointed out to me: https://review.openstack.org/10013215:46
jroll<3 python15:49
JoshNangShrews: ahh yeah, i've had that bite me before. i'm pretty sure all the oslo loopingcalls use that to keep state between loops15:49
*** HenryG has joined #openstack-ironic15:54
ShrewsJoshNang: well it's a good thing the loopingcalls aren't used all over the place  </end sarcasm>15:55
JoshNangheh15:57
*** dlaube has joined #openstack-ironic15:59
jrollShrews: ever seen this? fresh clone of testr http://paste.openstack.org/show/95620/16:22
*** comstud is now known as bearhands16:26
*** derekh has quit IRC16:26
*** chuckC has quit IRC16:27
*** ellenh has joined #openstack-ironic16:34
openstackgerritAngus Thomas proposed a change to openstack/ironic: Removes get_port_by_vif  https://review.openstack.org/11459516:37
openstackgerritAngus Thomas proposed a change to openstack/ironic: Removes get_port_by_vif.  https://review.openstack.org/11459516:39
*** Hefeweizen has quit IRC16:40
openstackgerritAngus Thomas proposed a change to openstack/ironic: Removes get_port_by_vif.  https://review.openstack.org/11459516:40
openstackgerritAngus Thomas proposed a change to openstack/ironic: Removes get_port_by_vif.  https://review.openstack.org/11459516:42
NobodyCamdtantsur|afk: please ping /me when your back16:43
openstackgerritAngus Thomas proposed a change to openstack/ironic: Removes get_port_by_vif  https://review.openstack.org/11459516:44
openstackgerritAngus Thomas proposed a change to openstack/ironic: Removes get_port_by_vif  https://review.openstack.org/11459516:45
lucasagomeswoah16:45
NobodyCamhey lucasagomes got a question 'bout tests16:50
NobodyCamhappen to have a free second16:50
lucasagomesNobodyCam, hey yes16:50
lucasagomessure16:50
NobodyCam:)16:50
NobodyCamre: dtantsur's comment on https://review.openstack.org/#/c/114281/8/ironic/tests/conductor/test_manager.py16:51
NobodyCamI have attempted to use the decorator but something is not working when I do that?16:53
lucasagomesNobodyCam, hmm lemme see16:55
NobodyCamnode.reservation is not cleared and spawn is called three times16:55
NobodyCam:)16:55
lucasagomesNobodyCam, oh wondering if it was because if you mock at the method level16:57
lucasagomesthis line16:57
lucasagomesthread = self.service._spawn_worker(lambda: None)16:57
lucasagomeswill actually return the mock16:57
lucasagomescause u just mocked _spawn_worker16:57
lucasagomesbut seems unreleated to the reservation thing16:58
NobodyCamthats what thru me off17:00
Shrewsjroll: umm, nope17:00
Shrewsjroll: were you making changes to one of the tests when that happened?17:01
jrolloh wait17:01
jrollit's because I don't have a config file :/17:01
*** openstackgerrit has quit IRC17:02
Shrewsjroll: i have seen tempest vomit similarly one stupid things, like a misspelled import or such17:02
Shrewsnot helpful17:02
jrollright17:03
jrolltestr is awesome like that17:03
Shrewsreally17:03
*** openstackgerrit has joined #openstack-ironic17:03
*** harlowja_away is now known as harlowja17:03
lucasagomesNobodyCam, wondering if when you had that thread = service._spawn_worker before17:04
lucasagomesoutside the mock17:04
lucasagomeswas calling release_resources (which will call release_node and clean up the reservation)17:04
lucasagomeswhere when you mock at the method level, you will invoke do_node_deploy which will call reserve_node and set the hostname17:05
lucasagomesbut release_node will never be called since it's runs on the __exit__() at task_manager17:05
lucasagomescorreting... release_node will never be called because spawn_worker is now mocked so the __exit__() method will never run (which is where release_node is invoked in the task_manager.py)17:06
*** chuckC has joined #openstack-ironic17:14
lucasagomesNobodyCam, what if instead of mocking _spawn_worker you mock the deploy() method?17:14
lucasagomesNobodyCam, http://paste.openstack.org/show/95650/17:16
NobodyCamahh, I see ... let me try that... Ty lucasagomes :)17:17
lucasagomesNobodyCam, it sounds even better, cause by mocking _spawn_worker you would mock the _do_node_deploy() complete... where mocking only the driver.deploy() you just mock the driver specific call17:18
lucasagomesand run the manager code stuff17:18
NobodyCamya17:18
NobodyCamthat was it TY lucasagomes :)17:22
lucasagomesNobodyCam, np yw!17:22
*** shakamunyi has joined #openstack-ironic17:24
Shrewslucasagomes: grrr... hitting that same bug you encountered earlier17:24
lucasagomesShrews, :( which one?17:25
Shrewshttps://bugs.launchpad.net/openstack-ci/+bug/131106617:25
lucasagomesShrews, >.<17:25
lucasagomesShrews, well at least it's a resource problem not a bug in the code17:25
Shrewstop bug in the recheck list17:25
openstackgerritChris Krelle proposed a change to openstack/ironic: Allow rebuild of node in ERROR and DEPLOYFAIL state  https://review.openstack.org/11428117:26
lucasagomesyeah... it's causing pretty much all patches to fail in the gate17:26
* lucasagomes clicks on this bug affects me17:29
lucasagomes:)17:29
lucasagomesaight, I will call it a day17:30
lucasagomesit's late and friday :)17:30
lucasagomeshave a good night everyone! enjoy the weekend17:30
NobodyCamhave a great weekend lucasagomes17:30
NobodyCam:)17:30
Shrewslater lucasagomes17:30
NobodyCamand Ty again17:31
lucasagomesnp :)17:31
lucasagomeslaterz17:31
*** lucasagomes is now known as lucas-dinner17:31
JoshNang~.~.~~.17:32
JoshNang...whoops. frozen screen :/17:33
NobodyCam:-p17:33
jrollheyyyyy so.17:36
jrollI think our gate might be busted17:36
jrollhttp://logs.openstack.org/75/95775/15/check/check-tempest-dsvm-virtual-ironic/e15e2fa/logs/devstacklog.txt.gz#_2014-08-15_14_31_20_38617:36
jrollI'm having this problem with a fresh install of devstack as well17:36
Shrewshrm, so... i was thinking that was due to 1311066, but maybe there is something else? seems to only be affecting the ironic jobs17:39
Shrewsugh17:39
jrollShrews: yeah, see that log there... seems like maybe we need to pass container format or something17:39
jrollI don't see evidence of something related to that changing, but who knows17:39
jrolloh, we use glanceclient master17:40
* jroll looks17:40
*** shakamunyi has quit IRC17:43
openstackgerritChris Krelle proposed a change to openstack/ironic: Allow rebuild of node in ERROR and DEPLOYFAIL state  https://review.openstack.org/11428117:43
*** Alexei_987 has quit IRC17:45
jrollShrews: I'm going to try this again with --container-format17:46
*** krtaylor is now known as krtaylor_away17:46
jrollstill curious what broke this17:47
Shrewsjroll: is this due to a recent change in glanceclient?17:47
Shrewsthat would suck17:47
* Shrews checks logs17:48
jrollit would17:49
jrollI poked around but couldn't find anything obvious17:49
jrolland.. glanceclient gets "schemas" from glance api17:49
jrollpresumably defaults are there17:49
*** penick has joined #openstack-ironic17:52
*** jistr is now known as jistr|afk17:52
Shrewsf***17:53
Shrewsjroll: https://review.openstack.org/11153817:53
Shrewsnotice, once again, the ironic job failed, but it was +A'd17:53
NobodyCamnice17:54
NobodyCam:(17:54
jrolloh17:54
jrollffs17:54
jrollShrews: want me to do the bug filing and kicking and screaming, or you got it?17:55
Shrewsjroll: trying to force feed a pb&banana down my mouth hole while debugging... can you kick it off?17:56
jrollShrews: I believe I have the bugfix right now17:56
jrollsure17:56
Shrewscan we have a voting job now????? plz???17:56
jrollsrsly17:57
jrollI bet this broke nova-bm, too :P17:57
*** shakamunyi has joined #openstack-ironic17:57
jrollugh, it's really hard to not be an asshole right now18:02
* Shrews updates whiteboard18:03
*** ellenh_ has joined #openstack-ironic18:04
adam_g:(18:04
jrollhttps://review.openstack.org/11461918:06
*** ellenh has quit IRC18:06
Shrewsawesome18:06
jrollhow the $%^&* do you land a change to lib/ironic without ironic's job passing18:06
Shrewsmaybe he was in the middle of debugging the change when it was approved?  /me says sheepishly18:07
jrolllol18:07
jrollI mean, as a core18:07
jrollI'm going to ping them all once jenkins runs18:08
Shrewsi think the meat of the problem is that there are so many transient gate bugs, that unless a job is voting, it's very easy to assume that a non-voting job failed b/c of one of the other bugs18:11
Shrewstl;dr -- unless someone is forced to look at it, it won't be looked at18:12
jrollI think the problem is people assume18:13
jrolllike, I'm not mad if you change lib/nova and it breaks ironic18:13
jrollbut the change was to lib/ironic18:13
jrolland ironic failed18:13
jrollyou can't ignore that18:14
*** ellenh_ has quit IRC18:18
NobodyCamwell can't and shouldn't are two diffenent things18:24
jroll>:(18:24
adam_gwe should probably be taking a bit of responsibility for catching those errors in the devstack review queue. i try to add a -1 to reviews for tempest/devstack if its consistently failing the ironic job18:27
adam_ghmm18:27
Shrewsoh, adam_g, that reminds me. once https://review.openstack.org/112134 lands, we need to put up another patch to disable cinder with ironic again18:28
adam_gi can modify http://no-carrier.net/~adam/openstack/ironic_gate_status.html to show the project for each review, that may help us catch these earlier and18:28
adam_gShrews, that would need to go to devstack-gate or config?18:28
Shrewsadam_g: it would need to be this: https://review.openstack.org/105599 (which was reverted)18:29
Shrewsdevstack-gate18:29
adam_gShrews, ohh right18:29
Shrewswe need those jobs converged first18:30
adam_gShrews, that'll be a fun one to coordinate18:30
*** ellenh_ has joined #openstack-ironic18:39
*** ellenh_ is now known as ellenh18:40
Shrewswoohoo. finally got a +A on the tempest rebuild test.18:48
NobodyCam:)18:48
*** jistr|afk has quit IRC18:50
ellenhsilly question: does ironic use openstack/stevedore or dreamhost/stevedore?18:53
NobodyCami would think openstack18:54
NobodyCam:-p18:54
kylestevI would hope openstack18:54
kylestevthat would be confusing18:54
ellenhi'm guessing openstack/stevedore, but the behavior isn't quite matching up with what the docs say...18:54
ellenhor maybe i just misspelled things :P18:54
NobodyCamhttps://github.com/openstack/ironic/blob/master/requirements.txt#L2118:55
NobodyCamshould be from here: https://pypi.python.org/pypi/stevedore/1.0.0.0a118:56
NobodyCamsource is listed as http://git.openstack.org/cgit/openstack/stevedore while the home page is listed as https://github.com/dreamhost/stevedore18:57
NobodyCamso your guess is as good as mine :-p18:57
ellenhthat's kinda awkward...18:58
ellenhhrm. thanks, i'll poke around some more :)18:58
NobodyCammaybe ask dhellmann?18:59
Shrewsopenstack took over hosting stevedore, iirc18:59
Shrewssame code base, but doubtful the github.com/dreamhost/stevedore code is updated19:00
ellenhyeah, dreamhost's hasn't been updated since 201319:00
Shrewspypi should be published from the openstack repo19:00
ellenhi'm trying to use an argument that exists in github/openstack/stevedore but not github/dreamhost/stevedore, and it's not doing anything19:01
ellenhso that's why i was suspicious19:01
ellenhhttps://github.com/openstack/stevedore/blob/master/stevedore/driver.py#L22 vs https://github.com/dreamhost/stevedore/blob/master/stevedore/driver.py#L2219:02
ellenhi have a on_load_failure_callback argument, but i'm seeing runtime exceptions instead of the exception i want, so...19:03
Shrewsellenh: what's the exception you're seeing?19:05
*** penick has quit IRC19:06
openstackgerritKyle Stevenson proposed a change to openstack/ironic-specs: Add spec for pluggable distributed synchronization  https://review.openstack.org/11463219:07
ellenhsame one as http://paste.openstack.org/show/95453/19:08
Shrewsellenh: is 'blah' defined in setup.cfg?19:09
ellenhno, that's the error i want to catch19:09
Shrewsah, i see19:09
Shrewsis the code up for review somewhere?19:10
ellenhyeah, let me just push up a WIP19:11
openstackgerritEllen Hui proposed a change to openstack/ironic: Make DHCP provider pluggable  https://review.openstack.org/11235119:13
ellenhShrews: ^ :)19:13
jrollkylestev: you should poke ironic folks to see what they think19:16
Shrewsellenh: so, i *think* that callback is called only if the defined entrypoint file is not found (i.e., it must still be defined in setup.cfg)19:20
Shrewsso, that's a different exception19:20
ellenhhmm.  yeah, i'm now 99% sure that it's not a problem with stevedore, since the driver_factory uses on_load_failure_callback19:21
jrollI think it's if an exceptino is raised when instantiating the class19:21
Shrewsjroll: right19:21
* jroll goes afk for a while19:22
ellenhyeah, you're right. https://github.com/openstack/stevedore/blob/master/stevedore/extension.py#L16919:23
kylestevHey devananda and NobodyCam, want to take a look at https://review.openstack.org/114632 when you get a chance?19:24
NobodyCamkylestev: ironic is in feature proposal freeze19:25
NobodyCamthat will need to go into K cycle19:25
NobodyCamahh just saw your comment19:25
kylestevNobodyCam: right, I put it in kilo and set the workflow to -119:26
kylestev:P19:26
*** jcoufal has quit IRC19:34
*** Hefeweizen has joined #openstack-ironic19:34
ellenhalright, things make sense now.  Thanks Shrews :)19:36
Shrewsnp19:36
*** athomas has quit IRC19:37
*** Mikhail_D_ltp has joined #openstack-ironic19:51
*** aignatov has quit IRC19:51
*** lucas-dinner has quit IRC19:52
*** tatyana has joined #openstack-ironic19:53
*** aignatov has joined #openstack-ironic19:57
NobodyCamone +2 on 11461920:01
*** aignatov has quit IRC20:07
*** Mikhail_D_ltp has left #openstack-ironic20:13
Shrewsno vote from the cantankerous Mr. J yet, though20:15
*** aignatov has joined #openstack-ironic20:16
Shrewsthe ironic jobs are all SUCCESS though20:16
NobodyCamjust three more jobs to go20:17
Shrewshrm, i'm guessing jeblair is going to be the only core around to approve that for us20:24
* Shrews pokes around for a core...20:26
NobodyCamty Shrews :)20:26
* NobodyCam needs mo coffee brb20:28
*** ellenh has quit IRC20:30
*** tatyana has quit IRC20:30
*** igordcard has joined #openstack-ironic20:31
NobodyCamwoo hoo20:35
NobodyCam114619 on its way down for landing20:35
NobodyCamTY Shrews20:35
Shrewsjust glad it didn't take us 3 days  :)20:37
*** igordcard has quit IRC20:45
*** igordcard has joined #openstack-ironic20:47
NobodyCamlol it still has to get past the gate :-p20:48
*** ellenh has joined #openstack-ironic20:49
openstackgerritEllen Hui proposed a change to openstack/ironic: Make DHCP provider pluggable  https://review.openstack.org/11235120:51
Shrewscalling it a day here, folks. see you all in a week20:51
NobodyCamhave a good one Shrews20:51
*** rwsu has quit IRC20:51
*** aignatov has quit IRC20:52
NobodyCam:)20:52
*** aignatov has joined #openstack-ironic20:54
NobodyCamkylestev: do you have another locking schema in mind?21:16
kylestevNobodyCam: yes. ZooKeeper.21:17
kylestevimplementing locking with ephemeral znodes is quite trivial and will mitigate deadlocks caused by the db-backed locking when a conductor dies21:19
NobodyCamI would like to see that as part of the spec, more so I can under stand the why do this question I generate while reading that spec21:24
kylestevjroll: ^ thoughts?21:25
chuckChi all: I'm working on a bug in neutron to allow neutron's port mac_address to be updatable so nic failure on an ironic server can be more easily handled21:29
chuckCwhat I need to know is how to simulate a nic failure on an ironic server that's really a VM21:30
chuckCthoughts?21:31
*** todd_dsm has joined #openstack-ironic21:36
adam_gchuckC, what do you mean by nic failure?21:38
NobodyCamifdown the nic21:38
NobodyCam?21:38
openstackgerritEllen Hui proposed a change to openstack/ironic: Make DHCP provider pluggable  https://review.openstack.org/11235121:39
chuckCIt would need to be a way to effect the simulated hardware, not something inside the server OS21:40
*** todd_dsm has quit IRC21:40
chuckCWhat I'm struggling with is if nic hardware fails on an ironic server, will neutron notice in any way?21:42
chuckCSo I wanted to try it, and of course I don't have baremetal hardware, just VMs21:44
NobodyCamchuckC: would hacking the xml work ? you could add <link state='down'/> for the nic21:46
chuckCNobodyCam: Thanks for the idea, I'll try it.21:47
*** coolsvap has quit IRC21:50
NobodyCamquick walkies ...brb21:58
*** coolsvap has joined #openstack-ironic22:04
NobodyCamugh 114619 fail :(22:19
*** igordcard has quit IRC22:25
*** shakamunyi has quit IRC22:30
*** igordcard has joined #openstack-ironic22:37
*** chuckC has quit IRC22:37
openstackgerritKyle Stevenson proposed a change to openstack/ironic-specs: Add spec for pluggable distributed synchronization  https://review.openstack.org/11463222:47
kylestevNobodyCam: https://review.openstack.org/#/c/114632/1..2/specs/kilo/pluggable-distributed-synchronization.rst - Let me know what you think. If you’d prefer not to deal with this on a Friday afternoon, I can remind you on Monday :)22:52
*** rwsu has joined #openstack-ironic23:06
NobodyCam:) please do... :)23:07
*** chuckC has joined #openstack-ironic23:08
kylestev:)23:09
openstackgerritEllen Hui proposed a change to openstack/ironic: Make DHCP provider pluggable  https://review.openstack.org/11235123:23
NobodyCamugggh23:34
*** rwsu has quit IRC23:35
*** ellenh has quit IRC23:39
NobodyCamreverify started... will check back later ... :)23:49
*** shakamunyi has joined #openstack-ironic23:51
*** igordcard has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!