Thursday, 2014-04-03

*** eguz has quit IRC00:02
openstackgerritJosh Gachnang proposed a change to openstack/ironic-python-agent: Adding oslo testing to improve debugging  https://review.openstack.org/8421200:03
*** blamar has quit IRC00:06
*** blamar has joined #openstack-ironic00:09
*** blamar has quit IRC00:26
*** matsuhashi has joined #openstack-ironic00:28
*** blamar has joined #openstack-ironic00:32
openstackgerritJosh Gachnang proposed a change to openstack/ironic-python-agent: Adding oslotest to improve debugging  https://review.openstack.org/8421200:33
*** Haomeng|2 has quit IRC00:36
openstackgerritJosh Gachnang proposed a change to openstack/ironic-python-agent: Adding oslotest to improve debugging  https://review.openstack.org/8421200:41
*** harlowja is now known as harlowja_away01:01
*** matsuhashi has quit IRC01:05
*** matsuhashi has joined #openstack-ironic01:06
*** linggao has quit IRC01:16
*** nosnos has joined #openstack-ironic01:29
*** ilives has joined #openstack-ironic01:41
*** ilives has quit IRC01:48
*** ilives has joined #openstack-ironic01:49
*** rloo has quit IRC02:45
*** matsuhashi has quit IRC03:05
lifelessdammit - devananda https://jenkins04.openstack.org/job/check-tripleo-ironic-undercloud-precise/152/console - thats my 30s delay patch03:07
lifelessdevananda: haven't dug in yet03:07
lifelesshttp://logs.openstack.org/61/84361/1/check-tripleo/check-tripleo-ironic-undercloud-precise/81791d5/03:08
NobodyCamlifeless: power state is NOSTATE is ssh working?03:12
NobodyCamperedioc task should set that to off03:13
russell_hlifeless: is there a long term fix for this?03:15
lifelessrussell_h: for what ?03:19
lifelessNobodyCam: the seed booted, so we know the creds in teh testenv json are valid03:20
NobodyCamlifeless: I see this in the conductor log. http://paste.openstack.org/show/qpJlh1VMtwwiOv2RKlGu/03:21
*** nosnos has quit IRC03:23
*** rameshg87 has joined #openstack-ironic03:29
*** coolsvap has joined #openstack-ironic03:32
russell_hlifeless: for this neutron issue03:33
lifelessNobodyCam: have a look in syslog, see if the node DHCPd03:33
russell_hlifeless: comments like03:35
russell_h"Horrible hack to mitigate bug 1300589" deserve some sort of long term plan IMO03:35
NobodyCamApr  3 02:14:52 ubuntu dnsmasq-dhcp[5126]: DHCPDISCOVER(tap1e536302-66) 52:54:00:ab:d5:3e03:35
NobodyCamApr  3 02:14:52 ubuntu dnsmasq-dhcp[5126]: DHCPOFFER(tap1e536302-66) 192.0.2.2 52:54:00:ab:d5:3e03:35
lifelessrussell_h: oh thats easy03:36
russell_hI was given to understand that Nova has a similar bug and they're adding some sort of callback thing03:36
lifelessrussell_h: generalise neutron's callback-to-nova to use a webhook or arbitrary url instead, so that you can say 'tell ironic at XXX when this complete'03:36
russell_hgotcha03:36
lifelessrussell_h: they added it but its not generic.03:36
russell_hlifeless: is there an issue open to genericize it?03:37
russell_hbasically, I'd like to see something like "TODO: refactor to block for neutron callback once bug xxxxx is resolved"03:38
lifelessrussell_h: I think that should be a joint ironic / neutron blueprint.03:40
lifelessNobodyCam: that looks like it DHCPd03:51
lifelessNobodyCam: so either the ramdisk failed03:51
NobodyCamya03:51
lifelessNobodyCam: or my fix wasn't aggressive enough03:51
NobodyCamand I would expect to see https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/ssh.py#L140-L14203:51
lifelessNobodyCam: can you check the timestamps03:51
NobodyCamif ssh failed03:51
lifelesswell it dhcpd so that means ssh power on worked03:52
NobodyCamyep...03:52
lifelessif we're not logging enough, perhaps ironics log default should be different :)03:52
NobodyCamwe may want to bump up a couple of levels until things are working smoothly :)03:53
lifelessNobodyCam: well, think of it as a production exercise03:53
lifelessNobodyCam: this is exactly what debugging production problems is like; so if we're not logging enough, operators will find it problematic too.03:54
*** eghobo has joined #openstack-ironic03:54
lifelessby the time you realise you need logs, they've already been written and its too late03:54
NobodyCamlifeless: yes!!!03:54
lifelessso - I'm fairly sure tripleo is running ironics default03:54
lifelessperhaps ironic should change its default :)03:54
russell_hI took that to the extreme once03:54
russell_hand logged actually everything03:54
lifelessrussell_h: so we did that with oopstools03:55
lifelessrussell_h: we logged *everything* for each request, into a bson struct03:55
lifelessrussell_h: and if the request failed, put that in a DB03:55
lifelessrussell_h: if the request worked, we threw it away and just logged the top level HTTPD style line03:55
lifeless[by put ina  DB, I mean tossed it at rabbit which tossed it at the DB]03:55
russell_hright03:55
russell_hthats basically what we were doing, but without the clever throwing away bit03:56
russell_hjson log messages03:56
NobodyCamahh you didn't pay for splunk03:56
russell_hhundreds per request, even for simple GETs03:56
lifelesswe used bson because we handle binary data in launchpad03:56
lifelessand we wrote a display console to show the timelines03:56
lifelessdebugging slow SQL etc was ---trivial--- with this03:57
russell_hNobodyCam: we could have bought a lot of splunk for what we spent on our graylog cluster03:57
NobodyCam:)03:57
russell_hactually probably not...03:57
russell_hyou still have to buy your own gear for splunk right?03:57
russell_hat least the license I used once was that way03:57
NobodyCamyou did when I last used them03:57
NobodyCamby the GB loged03:58
russell_hwe were doing gigabits of logs03:58
russell_hat pretty trivial load03:58
russell_hthen the ops guys made us turn down the logging :(03:58
NobodyCamlol03:58
russell_hthey were spending more time scaling graylog than the actual software03:58
russell_hbut debugging was glorious03:59
lifelessso the throwaway strat is important03:59
russell_hyeah, I like that03:59
lifelesse.g. AIUI google throwaway a large % of log messages03:59
lifelessjust randomly03:59
lifelessthe know enough will get through to show errors up03:59
*** harlowja_away is now known as harlowja04:00
*** nosnos has joined #openstack-ironic04:09
*** matsuhashi has joined #openstack-ironic04:15
NobodyCamlifeless: ssh.py 5 Loging lines 4 debug and one error. I'd like to add a few more at other levels, but it is late for me here04:18
NobodyCamI can do it first thing in the AM04:19
lifelessNobodyCam: so, I suggest taking over my patch04:25
lifelessand stabbing at it more and more until you can see a) whats wrong and b) fix it04:25
lifelessNobodyCam: then, when it passes, we can split that work out into small individual patches in a stack04:25
*** BadCub01 has quit IRC04:28
*** killer_prince has joined #openstack-ironic04:39
*** matsuhashi has quit IRC04:40
*** matsuhashi has joined #openstack-ironic04:46
*** eghobo has quit IRC05:20
*** pradipta_away is now known as pradipta05:27
*** Mikhail_D_ltp has joined #openstack-ironic05:27
*** subah has joined #openstack-ironic05:29
*** harlowja is now known as harlowja_away05:32
*** radsy has quit IRC05:38
lifelessNobodyCam: so05:49
lifelessR(tap1e536302-66) 52:54:00:ab:d5:3e05:49
lifelessApr  3 02:14:51 ubuntu dnsmasq-dhcp[5126]: DHCPOFFER(tap1e536302-66) 192.0.2.2 52:54:00:ab:d5:3e05:49
lifelessApr  3 02:14:52 ubuntu dnsmasq-dhcp[5126]: DHCPDISCOVER(tap1e536302-66) 52:54:00:ab:d5:3e05:49
lifelessApr  3 02:14:52 ubuntu dnsmasq-dhcp[5126]: DHCPOFFER(tap1e536302-66) 192.0.2.2 52:54:00:ab:d5:3e05:49
lifelessApr  3 02:14:54 ubuntu dnsmasq-dhcp[5126]: DHCPREQUEST(tap1e536302-66) 192.0.2.2 52:54:00:ab:d5:3e05:49
lifelessbah05:49
lifelessApr  3 02:14:43 ubuntu dnsmasq-dhcp[5126]: read /var/run/neutron/dhcp/f0317e27-e6e8-4869-bff1-6b8e0ea93572/opts05:49
lifelessthen the DHCP05:49
lifelessso that looks like the patch worked - just, only 8 seconds leeway05:50
lifelessthe cleanup failure is interesting and needs fixing :)05:51
lifelessNobodyCam: what abbout this?05:51
lifeless(wsme.api): 2014-04-03 02:11:05,105 WARNING Client-side error: Couldn't apply patch '[{'path': '/extra/vif_port_id', 'op': 'remove'}]'. Reason: u'vif_port_id'05:51
lifelessfrom API log ?05:52
lifelessI can't see the vendor callback in the api log05:52
lifelessmaybe we should be getting the console out of the VM ?05:53
openstackgerritAdam Gandelman proposed a change to openstack/ironic: Add tox target + script for nova driver tests  https://review.openstack.org/8403305:55
*** Mikhail_D_ltp has quit IRC06:02
openstackgerritJenkins proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/8395606:07
*** ndipanov has joined #openstack-ironic06:32
*** ndipanov has quit IRC06:32
*** matsuhashi has quit IRC06:36
*** matsuhashi has joined #openstack-ironic06:39
*** ndipanov has joined #openstack-ironic06:43
*** romcheg has joined #openstack-ironic06:52
*** romcheg has quit IRC06:54
*** mrda is now known as mrda_away06:57
*** Haomeng|2 has joined #openstack-ironic06:59
*** zigo has quit IRC07:04
*** zigo has joined #openstack-ironic07:08
*** mikal has quit IRC07:10
*** mikal has joined #openstack-ironic07:10
*** dshulyak has joined #openstack-ironic07:11
*** dshulyak has quit IRC07:13
*** ifarkas has joined #openstack-ironic07:22
*** romcheg has joined #openstack-ironic07:29
*** martyntaylor has quit IRC07:43
*** romcheg has left #openstack-ironic07:45
*** max_lobur has joined #openstack-ironic07:49
rameshg87Hi07:52
rameshg87i had a question related to diskimage-builder07:52
rameshg87anyone up there ? :-)07:52
*** martyntaylor has joined #openstack-ironic07:52
Haomeng|2rameshg87: welcome:)07:57
rameshg87hello Haomeng|2:07:57
rameshg87question i have is this ..07:57
Haomeng|2rameshg87: sure07:58
rameshg87does diskimage-builder cache the apt packages ?07:58
Haomeng|2rameshg87: not sure, do you have performance issue?07:58
rameshg87because everytime i try to create an image (fs image or ramdisk), it downloads from the internet07:58
rameshg87does it happen to you as well ?07:58
rameshg87it takes a lot of time :-)07:58
Haomeng|2rameshg87: depends on your apt source I think07:59
Haomeng|2Haomeng|2: yes07:59
Haomeng|2rameshg87: we can raise the question in tripleo irc also I think:)07:59
rameshg87it caches the ubuntu base image and checks the ubuntu cloud server whether it has changed, and uses the cached copy if it hasn't changed07:59
rameshg87Haomeng|2: oh, is there a separate irc for tripleo ?07:59
rameshg87i didn't know that :-)08:00
Haomeng|2rameshg87: yes08:00
Haomeng|2https://wiki.openstack.org/wiki/TripleO08:00
Haomeng|2diskimage-builder is in TripleO project08:00
*** jistr has joined #openstack-ironic08:01
rameshg87Haomeng|2: is it #triple ?08:01
Haomeng|2rameshg87: #tripleo08:01
Haomeng|2rameshg87: https://wiki.openstack.org/wiki/IRC08:02
Haomeng|2it is special, no openstack words:)08:02
rameshg87Haomeng|2: ah okay..finally got it :-)08:02
Haomeng|2rameshg87: :)08:02
rameshg87thanks08:02
Haomeng|2rameshg87: maybe it is in /var/cache/apt/archieve08:03
Haomeng|2can you check08:03
Haomeng|2it should be apt-get problem:)08:03
rameshg87Haomeng|2: but i think the apt-get installs are run within the chroot environment08:03
Haomeng|2rameshg87: maybe08:04
Haomeng|2rameshg87: check with default apt-get cache dir /var/cache/apt/archives/08:04
dtantsurMorning Ironic08:04
Haomeng|2dtantsur: morning08:04
*** derekh has joined #openstack-ironic08:05
Haomeng|2rameshg87: we can find a lot of deb files08:05
rameshg87Haomeng|2: yeah, but i am not sure if it is using them08:05
Haomeng|2rameshg87: default behavior should be:)08:05
Haomeng|2rameshg87: we can double check with tripleo guys:)08:06
Mikhail_D_wkMorning folks! :)08:06
rameshg87Haomeng|2: for my laptop it works fine.  once i install a package on my laptop, it doesn't download it again if i uninstall and reinstall.  so apt for my laptop i think is fine08:06
Haomeng|2rameshg87: :)08:06
rameshg87Haomeng|2: but since diskimage-builder does it within the chroot environment, it is not using my laptops /var/cache/apt/archives i think08:07
Haomeng|2Mikhail_D_wk: morning:)08:07
Haomeng|2rameshg87: yes, this is point08:07
Haomeng|2rameshg87: not the default enviroment08:07
rameshg87Haomeng|2: for yum i can see it has used ~/.cache/image-create/yum/x86_64/20/fedora08:07
Haomeng|2rameshg87: :)08:08
rameshg87but not for apt :-(08:08
Haomeng|2rameshg87: :)08:08
Haomeng|2rameshg87: raise this problem into tripleo irc, will get response I think:)08:08
rameshg87Haomeng|2: just wondering if i am missing something here and somebody knows caching for apt works for sure ..08:08
rameshg87Haomeng|2: yeah sure, will do :-)08:08
rameshg87Haomeng|2: thanks08:08
Haomeng|2rameshg87: :)08:09
Haomeng|2rameshg87: welcome08:09
*** lucasagomes has joined #openstack-ironic08:09
Haomeng|2rameshg87: this can help you  - http://www.tecmint.com/useful-basic-commands-of-apt-get-and-apt-cache-for-package-management/08:11
*** Mikhail_D_wk has left #openstack-ironic08:29
*** martyntaylor has quit IRC08:29
agordeevmorning Ironic :)08:30
lucasagomeslifeless, ping re 84361, I left a comment there about fixing the sleep in the tests08:32
lucasagomesmorning agordeev08:33
*** derekh has quit IRC08:33
lucasagomeslifeless, apart from that the patch lgtm08:34
lifelesslucasagomes: I don't have the bandwidth to push that further; care to just take it over?08:34
lucasagomeslifeless, ack08:35
lucasagomeswill send another patch set fixing it then :)08:35
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Workaround race with Neutron port updates  https://review.openstack.org/8436108:38
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Workaround race with Neutron port updates  https://review.openstack.org/8436108:39
*** athomas has joined #openstack-ironic08:40
*** lynxman has quit IRC08:41
*** lynxman has joined #openstack-ironic08:42
*** max_lobur has quit IRC08:43
agordeevlucasagomes: morning!08:43
*** martyntaylor has joined #openstack-ironic08:44
lucasagomes:)08:45
*** matsuhashi has quit IRC09:07
*** athomas has quit IRC09:08
*** matsuhashi has joined #openstack-ironic09:12
*** athomas has joined #openstack-ironic09:13
lucasagomescomstud, hi, i'd love to have a way to pull neutron and check whether the update_port() task was already finished or not09:16
lucasagomesinstead of having that sleep :/09:16
*** ilives has quit IRC09:18
*** ilives has joined #openstack-ironic09:18
*** Mikhail_D_wk has joined #openstack-ironic09:31
*** max_lobur has joined #openstack-ironic09:33
*** derekh has joined #openstack-ironic09:33
lifelesslucasagomes: rather than poll09:35
*** coolsvap has quit IRC09:35
lifelesslucasagomes: end the thread and wait for a callback09:35
lifelesslucasagomes: same as we do for for the image deploy09:35
*** max_lobur1 has joined #openstack-ironic09:36
lucasagomeslifeless, hmm... right, well that would introduce a lot of complexity with the locks, transitional state etc09:37
* lucasagomes gotta look better at the code to see how it's going to fit there09:37
*** max_lobur has quit IRC09:38
*** Mikhail_D_wk has quit IRC09:40
*** Mikhail_D_wk has joined #openstack-ironic09:41
*** matsuhashi has quit IRC09:59
*** matsuhashi has joined #openstack-ironic09:59
*** martyntaylor has quit IRC10:13
*** martyntaylor has joined #openstack-ironic10:15
openstackgerritSandhya Balakrishnan proposed a change to openstack/ironic: Ironic User Guide  https://review.openstack.org/8334310:17
*** coolsvap has joined #openstack-ironic10:25
openstackgerritSandhya Balakrishnan proposed a change to openstack/ironic: Ironic User Guide  https://review.openstack.org/8334310:27
*** vkozhukalov has joined #openstack-ironic10:38
*** coolsvap has quit IRC10:38
*** ndipanov has quit IRC10:44
*** matsuhashi has quit IRC10:48
*** matsuhashi has joined #openstack-ironic10:55
*** romcheg has joined #openstack-ironic11:10
*** romcheg1 has joined #openstack-ironic11:10
*** romcheg has quit IRC11:10
*** matsuhashi has quit IRC11:15
*** matsuhashi has joined #openstack-ironic11:15
*** matsuhashi has quit IRC11:19
*** matsuhashi has joined #openstack-ironic11:21
vkozhukalovjroll: which commit of openstack/oslo-incubator did you sync into ironic-python-agent? I need to add processutils and  config.generator oslo modules and, of course, I need to re-sync them but in order to keep them consistent I need to know commit. Or maybe we can sync stable/havana?11:34
*** pradipta is now known as pradipta_away11:40
*** rameshg87 has left #openstack-ironic11:41
*** matsuhashi has quit IRC11:52
*** subah has quit IRC12:19
*** linggao has joined #openstack-ironic12:29
agordeevadam_g: https://review.openstack.org/#/c/85013/12:31
*** ndipanov has joined #openstack-ironic12:33
*** jdob has joined #openstack-ironic12:37
*** killer_prince has quit IRC12:45
*** nosnos has quit IRC12:50
*** matty_dubs|gone is now known as matty_dubs12:51
openstackgerritPablo Fernando Cargnelutti proposed a change to openstack/ironic: Overwriting node_is_available in IronicDriver  https://review.openstack.org/8421412:56
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute utility and rootwrap module  https://review.openstack.org/8502513:00
*** romcheg1 has left #openstack-ironic13:01
*** romcheg1 has joined #openstack-ironic13:01
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute utility and rootwrap module  https://review.openstack.org/8502513:03
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added some infrastructure tools  https://review.openstack.org/8503013:21
*** killer_prince has joined #openstack-ironic13:23
*** rloo has joined #openstack-ironic13:24
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute utility and rootwrap module  https://review.openstack.org/8502513:35
*** jbjohnso_ has joined #openstack-ironic13:36
*** jgrimm has quit IRC13:45
*** ilives has quit IRC13:48
*** sabah has joined #openstack-ironic13:49
*** ilives has joined #openstack-ironic13:52
NobodyCamGood morning Ironic13:54
romcheg1Morning NobodyCam lucasagomes and everyone else!13:54
NobodyCammorning romcheg1 :)13:55
*** romcheg1 is now known as romcheg13:55
romchegfixed :)13:55
NobodyCam:)13:55
agordeevNobodyCam: romcheg morning!13:56
lucasagomesmorning romcheg NobodyCam13:56
NobodyCammorning lucasagomes13:57
NobodyCamlucasagomes: I see in scroll back you pucked up lifeless' patch13:57
NobodyCampicked even13:58
NobodyCamПӨBӨDΥCΛM need coffee13:58
lucasagomesNobodyCam, it was just a quick fix in the tests13:58
lucasagomesso that sleep doesn't affect the time to run the tests13:59
NobodyCamahh ok. So I still need to dig to what up with the TripleO ci14:00
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added some oslo modules  https://review.openstack.org/8503614:01
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Add DiskPartitioner  https://review.openstack.org/8339614:01
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Use DiskPartitioner  https://review.openstack.org/8339914:01
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Get rid of the swap partition  https://review.openstack.org/8372614:01
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Use GB instead of MB for swap  https://review.openstack.org/8378814:01
jbjohnso_question, would people cry a lot if I changed pyghmi to use M2Crypto instead of PyCrypto?14:02
lucasagomesjbjohnso_, hmmm, PyCrypto is py3 compatible isnt it?14:03
lucasagomeswhere M2Crypto is not14:03
lucasagomesidk how py3 friendly is pyghmi right now14:03
lucasagomesbut it sounds like a step back14:03
lucasagomeswhat r the reasons?14:03
jbjohnso_lucasagomes, well, was going to see if there was a good reason, performance I was thinking of measuring14:05
jbjohnso_lucasagomes, currently, 'while :; do dmesg; done' on a SOL console takes 3% of a 2.6 GHz CPU14:06
jbjohnso_mostly in pycrypto stuff14:06
jbjohnso_so was contemplating that14:06
lucasagomesjbjohnso_, and with the M2Crypto? how much improvement you get?14:06
jbjohnso_dunno, was deciding whether it was worth checking14:06
lucasagomesif it's not something like WOW I would stick to PyCrypto14:07
jbjohnso_if there was a 'no way', then not bother14:07
jbjohnso_I'll probably at least change to use python hashlib instead14:07
lucasagomesI think py3 is an important variable in the game14:07
jbjohnso_for hash14:07
lucasagomesright14:07
lucasagomesyeah sounds grand14:07
jbjohnso_the cipher stuff though14:07
jbjohnso_well, guess I'll change the hmac for now, see how much closer that gets me14:08
jbjohnso_and maybe consider opportunistic use of M2Crypto if it looks appealing14:09
*** sabah has quit IRC14:09
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added some infrastructure tools  https://review.openstack.org/8503014:10
jbjohnso_hmm, realizing why I see HMAC so high in the profiling results..14:11
jbjohnso_the AES in pycrypto is C code, the HMAC is pure python, guess that might help a great deal...14:11
lucasagomesheh yeah14:11
lucasagomesplus the problem with HMAC only accepting byte strings14:12
jbjohnso_well, python core and PyCrypto share that attribute14:12
lucasagomesyeah :( I remember u were testing14:12
jbjohnso_no big deal, doing an '.encode' is trivial14:12
jbjohnso_and that info is only processed once during login14:13
jbjohnso_the rest of the hmacs are things that are naturally bytearrays (packet data)14:13
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute utility and rootwrap module  https://review.openstack.org/8502514:15
lucasagomesyeah I remember the code you put on pyghmi to convert/tolerate unicode strings14:15
lucasagomesgood stuff14:15
jbjohnso_utf8 should be good enough for anyone14:16
lucasagomes+114:16
jbjohnso_some non-ascii passphrases may not be able to be 20 characters long though14:16
jbjohnso_and I have no idea if ipmitool can actually set non-ascii passphrases14:16
lucasagomesme neither14:17
lucasagomesworth testing tho14:17
openstackgerritYuriy Zveryanskyy proposed a change to openstack/ironic: Fix max retries message for _sync_power_states  https://review.openstack.org/8504414:18
jbjohnso_yeah, to have a password like 葵14:18
NobodyCam刀o乃odリcムᄊ14:18
romchegNobodyCam: шо то в дідька таке? :)14:19
lucasagomesheh14:20
NobodyCamromcheg: http://allfont.wapka.mobi/index.xhtml14:20
lucasagomesééããççç14:20
lucasagomes:D14:20
NobodyCamレucム丂ムgoᄊ乇丂14:21
romchegг๏๓ςђєﻮ14:21
dtantsurwow Oo14:21
NobodyCam:)14:21
NobodyCamdtantsur: see link I just posted14:21
dtantsur๔tคภtรยг14:22
NobodyCam:)14:22
lucasagomesgosh heh14:22
dtantsurI have one great IRC nick now :)14:22
dtantsurdoes IRC support non-latin nicks?14:22
NobodyCamthou irrsi wont let me set a utf8 nick14:22
jbjohnso_let's find out...14:22
lucasagomesis that thai?14:22
romchegdtantsur: No14:22
jbjohnso_nope...14:22
romchegI've just tried14:22
NobodyCamhehehe14:22
dtantsurromcheg, that's a pity :(14:22
jbjohnso_'erroneous nickname'14:22
NobodyCamyep14:23
jbjohnso_now that would have been an evil thing to do to tab completion of nicks14:23
NobodyCamheheheh14:23
NobodyCambrb14:23
jrollmorning ironic14:24
*** jgrimm has joined #openstack-ironic14:24
*** sseago_ has joined #openstack-ironic14:24
jrollvkozhukalov: afaik I synced master? I just ran their update.py script with the config in the agent repo...14:25
matty_dubsNobodyCam: ⓞⓗ, ⓣⓗⓘⓢ ⓘⓢ ⓒⓞⓞⓛ.14:25
jrollvkozhukalov: I'm not sure why it matters with commit14:25
jrollvkozhukalov: I would prefer master though14:25
vkozhukalovjroll: i'd prefer stable, afaik it is likely to get not working state when you sync unstable oslo14:27
NobodyCammorning jroll14:28
NobodyCammatty_dubs: hehehe14:28
jrollvkozhukalov: if something doesn't work, then we fix it upstream :)14:28
jrollvkozhukalov: maybe later on we can sync some stable release, but the agent doesn't even work right now, so...14:28
vkozhukalovok, will sync master then14:29
jrollthanks14:30
*** _LXXIII_ has joined #openstack-ironic14:30
jrollvkozhukalov: why do we need config.generator?14:31
agordeevjroll: morning14:31
NobodyCammorning agordeev :)14:31
jrollthe reason I ask, is because operationally, config files will be hard to manage in a ramdisk14:32
NobodyCamand morning matty_dubs14:32
jrollI'd prefer kernel or command line args14:32
*** _LXXIII has joined #openstack-ironic14:33
*** _LXXIII has quit IRC14:33
vkozhukalovjroll: i want agent to use oslo.config and config.generator is nothing than just a convenient way to generate config sample14:33
*** sabah has joined #openstack-ironic14:34
Mikhail_D_wkMorning/afternoon all! :)14:34
agordeevMikhail_D_wk: morning14:34
vkozhukalovjroll: but it is quite convenient for testing14:34
NobodyCamMorning Mikhail_D_wk14:35
jrollvkozhukalov: ok, let's back up then. I'm skeptical on using oslo.config at all. what kind of options are going in there?14:35
*** _LXXIII_ has quit IRC14:36
dtantsurMikhail_D_wk, мσяηιηg14:37
devanandamorning, all14:38
agordeevjroll: doesn't oslo.config allow to use config file and command line args simultaneously? IMBW14:38
agordeevdevananda: morning!14:38
NobodyCamgood mornig devananda14:38
vkozhukalovjroll: plenty of them: logging options, authentication options, extensions activation options, path options, etc14:38
Mikhail_D_wkmorning devananda :)14:38
lucasagomesmorning devananda vkozhukalov14:38
vkozhukalovlucasagomes: morning14:38
lucasagomesmorning Mikhail_D_wk14:39
jrollagordeev: does it support /proc/cmdline? :P14:39
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added oslo modules  https://review.openstack.org/8505214:39
jrollvkozhukalov: hmm14:39
jrollvkozhukalov: I guess let's get the reviews up and go from there14:39
devanandajroll: what about a kernel param hook for it to pull in a config file?14:39
jrolldevananda: as in, pull in a remote config file?14:40
devanandajroll: if the initial ramdisk is served from a static DHCP/TFTP server, even if taht chains up to iPXE, you wont have control over unique kernel params14:40
devanandajroll: so if the driver needs to determine start-up options for the agent itself, you'll need a config file or comnandline params, ie, oslo.config14:41
devanandai'm not sure that it does. just saying "if" :)14:41
jrolldevananda: we already have dynamic kernel params today, not from the driver though14:42
devanandaeg, for plugins. I think two approaches are both valid: config specifies which plugins to load || plugins are loaded by stevedore when requested via the agent's API14:42
devanandajroll: from where?14:42
jrollipxe.txt is served over http14:42
jrollso we have a little webapp sitting on our dhcp server14:42
jrollnot ideal by any means14:43
devanandanope14:43
jrollbut it's a quick hack we did to work around some other things14:43
devanandadriver needs to be able to programatically set things ilke kernel params14:43
jrollit's something I want to think more about, it's kind of a hard problem no matter which way we go14:43
devanandaserving ipxe.txt from ironic-api is an interesting idea ...14:43
jrollbut wdyt about yes that14:43
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Synced oslo modules  https://review.openstack.org/8503614:43
devanandaseems logical to me14:44
devanandawell14:44
devanandanetwork layer seapration14:44
jrollthe hard problem we solved with a dynamic ipxe.txt was, "how does the agent know what IP it's accessible at"14:44
jrollmmm14:45
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Get rid object model `dict` methods part 3  https://review.openstack.org/6410814:45
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Get rid object model `dict` methods part 1  https://review.openstack.org/6002514:45
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Get rid object model `dict` methods part 2  https://review.openstack.org/6233114:45
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Get rid object model `dict` methods part 5  https://review.openstack.org/6427814:45
devanandaDC/sec/ops folks want to keep the IPMI network separate, only bridging ironic-conductor to it, and then also lock down the node's early network access to prevent injection from other running instances14:45
jrollyeah, in decom mode, we won't want the agents to be able to access ironic-api14:45
devanandaright14:45
devanandawell14:46
devanandawe need the agent to access *an* ironic-api service for enrollment and just to talk to the driver14:46
russell_hright14:46
devanandaso pulling ipxe.txt from the same IP is not any change to networking14:46
russell_hshould that be "the ironic API"?14:46
devanandathere may be a public IP and a private IP for reaching "the ironic API" endpoint14:47
devanandadepends on deloyer's needs14:47
jrollrussell_h: well, we could have a "decom" ironic-api service that only allows lookup/heartbeat14:47
russell_hright, thats how I saw it14:47
russell_hdevananda: we're going to have an internal address that only allows whitelisted access to a few paths14:47
devanandarussell_h: have you looked at the auth stuff for implementing that?14:48
devananda(i haven't)14:48
jrollI haven't either14:48
russell_hnope, I was under the impression that something like this exists14:48
russell_hbut if it doesn't, I'll make it or something :)14:49
devanandai'm not sure. haven't looked enough at keystone v3's capabilities14:49
russell_hplus, nginx14:49
devanandammmm, more dependencies --14:49
russell_hright, if this is going to be the prescribed method the prescription should not say "deploy nginx with this configuration"14:50
devanandaright14:50
devanandawe should investigate existing methods before going down that route, though14:50
russell_hat any rate, conceptually, you feel we should stick to extending the existing API instead of building some new one for the agent to hit?14:51
devanandayes14:51
russell_hdevananda: what do you want to do about this: https://review.openstack.org/#/c/81919/14:52
russell_hemail never generated much traffic14:52
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added some infrastructure tools  https://review.openstack.org/8503014:53
devanandarussell_h: " I don't believe that this is a useful way for vendors to implement new consumer-facing functionality."14:55
devanandarussell_h: is that in reference to #1 or #2 in your email?14:55
russell_hI don't think its a useful way to accomplish #114:56
devanandarussell_h: in re-reading your email, I think it wasn't a clear distinction between the functionality of "node-specific" and "node-non-specific" passthru14:57
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute utility and rootwrap module  https://review.openstack.org/8502514:57
russell_hat least I can't envision what sort of consumer-facing API a vendor would want to add which isn't specific to a node14:57
devanandaI also dropped the ball in replying to it during the release preparation period14:57
russell_hor at least a chassis or something if they're getting wild14:58
devanandarussell_h: discover14:58
russell_hdiscover what? unregistered nodes?14:58
devanandayes14:58
russell_hinteresting14:58
russell_hhadn't thought of it that way14:58
devanandachassis-level and IPMI-net-range have both been suggested14:58
russell_hright, so those make sense14:59
devanandaeg, go discover 10.1.255.25514:59
jbjohnso_devananda, btw, I was going to add detection of snooped service processor and firmware identifiers14:59
russell_hand are probably where discover _should_ be implemented (if it should be implemented)14:59
jbjohnso_devananda, to my hardware management daemon14:59
jbjohnso_devananda, it would be able to passively detect a lot of things unobtrusively14:59
devanandajbjohnso_: hmm... hardware mgmt daemon? I'm sure you mentioned it and I'm just blanking on the reference14:59
jbjohnso_devananda, yeah, I should demo it...15:00
NobodyCamwe need better logging in ssh driver :-p15:00
jbjohnso_devananda, it can do pretty nifty console stuff now...15:00
devanandaNobodyCam: yes!!!!15:00
devanandaNobodyCam: i beleive adam_g put up a patch to squelch a lot of the debug noise15:00
jbjohnso_I was thinking of having a show and tell of the updated state of my daemon tomorrow using a crappy web conference15:00
jbjohnso_(anything that calls for a java plugin is crappy)15:01
NobodyCamdevananda: I am looking into the ci issue with triple O15:01
jbjohnso_even crappier because I have to use windows to share my screen15:01
jbjohnso_it's very weird to RDP into a windows vm to Xpra over to a linux box15:02
*** martyntaylor has quit IRC15:02
devanandajbjohnso_: you should, you know, just present it at OSCON or something ;)15:02
jbjohnso_but a sneak example15:02
openstackgerritMikhail Durnosvistov proposed a change to openstack/ironic: Unified object model  https://review.openstack.org/8506515:03
jbjohnso_for power state, one url to use could be http://localhost:4005/nodes/n1/power/state.json15:03
*** ndipanov_ has joined #openstack-ironic15:03
*** ndipanov__ has joined #openstack-ironic15:03
jbjohnso_the console throught http is not restful though15:04
*** ndipanov__ has quit IRC15:04
jbjohnso_it takes about 9-10 milliseconds to get power state for one node15:04
jbjohnso_actually, that's including starting wget15:04
*** ndipanov has quit IRC15:04
jbjohnso_the console part is the cool part though, well that and the configuration and the filesystem-like interactive cli15:05
devanandacomstud: if you have time, could you look at mikhail's patch series ^^15:05
devanandacomstud: it seems possibly overlapping with yours?15:05
NobodyCamdevananda: when you have a second can you un -2 https://review.openstack.org/#/c/6410015:05
devanandaNobodyCam: done15:06
NobodyCamty15:06
*** _LXXIII_ has joined #openstack-ironic15:06
devanandahah15:07
devanandaour queue went from < 1page to > 2 pages overnight :)15:07
NobodyCam:) yay juno15:09
*** dguerri_ is now known as _dguerri15:11
*** _dguerri is now known as dguerri_15:12
dtantsurAs to me, we still have things that I would call critical for I, like https://review.openstack.org/#/c/84497/ or https://review.openstack.org/#/c/84396/ :(15:13
vkozhukalovguys, I have a question. lucasagomes suggested to use parted python binding https://fedorahosted.org/pyparted/  for partitioner. I believe it is quite stable and it is written in C. But I think it is much more convenient to use subprocess + parted instead, because many people know how to use it, it is much easier to review, it is much easier to fix bugs. On the other hand there are lvm python bindings as well. Some of them pure python, othe15:13
*** martyntaylor has joined #openstack-ironic15:13
jbjohnso_devananda, I need a conference near my lazy self15:14
NobodyCamjbjohnso_: do like I do ... get an RV and call it home. then move the house to the conf :-p15:15
dtantsurvkozhukalov, at least they need to do this: https://fedorahosted.org/pyparted/ticket/5215:17
jbjohnso_so 4 consoles going full tilt in a vm on a 2.6 ghz core runs consistently between 8 and 11 percent cpu...15:17
devanandadtantsur: i agree that both of those are backport potential15:18
lucasagomesvkozhukalov, the reason why i didn't use the parted binds is becuase the process needs to run as root15:18
lucasagomesbut they are very useful15:18
dtantsurlucasagomes, the fact it's not on pypi may be a problem15:19
lucasagomesI used it in another project, I liked... but I wouldn't oppose to have the parted + subprocess tho15:19
lucasagomesdtantsur, yeah indeed15:19
* NobodyCam gets mo joe15:22
vkozhukalovlucasagomes: dtantsur: devananda: ok, thank you for your opinions15:24
vkozhukalovwill use commands + subprocess + rootwrap15:25
lucasagomesvkozhukalov, np :)15:25
lucasagomesvkozhukalov, ack15:25
devanandadtantsur: i've tagged both bugs with icehouse-backport-potential15:25
lucasagomesdtantsur, ops I forgot to add the 1MiB offset to that patch15:26
dtantsurlucasagomes, yes, thus -1 ;)15:27
lucasagomesdtantsur, heh, I fixed on the other one (long-term) https://review.openstack.org/#/c/83396/15:27
lucasagomesand forgot that one15:27
lucasagomeswill change it15:27
dtantsurlucasagomes, thanks!15:28
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Switch from PyCrypto HMAC to standard python hmac  https://review.openstack.org/8507515:32
NobodyCamwatching a jenins test can be much like watching paint dry15:34
NobodyCamjenkins even15:34
*** eghobo has joined #openstack-ironic15:47
*** dwalleck has joined #openstack-ironic15:49
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Synced oslo modules  https://review.openstack.org/8505215:49
NobodyCamwhere is the check-tripleo/check-tripleo-ironic-undercloud-precise job15:51
NobodyCamdevananda: ^^ do you know?15:51
*** dguerri_ is now known as _dguerri15:53
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Synced oslo modules  https://review.openstack.org/8505215:53
*** ndipanov_ has quit IRC15:54
*** _dguerri is now known as dguerri_15:56
NobodyCambouncing bubbies tell me it is bbt time... brb15:57
*** matty_dubs is now known as matty_dubs|lunch15:58
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: JsonPatch add operation on existing property  https://review.openstack.org/8508615:58
lucasagomesdevananda, ^ it wasn't fixed no, the 'add' as replace only worked for multi-level attributes (like extra, driver_info, prorperties)15:59
devanandaNobodyCam: last i heard, triple-ironic-seed is trustworthy, but undercloud is not. but IMBW16:02
devanandalucasagomes: ahhh. right. thanks16:02
NobodyCamyes looking in to that16:02
devanandai'm heading into the office today, will be on intermittently16:05
NobodyCamack :) say hi to folks from me16:05
*** ndipanov_ has joined #openstack-ironic16:08
lucasagomesNobodyCam, root_gb in nova is the "disk size" from flavor?16:12
*** jistr has quit IRC16:13
*** krtaylor has quit IRC16:13
NobodyCamlucasagomes: yes.. but I think it should be root_gb = disk_size - swap - eff16:14
lucasagomeshmmm yeah idk if they r doing that calculation tho16:15
* NobodyCam wander if we should have also had a reserved16:15
NobodyCamlucasagomes: i do not think we are16:17
devanandatheyre not. last i looked16:17
devanandaroot = disk. swap = swap. eph = eph16:17
lucasagomesyeah they are not |                        | u'pxe_root_gb': u'10',16:18
lucasagomesNobodyCam, in Ironic its correct we have 1 field for each16:18
lucasagomesNobodyCam, properties/local_gb == disk size16:18
lucasagomesroot_gb, swap_gb, ephemeral_gb16:18
lucasagomeswhcih is grand16:18
devanandaanother impedence mismatch between virt (just allocate what you ned) and physical (you have what you have, except for when you have a SAN)16:18
lucasagomesproblem is in nova16:18
lucasagomesdevananda, yeah, could be workarounded in the driver tho16:19
lucasagomesI mean Ironic driver16:19
NobodyCamso to debug this is issue are there objections to me setting debug true for a bit?16:19
lucasagomesNobodyCam, ?16:19
NobodyCamtripleo undercloud is having issues atm16:20
lucasagomesdevananda, ephemeral and swap is only a baremetal thing for nova?16:20
devanandaok, no16:20
devanandas/ok//16:20
devanandalucasagomes: those are for VMs too16:21
lucasagomesdevananda, so nova should do the calculation16:21
lucasagomesdisk == root_gb - swap_mb - ephemeral_gb16:21
devanandaif ironic exposes "disk size", then the nova ExactMatchDiskScheduer needs to do something16:21
devanandaya16:21
devanandaor16:21
devanandaif (disk != (root + swap + eph)) and not using_san():16:22
devananda  # don't match16:22
lucasagomesdevananda, yeah16:22
lucasagomes+116:22
devanandaor something16:22
devanandaok, i gotta run, bbiah16:22
lucasagomesI had one more thing heh16:22
lucasagomeswe can talk later about it then16:22
NobodyCamstay warm devananda :)16:22
devanandalucasagomes: pnig me on gtalk16:22
lucasagomesdevananda, ack16:22
openstackgerritRuby Loo proposed a change to openstack/python-ironicclient: Documentation for contributors  https://review.openstack.org/8509616:22
lucasagomesit's not critical anyway16:22
*** max_lobur1 has quit IRC16:23
*** sphoorti has joined #openstack-ironic16:25
*** sphoorti has quit IRC16:30
comstuddevananda: will do16:32
NobodyCampost bbt walkies...bbiafm16:36
comstuddevananda: just a 2 line patch right now that doesn't conflict16:36
*** stevehuang has quit IRC16:36
comstudbut leaving a msg16:36
*** ifarkas has quit IRC16:36
comstudah16:36
*** romcheg has quit IRC16:39
*** killer_prince has quit IRC16:42
openstackgerritJarrod Johnson proposed a change to stackforge/pyghmi: Switch from PyCrypto HMAC to standard python hmac  https://review.openstack.org/8507516:48
*** krtaylor has joined #openstack-ironic16:50
openstackgerritRuby Loo proposed a change to openstack/python-ironicclient: Adds documentation for ironicclient API  https://review.openstack.org/8510716:53
lucasagomesaight, I came to the office today so I'm heading home now16:55
lucasagomeshave a gnight devananda NobodyCam everyone16:55
NobodyCamhave a good night lucasagomes16:55
*** lucasagomes has quit IRC16:56
russell_hagordeev: do you all have some sort of JSON payload that describes the hardware in a box?17:06
russell_hagordeev: in Fuel or something related17:06
russell_h^ or anyone else in the know17:06
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added some oslo modules  https://review.openstack.org/8503617:07
*** derekh has quit IRC17:08
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added some infrastructure tools  https://review.openstack.org/8503017:09
openstackgerritVladimir Kozhukalov proposed a change to openstack/ironic-python-agent: Added execute utility and rootwrap module  https://review.openstack.org/8502517:12
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Adding a reference driver for the agent  https://review.openstack.org/8479517:13
vkozhukalovrussell_h: in Fuel discovery agent parses output of ohai and reduces it significally, and yes it reports on master node hardware info as JSON17:14
*** harlowja_away is now known as harlowja17:14
russell_hvkozhukalov: do you have a link to what that JSON looks like?17:14
russell_hvkozhukalov: also, any thoughts on whether you'd change it any17:14
russell_hvkozhukalov: I'm thinking about what the agent's discovery payload should look like17:15
russell_hvkozhukalov: and I'm not interested in re-inventing this particular wheel if you all already have something17:15
russell_hhopefully would make things easier for you all too17:15
pquernaohai.17:16
pquernajust a tiny dependency17:16
JayFohai is a pretty good tool, but it's ruby and pretty heavy for a python agent17:17
russell_hwe don't need to _use_ ohai17:17
JayFjust mimic the structures it uses17:17
russell_hif it makes sense17:17
russell_hwe don't _need_ to do that either, but if its good lets use it17:17
vkozhukalovrussell_h: in Fuel we recognize node by its MACs17:17
*** ilives has quit IRC17:17
*** rloo has quit IRC17:18
vkozhukalovrussell_h:  and it looks like that http://paste.openstack.org/show/74992/17:18
*** ilives has joined #openstack-ironic17:18
vkozhukalovpquerna: salt grains has pretty much the same functionality17:20
*** max_lobur has joined #openstack-ironic17:20
russell_hvkozhukalov: nice, thanks17:20
*** lazy_prince has joined #openstack-ironic17:20
*** lazy_prince is now known as killer_prince17:20
devanandaback17:20
NobodyCamwb17:21
*** athomas has quit IRC17:21
openstackgerritA change was merged to stackforge/pyghmi: Switch from PyCrypto HMAC to standard python hmac  https://review.openstack.org/8507517:23
devanandaJoshNang: how close is IPA's driver to the reference PXE driver? (i haven't looked yet)17:24
vkozhukalovrussell_h:  but actually, we are not limited by this particular format, it is just important for us that new format contains at least that amount of data about node or even more17:25
*** romcheg has joined #openstack-ironic17:25
JayFvkozhukalov: I think it's more of a matter of, we need a format, and if one already exists might as well use it as a base unless there's a good reason not to17:25
JayFvkozhukalov: and I don't see a reason we can't format the data in a similar + possibly compatible way17:25
JayFAlthough we'll certainly want to include more.17:26
JoshNangdevananda: it does deploy, tear down, and validate, uses vendor passthru.17:26
vkozhukalovrussell_h:  JayF: great17:26
JoshNangdevananda: i'm not totally sure what you mean.17:26
devanandaJoshNang: to be a DeployDriver, it must do those17:26
devanandahm, who's pcsforeducation ?17:27
russell_hthats JoshNang17:27
JayFJoshNang is pcsforeducation17:27
JoshNangyeah mea17:27
JoshNang*me17:27
devanandaoh17:27
JoshNangwhere did it pop up?17:28
devananda# TODO(pcsforeducation)17:28
devanandais in a few place17:28
JoshNangoh :) i'm pcsforeducation on github17:28
devanandaheh17:28
russell_hwhat name is that supposed to be?17:28
devanandai was about to say, please use your irc nic or gerrit user name17:28
NobodyCamwoo hoo I can read the WhiteBoard again :-p17:29
ShrewsFor the record, I find it odd that a nova API method called get_host_stats() is only used to get a single attribute called 'hypervisor_hostname'17:29
JoshNangack17:29
Shrewsdevananda: ^^^^ go figure17:30
devanandaShrews: ... right ...17:30
devanandathe nova driver API seems to have grown a bit since I stopped working in there a year ago17:31
Shrewsdevananda: so, i think i can simplify your patch a great deal. running it through devstack now17:31
devanandaShrews: ++17:32
*** matty_dubs|lunch is now known as matty_dubs17:35
*** vkozhukalov has quit IRC17:36
*** rainya has quit IRC17:43
*** sabah has quit IRC17:49
devanandaadam_g: have you run the virtual-ironic job on https://review.openstack.org/#/c/81958/7 ?17:53
*** pcargnel has joined #openstack-ironic17:54
pcargnelHi! Can someone help to figure out which is the error in this one? https://review.openstack.org/#/c/84214/17:57
pcargnelIs check-tripleo-ironic-undercloud-precise a new job in the builds? I didn't notice until yesterday17:57
openstackgerritChris Krelle proposed a change to openstack/ironic: Add INFO level logging to ssh.py  https://review.openstack.org/8512417:58
*** derekh has joined #openstack-ironic17:58
NobodyCampcargnel: I am looking into that now17:58
devanandapcargnel: I don't believe that check-tripleo-ironic-undercloud-precise is passing for anything yet17:59
NobodyCamI have a review up that should give us debug logges .. so we can figure out whats going on.17:59
adam_gdevananda, hmm no i haven't. i didnt know that it would be triggered there17:59
devanandaNobodyCam: last i heard from lifeless, check-tripleo-ironic-seed should be passing (and it does) but -undercloud is not17:59
devanandaadam_g: post a comment sayign "check experimental"17:59
adam_gdevananda, oh, but once that merges theres one minor change to devstack that would need to land to enable the driver testing17:59
NobodyCamyes17:59
devanandaadam_g: so the virtual-ironic job is in the experimental queues for devstack, devstack-gate, and tempest18:00
NobodyCamdevananda: I put up this https://review.openstack.org/8510118:00
devanandaadam_g: and we should run it for any patch we're making to those projects18:00
NobodyCamso I can get debug logs18:00
devanandaadam_g: as an experimental job, it only runs when someone triggers it with a comment18:00
adam_gdevananda, cool, but as i said that exercise wont run until devstack has a patch to set 'driver_enabled=True' in [baremetal] of tempest.conf18:01
devanandaadam_g: ahhh. ok. can we land that in devstack first? :)18:02
adam_gdevananda, yeah--may need to relax that crazy regex too to include a single scenario test18:03
adam_galso two more things CI related18:03
adam_ghttp://logs.openstack.org/99/83399/11/check/check-tempest-dsvm-virtual-ironic/dc5733d/logs/testr_results.html.gz18:03
adam_gis looking *much* better.. the two failures are actually bugs in tempest18:03
*** rwsu has quit IRC18:03
dwalleckwow. nice18:04
adam_ghttps://bugs.launchpad.net/tempest/+bug/130178218:04
adam_ghttps://bugs.launchpad.net/neutron/+bug/126549818:04
devanandaawesome18:05
NobodyCamnice... and now its Bagel time18:05
adam_gdevananda, and re: the nova unit tests not showing up in the results of https://review.openstack.org/#/c/84033/ -- its separate tox command that need to be run in addition to py27/py2718:06
adam_gso we will likely need additional gate-ironic-nova-python26 + 27 check jobs18:06
*** overlayer has joined #openstack-ironic18:06
adam_gunless there is some magic we can do to get the nova tests to actually run as part of the ironic suite, but i havne't been able to18:07
devanandai think there should be18:09
devanandalemme see what i can do18:09
adam_gdevananda, there are issues having testr import both set of tests (and ironic + nova together) during test discovery. i think comments in that review describe the issue18:10
openstackgerritJim Rollenhagen proposed a change to openstack/ironic: Add agent fields to virt driver  https://review.openstack.org/8513118:10
dwalleckadam_g: So it looks like the libvirt issue is fixed?18:13
adam_gdwalleck, yeah--i just bumped the job to use the icehouse UCA pocket instead of havana, where a newer libvirt package has all thebug fixes we're waiting on18:13
openstackgerritPablo Fernando Cargnelutti proposed a change to openstack/ironic: Change admin_url help in ironic driver  https://review.openstack.org/8414818:20
*** k4n0 has joined #openstack-ironic18:21
*** rloo has joined #openstack-ironic18:23
ShrewsNobodyCam: for https://review.openstack.org/85124, you might want to first check out my review: https://review.openstack.org/8449618:24
* NobodyCam looks18:24
*** rloo has quit IRC18:25
*** rloo has joined #openstack-ironic18:25
openstackgerritAndreas Jaeger proposed a change to openstack/ironic: Use xargs -0 instead of --null  https://review.openstack.org/8513518:26
*** rloo has quit IRC18:26
*** rloo has joined #openstack-ironic18:26
NobodyCamShrews: you mean set 84496 as Dependencies18:27
devanandaadam_g: jsut talked with clarkb about your proposed unit test stuff, he has a slightly different idea i'm giong to try18:27
ShrewsNobodyCam: i mean it might negate the need for your change.18:27
adam_gdevananda, was that on IRC somewhere or face-to-face? im interested to know what it is18:28
ShrewsNobodyCam: logs with DEBUG are much more readable with my proposal18:28
NobodyCamShrews: I added because we had no info level logging in ssh at all...18:28
NobodyCambut yes I can see how 84496 cleans up the debug level18:29
adam_gdevananda, if there were no issues importing nova/ironic together (oslo.config) it would be a simple as making nova available (either via git clone or test-requirements.txt) and adding the missing __init__.py to ironic/nova/tests/ so it is discovered and loaded with the others by testr18:29
NobodyCamShrews: we support verbose= in the conf file .. .but don't actually log much at info level18:29
NobodyCamnobodycamAir:ironic NobodyCam$ grep -r LOG.info *|wc -l18:32
NobodyCam3618:32
NobodyCamif you take common out we have only 4 lines of info logging18:34
NobodyCamso I think that we need more info level logging over all18:35
*** derekh has quit IRC18:37
*** dshulyak has joined #openstack-ironic18:38
*** dshulyak has quit IRC18:38
*** derekh has joined #openstack-ironic18:39
*** killer_prince has quit IRC18:42
devanandaadam_g: f2f, i'm in the office18:43
Shrewsugh... that moment when you realize that the LOG.debug() call you put in to help find a bug was actually causing another bug18:47
NobodyCamugggh :-p18:47
NobodyCamlog caused a error?18:47
Shrewsyeah18:48
JoshNangheh i was just debugging one of those last night. a bad mock was messing with log.exception but not log.error in some tests.18:49
NobodyCamnice18:50
NobodyCambrb18:50
openstackgerritA change was merged to openstack/ironic: Fix grammar in error string in pxe driver  https://review.openstack.org/8447118:51
devanandalifeless: having testr not barf on bad imports would, you know, be helpful. apparently that's fixed now?18:51
NobodyCamand back18:53
*** derekh has quit IRC18:54
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Adding a reference driver for the agent  https://review.openstack.org/8479519:05
dwalleckadam_g: Once you fixed the libvirt issue with RS, did you have any issues launching servers? I got everything working but I've had this server in the building state, trying to figure out where it's hung up19:05
adam_gdwalleck, no its been working okay for me , let me give you the localrc i've been using19:06
*** vkozhukalov has joined #openstack-ironic19:06
*** stevehuang has joined #openstack-ironic19:06
adam_gdevananda, you get better output now letting you know what failed to import in a human readable format19:07
dwalleckthanks. I believe it's jacked on waiting for a neutron port to come up. No errors though. Good times19:07
adam_gdwalleck, http://paste.ubuntu.com/7200113/  <- this is fairly close to what gets used in the devstack gate checks19:08
dwalleckThanks. It looks pretty close to what I'm using (I'm using what's in the dev docs). Let me try this and see if I have more luck19:10
adam_gdwalleck, if you watch syslog wihile the node is booting you can confirm its coming up with correct networking by watching it DHCP + tftp19:11
*** derekh has joined #openstack-ironic19:11
dwalleckthanks, I'll give it a go19:14
jrollNobodyCam: I think it was you I was asking about the fields thing in the virt driver - does this make sense to you? https://review.openstack.org/#/c/85131/19:17
NobodyCamjroll: if I am recalling what we chatted about .. ya...19:20
openstackgerritJim Rollenhagen proposed a change to openstack/ironic: Add agent fields to virt driver  https://review.openstack.org/8513119:21
jrollok, cool19:22
jrollthanks19:22
JayFcheck-tripleo-ironic-seed-precise are the failures that are supposed to be voting, and the undercloud is reliably broken, right?19:23
NobodyCamJayF: just waiting for debug log to dive in19:23
JayFNobodyCam: I'm asking this more in the context of reviewing19:24
NobodyCambeen in the queue for 3 hours now19:24
NobodyCamahh yes then it be broken19:24
JayFi.e. I probably shouldn't +1 things that are failing check-tripleo-ironic-seed-preicse, right?19:24
NobodyCam:-p19:24
NobodyCamyes seed is working19:24
JayFwhile you're looking, it's 'recheck experimental' to get it to rerun, correct?19:25
NobodyCamI use recheck nobug or recheck bug# most offten19:26
NobodyCamexperimental is for the tempest jobs ... I think....19:26
NobodyCamdevananda: ^^^19:26
JayFYeah I just didn't know if that retriggered the tripleo ones as wel19:26
JayF*well19:27
NobodyCamok be back in a few min...19:29
*** overlayer has quit IRC19:29
lifelessdevananda: it is, pending some pbr changes to release. you can pip install from bzr19:33
linggaomatty_dubs, ping19:36
*** mrda_away is now known as mrda19:42
matty_dubslinggao: pong19:44
matty_dubs(sorry, was AFK)19:44
linggaomatty_dubs, np. I am playing with the console code that sjing has put in. And found some problems. Just want to check with you to see what were the probmems when you were testing it.19:46
matty_dubslinggao: I had problem with shellinaboxd starting/shutting down19:47
matty_dubsAnd I think it was causing the node to stay locked19:47
linggaomatty_dubs I saw that problem.19:48
linggaothe pid file was not created.19:48
matty_dubsMy memory is a little  hazy; I think I was getting a .pid file, but the service couldn't be stopped.19:49
matty_dubsBut a lot could have changed since.19:49
linggaoI have a question for a different problem, should each node have different ipmi_terminal_port?19:50
linggaootherwise the console will run into each other if we open console for more than one nodes.19:50
matty_dubslinggao: I didn't consider that at first, and I don't think the cond considers it.19:50
matty_dubs*code19:50
matty_dubsBut yes, we will.19:51
matty_dubsSince only the first one can be on 420019:51
matty_dubs(Unless I am missing something fundamental.)19:51
linggaothen the admin assign a port for each node when registering a node?19:52
linggaoor should there be a terminal port pool that the code will assign the terminal port at run time when user request a console?19:53
*** ilives has quit IRC19:55
*** ilives has joined #openstack-ironic19:56
*** rwsu has joined #openstack-ironic19:56
matty_dubslinggao: You know, I honestly hadn't thought this through previously. The way it works now is that the port is assigned when the user registers the node.19:57
matty_dubsI don't know if that's ideal, though.19:57
matty_dubsI suppose it works.19:57
NobodyCamand back19:57
linggaomatty_dubs, thanks. I'll do some research in this area. I just worry about the port numbers will be all used if we have thousands of bm nodes.19:59
matty_dubslinggao: Yes, good point. It could also be a pain to have to manually assign.19:59
*** dwalleck has quit IRC19:59
linggaoDid you see other problems?19:59
matty_dubslinggao: I don't _remember_ any other problems, but it's possible I forgot. Are you hitting more?20:00
linggaoyes, like it does not pick up the path for shellinaboxd in the conf.20:01
linggaoironic node-get-console returns 'unicode' object has no attribute 'iteritems'20:01
*** harlowja has quit IRC20:02
linggaoI'll go a head fix them.20:02
linggaomatty_dubs, thanks a lot for your time.20:03
*** harlowja has joined #openstack-ironic20:03
*** ilives has quit IRC20:03
matty_dubsAh, ironic node-get-console didn't exist when I was doing this, so I wouldn't have hit that.20:04
matty_dubslinggao: No problem, thanks for picking this up! I'll be happy to help test/review as well.20:04
linggaomatty_dubs, that will be great!20:04
*** ilives has joined #openstack-ironic20:05
*** rwsu has quit IRC20:07
russell_hNobodyCam: thoughts on this? https://review.openstack.org/#/c/84814/20:08
*** rwsu has joined #openstack-ironic20:09
NobodyCamrloo: love the usually on line 75 of api_v1.rst (https://review.openstack.org/#/c/85107/1/doc/source/api_v1.rst)20:09
rlooNobodyCam: well, I am not familiar with all that code, so couldn't say for sure that it *always* :-)20:10
NobodyCam:)20:10
rlooalso, ClientException is deprecated -- I want to ask lucas about that...20:10
NobodyCamheheh20:10
*** rwsu has quit IRC20:11
*** rwsu has joined #openstack-ironic20:11
NobodyCamrussell_h: I like it.. Just gave it a quick look through..20:14
russell_hNobodyCam: enough to give it a +2?20:19
NobodyCamrussell_h: that one will need testing...20:20
NobodyCammy ci tests just started running so I'm watching the jenkins logs atm20:20
NobodyCambut I agree that hash ring is a big enough pice to have its own class :) so in theroy yes I do !!!20:21
russell_hNobodyCam: how do you feel about use of dbapi in ironic/common/20:22
russell_hI sort of feel like it oughtn't be there, but figured I'd do it the easy way and see how people feel20:23
NobodyCamthats one of the things i'd like to look closer at.. but a TODO: remove dbapi would cover my initial conserns20:24
devanandaback20:27
*** rwsu has quit IRC20:27
NobodyCamwb devananda20:27
devanandaadam_g: you mentioned a DuplicateOptError?20:28
*** rwsu has joined #openstack-ironic20:31
openstackgerritA change was merged to openstack/ironic-python-agent: Synced oslo modules  https://review.openstack.org/8505220:32
*** derekh has quit IRC20:35
*** rainya- has joined #openstack-ironic20:40
openstackgerritA change was merged to openstack/ironic-python-agent: Added some oslo modules  https://review.openstack.org/8503620:42
openstackgerritA change was merged to openstack/ironic-python-agent: Added some infrastructure tools  https://review.openstack.org/8503020:43
adam_gdevananda, yeah i believe that was the exception. use_syslog was the option, IIRC20:49
*** dkehn__ has joined #openstack-ironic20:50
*** dkehn_ has quit IRC20:52
*** dkehn__ is now known as dkehn_20:54
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Adding swift temp url support  https://review.openstack.org/8139120:57
*** romcheg has quit IRC20:58
openstackgerritJosh Gachnang proposed a change to openstack/ironic: Adding a reference driver for the agent  https://review.openstack.org/8479520:58
*** k4n0 has left #openstack-ironic20:59
*** matty_dubs is now known as matty_dubs|gone21:00
openstackgerritJosh Gachnang proposed a change to openstack/ironic-python-agent: Adding oslotest to improve debugging  https://review.openstack.org/8421221:03
openstackgerritA change was merged to openstack/ironic-python-agent: Adding oslotest to improve debugging  https://review.openstack.org/8421221:09
*** jdob has quit IRC21:10
*** linggao has quit IRC21:11
*** dwalleck has joined #openstack-ironic21:11
*** jbjohnso_ has quit IRC21:11
*** dkehn__ has joined #openstack-ironic21:15
*** dkehn__ has quit IRC21:16
*** dkehn__ has joined #openstack-ironic21:16
*** dkehn_ has quit IRC21:16
*** dwalleck has quit IRC21:20
*** rwsu has quit IRC21:22
*** martyntaylor has left #openstack-ironic21:22
*** dkehn_ has joined #openstack-ironic21:23
*** dkehn__ has quit IRC21:25
*** vkozhukalov has quit IRC21:30
NobodyCamwt... FATAL: Unable to delete script file /tmp/hudson5356793296994292431.sh21:32
NobodyCam2014-04-03 21:30:37.361 | hudson.util.IOException2: remote file operation failed: /tmp/hudson5356793296994292431.sh at hudson.remoting.Channel@179cda28:tripleo-precise-tripleo-test-cloud-34276121:32
*** dkehn__ has joined #openstack-ironic21:32
*** dkehn_ has quit IRC21:34
*** dkehn__ is now known as dkehn-x21:37
*** dwalleck has joined #openstack-ironic21:38
*** pcargnel has quit IRC21:40
openstackgerritDavid Shrewsbury proposed a change to openstack/ironic: Refactor nova.virt.ironic.driver get_host_stats  https://review.openstack.org/8385321:52
*** dwalleck has quit IRC21:55
NobodyCamoh :( jenkins just happens to have a break down running my test... hummm.... coincidence... :-p21:56
*** matty_dubs|gone is now known as matty_dubs21:56
ShrewsNobodyCam: you broke it! shame21:59
*** max_lobur has quit IRC22:00
NobodyCamno no really it just broke for other reasons22:00
ShrewsAlso, nova driver code makes me want to curl up in a corner and cry22:00
NobodyCam:-(22:00
NobodyCamShrews: we could write a google glasses app that would just block it from view.. you know put cute kitten pics or something like that over top of the code22:02
Shrews++22:03
ShrewsNobodyCam: So for 83853, it's ready for you to test again. I just -1'd it b/c I just realized that the driver unit test needs updated, but that shouldn't stop your testing.22:05
NobodyCamhumm /me now wounders if he could write a Gglasses app that would convert python code on the fly.. and display it in what ever lang the users wants22:05
Shrewsugh, and i missed the import22:07
Shrewsoh, no i didn't22:07
* Shrews needs beer22:07
NobodyCamShrews: ack... but so you know.. I know I have landed patches that I know broke the current nova driver test..22:08
JoshNangNobodyCam: you wouldn't want to do it for Glass. staring at that screen for more than about 10 seconds isn't very comfortable. But they do have an awesome translation app for non-programming languages that does in place text replacement. its amazing22:08
ShrewsNobodyCam: oh. with adam_g's recent work, i don't think we want to continue doing that. or he might get all stabby on us22:09
NobodyCamJoshNang: :)22:09
NobodyCamShrews: it was awhile ago22:09
NobodyCam*cough*22:09
adam_gShrews, we'll ideally have those running as checks in the test pipeline in one way or another22:10
NobodyCamShrews: http://www.rozmon.com/t_shirt_rozmon_canadian_bullet_proof_clothing.php22:13
NobodyCamalso good vs. atabbings22:13
NobodyCams/atabbings/stabbings/22:14
Shrewslol22:14
NobodyCamheheh22:14
devanandaadam_g: did you find a solution to that exception?22:15
adam_gdevananda, haven't, finsihing up this 2013.2.3 release. AFAIK oslo.config options are global and both nova and ironic use a use_syslog (among others, probably)22:16
adam_gdevananda, what did clark suggestion?22:17
devanandaadam_g: add http://tarballs.openstack.org/nova/nova-master.tar.gz#egg=nova22:19
devanandato test-requirements.txt22:19
devanandaadam_g: and put nova's unit tests in ironic/tests/nova/22:19
devanandabut this runs into that DuplicateOpt exc22:19
adam_gdevananda, right, that would be the easiest from a setup POV. i had tried something similar originally but hit those oslo issues22:19
*** dwalleck has joined #openstack-ironic22:20
adam_gmay be solvable in oslo.confg. i'm guessing it uses some dict thats key'd by option name.   illl take a look at that in a bit22:20
devanandaLOL22:20
devanandaso22:20
devanandaclarkb apapretly fileda  bug about this issue in oslo22:21
devanandawhich was promptly marked notabug22:21
adam_gha22:21
adam_g#?22:21
*** krtaylor has quit IRC22:22
devanandawas yeeeears ago, lol22:23
devananda*shrug*22:23
adam_goh ha22:24
adam_gill take a look at oslo.confg in a bit and get another bug filed that describes our use case. something tells me we are blazing trails here22:24
*** rwsu has joined #openstack-ironic22:26
*** dkehn-x has quit IRC22:36
devanandai'm not having any good luck here22:37
devanandayea22:37
devanandaadam_g: looks like the test discovery imports each module, which is triggering oslo.config to load the conf options into its global state22:52
devanandaadam_g: so putting nova unit tests under the path where ironic unit tests also live causes the test discovery to fail22:53
devanandaunless they're exlucded22:53
devanandaat least, that's our theory at the moment22:53
openstackgerritJosh Gachnang proposed a change to openstack/ironic-python-agent: Add BackOffLoopingCall with jitter  https://review.openstack.org/8430323:00
*** dwalleck has quit IRC23:00
adam_gdevananda, yeah23:05
adam_gtest_ironic_foo - > from ironic import foo -> oslo config opts registered23:05
adam_gtest_nova_foo - > from nova import foo -> oslo config opts registered23:05
*** matsuhashi has joined #openstack-ironic23:05
adam_ggonna start poking at that now.. had to eat something.23:06
adam_ghorrible portland pizza23:06
*** dkehn has joined #openstack-ironic23:06
dhellmannwhat's the issue with the config options?23:06
*** matty_dubs is now known as matty_dubs|gone23:07
*** dkehn is now known as dkehnx23:07
adam_gdhellmann, testr importing all tests during discovery ends up importing both nova and ironic, causing conflicts on options registered by both (eg, use_syslog)23:08
dhellmannthe ironic tests are loading nova code?23:09
* NobodyCam knew that was comming23:09
NobodyCamdhellmann: we have the ironic nova driver in our tree23:09
dhellmannceilometer has something similar, we had to put them in a separate test directory23:10
dhellmannhttp://git.openstack.org/cgit/openstack/ceilometer/tree/nova_tests/test_notifier.py23:10
dhellmannNobodyCam: I'm not judging, just checking that I understand :-)23:10
NobodyCamhehe ya :)23:10
NobodyCamits late in hte day23:10
adam_ghttp://paste.ubuntu.com/7200995/23:10
NobodyCamthe even23:10
adam_gdhellmann, they currently live in separate directories, but testr is going to import everything it discovers23:11
dhellmannadam_g: run 2 testr jobs, one for each directory23:11
dhellmannhmm, it appears ceilometer is no longer testing that nova notifier plugin at all, probably for a similar reason :-(23:12
adam_gdhellmann, tests get run via setup.py testr23:12
dhellmannadam_g: tests are run by "tox -e something" so you have control over that23:13
dhellmannand testr can be told which directory to look at -- it doesn't have to start with "."23:13
openstackgerritMichael Davies proposed a change to openstack/ironic: Handling validation in conductor consistently  https://review.openstack.org/8437423:18
*** jgrimm has quit IRC23:18
NobodyCamafternoon walkies...bbiafm23:27
devanandadhellmann: adam_g: sounds like you two are thinking along similar lines23:28
devanandaadam_g: the diff here is that I think we can add nova-latest-tarball (and mox) as test-requirements, then do something like what we've been discussing23:29
adam_gdevananda, maybe. banging on testr/subunit now23:29
devanandayea. i banged on it for a while between meetings too, didn't get past the DuplciateOpt error though23:30
adam_gdevananda, yeah--it would be ideal if we didn't have to change anything in gate and can just rely on jenkins calling tox -e py27/py26 and having the nova tests run23:30
devanandaso if we can't, then your current patch is good23:30
devanandait's fine if we need to have jenkins call a separate test23:30
dhellmanntox can run more than one command in the same env23:30
devanandabut when i showed teh bash script to clark, he was not happy23:30
devanandaI gotta run again .... back later23:31
adam_gdhellmann, AFAICS the starting dir for test discovery is defined in .testr.conf's test_command23:31
NobodyCamenjoy devananda23:31
dhellmannadam_g: you can pass it on the command line, too23:31
adam_gdhellmann, cool. will take a closer look when venv is done rebuilding23:31
dhellmannadam_g: oslo-incubator calls testr more than once, but not with a different directory23:32
dhellmannadam_g: hmm, I might be wrong about the test dir thing23:34
adam_gdhellmann, yeah, looking as olso its using regex23:34
dhellmannyeah23:34
adam_gdhellmann, which doenst help here--testr is going to import everythign before applying that regex23:34
dhellmanntry looking through ceilometer's history to see where they removed that test job -- jd__ or gordc may be able to help with that23:34
adam_gdhellmann, cool.23:35
dhellmannwhy are the config options clashing? are the defaults different?23:35
adam_gdhellmann, https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L50223:38
adam_gthe ironic code was originally a copy from nova, so it makes sense they both use 'use-syslog'23:39
dhellmannadam_g: that actually checks more than the name23:39
dhellmannit compares the 2 option objects, so if their settings are the same you don't get an error23:39
dhellmannthe error should be updated to complain about the difference23:39
adam_gright23:39
dhellmannso having 2 options with the same name is fine, as long as all of their other settings are the same23:40
dhellmannwhat is different between the nova and ironic settings?23:40
dhellmanncould it be the help string, for example?23:40
adam_gnot sure off the top of my head23:40
dhellmannadam_g: figuring that out might be the shorter path to make it work23:41
*** eghobo has quit IRC23:42
adam_gdhellmann, hmph looks like they're actually clashing on help messages23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!