Friday, 2014-09-05

adam_gjroll, oh cool. does that require some changes elsewhere to make those files available on tarballs.o.o?00:02
jrolladam_g: already landed00:02
jrolladam_g: this came out of sdague not liking us unpacking the tarball and whantot00:02
adam_gjroll, weird, okay00:03
jrolladam_g: so once that and another change land... need infra/config changes to do CI on IPA00:03
jrollwhich I've based off your refactor00:04
adam_gyeah, let me ping them now about that00:04
jrollidk if you have time to babysit that or want me to take it00:04
jrolloh, cool :)00:04
jrollhappy to help push that through if needed00:04
adam_gmaybe if you peste^Wping them too it will help :)00:04
jrollheh, I've pinged them enough today :P00:05
jrollbut happy to do so tomorrow :D00:05
openstackgerritChris Krelle proposed a change to openstack/ironic: Draft a 'pxe_logpower' driver  https://review.openstack.org/9895000:07
*** romcheg1 has quit IRC00:08
openstackgerritJosh Gachnang proposed a change to openstack/ironic-python-agent: Add versioning to Agent decommission  https://review.openstack.org/10685900:09
openstackgerritJosh Gachnang proposed a change to openstack/ironic-python-agent: Adding support for decommissioning  https://review.openstack.org/10437900:09
Haomengmrda: night:)00:13
mrdanight Haomeng!00:14
mrdayou must be up late?00:14
Haomengmy timezone is +8, now it is 8am:)00:17
Haomengguess you are in night:)00:17
mrdaHaomeng: It's 10am here for me00:19
Haomengmrda: :)00:20
mrdaI'm +9.500:20
Haomengmrda: :)00:20
*** igordcard has quit IRC00:24
*** rloo has quit IRC00:28
*** bandicot has quit IRC00:29
*** Ugallu has quit IRC00:46
*** chuckC has quit IRC00:50
*** GheRivero has quit IRC00:50
*** GheRivero has joined #openstack-ironic00:51
*** killer_prince is now known as lazy_prince01:29
lazy_princehi all.. pls take a bit of time to review https://review.openstack.org/#/c/103483/ and provide your valuable comments..01:38
*** rloo has joined #openstack-ironic01:42
*** nosnos has joined #openstack-ironic01:49
rloomrda: did you see https://review.openstack.org/#/c/111428/ failed pep801:49
mrdayes01:49
mrdarloo: am prepping a new fix and uploading now01:49
mrdasigh01:49
mrdathank you for noticing01:49
rloothx mrda.01:50
rloojust missing the from nova.i18n import _ line.01:50
rlootalk to you next week mrda. I'm signing out now. (yay)01:50
mrdahave fun rloo!01:51
*** rloo has quit IRC01:51
mrdawow, 111428 fails in check, but doesn't fail locally.  I for one welcome our new Heisenbug overlords.01:55
*** ryanpetrello has quit IRC01:56
*** ryanpetrello has joined #openstack-ironic02:00
*** eghobo has quit IRC02:11
*** harlowja is now known as harlowja_away02:27
*** lazy_prince is now known as killer_prince02:28
*** dkehnx has quit IRC02:37
*** dlaube has quit IRC02:49
*** eghobo has joined #openstack-ironic02:51
*** zz_naotok is now known as naotok03:03
*** vinbs has joined #openstack-ironic03:09
*** eghobo has quit IRC03:10
*** dlaube has joined #openstack-ironic03:22
*** dlaube has quit IRC03:23
*** wanyen has quit IRC03:28
*** mgoddard has quit IRC03:28
*** eghobo has joined #openstack-ironic03:45
*** pcrews has quit IRC04:01
*** Poornima has joined #openstack-ironic04:23
*** pcrews has joined #openstack-ironic04:41
*** rushiagr_away is now known as rushiagr04:47
*** pcrews has quit IRC04:48
*** kevinbenton has quit IRC05:03
*** kevinbenton has joined #openstack-ironic05:06
*** krtaylor has quit IRC05:12
*** nikunj2513 has joined #openstack-ironic05:15
*** rakesh_hs has joined #openstack-ironic05:20
*** krtaylor has joined #openstack-ironic05:20
*** killer_prince is now known as lazy_prince05:50
*** yuanying has joined #openstack-ironic05:59
*** yuanying_ has quit IRC05:59
openstackgerritOpenStack Proposal Bot proposed a change to openstack/ironic: Imported Translations from Transifex  https://review.openstack.org/11854806:08
*** k4n0 has joined #openstack-ironic06:25
*** Haomeng has quit IRC06:27
*** Haomeng has joined #openstack-ironic06:30
*** nikunj2513 has quit IRC06:34
*** nikunj2513 has joined #openstack-ironic06:36
*** eghobo has quit IRC06:37
*** bluex-pl has joined #openstack-ironic07:08
*** athomas has joined #openstack-ironic07:08
*** jistr has joined #openstack-ironic07:15
*** chenglch has joined #openstack-ironic07:27
*** bluex-pl has quit IRC07:30
*** jcoufal has joined #openstack-ironic07:32
*** igordcard has joined #openstack-ironic07:44
*** andreykurilin_ has joined #openstack-ironic07:46
*** ifarkas has joined #openstack-ironic07:47
*** derekh has joined #openstack-ironic07:54
*** agordeev has quit IRC07:58
*** naotok is now known as zz_naotok07:58
*** agordeev has joined #openstack-ironic08:05
*** ndipanov has joined #openstack-ironic08:07
*** igordcard has quit IRC08:11
*** dtantsur|afk is now known as dtantsur08:15
dtantsurMorning Ironic08:15
mrdaHey dtantsur08:19
*** romcheg has quit IRC08:25
dtantsurmrda, g'evening)08:29
*** romcheg has joined #openstack-ironic08:29
*** lucasagomes has joined #openstack-ironic08:31
yuriyzmorning Ironic dtantsur mrda08:34
dtantsuryuriyz, morning08:34
*** derekh has quit IRC08:37
mrdahi yuriyz romcheg08:39
lucasagomesmorning dtantsur mrda yuanying08:46
lucasagomesouch08:46
lucasagomesyuriyz,08:46
*** yongli has quit IRC08:48
openstackgerritLucas Alvares Gomes proposed a change to openstack/ironic: Remove unnecessary mapping from Agent drivers  https://review.openstack.org/11832608:54
dtantsurlucasagomes, morning. can we close ironic part of https://bugs.launchpad.net/ironic/+bug/1351010actually?08:54
lucasagomesdtantsur, +108:54
dtantsurok08:54
lucasagomesdtantsur, I don't think we have anything else to do apart from doc it08:54
*** bluex-pl has joined #openstack-ironic08:55
*** andreykurilin_ has quit IRC09:00
openstackgerritA change was merged to openstack/ironic: Use models.TimestampMixin from oslo.db  https://review.openstack.org/11919509:05
*** Haomeng|2 has joined #openstack-ironic09:07
*** Haomeng has quit IRC09:08
*** dtantsur is now known as dtantsur|brb09:10
mrdaNight all, see you Monday09:12
*** k4n0 has quit IRC09:16
*** MattMan has joined #openstack-ironic09:18
*** nikunj2513 has quit IRC09:24
*** igordcard has joined #openstack-ironic09:27
openstackgerritA change was merged to openstack/ironic: mock.assert_called_once() is not valid method  https://review.openstack.org/11923409:29
*** zz_naotok is now known as naotok09:31
*** bluex-pl has quit IRC09:31
*** GheRivero is now known as GheRivero_pto09:34
*** bluex-pl has joined #openstack-ironic09:37
*** pelix has joined #openstack-ironic09:39
openstackgerritYuriy Zveryanskyy proposed a change to openstack/ironic: Log ERROR power state in node_power_action()  https://review.openstack.org/11749909:41
*** bluex-pl has quit IRC09:41
*** bluex-pl has joined #openstack-ironic09:42
openstackgerritVinay B S proposed a change to openstack/ironic: Adds instructions for deploying instances on real hardware  https://review.openstack.org/11861409:45
*** chenglch has quit IRC09:54
*** vdrok_afk is now known as vdrok10:08
*** jcoufal has quit IRC10:16
*** derekh has joined #openstack-ironic10:23
*** bluex-pl has quit IRC10:26
lucasagomesyuriyz, should we tag bug 1358816 to RC1?10:37
yuriyzlucasagomes, IMO we should10:38
*** jcoufal has joined #openstack-ironic10:41
lucasagomesyuriyz, ack, will tag it10:43
*** Haomeng has joined #openstack-ironic10:47
*** Haomeng|2 has quit IRC10:47
*** dtantsur|brb is now known as dtantsur11:01
*** bluex-pl has joined #openstack-ironic11:04
*** bluex-pl has quit IRC11:09
*** bluex-pl has joined #openstack-ironic11:10
romchegMorning guys11:12
romchegI'm not very familiar with this internals of hashing. Could someone please explain me why big endian is chosen here? https://review.openstack.org/#/c/118932/4/ironic/common/hash_ring.py11:14
romchegIt's on L8511:14
*** teju has joined #openstack-ironic11:20
tejuhi all... need help with boot provisioning using ironic....the provisioning state is stuck in wait call-back11:22
romchegteju: For how long?11:24
romchegteju: Hi, btw :)11:24
tejuromcheg : it's been more than 20 min11:25
romchegteju: Is tftp server up and runnign?11:25
tejuromcheg : yes11:25
openstackgerritA change was merged to openstack/ironic: Unbreak debugging via testr  https://review.openstack.org/11889611:25
romchegteju: What about DHCP, can you see anything in the logs?11:25
tejuromcheg : when i ran the command to deploy, the baremetal server is powered ON, but it failed to get boot image ....i get error pxe e5311:26
romchegteju: Could you please search your syslog to find something like DHCPDISCOVER from your node?11:27
openstackgerritVladyslav Drok proposed a change to openstack/ironic: Translator functions cleanup part 1  https://review.openstack.org/11630311:29
openstackgerritVladyslav Drok proposed a change to openstack/ironic: Translator functions cleanup part 2  https://review.openstack.org/11858711:29
openstackgerritVladyslav Drok proposed a change to openstack/ironic: Translator functions cleanup part 3  https://review.openstack.org/11858811:30
romchegteju: AFAIR pxe e53 may be related to wrong configuration of the DHCP server11:30
openstackgerritVladyslav Drok proposed a change to openstack/ironic: Translator functions cleanup part 4  https://review.openstack.org/11859111:30
tejuromcheg : i am running dnsmasq11:31
*** Poornima has quit IRC11:31
romchegteju: It would be useful to run "neutron port-show <your_port_uuid>" to check the PXE configuration11:31
*** Poornima has joined #openstack-ironic11:32
tejuromcheg : i followed this link : http://ma.ttwagner.com/bare-metal-deploys-with-devstack-and-ironic/#comment-257811:34
openstackgerritYuriy Zveryanskyy proposed a change to openstack/ironic: Log ERROR power state in node_power_action()  https://review.openstack.org/11749911:37
*** vinbs has quit IRC11:38
dtantsurteju, "NOTE: This guide is fairly dated (originally authored in January)." ;)11:39
dtantsurMatt started updating it, but I doubt he finished11:39
*** lucasagomes is now known as lucas-hungry11:40
tejudtantsur : ok....when i ran the curl command mentioned in that link, the server is powered ON, but it failed to get boot image.....may be i am missing some conf11:41
tejuromcheg : http://pastebin.com/f2bnGhDv11:43
romchegteju: So I see that DHCP seems to be accessible for the node (that happened to be a problem in the past). Could you please show the detailed neutron port info?11:44
*** naotok is now known as zz_naotok11:46
*** Haomeng|2 has joined #openstack-ironic11:46
romchegyou can find the Neutron vif _id in the Ironic Port extra specs11:46
*** Haomeng has quit IRC11:48
tejuromcheg : i dont know whether I am making sense, but i am trying to boot provision without neutron...will this work?  i mean, neutron is just for providing IP for my bare metal server...instead I am using dnsmasq for dhcp11:48
romchegteju: Ah, makes sense11:48
tejuromcheg : that should work...right?11:49
romchegteju: than something has to set PXE options in the DHCP settings11:49
romcheglooks like the node ain't getting them11:50
tejuromcheg : yes....provisioning state change from wait call back to deploy failed11:50
romchegteju: because of the timeout, I believe11:51
tejuromcheg : one basic question....pls dont mind,,,, do I need any physical connection from bare metal server to openstack server?11:52
romchegteju: So the node has to be able to connect both to the DHCP server and to the conductor11:54
romchegteju: and to the ironic api too11:54
*** k4n0 has joined #openstack-ironic11:55
romchegteju: and of course to the tftp server too11:56
tejuromcheg : yes....this gets complicated bcoz i am running all openstack services(keystone,glance,nova,neutron,ironic) in a VM11:57
tejuromcheg : and I am trying to boot a real bare metal server11:57
tejuromcheg : thanks for ur time....i will try to fix dhcp issues11:58
romchegteju: That is not very complicated, I have the same kind of setup on my lab (the only difference is that I use neutron for configuring dhcp)11:58
romchegteju: basically you can add another interface to the VM to the same network your bm node is11:58
romchegteju: and set up all required PXE params to the dnsmasq12:00
tejuromcheg : or I can run dhcp server in a physical server rather than in VM and connect this server to bare metal server...this may work...right?12:01
*** openstackgerrit has quit IRC12:01
*** openstackgerrit has joined #openstack-ironic12:03
romchegteju: node still has to have access to the tftp server and to the ironic api to send the callback12:04
tejuromcheg : yes, once the bare metal node gets IP from dhcp server, it will be able to talk to all openstack services, tftp,12:06
tejuromcheg : bcoz all the IPs for physical server and VMs are in the same subnet12:07
romchegteju: then it should work12:08
tejuromcheg : once again...thanks... i will try to get pxe boot work12:09
romchegteju: you're always welcome ;)12:09
openstackgerritDerek Higgins proposed a change to openstack/ironic: Revert "Nova review updates for _node_resource"  https://review.openstack.org/11935712:11
tejuNo VIFs found for node f8908f2c-2abf-4ec5-b25c-726793f4983b when attempting to update Neutron DHCP BOOT options.12:11
derekhTripleo Ironic CI tests are failing : bug 1365750 : the revert ^^ seems to fix it12:11
dtantsurderekh, hmm.. this change was requested during Nova review, I guess you want to talk to nova folks...12:12
dtantsurmrda, lucas-hungry ^^^12:12
*** teju has left #openstack-ironic12:14
derekhdtantsur: hmm, maybe it depends on a nova patch thats yet to merge ?12:14
dtantsurIIRC yes12:14
dtantsurdon't hav e link at hand12:14
*** bluex-pl has quit IRC12:18
*** nosnos has quit IRC12:19
*** nosnos has joined #openstack-ironic12:20
*** nosnos has quit IRC12:24
*** bluex-pl has joined #openstack-ironic12:24
*** Poornima has quit IRC12:26
*** rakesh_hs has quit IRC12:34
openstackgerritDmitry Tantsur proposed a change to openstack/python-ironicclient: Add sort_key and sort_dir parameters to *-list  https://review.openstack.org/11455512:36
*** bigjools has quit IRC12:38
*** bigjools has joined #openstack-ironic12:40
*** bigjools has joined #openstack-ironic12:40
lucas-hungryderekh, uuu, yup... actually that was something they required us to have12:45
lucas-hungryto canonicalize the arch12:45
*** lucas-hungry is now known as lucasagomes12:45
*** viktors|afk is now known as viktors12:45
viktorsdtantsur: hi!12:46
lucasagomesI think it's on part 3 patch set 1412:46
dtantsurviktors, hi12:46
lucasagomesthis discussion started12:46
lucasagomesderekh, will grab the link12:46
viktorsdtantsur: does someone suppose to fix bug 1347604 (ironic-dbsync fails on SQLite due to bad support of migrations) ? )12:47
derekhlucasagomes: ok, we've reverted it in tripleo CI, I'll remove the revert once its merged12:47
lucasagomesderekh, https://review.openstack.org/#/c/111425/14/nova/virt/ironic/driver.py12:47
dtantsurviktors, depending on definition of "fix". I'll be happy to close it, once we fix the docs12:47
lucasagomesderekh, it was just approved in nova12:47
derekhlucasagomes: back in a few minutes, gotta get of train12:47
derekhlucasagomes: ok, will try it in a few minutes12:48
lucasagomesderekh, ack, right yeah we have to sync about it and find a consensus12:48
romcheglucasagomes, dtantsur, could you please check out my question about endianess ^12:48
lucasagomesromcheg, sorry didn't see that12:48
lucasagomeslemme see12:48
*** derekh has quit IRC12:48
romcheglucasagomes: thanks!12:49
viktorsdtantsur: yes, I mean documentation fixes. My English is far from perfect, so  I'll be glad, if someone with good english will fix the docs :)12:49
dtantsurviktors, then unassign yourself with the comment and someone will pick it12:50
viktorsdtantsur: ok, will do12:50
lucasagomesromcheg, I think ur comment is accurate. I'm not sure why big endian, but I don't think if that makes any difference too12:51
lucasagomesromcheg, I know where that came from tho, I've the article somewhere about the consitent hashing ring12:52
lucasagomeswhere we based ours on12:52
lucasagomeswill grab the link12:52
romcheglucasagomes: let's wait for lifeless then :)12:52
lucasagomes(if I find)12:52
lucasagomesromcheg, we may find the answer in the article, out of the top of my head I don't remmeber12:52
lucasagomesromcheg, here we go http://greg.brim.net/page/building_a_consistent_hashing_ring.html12:53
romcheglucasagomes: thanks!12:53
*** Haomeng|2 has quit IRC12:54
romcheglucasagomes: at least I can see they use big endian too :D12:55
lucasagomesromcheg, heh yeah12:57
*** Haomeng|2 has joined #openstack-ironic12:57
* romcheg starts suspecting someone did copy&paste12:57
jbjohnso__well I just reviewed redfish specification and gave feedback... I frankly don't see it working in the intended role very well13:10
*** derekh has joined #openstack-ironic13:11
*** pelix has quit IRC13:13
*** Haomeng|2 has quit IRC13:25
*** Haomeng|2 has joined #openstack-ironic13:27
*** pcrews has joined #openstack-ironic13:33
*** dhellmann is now known as dhellmann_13:40
*** bigjools has quit IRC13:42
rushiagrlucasagomes: hi.. I have resubmitted the patch https://review.openstack.org/#/c/118554/ (Move metadata filtering logic to utils.py), incorporating your comments. Please have a relook when you have time13:44
rushiagrlucasagomes: and thanks for the review :)13:44
openstackgerritDmitry Tantsur proposed a change to openstack/ironic: EXPERIMENTAL Implement hardware discovery setup in PXE driver  https://review.openstack.org/11003113:44
openstackgerritDmitry Tantsur proposed a change to openstack/ironic: Add newly_discovered column to Node object  https://review.openstack.org/10738913:44
*** bigjools has joined #openstack-ironic13:45
* lucasagomes feels like he should go to #openstack-nova13:45
lucasagomesrushiagr, sure will do! thanks for addressing it13:45
rushiagrlucasagomes: heh13:45
*** r-daneel has joined #openstack-ironic13:53
*** SpamapS has quit IRC13:59
mariosquick q: when talking about actual bare metal machines and discovery, will we still create an entry in Ironic for any new machines (populating only ipmi creds i guess). i.e. same as for the virt case? will we still need to use MAC to match against discovered info?13:59
marios(ok that was 2 questions)14:00
*** SpamapS_ has joined #openstack-ironic14:02
*** datajerk has joined #openstack-ironic14:03
dtantsurmarios, if you mwan what I'm prototyping (there's no official discovery in Ironic), then 1. yes, 2. no14:03
dtantsur* mean actually14:03
mariosdtantsur: thanks! (just noticed those reviews earlier in the chat, do they answer my questions?)14:04
dtantsurmarios, that' my code, but it's in the middle of refactoring14:05
mariosdtantsur: i guess that means the 'maintenance mode' could/will be used for baremetal (at least right now)14:05
dtantsurmarios, likely will14:06
mariosdtantsur: am asking because i'm converting the dnsmasq script to element, so we can change the logic there14:06
mariosdtantsur: k thanks matey14:06
mariosdtantsur: (one last thing, will be easier once i get a review out and we can discuss specifics, but): doing it this way is quite expensive (to get a list of MAC)... get nodes where maintenance=False (OK), but then for each get ports (expensive)14:07
mariosdtantsur: so wanted to check if there would be difference in baremetal case that might make it better in this respect14:08
dtantsurmarios, for real thing (not virt) you don't have to check for maintenance, just black-list everything in ports14:08
mariosdtantsur: I C. NO MAC!14:08
mariosdtantsur: yes, there will be no longer any need to match with the MAC right14:09
mariosdtantsur: so ironic entry won't have a mac (or port? or port without mac, whatever) for new nodes14:09
dtantsurright. at least that's how I see it14:09
mariosdtantsur: yes seems right. which is great. (now just need to be able to distinguish if we are talking about vms or bm and am in business, as the script will now depend)14:10
mariosdtantsur: thanks again very helpful14:12
dtantsurnp)14:12
*** ndipanov is now known as ndipanoff14:16
*** Haomeng|2 has quit IRC14:18
jrollgood morning ironic :)14:22
romcheglucasagomes: in your last comment on 117499 you said that you are approving the patch but I don't see +A. Didn't you miss it?14:26
*** Haomeng|2 has joined #openstack-ironic14:27
romchegMorning jroll!14:27
lucasagomesromcheg, oh it it was patch set 714:27
lucasagomesyuriyz, posted a new one rebasing it14:27
lucasagomeswill re-approve14:27
lucasagomesdone14:28
romchegHmm, gerrit can apply old scores after a rebase again :)14:28
Shrewsmordred: ooh, dox sounds interesting14:30
mordredShrews: it needs a little bit more work...14:31
jrollromcheg: morning :)14:31
jrolland lucasagomes and Shrews and mordred and dtantsur :)14:31
mordredmorning jroll14:31
Shrewsmordred: interestingly, i had a very similar idea a month or so back (using docker for tests)14:32
Shrewsjroll: morning14:32
dtantsurjroll, Shrews, mordred, morning14:32
Shrewshey dtantsur o/14:33
jrollShrews: seen something like this before? http://logs.openstack.org/65/119165/3/check/check-tempest-dsvm-virtual-ironic/3720214/logs/screen-ir-api.txt.gz?level=WARNING14:33
jrollall sorts of weird port manipulation problems14:33
mordredShrews: it makes TOTAL sense for things like unittests/locally for the dev14:33
mordredShrews: especially since it's got the speed and caching thing sorted14:33
Shrewsmordred: exactly. i had it in mind for running ironic scenario tests locally instead of in tempest14:33
mordredShrews: it's too bad iscsi isn't namespaced...14:33
Shrewsjroll: looking14:34
Shrewsjroll: O.o14:34
Shrewsjroll: is this happening in several reviews, or just this one?14:35
jrollShrews: just this one afaik?14:36
jrollI might just recheck it14:36
jrollmordred: maybe we should replace iscsi from ironic :)14:36
jrollowait14:36
jrolldoing that already14:36
jrollunfortunately, I don't think it will actually go away :/14:37
NobodyCamGood morning Ironic14:39
NobodyCamTGIF!14:39
jrollmorning NobodyCam, TGIF indeed14:40
NobodyCammorning mordred :)14:40
NobodyCammorning jroll :)14:40
jrollShrews: rechecking14:41
Shrewsjroll: this is really odd. checking for that first chassis id in ir-cond shows nothing. it's like the wrong api server was being used or something14:42
jrollShrews: it's like openstack doesn't work or something14:43
*** dhellmann_ is now known as dhellmann14:43
Shrewsjroll: lol. the first node though (38f3eb1a) was actually successfully deleted according to ir-cond. so weird14:43
jrollyeah. completely lost.14:44
lucasagomesjroll, yo morning14:45
jrollheya :)14:46
Shrewsjroll: so, these are normal14:50
Shrewsjroll: the tempest test deletes the node, then attempts to get it again to make sure that NotFound is raised14:50
jrollShrews: I was looking at this weirdness: http://logs.openstack.org/65/119165/3/check/check-tempest-dsvm-virtual-ironic/3720214/logs/screen-ir-api.txt.gz?level=WARNING#_2014-09-05_04_17_44_21414:51
jrolland the couple lines after14:51
jrollShrews: and, the test failed, to be clear14:51
Shrewsjroll: yeah. looking at that one now (the earlier ones left me confused, though)14:51
Shrewsthe *real* error is in the baremetal scenario test: Details: Timed out waiting to get Ironic node by instance id 33822cbf-59d5-4870-acad-44bde8961f8714:53
jrollright14:54
Shrewsthat i've seen before, though rarely14:55
* Shrews blames a slow node14:55
*** k4n0 has quit IRC14:57
jrollwe're only 2 +A votes away from nova driver landing :)15:01
NobodyCamwoo hoo15:01
*** bluex-pl has quit IRC15:05
*** Guest18621 is now known as annegentle15:16
*** pelix has joined #openstack-ironic15:17
*** datajerk has quit IRC15:17
*** vdrok is now known as vdrok_afk15:24
dtantsurNobodyCam, morning and TGIF15:32
openstackgerritDmitry Tantsur proposed a change to openstack/ironic: EXPERIMENTAL Implement hardware discovery setup in PXE driver  https://review.openstack.org/11003115:33
dtantsuractually calling it a day for today :) have a nice weekend15:36
*** dtantsur is now known as dtantsur|afk15:36
NobodyCammorning dtantsur|afk15:37
NobodyCamhave a good weekend dtantsur|afk15:38
dtantsur|afk:)15:38
NobodyCamjust some general fyi. I will be not online next Friday!15:40
*** slagle_ is now known as slagle15:41
lazy_princeJoshNang: I am trying to use https://github.com/rackerlabs/ironic-neutron-plugin15:42
lazy_princeJoshNang: As per https://github.com/rackerlabs/ironic-neutron-plugin/blob/master/etc/neutron.conf.dist, i am using mechanism_drivers = ironic15:43
lazy_princeJoshNang: but it throws error...15:44
jrollmorgabra: ^^15:44
lucasagomesjrist, w00t15:47
lucasagomesmorning NobodyCam15:47
lucasagomesdtantsur|afk, g'night15:47
jrolllazy_prince: morgabra might know more about that, I could help you later today too15:47
jristI need to change my nick, don't I jroll, lucasagomes ?15:48
jrollhehehe, somebody does15:48
lucasagomesjrist, oh gosh again15:48
lucasagomeslol15:49
jrollOMG15:49
jrolloh15:49
jrollthat one was intentional :P15:49
lucasagomesjrist, how are u doing :)15:49
lucasagomesyeah that was intentional jroll :D15:49
jristI was born in 1981, so can I lay claim, jroll?15:49
jrolldamn.15:49
jristalso mine is alphabetically in front15:49
jristlucasagomes: great, you?15:49
* jroll is a youngin, I guess15:49
*** jistr has quit IRC15:49
lucasagomesjrist, all is well :D15:49
jristhaha15:50
jristmaking me feel old15:50
NobodyCam81 gah /me is old15:52
jrollI won't tell you when I was born, then :P15:52
* NobodyCam was born before 1970 :-p15:52
lucasagomesheh15:55
lucasagomes89 FTW!15:55
NobodyCamgah /me gets his cain and hearing aid and trundles away15:56
jrollyay, I'm not the youngest15:56
jrollLOL15:56
NobodyCamlol15:56
lucasagomes;(15:57
NobodyCamhehehe15:57
jrollthis passed jenkins now15:57
jrollhttps://review.openstack.org/#/c/118693/15:57
jrollit's already in nova, one of y'all want to approve it?15:57
lazy_princemorgabra: here is the complete log in case if you need it. http://paste.openstack.org/show/106614/15:58
jrolloooo colors15:58
NobodyCamjroll: landing15:59
jrolllazy_prince: to be clear, you pip install'd this plugin, right?15:59
jrollNobodyCam: thanks!15:59
*** dlaube has joined #openstack-ironic15:59
lazy_princenope.. i pulled it from github16:00
lazy_princejroll: devstack did that for me16:00
jrollare you sure devstack pip installed it?16:01
jrollyou need to do that16:01
lazy_princeactually devstack did setup neutron for me and then i stopped it and used the instructions from https://github.com/rackerlabs/ironic-neutron-plugin16:01
jrollor python setup.py install16:01
jrollor whatever16:01
jrollto load the entrypoints16:01
jrollexport PYTHONPATH=$PYTHONPATH:/path/to/ironic-neutron-plugin16:02
jrollmight also work, idk if you did that16:02
lazy_princei am sorry.. which plugin, ironic-neutron-plugin (neutron plugin), i copied it manually after stopping neutron service to neutron directory..16:03
jrollyeah16:03
jrollyou need to cd to ironic-neutron-plugin16:03
jrollpython setup.py install16:03
jrollthat's how the entrypoint in setup.cfg get loaded, so that stevedore knows about them16:04
lazy_princeokay let me do that..16:04
jrolllazy_prince: what sort of switch will you be using this with?16:04
lazy_princeI will be using HP switch..16:05
jrollalso... we still need to upstream our code that uses this16:05
jrollok, you'll likely have to write a driver, then, too16:05
jrollin here: https://github.com/rackerlabs/ironic-neutron-plugin/tree/master/ironic_neutron_plugin/drivers16:05
lazy_princeI have finished the driver code for it16:05
jrollneat :)16:05
lazy_princeand wanted to test end-to-end.. thats where i am getting into issues..16:06
jrollright16:06
jrollthe upstreaming part of this is going to be interesting16:07
lazy_princepython setup.py install failed with error about EnvironmentError: mysql_config not found16:07
jrollthere's other people basically writing this in neutron16:07
jrollapt-get install libmysqlclient-dev16:07
*** igordcard has quit IRC16:07
jroll(strangely, that should already be installed in devstack, I think)16:08
lazy_princewell.. i am trying to see if it works for me.. it is not meant for production.. when the support comes from neutron, we will use that..16:08
jrollright, ok16:08
jrollwould you be interested in our current code that interacts with this?16:09
NobodyCamlol - this is cute: https://www.youtube.com/watch?v=YoB8t0B4jx4&t=3516:09
jrollwe still haven't put it in gerrit, because it likely won't land as-is16:09
lazy_princeyup..16:09
jrollok, I'll try to get that up next week16:10
lucasagomesNobodyCam, lol16:10
lucasagomespoor dog hah16:10
NobodyCamhahaha16:10
jrolllol that's awesome16:11
* jroll goes shopping for one16:11
NobodyCam:)16:11
NobodyCamlooks like they had a lot of fun with it16:11
lazy_princejroll: after running 'apt-get install libmysqlclient-dev' followed by 'python setup.py install' worked16:12
jrollcool :)16:12
* jroll goes afk for a while16:12
lazy_princethanks for the guidance.. now will start testing..16:12
jrollbbiab16:12
lazy_princejroll: and will wait for the current code16:13
jrollok :)16:13
ShrewsNobodyCam: awww. that poor puppy16:14
NobodyCam[s]he seems to be having fun16:15
Shrewsor will be left with an inferiority complex thinking no one likes her16:16
jrolllazy_prince: this is the very short version of how onmetal does it: https://gist.github.com/jimrollenhagen/9f80ba59d2a031a130f816:16
*** vinbs has joined #openstack-ironic16:17
jrollbbl16:18
*** mordred has quit IRC16:18
*** mordred has joined #openstack-ironic16:18
*** vinbs has quit IRC16:22
lazy_princejroll: i am doing something similar.. i pulled it from the ironic patch written by JoshNag. thanks to him..16:24
openstackgerritChris Krelle proposed a change to openstack/ironic: Proxy Nova baremetal commands to Ironic  https://review.openstack.org/11631616:24
JayFlazy_prince: JoshNang works with both myself and jroll, so that's the same code pretty much :P16:24
lazy_princeJayF: something failed with neutron while running devstack. let me c..16:26
lazy_princehere is the relevant devstack log http://paste.openstack.org/show/106629/16:29
*** ndipanoff has quit IRC16:30
lucasagomesaight I will call it a day16:36
lucasagomeshave a good night everybody!16:36
lucasagomesenjoy the weekend16:36
NobodyCamhave a good weekend lucasagomes16:36
*** lucasagomes is now known as lucas-dinner16:37
morgabralazy_prince: hey, sorry. Did you run setup.py at all?16:38
*** eghobo has joined #openstack-ironic16:38
morgabralazy_prince: try 'python setup.py develop'16:38
lazy_princehi morgabra16:38
morgabrait builds the config file that stevedore reads for drivers16:38
lazy_princemorgabra: i did python setup.py install.. and it solved my problem.16:39
morgabracool, sorry for the late reply. hit me up if you run into anything else16:39
lazy_princemorgabra: jroll helped me get past. but i have another problem.. http://paste.openstack.org/show/106629/16:40
lazy_princemorgabra: well.. in another 20mins i will call it a day.. till then i will continue to bug you..16:41
morgabraneutron router-interface-add 8f96ab0c-da3d-49f0-8bfd-a1caa697fd0b 6380f776-ba25-4e4a-83a7-8c961ddc5eec fails, hmm16:41
lazy_princehave you seen this before..?16:42
* NobodyCam does a quick run to starbuck..bbiaf16:42
morgabraI haven't tried to run this in devstack at all yet16:42
morgabraI didn't change any extensions that ml2 loads16:43
lazy_princeoh.. by the way, i have intgrated your plugin with devstack and i am running it with devstack.. could that be the issue..?16:43
morgabraso if it is supposed to work for ml2 I'm not sure what's up16:43
morgabrawell, maybe. Can you look in the neutron application log for the error?16:43
lazy_princeFile "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/exc_filters.py", line 235, in _raise_mysql_table_doesnt_exist_asis16:44
morgabrahrm, I'm not sure what that's about tbh.16:45
*** derekh has quit IRC16:45
morgabraI know the latest version of neutron changed to not auto-sync db models16:45
morgabraso maybe devstack isn't using the one specified in requirements.txt16:46
lazy_princelet me get you complete trace..16:46
*** rushiagr is now known as rushiagr_away16:47
lazy_princemorgabra: got the log. here it is http://paste.openstack.org/show/106636/16:50
morgabrayeah, db didn't get synced16:50
lazy_princecan i do that manually..?16:51
morgabrayou could try 'neutron-db-manage --config-file /path/to/neutron.conf upgrade head16:52
morgabrabut if it were using the correct version of neutron it would create the tables automatically when the service starts16:53
morgabra(I need to update this at some point :/)16:53
lazy_princeaha.. i am using the Top of Trunk of Neutron..16:53
lazy_princecan you let me know when you update it..? I will be waiting for it..16:54
lazy_princemorgabra: now that you are here, let me ask you few things..16:56
lazy_princemorgabra: https://github.com/openstack-dev/devstack/blob/master/lib/neutron_plugins/ml2#L5116:56
lazy_princedo you know what values I should use for your plugin..?16:56
lazy_princemorgabra: and on L52 in the same file16:57
*** penick has joined #openstack-ironic16:58
morgabrahmm. I'm not sure how devstack works there. It's just pointing to another config file16:58
lazy_princemorgabra: yup.. but if i do not use config file devstack fails..16:59
lazy_princemorgabra: and since you are extending ml2_plugin, it makes sense to use ml2 as base for devstack too..17:00
*** dguerri is now known as dguerri`afk17:00
lazy_princei could be worng too.. on 2nd thought, this is all neutron related.. should we switch to neutron channel so that others can also participate..?17:01
*** viktors is now known as viktors|afk17:01
morgabrawell, this is kind of a one-off thing17:01
lazy_princeokay.. i will leave now.. can i catch you on Monday same time..?17:02
*** penick has quit IRC17:02
morgabrayeah I'll be around17:02
lazy_princeokay.. c ya on Monday then..17:02
*** openstackgerrit has quit IRC17:04
* NobodyCam is back17:05
*** harlowja_away is now known as harlowja17:05
*** jcoufal has quit IRC17:05
*** penick has joined #openstack-ironic17:11
*** HenryG is now known as HenryThe8th17:12
*** MattMan has left #openstack-ironic17:16
*** pelix has quit IRC17:17
*** dhellmann is now known as dhellmann_17:23
*** lazy_prince is now known as killer_prince17:31
*** rushiagr_away is now known as rushiagr17:44
*** igordcard has joined #openstack-ironic17:52
*** kfox1111 has joined #openstack-ironic17:53
kfox1111so, you can have some nova compute's configured with ironic and some with kvm, then you can use host aggregates to associate flavors to hypervisors, and one nova schedular will then be happy with the setup?17:57
*** dhellmann_ is now known as dhellmann18:02
*** eghobo has quit IRC18:04
NobodyCamkfox1111: I have seen it work with compute zones, but that was like 8 months to a year ago18:06
JayFkfox1111: In our deployment, we segreate out different hypervisors into different nova cells, but afaik there's nothing that requires that other than making our management practices a bit more sane :)18:08
*** penick has quit IRC18:09
*** openstackgerrit has joined #openstack-ironic18:11
*** penick has joined #openstack-ironic18:21
*** yjiang5 has joined #openstack-ironic18:23
yjiang5Hi, all, are there any administrator guide for ironic? I didn't find any in https://wiki.openstack.org/wiki/Ironic . (Is this the right channel, or I need to go to #openstack channel?)18:25
NobodyCamthis is the Ironic channel18:25
NobodyCamyjiang5: have you seen http://docs.openstack.org/developer/ironic/#admin-guide18:26
yjiang5NobodyCam: Aha, thanks. Didn't realize it's in the title of "developer documentation". Will have a look.18:27
NobodyCamya, we'll get around to moving it around later, lol18:28
yjiang5NobodyCam: cool, thanks.18:29
kfox1111Ok. thanks for the help.18:30
*** romcheg1 has joined #openstack-ironic18:38
*** ifarkas has quit IRC18:44
*** lucas-dinner has quit IRC18:45
* NobodyCam wanders afk for a few min18:54
*** openstackgerrit has quit IRC19:01
*** openstackgerrit has joined #openstack-ironic19:03
*** rushiagr is now known as rushiagr_away19:05
NobodyCamanyone happen to know if BOOTx64.EFI is used for 32 and 64 bit arch's?19:10
kfox1111whats the status of ironic integration/baremetal deprecation?19:14
NobodyCamkfox1111: as in... the nova driver has yet to land but we're very close. see https://review.openstack.org/#/c/111223 as the start of the nova patches19:17
kfox1111ah. cool.19:17
kfox1111can you run a juno ironic with an icehouse everything else?19:21
NobodyCamhummm, not sure, I think the nova api changed at some point that would break things19:24
kfox1111ah. then I'll probably just wait for juno to play then...19:27
*** penick has quit IRC19:27
NobodyCamkfox1111: you can devstack for just playing with ironic19:31
jrollyeah, nova api has changed19:34
kfox1111didn't meen that kind of playing. playing with a deployment with the intent of using it soonish. I believe it will work, just want to play with how to actually deploy it for use.19:34
kfox1111jroll: thanks.19:34
jrollkfox1111: just run master of everything :D19:34
kfox1111jroll: "What could go wrong (TM)" :)19:35
jroll:)19:35
kfox1111"imagine everything on fire" ;)19:35
jrollit's really not that bad19:35
kfox1111probably not. just been bit one too many times by bad updates.19:36
*** andreykurilin_ has joined #openstack-ironic19:36
jrollright19:36
kfox1111"Oh yeah, we upgraded the database schema.... you need to sync that..."19:36
jrollyeah, good testing is required19:36
kfox1111I'm interested in if something like solum or tripleo will help with that.19:37
jrollyeah, dunno19:38
kfox1111Openstack's got great testing infrastructure, but it doesn't help to ensure what was tested, is whats tested when we deploy it.19:38
jrollright, you need to do downstream testing as well19:38
kfox1111would be great if it was easy to reuse the testing infrastructure locally.19:38
jrollit's all in puppet and whatnot19:39
kfox1111or alternately, tripleo tests things, and then releases the exact images they tested.19:39
* jroll cringes19:39
JayFIt's a problem of basically unlimited space19:40
JayFpeople should be responsible enough with versioning that everything at any version works with everything at any version19:40
JayFbut that's as likely to happen as world peace :)19:40
JayFso I'd maybe just wait until juno-rc1 is cut, use that to play around with, then for prod you can use juno19:40
kfox1111jayf: you only need to keep n most up to date versions of the images or something. not every possible image.19:41
JayFor run master, like we (basically) do, but have to have stronger internal testing processes19:41
JayFkfox1111: if you're talking about the tripleo images idea, I really dislike that for a multitude of reasons19:41
kfox1111yeah.19:41
JayFnot the least of which being that tripleo is /a/ way to deploy Openstack not /the/ way to deploy Openstack :)19:41
kfox1111jayf: sure. but that shouldn't preclude them from providing images for those who don't want to build them themselves. it lowers the bar to entry greatly.19:42
JayFI firmly disagree, but honestly, I dislike image-based deployment as a pattern generally :)19:42
kfox1111you should be able to deploy openstack with puppet on redhat using rdo, or trunk on ubunto, or build your own images, or just get some prebuilt ones. all options are good in different ways.19:43
*** penick has joined #openstack-ironic19:43
kfox1111JayF: we've been back and forth on it for a long time. both have advantages.19:44
kfox1111images boot much faster, and easy to relyably get exactly what you want.19:44
JayFbut lots and lots of overhead just to patch one thing19:44
JayFlike, say, heartbleed19:44
kfox1111kickstart style deployment is easier to deploy but more chance of random spring comes out.19:44
*** chuckC has joined #openstack-ironic19:45
JayFThe magic isn't in using images19:45
JayFthe magic is in the validation19:45
kfox1111yeah. thats where something like disk image builder is very interesting. you can take one image, do a yum upgrade in it easily, and make another. you can then fix heartblead fairly easily while still doing images.19:45
JayFjust by using images you can *try* to push validation work back upstream19:45
*** andreykurilin_ has quit IRC19:45
*** andreykurilin_ has joined #openstack-ironic19:46
jrollrolling upgrades are more complex with image19:46
kfox1111I see the kickstart style image building, as absolutely needed in order to manage creating/updating images. images though have other advantages. I think using one to build the other has a lot of benifits.19:46
kfox1111jroll: true. but ideally, you have a tool like heat that will do the rolling updates for you.19:47
jrollalso the, 'how do I distribute updates to 40k hypervisor hosts?' question is a bit harder with full images19:47
*** penick has quit IRC19:47
jrolldistribute updates quickly*19:48
kfox1111true: but also harder the other direction too. which of my 40k hypervisor hosts got bad update X and need fixing?19:48
NobodyCamjroll: isn't that what tripleO addresses?19:48
jrollwhat bad update x?19:48
jrolldon't push bad updates to production :P19:48
jrollor have a way to pluck the version out?19:48
NobodyCamno 40 k node updates19:48
jrollNobodyCam: what does tripleo address?19:48
jrollwhy?19:48
jrollI have 40k compute nodes, they all need new n-cpu code19:49
jrollright?19:49
NobodyCamya so perserve ephermal and update them19:49
NobodyCam:-p19:49
NobodyCam(/me is giving jroll a hard time)19:50
JayFthat was a high quality troll19:50
JayFI felt my blood pressure going up19:50
NobodyCamhehehe19:50
NobodyCam:-p19:50
jrolllolol19:50
NobodyCamit is friday19:50
jrollhigh five19:50
kfox1111There was an idea floated at some point that was a light weight image update. rather then do a full image replacement, patch it with the diff between two images. kind of half way inbetween.19:51
kfox1111jroll: ideally you would never push out a bad version of something. stuff happens though. :/19:51
kfox1111:)19:51
jrollright, have a way to get a version19:52
*** chuckC has quit IRC19:52
*** chuckC has joined #openstack-ironic19:53
*** athomas has quit IRC20:02
kfox1111we currently use cobbler + anaconda to deploy bare metal nodes. we use cobbler to sync rpmsets and have production/testing repo sets. going forward is pretty easy with this setup, but if we ever have to roll something back, its kind of a pain.20:05
*** penick has joined #openstack-ironic20:05
NobodyCamkfox1111: I've interviewed with folk who never rolled back only forward (thou that was many years ago)20:09
jroll^^20:09
jrollroll forward or bust20:09
NobodyCamactually they had a pretty cool roll out20:09
kfox1111we often will push forward since its so hard to roll back. but it causes outages that are longer then they should be. can be hours instead of minutes. thats the drawback to that approach. when something unexpected goes wrong, it is down time in which you have to figure out what unexpected happened, rather then roll back to a known state quickly, and take nondown time to figure it out.20:11
NobodyCamthe code did automatic a/b testing, so all servers had all the code but only some would run the new code and if blewup theyd just stop serving that rev20:12
kfox1111ah. a/b is a kind of image. ;)20:12
NobodyCamkfox1111: http://en.wikipedia.org/wiki/A/B_testing20:13
kfox1111yeah.20:16
*** romcheg1 has quit IRC20:18
jrolladam_g: check out https://review.openstack.org/#/c/112693 when you have a chance?20:54
jrolladam_g: relies on https://review.openstack.org/#/c/11947320:54
openstackgerritlinggao proposed a change to openstack/ironic: Minor fixes for ipminative console support  https://review.openstack.org/11947520:54
adam_gjroll, sure will take a look in a bit20:55
jrollthanks :)20:55
jrolljust want to make sure that does what I think it does (see commit message)20:55
*** r-daneel has quit IRC21:12
kfox1111so, how does cells and neutron work. do you have a neutron per cell or is it shared. Is that an issue with ironic and kvm in the same cloud?21:24
*** openstackgerrit has quit IRC21:31
*** openstackgerrit has joined #openstack-ironic21:33
adam_gjroll, looks good but i think you've forgotten to add check-tempest-dsvm-ironic-agent_ssh-src to projects.yaml?21:34
*** andreykurilin_ has quit IRC21:43
*** eghobo has joined #openstack-ironic22:02
NobodyCamgah friday if fighting to distract me at every turn22:09
jrolladam_g: exactly why I was looking for more eyes22:17
jrollNobodyCam: you have no idea, so many context switches today :/22:17
NobodyCam:(22:25
NobodyCamjust fyi: I am here in but brain is shutting down :-p22:26
*** dlaube has quit IRC22:29
lifelessromcheg: hi, you wanted me?22:40
mikalPlease brace for impact22:50
jroll:o22:50
greghaynesmikal: O_o22:50
NobodyCam:/22:50
jrolloshi22:50
jrolllast nova patch has +222:50
jrollerr22:50
jroll2x +222:50
jrollthis is happening22:50
NobodyCamomg no22:50
NobodyCamdevananda: ^^^^^^^^^^22:51
jrolleverything does22:51
JayFhttp://i.kinja-img.com/gawker-media/image/upload/s--Gwlhw4RX--/qtr9wvgsujidg5zri4im.gif22:51
mikalNobodyCam: want me to -2?!?22:51
* jroll hands mikal the giant red button22:51
* NobodyCam looks for his GUN22:51
mikalNobodyCam: https://twitter.com/dolphm/status/505787977512275968/photo/122:52
jrollnice22:55
jrollmy kind of party22:55
mikalOh, good news22:56
mikalThis is going to fail22:56
mikalAnd its my fault22:56
NobodyCammikal: lol22:56
mikalRemember, vote one flailing guy22:56
*** harlowja is now known as harlowja_away23:00
mikalOk, new plan23:01
mikalWe're going to have a nap instead23:01
mikalOr bacon23:01
mikalOne or the other23:02
jrollwhy you gotta tease us like that23:02
kfox1111bacon! :)23:02
NobodyCamya the answer has to be "Bacon"23:05
mikaljroll: cause I hate freedome23:05
mikalAlso, IT BEGINS AGAIN23:05
jrolllol23:06
adam_ghttp://iheartjimmy.com/wp-content/uploads/2012/07/tumblr_m4zmrtA1Bf1qheqxz.gif23:07
NobodyCamlol adam_g that is so 70's23:08
* mikal wanders off now that he's done a thing23:10
*** eguz has joined #openstack-ironic23:10
*** eghobo has quit IRC23:13
jrollmikal: why's it going to fail?23:14
jroll(other than it's running openstack CI)23:15
*** dividehex has quit IRC23:15
NobodyCamjroll: boo23:16
*** igordcard has quit IRC23:16
NobodyCammore like because it has to make its way through the gate23:16
mikaljroll: the first time had a pep8 fail in my blocker patch23:16
mikaljroll: the second time around fixes that23:16
NobodyCamgate / gauntlet23:16
jrollmikal: lol.23:16
* mikal runs away to eat the bacon of victory23:18
mikalIn the muffin of pep8 shame23:18
*** dividehex has joined #openstack-ironic23:21
NobodyCamlol23:22
*** penick has quit IRC23:23
*** harlowja_away is now known as harlowja23:31
mrdaJust to chime in23:37
mrda\o/23:37
* mrda wonders whether he should weekend, or sit here watching +A'd patches make their way through check and gate23:42
jrollit's da weekend23:42
mrdajroll: is that a question or a statement?23:44
jrollstatement23:44
jrollI am like... 20 minutes away from weekend23:45
mrdabut this is my baby :)23:45
jrollhehe23:45
jrolllet mikal babysit it for the weekend :)23:45
mrdaAnd I have 2 hours until I have to coach a basketball team :)23:45
jrolllol if you think it can get through any check/gate jobs in two hours23:45
mrdaI can only hope23:46
mrdamaybe check23:46
mrdafor a few of them23:46
mrdacheck is under 90 minutes right now23:46
mrdao.O23:46
jrollyeah, but the gate was super backed up last I checked23:48
mrdanah, gate is only 16 hours!23:49
jrollonly.23:49
mrdabetter than 30+ hours for I23:49
jrollI guess23:49

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!