*** michchap_ has joined #tripleo | 00:01 | |
SpamapS | lifeless: deleted baremetal node that pointed to it. It was being chosen "next" all the time.. then failing, then the thing would move on to the next that presumably works better. Not definitive proof yet, but the machine does look hosed | 00:02 |
---|---|---|
*** michchap has quit IRC | 00:03 | |
*** toci-bot has joined #tripleo | 00:04 | |
toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_FnRhGRy/ | 00:04 |
*** toci-bot has quit IRC | 00:04 | |
*** asalkeld has joined #tripleo | 00:05 | |
*** csd has quit IRC | 00:08 | |
lifeless | SpamapS: have you pdated the machine-inforamtion-tab.txt file for it? | 00:08 |
SpamapS | lifeless: not yet no still investigating | 00:09 |
*** csd has joined #tripleo | 00:10 | |
lifeless | SpamapS: please do - just a 'am looking' so that it doesn't get added back in or anything | 00:11 |
SpamapS | yeah I left a note in the in use by | 00:11 |
lifeless | thanks | 00:13 |
SpamapS | lifeless: running into "EOD stage 1" .. but i am hopeful. :) | 00:15 |
SpamapS | "Waiting for link-up on net0" ... slooooowww. | 00:16 |
SpamapS | and now freecloud0033 is getting the PXE, but failing trying to load pxelinux.cfg | 00:23 |
SpamapS | Trying to load: pxelinux.cfg/01-78-e7-d1-23-94-15 ok | 00:23 |
SpamapS | Boot failed: press a key to retry, or wait for reset.. | 00:23 |
lifeless | one thing to check is the mac address | 00:29 |
lifeless | make sure the port is registered in neutron | 00:29 |
lifeless | (78-e7-d1-23-94-15) | 00:29 |
SpamapS | it is.. | 00:30 |
SpamapS | and in fact we get the right DHCP address | 00:30 |
lifeless | the pxe config file isn't available? | 00:31 |
SpamapS | it is available, and it gets downloaded ^^^ " ok" | 00:31 |
*** csd has quit IRC | 00:33 | |
SpamapS | I think maybe ilo textcons was confusing me | 00:33 |
*** csd has joined #tripleo | 00:33 | |
SpamapS | trying conman | 00:33 |
SpamapS | but.. out of time. :( | 00:33 |
lifeless | ohbai | 00:33 |
SpamapS | oh and now it booted | 00:36 |
SpamapS | lifeless: I still wonder about flakiness | 00:36 |
lifeless | I am strongly suspicious of the mellanox driver situation | 00:39 |
*** cd-undercloud has joined #tripleo | 00:44 | |
cd-undercloud | ************** overcloud complete status=2 ************ | 00:44 |
*** cd-undercloud has quit IRC | 00:44 | |
*** jayg is now known as jayg|g0n3 | 00:50 | |
asalkeld | hi guys, do you still have auto generated cfntools guest images? | 00:50 |
asalkeld | what's the new version of: http://jenkins.tripleo.org:8080/job/autobuilt-images/elements=fedora%20vm%20heat-cfntools/lastSuccessfulBuild/artifact/fedora-vm-heat-cfntools.qcow2 | 00:51 |
lifeless | asalkeld: we don't at the moment | 00:52 |
asalkeld | :( | 00:52 |
*** michchap_ has quit IRC | 00:56 | |
*** michchap has joined #tripleo | 00:59 | |
*** rongze has joined #tripleo | 01:23 | |
*** rongze has quit IRC | 01:27 | |
openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface https://review.openstack.org/57776 | 01:33 |
openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface https://review.openstack.org/57776 | 01:35 |
*** boris-42 has joined #tripleo | 01:36 | |
*** nosnos has joined #tripleo | 01:40 | |
*** arata has joined #tripleo | 01:40 | |
*** nosnos has quit IRC | 01:40 | |
*** nosnos has joined #tripleo | 01:41 | |
*** cd-undercloud has joined #tripleo | 01:52 | |
cd-undercloud | ************** overcloud complete status=1 ************ | 01:52 |
*** cd-undercloud has quit IRC | 01:52 | |
*** boris-42 has quit IRC | 01:55 | |
*** boris-42 has joined #tripleo | 01:56 | |
*** boris-42 has quit IRC | 01:58 | |
*** morazi has quit IRC | 02:02 | |
*** rongze has joined #tripleo | 02:03 | |
*** csd has quit IRC | 02:10 | |
*** csd has joined #tripleo | 02:11 | |
*** ccrouch has quit IRC | 02:11 | |
*** ccrouch has joined #tripleo | 02:12 | |
*** ccrouch has quit IRC | 02:16 | |
*** boris-42 has joined #tripleo | 02:17 | |
*** victor_lowther has quit IRC | 02:32 | |
*** edmund has quit IRC | 02:44 | |
*** cd-undercloud has joined #tripleo | 02:52 | |
cd-undercloud | ************** overcloud complete status=2 ************ | 02:52 |
*** cd-undercloud has quit IRC | 02:52 | |
*** boris-42 has quit IRC | 02:57 | |
*** edmund has joined #tripleo | 03:07 | |
*** ccrouch has joined #tripleo | 03:20 | |
*** arata has quit IRC | 03:22 | |
*** CaptTofu_ has quit IRC | 03:26 | |
*** CaptTofu has joined #tripleo | 03:26 | |
lifeless | jamezpolley: oh, and devtest is special - it gets turned into docs. | 03:30 |
lifeless | jamezpolley: dig around in-tree a little to see that | 03:30 |
*** cd-undercloud has joined #tripleo | 03:34 | |
cd-undercloud | ************** overcloud complete status=1 ************ | 03:34 |
*** cd-undercloud has quit IRC | 03:34 | |
*** edmund has quit IRC | 03:46 | |
*** edmund has joined #tripleo | 04:01 | |
*** julim has joined #tripleo | 04:13 | |
*** toci-bot has joined #tripleo | 04:14 | |
toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_zqdhVcF/ | 04:14 |
*** toci-bot has quit IRC | 04:14 | |
*** arata has joined #tripleo | 04:18 | |
*** julim has quit IRC | 04:23 | |
*** cody-somerville has quit IRC | 04:27 | |
SpamapS | lifeless: ever dig through these a bit: http://www.mellanox.com/related-docs/prod_software/Mellanox_EN_for_Linux_Release_Notes_v2_0-3_0_0.pdf ? | 04:29 |
*** cd-undercloud has joined #tripleo | 04:40 | |
cd-undercloud | ************** overcloud complete status=2 ************ | 04:40 |
*** cd-undercloud has quit IRC | 04:40 | |
*** cody-somerville has joined #tripleo | 04:42 | |
SpamapS | Ok I think status=2 is always this failure: | 04:42 |
SpamapS | Authorization Failed: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection t | 04:42 |
SpamapS | o http://138.35.77.4:5000/v2.0/tokens) | 04:42 |
lifeless | SpamapS: not so much. | 04:51 |
*** rongze has quit IRC | 04:53 | |
SpamapS | lifeless: when you've unloaded/reloaded the mlx driver.. did you leave notes somewhere? | 04:54 |
SpamapS | lifeless: I"m seeing tons of really odd intermittent network behavior | 04:54 |
SpamapS | lifeless: crazy thought - bad cable maybe? :-P | 04:54 |
*** arata has quit IRC | 05:00 | |
*** matsuhashi has joined #tripleo | 05:06 | |
lifeless | SpamapS: unload the driver, reload, remove from bridge, add to bridge | 05:08 |
lifeless | SpamapS: it's all in the console session history | 05:09 |
*** akuznetsov has joined #tripleo | 05:15 | |
*** matsuhashi has quit IRC | 05:16 | |
SpamapS | lifeless: "console session" ? | 05:20 |
SpamapS | lifeless: remember that I am screen stupid | 05:20 |
SpamapS | lifeless: so if there is some screen somewhere, I have no clue how to attach to it. :-/ | 05:20 |
*** matsuhashi has joined #tripleo | 05:23 | |
*** rongze has joined #tripleo | 05:24 | |
lifeless | on the bastion | 05:24 |
lifeless | screen -ls | 05:24 |
*** vkozhukalov has joined #tripleo | 05:24 | |
*** edmund has quit IRC | 05:25 | |
lifeless | then screen -r 15804.ilos or 14980.ilos (they are different ilo sessions screens) | 05:25 |
*** rpodolyaka1 has joined #tripleo | 05:32 | |
*** rongze has quit IRC | 05:33 | |
SpamapS | lifeless: ty | 05:37 |
SpamapS | lifeless: so if that clears it up.. I think we should consider migrating our undercloud to a new machine... | 05:37 |
SpamapS | lifeless: perhaps one with saucy | 05:38 |
*** michchap has quit IRC | 05:40 | |
*** michchap has joined #tripleo | 05:40 | |
*** cd-undercloud has joined #tripleo | 05:44 | |
cd-undercloud | ************** overcloud complete status=1 ************ | 05:44 |
*** cd-undercloud has quit IRC | 05:44 | |
*** arata has joined #tripleo | 06:06 | |
*** arata has quit IRC | 06:07 | |
lifeless | SpamapS: no reason to believe other machines will be better | 06:08 |
lifeless | SpamapS: and AIUI saucy doesn't have the fix... IMBW | 06:08 |
*** vkozhukalov has quit IRC | 06:18 | |
*** michchap has quit IRC | 06:19 | |
SpamapS | lifeless: is there an explicit fix? Something in the back of my brain is saying yes. | 06:24 |
lifeless | SpamapS: I linked stuff in trello | 06:24 |
* SpamapS knew there was something | 06:25 | |
*** michchap has joined #tripleo | 06:26 | |
SpamapS | lifeless: its not so much that I think other hardware will be better. It is that if other hardware is not better, that is _data_. | 06:26 |
lifeless | SpamapS: ack; thing is, we don't really have a facility for moving the undercloud around :( | 06:28 |
lifeless | SpamapS: if you recall we deployed perfectly for the first 3-4 days | 06:29 |
SpamapS | Yes I do recall that. | 06:29 |
lifeless | SpamapS: we could try a reboot, but I'm skeptical. | 06:29 |
lifeless | SpamapS: so, I guess, it is data. | 06:30 |
lifeless | SpamapS: I'm worried about achieving it | 06:30 |
SpamapS | lifeless: http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-2.0-3.0.0&mname=MLNX_OFED_LINUX-2.0-3.0.0-ubuntu12.04-x86_64.iso right? | 06:33 |
lifeless | SpamapS: I guess | 06:33 |
lifeless | SpamapS: I just have the pointer to JIRA same as you | 06:33 |
SpamapS | ah I think I was booted from JIRA after the HPCC move | 06:34 |
lifeless | you should have been reinstated | 06:36 |
lifeless | if not, there is a clicky clicky thing to get back in | 06:36 |
*** rongze has joined #tripleo | 06:37 | |
*** matsuhashi has quit IRC | 06:44 | |
*** matsuhas_ has joined #tripleo | 06:49 | |
SpamapS | lifeless: so looking at the driver source in that iso.. | 06:49 |
SpamapS | lifeless: there are some fixes that look like "zero out previously used memory" | 06:49 |
*** rpodolyaka1 has quit IRC | 06:51 | |
SpamapS | and one thing where the old one waits for a timeout then moves on, where the new one waits for a timeout, if it is reached, warns and retries, and then waits unconditionally for the same thing the other one would skip if the timeout was reached | 06:51 |
SpamapS | seems like it might be worth trying the new driver | 06:52 |
* SpamapS checks to see if saucy already has this | 06:53 | |
* SpamapS should probably check trusty too | 06:53 | |
*** matsuhas_ has quit IRC | 06:56 | |
lifeless | 6 techcon papers to go | 06:57 |
*** cd-undercloud has joined #tripleo | 06:58 | |
cd-undercloud | ************** overcloud complete status=2 ************ | 06:58 |
*** cd-undercloud has quit IRC | 06:58 | |
lifeless | it's a little worrying that every LXC in Ubuntu was broken recently (AFAICT) | 06:59 |
*** jprovazn has joined #tripleo | 07:01 | |
*** matsuhashi has joined #tripleo | 07:03 | |
*** sdake_ has quit IRC | 07:08 | |
*** akuznetsov has quit IRC | 07:09 | |
jprovazn | lifeless, you still around? | 07:18 |
lifeless | jprovazn: somewhat | 07:19 |
jprovazn | phew | 07:19 |
jprovazn | question about this: | 07:19 |
jprovazn | https://etherpad.openstack.org/p/tripleo-icehouse-scaling-design | 07:19 |
jprovazn | lifeless, it seems that next step is to add logstash and/or collectd to monitor undercloud | 07:20 |
jprovazn | lifeless, anybody is on it? | 07:20 |
lifeless | jprovazn: not at the moment | 07:22 |
lifeless | jprovazn: we're working from the MVP in trello | 07:22 |
lifeless | jprovazn: something that needs doing right now is migrating elements to store state in /mnt/state | 07:22 |
*** cwolferh has quit IRC | 07:23 | |
jprovazn | lifeless, by state you mean any state and any element? e.g. for yum/apt save package database in /mnt/state? | 07:23 |
lifeless | any files we want to be modifying in a running image-update image | 07:24 |
lifeless | so not yum/apt | 07:24 |
lifeless | but nova.conf | 07:24 |
lifeless | sshd host keys | 07:24 |
lifeless | mysql database files | 07:24 |
lifeless | nova instance image files | 07:24 |
jprovazn | ah, thanks | 07:30 |
jprovazn | lifeless, when working on an element, should a task in trello be created? | 07:31 |
*** rushiagr has joined #tripleo | 07:31 | |
lifeless | jprovazn: only if you start to run into coordination issues; tere is already a card for this overall thing I think | 07:31 |
lifeless | jprovazn: yeah, third down in the current MVP column | 07:32 |
lifeless | jprovazn: just add your name to the names on the card, and folk will know you're also working on that broad theme. | 07:32 |
lifeless | jprovazn: there are lots of elements to update, likely there won't be any conflicts :) | 07:32 |
jprovazn | lifeless, ok | 07:32 |
jprovazn | lifeless, and completely different question - how about HA? https://etherpad.openstack.org/p/tripleo-icehouse-ha-production-configuration | 07:33 |
lifeless | jprovazn: noone's working on it yet - its just a little over the horizon | 07:33 |
jprovazn | lifeless, this is in status of finding out what should be done or is there already cleaner idea? | 07:33 |
lifeless | the HA spec outlines what we'll do when we get to that point - and we'll need that to get CI/CD fully live so it's also quite important. | 07:34 |
openstackgerrit | A change was merged to openstack/tripleo-incubator: Add the nova V3 api endpoint https://review.openstack.org/57533 | 07:34 |
lifeless | basically I'm delighted to see any work that takes us towards the full CI/CD story :) | 07:34 |
lifeless | jprovazn: I would most like to see folk hacking on the /mnt/state stuff for now. | 07:35 |
lifeless | as rpodolyaka and jog0 are hacking on the nova rebuild support | 07:35 |
jprovazn | lifeless, great, I will be happy to help with /mnt/state right now, but I'd like also put some attention into HA at the momments I'm blocked/wait for something | 07:36 |
lifeless | jprovazn: fair enough! | 07:37 |
GheRivero | morning TripleO | 07:38 |
rpodolyaka | morning all | 07:38 |
jprovazn | morning | 07:40 |
*** akuznetsov has joined #tripleo | 07:41 | |
*** jtomasek has joined #tripleo | 07:51 | |
*** akuznetsov has quit IRC | 07:51 | |
*** vkozhukalov has joined #tripleo | 07:52 | |
*** ifarkas has joined #tripleo | 07:52 | |
*** ifarkas has quit IRC | 07:53 | |
*** ifarkas has joined #tripleo | 07:54 | |
openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fix PATH in pypi-mirror cron job https://review.openstack.org/57724 | 07:56 |
*** arata has joined #tripleo | 08:14 | |
*** cd-undercloud has joined #tripleo | 08:14 | |
cd-undercloud | ************** overcloud complete status=1 ************ | 08:14 |
*** cd-undercloud has quit IRC | 08:14 | |
SpamapS | hm ok saucy does not have the latest mellanox either | 08:20 |
SpamapS | ok, 3.12.0 doesn't either | 08:27 |
lifeless | that would be too easy, right? | 08:28 |
SpamapS | yeah | 08:28 |
SpamapS | I guess it is time to learn how to use dkms | 08:29 |
SpamapS | that or ask them to plug in 04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) | 08:29 |
lifeless | noo | 08:30 |
lifeless | dkms :) | 08:30 |
*** nosnos_ has joined #tripleo | 08:31 | |
*** nosnos has quit IRC | 08:35 | |
*** cody-somerville has quit IRC | 08:44 | |
*** akuznetsov has joined #tripleo | 08:44 | |
*** cody-somerville has joined #tripleo | 08:45 | |
SpamapS | lifeless: FYI, screen says those ilos sessions are not shared | 08:45 |
*** athomas has joined #tripleo | 08:46 | |
lifeless | SpamapS: 15804.ilos (10/07/2013 03:00:54 PM) (Multi, attached) | 08:46 |
*** jtomasek has quit IRC | 08:46 | |
lifeless | SpamapS: 'Multi' | 08:46 |
SpamapS | $ screen -r robertc/15804.ilos | 08:46 |
SpamapS | Must run suid root for multiuser support. | 08:46 |
SpamapS | $ sudo screen -r robertc/15804.ilos | 08:47 |
SpamapS | Access to session denied. | 08:47 |
lifeless | hu | 08:47 |
lifeless | try | 08:47 |
lifeless | screen -r 15804.ilos ? | 08:47 |
SpamapS | $ screen -r 15804.ilos | 08:47 |
SpamapS | There is no screen to be resumed matching 15804.ilos. | 08:47 |
lifeless | nuts | 08:48 |
lifeless | try screen -Ax robertc/15804.ilos | 08:48 |
SpamapS | denied | 08:49 |
lifeless | both with and without sudo ? | 08:49 |
SpamapS | isn't there a ctrl-A you have to do to give me access? | 08:49 |
SpamapS | w/o sudo it whines about suid | 08:49 |
lifeless | I dunno, Ng set this up originally | 08:50 |
lifeless | huh | 08:50 |
lifeless | you're not in the users gorup | 08:50 |
lifeless | *group* | 08:50 |
*** jtomasek has joined #tripleo | 08:50 | |
lifeless | add yourself to that, log out and in, try agin | 08:50 |
SpamapS | denied | 08:52 |
SpamapS | also tried making screen suid root | 08:52 |
SpamapS | now denied as my regular user | 08:52 |
lifeless | uhm, no there was no need to chmod the screen binary | 08:52 |
lifeless | you'll want to put that back to unbreak it | 08:53 |
SpamapS | was just testing | 08:53 |
lifeless | sigh | 08:54 |
lifeless | robertc@bm-aw1az2-freecloud0001:/home/shared$ screen -Ax ilos | 08:54 |
lifeless | Directory '/var/run/screen' must have mode 755. | 08:54 |
lifeless | I wish you hadn't done that test | 08:54 |
SpamapS | lifeless: hence... conman. :-/ | 08:54 |
lifeless | on a box where it's known to work for everyone else | 08:54 |
SpamapS | lifeless: really? that borkens things? | 08:54 |
lifeless | SpamapS: it may have been suid already | 08:54 |
SpamapS | brittle little thing :-P | 08:54 |
SpamapS | it was not | 08:54 |
SpamapS | I checked | 08:54 |
SpamapS | it was sgid | 08:54 |
SpamapS | and is now | 08:55 |
lifeless | ok, I raced with you | 08:55 |
lifeless | its ok now | 08:55 |
SpamapS | and /var/run/screen is the same as other working boxes | 08:55 |
lifeless | ok, try screen -Ax ilos now | 08:56 |
lifeless | no sudo | 08:56 |
SpamapS | -ls shows none | 08:59 |
SpamapS | -Ax says no sockets found in /var/run/screen/S-clint | 08:59 |
*** jistr has joined #tripleo | 09:00 | |
lifeless | try with ilos2? | 09:01 |
SpamapS | nothing. :( | 09:02 |
lifeless | robertc/ilos2? | 09:02 |
SpamapS | "Must run suid root for multiuser support" | 09:04 |
lifeless | bear with me a sec | 09:04 |
*** nosnos_ has quit IRC | 09:04 | |
SpamapS | just building modules | 09:05 |
*** nosnos has joined #tripleo | 09:05 | |
lifeless | SpamapS: ok, try screen -Ax robertc/ilos1 | 09:08 |
SpamapS | "Must run suid root for multiuser support. | 09:08 |
lifeless | ok, try sudo that | 09:08 |
SpamapS | Access to session denied. | 09:09 |
lifeless | ok, acess denied | 09:09 |
lifeless | try now | 09:09 |
SpamapS | \o/ | 09:09 |
SpamapS | so glad I could use your time to get that done | 09:10 |
lifeless | try ilos2, no sudo ? | 09:11 |
SpamapS | still same | 09:11 |
lifeless | ok, w/sudo should work | 09:12 |
SpamapS | so, the ofed thing actually came with a dkms package already | 09:12 |
SpamapS | lifeless: thoughts on installing the newer driver? | 09:12 |
lifeless | SpamapS: test it in a baremetal image first :) | 09:12 |
SpamapS | ah now that's thinking :) | 09:13 |
SpamapS | ok, will attack that tomorrow | 09:13 |
SpamapS | ofed source is in ~heat-admin/mlnx-ofed-kernel-2.0 if you get bored | 09:13 |
SpamapS | debuild binary produces the dkms package | 09:13 |
* SpamapS heads to bed | 09:13 | |
lifeless | SpamapS: check iols2 fist? | 09:14 |
lifeless | ilos2 first? | 09:14 |
*** derekh has joined #tripleo | 09:15 | |
*** cd-undercloud has joined #tripleo | 09:15 | |
cd-undercloud | ************** overcloud complete status=1 ************ | 09:15 |
*** cd-undercloud has quit IRC | 09:15 | |
*** marun has joined #tripleo | 09:16 | |
Ng | mornint | 09:18 |
derekh | overcloud-notcompute-5myoxsyixxnk stuck in BUILD state ... | 09:18 |
derekh | on cd-overcloud | 09:19 |
Ng | lifeless: SpamapS: for talking to the ilos I just apply a giant hammer of awfulness and chmod my pts so I can su to the user who owns the screen session and attach | 09:20 |
Ng | lifeless: SpamapS: but if you just want to talk to serial console for shell, remember conman is there and works :) | 09:20 |
*** akuznetsov has quit IRC | 09:20 | |
lifeless | Ng: ah, I thought you had done something nice | 09:20 |
lifeless | Ng: I've added root to the acls | 09:20 |
Ng | cool | 09:20 |
lifeless | so if anyone does screen -S ilo1 -c ilo_sreenrc1 | 09:21 |
Ng | ideally we'd have a role user to run the screen sessions as, I think | 09:21 |
lifeless | and again with 1/2 | 09:21 |
lifeless | then sudo screen -Ax user/ilos1 | 09:21 |
lifeless | will work | 09:21 |
*** marun has quit IRC | 09:22 | |
*** marun has joined #tripleo | 09:23 | |
*** lucasagomes has joined #tripleo | 09:23 | |
lifeless | derekh: we suspect flaky drivers | 09:27 |
lifeless | derekh: SpamapS has about 1/2 through getting a new mellanox driver together | 09:27 |
derekh | lifeless: ok | 09:27 |
*** boris-42 has joined #tripleo | 09:29 | |
*** matsuhashi has quit IRC | 09:29 | |
lifeless | derekh: I'm just resetting the firmware now | 09:30 |
*** matsuhashi has joined #tripleo | 09:30 | |
lifeless | but I'm off to bed after that | 09:30 |
derekh | lifeless: ok, if there is anything I can help out with let me know (although I know pretty much nothing about mellanox drivers) | 09:31 |
lifeless | will do | 09:31 |
lifeless | rmmod has hung... sure sign that this was the issue | 09:32 |
lifeless | [3893443.151304] mlx4_core 0000:05:00.0: mlx4_cmd_post:cmd_pending failed | 09:32 |
lifeless | [3893451.633588] INFO: task ovs-vswitchd:9303 blocked for more than 120 seconds.6e:06 | 09:32 |
*** akuznetsov has joined #tripleo | 09:32 | |
lifeless | [3893451.633659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" | 09:33 |
lifeless | SpamapS: ^ | 09:33 |
Ng | ouch | 09:33 |
*** boris-42 has quit IRC | 09:34 | |
lifeless | still going | 09:34 |
*** jtomasek has quit IRC | 09:40 | |
*** martyntaylor has joined #tripleo | 09:41 | |
*** jtomasek has joined #tripleo | 09:43 | |
lifeless | still doing it's thing on the ilo textcnos | 09:46 |
lifeless | I'm going to leave it churning away | 09:46 |
lifeless | it may come back, it may not - if not check br-ctlplane has the right ports et | 09:46 |
*** arata has quit IRC | 09:48 | |
derekh | tempest getting close | 10:06 |
derekh | Ran 1236 tests in 3675.426s | 10:06 |
derekh | FAILED (id=0, failures=5, skips=64) | 10:06 |
derekh | although it took an hour | 10:06 |
*** pblaho has joined #tripleo | 10:11 | |
openstackgerrit | Marios Andreou proposed a change to openstack/tripleo-heat-templates: Make merge.py invokable from another script https://review.openstack.org/56947 | 10:17 |
*** marun has quit IRC | 10:45 | |
*** cody-somerville has quit IRC | 10:46 | |
*** marun has joined #tripleo | 10:47 | |
*** cody-somerville has joined #tripleo | 10:48 | |
*** pblaho has quit IRC | 10:52 | |
*** marun has quit IRC | 10:56 | |
*** marun has joined #tripleo | 10:56 | |
*** marun has quit IRC | 10:58 | |
*** cody-somerville has quit IRC | 11:02 | |
*** athomas has quit IRC | 11:07 | |
*** cody-somerville has joined #tripleo | 11:07 | |
*** rdo has quit IRC | 11:07 | |
*** athomas has joined #tripleo | 11:15 | |
*** max_lobur_afk has quit IRC | 11:30 | |
*** max_lobur has joined #tripleo | 11:33 | |
openstackgerrit | Imre Farkas proposed a change to openstack/tripleo-incubator: Add support to register horizon as an endpoint https://review.openstack.org/57906 | 11:54 |
*** cody-somerville has quit IRC | 11:58 | |
*** matsuhashi has quit IRC | 12:09 | |
*** matsuhashi has joined #tripleo | 12:10 | |
*** matsuhashi has quit IRC | 12:15 | |
*** ifarkas has quit IRC | 12:18 | |
*** ifarkas has joined #tripleo | 12:19 | |
*** akuznetsov has quit IRC | 12:21 | |
*** michchap has quit IRC | 12:23 | |
*** michchap has joined #tripleo | 12:23 | |
*** CaptTofu has quit IRC | 12:28 | |
*** CaptTofu has joined #tripleo | 12:28 | |
*** lucasagomes is now known as lucas-hungry | 12:29 | |
openstackgerrit | Dan Prince proposed a change to openstack-infra/tripleo-ci: Set OVERCLOUD_IP correctly for all-in-one. https://review.openstack.org/57912 | 12:34 |
*** michchap has quit IRC | 12:45 | |
*** michchap has joined #tripleo | 12:46 | |
ifarkas | marios, ping | 12:56 |
*** rdopieralski has joined #tripleo | 12:58 | |
*** bauzas has quit IRC | 12:58 | |
marios | ifarkas: hey | 13:00 |
*** marun has joined #tripleo | 13:00 | |
*** akuznetsov has joined #tripleo | 13:01 | |
ifarkas | marios, hey, regarding the horizon endpoint patch, I am working on it to add the overcloud horizon url to tuskar-ui. The patch is more general as it just adds support to register the horizon service, so it's definitely not only an overcloud endpoint blueprint specific change. What else do you think I should add to the commit message? | 13:01 |
*** CaptTofu has quit IRC | 13:02 | |
*** akuznetsov has quit IRC | 13:02 | |
*** CaptTofu has joined #tripleo | 13:03 | |
marios | ifarkas: heh, i don't know you could write any number of things. just thought it looked odd to not have at least one sentence there explaining the commit for sake of posterity | 13:04 |
marios | "right now horizon isn't registered as an endpoint, so services like <foo> (tuskar/ui) can't bla" | 13:06 |
marios | example | 13:06 |
ifarkas | marios, ok | 13:07 |
openstackgerrit | Imre Farkas proposed a change to openstack/tripleo-incubator: Add support to register horizon as an endpoint https://review.openstack.org/57906 | 13:11 |
*** lsmola has quit IRC | 13:11 | |
marios | ifarkas: thanks | 13:15 |
ifarkas | marios, than you for your hint ;-) | 13:15 |
ifarkas | *thank | 13:15 |
*** bauzas1 has joined #tripleo | 13:18 | |
openstackgerrit | Imre Farkas proposed a change to openstack/tripleo-incubator: Add support to register horizon as an endpoint https://review.openstack.org/57906 | 13:19 |
marios | ifarkas: hehe, thanks :) | 13:19 |
ifarkas | marios, thanks for the review. I think it's better to fix this small issue ;-) | 13:20 |
*** pblaho has joined #tripleo | 13:33 | |
*** jdob has joined #tripleo | 13:33 | |
*** lucas-hungry is now known as lucasagomes | 13:34 | |
*** nosnos has quit IRC | 13:38 | |
*** akuznetsov has joined #tripleo | 13:39 | |
*** julim has joined #tripleo | 13:47 | |
*** CaptTofu has quit IRC | 13:53 | |
*** akuznetsov has quit IRC | 13:53 | |
*** CaptTofu has joined #tripleo | 13:53 | |
*** dprince has joined #tripleo | 13:55 | |
*** marun has quit IRC | 13:57 | |
*** marun has joined #tripleo | 13:58 | |
*** michchap has quit IRC | 13:58 | |
*** michchap has joined #tripleo | 14:01 | |
*** tzumainn has joined #tripleo | 14:02 | |
*** rdopieralski has quit IRC | 14:04 | |
*** morazi has joined #tripleo | 14:07 | |
*** akuznetsov has joined #tripleo | 14:07 | |
openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: Adding a geard element https://review.openstack.org/57927 | 14:09 |
marios | derekh: s/borker/broker | 14:11 |
*** bauzas1 is now known as bauzas | 14:11 | |
derekh | marios: thanks fixing | 14:11 |
openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: Adding a geard element https://review.openstack.org/57927 | 14:12 |
*** boris-42 has joined #tripleo | 14:13 | |
*** julim has quit IRC | 14:13 | |
*** michchap has quit IRC | 14:14 | |
*** michchap has joined #tripleo | 14:14 | |
*** julim has joined #tripleo | 14:20 | |
*** jayg|g0n3 is now known as jayg | 14:21 | |
*** lsmola has joined #tripleo | 14:23 | |
rpodolyaka | I'm a bit worried about "The server is not designed for production use under load" line :) | 14:24 |
*** jdob has quit IRC | 14:31 | |
*** jdob has joined #tripleo | 14:31 | |
*** boris-42 has quit IRC | 14:32 | |
*** boris-42_ has joined #tripleo | 14:32 | |
*** cody-somerville has joined #tripleo | 14:34 | |
*** rushiagr has quit IRC | 14:41 | |
*** shakayumi has joined #tripleo | 14:44 | |
derekh | rpodolyaka: ya I hear ya, but from what I can gather its what infra use for all the gating stuff | 14:44 |
derekh | rpodolyaka: https://github.com/openstack-infra/gear | 14:45 |
rpodolyaka | yeah, just wanted to say I'm totally ok with that as it doesn't seem we have too much choice here | 14:47 |
dkehn | morning all | 14:49 |
*** martyntaylor has quit IRC | 14:52 | |
*** jdob has quit IRC | 14:52 | |
*** jdob has joined #tripleo | 14:52 | |
*** jtomasek has quit IRC | 14:53 | |
*** bnemec is now known as beekneemech | 14:53 | |
*** boris-42_ has quit IRC | 14:58 | |
NobodyCam | Good Morning TripleO | 15:03 |
*** lucasagomes has quit IRC | 15:06 | |
*** pblaho has quit IRC | 15:06 | |
*** marun has quit IRC | 15:08 | |
*** marun has joined #tripleo | 15:09 | |
*** jtomasek has joined #tripleo | 15:19 | |
*** lucasagomes has joined #tripleo | 15:21 | |
*** shakayumi has quit IRC | 15:22 | |
*** edmund has joined #tripleo | 15:23 | |
*** shakayumi has joined #tripleo | 15:23 | |
*** marun has quit IRC | 15:29 | |
*** pblaho has joined #tripleo | 15:32 | |
*** rongze has quit IRC | 15:45 | |
*** ifarkas has quit IRC | 15:47 | |
*** shakayumi has quit IRC | 15:48 | |
openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: Adding a geard element https://review.openstack.org/57927 | 15:49 |
openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net https://review.openstack.org/57763 | 15:50 |
openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net https://review.openstack.org/57763 | 15:52 |
*** rongze has joined #tripleo | 16:04 | |
*** UtahDave has joined #tripleo | 16:06 | |
*** akuznetsov has quit IRC | 16:08 | |
*** vkozhukalov has quit IRC | 16:17 | |
*** rongze_ has joined #tripleo | 16:32 | |
*** rongze has quit IRC | 16:32 | |
*** max_lobur has quit IRC | 16:33 | |
*** jtomasek has quit IRC | 16:36 | |
*** akuznetsov has joined #tripleo | 16:38 | |
*** csd has quit IRC | 16:44 | |
*** dkehn has quit IRC | 16:59 | |
*** derekh has quit IRC | 17:00 | |
*** dkehn has joined #tripleo | 17:00 | |
*** julim has quit IRC | 17:05 | |
*** athomas has quit IRC | 17:06 | |
*** julim has joined #tripleo | 17:06 | |
*** rpodolyaka has quit IRC | 17:12 | |
*** athomas has joined #tripleo | 17:12 | |
openstackgerrit | Petr Blaho proposed a change to openstack/python-tuskarclient: [WIP] Adds help for subcommands https://review.openstack.org/56257 | 17:16 |
*** arata has joined #tripleo | 17:17 | |
*** arata has left #tripleo | 17:17 | |
*** pblaho has quit IRC | 17:20 | |
*** jistr has quit IRC | 17:27 | |
*** athomas has quit IRC | 17:36 | |
*** rushiagr has joined #tripleo | 17:36 | |
*** hewbrocca has quit IRC | 17:37 | |
*** lsmola has quit IRC | 17:41 | |
openstackgerrit | Dan Prince proposed a change to openstack/tripleo-heat-templates: Use merge.py for the undercloud templates. https://review.openstack.org/57994 | 17:50 |
*** rongze has joined #tripleo | 17:57 | |
* Ng out for dinner | 17:58 | |
*** rongze_ has quit IRC | 18:01 | |
*** akuznetsov has quit IRC | 18:02 | |
*** UtahDave has quit IRC | 18:09 | |
*** akuznetsov has joined #tripleo | 18:11 | |
openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Make undercloud-vm.yaml. https://review.openstack.org/57997 | 18:23 |
*** rushiagr has quit IRC | 18:27 | |
openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Make undercloud-vm.yaml. https://review.openstack.org/57999 | 18:30 |
openstackgerrit | Dan Prince proposed a change to openstack-infra/tripleo-ci: Make undercloud-vm.yaml. https://review.openstack.org/58000 | 18:31 |
*** rushiagr has joined #tripleo | 18:33 | |
*** rongze has quit IRC | 18:42 | |
*** lucasagomes has quit IRC | 18:43 | |
SpamapS | lifeless: when you're around.. it isn't clear to me what happened with your rmmod. | 18:43 |
dprince | SpamapS: does that mean the CD cloud is down? | 18:49 |
SpamapS | dprince: it is turned off right now | 18:51 |
SpamapS | dprince: we were going to try a new mellanox driver actually | 18:51 |
dprince | SpamapS: I'm a bit out of the loop on the mellanox situation. Last I used mellanox it was for IB... Is there a tl;dl? | 18:52 |
SpamapS | dprince: our hardware has Mellanox 10Gb ethernet cards that we use for connection to the network. We've had serious driver flakiness. | 18:56 |
SpamapS | dprince: rmmod/insmod clears them for a while. | 18:56 |
dprince | SpamapS: we do this on the undercloud? | 18:56 |
SpamapS | dprince: yeah, that one seems to be the one with the problem. | 19:00 |
SpamapS | dprince: only because it has been running for so long we suspect | 19:01 |
*** sdake_ has joined #tripleo | 19:10 | |
*** sdake_ has quit IRC | 19:10 | |
*** sdake_ has joined #tripleo | 19:10 | |
lifeless | SpamapS: hi | 19:18 |
lifeless | SpamapS: when I did rmmod mlx4_en mlx4_core; modprobe mlx | 19:18 |
lifeless | 4_en; sleep 1; ip address del 10.10.16.169/26 dev eth2; ovs-vsctl del-port br-ct | 19:18 |
lifeless | lplane eth2; ovs-vsctl add-port br-ctlplane eth2 | 19:18 |
lifeless | SpamapS: on a console session | 19:18 |
lifeless | SpamapS: it spat out that cmd error for 20m before I said enough and went to bed | 19:18 |
*** vkozhukalov has joined #tripleo | 19:25 | |
*** jdob has quit IRC | 19:28 | |
*** jdob has joined #tripleo | 19:32 | |
SpamapS | lifeless: ok, so you killed it, it never actually rmmodded? | 19:32 |
*** rpodolyaka has joined #tripleo | 19:38 | |
*** rongze has joined #tripleo | 19:43 | |
dprince | lifeless: got time for a question? | 19:44 |
*** rongze has quit IRC | 19:48 | |
*** dprince has quit IRC | 19:52 | |
*** rpodolyaka has quit IRC | 19:55 | |
*** toci-bot has joined #tripleo | 20:04 | |
toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_qInnh5J/ | 20:04 |
*** toci-bot has quit IRC | 20:04 | |
*** akuznetsov has quit IRC | 20:06 | |
*** akuznetsov has joined #tripleo | 20:11 | |
*** jprovazn has quit IRC | 20:12 | |
*** cwolferh has joined #tripleo | 20:13 | |
*** akuznetsov has quit IRC | 20:16 | |
*** rushiagr has quit IRC | 20:18 | |
openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface https://review.openstack.org/57776 | 20:25 |
openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface https://review.openstack.org/57776 | 20:58 |
*** vipul is now known as vipul-away | 21:29 | |
*** vipul-away is now known as vipul | 21:29 | |
*** toci-bot has joined #tripleo | 21:31 | |
toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_4YFSr4g/ | 21:31 |
*** toci-bot has quit IRC | 21:31 | |
openstackgerrit | Clint Byrum proposed a change to openstack/tripleo-image-elements: In tripleo-cd log hardware to instance map https://review.openstack.org/57749 | 21:32 |
*** jistr has joined #tripleo | 21:33 | |
*** vipul is now known as vipul-away | 21:37 | |
*** vipul-away is now known as vipul | 21:37 | |
openstackgerrit | Clint Byrum proposed a change to openstack/tripleo-image-elements: Log pypi-mirror cron job output to syslog https://review.openstack.org/57725 | 21:41 |
Ng | SpamapS: very +1 capturing cron output with logger :) | 21:47 |
SpamapS | Ng: it's one of those "why doesn't cron just do this?" things. | 21:47 |
Ng | SpamapS: I can kinda see an argument that such a default would mean dumping giant pages of text into syslog, but, I agree | 21:48 |
SpamapS | default no | 21:50 |
SpamapS | but just.. do it | 21:50 |
SpamapS | like, OUTPUT=syslog or something | 21:50 |
Ng | there is still my vomitous option of procmailling into logger ;) | 21:51 |
*** vkozhukalov has quit IRC | 21:51 | |
openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Set OVERCLOUD_IP correctly for all-in-one. https://review.openstack.org/57912 | 21:57 |
*** julim has quit IRC | 21:59 | |
*** jistr has quit IRC | 22:10 | |
SpamapS | Ng: wtf are you doing up? | 22:10 |
SpamapS | Oh wait it's only 2200 there | 22:11 |
SpamapS | I'm seriously going to buy a globe that rotates a sun-like light bulb and paste all of your pictures on the places you live to see what time it is | 22:11 |
* SpamapS may have just invented the next booth schwag for the openstack summit : | 22:12 | |
clarkb | SpamapS: google earth/maps can do realtime sunlight and weather if you zoom out far enough | 22:12 |
clarkb | but physical device is much more interesting | 22:12 |
SpamapS | right | 22:12 |
SpamapS | I could put earth on the one screen I don't really use.. | 22:13 |
SpamapS | If it can actually show me peoples' G+ pictures in their current location (or at least profile location) that would be money. | 22:13 |
Ng | SpamapS: you don't need the globe thing, just add timezones for all the people you care about the wakefulness of, to the datetime dropdown at the top right of your screen :) | 22:15 |
Ng | I typically have Pacific, Easter, UTC, Australia and New Zealand in there | 22:16 |
*** jdob has quit IRC | 22:18 | |
Ng | hmm, do not like that gerrit lists a review as having a +2, but when I go into it, I see that it has a more recent -1 from a core person | 22:19 |
*** morazi has quit IRC | 22:23 | |
clarkb | it is biggest value wins where -2 is biggest value :) | 22:23 |
openstackgerrit | A change was merged to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net https://review.openstack.org/57763 | 22:25 |
Ng | clarkb: yeah I'm not sure how it should really display, maybe some icon indicating dissent in the ranks ;) | 22:26 |
Ng | is there an emoji of a mob fighting? | 22:26 |
openstackgerrit | Dan Prince proposed a change to openstack-infra/tripleo-ci: Make undercloud-vm.yaml. https://review.openstack.org/58000 | 22:33 |
openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Make undercloud-vm.yaml. https://review.openstack.org/57999 | 22:34 |
SpamapS | Ng: I have it. I don't want to click anywhere :) | 22:37 |
Ng | maybe now that we have bendable LCD someone could make you a globe that displays xearth :) | 22:37 |
SpamapS | Ng: I disagree actually. You still should be reviewing things with -1's... at least you should be reviewing said -1 to see if you agree. -1 is not "nobody else review this". It is "I don't like it." | 22:38 |
Ng | SpamapS: yeah it's not so much because I want to just skip it, more that if I can see there is dissent I can go and check it out | 22:39 |
lifeless | SpamapS: it did eventually I think | 22:40 |
Ng | but realistically I'm going to go in anyway, so it's no biggie | 22:40 |
SpamapS | lifeless: ah ok | 22:40 |
lifeless | SpamapS: did it come good ? | 22:40 |
SpamapS | Ng: right, I actually think the "last touched" queue is more important. As in, look at things that have been touched least recently. | 22:40 |
SpamapS | lifeless: tripleo-cd was still off, so don't know for sure | 22:44 |
SpamapS | lifeless: going to watch it run now | 22:44 |
SpamapS | I do wonder if we have a race with init-keystone and o-r-c somehow in the result=2 cases. | 22:44 |
jamezpolley | @SpamapS I was more confused by you being online last night when it was ~2000 my time | 22:45 |
lifeless | SpamapS: remember init-keystone is synced by a wait condition | 22:47 |
lifeless | SpamapS: so, no. | 22:47 |
SpamapS | jamezpolley: indeed, I have to _EARN_ my sleep time. :) | 22:48 |
SpamapS | lifeless: unless after the wait condition is fired, we run orc again and are mid-restarting | 22:48 |
SpamapS | hypothesis .. not findings | 22:48 |
*** UtahDave has joined #tripleo | 22:52 | |
openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface https://review.openstack.org/57776 | 22:52 |
*** sdake_ has quit IRC | 23:04 | |
*** toci-bot has joined #tripleo | 23:07 | |
toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_3mrvTb5/ | 23:07 |
*** toci-bot has quit IRC | 23:07 | |
*** cd-undercloud has joined #tripleo | 23:28 | |
cd-undercloud | ************** overcloud complete status=1 ************ | 23:28 |
*** cd-undercloud has quit IRC | 23:28 | |
SpamapS | Unable to find network with name 'ext-net' | 23:39 |
SpamapS | and yet, ext-net exists now | 23:40 |
SpamapS | lifeless: is there any reason we actually build the user image? | 23:40 |
SpamapS | lifeless: couldn't we just upload base ubuntu? | 23:41 |
*** noslzzp has quit IRC | 23:41 | |
*** edmund has quit IRC | 23:42 | |
lifeless | SpamapS: we could today; We'll want a heat enabled image to check heat in the cloud though | 23:43 |
SpamapS | ok so this might be that asynchronous network schedling thing | 23:44 |
SpamapS | scheduling | 23:44 |
SpamapS | the command that failed was neutron floatingip-create ext-net --port-id "${PORT//[[:space:]]/}" | 23:45 |
SpamapS | which works fine | 23:45 |
SpamapS | I've heard this before | 23:45 |
SpamapS | "Neutron is liar" was the title of a lightning talk slide. :) | 23:45 |
SpamapS | | fault | {u'message': u'Unexpected vif_type=binding_failed', u'code': 500, u'created': u'2013-11-22T19:05:22Z'} | | 23:48 |
SpamapS | actually that was the problem | 23:48 |
SpamapS | but.. no idea what it actually means | 23:48 |
SpamapS | lifeless: seems that our network driver problems were some of the issues, but we do have something really breaking the overcloud | 23:49 |
lifeless | http://www.gossamer-threads.com/lists/openstack/dev/33184 | 23:49 |
lifeless | SpamapS: thus my point about treating this like production even though we don't have state - keeps bitrot low :) | 23:50 |
*** edmund has joined #tripleo | 23:50 | |
lifeless | SpamapS: neutron agent-list? | 23:50 |
lifeless | http://www.gossamer-threads.com/lists/openstack/dev/33218#33218 | 23:51 |
SpamapS | lifeless: doh, tripleo-cd wiped it out again :( | 23:51 |
SpamapS | should have stopped it | 23:51 |
lifeless | nvm | 23:51 |
SpamapS | I added the bridge_mapping in the ml2_plugin.ini and now everything works | 23:52 |
lifeless | SpamapS: so - look at neutron agent logs next time I think | 23:52 |
* SpamapS checks | 23:52 | |
SpamapS | should have been quotes on that last thing | 23:52 |
SpamapS | we don't even have an ml2_plugin.ini | 23:53 |
lifeless | SpamapS: where are you looking? | 23:53 |
SpamapS | ok apparently it is usually configured in ovs_neutron_plugin.ini which we do | 23:54 |
SpamapS | so maybe neutron.ovs.bridge_mappings wasn't set right | 23:54 |
lifeless | elements/neutron/os-config-applier/etc/neutron/plugins/ml2/ml2_conf.ini | 23:54 |
*** UtahDave has quit IRC | 23:55 | |
SpamapS | _conf isn't _plugin ..? | 23:55 |
* SpamapS should probably just RTFM on bridge_mapping and see if we are doing it right | 23:55 | |
SpamapS | bridge_mappings: '' | 23:56 |
*** jayg is now known as jayg|g0n3 | 23:58 | |
SpamapS | lifeless: http://docs.openstack.org/network-admin/admin/content/app_demo_single_router.html <-- we are that right? | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!