Wednesday, 2018-09-26

*** lixiaoy1 has joined #openstack-cinder00:32
*** LiangFang has joined #openstack-cinder00:34
*** yikun has joined #openstack-cinder00:52
openstackgerritMerged openstack/cinder master: api-ref: add docs for os-migrate_volume API  https://review.openstack.org/59537900:58
openstackgerritMerged openstack/cinder master: Avoid using dict.get() in assertions  https://review.openstack.org/59737700:58
*** Dinesh_Bhor has joined #openstack-cinder01:23
openstackgerritMerged openstack/cinder master: Drop legacy backup service support  https://review.openstack.org/59537201:54
openstackgerritMerged openstack/cinder master: Fix api-ref title levels and index  https://review.openstack.org/59202601:54
openstackgerritMerged openstack/cinder master: Fix deprecation warnings  https://review.openstack.org/59583301:54
*** erlon has quit IRC02:01
*** Dinesh_Bhor has quit IRC02:02
*** zhaochao has joined #openstack-cinder02:07
*** Dinesh_Bhor has joined #openstack-cinder02:11
*** psachin has joined #openstack-cinder02:39
*** Bhujay has joined #openstack-cinder02:49
*** imacdonn has quit IRC02:50
*** imacdonn has joined #openstack-cinder02:50
openstackgerritLisaLi proposed openstack/cinder-specs master: Driver reinitialization after failure  https://review.openstack.org/59950502:51
openstackgerritYikun Jiang proposed openstack/cinder-specs master: Add volume re-image api  https://review.openstack.org/60531703:04
*** Bhujay has quit IRC03:07
*** LiangFang has quit IRC03:11
openstackgerritMerged openstack/cinder master: Fix some inconsistencies in messages api ref  https://review.openstack.org/60329303:11
openstackgerritMerged openstack/cinder master: Fix some inconsistencies in qos-specs api ref  https://review.openstack.org/60402003:11
openstackgerritMerged openstack/cinder master: ZFSSA iSCSI implement get_manageable_volumes()  https://review.openstack.org/60423003:11
*** gnufied has quit IRC03:23
*** gnufied has joined #openstack-cinder03:40
*** gnufied has quit IRC03:40
*** dave-mccowan has quit IRC03:44
*** dave-mccowan has joined #openstack-cinder03:46
*** itlinux has joined #openstack-cinder03:55
*** pcaruana has joined #openstack-cinder04:14
*** pcaruana has quit IRC04:38
*** Dinesh_Bhor has quit IRC04:47
*** Dinesh_Bhor has joined #openstack-cinder04:54
*** e0ne has joined #openstack-cinder05:08
*** abishop_ has quit IRC05:08
*** abishop has joined #openstack-cinder05:12
*** Bhujay has joined #openstack-cinder05:26
*** Bhujay has quit IRC05:32
*** dave-mccowan has quit IRC05:39
*** pcaruana has joined #openstack-cinder05:43
*** e0ne has quit IRC05:49
*** Bhujay has joined #openstack-cinder05:49
*** jistr has quit IRC05:55
*** jistr has joined #openstack-cinder05:56
*** dpawlik has joined #openstack-cinder06:23
whoami-rajatHi tommylikehu , need some help regarding my patch related to multiattach, https://review.openstack.org/#/c/604040/4 (don't refer latest patchset), Zuul passes but the CI seems to be failing where the tempest test fail with timeout and can't reach volume-type-2 . https://github.com/openstack/tempest/blob/master/tempest/api/volume/admin/test_volume_retype.py#L81 here.06:51
tommylikehuwhoami-rajat:  will take a look a little later today:)06:52
whoami-rajattommylikehu: ok, thanks.06:53
*** LiangFang has joined #openstack-cinder07:00
*** rcernin has quit IRC07:02
*** alexchadin has joined #openstack-cinder07:03
*** amoralej|off is now known as amoralej07:06
*** Dinesh_Bhor has quit IRC07:15
*** psachin has quit IRC07:21
*** alexchadin has quit IRC07:25
*** psachin has joined #openstack-cinder07:26
tobias-urdinjungleboyj: no worries, didn't read anything in the release notes about resource_filters.json though, was it enforced silently because that file did not exist in queens and there was no issues then?07:45
tobias-urdina release note just saying that that file is used would have helped when searching through the notes for "query" since there was issue with query options for api calls07:45
*** alexchadin has joined #openstack-cinder07:51
*** rcernin has joined #openstack-cinder07:56
*** alexchadin has quit IRC07:57
*** Dinesh_Bhor has joined #openstack-cinder07:58
*** alexchadin has joined #openstack-cinder07:59
*** e0ne has joined #openstack-cinder08:10
*** moshele has joined #openstack-cinder08:12
*** gkadam has joined #openstack-cinder08:16
*** Emine has joined #openstack-cinder08:17
*** e0ne has quit IRC08:20
*** jistr has quit IRC08:30
*** jistr has joined #openstack-cinder08:31
*** LiangFang has quit IRC08:49
*** lixiaoy1 has quit IRC08:53
tobias-urdinjungleboyj: is it a regression that cinderclient volume_snapshots.list() doesn't output the user_id for the snapshot?08:55
tobias-urdinthat's breaking ceilometers volume.snapshots.size metric08:55
*** Dinesh_Bhor has quit IRC09:03
*** e0ne has joined #openstack-cinder09:04
*** alexchadin has quit IRC09:06
tobias-urdinsorry for noice again, probably ceilometer not using api_version >= 3.4109:08
*** moshele has quit IRC09:10
*** rcernin has quit IRC09:16
*** alexchadin has joined #openstack-cinder09:20
openstackgerritNeha Alhat proposed openstack/cinder master: Enable split logging for cinder-keystoneclient interaction  https://review.openstack.org/57976509:21
*** alexchadin has quit IRC09:25
*** Dinesh_Bhor has joined #openstack-cinder09:28
*** alexchadin has joined #openstack-cinder09:42
openstackgerritMerged openstack/cinder master: Add comment to online data migration command  https://review.openstack.org/60168409:42
openstackgerrityenai proposed openstack/os-brick master: Remove meanless debug log  https://review.openstack.org/60538009:50
openstackgerritYikun Jiang proposed openstack/cinder-specs master: Add volume re-image api  https://review.openstack.org/60531709:52
*** Dinesh_Bhor has quit IRC10:10
*** Dinesh_Bhor has joined #openstack-cinder10:16
*** Dinesh_Bhor has quit IRC10:18
*** e0ne has quit IRC10:27
*** Luzi has joined #openstack-cinder10:42
*** alexchadin has quit IRC10:49
*** ganso has joined #openstack-cinder10:50
*** alexchadin has joined #openstack-cinder10:58
*** alexchadin has quit IRC11:03
*** erlon has joined #openstack-cinder11:07
*** luizbag has joined #openstack-cinder11:11
*** psachin has quit IRC11:13
*** pcaruana has quit IRC11:15
*** psachin has joined #openstack-cinder11:27
*** pooja-jadhav is now known as pooja_jadhav11:33
*** lixiaoy1 has joined #openstack-cinder11:34
*** brinzhang has joined #openstack-cinder11:42
*** e0ne has joined #openstack-cinder11:53
*** savvas has joined #openstack-cinder11:55
*** alexchadin has joined #openstack-cinder11:57
*** dpawlik has quit IRC11:58
*** alexchadin has quit IRC12:17
*** alexchadin has joined #openstack-cinder12:25
*** lixiaoy1 has quit IRC12:32
*** amoralej is now known as amoralej|lunch12:33
*** psachin has quit IRC12:41
*** zhaochao has quit IRC12:49
*** alexchadin has quit IRC13:02
*** mriedem has joined #openstack-cinder13:03
*** alexchadin has joined #openstack-cinder13:17
*** dustins has joined #openstack-cinder13:24
*** _hemna has joined #openstack-cinder13:35
*** abishop has quit IRC13:40
*** amoralej|lunch is now known as amoralej13:41
*** alexchadin has quit IRC13:44
*** gnufied has joined #openstack-cinder13:45
*** alexchadin has joined #openstack-cinder13:46
*** brinzhang has quit IRC13:49
*** alexchadin has quit IRC13:50
_hemnasup peeps13:54
*** abishop has joined #openstack-cinder14:09
jungleboyj_hemna:  Morning sir.14:16
jungleboyjtobias-urdin:  O dpn14:19
jungleboyjWow, guess I am not ready to type this morning.14:19
openstackgerritRajat Dhasmana proposed openstack/cinder master: Fix multiattach set to false after retype  https://review.openstack.org/60404014:24
_hemnanot awake yet?14:25
_hemnaheh14:25
jungleboyj_hemna:  No.14:25
_hemnaI'm not sure I want to adult today14:25
jungleboyj_hemna: OMG, I hear you.14:25
*** Luzi has quit IRC14:27
jungleboyjtobias-urdin:  So, it looks like the resource_filters were added almost a year ago.  The only change to that file that went in for this release was supporting availability-zone type.  With that said, when it merged it didn't really have a good explanation as to the possible impact on end users.  So that definitely needs improvement.14:29
jungleboyjtobias-urdin:  As for the  volume_snapshots.list() it sounds like you nailed that one.  We have had a few people confused by things due to not specifying the microversion.  That was a problem that we discussed at the PTG.  We have updated the help text so that it will show all commands regardless of the requested microversion.  Hoping that will reduce some confusion.14:32
openstackgerritMerged openstack/cinder master: Fix bug of renaming volume with same name  https://review.openstack.org/60227514:39
*** Bhujay has quit IRC14:44
*** LiangFang has joined #openstack-cinder14:49
*** pcaruana has joined #openstack-cinder15:01
tobias-urdinjungleboyj: ack, thanks15:01
jungleboyjtobias-urdin:  Welcome.  Were you upgrading from Queens to Rocky?15:12
tobias-urdinyeah https://bugs.launchpad.net/ceilometer/+bug/179447515:13
openstackLaunchpad bug 1794475 in Ceilometer "volume.snapshot.size broken due to user_id attributeerror" [Undecided,New]15:13
tobias-urdinmanually fixing that for now, can't we do some cooler version discovery here https://github.com/openstack/ceilometer/blob/stable/rocky/ceilometer/volume/discovery.py#L3215:13
*** finix has joined #openstack-cinder15:17
tobias-urdini still dont understand why it would stop working, iirc it worked in queens without any issue15:17
tobias-urdinthat's why i wondered if it might be a issue somewhere or a regression introduced15:17
*** dpawlik has joined #openstack-cinder15:32
*** dpawlik has quit IRC15:36
*** savvas has quit IRC15:37
*** finixlei has joined #openstack-cinder15:38
*** finix has quit IRC15:41
*** dklyle has joined #openstack-cinder15:45
*** Bhujay has joined #openstack-cinder15:46
*** xyang has joined #openstack-cinder15:47
whoami-rajatjungleboyj: https://review.openstack.org/#/c/604067/2 i think this patch needs to add a unit test for the case where it will fail with 3 default filters but will pass with CapabilitiesFilter15:48
jungleboyjwhoami-rajat:  Wow,  surprised I missed that.15:49
jungleboyjwhoami-rajat:  You are right.15:49
whoami-rajatjungleboyj: maybe its a small fix that's why.15:52
jungleboyjwhoami-rajat:  But you are right.15:54
whoami-rajatjungleboyj: 🙂15:56
*** dave-mccowan has joined #openstack-cinder16:01
*** LiangFang_ has joined #openstack-cinder16:07
*** LiangFang has quit IRC16:08
*** Emine has quit IRC16:16
*** tpsilva has joined #openstack-cinder16:19
*** gkadam has quit IRC16:25
*** Bhujay has quit IRC16:29
*** LiangFang_ has quit IRC16:30
_hemnageguileo, jungleboyj so, I'm sorry I wasn't able to attend the PTG this time around to talk about the cinderlib and extracting drivers from tree16:39
_hemnabut I was just wondering how that all went16:39
geguileo_hemna: we didn't discuss that topic, it took us the whole hour to decide we were going to merge it into the cinder repo16:42
_hemnaah ok16:42
_hemnaat some point I'd like to bring up a few issues that I have with the current driver mechanism in cinder16:42
*** e0ne has quit IRC16:43
*** dave-mccowan has quit IRC16:49
tpsilvajungleboyj: ping16:57
jungleboyjtpsilva: Hold on.  Wrapping up meeting.16:58
tpsilvaalright16:58
openstackgerritMerged openstack/cinder master: Remove unecessary pass  https://review.openstack.org/60517316:58
tpsilvajungleboyj: sorry, thought it was finished :)16:58
tpsilvasorry, messed up the schedule for the meetings17:00
jungleboyjtpsilva:  What's up?17:03
tpsilvajungleboyj: hey17:03
tpsilvaI got a -2 on this: https://review.openstack.org/#/c/597193/17:04
tpsilvabut its parent was merged17:04
tpsilvahttps://review.openstack.org/#/c/597192/17:04
tpsilvaI don't remember who exactly talked about this in the past, but it was agreed to be merged17:04
jungleboyjtpsilva:  Looking.17:06
jungleboyjtpsilva:  And I merged it.  *laughing*17:08
jungleboyjOk.  Looking at it again I think I can re-address.17:08
tpsilvaalright17:08
tpsilvathank you!17:08
jungleboyjtpsilva:  Sorry about that.17:09
tpsilvajungleboyj: no problem17:09
jungleboyj_hemna: Topic for the mid-cycle?17:14
_hemnasure17:15
*** dpawlik has joined #openstack-cinder17:16
*** dpawlik has quit IRC17:16
*** dpawlik has joined #openstack-cinder17:17
openstackgerritChuck Short proposed openstack/cinder master: inspur: Fix raise  https://review.openstack.org/60549517:36
*** dpawlik has quit IRC17:47
imacdonn_hemna: still around ?17:49
_hemnayah17:50
imacdonnpls review https://review.openstack.org/593955 if you have a moment ... ta17:50
_hemnaok17:51
imacdonntnx17:51
_hemnaprobably should have unit tests17:53
imacdonnyeah, that seems fair .. thanks17:56
_hemnado you remember where the dummer driver is?18:00
_hemnafound it18:00
_hemnafake_driver18:00
openstackgerritMerged openstack/cinder stable/queens: VMAX driver - Block revert to snapshot for replicated volumes  https://review.openstack.org/59704918:05
*** amoralej is now known as amoralej|off18:20
*** dpawlik has joined #openstack-cinder18:21
*** pcaruana has quit IRC18:24
*** dpawlik has quit IRC18:25
jungleboyjI have submitted our forum topics and added links from the Wiki to the submissions.18:34
jungleboyjmriedem:  Ping.18:49
mriedemPONG18:50
*** abishop_ has joined #openstack-cinder18:58
*** jdillaman has quit IRC18:58
*** jdillaman has joined #openstack-cinder18:59
*** abishop has quit IRC19:00
jungleboyjmriedem:  Hey.19:01
jungleboyjmriedem:  Have a couple questions about scheduling VMs in a heterogeneous environment.19:01
*** dave-mccowan has joined #openstack-cinder19:05
*** rtjure has quit IRC19:05
*** rtjure has joined #openstack-cinder19:08
*** asbishop has joined #openstack-cinder19:12
*** asbishop is now known as abishop19:13
*** abishop_ has quit IRC19:15
*** e0ne has joined #openstack-cinder19:15
mriedemjungleboyj: ok19:24
mriedemi might have a half assed answer19:24
jungleboyjmriedem: I will take half assed.19:24
jungleboyjmriedem: So, say I have two different kinds of compute nodes.19:25
jungleboyjSome are HCI with Ceph and some only have Ephemeral storage.19:25
*** Emine has joined #openstack-cinder19:26
jungleboyjIs there some way with flavors or something to say I want this default VM config to go to they systems that are HCI (using Ceph for the volume backing) and the other config I want to go to the compute nodes just using ephemeral storage?19:26
mriedemso like 3 compute servers on the same ceph pool in the same rack?19:26
mriedemhost aggregates is how you can tie flavors to certain sets of compute hosts19:27
jungleboyjRight and then three compute nodes without Ceph that we want to just use ephemeral local storage.19:27
mriedemso when you say ephemeral you mean local to the compute host right?19:28
jungleboyjmriedem:  Right.19:28
mriedemb/c compute hosts connected to a ceph cluster can still have "ephemeral" root disks19:28
mriedemi.e. non-volume-backed vms19:28
mriedemyeah so if you're using libvirt, there is an backend config to say you want hosts using qcow2 or raw local disk images and the other compute hosts you configure to use the ceph cluster19:29
mriedemand then put those different compute hosts in aggregates,19:29
jungleboyjmriedem:  Right, but if we have a flavor that uses a host aggregate and then say that we want to use Ceph as the volume-backing for the VM, it will go to the right place.19:29
mriedemand then associate your 2 flavors to each host aggregate19:29
jungleboyjmriedem:  Sweet.  That was what I was hoping you would say.19:30
*** e0ne has quit IRC19:30
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#libvirt.images_type19:30
mriedemvolume-backed is a bit different,19:30
mriedemsince it doesn't have anything to do with that images_type config19:30
mriedemsince nova does'nt control the storage for the root disk in the case of a volume-backed vm19:31
mriedempinning volume-backed vms to specific compute hosts gets a bit hairier with stuff like AZs19:32
jungleboyjOk, but if we wanted to do that we just need to specify the right aggregate and then say we want volume backed with the right flavor.19:32
mriedemi.e. affinity wise you want the ceph volume-backed vm to be closer to the compute hosts sharing the same ceph cluster i'd imagine19:32
mriedemwell,19:32
jungleboyjWell, it is HCI, so they are going to be close.19:33
mriedemyeah if the user is using the ceph flavor19:33
mriedemso boot from volume with rbd volume type and then use the flavor pinned to the ceph compute hosts19:33
jungleboyjmriedem:  Right.19:33
jungleboyjThis is a PoC so it doesn't have to be super fancy right now.19:33
mriedemthe user could use the non-ceph flavor and it would still work as long as the other computes can connect to the rbd volume backend19:34
mriedembut that's where i said things get weird with stuff like AZs19:34
jungleboyjRight.19:34
jungleboyjJust wanted to make sure I wasn't talking out my ass when I said that we should be able to have two different flavors.19:35
jungleboyjTastes like Ceph and doesn't taste like Ceph.19:35
mriedemhttps://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#host-aggregates-and-availability-zones19:35
mriedemhttps://docs.openstack.org/nova/latest/user/aggregates.html19:36
jungleboyjmriedem:  Thanks.  I will take a look.19:36
mriedemwhen you start needing to restrict a vm and volume to the same set of hosts, that's when https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach comes in19:36
mriedembut that configuration has it's own set of bugs...19:37
mriedemsorrison: can tell you all about it19:37
mriedemtell miro i said hi19:39
jungleboyjmriedem:  Will do.  :-)20:03
jungleboyjHe is keeping me busy.20:03
*** luizbag has quit IRC20:11
*** evrardjp has joined #openstack-cinder20:11
*** jarbassaidai has joined #openstack-cinder20:33
jungleboyjmriedem:  Ah, I just re-read the discussion and understand why we were talking past each other at one point.  There RBD backing for ephemeral storage which is a different concept from volume backed images.  Gotcha.20:48
* jungleboyj has a volume centric view. :-)20:48
mriedemyar20:48
jungleboyjYar20:48
mriedemthis is why we've (nova) talked about replacing those various ephemeral image backends in nova with a cinder backend that just does boot from volume for all of those other types of things, like scaleio, rbd, ploop, etc20:49
*** dustins has quit IRC20:49
jungleboyjmriedem:  What happened to that discussion.  People were excited about it for a bit.20:49
mriedembut some people want local disk for HPC20:49
mriedem*people that were not going to actually work on making it happen were excited about it for a bit20:49
mriedemhttps://review.openstack.org/#/c/511965/ was a semi compromise after the boston summit20:50
mriedemwhere i vomited some ideas into a spec20:50
mriedemwe talked about ^ again at the last ptg, and i told people to read that and weigh in on if it's something nova should do20:50
mriedemtl;dr put a flag on flavors saying the vm will be volume-backed20:51
mriedemand nova orchestrates the stuff for you20:51
mriedemso you don't need to pass the ugly block_device_mapping_v2 dict-o-doom20:51
jungleboyjHe he.20:51
*** abishop has quit IRC21:03
*** erlon has quit IRC21:04
jungleboyjmriedem:  Maybe a topic to bring up again at some point.21:05
mriedemlyarwood has proposed a forum session that it could tie into so i added a comment about it there21:13
openstackgerritJon Bernard proposed openstack/cinder master: RBD: add support for multiattach  https://review.openstack.org/59582721:27
imacdonnjungleboyj: working on unit test(s) for https://review.openstack.org/#/c/593955/ ... it occurs to me; should there even be an option to disable the v3 API? It's the only non-deprecated one we have........22:00
*** dklyle has quit IRC22:02
jungleboyjWell right now first option doesn't make sense. In the future if we go to a V4 we may need to disable but I think the hope is we never have to.22:04
jungleboyj*the option22:05
imacdonnhmm, right, yeah22:05
imacdonnstill, seems that the option to not enable v3 should only come into existence when there's a v422:08
imacdonnI'll just use ddt for now and apply to 2.0 and 3.022:08
imacdonnside-note: I think there's a minor bug here - this test doesn't do anything version specific, so there's no need (that I can see) to use ddt: https://github.com/openstack/cinder/blob/master/cinder/tests/unit/api/test_versions.py#L6222:11
*** ganso has quit IRC22:17
imacdonnjungleboyj: so I want to submit my update to smcginnis's change, with this new unit test .. how do I handle this? He's got "Signed-off-by" in hos commit message. I'm thinking I remove that, and add a Co-Authored-By instead ?22:19
*** ThomasWhite has joined #openstack-cinder22:20
ThomasWhiteHi, I'm currently running cinder 10.0.7 with a Ceph RBD backend, and I am trying to find a way to rate-limit the activity of snapshots, backups etc. I have found the volume_copy_bps_limit parameter and set it to 20971520 in the hope of limiting it to 200MB/s, but it seems to have had absolutely no effect. Have I misunderstood the usage of the parameter?22:22
jungleboyjJust add the co-authored.22:22
imacdonnjungleboyj: and leave the "Signed-off-by"? ok...22:23
imacdonnjungleboyj: er, sorry, dumb Q .. should the co-authored-by be me or him? not sure how gerrit handles change of submitter ....22:24
* imacdonn finds and starts reading https://openstack.nimeyo.com/27153/openstack-dev-gerrit-co-authors-and-ticket-stealing22:25
ThomasWhiteFor reference, my present cinder.conf: https://pastebin.com/CKs09QEK22:28
imacdonnThomasWhite: haven't looked closely, but I think that option just rate-limits a single copy transaction ... doesn't span transactions, nor limit transactions per time period22:34
*** rcernin has joined #openstack-cinder22:35
ThomasWhiteimacdonn, even on single transactions, still seeing up to 1000 MB/s for both read and write individually.22:40
imacdonnThomasWhite: what release are you running? and have you applied that config on the cinder-volume host(s) ?22:42
ThomasWhite10.0.7/Ocata22:42
ThomasWhiteNo, set it to [DEFAULT], I did try the per-host too and same result of no impact22:43
*** dpawlik has joined #openstack-cinder22:48
imacdonnThomasWhite: I meant; did you add it to cinder.conf on the host(s) that run the cinder-volume service, as opposed to, say, the one(s) running cinder-api, etc ?22:52
*** dpawlik has quit IRC22:52
*** rcernin has quit IRC22:53
*** rcernin has joined #openstack-cinder22:55
jungleboyjimacdonn:  So, Sean originally submitted it.  If you push up another patch you will show up as the Committer but Sean will be the Author.23:00
jungleboyjI believe that is how that shows up.23:00
jungleboyjAnyway, If you want to add yourself you would be co-author since you are helping out.23:00
imacdonnjungleboyj: yeah, that's what I figured - will try it .. currently trying to figure out why some VMAX unit tests are puking when I do a full 'tox -e py27'23:01
jungleboyjStrange.  Good luck.  I need to go get food for my offspring.  :-)23:02
imacdonnjungleboyj: tnx .. later23:02
*** tpsilva has quit IRC23:19
*** dklyle has joined #openstack-cinder23:26
*** mriedem is now known as mriedem_away23:30
openstackgerritiain MacDonnell proposed openstack/cinder master: Exclude disabled API versions from listing  https://review.openstack.org/59395523:31
*** finixlei has quit IRC23:40
*** rcernin_ has joined #openstack-cinder23:41
*** rcernin has quit IRC23:43
imacdonnsmcginnis: looks like I stole the author on that change .... not intentionally!23:43
*** openstackgerrit has quit IRC23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!