Friday, 2019-08-09

« Thursday, 2019-08-08 Index Saturday, 2019-08-10 »
*** ivve has quit IRC00:32
*** whoami-rajat has quit IRC00:34
*** nafiux has quit IRC00:58
*** spsurya has joined #openstack-kolla01:05
*** igordc has quit IRC01:25
*** whoami-rajat has joined #openstack-kolla01:32
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550201:39
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550201:59
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550202:07
*** kevinz has joined #openstack-kolla02:10
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550202:25
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550202:31
*** altlogbot_0 has quit IRC02:44
*** altlogbot_1 has joined #openstack-kolla02:45
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550202:54
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550203:20
*** redrobot has quit IRC03:22
*** factor has quit IRC04:12
*** factor has joined #openstack-kolla04:12
*** shyamb has joined #openstack-kolla04:23
*** dave-mccowan has quit IRC04:25
*** shyamb has quit IRC04:37
*** nafiux has joined #openstack-kolla05:00
*** shyamb has joined #openstack-kolla05:33
*** cah_link has joined #openstack-kolla05:56
*** shyamb has quit IRC05:59
*** shyamb has joined #openstack-kolla06:04
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP]Move stop acton to kolla model from host  https://review.opendev.org/67550206:06
*** Wasaac_ has joined #openstack-kolla06:09
*** Wasaac_ has quit IRC06:13
*** shyamb has quit IRC06:17
*** shyamb has joined #openstack-kolla06:33
*** kevinz has quit IRC06:36
*** nafiux has quit IRC06:37
*** shyam89 has joined #openstack-kolla06:38
*** shyamb has quit IRC06:42
openstackgerritzhulingjie proposed openstack/kolla-cli master: Move stop acton to kolla model from host  https://review.opendev.org/67550206:44
yoctozeptomorning06:44
*** rpittau|afk is now known as rpittau06:52
*** shyam89 has quit IRC06:56
*** shyamb has joined #openstack-kolla07:07
openstackgerritRadosław Piliszek proposed openstack/kolla master: Ensures monasca-thresh data and worker dirs exist and are empty on start up  https://review.opendev.org/67477907:21
openstackgerritgugug proposed openstack/kolla-cli master: Use inventory object to validate the hosts and services  https://review.opendev.org/67554207:23
hrwmorning07:25
hrwmnasiadka: built master debian/source images with your patch and deployed just fine07:26
mnasiadkahrw: awesome, because it merged ;)07:26
hrwmnasiadka: https://i.imgur.com/G2p2RfK.png07:27
*** shyamb has quit IRC07:45
*** gfidente has joined #openstack-kolla08:04
*** dougsz has joined #openstack-kolla08:11
mgoddardniedbalski: does openstack exporter 0.2.1 work with the version of prometheus in kolla?08:12
*** ivve has joined #openstack-kolla08:15
openstackgerritMark Goddard proposed openstack/kolla master: DNM: Use python 3.6 in CentOS source images via IUS  https://review.opendev.org/67536708:16
*** Wasaac_ has joined #openstack-kolla08:17
yoctozeptomgoddard: hi, please see my k-a fwaas patches -  I think we need to (re)discuss testing strategies regarding stuff we support: if we gated with it enabled then there would be no issue - this might probably go after we establish the support matrix but is very important for the upcoming cycles imho08:22
mgoddardyoctozepto: sure. always a tradeoff - enabling more things means longer runs and more things to go wrong08:24
mgoddardfirst thing is test matrix, then we can easily prioritise the gaps08:24
mgoddard"easily"08:24
*** jistr is now known as jistr|afk08:24
yoctozeptoyeah "easily"08:26
*** lemko has joined #openstack-kolla08:36
hrwmnasiadka: niedbalski lives few hours earlier timezone. will be in 4-6h08:39
mnasiadkaSo has anybody started to do this damn matrix?:)08:40
hrwmgoddard: https://review.opendev.org/#/c/672700/ drops releasenote about adding option which got dropped in this patch. fine or not?08:40
hrwmgoddard: yoctozepto is against, I am in favour08:41
mgoddardhrw: we don't need to change old release notes. It won't affect the release notes of current release08:41
yoctozeptohrw: I'm against because it feels weird considering how releasenotes work08:41
yoctozeptoexactly08:41
hrwok08:42
mnasiadkaWe are removing functionality without deprecation or backwards compatibility?:)08:42
mgoddardmnasiadka: yeah I'm not exactly happy about it08:43
yoctozeptomnasiadka: shh ;D08:43
mnasiadkaUhh, not nice...08:43
yoctozeptouh-oh, it's bad the way it is though08:44
*** kevinz has joined #openstack-kolla08:47
hrwanyone has alternative which handle old option?08:48
*** Wasaac_ has quit IRC08:48
hrwwithout crazy shell code to split every option, map it to daemon.json option etc?08:48
*** Wasaac_ has joined #openstack-kolla08:48
yoctozeptohrw: one coming to mind is copying the line systemd already knowns and appending to it if the old option is used08:49
mgoddardhrw: could leave only docker_custom_option in the systemd unit file, and only create it if docker_custom_option is set08:49
yoctozeptoand json for the other ones08:49
mnasiadkaYe08:50
hrwfeel free to edit patch08:50
mnasiadkaYeah, and deprecate it :)08:50
yoctozeptoit's far from urgent but we should deprecate it in train08:51
hrwwho crossed entries in https://etherpad.openstack.org/p/kolla-train-image-evaluation doc? please follow such edits with patches08:57
hrwand if we are going to remove images should we 'deprecate' them in train and remove in ucycle or just remove/08:59
*** shyamb has joined #openstack-kolla09:04
openstackgerritChandan Kumar (raukadah) proposed openstack/kolla master: [DNM] RHEL8 container build check  https://review.opendev.org/67556409:09
yoctozeptohrw: mgoddard?09:12
yoctozeptoI think only me and him edited this page09:12
yoctozeptoI added deprec as comment though09:12
yoctozeptomaybe ohwhyosa? he's the owner09:13
mgoddardI crossed out a few because the projects seemed dead, or were for kolla-k8s09:14
mgoddarddead project we should just remove09:14
hrwmgoddard: so kube* go to trash?09:15
mgoddardkolla-k8s stuff I guess people may have used, maybe we need to deprecate?09:15
mgoddardnot sure really09:15
hrwmgoddard: let make a list of 'going to deprecate' and feel free to add whatever there. then we send list to openstack-dev and wait for complaints?09:16
mgoddardhrw: problem is not all users read that list09:16
yoctozeptohrw: good start ;-)09:16
hrwmgoddard: but it will show developers that we are going to ignore their projects too09:17
mgoddardit's ok for 'going to deprecate', but probably not for 'going to remove'09:17
mgoddardhrw: true09:17
hrwdeprecate == remove. just a bit later09:17
yoctozeptoany better comm channels?09:18
hrwwe have 337 directories in docker/ and core team is quite small09:18
hrwyoctozepto: we have release notes where all those will be listed09:19
hrwand this is git - if someone will miss something then it is a matter of digging it out of history09:19
yoctozeptosure, just meant for opinion gathering09:19
hrwyoctozepto: openstack-users ML? if there is such one09:20
hrwwhereeved we try there will be people who not read09:20
yoctozeptoobviously the goal is to make as many as we can read it actually09:20
hrwI just dislike people so ignore them09:21
hrwthat's why I also not fit for PTL position09:21
yoctozeptono problem, we, aliens, hate them from time to time, then we eat them and hate is gone09:22
hrwhttps://review.opendev.org/#/c/673311/ anyone? prometheus exported update09:23
yoctozeptojoking aside, understood ;-)09:23
yoctozeptomgoddard already asked niedbalski09:23
yoctozeptore prometheus compat09:24
hrwok09:24
hrwbbiab09:24
*** Wasaac_ has quit IRC09:35
*** Wasaac_ has joined #openstack-kolla09:58
*** Wasaac_ has quit IRC10:00
*** shyamb has quit IRC10:01
*** Wasaac_ has joined #openstack-kolla10:02
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Mark several images as deprecated  https://review.opendev.org/67556810:03
hrwkind of patch to list all images. added into etherpad page10:03
*** Wasaac_ has quit IRC10:04
openstackgerritMerged openstack/kolla-ansible master: Support namespacing RabbitMQ logs  https://review.opendev.org/67539710:04
*** Wasaac_ has joined #openstack-kolla10:05
*** shyamb has joined #openstack-kolla10:17
*** jistr|afk is now known as jistr10:18
hrwok, I have set of ansible rules to set distro_python_version for k-a10:39
hrwbut where it should be used to be available for whole deployment?10:39
hrw      - name: set distro Python version10:39
hrw        set_fact:10:39
hrw          distro_python_version: "3.7"10:39
hrw        when: kolla_base_distro == 'debian'10:39
hrwthat kind - easy and working10:40
hrwansible/gather_facts.yml looks good. always called ;d10:45
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Set 'distro_python_version' variable  https://review.opendev.org/67558110:53
hrwprobably needs release note, probably should be in other file. open for reviews10:53
*** Wasaac_ has quit IRC11:01
mgoddardhrw: it doesn't need to be a task - we can put a variable in group_vars/all.yml with the necessary logic11:04
hrwmgoddard: tried, but failed11:05
mgoddardhrw: lots of set_facts can get confusing11:05
mgoddardand requires more code11:05
mgoddardI'll check your patch11:06
mgoddardhrw: do it as a map lookup: distro_python_version: "{{ distro_python_version_map[kolla_base_distro] }}"11:07
mgoddardanyway, off now11:07
mgoddardo/11:07
hrwo!11:07
*** Wasaac has joined #openstack-kolla11:08
*** Wasaac has left #openstack-kolla11:09
*** Wasaac has joined #openstack-kolla11:09
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Set 'distro_python_version' variable  https://review.opendev.org/67558111:11
*** kplant has joined #openstack-kolla11:11
openstackgerritMichal Nasiadka proposed openstack/kolla master: Add OPENSTACK_RELEASE to before-rc1  https://review.opendev.org/67558811:14
mnasiadkahmm11:18
mnasiadkaseems we have a grafana build error11:18
mnasiadkahttps://logs.opendev.org/86/675386/2/gate/kolla-build-centos-source/f15bf1b/kolla/build/000_FAILED_grafana.txt.gz11:18
*** gfidente has quit IRC11:21
*** dpawlik has quit IRC11:21
*** dpawlik has joined #openstack-kolla11:22
hrw[Fri Aug 09 11:15:26.249697 2019] [wsgi:error] [pid 36:tid 281473679282576] WARNING:root:Use of this 'djano.wsgi' file has been deprecated since the Rocky release in favor of 'wsgi.py' in the 'openstack_dashboard' module. This file is a legacy naming from before Django 1.4 and an importable 'wsgi.py' is now the default. This file will be removed in the T release cycle.11:23
hrwhorizon11:23
hrw[Fri Aug 09 11:15:29.631565 2019] [wsgi:error] [pid 36:tid 281473679282576] [remote 172.16.101.38:38926] ERROR:django.security.DisallowedHost:Invalid HTTP_HOST header: '10.101.16.4'. You may need to add '10.101.16.4' to ALLOWED_HOSTS.11:23
hrwha! that's why horizon do not like me11:23
hrwbut ALLOWED_HOSTS = ['*'] is in config.11:28
*** gfidente has joined #openstack-kolla11:29
hrwsomeone has train deployed with working horizon? debian or ubuntu11:30
*** kevinz has quit IRC11:32
*** kevinz has joined #openstack-kolla11:32
hrwhttps://review.opendev.org/#/c/674886/ guys11:33
*** rohit02 has joined #openstack-kolla11:34
kplantthat is so much cleaner11:46
hrw[Fri Aug 09 11:51:31.741083 2019] [wsgi:error] [pid 31:tid 281473325560208] WARNING:openstack_dashboard.settings:No local_settings file found.11:52
hrwone step at time11:52
kplantin containers where /var/run/docker.sock is bound, would it make sense to also create a group called "docker" or "dockerhost" with the same gid as the "docker" group on the host?11:55
kplantsomething like 'groupadd -g $(getent group docker | cut -d: -f3) dockerhost' in the entrypoint?11:55
*** kevinz has quit IRC12:01
*** rohit02 has quit IRC12:04
openstackgerritMerged openstack/kolla-ansible master: Fix FWaaS service provider (v2, Stein issue)  https://review.opendev.org/67543912:05
*** shyamb has quit IRC12:07
hrwkplant: what for?12:14
openstackgerritMerged openstack/kolla stable/queens: ceph mds cap 'allow *' in extend_start  https://review.opendev.org/67504312:15
kplantwell, otherwise the socket is not readable by anyone who isn't root12:15
kplantif you present a docker group within the container, you could easily pop a user in there and allow them to read from the socket12:15
hrwif you exec into running container then '-u root' and done12:17
kplantright, but for something like telegraf - that shouldn't run as root12:17
kplantit'd be much easier to 'gpasswd -a telegraf docker'12:17
hrwkplant: if for some reason you need access to something in running container then 'exec -u root' gives you shell in it. which has nothing with what runs there by default12:19
hrwright?12:19
kplantyes but i'm not talking about shell12:19
kplanti'm talking about telegraf gathering stats from docker about what the containers on my hosts are doing12:19
kplantso i can then have them in grafana12:19
hrwah, now you are talking12:20
kplant(-:12:20
hrwkplant: submit a patch?12:21
kplanthappy to - i'm just trying to think the best way to do it12:23
kplantand if it's even a good idea12:23
*** kevinz has joined #openstack-kolla12:34
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Bump up fluentd  https://review.opendev.org/67561412:35
hrwshit. looks like we still need systemd overwrite ;(12:38
hrwCI use of -H flag12:38
hrw"Note: You cannot set options in daemon.json that have already been set on daemon startup as a flag. On systems that use systemd to start the Docker daemon, -H is already set, so you cannot use the hosts key in daemon.json to add listening addresses. "12:39
hrwRT@#%@!RRFWJEIOGHFIO~!12:48
hrwguys: I need ExecStart line from /lib/systemd/system/docker.service on ubuntu and centos12:54
hrws/guys/folks12:54
*** kevinz has quit IRC12:55
kplantsure12:55
kplant1s12:55
kplantcentos: `ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock`12:55
kplantthat ives in /usr/lib/systemd/system/docker.service12:56
hrwthanks12:56
hrwkplant: that's docker-ce 18.09+, right?12:56
kplant[root@chrnc-dev-os-control-01 ~]# docker --version12:56
kplantDocker version 18.09.7, build 2d0083d12:56
hrwcool12:57
*** kevinz has joined #openstack-kolla12:57
*** henriqueof has joined #openstack-kolla12:58
goldyfruit_o/12:58
*** Guest1857 has joined #openstack-kolla12:59
*** mkarpiarz has joined #openstack-kolla13:02
hrwuf. docker-ce has same docker.service on centos/debian/ubuntu13:03
hrwyoctozepto, mnasiadka: let kill legacy docker: https://review.opendev.org/#/c/674886/13:05
hrwI then can remove 'dockerd or docker daemon' part13:07
*** Wasaac_ has joined #openstack-kolla13:10
*** kevinz has quit IRC13:23
*** kevinz has joined #openstack-kolla13:24
*** kevinz has quit IRC13:25
*** BjoernT has joined #openstack-kolla13:28
*** jistr is now known as jistr|call13:29
hrwnewer version of 672700 looks promising13:29
*** Wasaac_ has quit IRC13:32
*** Wasaac__ has joined #openstack-kolla13:33
*** ivve has quit IRC13:33
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Modernize a way of configuring Docker daemon  https://review.opendev.org/67270013:37
hrwhope to include all comments13:37
*** jistr|call is now known as jistr13:38
*** Wasaac__ has quit IRC13:48
*** Wasaac_ has joined #openstack-kolla13:48
*** Wasaac_ has quit IRC13:48
*** spsurya has quit IRC13:54
*** cah_link has quit IRC14:25
*** dpawlik has quit IRC14:26
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix FWaaS service provider (v2, Stein issue)  https://review.opendev.org/67563414:26
dougszWhich flavours of machines do we have in CI? Are they all the same, or can you choose according to the job?14:26
yoctozeptohrw: done, now: https://review.opendev.org/67411414:26
yoctozeptodougsz: mix and match14:27
yoctozeptonow ubuntu deploy ubuntu, centos centos14:27
yoctozeptoaio, multinode14:27
yoctozepto;D14:27
dougszThose are defined in the Vagrant file?14:27
openstackgerritMerged openstack/kolla-cli master: Move kolla prechecks action into kolla_actiom model  https://review.opendev.org/67514014:28
yoctozeptodougsz: no, vagrant is something abandoned, it does not work properly atm afaik14:29
yoctozeptoCI is defined in the *zuul* files14:29
yoctozeptohrw: why not do systemctl show to get the startpath?14:30
dougszyeah it's borked by default, but it does work if you bring it up with `--no-parallel`!14:30
yoctozeptoalso, why modify -H?14:30
openstackgerritMerged openstack/kolla-cli master: Use inventory object to validate the hosts and services  https://review.opendev.org/67554214:30
dougszthanks yoctozepto14:30
yoctozeptodougsz: could be, did not try further because it uses unsupported kolla-cli anyway14:30
yoctozeptothere is some progress in it though14:31
yoctozeptono problem dougsz14:31
dougszthat's true14:31
yoctozeptodougsz: you have some Polish roots? your surname sounds Polish14:31
openstackgerritMerged openstack/kolla-cli master: Use kolla-cli owner url as the home page  https://review.opendev.org/67240614:36
dougszyoctozepto: 25% polish, my grandfather was from Volhynia, hometown now demolished14:36
*** Wasaac has quit IRC14:37
*** Wasaac has joined #openstack-kolla14:37
yoctozeptodougsz: understood, sad14:37
*** ivve has joined #openstack-kolla14:39
*** cah_link has joined #openstack-kolla14:39
openstackgerritzhulingjie proposed openstack/kolla-cli master: Move stop acton to kolla model from host  https://review.opendev.org/67550214:43
yoctozeptohrw: checked, that was for Zun, uh-huh14:44
yoctozeptostill, the point about using systemctl show stands14:44
yoctozeptogive it a whirl14:45
hrwyoctozepto: https://review.opendev.org/674114 fails on ceph again?15:01
hrwyoctozepto: screw 'ssytemctl show'. all supported host systems use docker-ce and all have the same systemd/docker.service15:03
*** shyamb has joined #openstack-kolla15:15
*** shyamb has quit IRC15:20
*** shyam89 has joined #openstack-kolla15:20
*** shyam89 has quit IRC15:20
openstackgerritzhulingjie proposed openstack/kolla-cli master: Add --hosts and --services support for reconfigure action  https://review.opendev.org/67564315:25
openstackgerritMerged openstack/kolla-ansible master: Remove support for Docker legacy packages  https://review.opendev.org/67488615:27
hrwyay!15:27
hrwno more 'docker daemon'15:27
openstackgerritzhulingjie proposed openstack/kolla-cli master: [WIP] add reconfigure unit test  https://review.opendev.org/67564615:31
yoctozeptohrw: one failed mariadb but the other ceph-nfs again15:34
hrw;(15:34
yoctozeptothat was 5 Aug15:35
yoctozeptocould be not all patches in?15:35
yoctozeptoyeah, in 6 Aug15:35
yoctozeptohence why15:36
hrwok, once pass will push it15:39
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Modernize a way of configuring Docker daemon  https://review.opendev.org/67270015:43
hrwtabs removed15:44
*** atmark has quit IRC16:08
*** atmark has joined #openstack-kolla16:11
openstackgerritgugug proposed openstack/kolla-cli master: Move stop acton to kolla model from host  https://review.opendev.org/67550216:12
*** henriqueof has quit IRC16:14
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Copy Nova role as a basis for the Nova cell role  https://review.opendev.org/67565816:15
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: [WIP] Factor out nova-cell role  https://review.opendev.org/67565916:15
*** Guest1857 is now known as redrobot16:17
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Fixes Monasca log transformer UTC offset exception  https://review.opendev.org/67566016:21
*** jonaspaulo has joined #openstack-kolla16:30
*** rpittau is now known as rpittau|afk16:35
openstackgerritMark Goddard proposed openstack/kolla master: DNM: Use python 3.6 in CentOS source images via IUS  https://review.opendev.org/67536716:44
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Copy Nova role as a basis for the Nova cell role  https://review.opendev.org/67565816:44
*** jovial[m] has quit IRC16:45
*** ohwhyosa has quit IRC16:45
*** irclogbot_0 has quit IRC16:47
*** irclogbot_1 has joined #openstack-kolla16:48
*** irclogbot_1 has quit IRC16:49
*** irclogbot_1 has joined #openstack-kolla16:50
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add custom filters for checking services  https://review.opendev.org/66592216:54
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check containers during kolla-ansible check  https://review.opendev.org/59973516:54
*** dougsz has quit IRC16:59
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561417:04
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check for CRITICAL, WARNING and ERROR log messages in CI  https://review.opendev.org/64958317:04
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Run nova-status upgrade check after DB sync and migrations  https://review.opendev.org/66564317:09
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561417:12
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check for CRITICAL, WARNING and ERROR log messages in CI  https://review.opendev.org/64958317:15
*** igordc has joined #openstack-kolla17:17
*** lemko has quit IRC17:33
*** lemko has joined #openstack-kolla17:47
*** gfidente has quit IRC18:05
*** factor has quit IRC18:06
*** icarusfactor has joined #openstack-kolla18:06
openstackgerritMerged openstack/kolla-ansible stable/stein: Remove FWaaS v1 related options  https://review.opendev.org/66840318:24
openstackgerritKeith Plant proposed openstack/kolla-ansible master: Configure Telegraf to monitor Docker containers  https://review.opendev.org/67542118:31
*** BjoernT has quit IRC19:44
*** kplant has quit IRC19:53
*** whoami-rajat has quit IRC20:12
*** lemko has quit IRC20:43
*** ktibi has joined #openstack-kolla20:59
*** EmilienM is now known as EmilienM|pto21:01
*** igordc has quit IRC21:01
*** jonaspaulo has quit IRC22:23
*** cah_link has quit IRC22:49
*** cah_link has joined #openstack-kolla22:52
*** cah_link has quit IRC22:57
*** ktibi has quit IRC23:32
*** weshay is now known as weshay_pto23:37
« Thursday, 2019-08-08 Index Saturday, 2019-08-10 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!