| *** yolanda has quit IRC | 04:18 | |
| *** bhavik1 has joined #zuul | 05:28 | |
| tobiash | jeblair: hi, I have a question about https://review.openstack.org/#/c/396719/ | 05:49 |
|---|---|---|
| tobiash | is snapshot support really going to be removed? | 05:49 |
| tobiash | I currently have two use cases which rely on this mechanism | 05:50 |
| tobiash | I didn't manage to prepopulate docker images during diskimage build | 05:51 |
| tobiash | and creating windows images is also not quite easy with diskimage builder | 05:51 |
| tobiash | my current workflow for linux nodes is creating them via diskimage builder and use that as base image | 05:56 |
| tobiash | the setup scripts then iterate over a specific tag in a docker registry and fetch all images with this tag | 05:57 |
| tobiash | for that step docker needs to run which I didn't get to work during diskimage creation | 05:58 |
| *** abregman has joined #zuul | 06:05 | |
| *** pabelanger has quit IRC | 06:28 | |
| *** pabelanger has joined #zuul | 06:34 | |
| *** yolanda has joined #zuul | 07:33 | |
| *** abregman_ has joined #zuul | 08:37 | |
| *** abregman has quit IRC | 08:39 | |
| *** abregman_ has quit IRC | 08:47 | |
| *** abregman has joined #zuul | 08:47 | |
| *** hashar has joined #zuul | 08:51 | |
| *** abregman has quit IRC | 09:12 | |
| *** abregman has joined #zuul | 09:18 | |
| *** openstackgerrit has quit IRC | 09:47 | |
| *** openstackgerrit has joined #zuul | 09:48 | |
| *** bhavik1 has quit IRC | 09:54 | |
| *** bcoca has joined #zuul | 13:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable TestMergerRepo() class for testing https://review.openstack.org/397189 | 13:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable TestMergerRepo() class for testing https://review.openstack.org/397189 | 13:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_head_is_dequeued_once test https://review.openstack.org/396803 | 13:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_noop_job test https://review.openstack.org/396807 | 13:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_merger_repack_large_change test https://review.openstack.org/396805 | 13:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Remove includes and parameter functions https://review.openstack.org/394644 | 13:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Add tenant support to zuul client https://review.openstack.org/393850 | 13:47 |
| *** abregman is now known as abregman|mtg | 15:44 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_gate test https://review.openstack.org/397277 | 16:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_gate / test_crd_multiline / test_crd_gate_reverse https://review.openstack.org/397277 | 16:26 |
| *** abregman|mtg is now known as abregman | 16:27 | |
| *** abregman has quit IRC | 17:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_check job https://review.openstack.org/397317 | 17:32 |
| pabelanger | jeblair: clarkb: mordred: ^ was a little tricky, in getting access to the JobDir of a build. Let me know what you think of that approach | 17:32 |
| *** rattboi has joined #zuul | 17:34 | |
| mordred | pabelanger: it doesn't seem extra crazy to me | 17:34 |
| clarkb | thats because zuul is cleaning up the jobdir on its own when job completes? | 17:35 |
| pabelanger | Ya | 17:36 |
| pabelanger | we could override the JobDir path, maybe place it into the test_root path for ZuulTestCase. Use JobDir(keep=True), then let the ZuulTestCase handler did the test_root folder | 17:38 |
| pabelanger | then we don't need to hold_jobs_in_build | 17:38 |
| clarkb | I think thats probably more invasive | 17:38 |
| pabelanger | for sure | 17:39 |
| clarkb | the upside to this is its fairly minimal, basically just don't execute tests while we check things | 17:39 |
| clarkb | I like this solution | 17:40 |
| pabelanger | ack | 17:44 |
| *** hashar is now known as hasharAway | 17:51 | |
| *** Shuo has joined #zuul | 17:55 | |
| Shuo | how to start participating in zuul v3 and take a task from the storyboard? https://storyboard.openstack.org/#!/story/2000791 | 18:09 |
| pabelanger | Shuo: we should be having a meeting today at 22:00UTC for zuul, I suspect we'll be going over a few things | 18:31 |
| Shuo | pabelanger: what time is 22:00UTC for Pacific Time? And also, is the meeting conducted through this channel? (Love to participate) | 18:32 |
| clarkb | Shuo: it is 1833UTC right now | 18:33 |
| pabelanger | Shuo: http://eavesdrop.openstack.org/#Zuul_Meeting for more info | 18:33 |
| pabelanger | lists time and location | 18:33 |
| Shuo | pabelanger: clarkb: cool, thanks. | 18:35 |
| Shuo | pabelanger: looks like today the first meeting? Link from "Logs from past meetings" is empty (not accessible). | 18:39 |
| pabelanger | Shuo: yup! | 18:40 |
| pabelanger | Shuo: prior to todays meeting, we used the project infrastructure meeting timeslot for some zuulv3 discussions. | 18:41 |
| * Shrews wonders about the agenda | 18:41 | |
| jeblair | Shrews: 'create an agenda' is on the agenda. it's a real bootstrapping meeting. :) | 18:46 |
| jeblair | tobiash: that's the plan -- at least for the work we're doing in preparation for zuulv3. we stopped using snapshot image builds a long time ago.... maybe greghaynes or one of the other dib folks could help address the docker issue you have? | 18:51 |
| jeblair | tobiash: however, if folks really want a snapshot image builder, and are willing to step up and do some work on it, it *is* compatible with the new design of nodepool, and i would be happy to help describe how to implement it. | 18:53 |
| clarkb | (I am going to guess the docker daemon and needing a socket are the problems with doing that in dib, likely solveable by using "outer" docker to populate store in chroot) | 18:55 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_branch test https://review.openstack.org/397337 | 18:59 |
| Shuo | jeblair: regarding image-base or not, are we talking about the zuul installation itself? | 18:59 |
| jeblair | Shuo: no, that's an unrelated topic about how nodepool gets the images it uses to launch worker nodes. we used to use a snapshot approach (launch a server, do stuff, snapshot the vm, then use that to launch the worker nodes). but we switched to diskimage-builder because the image ended up being different in every cloud we used (because they started out different). | 19:01 |
| Shuo | pabelanger: saw your update at https://storyboard.openstack.org/#!/story/2000791, where is your repo on github? (you gave the openstack git endpoint, but not github one) | 19:05 |
| pabelanger | Shuo: we don't do any development on github, just a best effort to mirror there. I'd recommend using git.o.o. The mirror is http://github.com/openstack | 19:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_cycle_join test https://review.openstack.org/397340 | 19:13 |
| *** phschwartz has joined #zuul | 19:23 | |
| openstackgerrit | Merged openstack-infra/nodepool: Transition ZK API from dict to object model https://review.openstack.org/394592 | 19:23 |
| Shuo | pabelanger: does your ansible script assume an all-in-one situation? | 19:23 |
| openstackgerrit | Merged openstack-infra/nodepool: Combine ZKTestCase with DBTestCase https://review.openstack.org/383962 | 19:24 |
| pabelanger | Shuo: no, I've done my best to make the role multi host | 19:24 |
| openstackgerrit | Merged openstack-infra/nodepool: Supply ZK connection information to test configs https://review.openstack.org/383963 | 19:25 |
| openstackgerrit | Merged openstack-infra/nodepool: Add getMostRecentBuildImageUpload method to zk https://review.openstack.org/383964 | 19:25 |
| openstackgerrit | Merged openstack-infra/nodepool: Assume diskimage and image names are the same https://review.openstack.org/383965 | 19:25 |
| openstackgerrit | Merged openstack-infra/nodepool: Update waitForImage test method for ZK https://review.openstack.org/383966 | 19:25 |
| openstackgerrit | Merged openstack-infra/nodepool: Use diskimage name when looking up image on disk https://review.openstack.org/396422 | 19:25 |
| openstackgerrit | Merged openstack-infra/nodepool: Override the cleanup interval in builder fixture https://review.openstack.org/396423 | 19:26 |
| openstackgerrit | Merged openstack-infra/nodepool: Add __repr__ methods to ZK objects https://review.openstack.org/396424 | 19:26 |
| Shrews | yay for merging the things | 19:26 |
| pabelanger | http://git.openstack.org/cgit/openstack/windmill/tree/playbooks/group_vars/zuul-merger.yaml would be an example of how to only setup zuul-merger | 19:26 |
| openstackgerrit | Merged openstack-infra/nodepool: Re-enable test_dib_image_list https://review.openstack.org/383967 | 19:26 |
| pabelanger | Shrews: exciting | 19:27 |
| jeblair | Shrews: yeah, my last change failed in zuul, but that test passes when i run locally. :/ | 19:28 |
| openstackgerrit | Merged openstack-infra/nodepool: Re-enable test_image_list_empty https://review.openstack.org/396449 | 19:28 |
| jeblair | i rechecked to see if it's transient | 19:28 |
| Shrews | i do not know how to make my sb TODO go away, but maybe SpamapS can do things with it | 19:28 |
| SpamapS | Shrews: did you add a Task: header? | 19:29 |
| Shuo | pabelanger: quite a while ago, I did something of Vagrant + Ansible (for a different stack we used back end, and a lot of further work has been moved to internal development) https://github.com/shuoy/logging-for-openstack. Basically, the idea/goal of that was bring up a demo from a laptop. Let me explain a bit to see if the idea is compatible to your work... | 19:29 |
| Shrews | SpamapS: i can basically do nothing with storyboard since i have no permissions. i'm failing to see its usefulness, tbh | 19:29 |
| SpamapS | whhherrrrt? | 19:29 |
| SpamapS | oh the board | 19:29 |
| SpamapS | Shrews: I said in the email, just ask for perms if you want to move the cards. | 19:29 |
| Shuo | 1. we'd like to isolate the demo environments from what kind of hosts the demo runs on. So, all the ansible work are on top of vagrant VMs. | 19:30 |
| SpamapS | Shrews: you have card moving access. | 19:30 |
| Shuo | pabelanger: 1. we'd like to isolate the demo environments from what kind of hosts the demo runs on. So, all the ansible work are on top of vagrant VMs. | 19:30 |
| SpamapS | Shrews: but all that I really want to make sure is that people claim work so we don't re-work stuff | 19:31 |
| pabelanger | Shuo: I have been toying around with the idea of using lxc to further isolate things, for testing / demo at least. | 19:31 |
| Shuo | pabelanger: 2. the kind of VMs cluster (be it a single node or multiple nodes) is configurable by the demo-er (simpley change a yaml file) | 19:32 |
| pabelanger | Shuo: otherwise, I use a cloud for my local testing, where I spin up multiple VMs | 19:32 |
| Shuo | pabelanger: after the 'substrate' VM cluster is setup through vagrant up, run the ansible script in the bootstrap machine and the ansible script can automatically reflect the nature of the vagrant cluster. | 19:33 |
| Shuo | pabelanger: in this idea, we can easily migrate the work from vagrant to AWS instnaces. | 19:34 |
| pabelanger | sure, that works. That is great thing about ansible, just need an SSH connection and things should work as expected. Assuming both are running the same OS | 19:34 |
| clarkb | ansible can even spin up the VMs/containers | 19:35 |
| clarkb | no need for vagrant | 19:35 |
| pabelanger | yup | 19:35 |
| pabelanger | I haven't had time to use the lxc task, but the ansible openstack team uses it I believe | 19:35 |
| pabelanger | obviously, os_server works well too :) | 19:36 |
| *** abregman has joined #zuul | 19:38 | |
| jeblair | pabelanger: i left a comment on the tenant/json change | 19:38 |
| pabelanger | jeblair: ack, just looking now | 19:39 |
| Shrews | I find it odd that our Backlog list on our board does not list all tasks within an individual story, only one. | 19:44 |
| Shrews | Is that how it's supposed to work? | 19:44 |
| Shrews | If so, it should just use the story title and not a random task title from the story, IMO | 19:45 |
| Shrews | perhaps i should take this to #storyboard :) | 19:45 |
| jeblair | Shrews: i think a list can have either stories or tasks in it | 19:47 |
| jeblair | Shrews: and i believe they were/are manually populated | 19:47 |
| jeblair | Shrews: i would expect a story entry to have the story title, though. does one not? | 19:47 |
| Shrews | jeblair: https://storyboard.openstack.org/#!/story/2000768, for example | 19:48 |
| Shrews | shows up as "Nodepool changes" in our Backlog, though i just manually edited that one. Perhaps i missed a step | 19:48 |
| Shrews | same with https://storyboard.openstack.org/#!/story/2000767, which i did not change | 19:50 |
| Shrews | I've yet to see one that isn't using a task title, TBH | 19:51 |
| jeblair | Shrews: which task title in 2000767 do you see in the backlog? | 19:53 |
| Shrews | jeblair: "switch nodepool...", though i moved it to In Progress from Todo | 19:54 |
| jeblair | Shrews: i believe that is a task entry | 19:55 |
| Shrews | jeblair: yep, which is my point | 19:55 |
| Shrews | where are the other tasks? | 19:55 |
| jeblair | Shrews: they have not been added. but can be added manually | 19:55 |
| jeblair | Shrews: (there is nothing automatic about this workboard) | 19:56 |
| Shuo | pabelanger: after cloning the repo, what am I suppose to do -- the doc does not seem to instruct it? (I am on an ec2 instance) | 19:58 |
| greghaynes | tobiash: Hey, dib should be able to make docker images just fine, if it cant for some reason I'd be happy to look in to it. The windows thing is wierd, I dont know what making a windows image would even mean since MS releases them, dib is totally capable of wgeting an image and outputting it though... | 20:01 |
| clarkb | greghaynes: I think in this case its not making a docker image, its fetching docker images from a docker registry as part of the dib image build | 20:03 |
| greghaynes | aha | 20:04 |
| clarkb | greghaynes: but that likely doesn't work due to socket and docker daemon | 20:04 |
| greghaynes | I wonder if theres some way to just cache the image data and feed it in to docker later | 20:04 |
| greghaynes | the docker image data, that is | 20:05 |
| pabelanger | Shuo: it is an ansible role, so you'll need to build up a playbook to consume it | 20:05 |
| greghaynes | tobiash: one option is we could make an element which docker save's a bunch of images from the hypervisor, copies them in to the image, then docker load's them on boot | 20:09 |
| clarkb | greghaynes: tobiash or maybe use an external docker process to write into the chroot | 20:10 |
| greghaynes | ooo | 20:10 |
| clarkb | a lot of our caching setup works that way so you get a cache :) | 20:10 |
| greghaynes | Yea, if someone has the docker-fu to know how to do that it'd be awesome | 20:10 |
| SpamapS | Shrews: many of the stories didn't have any tasks broken out, so they were just added as a single task. The ones that were added as stories were those that, by working, would naturally generate more tasks. | 20:18 |
| SpamapS | Shrews: I think we'll work on a better set of instructions (something shorter than a 6 paragraph email) that explains what we're tryign to do with the board, and maybe relieves you from feeling uncertain about what to do with it. | 20:22 |
| pabelanger | jeblair: I'm going to rebase away the tenant/json change, since there might be some back and forth on that change. And don't want to spam 15 patches at a time | 20:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_check job https://review.openstack.org/397317 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_check_smtp_pool test https://review.openstack.org/396707 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable TestMergerRepo() class for testing https://review.openstack.org/397189 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_live_reconfiguration test https://review.openstack.org/393488 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_disable_at test https://review.openstack.org/396785 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_check_reconfiguration test https://review.openstack.org/396788 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_head_is_dequeued_once test https://review.openstack.org/396803 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_gate / test_crd_multiline / test_crd_gate_reverse https://review.openstack.org/397277 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_cycle_join test https://review.openstack.org/397340 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_repo_deleted test https://review.openstack.org/396703 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_check_unknown test https://review.openstack.org/396798 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_gate_unknown / test_crd_undefined_project https://review.openstack.org/396799 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_noop_job test https://review.openstack.org/396807 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_crd_branch test https://review.openstack.org/397337 | 20:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Re-enable test_merger_repack_large_change test https://review.openstack.org/396805 | 20:34 |
| *** abregman has quit IRC | 20:55 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Update webapp status json to support tenants https://review.openstack.org/391681 | 21:00 |
| jhesketh | Morning | 21:26 |
| *** rcarrillocruz has joined #zuul | 21:30 | |
| jeblair | reminder: our first meeting is in ~30 mins | 21:31 |
| rcarrillocruz | ++ | 21:36 |
| mordred | morning jhesketh ! | 21:37 |
| jlk | jeblair: which channel? | 21:53 |
| *** rmoe has joined #zuul | 21:56 | |
| jeblair | jlk: #openstack-meeting-alt (http://eavesdrop.openstack.org/#Zuul_Meeting for reference) | 21:57 |
| jeblair | hey everyone! zuul meeting now in #openstack-meeting-alt yay! | 22:00 |
| pabelanger | yay | 22:00 |
| jhesketh | sigh, my teleconf is running late (sorry I didn't join earlier) | 22:18 |
| jeblair | pabelanger: my thought is that if you have run into a test where you tried to enable it, but found it too complex -- how do we make sure people can see that? | 23:01 |
| *** rcarrillocruz has quit IRC | 23:03 | |
| pabelanger | jeblair: Good question, I've been trying to hack on them locally. But likely need a place to track them | 23:03 |
| pabelanger | test_client_enqueue_ref is a good example | 23:03 |
| pabelanger | I have code for it, but don't understand why right now | 23:03 |
| pabelanger | test_crd_cycle_join was another one, but I just figured that out this morning. I was working on it off and on for the last 3 days | 23:05 |
| pabelanger | JobDir() location for git_root | 23:05 |
| jeblair | pabelanger: how about you make a task for it, and add some notes to it? feel free to assign yourself or not depending on whether you want to claim it, but at least this way, there will be some history and shared information | 23:05 |
| jeblair | pabelanger: also, i think at some point, you should work with me on the issues | 23:06 |
| pabelanger | jeblair: sure, I can make a few issues in the morning, dump some history into it. | 23:07 |
| Shuo | to understand how zuul serializes the incoming commits and speculative executing it, which piece of code should I start reading? | 23:07 |
| pabelanger | jeblair: agreed, I don't think it will take long to dive into the issues. Like you said, just scheduling the time for it. | 23:07 |
| jeblair | basically, i should have a work queue. so when you hit a wall, put something into my queue and i'll pop things off | 23:07 |
| jeblair | pabelanger: ^ | 23:08 |
| pabelanger | jeblair: okay | 23:08 |
| jeblair | Shuo: that's the pipeline managers. independentpipelinemanager and dependentpipelinemanager share a parent class basepipelinemanager | 23:08 |
| clarkb | jeblair: re https://review.openstack.org/#/c/396719/ and cleaning things up, that might be a good opportunity to rename the snapshot image db table? | 23:12 |
| clarkb | though probably needs to be a followup change since thats probably not the smallest change | 23:12 |
| jeblair | clarkb: well, actually it needs to go away | 23:12 |
| jeblair | since it's replaced by zookeeper | 23:13 |
| clarkb | oh right nodepool will check zk for those uuids | 23:13 |
| jeblair | yeah, so building on that change should include changes which switch to using the zk api to get the image uuid and also remove the snapshot db table | 23:14 |
| jeblair | Shrews, pabelanger: ^ | 23:14 |
| mordred | jeblair: didn't I already see a patchto use the zk api to get the image uuid? or am I imagining that? | 23:14 |
| jeblair | (as well as changes to make the commands work, and changes to make the tests work) | 23:14 |
| clarkb | mordred: I think that was just enough change to make a single test pass, likely needs much more updatse like that | 23:15 |
| jeblair | mordred: i wrote the method that will be used (the "get me the right image" method), and yeah, used it in a test as clarkb says | 23:15 |
| pabelanger | I have to step away now, but will catch up on the backscroll when I return | 23:15 |
| jeblair | clarkb: though honestly, that's just about the only interaction nodepoold needs to have with it | 23:15 |
| clarkb | jeblair: I think there are other places like command to list images | 23:16 |
| clarkb | also I think allocation may have somethign that double checks it can create instances if an image exists for it? | 23:16 |
| jeblair | clarkb: yes, the commands need updating, but i'm pretty sure nodepoold (note the d) only needs that single interface with zk. | 23:16 |
| jeblair | clarkb: i think the allocator will use that same method | 23:18 |
| clarkb | ah ok | 23:18 |
| jeblair | the successor to getCurrentSnapshotImage(provider, image) | 23:19 |
| jeblair | which shows up 4 times in nodepoold (launch node, launch subnode, allocator, and, weirdly, right before launching a node) | 23:21 |
| Shuo | jeblair: how does this queue, i.e., the final source of truth for the sequence, get persisted? so that the enqueue and dequeue with mutex can operate on this central piece of information? | 23:28 |
| jeblair | Shuo: it's just an in-memory data structure; it doesn't persist at all | 23:33 |
| jeblair | Shuo: the triggers (ie, gerrit) push events into an event queue, and the scheduler processes those events in order, some of which may cause items to be enqueued into the pipelines | 23:34 |
| jeblair | Shuo: (the pipeline managers then handle the sequencing inside of the pipeline and running jobs in parallel) | 23:35 |
| Shuo | jeblair: thanks. if zuul scheduler process gets bogged down for some reason, would that cause any issue (e.g., lose some tests)? | 23:36 |
| jeblair | Shuo: no, it just gets very slow | 23:36 |
| Shuo | jeblair: I may used the wrong term. I meant when the server/process holding that piece of in-memory data structure (which is 'pipelines' I think) is restarted, the items that triggers/clients believe they have successfully enqued for future processing will be lost along with that restart, but the clients/triggers does not know 'something is lost' | 23:41 |
| Shuo | jeblair: what effect this restart would cause? out-of-order or some commits won't be processed for a period of time? | 23:42 |
| *** phschwartz has quit IRC | 23:49 | |
| *** phschwartz has joined #zuul | 23:49 | |
| *** phschwartz has quit IRC | 23:50 | |
| *** phschwartz has joined #zuul | 23:51 | |
| *** hasharAway has quit IRC | 23:52 | |
| jeblair | Shuo: oh yeah, when it's restarted all state is lost | 23:57 |
| jeblair | we should improve that in v3 | 23:57 |
| *** Shuo has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!