| *** markvoelker has quit IRC | 00:06 | |
| *** jbadiapa has quit IRC | 00:10 | |
| *** ironfoot has joined #openstack-ansible | 00:12 | |
| *** ansmith has quit IRC | 00:17 | |
| *** macza has quit IRC | 00:18 | |
| *** ironfoot has quit IRC | 00:24 | |
| *** ironfoot has joined #openstack-ansible | 00:27 | |
| *** ironfoot has quit IRC | 00:31 | |
| *** tosky has quit IRC | 00:32 | |
| *** ironfoot has joined #openstack-ansible | 00:32 | |
| *** ansmith has joined #openstack-ansible | 00:59 | |
| *** strobelight has joined #openstack-ansible | 01:01 | |
| cloudnull | evenings | 01:23 |
|---|---|---|
| *** markvoelker has joined #openstack-ansible | 01:35 | |
| *** markvoelker has quit IRC | 01:40 | |
| jamesdenton | howdy | 01:40 |
| jamesdenton | cloudnull i see you're implementing skydive? | 01:49 |
| cloudnull | indeed :) | 01:49 |
| cloudnull | https://github.com/openstack/openstack-ansible-ops/tree/master/skydive | 01:49 |
| jamesdenton | subscribed | 01:49 |
| jamesdenton | just didn't have time to play | 01:49 |
| cloudnull | ^ its still a little rough, though its been implemented | 01:49 |
| * cloudnull currently working on improving the docs | 01:50 | |
| jamesdenton | right on | 01:50 |
| cloudnull | its cool though, works great in my home lab | 01:50 |
| jamesdenton | i will be trying it out this week, hopefully. full plate, so we'll see | 01:50 |
| cloudnull | probably need to pick some brains on getting the default fabric generation to be right | 01:50 |
| cloudnull | but works well enough | 01:51 |
| * cloudnull from my laymen perspective | 01:51 | |
| jamesdenton | so humble | 01:51 |
| jamesdenton | you have time for a quick review? | 01:52 |
| *** ansmith has quit IRC | 01:54 | |
| *** mino_ has joined #openstack-ansible | 01:58 | |
| *** markvoelker has joined #openstack-ansible | 01:59 | |
| *** macza has joined #openstack-ansible | 02:06 | |
| *** markvoelker has quit IRC | 02:07 | |
| prometheanfire | cloudnull: did you check out the rebase failures? | 02:09 |
| prometheanfire | for networkd stuffs | 02:09 |
| prometheanfire | https://review.openstack.org/592596 | 02:09 |
| *** macza has quit IRC | 02:11 | |
| *** TxGirlGeek has quit IRC | 02:23 | |
| *** strobelight_ has joined #openstack-ansible | 02:37 | |
| *** strobelight is now known as Guest97276 | 02:37 | |
| *** strobelight_ is now known as strobelight | 02:39 | |
| *** Guest97276 has quit IRC | 02:41 | |
| *** vnogin has joined #openstack-ansible | 02:46 | |
| *** vnogin has quit IRC | 02:51 | |
| partlycloudy | hi folks, i still have some questions about the ceph production example at https://docs.openstack.org/openstack-ansible/rocky/user/ceph/full-deploy.html. | 02:51 |
| partlycloudy | in this example, if we use br-storage for ceph replication/recovery traffic, why we still need to assign ips to the br-storage on compute nodes, as per the comment suggested? | 02:52 |
| * cloudnull sadly sucks at just about all things ceph | 02:55 | |
| cloudnull | I believe the IP requirements is to ensure the compute nodes have a route back to the storage nodes. | 02:56 |
| cloudnull | prometheanfire i've not | 02:57 |
| cloudnull | jamesdenton I do whats up ? | 02:58 |
| partlycloudy | in the Cookbook (4th ed), i see br-storage is used by infra/compute/storage nodes as the public network. | 03:01 |
| partlycloudy | hence i understand the purpose of having ips to the br-storage on compute nodes. | 03:03 |
| *** strobelight has quit IRC | 03:07 | |
| partlycloudy | but in the ceph-prod-example, it seems the br-storage is used as the cluster network (replication/recovery). so i think there is no need for the compute nodes to have ips on the br-storage. right? | 03:09 |
| *** TxGirlGeek has joined #openstack-ansible | 03:19 | |
| *** DanyC has joined #openstack-ansible | 03:20 | |
| *** DanyC has quit IRC | 03:24 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Improve the documentation for skydive https://review.openstack.org/630873 | 03:24 |
| cloudnull | partlycloudy i guess if its only used for replication then there'd be no need for it on the compute node. | 03:25 |
| partlycloudy | cloudnull: thank you! that makes sense to me. i will see if i hit any errors with no ips on the compute's br-storage in this case. | 03:31 |
| cloudnull | partlycloudy sorry i dont have a better answer for you, | 03:33 |
| partlycloudy | thank you, cloudnull. that was really helpful. i'll try it and let you know what i find. | 03:39 |
| jamesdenton | cloudnull was gonna see if i could get your thoughts on https://review.openstack.org/#/q/topic:bug/1810319, as well as these two: https://review.openstack.org/#/q/owner:james.denton%2540rackspace.com+project:openstack/openstack-ansible-ops | 03:40 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Improve the documentation for skydive https://review.openstack.org/630873 | 03:41 |
| cloudnull | jamesdenton sure thing | 03:41 |
| jamesdenton | thank you much. no rush. i'm callin it. | 03:42 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Improve the documentation for skydive https://review.openstack.org/630873 | 03:49 |
| cloudnull | have a good night jamesdenton | 03:49 |
| jamesdenton | thanks buddy | 03:50 |
| *** TxGirlGeek has quit IRC | 03:51 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Improve the documentation for skydive https://review.openstack.org/630873 | 04:04 |
| *** udesale has joined #openstack-ansible | 04:05 | |
| *** dave-mccowan has quit IRC | 04:11 | |
| *** udesale has quit IRC | 04:15 | |
| *** udesale has joined #openstack-ansible | 04:16 | |
| *** udesale has quit IRC | 04:26 | |
| *** radeks has joined #openstack-ansible | 04:33 | |
| *** markvoelker has joined #openstack-ansible | 05:05 | |
| *** partlycloudy has quit IRC | 05:10 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Improve the documentation for skydive https://review.openstack.org/630873 | 05:18 |
| *** chkumar|out is now known as chandankumar | 05:38 | |
| *** Bhujay has joined #openstack-ansible | 05:43 | |
| *** Bhujay has quit IRC | 05:44 | |
| *** Bhujay has joined #openstack-ansible | 05:45 | |
| *** Bhujay has quit IRC | 05:46 | |
| *** Bhujay has joined #openstack-ansible | 05:46 | |
| *** vnogin has joined #openstack-ansible | 05:46 | |
| *** markvoelker has quit IRC | 05:47 | |
| *** markvoelker has joined #openstack-ansible | 05:48 | |
| *** vnogin has quit IRC | 05:51 | |
| *** markvoelker has quit IRC | 05:59 | |
| *** udesale has joined #openstack-ansible | 06:05 | |
| *** udesale has quit IRC | 06:12 | |
| *** udesale has joined #openstack-ansible | 06:13 | |
| *** udesale has quit IRC | 06:33 | |
| *** udesale has joined #openstack-ansible | 06:33 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_heat master: [DNM] heat tempest tests finding https://review.openstack.org/630694 | 06:58 |
| *** Bhujay has quit IRC | 07:01 | |
| *** Bhujay has joined #openstack-ansible | 07:10 | |
| *** Bhujay has quit IRC | 07:11 | |
| *** Bhujay has joined #openstack-ansible | 07:11 | |
| *** DanyC has joined #openstack-ansible | 07:12 | |
| *** jbadiapa has joined #openstack-ansible | 07:15 | |
| *** DanyC has quit IRC | 07:17 | |
| *** kopecmartin|off is now known as kopecmartin | 07:35 | |
| *** yetiszaf has joined #openstack-ansible | 07:35 | |
| *** hamzaachi has joined #openstack-ansible | 07:58 | |
| *** radeks has quit IRC | 08:08 | |
| *** hamzaachi has quit IRC | 08:18 | |
| *** hamzaachi has joined #openstack-ansible | 08:18 | |
| *** hamzaachi has quit IRC | 08:22 | |
| *** hamzaachi has joined #openstack-ansible | 08:22 | |
| *** ostackz has joined #openstack-ansible | 08:22 | |
| *** hamzaachi has quit IRC | 08:24 | |
| *** hamzaachi has joined #openstack-ansible | 08:24 | |
| *** hamzaachi has quit IRC | 08:26 | |
| *** hamzaachi has joined #openstack-ansible | 08:26 | |
| *** hamzaachi has quit IRC | 08:27 | |
| *** hamzaachi has joined #openstack-ansible | 08:28 | |
| ostackz | trying minor upgrade 18.1.1 -> 18.1.2 but playbooks fail on TASK [rabbitmq_server : Apply rabbitmq policies], turns out beam.smp process is down in all rabbitmq containers. | 08:35 |
| ostackz | Tried restarting containers - then beam.smp is up, but if rerun playbooks fails again and beam.smp down again. More details https://pastebin.com/raw/UzZXqXbG Please suggest how to solve this. | 08:35 |
| *** tosky has joined #openstack-ansible | 08:44 | |
| chandankumar | odyssey4me: Hello | 09:00 |
| chandankumar | odyssey4me: http://logs.openstack.org/82/627482/42/check/python-tempestconf-tempest-os-tempest/b11cbd6/ara-report/result/4d9ded1f-91ee-425c-8e9c-29dca4d723ea/ | 09:02 |
| chandankumar | odyssey4me: I think I also need to put interface {regionname} in a new var | 09:03 |
| chandankumar | odyssey4me: what do you say? | 09:03 |
| jenglisch | [04:25] <cloudnull> partlycloudy i guess if its only used for replication then there'd be no need for it on the compute node. << the (in ceph termed) cluster network don't need access to anything but other osds | 09:08 |
| *** hamzaachi_ has joined #openstack-ansible | 09:10 | |
| *** hamzaachi_ has quit IRC | 09:11 | |
| *** hamzaachi_ has joined #openstack-ansible | 09:13 | |
| *** hamzaachi has quit IRC | 09:14 | |
| jrosser | i think that the setup in the ceph example should be improved | 09:15 |
| jrosser | br-storage on the compute nodes is really for rbd clients of ceph | 09:15 |
| jenglisch | jrosser: actually i think the example should be changed to be alike of partlycloudy's | 09:26 |
| jrosser | well, perhaps | 09:27 |
| jenglisch | oh, that's what you wrote | 09:27 |
| jrosser | br-storage should have an ip on the storage network | 09:27 |
| jrosser | and i'm not really sure that the example setup should be putting ceph traffic on the mgmt network | 09:27 |
| jenglisch | actually i proposed the change required for this, since i thought it's already the default setup to put the public network of ceph on br-storage | 09:28 |
| jenglisch | https://review.openstack.org/#/c/628625/ | 09:28 |
| jenglisch | and then the example would be broken | 09:28 |
| jenglisch | i'll extend it, and move ceph's public network to br-storage | 09:28 |
| jrosser | yeah, the same setup needs to cover the case where OSA deploys ceph, or an external ceph cluster is accessible via br-storage and the storage network | 09:29 |
| *** pcaruana has joined #openstack-ansible | 09:30 | |
| *** hamzaachi_ has quit IRC | 09:33 | |
| *** hamzaachi_ has joined #openstack-ansible | 09:33 | |
| *** hamzaachi_ has quit IRC | 09:38 | |
| *** hamzaachi_ has joined #openstack-ansible | 09:38 | |
| *** jungleboyj has quit IRC | 09:39 | |
| *** jungleboyj has joined #openstack-ansible | 09:42 | |
| *** DanyC has joined #openstack-ansible | 09:43 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use usr/local/share/ansible/roles for data_files https://review.openstack.org/630917 | 09:45 |
| odyssey4me | chandankumar if the region and endpoint aren't already parameterised, then yeah, that's probably useful to enable | 09:49 |
| chandankumar | odyssey4me: working on that | 09:50 |
| chandankumar | odyssey4me: regarding heat tempest test patch | 09:51 |
| chandankumar | odyssey4me: tempest logs are not getting generated | 09:51 |
| *** nyloc has quit IRC | 09:51 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use usr/local/share/ansible/roles for data_files https://review.openstack.org/630917 | 09:52 |
| odyssey4me | chandankumar I see them: http://logs.openstack.org/94/630694/3/check/openstack-ansible-deploy-aio_metal_heat-ubuntu-bionic/1f68af5/logs/openstack/aio1-utility/stestr_results.html | 09:53 |
| chandankumar | odyssey4me: oh I didnot checked that directory | 09:53 |
| *** DanyC has quit IRC | 09:54 | |
| *** DanyC has joined #openstack-ansible | 09:55 | |
| ostackz | Hi, odyssey4me, trying minor upgrade 18.1.1 -> 18.1.2 but playbooks fail on TASK [rabbitmq_server : Apply rabbitmq policies], turns out beam.smp process is down in all rabbitmq containers. | 09:59 |
| ostackz | Tried restarting containers - then beam.smp is up, but if rerun playbooks fails again and beam.smp down again. More details https://pastebin.com/raw/UzZXqXbG Could you suggest how to solve this? | 10:00 |
| odyssey4me | ostackz hrm, I can't recall the specific command to restart rabbitmq entirely - it's in that role I think | 10:00 |
| odyssey4me | otherwise just restart the containers one at a time? | 10:01 |
| *** hamzaachi_ has quit IRC | 10:03 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests https://review.openstack.org/630695 | 10:06 |
| *** nyloc has joined #openstack-ansible | 10:15 | |
| chandankumar | odyssey4me: regarding heat tempest failure, I need to put heat related tempest configuration | 10:17 |
| odyssey4me | chandankumar so something's missing in the tempest role to make it work? | 10:17 |
| chandankumar | odyssey4me: yes, adding to my list 1:1 | 10:17 |
| odyssey4me | chandankumar thanks - appreciate it | 10:17 |
| chandankumar | odyssey4me: tests failed as auth_url used in heat section missing | 10:18 |
| odyssey4me | isn't it supposed to get that from the service catalog? | 10:18 |
| chandankumar | odyssey4me: https://github.com/openstack/heat/blob/master/heat_integrationtests/prepare_test_env.sh#L32 | 10:18 |
| chandankumar | odyssey4me: we need to specify that particularly otherwise it will not work | 10:19 |
| chandankumar | odyssey4me: I will get that working | 10:19 |
| odyssey4me | I see, ok - thanks. | 10:19 |
| *** vnogin has joined #openstack-ansible | 10:22 | |
| *** udesale has quit IRC | 10:25 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone stable/rocky: Remove legacy nginx config for keystone-wsgi-admin https://review.openstack.org/629430 | 10:27 |
| *** aedc has joined #openstack-ansible | 10:28 | |
| *** chandankumar has quit IRC | 10:31 | |
| *** arxcruz|ruck has quit IRC | 10:34 | |
| *** udesale has joined #openstack-ansible | 10:35 | |
| *** chandan_kumar has joined #openstack-ansible | 10:35 | |
| *** mino_ has quit IRC | 10:37 | |
| *** electrofelix has joined #openstack-ansible | 10:48 | |
| openstackgerrit | kourosh vivan proposed openstack/openstack-ansible-os_keystone master: Deployment host can provide shibboleth SP key-pair https://review.openstack.org/620324 | 10:48 |
| *** macza has joined #openstack-ansible | 10:50 | |
| *** macza has quit IRC | 10:54 | |
| *** udesale has quit IRC | 11:01 | |
| ostackz | odyssey4me, regarding that minor upgrade - tried to restart and leave running only 1 rabbitmq conatainer(container count 3, but beam.smp running only in first), then rerun openstack-ansible -e rabbitmq_upgrade=true setup-infrastructure.yml. It ended like before - all rabbitmq processes are down. | 11:04 |
| ostackz | Here are playbook details https://pastebin.com/raw/UzZXqXbG Seems that playbooks stop rabbitmq cluster nodes but fail later because rabbitmq process is down. There is no starting task after "Ensure RabbitMQ node [0] is stopped". | 11:04 |
| odyssey4me | ostackz beam.smp is only part of the service - not the whole thing | 11:04 |
| odyssey4me | ostackz which tag/release is this? | 11:05 |
| ostackz | 18.1.2 | 11:05 |
| *** arxcruz has joined #openstack-ansible | 11:05 | |
| odyssey4me | ostackz does that have this patch in the rabbitmq role? https://review.openstack.org/#/c/625200/1 | 11:06 |
| odyssey4me | nope, looks like that just missed it | 11:07 |
| odyssey4me | stable/rocky has it | 11:07 |
| odyssey4me | although another way around the issue is to just wipe out the rabbitmq containers and recreate them | 11:07 |
| odyssey4me | then you have no need to do the upgrade, because it's a fresh install | 11:08 |
| ostackz | odyssey4me, I just did git checkout 18.1.2 yesterday, isn't it Rocky stable? | 11:08 |
| odyssey4me | ostackz it is, but there was a bug which was found and resolved after which is still in testing | 11:08 |
| ostackz | yes, I can do that, because fresh insstall did not have issues with rabbitmq | 11:08 |
| odyssey4me | you can cherry-pick that patch into your rabbitmq role | 11:09 |
| ostackz | odyssey4me, thanks! At least I know there is issue that needs fixing in playbooks not my config. Will look for info how to cherry-pick, I'm confident there is some piece of doc for those who are not familiar with git and stuff, time to learn something new :) or recreate containers as alternative. | 11:15 |
| odyssey4me | ostackz in that url - on the top right is a 'download' drop-down, with 'cherry-pick' as an option, and a little clipboard on the right to copy the command to clipboard | 11:16 |
| odyssey4me | then cd /etc/ansible/roles/rabbitmq_server; <paste command> | 11:16 |
| odyssey4me | that should do the trick | 11:16 |
| ostackz | odyssey4me, thanks a lot! will note that down | 11:17 |
| *** openstackgerrit has quit IRC | 11:22 | |
| *** mwe1 has quit IRC | 11:23 | |
| *** ygk_12345 has joined #openstack-ansible | 11:24 | |
| ygk_12345 | odyssey4me: Hi, Does OSA support BIND dns for designate as backend ? | 11:24 |
| odyssey4me | ygk_12345 it supports whatever designate supports | 11:27 |
| odyssey4me | ygk_12345 we do not deploy the DNS service though - it is expected to already be running | 11:28 |
| ygk_12345 | odyssey4me: i dont find that description here https://docs.openstack.org/openstack-ansible-os_designate/latest/ | 11:28 |
| odyssey4me | ygk_12345 what description? | 11:28 |
| ygk_12345 | odyssey4me: the backend DNS softwares it supports | 11:28 |
| ygk_12345 | odyssey4me: like BIND or powerdns etc | 11:28 |
| odyssey4me | ygk_12345 we simply deploy the designate service - so whatever designate supports, we support | 11:29 |
| odyssey4me | that's something that the service supports - not us | 11:29 |
| ygk_12345 | odyssey4me: ok i will try to deploy | 11:29 |
| ygk_12345 | odyssey4me: ok | 11:29 |
| *** mwe1 has joined #openstack-ansible | 11:31 | |
| ygk_12345 | odyssey4me: one more query | 11:34 |
| ygk_12345 | odyssey4me: is there detailed procedure of what variables to include in user_cofig and user_variable files for designate ? | 11:34 |
| *** udesale has joined #openstack-ansible | 11:36 | |
| odyssey4me | ygk_12345 for all roles, the defaults in the role are the documented override variables you can use - they are all optional | 11:36 |
| ygk_12345 | odyssey4me: ok | 11:36 |
| odyssey4me | all overrides are implemented in host_vars/group_vars or user_variables (which are global)... openstack_user_config should be avoided for overrides because they persist in the inventory | 11:37 |
| *** openstackgerrit has joined #openstack-ansible | 11:37 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Add tempest_region_name var for setting interface https://review.openstack.org/630942 | 11:37 |
| chandan_kumar | odyssey4me: ^^ there are two more things, domain & domain-id as well as endpoint type what to do for that? | 11:38 |
| odyssey4me | chandan_kumar interface and endpoint are aliases - I can't remember which one is the primary, but it'd be nice to get that consistently used | 11:39 |
| chandan_kumar | odyssey4me: I changed both of them then amended | 11:40 |
| chandan_kumar | odyssey4me: good to know that | 11:40 |
| chandan_kumar | updating it | 11:40 |
| odyssey4me | then as I recall, the domain argument can be a name or id - and there is no domain id parameter, except for legacy... and those are upstream bugs which need fixing | 11:40 |
| chandan_kumar | odyssey4me: ok | 11:41 |
| *** ansmith has joined #openstack-ansible | 11:59 | |
| *** DanyC has quit IRC | 12:10 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Add tempest_interface_name var for interface/endpoint_type https://review.openstack.org/630942 | 12:11 |
| *** DanyC has joined #openstack-ansible | 12:13 | |
| *** ansmith has quit IRC | 12:16 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: DNM - Testing ARA upgrades https://review.openstack.org/537360 | 12:26 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest master: Fix tempest workspace path https://review.openstack.org/628182 | 12:29 |
| *** radeks has joined #openstack-ansible | 12:31 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use tempest_domain_name var for setting domain/domain-id https://review.openstack.org/630957 | 12:34 |
| guilhermesp | odyssey4me: based on your comment, would be good then to add an AND conditional to check if the current workspace is the same workspace_path? https://review.openstack.org/#/c/628182/ | 12:42 |
| *** vnogin has quit IRC | 12:43 | |
| odyssey4me | guilhermesp I would think that it needs to check if it is NOT the same as the current workspace path | 12:44 |
| *** crazzy has joined #openstack-ansible | 12:49 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Make container_interface optional for provider network https://review.openstack.org/630964 | 12:57 |
| *** chandan_kumar has quit IRC | 12:58 | |
| jamesdenton | jrosser what are your thoughts on this? https://review.openstack.org/#/c/630964/. Looking to simplify provider network definitions in the age of neutron-deployed-on-bare-metal | 12:58 |
| *** chandan_kumar has joined #openstack-ansible | 12:59 | |
| *** radeks has quit IRC | 13:00 | |
| * jrosser looks | 13:01 | |
| *** radeks has joined #openstack-ansible | 13:01 | |
| jrosser | jamesdenton: somewhat relatedly we place neutron ml2.conf files inside the neutron server containers, thats also a bit confusing | 13:04 |
| *** vnogin has joined #openstack-ansible | 13:04 | |
| jamesdenton | i think it might actually be necessary, at least so neutron-server knows the provider labels. i could be wrong | 13:05 |
| jrosser | right | 13:06 |
| jrosser | oh yes i see on the running processes it's loaded ml2_conf.ini | 13:06 |
| jamesdenton | but certainly any provider mapping wouldn't be - but i think that's in the respective agent config file | 13:06 |
| *** mkuf_ has joined #openstack-ansible | 13:07 | |
| jamesdenton | it was a mess (upstream) not all that long ago | 13:07 |
| mgariepy | jenglisch, odyssey4me mnaser concerning: https://review.openstack.org/#/c/629330/ rtcsync would sync every 11 minutes, not 11 seconds | 13:08 |
| jenglisch | mgariepy: you're correct, my bad | 13:09 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-ops master: Add mnaio_data_disk_suffix var to support NVMe partition naming https://review.openstack.org/630375 | 13:09 |
| jenglisch | hm | 13:09 |
| mgariepy | and with the option rtcfile and rtcautotrim timedatectl show N/A for rtc time. | 13:10 |
| *** mkuf has quit IRC | 13:10 | |
| mgariepy | also i can set/read the RTC with hwclock | 13:10 |
| mgariepy | with rtcsync it works fine. | 13:10 |
| mgariepy | any objection to switch to rtcsync ? | 13:11 |
| *** Guest1750 has joined #openstack-ansible | 13:14 | |
| *** Guest1750 is now known as maxbab | 13:16 | |
| jrosser | jamesdenton: that looks good to make container_network optional there, all of that stuff makes me nervous that something is going to mis(use) that key somewhere unexpected | 13:18 |
| jrosser | sory container_interface | 13:19 |
| jamesdenton | container_interface, you mean? | 13:19 |
| jamesdenton | l | 13:19 |
| jamesdenton | kk | 13:19 |
| *** ygk_12345 has quit IRC | 13:19 | |
| jenglisch | as far as i understood the documentation "rtcsync" syncs the RTC every 11 minutes regardless the RTC time, a coworker of mine suggested this approach since a lot of writes to the CMOS may be bad, hence this option should just check the RTC and only adjust it if it's more than 10 seconds off | 13:19 |
| jenglisch | however i don't have any source for the assumption of my coworker currently, i'm also ok with using rtcsync | 13:19 |
| jenglisch | it's not unusual to call hwclock --synctohc on a daily basis and/or on shutdown, i don't exactly know what the linux kernel does on rtcsync every 11 minutes | 13:20 |
| *** arxcruz is now known as arxcruz|ruck | 13:21 | |
| *** dave-mccowan has joined #openstack-ansible | 13:29 | |
| *** radeks has quit IRC | 13:29 | |
| mgariepy | jenglisch, if it would work correctly and i could verify,i wouldn't mind rtcfile + autotrim | 13:29 |
| mgariepy | but when those are set in chrony.conf i cannot read the rtc anymore. | 13:29 |
| mgariepy | unless i stop chrony.. | 13:29 |
| jenglisch | whats your usecase of reading from /dev/rtc* directly? | 13:31 |
| mgariepy | validate it's sync. | 13:31 |
| jenglisch | typically i'd use the system time. the rtc is just used in case of a reboot | 13:31 |
| jenglisch | hm | 13:31 |
| jenglisch | so it breaks your monitoring system? | 13:31 |
| mgariepy | hwclock or timedatectl is not able to read | 13:31 |
| mgariepy | https://paste.ubuntu.com/p/pdYQ5hgfcv/ | 13:33 |
| mgariepy | i had some issue when rebooting nodes that the time is not set quite correcly, so i want to validarte the rtc is set ok before rebooting. | 13:34 |
| *** udesale has quit IRC | 13:36 | |
| *** udesale has joined #openstack-ansible | 13:36 | |
| speedmann | Hi there, when trying to install octavia i get an error that libpython2.7.so.1.0 can not be found. We are running release 18.1.0. Is this a known issue with the octavia container? | 13:36 |
| speedmann | sorry, this error occurs when starting octavia-api uwsgi | 13:37 |
| mgariepy | also syncing every 11 minutes isn't that much, and it was like this for several years on other of my system never saw any issue about it. | 13:38 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Improve the documentation for skydive https://review.openstack.org/630873 | 13:45 |
| jrosser | speedmann: this is most likley a bug, we have been trying to reduce the number of packages globally installed into containers and have each role be more specific about which packages are needed | 13:55 |
| jrosser | speedmann: quite possible that octavia needs a patch to fix that for libpython | 13:55 |
| jamesdenton | Couldn't help but notice OSA wasn't listed here: https://releases.openstack.org/stein/index.html. Oversight or intentional? | 13:56 |
| odyssey4me | hmm, mnaser that's odd | 13:57 |
| odyssey4me | oh, it's because we haven't done a m1 release | 13:57 |
| odyssey4me | evrardjp ? | 13:57 |
| jamesdenton | ahh ok | 13:57 |
| jamesdenton | aren't we in m2 territory? lol | 13:58 |
| odyssey4me | we were supposed to do a m1 release in october | 13:58 |
| speedmann | jrosser thank you, i will open a bug report for this so someone can have a close look | 13:58 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest master: Fix tempest workspace path https://review.openstack.org/628182 | 13:59 |
| *** vnogin has quit IRC | 14:01 | |
| ostackz | odyssey4me, your patch helped, I successfully run setup-hosts.yml and setup-infrastructure.yml, but now failing on setup-openstack.yml - TASK [os_keystone : Bootstrap keystone admin and endpoint]. | 14:04 |
| ostackz | What exactly it is supposed to do if I already have rocky setup? I guess bootstrapping should be skipped then? | 14:04 |
| odyssey4me | ostackz no, it always runs - what's the failure? | 14:04 |
| odyssey4me | a debug log paste would be useful | 14:05 |
| noonedeadpunk | Folks, getting "Package '+python-neutron-vpnaas-dashboard' not found." in opensuse-423 aio_distro install. Is anyone aware about this problem? | 14:05 |
| ostackz | odyssey4me FAILED - RETRYING: Bootstrap keystone admin and endpoint (1 retries left). | 14:05 |
| odyssey4me | noonedeadpunk is that master or rocky? | 14:05 |
| noonedeadpunk | rocky | 14:05 |
| odyssey4me | ostackz a paste (on a paste service) of a debug log of that task please | 14:05 |
| odyssey4me | noonedeadpunk check if you have https://review.openstack.org/629194 or https://review.openstack.org/629177 depending on the branch | 14:06 |
| odyssey4me | noonedeadpunk oh, actually - that seems like an outdated mirror - because that's the new package name | 14:07 |
| noonedeadpunk | odyssey4me: ah, great, patch is probably on it's way to zuul gates | 14:08 |
| odyssey4me | noonedeadpunk those merged yesterday | 14:08 |
| noonedeadpunk | https://review.openstack.org/#/c/630429/ still failing with this. Probably, regular update still wasn't applied? | 14:09 |
| *** ansmith has joined #openstack-ansible | 14:10 | |
| odyssey4me | noonedeadpunk I think stable/rocky will need an anrible-role-requirements update to include https://review.openstack.org/629194 | 14:10 |
| noonedeadpunk | oh... yeah, might be the case | 14:10 |
| noonedeadpunk | do we need evrardjp for this? | 14:11 |
| noonedeadpunk | clean forgot about ansible-role-requirements | 14:12 |
| odyssey4me | noonedeadpunk nope, you can just update the one SHA in a review | 14:12 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: [DNM] - Testing openSUSE nova role https://review.openstack.org/495767 | 14:14 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible stable/rocky: Updated SHA for os_horizon role https://review.openstack.org/630982 | 14:16 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible stable/rocky: Fixes empty metal_query https://review.openstack.org/630429 | 14:17 |
| noonedeadpunk | odyssey4me thanks for the help | 14:19 |
| ostackz | odyssey4me here is the faillog https://pastebin.com/raw/9sGPVEQu To me 'mesg: ttyname failed: Inappropriate ioctl for device\n' seems strange, but dont know what is it about. Lacks terminal device? | 14:19 |
| odyssey4me | ostackz that's cosmetic - don't worry about it | 14:19 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Add tempest_interface_name var for setting interface https://review.openstack.org/630942 | 14:20 |
| odyssey4me | ostackz oh yeah, there's a password there so it won't output stdout/stderr... can you execute that command by hand in the container and see what it says? | 14:21 |
| ostackz | odyssey4me it executes with no output, like successful | 14:21 |
| odyssey4me | ostackz ORLY? do 'echo $?' to see the return code? | 14:22 |
| ostackz | echo $? returns 1 though | 14:22 |
| *** shyamb has joined #openstack-ansible | 14:22 | |
| *** Soopaman has joined #openstack-ansible | 14:23 | |
| ostackz | odyssey4me I was guessing https and self-signed cert issue, but in task there seems to be http used only | 14:24 |
| odyssey4me | ostackz so that's why the task is failing... but why the command is failing is anyone's guess | 14:25 |
| odyssey4me | maybe there's a --debug that can be used? | 14:25 |
| odyssey4me | FYI that command interacts directly with the DB - no certs involved | 14:25 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use tempest_domain_name var for setting domain https://review.openstack.org/630957 | 14:26 |
| chandan_kumar | cloudnull: jrosser https://review.openstack.org/630917 is good to go | 14:26 |
| odyssey4me | guilhermesp you left a swap file in https://review.openstack.org/#/c/628182/ | 14:27 |
| ostackz | odyssey4me with -d got something in journalctl, please see bottom of https://pastebin.com/raw/9sGPVEQu Seems to be issue with role/domain relations | 14:31 |
| odyssey4me | ostackz oh, that's curious - I wonder if the folks in #openstack-keystone can help understand what's going on there and how best to resolve it | 14:33 |
| jrosser | jamesdenton: i've not dug into it any more but this seems to be one of the only uses of container_interface http://git.openstack.org/cgit/openstack/openstack-ansible-plugins/tree/library/provider_networks#n268 | 14:33 |
| ostackz | odyssey4me could be that I have created domains with admin roles inside and this confuses bootstrap playbooks | 14:34 |
| odyssey4me | ostackz it could be, but if that's the case then there may be a bug in the tool and they would want to know about that | 14:35 |
| jamesdenton | jrosser agreed, and only if is_metal=false | 14:35 |
| jamesdenton | which, for our default deploys is now the case | 14:35 |
| jamesdenton | for the neutron agent container, anyway | 14:35 |
| *** maxbab has quit IRC | 14:36 | |
| ostackz | odyssey4me sure, will try to reach someone in openstack-keystone, thanks! | 14:37 |
| jrosser | jamesdenton: yes so i think it looks ok to remove | 14:37 |
| jamesdenton | right on, thanks | 14:37 |
| jamesdenton | i'll go thru and update docs to make it easier to understand its purpose | 14:37 |
| openstackgerrit | Marc Gariépy (mgariepy) proposed openstack/ansible-hardening master: Switch to rtcsync for chrony https://review.openstack.org/630996 | 14:39 |
| *** mkuf_ has quit IRC | 14:39 | |
| *** mkuf_ has joined #openstack-ansible | 14:43 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest master: Fix tempest workspace path https://review.openstack.org/628182 | 14:43 |
| *** Soopaman has quit IRC | 14:48 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Use usr/local/share/ansible/roles for data_files https://review.openstack.org/630917 | 14:49 |
| *** shyamb has quit IRC | 14:53 | |
| jrosser | noonedeadpunk: is this another one you've seen? gnocchi.cli.api: Unable to find `uwsgi' | 14:54 |
| *** Bhujay has quit IRC | 15:02 | |
| noonedeadpunk | jrosser: I guess, no.. When does it happens? As I remeber OSA dosen't use native gnocchi-api service. Instead it configures apache to serve wsgi | 15:04 |
| *** mkuf_ has quit IRC | 15:04 | |
| noonedeadpunk | so it shouldn't need uwsgi in these conditions | 15:04 |
| *** mkuf has joined #openstack-ansible | 15:05 | |
| jrosser | i'm going through logs for everything and trying to fix up errors generally | 15:05 |
| jrosser | i did find this https://github.com/gnocchixyz/gnocchi/issues/478 | 15:05 |
| noonedeadpunk | I've had this error, but on my manual deployment (not via OSA), and I've catched it during Gnocchi 3->4 upgrade | 15:06 |
| noonedeadpunk | and yeah, it was installed with distro packages on ubuntu | 15:06 |
| noonedeadpunk | but with osa you don't need gnocchi-api and, accordingly, uwsgi | 15:06 |
| *** francois has quit IRC | 15:07 | |
| noonedeadpunk | And I had to correct this line https://github.com/gnocchixyz/gnocchi/blob/4.2.4/gnocchi/cli/api.py#L76 as in ubuntu package it tried to use shutil method, which is available only since python 3.4 | 15:08 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: README cleanup and additional notes about limits https://review.openstack.org/631004 | 15:08 |
| cloudnull | mornings | 15:08 |
| *** francois has joined #openstack-ansible | 15:09 | |
| francois | Hello and happy new year! | 15:11 |
| francois | I'm trying to understand how dns resolution should work in a rocky deployment using ubuntu 18.04 and the unbound containers | 15:12 |
| francois | more specifically how it works when one of the unbound container is down | 15:12 |
| jamesdenton | mornin cloudnull | 15:12 |
| francois | when one of the three controller is down, all openstack apis are super slow due to dns timeouts | 15:13 |
| noonedeadpunk | jrosser under gnocchi-api I meant systemctl service, but not API in it's general meaning | 15:13 |
| *** tbarron has joined #openstack-ansible | 15:16 | |
| jamesdenton | maybe load balancing dns would help? | 15:20 |
| jrosser | the default resolver timeout is 5s iirc | 15:24 |
| francois | yep, linux's libc seems to use 5s by default | 15:26 |
| francois | in an OSA deployment "option timeout:1 rotate" is added to resolv.conf but only one bare metal machines | 15:27 |
| jrosser | it quite possibly still hits that for every name resolution though, so its bad -> less bad? | 15:28 |
| francois | the LXC containers don't have this option set, probably because of systemd-resolved | 15:28 |
| francois | jrosser: yes probably | 15:28 |
| francois | i think that systemd-resolved actually queries all resolvers in parallel | 15:29 |
| francois | but not the way it's currently configured it seems | 15:29 |
| *** TxGirlGeek has joined #openstack-ansible | 15:30 | |
| fnpanic | hi | 15:34 |
| fnpanic | using all resolvers in parallel is not default as far as i know | 15:35 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest master: Fix tempest workspace path https://review.openstack.org/628182 | 15:37 |
| *** partlycloudy has joined #openstack-ansible | 15:40 | |
| francois | fnpanic: hi, with systemd-resolved? for dnsmasq, the --all-server option needs to be set | 15:41 |
| fnpanic | francois: yes with systemd-resolved | 15:46 |
| jrosser | the internet seems to say that it is wrong to leave dns failover up to the client | 15:47 |
| *** triceracops has joined #openstack-ansible | 15:50 | |
| *** radeks has joined #openstack-ansible | 15:57 | |
| fnpanic | jrosser: ok, so you suggest adding VIP for each unbound container and failing them over? | 15:57 |
| francois | jrosser: yeah, we should probably anycast the dns resolver ip but it's not currently the case :( | 15:57 |
| fnpanic | anycast is a bit extrem and you need the infra to implement it | 15:58 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Make container_interface optional for provider network https://review.openstack.org/630964 | 16:00 |
| *** radeks has quit IRC | 16:04 | |
| *** errr has quit IRC | 16:07 | |
| *** errr has joined #openstack-ansible | 16:10 | |
| *** partlycloudy has quit IRC | 16:14 | |
| chandan_kumar | odyssey4me: jrosser cloudnull https://review.openstack.org/#/c/630957/ and https://review.openstack.org/630942 | 16:15 |
| chandan_kumar | please have a look thanks :-) | 16:15 |
| chandan_kumar | odyssey4me: Do I need to keep an eye of ansible provided openstack modules in order to avoid breakage in future? | 16:26 |
| chandan_kumar | as we are consuming in os_tempest side | 16:26 |
| openstackgerrit | Dave Wilde (d34dh0r53) proposed openstack/openstack-ansible-os_tempest master: Enable port security https://review.openstack.org/617719 | 16:29 |
| odyssey4me | chandan_kumar yep, we're relying more and more on the upstream ansible modules - so any assistance with the bugs there would be very helpful. | 16:31 |
| chandan_kumar | odyssey4me: will add it to my list and where to start poking any pointers? | 16:32 |
| *** ansmith_ has joined #openstack-ansible | 16:32 | |
| odyssey4me | chandan_kumar bugs: https://github.com/ansible/ansible/issues?q=is%3Aopen+is%3Aissue+label%3Aopenstack | 16:33 |
| odyssey4me | chandan_kumar prs: https://github.com/ansible/ansible/pulls?q=is%3Aopen+is%3Apr+label%3Aopenstack | 16:33 |
| * chandan_kumar adds to my list | 16:33 | |
| *** ansmith has quit IRC | 16:35 | |
| francois | fnpanic: accordding to upstream doc (https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html), it looks it does parallel queries by default | 16:36 |
| francois | > (Note that more than one link might have this same "best matching" search/route-only domain configured, in which case the query is sent to all of them in parallel) | 16:36 |
| francois | I'll give a try with "lxc_container_enable_resolved: false" because I'm pretty sure there's some clash between unbound/resolvconf and systemd-resolved going on | 16:38 |
| *** hamzy_ has quit IRC | 16:43 | |
| fnpanic | ok. let me know how it went | 16:50 |
| fnpanic | is ist possible to have keeplaived ping more then one address? | 16:52 |
| fnpanic | keepalived_ping_address: | 16:52 |
| *** ansmith has joined #openstack-ansible | 16:54 | |
| *** ansmith_ has quit IRC | 16:56 | |
| francois | fnpanic: no, looking at the code only a single ip is supported | 16:58 |
| chandan_kumar | evrardjp: odyssey4me: suse ci is passing now a days, can we make it voting? | 17:02 |
| *** udesale has quit IRC | 17:04 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: README cleanup and additional notes about limits https://review.openstack.org/631004 | 17:05 |
| *** macza has joined #openstack-ansible | 17:08 | |
| odyssey4me | chandan_kumar we have a very small suse supporting group and will let them decide on that | 17:12 |
| *** macza_ has joined #openstack-ansible | 17:13 | |
| *** macza has quit IRC | 17:14 | |
| *** DanyC has quit IRC | 17:48 | |
| *** DanyC has joined #openstack-ansible | 17:48 | |
| *** DanyC has quit IRC | 17:53 | |
| *** hamzaachi has joined #openstack-ansible | 18:01 | |
| *** pcaruana has quit IRC | 18:01 | |
| *** TxGirlGeek has quit IRC | 18:04 | |
| *** TxGirlGeek has joined #openstack-ansible | 18:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Use tempest_domain_name var for setting domain https://review.openstack.org/630957 | 18:07 |
| *** ansmith_ has joined #openstack-ansible | 18:26 | |
| *** ansmith has quit IRC | 18:29 | |
| *** hamzaachi has quit IRC | 18:38 | |
| *** hamzy_ has joined #openstack-ansible | 18:44 | |
| *** openstacking_123 has joined #openstack-ansible | 18:46 | |
| openstacking_123 | Hope everyone is welll | 18:51 |
| *** ansmith_ has quit IRC | 18:52 | |
| openstacking_123 | Anyone ever run into an issue where rabbitmq can no longer find the exchange for Openstack services? | 18:52 |
| openstacking_123 | ```operation basic.publish caused a channel exception not_found: no exchange 'reply_b91e0012696d45e7a32e3265a5ab22fe' in vhost '/nova'``` | 18:52 |
| *** kopecmartin is now known as kopecmartin|off | 18:52 | |
| openstacking_123 | Rabbit looks healthy as far as I can tell but this issue popped up at random for me yesterday | 18:53 |
| spotz | openstacking_123: I haven't run into this but can you run basic rabbit troubleshooting commands successfully? | 18:59 |
| spotz | And also check space...:) | 18:59 |
| *** hamzy_ has quit IRC | 18:59 | |
| *** hamzy_ has joined #openstack-ansible | 19:00 | |
| openstacking_123 | Yeah they look good but I think maybe this happened https://bugs.launchpad.net/oslo.messaging/+bug/1789177 | 19:00 |
| openstack | Launchpad bug 1789177 in oslo.messaging "RabbitMQ fails to synchronize exchanges under high load" [Undecided,Fix released] - Assigned to Oleg Bondarev (obondarev) | 19:00 |
| openstacking_123 | We run DVR so we would be running quite a bit of rabbit connections | 19:00 |
| openstacking_123 | going to see if we have that patch | 19:00 |
| *** ansmith_ has joined #openstack-ansible | 19:01 | |
| *** hamzy_ has quit IRC | 19:13 | |
| *** hamzy_ has joined #openstack-ansible | 19:31 | |
| *** electrofelix has quit IRC | 19:46 | |
| *** vnogin has joined #openstack-ansible | 20:06 | |
| *** hamzy_ has quit IRC | 20:12 | |
| *** vnogin has quit IRC | 20:22 | |
| *** openstacking_123 has quit IRC | 20:37 | |
| *** DanyC has joined #openstack-ansible | 20:57 | |
| *** DanyC has quit IRC | 20:58 | |
| *** DanyC has joined #openstack-ansible | 20:59 | |
| *** DanyC_ has joined #openstack-ansible | 21:03 | |
| *** DanyC has quit IRC | 21:04 | |
| *** hamzaachi has joined #openstack-ansible | 21:07 | |
| *** strattao has joined #openstack-ansible | 21:16 | |
| strattao | why does os_tempest create a public network associated with a project, but not a public subnet? | 21:20 |
| *** hamzy_ has joined #openstack-ansible | 21:21 | |
| strattao | just throwin’ that question out into the void… | 21:24 |
| strattao | Actually, a better question is, why does os_tempest create a default public network at all? I want to test my existing public network configuration. | 21:24 |
| strattao | Hopefully some tempest guru can chime in… @odyssey4me seems to be the biggest contributor to that repo so maybe… | 21:27 |
| strattao | unless I’m just barking up the wrong tree to test my deployment… | 21:29 |
| *** hamzy_ has quit IRC | 21:29 | |
| *** openstacking_123 has joined #openstack-ansible | 21:39 | |
| odyssey4me | strattao the os_tempest role has some resource setup which was originally for the purpose of our own testing (back in icehouse), not really for general use - over time that's changing, and if there's something that'd make it work better for more general use, we're happy to receive bug reports/patches | 21:41 |
| odyssey4me | strattao that said, tempest is a bit obscure for our users, so we started developing 'healthcheck-*' playbooks in rocky for more readable and understandable output which may be something you prefer | 21:42 |
| odyssey4me | we'd also be happy to see those grow to be more full fledged | 21:43 |
| strattao | odyssey4me, as always, thank you for your response! | 21:50 |
| jrosser | strattao: i think if you put in the details of your existing external network, overriding the tempest defaults then it will just use that? | 21:53 |
| strattao | jrosser, it’s close, but there aren’t overrides for everything os_tempest wants to create | 21:54 |
| jrosser | can we fix that? :) | 21:54 |
| *** ansmith_ has quit IRC | 21:54 | |
| strattao | sure! it’s just time, right? I’m still getting familiar with options for testing/benchmarking my deployment to make sure that everything is good to go. Seems like rally and tempest are my best bets. Will take a closer look at the healthcheck-* stuff. | 21:57 |
| jrosser | i have an ansible role somewhere for refstack too | 21:57 |
| strattao | other than rally, tempest, healthcheck, what other tools should I be looking at to validate/benchmark a production deployment? | 21:58 |
| jrosser | i use this https://refstack.openstack.org | 22:00 |
| jrosser | which is shiny around tempest | 22:00 |
| *** openstacking_123 has quit IRC | 22:11 | |
| *** hamzaachi has quit IRC | 22:12 | |
| *** strattao has quit IRC | 22:49 | |
| *** bgmccollum has quit IRC | 22:53 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add a default retention policy for skydive indexes https://review.openstack.org/631099 | 23:36 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!