openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 00:00 |
---|---|---|
*** jaosorior has quit IRC | 00:05 | |
*** tosky has quit IRC | 00:13 | |
*** tongl has quit IRC | 00:24 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Capture ramdisk test run logs https://review.openstack.org/619400 | 00:25 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix unit tests for elements https://review.openstack.org/619387 | 00:25 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "Make tripleo-buildimage-overcloud-full-centos-7 non-voting" https://review.openstack.org/620201 | 00:25 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 00:25 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 00:25 |
dtrainor | Can someone tell me more about ComputeCapabilitiesFilter? Does this filter based on known information from Ironic, or is this information gathered from the node when the node is about to be used in a deployment, e.g. over the wire instead of from Ironic's cache? | 00:27 |
dtrainor | I ask because I see some weird values in here - negative RAM amounts, 0 disk size amounts etc. Wondering if I've got a bad or buggy bios that's spitting out this information. The introspection data for the node seems reasonable and sane. | 00:28 |
*** ade_lee has quit IRC | 00:40 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 00:49 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 00:49 |
*** wolverin_ has quit IRC | 00:59 | |
*** wolverineav has joined #tripleo | 01:00 | |
*** chem has quit IRC | 01:01 | |
*** wolverineav has quit IRC | 01:02 | |
*** wolverineav has joined #tripleo | 01:02 | |
*** ramishra has quit IRC | 01:13 | |
*** boazel_ has joined #tripleo | 01:19 | |
*** boazel has quit IRC | 01:22 | |
*** bfournie has joined #tripleo | 01:33 | |
*** dsneddon has quit IRC | 01:59 | |
*** dsneddon has joined #tripleo | 02:16 | |
*** mcornea has quit IRC | 02:17 | |
*** sai_p has quit IRC | 02:23 | |
*** Chaserjim has quit IRC | 02:25 | |
*** mschuppert has quit IRC | 02:25 | |
*** mrsoul has quit IRC | 02:25 | |
*** dsneddon has quit IRC | 02:25 | |
*** dsneddon has joined #tripleo | 02:33 | |
*** ramishra has joined #tripleo | 02:35 | |
*** dsneddon has quit IRC | 02:37 | |
*** zaneb has quit IRC | 02:41 | |
*** psachin has joined #tripleo | 02:42 | |
openstackgerrit | Merged openstack/ansible-role-redhat-subscription master: Fix table in README https://review.openstack.org/620345 | 02:45 |
*** rh-jelabarre has quit IRC | 03:03 | |
*** wolverineav has quit IRC | 03:12 | |
openstackgerrit | Merged openstack/tripleo-validations stable/rocky: Fix containerized services task https://review.openstack.org/620227 | 03:13 |
openstackgerrit | Merged openstack/tripleo-validations stable/rocky: Fix the ceilometerdb-size validation in the containerized undercloud https://review.openstack.org/620040 | 03:13 |
*** ramishra has quit IRC | 03:15 | |
*** wolverineav has joined #tripleo | 03:17 | |
*** wolverineav has quit IRC | 03:24 | |
*** dsneddon has joined #tripleo | 03:24 | |
*** ramishra has joined #tripleo | 03:35 | |
openstackgerrit | Merged openstack/python-tripleoclient stable/rocky: Run online data migrations during undercloud/standalone upgrades https://review.openstack.org/620083 | 03:51 |
*** wolverineav has joined #tripleo | 03:53 | |
*** ykarel|away has joined #tripleo | 03:55 | |
*** wolverineav has quit IRC | 03:59 | |
*** kopecmartin|off has quit IRC | 04:34 | |
*** artom has quit IRC | 04:42 | |
openstackgerrit | Merged openstack/tripleo-heat-templates master: Don't set boot_option=local on baremetal flavors https://review.openstack.org/618727 | 04:45 |
*** artom has joined #tripleo | 04:46 | |
*** dsneddon has quit IRC | 04:47 | |
*** janki has joined #tripleo | 04:48 | |
*** udesale has joined #tripleo | 04:54 | |
openstackgerrit | Merged openstack/diskimage-builder master: Capture ramdisk test run logs https://review.openstack.org/619400 | 04:57 |
*** kopecmartin has joined #tripleo | 05:00 | |
*** thrash is now known as thrash|g0ne | 05:07 | |
janki | HI. Can I get help merging these puppet-neutron cherry-picks https://review.openstack.org/#/c/618719/ https://review.openstack.org/#/c/618718 | 05:08 |
*** pcaruana has joined #tripleo | 05:09 | |
openstackgerrit | Jaganathan Palanisamy proposed openstack/os-net-config master: Adding pci address in SRIOV config https://review.openstack.org/620456 | 05:13 |
*** ykarel|away has quit IRC | 05:16 | |
*** tzumainn has quit IRC | 05:25 | |
*** ykarel|away has joined #tripleo | 05:31 | |
*** ykarel|away is now known as ykarel | 05:31 | |
*** dsneddon has joined #tripleo | 05:44 | |
*** dsneddon has quit IRC | 05:49 | |
*** fpan has quit IRC | 05:55 | |
*** fpan has joined #tripleo | 05:59 | |
*** ratailor has joined #tripleo | 06:00 | |
*** fpan has quit IRC | 06:12 | |
*** fpan has joined #tripleo | 06:14 | |
*** aedc has quit IRC | 06:25 | |
*** ahosam has joined #tripleo | 06:29 | |
*** saneax has joined #tripleo | 06:30 | |
*** jaosorior has joined #tripleo | 06:36 | |
*** jfrancoa has joined #tripleo | 06:39 | |
*** hjensas has quit IRC | 06:40 | |
*** dhill_ has quit IRC | 06:40 | |
*** udesale has quit IRC | 06:43 | |
*** udesale has joined #tripleo | 06:44 | |
*** dsneddon has joined #tripleo | 06:45 | |
*** udesale has quit IRC | 06:45 | |
*** udesale has joined #tripleo | 06:46 | |
*** psachin has quit IRC | 06:47 | |
*** dhill_ has joined #tripleo | 06:48 | |
*** ramishra has quit IRC | 06:48 | |
*** hakhande_ has joined #tripleo | 06:49 | |
janki | tobias-urdin, hey. Can you help review these puppet-neutron cherry-picks please https://review.openstack.org/#/c/618718/ https://review.openstack.org/#/c/618719/ | 06:49 |
*** dhill_ has quit IRC | 06:54 | |
*** dsneddon has quit IRC | 06:54 | |
Tengu | hello there :) | 06:55 |
*** yprokule has joined #tripleo | 06:55 | |
*** udesale has quit IRC | 06:57 | |
*** rcernin has quit IRC | 06:57 | |
bandini | is this rpm dep issue known? http://logs.openstack.org/43/613043/3/gate/tripleo-ci-centos-7-containers-multinode/887f3c9/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz#_2018-11-21_19_09_03 | 06:58 |
Tengu | again ?! | 06:59 |
Tengu | ah, nah, old. | 06:59 |
*** chkumar|away is now known as chkumar|ruck | 06:59 | |
Tengu | bandini: should be corrected by now | 06:59 |
* bandini rechecks | 07:00 | |
bandini | thanks! | 07:00 |
Tengu | bandini: don't push 7 days old logs in here please X(. Almost had a heart attack | 07:00 |
bandini | 7 days? | 07:00 |
bandini | ooh | 07:00 |
Tengu | yeah ;) | 07:00 |
* bandini looks in the gate | 07:00 | |
*** gkadam has joined #tripleo | 07:00 | |
Tengu | gate is green | 07:00 |
Tengu | :) | 07:00 |
marios | bandini: that was one fixed | 07:01 |
marios | the selinux issue was a few days ago | 07:01 |
marios | where you been man?! | 07:01 |
bandini | lol i just do escalations these days | 07:01 |
marios | chkumar|ruck: are we still hitting that? | 07:01 |
*** wolverineav has joined #tripleo | 07:01 | |
bandini | meh 13h still in the gate. oh well. | 07:01 |
Tengu | marios: nope, I can confirm, it's working fine since a few days already :) | 07:02 |
*** moguimar has joined #tripleo | 07:02 | |
marios | ack Tengu | 07:02 |
bandini | marios: it's fine. I did not notice the review is still in the gates | 07:02 |
marios | bandini: you give us all heart attacks. i spilt my coffee | 07:02 |
marios | but its fine | 07:02 |
marios | :D | 07:02 |
bandini | marios: lol ON YOUR LAPTOP?!? AGAIN!? | 07:02 |
marios | sec i'll get the hairdrier | 07:02 |
bandini | rotfl | 07:02 |
Tengu | uhuhuh | 07:03 |
chkumar|ruck | marios: nope | 07:03 |
marios | thanks chkumar|ruck sorry it seems old logs | 07:03 |
Tengu | and I didn't even get my first cup of tea of the day. | 07:03 |
Tengu | bandini: you're a monster | 07:03 |
*** slaweq has joined #tripleo | 07:04 | |
bandini | lol I get that a lot these days | 07:04 |
marios | haha | 07:04 |
bandini | cannot be a coincidence | 07:04 |
Tengu | :3 | 07:04 |
*** ramishra has joined #tripleo | 07:05 | |
*** wolverineav has quit IRC | 07:06 | |
*** radeks has joined #tripleo | 07:07 | |
*** quiquell|off is now known as quiquell | 07:09 | |
*** skramaja has joined #tripleo | 07:15 | |
openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: Catch socket.timeout waiting port api https://review.openstack.org/619741 | 07:16 |
chkumar|ruck | We are seeing some pypi mirror timed in some jobs | 07:17 |
*** ramishra has quit IRC | 07:18 | |
*** ramishra has joined #tripleo | 07:21 | |
*** mschuppert has joined #tripleo | 07:29 | |
*** dhill_ has joined #tripleo | 07:29 | |
*** cylopez has joined #tripleo | 07:33 | |
*** cylopez has left #tripleo | 07:34 | |
*** ksambor has joined #tripleo | 07:36 | |
*** homegrown has quit IRC | 07:41 | |
*** ykarel is now known as ykarel|lunch | 07:41 | |
*** quiquell is now known as quiquell|brb | 07:43 | |
*** dciabrin has quit IRC | 07:44 | |
*** lennyb_ has joined #tripleo | 07:46 | |
*** homegrown has joined #tripleo | 07:48 | |
*** udesale has joined #tripleo | 07:53 | |
*** hjensas has joined #tripleo | 07:53 | |
*** dsneddon has joined #tripleo | 07:54 | |
*** udesale has quit IRC | 07:58 | |
*** dsneddon has quit IRC | 07:59 | |
*** agurenko has joined #tripleo | 08:00 | |
openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 08:00 |
*** suuuper has joined #tripleo | 08:01 | |
openstackgerrit | Carlos Camacho proposed openstack/instack-undercloud stable/rocky: Fix instack-undercloud PEP8 failures and requirements constraints https://review.openstack.org/619467 | 08:03 |
*** jtomasek has joined #tripleo | 08:08 | |
openstackgerrit | Vieri proposed openstack/python-tripleoclient master: add python 3.6 unit test job https://review.openstack.org/620524 | 08:08 |
*** aedc has joined #tripleo | 08:08 | |
*** quiquell|brb is now known as quiquell | 08:10 | |
*** udesale has joined #tripleo | 08:11 | |
*** dsneddon has joined #tripleo | 08:13 | |
openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Add a dual-bond example NIC config template https://review.openstack.org/604572 | 08:13 |
dtantsur|mtg | morning folks, can I get a 2nd +2 on this docs patch please? https://review.openstack.org/#/c/619287/ | 08:14 |
*** amoralej|off is now known as amoralej | 08:22 | |
*** xek_ has joined #tripleo | 08:22 | |
openstackgerrit | Merged openstack/tripleo-docs master: Document using direct deploy with ironic https://review.openstack.org/619287 | 08:27 |
*** florianf|afk is now known as florianf | 08:30 | |
*** ahosam has quit IRC | 08:31 | |
*** ahosam has joined #tripleo | 08:31 | |
openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common stable/queens: Correct handling of capabilities on node update https://review.openstack.org/618981 | 08:31 |
tobias-urdin | janki: ack | 08:34 |
chkumar|ruck | Tengu: Hello | 08:41 |
chkumar|ruck | Tengu: I need some help to understand https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/840c38b/logs/undercloud/var/log/heat-launcher/undercloud_deploy-itinp1/heat.log.txt.gz?level=ERROR | 08:41 |
*** agurenko has quit IRC | 08:41 | |
chkumar|ruck | Tengu: during undercloud install at heat stack create it is giving internal server error | 08:41 |
chkumar|ruck | Tengu: Please have a look | 08:41 |
Tengu | I'll try. | 08:41 |
*** hberaud|gone is now known as hberaud | 08:43 | |
Tengu | hmm. therve maybe? -^^ | 08:43 |
therve | Looking | 08:43 |
therve | Ah | 08:43 |
therve | Zane broke it | 08:43 |
Tengu | :) | 08:43 |
therve | And I approved it | 08:43 |
Tengu | rooohhh | 08:44 |
Tengu | at least you know what to do/where to look | 08:44 |
therve | Yeah :/ | 08:45 |
chkumar|ruck | therve: https://bugs.launchpad.net/tripleo/+bug/1805589 | 08:45 |
openstack | Launchpad bug 1805589 in tripleo "[master] container build job failed while deploying undercloud with HTTPInternalServerError: ERROR: Internal Error" [Critical,Triaged] | 08:45 |
chkumar|ruck | therve: Tengu :-) | 08:45 |
therve | There are 2 changes stacked, wonder if it'll work | 08:47 |
chkumar|ruck | therve: can you pass me the changes so that i can test it in rdo cloud? | 08:48 |
therve | chkumar|ruck: https://review.openstack.org/620532 | 08:48 |
*** tosky has joined #tripleo | 08:51 | |
chkumar|ruck | therve: testing here https://review.rdoproject.org/r/#/c/17348/ | 08:52 |
*** chem has joined #tripleo | 08:57 | |
*** dmacpher has joined #tripleo | 08:57 | |
*** jpena|off is now known as jpena | 08:57 | |
*** akrivoka has joined #tripleo | 08:59 | |
*** jpich has joined #tripleo | 09:02 | |
*** aedc has quit IRC | 09:02 | |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Invoke redhat-subscription role during upgrade. https://review.openstack.org/610493 | 09:04 |
openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Fix failure to install atop on fedora28 https://review.openstack.org/619595 | 09:05 |
*** shardy has joined #tripleo | 09:10 | |
*** ahosam has quit IRC | 09:10 | |
*** ahosam has joined #tripleo | 09:10 | |
*** owalsh_ is now known as owalsh | 09:12 | |
*** cfontain has joined #tripleo | 09:16 | |
*** bogdando has joined #tripleo | 09:18 | |
*** dmacpher_ has joined #tripleo | 09:23 | |
*** dmacpher has quit IRC | 09:25 | |
openstackgerrit | Debo Zhang proposed openstack/tripleo-docs master: Fix typo https://review.openstack.org/620543 | 09:32 |
*** ykarel|lunch is now known as ykarel | 09:39 | |
shardy | simple review https://review.openstack.org/#/c/607180/ if anyone has a moment? | 09:39 |
*** paramite has quit IRC | 09:40 | |
*** paramite|clone has joined #tripleo | 09:40 | |
shardy | Also need eyes on https://review.openstack.org/#/c/605492/ and https://review.openstack.org/#/c/605682/ which I've been trying to land for 2 months :) | 09:40 |
openstackgerrit | Merged openstack/tripleo-specs master: Replacing the link in HACKING.rst https://review.openstack.org/618366 | 09:43 |
*** akrivoka has quit IRC | 09:44 | |
*** akrivoka has joined #tripleo | 09:44 | |
*** agurenko has joined #tripleo | 09:46 | |
*** ahosam has quit IRC | 09:49 | |
openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Fix the Docker command to get the open_files_limit https://review.openstack.org/620545 | 09:54 |
openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: Add '/usr/share/ansible-modules' at ansible.cfg https://review.openstack.org/620292 | 10:00 |
*** agurenko has quit IRC | 10:06 | |
*** holser_ has joined #tripleo | 10:06 | |
quiquell | marios: ^ | 10:06 |
Tengu | jbadiapa: are you around? I have a question regarding rsyslog config with tripleo :) | 10:09 |
*** tobias-urdin has quit IRC | 10:10 | |
*** artom has quit IRC | 10:11 | |
*** derekh has joined #tripleo | 10:12 | |
marios | quiquell: just back from school run... looking | 10:15 |
*** dougbtv has quit IRC | 10:15 | |
*** tobias-urdin has joined #tripleo | 10:17 | |
*** ahosam has joined #tripleo | 10:19 | |
marios | quiquell: so trying it with recheck on scen1 did you get a good result locally? is kinda hacky but if that is how they are writing out the config file already... | 10:20 |
marios | http://logs.openstack.org/08/619508/8/check/tripleo-ci-centos-7-scenario001-standalone/971275d/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-11-27_12_35_50 | 10:20 |
marios | didn't notice that before "Generating default ansible config" | 10:20 |
quiquell | marios: Doing some cleanup at my thingy, was not able to run it locally | 10:21 |
holser_ | jfrancoa, ccamacho, chem, lbezdick, yprokule, jistr - data provided from lbezdick wrapped into graph for GSS presentation - https://plot.ly/~holser/17/cache-packages-on-all-nodes-update-kernel-on-uc-reboot-uc-with-kexec-uc-minor-up/ | 10:21 |
quiquell | marios: Just ensured that unit tests were ok | 10:21 |
jfrancoa | holser_: 404..link doesn't work :-S (and I'm logged in) | 10:25 |
evrardjp | hola | 10:25 |
holser_ | I deleted it :D | 10:26 |
holser_ | I will give another one | 10:26 |
marios | quiquell: how about just adding an export here https://github.com/openstack/tripleo-quickstart-extras/blob/3be1290957fcc1b58e8c7ba6677afe578075bd25/roles/standalone/tasks/main.yml#L55 | 10:33 |
*** jistr is now known as jistr|mtg | 10:34 | |
openstackgerrit | gaobin proposed openstack/tripleo-heat-templates master: Exclude redundant letters https://review.openstack.org/620550 | 10:35 |
*** salmankhan has joined #tripleo | 10:35 | |
*** wolverineav has joined #tripleo | 10:37 | |
openstackgerrit | Thomas Herve proposed openstack/tripleo-common master: Don't accumulate ansible output uselessly https://review.openstack.org/620552 | 10:38 |
*** wolverineav has quit IRC | 10:42 | |
marios | quiquell: going to try that with scen4 for an alternative/incase | 10:44 |
*** salmankhan has quit IRC | 10:45 | |
*** cylopez has joined #tripleo | 10:45 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: WIP Add ANSIBLE_LIBRARY to linting so we can find modules https://review.openstack.org/620556 | 10:45 |
marios | quiquell: ^ this is what i mean | 10:46 |
openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Add tripleo-ci-centos-7-scenario004-standalone job https://review.openstack.org/619520 | 10:48 |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Fix address for glusterfs container images https://review.openstack.org/620557 | 10:49 |
*** salmankhan has joined #tripleo | 10:53 | |
openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Add puppet-tripleo container https://review.openstack.org/619744 | 10:54 |
openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Optimize base and upper layer for containers https://review.openstack.org/620310 | 10:54 |
bogdando | jistr|mtg: ^^ | 10:55 |
bogdando | jaosorior: ^^ | 10:55 |
bogdando | PTAL, as you have spare time | 10:55 |
*** hberaud is now known as hberaud|lunch | 10:56 | |
*** ahosam has quit IRC | 10:57 | |
*** cylopez has left #tripleo | 11:02 | |
mfedosin | weshay, therve hi :) I finally found the cause of the issue with the reproducer script | 11:04 |
mfedosin | actually there are several issues, and their combination leads to the failure on Fedora 29 | 11:04 |
mfedosin | first: there is a bug with python3-openstackclient | 11:05 |
mfedosin | $ openstack-3 image list --long | 11:05 |
mfedosin | issubclass() arg 1 must be a class | 11:05 |
mfedosin | --long option doesn't work there | 11:06 |
mfedosin | $ openstack image list --long | 11:06 |
mfedosin | fdfd5d39-a76d-45df-abec-17f768ba3054 | 11:06 |
openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: Add '/usr/share/ansible-modules' at ansible.cfg https://review.openstack.org/620292 | 11:07 |
mfedosin | second: when we create a venv the default python version is used here https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2#L234 | 11:07 |
mfedosin | for Fedora it is python 3 | 11:07 |
mfedosin | and third: we don't need --long option here https://github.com/openstack/tripleo-quickstart-extras/blob/da02a39069605658a8ad787c21da09e7c2daa6a1/roles/multinodes/tasks/main.yml | 11:08 |
mfedosin | ID is included in the output by default | 11:08 |
mfedosin | I think we need to fix all three issues | 11:08 |
openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Fix new node detection https://review.openstack.org/618636 | 11:17 |
openstackgerrit | Merged openstack/tripleo-upgrade stable/rocky: Run online data migration before update/upgrade converge. https://review.openstack.org/619968 | 11:17 |
openstackgerrit | Merged openstack/os-apply-config stable/ocata: fixes how os-apply-config handles invalid json https://review.openstack.org/620306 | 11:17 |
openstackgerrit | Merged openstack/tripleo-heat-templates master: mysql: do not overwrite password file during docker-puppet https://review.openstack.org/602969 | 11:17 |
openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: Catch socket.timeout waiting port api https://review.openstack.org/619741 | 11:19 |
*** hgibson has quit IRC | 11:22 | |
ramishra | mfedosin: https://review.openstack.org/#/c/618137/ should fixe the issubclass() issue with osc | 11:24 |
ramishra | mfedosin: I had the issue with 3.7 which I proposed a fix for | 11:24 |
openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Optimize config containers used for puppet steps https://review.openstack.org/620061 | 11:26 |
mfedosin | ramishra, awesome! I'll review it | 11:31 |
openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Optimize config containers used for puppet steps https://review.openstack.org/620061 | 11:33 |
openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Optimize config containers used for puppet steps https://review.openstack.org/620061 | 11:33 |
openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: Add '/usr/share/ansible-modules' at ansible.cfg https://review.openstack.org/620292 | 11:47 |
*** raildo has joined #tripleo | 11:48 | |
openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: Add '/usr/share/ansible-modules' at ansible.cfg https://review.openstack.org/620292 | 11:49 |
openstackgerrit | Carlos Camacho proposed openstack/tripleo-upgrade stable/rocky: Do not adjust roles data for minor updates. https://review.openstack.org/620580 | 11:56 |
*** hberaud|lunch is now known as hberaud | 12:01 | |
*** aufi has joined #tripleo | 12:02 | |
*** jpena is now known as jpena|lunch | 12:02 | |
*** hjensas has quit IRC | 12:02 | |
bogdando | PTAL: https://review.openstack.org/#/c/613300/7 | 12:07 |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Replace scenario001-multinode with scenario001-standalone https://review.openstack.org/620305 | 12:09 |
*** aufi has quit IRC | 12:09 | |
*** holser_ is now known as holser|lunch | 12:09 | |
openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo master: Replace scenario001-multinode with scenario001-standalone https://review.openstack.org/620304 | 12:09 |
*** ansmith has quit IRC | 12:17 | |
*** dougbtv has joined #tripleo | 12:19 | |
*** janki has quit IRC | 12:20 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/rocky: [rocky] Replace scenario001-multinode with scenario001-standalone https://review.openstack.org/620588 | 12:21 |
*** bfournie has quit IRC | 12:25 | |
*** ratailor has quit IRC | 12:26 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/queens: [queens] Replace scenario001-multinode with scenario001-standalone https://review.openstack.org/620589 | 12:26 |
*** artom has joined #tripleo | 12:29 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/pike: [pike] Replace scenario001-multinode with scenario001-standalone https://review.openstack.org/620591 | 12:30 |
*** ahosam has joined #tripleo | 12:33 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/ocata: [ocata] Replace scenario001-multinode with scenario001-standalone https://review.openstack.org/620594 | 12:34 |
marios | quiquell: damn scen 4 failed with a different error didn't get to linting http://logs.openstack.org/20/619520/10/check/tripleo-ci-centos-7-scenario004-standalone/517d04c/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-11-28_12_26_13 | 12:36 |
quiquell | marios: no kidding :-/ | 12:37 |
quiquell | marios: looks like something is broken at puppet-keyston or the like, that's why we need this jobs to gate them with this | 12:39 |
quiquell | marios: also my review is broken :-/ | 12:44 |
quiquell | marios: we are not lucky with scn today | 12:44 |
openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Render playbooks minimal, move everything to roles https://review.openstack.org/607288 | 12:52 |
*** dciabrin has joined #tripleo | 12:53 | |
*** beagles is now known as beagles_mtgs | 12:55 | |
*** rh-jelabarre has joined #tripleo | 12:56 | |
*** redrobot has joined #tripleo | 13:00 | |
*** thrash|g0ne is now known as thrash | 13:00 | |
*** rlandy has joined #tripleo | 13:00 | |
*** jpena|lunch is now known as jpena | 13:02 | |
*** bfournie has joined #tripleo | 13:02 | |
*** dougbtv has quit IRC | 13:03 | |
*** artom has quit IRC | 13:03 | |
*** morazi has quit IRC | 13:04 | |
*** morazi has joined #tripleo | 13:06 | |
dtantsur|mtg | Hi all! would appreciate some eyes on the easy backport: https://review.openstack.org/#/c/618981/ | 13:08 |
*** amoralej is now known as amoralej|lunch | 13:12 | |
*** trown|outtypewww is now known as trown | 13:12 | |
*** ahosam has quit IRC | 13:13 | |
*** artom has joined #tripleo | 13:16 | |
*** holser|lunch is now known as holser_ | 13:16 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Replace scenario001-multinode with scenario001-standalone https://review.openstack.org/620596 | 13:18 |
*** abishop has joined #tripleo | 13:21 | |
*** jistr|mtg is now known as jistr | 13:21 | |
openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade stable/queens: Add timeout to openstack command https://review.openstack.org/620597 | 13:26 |
*** ykarel has quit IRC | 13:29 | |
*** dprince has joined #tripleo | 13:33 | |
*** eck` is now known as eck`gone | 13:37 | |
*** aufi has joined #tripleo | 13:39 | |
*** salmankhan1 has joined #tripleo | 13:42 | |
shardy | dtantsur|mtg: done, when you're done in mtg I have a question for you re metalsmith | 13:42 |
*** salmankhan has quit IRC | 13:43 | |
*** salmankhan1 is now known as salmankhan | 13:43 | |
dtantsur|mtg | shardy: I'm done in mtg on Friday, so ask ahead :) | 13:43 |
shardy | dtantsur|mtg: Hey! | 13:43 |
dtantsur|mtg | o/ | 13:43 |
shardy | dtantsur|mtg: so I've been thinking of ways to drive Ironic standalone/all-in-one, and I had a (slightly crazy) idea | 13:43 |
shardy | would you be open to a mode in metalsmith where we launch the ironic services (with fake RPC and sqlite DB) in a container, like we do with heat for the standalone tripleo deployment? | 13:44 |
shardy | AFAICS the exact same pattern would work, and metalsmith could drive the API, just with an ephemeral Ironic | 13:44 |
dtantsur|mtg | shardy: metalsmith does not lunch ironic, you're confusing it with bifrost. what you're talking about is bifrost with containers :) | 13:45 |
*** mcornea has joined #tripleo | 13:45 | |
shardy | dtantsur|mtg: No, I'm saying metalsmith *could* launch a single container with Ironic inside | 13:45 |
shardy | https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/heat_launcher.py | 13:46 |
*** ykarel has joined #tripleo | 13:46 | |
dtantsur|mtg | shardy: well, that would increase the scope of metalsmith many times. I really like this idea, I just don't think it belongs in metalsmith. | 13:46 |
shardy | dtantsur|mtg: like that ^^ | 13:46 |
dtantsur|mtg | yeah, I know how it works | 13:46 |
shardy | dtantsur|mtg: Ok, cool - I'm just looking for somewhere to prototype it | 13:46 |
dtantsur|mtg | shardy: it sounds like we need a new repo and an ansible role | 13:46 |
shardy | ideally I don't want to use ansible, a standalone CLI tool would fit my current use-case better I think | 13:47 |
openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy https://review.openstack.org/620601 | 13:47 |
dtantsur|mtg | Why? I think ansible is very suitable to what you're trying. | 13:47 |
*** salmankhan has quit IRC | 13:47 | |
*** jcoufal has joined #tripleo | 13:47 | |
shardy | dtantsur|mtg: It fits the (k8s) plugin model I'm looking at better, but I guess it could run ansible | 13:48 |
shardy | dtantsur|mtg: if we did that, we'd have ansible start the container, then drive metalsmith? | 13:48 |
dtantsur|mtg | yep | 13:48 |
shardy | I had in mind more of a one-shot tool with a single configuration input, but that could work I guess | 13:48 |
*** skramaja has quit IRC | 13:49 | |
shardy | dtantsur|mtg: Ok, thanks for the discussion | 13:49 |
dtantsur|mtg | there will be two sides: the container side and the outside (orchestrating) side | 13:49 |
shardy | dtantsur|mtg: FWIW the reason I though of metalsmith is it could be an optional "standalone" mode, e.g you either bring your own ironic or it spawns one for you | 13:50 |
dtantsur|mtg | I think the latter should be ansible, the former is probably a bit of python | 13:50 |
shardy | with minimal dependencies, e.g just launch one container | 13:50 |
openstackgerrit | Raviv proposed openstack/tripleo-upgrade master: Add container image prepare command before undercloud update https://review.openstack.org/620603 | 13:50 |
*** dhill_ has quit IRC | 13:51 | |
shardy | dtantsur|mtg: if you had a fixed input (say list of nodes and end state) I suppose the orchestrating could even happen in the container, but yeah splitting those makes logical sense | 13:51 |
dtantsur|mtg | shardy: fwiw I think this all is just ugly hacks to avoid fixing what openstack does wrong (e.g. rpc) | 13:51 |
shardy | I'm just trying to bend it to fit in the k8s world ;) | 13:51 |
dtantsur|mtg | well, *something* has to be outside of the container | 13:52 |
shardy | dtantsur|mtg: well you could just pass in the config and have it exit when done | 13:52 |
dtantsur|mtg | and if you need to troubleshoot it - you're screwed | 13:53 |
dtantsur|mtg | I would really prefer we do not do this OR at least do not ship it to customers.. | 13:53 |
shardy | dtantsur|mtg: it's exactly what we do for the standalone installer, it has the option of not exiting on failure | 13:53 |
shardy | dtantsur|mtg: sure I'm just brainstorming at this point | 13:53 |
dtantsur|mtg | right, and one thing tripleo is not know for is ease of troubleshooting ;) | 13:54 |
shardy | dtantsur|mtg: meh, I think standalone mode is a big improvement from that perspective, but this is more about how we could cleanly reuse Ironic for k8s without a full undercloud available | 13:55 |
shardy | launching a container fits quite well into that model | 13:55 |
shardy | dtantsur|mtg: thanks for the ideas, I'll think about it some more :) | 13:55 |
dtantsur|mtg | to be clear, launching 3-4 containers is not a huge issue either | 13:55 |
dtantsur|mtg | this IS the k8s model, after all | 13:55 |
*** janki has joined #tripleo | 13:55 | |
shardy | dtantsur|mtg: yeah, but it's about 10 containers including the DB/RPC, even if you want to deploy 3 nodes | 13:57 |
shardy | I was looking for ways to reduce that, particularly when state is managed externally | 13:57 |
dtantsur|mtg | shardy: 6 I think. remove database, you got 5. and if we could do something about RPC requiring a heavy server, could be 4 | 13:58 |
dtantsur|mtg | mmm, even 3? ironic-api, ironic-conductor, ironic-pxe. | 13:59 |
shardy | dtantsur|mtg: yeah I guess, maybe fake RPC could work with a shared mount or something | 14:00 |
shardy | I suppose the exact number isn't important, I was thinking along the heat-all lines for this though | 14:00 |
*** zaneb has joined #tripleo | 14:00 | |
shardy | wrapping 3 services, fake RPC and the DB in one container has the same effect without any code changes | 14:01 |
dtantsur|mtg | shardy: you need to write some code to start ironic in one process (to use fake). on the other hand, if we could create a sane RPC backend.. | 14:01 |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Invoke redhat-subscription role during upgrade. https://review.openstack.org/610493 | 14:02 |
openstackgerrit | Thomas Herve proposed openstack/tripleo-common master: Don't accumulate ansible output uselessly https://review.openstack.org/620552 | 14:02 |
*** amoralej|lunch is now known as amoralej | 14:05 | |
*** salmankhan has joined #tripleo | 14:06 | |
*** saneax has quit IRC | 14:08 | |
*** tzumainn has joined #tripleo | 14:10 | |
*** janki has quit IRC | 14:11 | |
*** janki has joined #tripleo | 14:12 | |
*** dsneddon has quit IRC | 14:13 | |
*** wolverineav has joined #tripleo | 14:14 | |
*** ansmith has joined #tripleo | 14:16 | |
*** wolverineav has quit IRC | 14:19 | |
*** eck`gone is now known as eck` | 14:21 | |
*** udesale has quit IRC | 14:24 | |
*** udesale has joined #tripleo | 14:25 | |
openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy https://review.openstack.org/620601 | 14:26 |
Tengu | bogdando: quick question: why would you prefere "reloaded" over "restarted" for the rsyslog service config? | 14:30 |
bogdando | Tengu: to not lost any messages | 14:30 |
*** mjturek has joined #tripleo | 14:30 | |
*** dhill_ has joined #tripleo | 14:30 | |
bogdando | just a nit tho | 14:31 |
Tengu | bogdando: corrected the nit - the reloaded thing «should» work. | 14:31 |
*** dsneddon has joined #tripleo | 14:31 | |
Tengu | but in fact, losing message isn't a real concern I think in this precise case. It won't restart haproxy unless the config is pushed/changed, and this happens in only two cases: new deploy; update | 14:32 |
*** aufi has quit IRC | 14:40 | |
bogdando | hi folks. While its clear how to amend distgits in RDO, where shall I seek to fix cronie et al? | 14:49 |
bogdando | mwhahaha: around? | 14:50 |
openstackgerrit | Natal Ngétal proposed openstack/tripleo-docs master: [Documentation] Fix link bugs. https://review.openstack.org/620615 | 14:50 |
mwhahaha | bogdando: what's up? | 14:52 |
bogdando | mwhahaha: While its clear how to amend distgits in RDO, where shall I seek to fix cronie et al? | 14:52 |
bogdando | I want to do that https://review.rdoproject.org/r/#/q/topic:base-container-reduction as well | 14:52 |
mwhahaha | bogdando: i don't know where the cronie upstream is. probably in fedora somewhere | 14:53 |
bogdando | sigh | 14:53 |
bogdando | okay | 14:53 |
bogdando | https://review.rdoproject.org/r/#/c/17554/1 :( | 14:56 |
bogdando | mwhahaha: do you know where can I find RDO project in bz to file a bug?.. | 14:58 |
bogdando | for built system... | 14:58 |
bogdando | build* | 14:58 |
mwhahaha | bogdando: it's under community | 14:58 |
mwhahaha | in bz | 14:58 |
bogdando | thanks! | 14:59 |
*** dhill_ has quit IRC | 15:02 | |
openstackgerrit | Mike Fedosin proposed openstack/tripleo-heat-templates master: WIP: Add a role paramter do disable docker reconfiguration during updating https://review.openstack.org/620621 | 15:02 |
*** hjensas has joined #tripleo | 15:05 | |
weshay | jaosorior, http://logs.openstack.org/39/619039/1/gate/openstack-tox-py35/1f6a04a/testr_results.html.gz | 15:05 |
weshay | failing the gate | 15:05 |
*** aufi has joined #tripleo | 15:06 | |
*** aufi has quit IRC | 15:06 | |
openstackgerrit | Carlos Camacho proposed openstack/tripleo-upgrade master: Do not adjust roles data for minor updates. https://review.openstack.org/620262 | 15:11 |
openstackgerrit | Carlos Camacho proposed openstack/tripleo-upgrade master: Do not adjust roles data for minor updates. https://review.openstack.org/620262 | 15:13 |
openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient master: Catch websocket connection close errors in cleanup https://review.openstack.org/620625 | 15:13 |
bogdando | mwhahaha: prolly that's it https://git.centos.org/blob/rpms!cronie.git/0dca7f3c17496d60a79f0c35761a8c8bfe6aba4f/SPECS!cronie.spec | 15:13 |
mwhahaha | bogdando: that's for centos which comes from RHEL | 15:14 |
mwhahaha | bogdando: but future versions might be in fedora | 15:14 |
bogdando | o.O | 15:14 |
openstackgerrit | Daniel Alvarez proposed openstack/tripleo-upgrade master: Do not adjust roles data for minor updates. https://review.openstack.org/620262 | 15:14 |
openstackgerrit | Carlos Camacho proposed openstack/tripleo-upgrade master: Do not adjust roles data for minor updates. https://review.openstack.org/620262 | 15:15 |
*** chkumar|ruck is now known as chkumar|away | 15:17 | |
*** dhill_ has joined #tripleo | 15:17 | |
openstackgerrit | Carlos Camacho proposed openstack/tripleo-upgrade master: Do not adjust roles data for minor updates. https://review.openstack.org/620262 | 15:19 |
openstackgerrit | Mike Fedosin proposed openstack/tripleo-heat-templates master: WIP: Add a role paramter do disable docker reconfiguration during updating https://review.openstack.org/620621 | 15:19 |
openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient master: Remove timeout in config-download https://review.openstack.org/620627 | 15:20 |
openstackgerrit | Carlos Camacho proposed openstack/tripleo-upgrade master: Do not adjust roles data for minor updates. https://review.openstack.org/620262 | 15:21 |
mwhahaha | weshay: ok so who is ruck/rover and why is the gate 22hrs behind? do we know what jobs are failing? is it the stable scenarios? | 15:23 |
weshay | mwhahaha, we are picking the new ruck/rover today.. transition is today | 15:23 |
weshay | I've been poking at it.. a bit as we're in mtgs | 15:23 |
weshay | latest failure http://logs.openstack.org/39/619039/1/gate/openstack-tox-py35/1f6a04a/testr_results.html.gz | 15:24 |
mwhahaha | dat line break | 15:24 |
bogdando | mwhahaha: what about https://github.com/puppetlabs/puppet/blob/master/ext/redhat/puppet.spec.erb | 15:25 |
bogdando | is it the most current place to distribute the change around? | 15:25 |
bogdando | still failing to locate those fedoraproject specs... | 15:26 |
mwhahaha | bogdando: that one number80 handles for us in fedora, check with him | 15:26 |
mwhahaha | bogdando: though what are you trying to remove from the puppet spec? | 15:26 |
bogdando | mwhahaha: systemd | 15:27 |
mwhahaha | uh no | 15:27 |
bogdando | please see for the topic updates | 15:27 |
*** ade_lee has joined #tripleo | 15:27 | |
mwhahaha | cause that'll break the actual package for normal usage | 15:27 |
mwhahaha | i don't think that's a good idea | 15:27 |
bogdando | mwhahaha: can you imagine a sane case when one installs puppet onto a host w/o systemd? | 15:27 |
mwhahaha | techincally you could, so yes? | 15:28 |
mwhahaha | the problem is that puppet assumes systemd in the rhel logic | 15:28 |
bogdando | and for %post et al macros, there is %systemd_ordering | 15:28 |
mwhahaha | so you could break something | 15:28 |
hjensas | bogdando: that spec has %if 0%{?_with_systemd} . So removing that could (should) be done with rpm environment when building the package? Not by changeing the spec file? | 15:28 |
bogdando | hjensas: perhaps it is, just not sure I found a proper place to fix it globally to see the change in tripleo | 15:29 |
bogdando | just like for cronie as well, still can't find it | 15:29 |
openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: Remove ties between ceilometer and panko https://review.openstack.org/620402 | 15:29 |
bogdando | dprince: > the problem is that puppet assumes systemd in the rhel logic | 15:30 |
bogdando | ;( | 15:30 |
bogdando | we prolly cannot fix it for puppet | 15:30 |
bogdando | in packaging | 15:30 |
*** vinaykns has joined #tripleo | 15:30 | |
mwhahaha | it just so happens that we're hacking puppet to ignore all that stuff in containers | 15:30 |
mwhahaha | but i realy don't think the correct fix is to hack all the pacakges up | 15:30 |
sshnaidm | dtantsur|mtg, do you know maybe what could be a problem with ovb here? https://logs.rdoproject.org/32/614632/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/fac9e0e/logs/undercloud/var/log/extra/errors.txt.gz#_2018-11-28_12_04_25_514 | 15:30 |
openstackgerrit | John Fulton proposed openstack/puppet-tripleo master: WIP/DNM: Allow user to define tuned_profile https://review.openstack.org/613698 | 15:31 |
bogdando | I'm fine to keep those a beloved pair together as long as we drop it from the base layer :D | 15:31 |
mwhahaha | i'm not sure this is a good use of resources as i'm still not sold on it's benefits | 15:33 |
Tengu | mwhahaha: care to pas the gate as red? apparently until https://review.openstack.org/#/c/620457/ is merged.... and https://review.openstack.org/#/c/620532/ might also help, after that. | 15:33 |
Tengu | *pass (in the topic) | 15:33 |
mwhahaha | Tengu: we don't track the heat gates? | 15:33 |
dtantsur|mtg | sshnaidm: I suspect the real problem is deployment timeout. and this may be just a side effect of nova trying to clean up. | 15:33 |
bogdando | mwhahaha: not sold for removing 190MB of systemd in each container? | 15:34 |
bogdando | (not saying of security fixes shall be applied for packages still, even if not used) | 15:34 |
mwhahaha | bogdando: once again it's not each container | 15:34 |
bogdando | really?? | 15:34 |
bogdando | it is | 15:34 |
*** Vorrtex has joined #tripleo | 15:34 | |
mwhahaha | bogdando: it's just 190mb period | 15:34 |
mwhahaha | no it's not | 15:34 |
bogdando | it is :D | 15:34 |
mwhahaha | it's 190 from a layer that's shared | 15:34 |
Tengu | mwhahaha: well, promotion blocker and the like ;). nothing can merge for now. | 15:34 |
mwhahaha | no. it's not | 15:34 |
mwhahaha | Tengu: that's not related to our gate | 15:34 |
bogdando | from maintenance pov - it is | 15:34 |
bogdando | from space consumed - not | 15:35 |
Tengu | mwhahaha: okay... | 15:35 |
mwhahaha | it's not any maintence overhead for us | 15:35 |
mwhahaha | we rebuild all this stuff each time anyway | 15:35 |
mwhahaha | we aren't patching a single layer when we update | 15:35 |
*** gkadam has quit IRC | 15:35 | |
mwhahaha | and we don't change systemd that often | 15:35 |
mwhahaha | we have more important issues than this when it comes to tripleo | 15:36 |
weshay | mwhahaha, new ruck rover is marios ( leaving for the day) and ssbarnea|bkp2 | 15:36 |
weshay | mwhahaha, I'll help in a minute | 15:36 |
bogdando | so if some CVE released for systemd we say to all we don't care, keep using vulnerable containers and hope no one can use that dead code, right? | 15:37 |
*** beagles_mtgs is now known as beagles | 15:37 | |
bogdando | mwhahaha: I got the idea then | 15:37 |
mwhahaha | bogdando: that's what people do today when they don't update between releases | 15:37 |
mwhahaha | bogdando: so this is not a foreign concept | 15:37 |
Tengu | well, responsability is not the same. | 15:38 |
mwhahaha | honestly this is the wrong end of things to be addressing for improvements | 15:38 |
bogdando | between releases? I'm talking about a feature parity with hitting yum update | 15:38 |
Tengu | either the USER decides "no update", or WE decide "no update" | 15:38 |
bogdando | to get all CVE fixes in | 15:38 |
mwhahaha | bogdando: downstream we have people who are still running 1+ year code | 15:38 |
bogdando | isn't that an established and recommended practice for hosts? | 15:38 |
mwhahaha | yes but that's an end user responsibility | 15:38 |
mwhahaha | not something *we* need to solve for | 15:38 |
mwhahaha | we provide them the latest, if they choose not to use it that's not something we can force | 15:39 |
bogdando | but we should not alias yum update to noop right? | 15:39 |
bogdando | that's that we do for containers ignoring dead code sitting there | 15:39 |
mwhahaha | the update process assumes they are pulling new containers and doing a host yum update | 15:39 |
mwhahaha | if a customer choose not to update the containers they'll likely run into code issues anyway | 15:39 |
bogdando | ok I give up | 15:39 |
mwhahaha | yes i want to get rid of more code we have in containers, no i don't think it should be at a complexity cost | 15:40 |
mwhahaha | tripleo is already too complex for it's own good | 15:40 |
*** ykarel is now known as ykarel|away | 15:41 | |
mwhahaha | so we need to be very careful on how we tackle these things | 15:41 |
*** quiquell is now known as quiquell|off | 15:41 | |
mwhahaha | i'd rather we properly develop something to get us off puppet than continue to hack at it | 15:41 |
*** udesale has quit IRC | 15:43 | |
*** udesale has joined #tripleo | 15:43 | |
weshay | it would be really nice to have upstream qa +2 some things https://review.openstack.org/#/q/topic:oooq/elastic-recheck+(status:open+OR+status:merged) | 15:43 |
weshay | mwhahaha, would alert bugs on the failing gate issues be helpful atm? | 15:43 |
*** akrivoka has quit IRC | 15:43 | |
mwhahaha | weshay: probably not, can we identify what jobs are failing | 15:44 |
*** akrivoka has joined #tripleo | 15:44 | |
weshay | http://logs.openstack.org/67/619167/2/gate/tripleo-buildimage-overcloud-full-centos-7/8ed7d0b/job-output.txt.gz | 15:45 |
bogdando | mwhahaha: makes sense for future indeed. But I wish us luck deploying Queens over 10,000 of computes with all those fat containers distributed over WAN then... | 15:45 |
*** panda|rover is now known as panda|pto | 15:45 | |
mwhahaha | bogdando: something like this likely cannot be backported to queens | 15:45 |
bogdando | that's why I wish us luck | 15:46 |
mwhahaha | so i'm not even sure that's feasible today | 15:46 |
*** ramishra has quit IRC | 15:46 | |
bogdando | feasible or not, that's the case (DCN) we have on radart | 15:48 |
bogdando | and it is expected to work with Queens | 15:48 |
bogdando | slagle: WDYT? | 15:49 |
mwhahaha | so glad we're forward thinking | 15:49 |
bogdando | not really forward, that's something my team already testing | 15:50 |
bogdando | not yet for 10000 of comuptes tho )) | 15:50 |
slagle | bogdando: i think that optimizing container image size is quite premature given that we can't even deploy 300+ nodes at all with tripleo | 15:51 |
mwhahaha | figuring out how to retrofit something is not forward thinking | 15:51 |
slagle | bogdando: so trying to solve for 10,000 by reducing container image size is...not going to help all that much | 15:51 |
bogdando | slagle: ack, I agree that's premature then | 15:51 |
bogdando | let's wait until real cases for DCN | 15:52 |
openstackgerrit | Mike Fedosin proposed openstack/tripleo-heat-templates master: WIP: Add a role paramter do disable docker reconfiguration during updating https://review.openstack.org/620621 | 15:52 |
bogdando | and fix it adhoc | 15:52 |
bogdando | perhaps using even more advanced band-aiding technics | 15:52 |
hjensas | mwhahaha: I like puppet, but I also like the idea of removing it from containers. If we where to move down the ansible path, like for chrony. Could we stay compatible with customers customizations done using the puppet config classes? | 15:53 |
mwhahaha | hjensas: chrony isn't containerized. that's the thing that has to be solved for if we remove puppet, how does that impact the end user's customizations | 15:54 |
weshay | mwhahaha, infra and tox errors, seeing a disproportionate amount of timeouts in ovh BS1 http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1&var-region=ovh-bhs1 | 15:54 |
mwhahaha | which is why i'd rather see a solution for that than hacking bits | 15:54 |
* weshay chats w/ infra | 15:54 | |
*** rlandy is now known as rlandy|brb | 15:55 | |
*** dsneddon has quit IRC | 15:55 | |
hjensas | mwhahaha: yes, the user feedback from last weeks was clear that they don't like it when they have to change all their customization between releases. | 15:55 |
*** ksambor has quit IRC | 15:55 | |
mwhahaha | hjensas: yes but at somepoint we have to stop carrying backwards compatibilities | 15:55 |
mwhahaha | they also don't like how slow and complex things are :D | 15:56 |
mwhahaha | so it's a trade off, do you want a more reliable deployment or the same config method from Liberty | 15:56 |
mwhahaha | but i don't think we do a good job at exposing that to a user or communicating these choices | 15:57 |
fultonj | some good discussions in here today | 15:58 |
openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-heat-templates master: Mount puppet-generated directory for tripleo-validations https://review.openstack.org/620642 | 15:59 |
bogdando | just a thinking loudly, really, after dealing with lightweight images of etcd or CNI images in k8s world, its weird to see such an acceptance degree of images including OS distro in it, like that's fine | 15:59 |
mwhahaha | i don't think it's fine | 15:59 |
mwhahaha | but i don't think the proposed solutions are an improvement | 15:59 |
mwhahaha | because of the complexity required | 15:59 |
bogdando | just my take of this, don't take it personally :) I know it's fine as far as it works | 15:59 |
mwhahaha | and the impact | 15:59 |
hjensas | The feedback for years was, easier to troubleshoot, less complex. I feel we are adding layer after layer, and this side-car container shared volume mount magic adds another layer. -1 to that. | 15:59 |
bogdando | as long* | 16:00 |
openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 16:00 |
openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 16:00 |
mwhahaha | i hate the size and complexity we're dealing with and do not want to add to it unless we get something significant | 16:00 |
openstackgerrit | Derek Higgins proposed openstack/tripleo-docs master: [WIP] Add documentation on using net-ansible in overcloud https://review.openstack.org/620101 | 16:00 |
bogdando | well, mwhahaha what is complexity? a pod to config a container? | 16:00 |
bogdando | that's the most complex code | 16:01 |
*** suuuper has quit IRC | 16:01 | |
mwhahaha | all of it, the additional complexities added to support a concept and build such a concept. from the proposed patches i do not see a reduction in the complexity | 16:02 |
bogdando | https://review.openstack.org/#/c/620061/7/docker/docker-puppet.py does it look so complex? perhaps switching to native k8s pods or at least podman pods can make it looking even more simple | 16:02 |
mwhahaha | it all comes at a dangerous regression cost as well | 16:02 |
bogdando | what else am I missing as other complexity? | 16:02 |
*** dtrainor has quit IRC | 16:02 | |
mwhahaha | the use cases and new workflows required to support such a thing. i'm not sure we have a 1:1 mapping of the impacts | 16:03 |
*** dtrainor has joined #tripleo | 16:03 | |
mwhahaha | for example removing the puppet and not using the host modules comes at an increased risk of breaking updates/upgrades | 16:03 |
bogdando | that's only for docker puppet steps | 16:03 |
bogdando | and touches nothing else | 16:03 |
mwhahaha | which is kinda important | 16:03 |
bogdando | right, but takes a 15 min of deployment | 16:04 |
*** mcornea has quit IRC | 16:04 | |
bogdando | in ideal world ofc | 16:04 |
bogdando | w/o pull failing | 16:04 |
mwhahaha | that speed is not improved by this | 16:04 |
bogdando | (pulling images was not counted in those 15 min, so it's like 45 min more for a reliable LAN) | 16:04 |
mwhahaha | it's not faster processing, it's not less code being pulled down | 16:04 |
mwhahaha | it's just shuffling the same stuff around | 16:04 |
bogdando | I mean that part of deployment is not that I can see as complexity critical | 16:05 |
bogdando | whatever we do to that part, it's like boiling the ocean of complexity with a tiny heater | 16:06 |
slagle | bogdando: thinking outside the box, i don't know that I even see us running puppet 10k times for 10k nodes. | 16:07 |
slagle | bogdando: tripleo is very inefficient in this regard in lots of places. redoing the same actions for each node, when it's all almost identical | 16:08 |
bogdando | that's a nice item to add to easy-starts :) | 16:08 |
bogdando | https://trello.com/b/oXWIAgJ0/edge-dcn-tripleo-easy-start | 16:08 |
bogdando | I mean we'd still have to address that for Queens sooner or later, right? | 16:09 |
bogdando | or did I get that DCN message wrong?.. | 16:09 |
bogdando | it may be... | 16:09 |
bogdando | slagle: | 16:09 |
slagle | bogdando: i dont know what message you have | 16:10 |
bogdando | so backporting config-download looks like an essential thing to do from that PoV | 16:10 |
slagle | bogdando: but i've not come away that the end goal is to get an ideal edge solution for 10k nodes in queens | 16:10 |
bogdando | slagle: on the team meetings, it was we want to support DCN for Queens (downstream at least) | 16:10 |
bogdando | I did | 16:11 |
bogdando | but for 30-40k tho :D | 16:11 |
bogdando | perhaps that was me listening bad | 16:11 |
bogdando | which triggered panic mode with my DF hat on | 16:12 |
bogdando | sorry for that :D | 16:12 |
openstackgerrit | Carlos Camacho proposed openstack/instack-undercloud stable/rocky: Fix instack-undercloud PEP8 failures and requirements constraints https://review.openstack.org/619467 | 16:12 |
*** zbitter has joined #tripleo | 16:12 | |
slagle | many interested parties will always say they want such and such in such and such release. it's our job to say what is possible and feasible and in what timeframe | 16:13 |
bogdando | indeed | 16:14 |
slagle | but regadless, i don't think any of these scale issues are going to be solved by many micro-optimizations | 16:14 |
shardy | slagle: Hey any chance of a review on https://review.openstack.org/#/c/605492/ https://review.openstack.org/#/c/605682 and https://review.openstack.org/#/c/607180/ when you get a moment? | 16:14 |
slagle | we need to be thinking in different terms | 16:14 |
bogdando | hm, the very nature of our daily work is those many micro chagnes | 16:14 |
bogdando | no one likes to review 50k locs | 16:14 |
shardy | two are related to the bootstrap bug we discussed in Berlin and have been around for quite a while | 16:14 |
bogdando | so I'm not sure I agree | 16:15 |
bogdando | step by step, that's the way we do things | 16:15 |
*** zaneb has quit IRC | 16:15 | |
*** zbitter is now known as zaneb | 16:16 | |
slagle | shardy: sure, will take a look | 16:16 |
shardy | thanks! | 16:17 |
bogdando | and sometimes tweaking a single sysctl may be a feasible micro optimization for large scale | 16:18 |
bogdando | or readconnection timeout for galera ;) | 16:18 |
bogdando | I can be telling examples for long | 16:18 |
*** mcornea has joined #tripleo | 16:19 | |
*** agopi is now known as agopi|food | 16:20 | |
dtrainor | jaosorior, i think this may need a backport to rocky, thoughts? https://bugs.launchpad.net/tripleo/+bug/1799724 | 16:20 |
openstack | Launchpad bug 1799724 in tripleo "During undercloud installation /etc/hosts gets bad entry: '192.168.0.1 192.168.24.2'" [Medium,Fix released] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles) | 16:20 |
dtrainor | we have a downstream bug that sounds familiar, i'm going to test today | 16:21 |
dtrainor | *sounds related | 16:21 |
openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/rocky: Standalone: Disable persisting VIPs on /etc/hosts https://review.openstack.org/620648 | 16:22 |
openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-common master: Stops growth of massive dentry cache growth https://review.openstack.org/620649 | 16:22 |
mwhahaha | dtrainor: likely,-^ is the first cherry pick. the other isn't clean so i'll take a look at it in a bit | 16:22 |
openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Fix Swift S3 API configuration https://review.openstack.org/620650 | 16:24 |
openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient stable/rocky: Undercloud: Disable persisting VIPs in /etc/hosts https://review.openstack.org/620652 | 16:27 |
openstackgerrit | Merged openstack/ansible-role-redhat-subscription master: sat6: pull & deploy katello certs https://review.openstack.org/620176 | 16:30 |
*** hakhande_ has quit IRC | 16:32 | |
*** rlandy|brb is now known as rlandy | 16:34 | |
*** udesale has quit IRC | 16:34 | |
openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Add TunedCustomProfile parameter and HCI Ceph filestore environment https://review.openstack.org/613699 | 16:38 |
*** hakhande_ has joined #tripleo | 16:38 | |
*** hakhande_ is now known as hakhande | 16:38 | |
*** yprokule has quit IRC | 16:41 | |
*** rbrady has joined #tripleo | 16:42 | |
openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs master: Update the baremetal overcloud documentation with Stein changes https://review.openstack.org/620658 | 16:59 |
*** pcaruana has quit IRC | 17:03 | |
openstackgerrit | Merged openstack/ansible-role-redhat-subscription master: Make it more explicit that rhsm_activation_key and rhsm_repos are exclusive https://review.openstack.org/620380 | 17:03 |
openstackgerrit | Janki Chhatbar proposed openstack/puppet-tripleo stable/queens: Pass variable from puppet-tripleo to puppet-neutron https://review.openstack.org/620013 | 17:08 |
*** cfontain has quit IRC | 17:09 | |
openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient master: Catch websocket connection close errors in cleanup https://review.openstack.org/620625 | 17:20 |
*** ssbarnea|bkp2 has quit IRC | 17:20 | |
*** pbandark has joined #tripleo | 17:25 | |
*** ssbarnea has joined #tripleo | 17:29 | |
*** bogdando has quit IRC | 17:34 | |
openstackgerrit | Janki Chhatbar proposed openstack/puppet-tripleo stable/rocky: Pass variable from puppet-tripleo to puppet-neutron https://review.openstack.org/620012 | 17:38 |
*** jpich has quit IRC | 17:40 | |
*** janki has quit IRC | 17:41 | |
*** trown is now known as trown|lunch | 17:48 | |
Hobbestigrou | marios, please can you review this patch https://review.openstack.org/#/c/620615/ ? | 17:49 |
*** florianf is now known as florianf|afk | 17:50 | |
*** hakhande has quit IRC | 17:56 | |
*** sshnaidm is now known as sshnaidm|afk | 18:01 | |
openstackgerrit | Jill Rouleau proposed openstack/paunch master: WIP: Add systemd healthchecks to podman https://review.openstack.org/620372 | 18:02 |
dtrainor | Anyone have a few minutes to talk to me more about flavor resources, particularly the default CUSTOM_BAREMETAL resource assigned to the 'resource' property of a flavor, why its value is set to 1, and how this applies to node tagging, in particular a node's profile:<tag> property, if at all? | 18:05 |
dtrainor | a node also has a resource_class property, it's not clear if this should match a flavor's resource property either | 18:05 |
dtrainor | i suspect node tagging not working because of this in downstream queens | 18:06 |
*** aufi has joined #tripleo | 18:06 | |
*** jfrancoa has quit IRC | 18:06 | |
*** hberaud is now known as hberaud|gone | 18:07 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: [DNM] Test Centos 7.6 https://review.openstack.org/618832 | 18:07 |
openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: [DNM] Test workflow with Centos7.6 https://review.openstack.org/618780 | 18:07 |
dtrainor | it would seem to me that each flavor should have a CUSTOM_BAREMETAL_<role>, or something that matches which role a node is tagged with. The node's resource_class property should match the flavor's resource:CUSTOM_BAREMETAL_<role> | 18:08 |
*** jpena is now known as jpena|off | 18:09 | |
*** shardy has quit IRC | 18:11 | |
dtrainor | dtantsur|mtg, am I on the right track here? | 18:12 |
openstackgerrit | Jill Rouleau proposed openstack/tripleo-ansible master: Add podman_container module https://review.openstack.org/620423 | 18:16 |
*** dmacpher_ has quit IRC | 18:17 | |
*** wolverineav has joined #tripleo | 18:23 | |
openstackgerrit | Dan Radez proposed openstack/puppet-tripleo stable/queens: Add manifest for networking_ansible ML2 plugin https://review.openstack.org/620674 | 18:24 |
*** dsneddon has joined #tripleo | 18:25 | |
*** salmankhan has quit IRC | 18:26 | |
openstackgerrit | Dan Radez proposed openstack/tripleo-common stable/queens: Install python2-networking-ansible in neutron-server https://review.openstack.org/620675 | 18:27 |
*** holser_ has quit IRC | 18:27 | |
*** agopi|food is now known as agopi | 18:33 | |
*** amoralej is now known as amoralej|off | 18:36 | |
*** wolverineav has quit IRC | 18:50 | |
*** akrivoka has quit IRC | 18:51 | |
*** wolverineav has joined #tripleo | 18:51 | |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Set container images for openshift 3.11 https://review.openstack.org/613165 | 18:56 |
*** trown|lunch is now known as trown | 18:59 | |
*** preece has joined #tripleo | 19:00 | |
*** ykarel|away has quit IRC | 19:03 | |
*** kopecmartin is now known as kopecmartin|off | 19:15 | |
*** eck` is now known as eck`gone | 19:23 | |
*** radeks_ has joined #tripleo | 19:25 | |
*** radeks has quit IRC | 19:26 | |
*** aufi has quit IRC | 19:29 | |
*** aufi has joined #tripleo | 19:51 | |
*** radeks has joined #tripleo | 19:53 | |
*** radeks_ has quit IRC | 19:55 | |
*** wolverineav has quit IRC | 20:05 | |
*** wolverineav has joined #tripleo | 20:07 | |
*** eck`gone is now known as eck` | 20:16 | |
*** wolverineav has quit IRC | 20:16 | |
openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo stable/queens: Dell EMC SC: Add use_multipath_for_image_xfer https://review.openstack.org/616687 | 20:19 |
openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo stable/rocky: Dell EMC SC: Add use_multipath_for_image_xfer https://review.openstack.org/616685 | 20:22 |
weshay | fyi.. https://review.openstack.org/#/c/620630/2/roles/configure-mirrors/templates/etc/pip.conf.j2 | 20:23 |
*** zaneb has quit IRC | 20:26 | |
*** zaneb has joined #tripleo | 20:27 | |
*** openstackgerrit has quit IRC | 20:36 | |
*** ahosam has joined #tripleo | 20:48 | |
*** dprince has quit IRC | 20:48 | |
*** wolverineav has joined #tripleo | 20:48 | |
*** raildo has quit IRC | 20:54 | |
weshay | mwhahaha, leaving the gate queue as is? | 20:55 |
mwhahaha | unless you've got something specific to fix it | 20:56 |
mwhahaha | at the moment i'm not sure what is causing the backup (where we're failing) | 20:56 |
weshay | mwhahaha, I hear AI is the all the rage | 20:56 |
mwhahaha | i checked zuul and it seemed to be various timeouts but nothing consistent in terms of branches or jobs | 20:56 |
weshay | as usual, not one thing | 20:56 |
weshay | aye | 20:56 |
*** vpickard is now known as vpickard_ | 20:56 | |
mwhahaha | i was pondering if removing the scenarios from the stable gates might help | 20:56 |
mwhahaha | because we were merging things fine w/o them for a few days | 20:57 |
weshay | meh.. updates and containers-multinode is failing as much as scenarios on stable branches | 20:58 |
* weshay blames, yum, pip and ovh BHS1 | 20:59 | |
weshay | and mwhahaha favorite band | 20:59 |
mwhahaha | :o | 21:00 |
weshay | oh and Queue: starlingx | 21:00 |
*** wolverineav has quit IRC | 21:02 | |
*** wolverineav has joined #tripleo | 21:06 | |
*** openstackgerrit has joined #tripleo | 21:08 | |
openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates master: Optional ICMP validation of controllers and gateways https://review.openstack.org/620698 | 21:08 |
openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates master: Optional ICMP validation of controllers and gateways https://review.openstack.org/620698 | 21:09 |
openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates master: Optional ICMP validation of controllers and gateways https://review.openstack.org/620698 | 21:11 |
openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates master: Optional ICMP validation of controllers and gateways https://review.openstack.org/620698 | 21:13 |
*** wolverineav has quit IRC | 21:22 | |
*** wolverineav has joined #tripleo | 21:23 | |
*** wolverineav has quit IRC | 21:23 | |
*** wolverineav has joined #tripleo | 21:23 | |
*** aufi has quit IRC | 21:23 | |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Set container images for openshift 3.11 https://review.openstack.org/613165 | 21:28 |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Set container images for openshift 3.11 https://review.openstack.org/613165 | 21:31 |
*** radeks has quit IRC | 21:32 | |
*** bfournie has quit IRC | 21:36 | |
*** dprince has joined #tripleo | 21:38 | |
openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Remove non-voting jobs from the gate https://review.openstack.org/620705 | 21:39 |
*** dprince has quit IRC | 21:43 | |
*** rcernin has joined #tripleo | 21:50 | |
openstackgerrit | Trevor Vardeman proposed openstack/tripleo-common master: Add manifest list image support to the kolla builder https://review.openstack.org/620397 | 21:51 |
*** ansmith has quit IRC | 21:59 | |
*** Vorrtex has quit IRC | 21:59 | |
*** rcernin has quit IRC | 22:03 | |
*** jcoufal has quit IRC | 22:06 | |
*** jcoufal has joined #tripleo | 22:11 | |
*** abishop has quit IRC | 22:11 | |
*** rcernin has joined #tripleo | 22:12 | |
*** trown is now known as trown|outtypewww | 22:14 | |
*** xek_ has quit IRC | 22:19 | |
*** rh-jelabarre has quit IRC | 22:24 | |
*** artom has quit IRC | 22:26 | |
*** artom has joined #tripleo | 22:28 | |
*** artom has quit IRC | 22:29 | |
*** artom has joined #tripleo | 22:29 | |
*** cfontain has joined #tripleo | 22:31 | |
*** jcoufal has quit IRC | 22:41 | |
*** cfontain has quit IRC | 22:42 | |
*** derekh has quit IRC | 22:43 | |
*** derekh has joined #tripleo | 22:44 | |
*** derekh has quit IRC | 22:44 | |
*** slaweq has quit IRC | 22:46 | |
*** ahosam has quit IRC | 22:51 | |
*** trown|outtypewww has quit IRC | 22:54 | |
*** trown has joined #tripleo | 22:55 | |
*** wolverineav has quit IRC | 23:01 | |
*** wolverineav has joined #tripleo | 23:02 | |
*** mcornea has quit IRC | 23:02 | |
*** wolverineav has quit IRC | 23:03 | |
*** wolverineav has joined #tripleo | 23:03 | |
*** vinaykns has quit IRC | 23:11 | |
*** rlandy is now known as rlandy|bbl | 23:33 | |
openstackgerrit | Merged openstack/python-tripleoclient stable/rocky: Add docker bridge ip option to the undercloud.conf https://review.openstack.org/619167 | 23:35 |
openstackgerrit | Merged openstack/tripleo-ui master: Enable deletion of deployment upon successful deployment https://review.openstack.org/620332 | 23:35 |
mwhahaha | 30 hours later, merged. dat speed | 23:36 |
*** rpioso is now known as rpioso|afk | 23:41 | |
*** wolverineav has quit IRC | 23:42 | |
*** wolverineav has joined #tripleo | 23:44 | |
mwhahaha | meh timeout at the top of the gate. glorious | 23:46 |
* mwhahaha gives up | 23:46 | |
fultonj | :( | 23:46 |
openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui stable/rocky: Enable deletion of deployment upon successful deployment https://review.openstack.org/620752 | 23:50 |
jrist | honza, dtrainor, florianf|afk, jtomasek https://review.openstack.org/#/c/620752/ +2 plz | 23:50 |
*** wolverineav has quit IRC | 23:51 | |
*** slaweq has joined #tripleo | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!