| *** tosky has quit IRC | 00:05 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Fix old docs listing octavia as experimental https://review.openstack.org/636776 | 00:06 |
|---|---|---|
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/639544 | 00:06 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/639533 | 00:06 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests https://review.openstack.org/639550 | 00:06 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-repo_server master: pypiserver: drop pypi server https://review.openstack.org/648556 | 00:10 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-repo_server master: Test using integrated build https://review.openstack.org/648563 | 00:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_almanach master: Add bugs url link to README https://review.openstack.org/622126 | 00:15 |
| *** ianychoi has quit IRC | 00:17 | |
| *** ianychoi has joined #openstack-ansible | 00:18 | |
| *** weshay has quit IRC | 00:19 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create master: Add bugs url link to README https://review.openstack.org/622117 | 00:19 |
| *** weshay has joined #openstack-ansible | 00:20 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: gate: cache /etc/pip.conf inside containers https://review.openstack.org/648566 | 00:30 |
| *** markvoelker has joined #openstack-ansible | 00:40 | |
| *** hamzy has joined #openstack-ansible | 00:44 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: developer_mode: drop vars https://review.openstack.org/648569 | 01:00 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: Test using integrated build https://review.openstack.org/647856 | 01:00 |
| *** yolanda has quit IRC | 01:07 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Test using integrated build https://review.openstack.org/648509 | 01:11 |
| *** yolanda has joined #openstack-ansible | 01:15 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: ironic: deploy swift at the same time https://review.openstack.org/648574 | 01:19 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: gate: cache /etc/pip.conf inside containers https://review.openstack.org/648566 | 01:25 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: swift: move to br-storage always https://review.openstack.org/648575 | 01:31 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Test using integrated build https://review.openstack.org/648512 | 01:31 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_swift master: Test using integrated build https://review.openstack.org/648506 | 01:32 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/639544 | 01:37 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Fix old docs listing octavia as experimental https://review.openstack.org/636776 | 01:37 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/639533 | 01:38 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_gnocchi master: Test using integrated build https://review.openstack.org/648515 | 01:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_aodh master: Test using integrated build https://review.openstack.org/647851 | 01:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_panko master: Test using integrated build https://review.openstack.org/648514 | 01:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ceilometer master: Test using integrated build https://review.openstack.org/648513 | 01:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Update role for new source build process https://review.openstack.org/648463 | 01:51 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_swift master: Update role for new source build process https://review.openstack.org/648465 | 01:51 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: barbican: run without SSL inside gate for metal jobs https://review.openstack.org/648234 | 01:57 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: gate: switch all public URLs to use HTTP for metal https://review.openstack.org/648579 | 01:57 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: barbican: run without SSL inside gate for metal jobs https://review.openstack.org/648234 | 01:59 |
| *** cshen has joined #openstack-ansible | 02:11 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: developer_mode: drop vars https://review.openstack.org/648569 | 02:11 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: Test using integrated build https://review.openstack.org/647856 | 02:11 |
| *** cshen has quit IRC | 02:15 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_tempest master: Test using integrated build https://review.openstack.org/648580 | 02:16 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_tacker master: Test using integrated build https://review.openstack.org/648581 | 02:22 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_trove master: Test using integrated build https://review.openstack.org/648583 | 02:28 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_designate master: Test using integrated build https://review.openstack.org/648584 | 02:31 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: zuul: bump timeout to 3 hours https://review.openstack.org/648585 | 02:36 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_designate master: Test using integrated build https://review.openstack.org/648584 | 02:38 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Test using integrated build https://review.openstack.org/648509 | 02:39 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Test using integrated build https://review.openstack.org/648512 | 02:58 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_swift master: Test using integrated build https://review.openstack.org/648506 | 03:10 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests https://review.openstack.org/639550 | 03:10 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_swift master: Update role for new source build process https://review.openstack.org/648465 | 03:10 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Replace usage of netloc filters to urlsplit https://review.openstack.org/648190 | 03:11 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: missing document start "---" https://review.openstack.org/614107 | 03:11 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Update role for new source build process https://review.openstack.org/648463 | 03:11 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/639533 | 03:11 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Break out config files for each octavia service https://review.openstack.org/603228 | 03:12 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Update role for new source build process https://review.openstack.org/648231 | 03:12 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/639544 | 03:12 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Fix old docs listing octavia as experimental https://review.openstack.org/636776 | 03:12 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: repo: stop using repo_build + pip_install https://review.openstack.org/648477 | 03:20 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: gate: cache /etc/pip.conf inside containers https://review.openstack.org/648566 | 03:20 |
| *** weshay has quit IRC | 03:24 | |
| *** weshay has joined #openstack-ansible | 03:24 | |
| *** weshay has quit IRC | 03:24 | |
| *** weshay has joined #openstack-ansible | 03:25 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_tempest master: manila: add image_with_share_tools https://review.openstack.org/648593 | 03:28 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: Test using integrated build https://review.openstack.org/647856 | 03:28 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_trove master: Test using integrated build https://review.openstack.org/648583 | 03:35 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: zuul: bump timeout to 3 hours https://review.openstack.org/648585 | 03:36 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_horizon master: Test using integrated build https://review.openstack.org/648595 | 03:49 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_magnum master: Test using integrated build https://review.openstack.org/648597 | 03:52 |
| *** raukadah is now known as chandankumar | 03:53 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_rally master: Test using integrated build https://review.openstack.org/648598 | 03:56 |
| *** udesale has joined #openstack-ansible | 03:56 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_sahara master: Test using integrated build https://review.openstack.org/648599 | 03:58 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_mistral master: Test using integrated build https://review.openstack.org/648600 | 04:00 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_designate master: Test using integrated build https://review.openstack.org/648584 | 04:02 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_mistral master: Test using integrated build https://review.openstack.org/648600 | 04:03 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Test using integrated build https://review.openstack.org/648509 | 04:07 |
| *** cshen has joined #openstack-ansible | 04:10 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Test using integrated build https://review.openstack.org/648512 | 04:14 |
| *** cshen has quit IRC | 04:14 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Replace usage of netloc filters to urlsplit https://review.openstack.org/648190 | 04:15 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: missing document start "---" https://review.openstack.org/614107 | 04:15 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Update role for new source build process https://review.openstack.org/648463 | 04:15 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/639533 | 04:15 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/639544 | 04:16 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Break out config files for each octavia service https://review.openstack.org/603228 | 04:16 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Update role for new source build process https://review.openstack.org/648231 | 04:16 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Fix old docs listing octavia as experimental https://review.openstack.org/636776 | 04:16 |
| johnsom | mnaser: dude, what is going on? Like 12 rebases today. | 04:17 |
| mnaser | johnsom: working on getting os_octavia passing in our integrated gate | 04:18 |
| mnaser | and rebasing previous old patches on top of it so we land them all together :> | 04:18 |
| mnaser | sorry for the noise, I can remove you from the review in the meantime | 04:18 |
| johnsom | No worries, just wondering if you needed a second set of eyes on something. | 04:19 |
| mnaser | johnsom: slowly making my way to get the integrated gate running for octavia, thanks for the thought :) | 04:20 |
| johnsom | Looking forward to some solid gates for the ansible stuff. | 04:20 |
| mnaser | johnsom: yeah with these changes this stuff is going to be super solid, im really happy, we're uncovering a lot of things. | 04:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: missing document start "---" https://review.openstack.org/614115 | 04:25 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: fix tox python3 overrides https://review.openstack.org/606786 | 04:27 |
| *** logan- has joined #openstack-ansible | 04:33 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_trove master: Test using integrated build https://review.openstack.org/648583 | 04:49 |
| mnaser | cores: I tried to get as much patches passing as possible, there's a whole bunch of things depending on things, so appreciate as much reviews to get as much stuff to land as possible. if you see anything broken (of my reviews), please go ahead and fix it (till I come in tomorrow AM, to speed things up). we need to land as much of this as we can for our Saturday cutoff :) | 04:51 |
| mnaser | thanks! <3 | 04:51 |
| *** shyamb has joined #openstack-ansible | 05:20 | |
| *** shyamb has quit IRC | 05:47 | |
| *** shyamb has joined #openstack-ansible | 05:54 | |
| *** cshen has joined #openstack-ansible | 06:10 | |
| *** cshen has quit IRC | 06:14 | |
| *** SmearedBeard has joined #openstack-ansible | 06:25 | |
| *** udesale has quit IRC | 06:41 | |
| *** udesale has joined #openstack-ansible | 06:41 | |
| *** SmearedBeard has quit IRC | 06:51 | |
| *** DanyC has joined #openstack-ansible | 06:57 | |
| *** kopecmartin|off is now known as kopecmartin | 07:07 | |
| *** cshen has joined #openstack-ansible | 07:26 | |
| jrosser | reviews on this would be good: https://review.openstack.org/#/c/648551/ it's a depends-on for a ton of stuff | 07:31 |
| *** shyamb has quit IRC | 07:34 | |
| *** luksky has joined #openstack-ansible | 07:35 | |
| *** shyamb has joined #openstack-ansible | 07:43 | |
| evrardjp | in the future we should probably remove the requirements install branch and use the project's install branch as a basis to find requirements install branch, what do you think? | 07:51 |
| evrardjp | reduction of variables, and improves installability | 07:51 |
| *** shyamb has quit IRC | 07:54 | |
| *** fghaas has joined #openstack-ansible | 07:54 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: developer_mode: drop vars https://review.openstack.org/648569 | 08:01 |
| fnpanic | good morning | 08:10 |
| *** nowster has quit IRC | 08:13 | |
| *** nowster has joined #openstack-ansible | 08:13 | |
| *** tosky has joined #openstack-ansible | 08:15 | |
| *** DanyC has quit IRC | 08:23 | |
| *** DanyC has joined #openstack-ansible | 08:26 | |
| *** gillesMo has joined #openstack-ansible | 08:27 | |
| *** DanyC has quit IRC | 08:34 | |
| *** shyamb has joined #openstack-ansible | 08:37 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari master: Fixed variable in example playbook https://review.openstack.org/648644 | 08:39 |
| *** udesale has quit IRC | 08:57 | |
| *** udesale has joined #openstack-ansible | 08:58 | |
| *** goldenfri has quit IRC | 09:04 | |
| *** goldenfri has joined #openstack-ansible | 09:05 | |
| CeeMac | morning | 09:06 |
| zhongjun2 | morning | 09:06 |
| *** udesale has quit IRC | 09:07 | |
| *** shyamb has quit IRC | 09:09 | |
| *** shyamb has joined #openstack-ansible | 09:09 | |
| *** udesale has joined #openstack-ansible | 09:10 | |
| *** DanyC has joined #openstack-ansible | 09:13 | |
| *** pcaruana has joined #openstack-ansible | 09:15 | |
| *** udesale has quit IRC | 09:15 | |
| *** cshen has quit IRC | 09:19 | |
| *** priteau has joined #openstack-ansible | 09:33 | |
| *** aedc has joined #openstack-ansible | 09:33 | |
| *** SmearedBeard has joined #openstack-ansible | 09:34 | |
| *** electrofelix has joined #openstack-ansible | 09:37 | |
| odyssey4me | morning all | 09:39 |
| *** cshen has joined #openstack-ansible | 09:39 | |
| odyssey4me | evrardjp it'd be nice to simplify things, yeah - the general idea I had was to have something that would move the pins from the integrated repo into the roles, then automate a proposal to change those on a weekly basis | 09:41 |
| odyssey4me | great work yesterday mnaser, wow! I'll start working through failures and see what I can do to help get them mooving | 09:45 |
| odyssey4me | mnaser the reason horizon is not setup on the integrated build for metal builds has something to do with conflicting ports IIRC, evrardjp would have to share more details if he can remember | 09:46 |
| *** shyamb has quit IRC | 09:52 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: metal: sync all containers to add is_metal https://review.openstack.org/648292 | 09:55 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: gate: switch all public URLs to use HTTP for metal https://review.openstack.org/648579 | 09:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_server master: pypiserver: drop pypi server https://review.openstack.org/648556 | 10:02 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_server master: Test using integrated build https://review.openstack.org/648563 | 10:03 |
| *** shyamb has joined #openstack-ansible | 10:05 | |
| evrardjp | odyssey4me: probably ... + I didn't care. :p | 10:12 |
| evrardjp | most likely horizon + repo server were running on http/https | 10:12 |
| evrardjp | so if repo server goes away we can do it | 10:12 |
| evrardjp | (it was also very different configuration + apt proxy at taht time | 10:13 |
| odyssey4me | evrardjp ah, that makes sense - nginx (repo server) and apache (horizon) can't share a host | 10:13 |
| odyssey4me | well, they could - but only if they don't conflict on ports | 10:13 |
| evrardjp | yes on the same port | 10:13 |
| evrardjp | :) | 10:13 |
| evrardjp | correct | 10:14 |
| evrardjp | so all this wiring for something I didn't care... I didn't bother | 10:14 |
| *** CobHead has quit IRC | 10:14 | |
| evrardjp | it's like haproxy -- I didn't see the reason why to add this layer, when you could directly hit the services | 10:15 |
| evrardjp | if you have a single node that has no point to me | 10:15 |
| evrardjp | if you start to have multiple backends... that's a different story | 10:15 |
| openstackgerrit | Merged openstack/openstack-ansible master: nova: refactor common playbook to use role instead of include_role https://review.openstack.org/648551 | 10:16 |
| openstackgerrit | Merged openstack/openstack-ansible master: zuul: bump timeout to 3 hours https://review.openstack.org/648585 | 10:16 |
| openstackgerrit | Merged openstack/openstack-ansible master: Add zero-container AIO distro install jobs https://review.openstack.org/647840 | 10:16 |
| *** CobHead has joined #openstack-ansible | 10:16 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Remove experimetal scenarios https://review.openstack.org/647897 | 10:16 |
| evrardjp | I believe we should have playbooks that are different for installing in the first time, so that we don't run the serial way nowadays by default. It will simplify and maybe run faster because role tests for 1h30 that's long | 10:17 |
| evrardjp | on metal | 10:17 |
| odyssey4me | evrardjp well, the repo still has a function in a single node build, because it compiles the wheels and that speeds up deployment, especially because many services use the same python packages as deps | 10:17 |
| evrardjp | but is that a webserver? | 10:17 |
| evrardjp | or a file cache? | 10:17 |
| odyssey4me | so for now we'll need to make the horizon test use a containerised build for testing, just to get things to a state where all tests use the integrated build... then we can figure out how to solve the clashing web servers later | 10:18 |
| evrardjp | yes | 10:18 |
| odyssey4me | oh sure, it could just be a file cache - but that would require adding a ton of complexity for a single node use-case which I don't think is worth it | 10:18 |
| evrardjp | I didn't know what you were trying to solve with this conversation :) | 10:18 |
| evrardjp | ok | 10:18 |
| evrardjp | it's fine let's not make things more complex :) | 10:19 |
| evrardjp | http works | 10:19 |
| evrardjp | :p | 10:19 |
| *** hamzaachi has joined #openstack-ansible | 10:20 | |
| *** aedc has quit IRC | 10:22 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon master: Test using integrated build https://review.openstack.org/648595 | 10:22 |
| *** aedc has joined #openstack-ansible | 10:22 | |
| odyssey4me | :) I was just trying to recall why we couldn't have horizon in the metal builds... so thanks for reminding me :) | 10:23 |
| *** shyam89 has joined #openstack-ansible | 10:24 | |
| *** shyamb has quit IRC | 10:28 | |
| evrardjp | :) | 10:33 |
| *** hamzaachi has quit IRC | 10:35 | |
| *** aedc has quit IRC | 10:36 | |
| *** hamzaachi has joined #openstack-ansible | 10:37 | |
| *** shyam89 has quit IRC | 10:37 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_zun master: Update role for new source build process https://review.openstack.org/648380 | 10:44 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: repo: stop using repo_build + pip_install https://review.openstack.org/648477 | 10:45 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: repo: stop using repo_build + pip_install https://review.openstack.org/648477 | 10:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: gate: cache /etc/pip.conf inside containers https://review.openstack.org/648566 | 10:50 |
| *** SmearedBeard has quit IRC | 10:58 | |
| *** SmearedBeard has joined #openstack-ansible | 10:59 | |
| *** shyam89 has joined #openstack-ansible | 11:01 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Add telemetry service bundle to AIO tests https://review.openstack.org/647921 | 11:16 |
| *** hamzaachi has quit IRC | 11:27 | |
| *** shyam89 has quit IRC | 11:39 | |
| *** shyam89 has joined #openstack-ansible | 11:40 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.openstack.org/645937 | 11:55 |
| *** Pandry has joined #openstack-ansible | 11:59 | |
| *** shyam89 has quit IRC | 11:59 | |
| openstackgerrit | Merged openstack/openstack-ansible master: ironic: deploy swift at the same time https://review.openstack.org/648574 | 12:05 |
| openstackgerrit | Merged openstack/openstack-ansible master: swift: move to br-storage always https://review.openstack.org/648575 | 12:05 |
| *** weshay is now known as weshay|rover | 12:06 | |
| *** luksky has quit IRC | 12:17 | |
| *** izake has joined #openstack-ansible | 12:21 | |
| izake | Greetings all | 12:24 |
| izake | I am currently in the process of spinning up Openstack rocky with opesntack ansible (18.1.3) | 12:25 |
| izake | I am getting an error on keystone setup at task: "TASK [os_keystone : Add service project] " | 12:25 |
| izake | Error: Cloud default was not found | 12:26 |
| izake | (I can give the entire stack trace, if that would be more helpful?) | 12:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: metal: sync all containers to add is_metal https://review.openstack.org/648292 | 12:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: Test using integrated build https://review.openstack.org/648106 | 12:27 |
| *** markvoelker has quit IRC | 12:32 | |
| *** cshen has quit IRC | 12:35 | |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: User guide https://review.openstack.org/644570 | 12:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress master: Test using integrated build https://review.openstack.org/648108 | 12:41 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Test using integrated build https://review.openstack.org/648599 | 12:43 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Replace role tests with integrated build tests https://review.openstack.org/647862 | 12:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Test using integrated build https://review.openstack.org/647847 | 12:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Test using integrated build https://review.openstack.org/648581 | 12:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Test using integrated build https://review.openstack.org/648597 | 12:48 |
| odyssey4me | izake that means that the clouds.yaml file wasn't laid down, so it's likely there was an earlier failure which took the host out of the rest of the play | 12:49 |
| odyssey4me | if you can, take a look for any other failures earlier in that play | 12:49 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Test using integrated build https://review.openstack.org/648580 | 12:49 |
| izake | Yeah I gather as much as per the `os_project` ansible module documentation | 12:50 |
| izake | if I recall now there wasn't any specific failures before hand, but will dig a bit further | 12:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Test using integrated build https://review.openstack.org/647843 | 12:53 |
| *** luksky has joined #openstack-ansible | 12:54 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: Test using integrated build https://review.openstack.org/648584 | 12:55 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Test using integrated build https://review.openstack.org/647848 | 12:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Test using integrated build https://review.openstack.org/647844 | 12:59 |
| openstackgerrit | Merged openstack/openstack-ansible master: metal: sync all containers to add is_metal https://review.openstack.org/648292 | 13:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral master: Test using integrated build https://review.openstack.org/648600 | 13:04 |
| *** cshen has joined #openstack-ansible | 13:04 | |
| *** gillesMo has quit IRC | 13:09 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: Update role for new source build process https://review.openstack.org/648380 | 13:11 |
| *** dxiri has joined #openstack-ansible | 13:17 | |
| *** dxiri has quit IRC | 13:18 | |
| mnaser | good morning! | 13:22 |
| mnaser | woo so many merges | 13:23 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Test using integrated build https://review.openstack.org/648512 | 13:26 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/639533 | 13:26 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: missing document start "---" https://review.openstack.org/614107 | 13:27 |
| guilhermesp | mnaser morning! | 13:27 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Replace usage of netloc filters to urlsplit https://review.openstack.org/648190 | 13:27 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ironic master: Update role for new source build process https://review.openstack.org/648463 | 13:27 |
| mnaser | evrardjp: is swift3 no longer a thing? | 13:28 |
| evrardjp | it's part of swift now | 13:28 |
| *** d3n14l has joined #openstack-ansible | 13:28 | |
| evrardjp | can be done in a later commit. I am just afraid of old constraints staying there and breaking the build though. | 13:29 |
| odyssey4me | mnaser evrardjp ah yeah | 13:29 |
| mnaser | evrardjp: I will push a follow up patch on that to clean it up not to have to run ci again and rebase things | 13:30 |
| *** CobHead has quit IRC | 13:31 | |
| evrardjp | mnaser: great | 13:32 |
| mnaser | evrardjp, odyssey4me: does it not exist anymore at all per say or it is still called swift3 but lives inside swift repo? | 13:33 |
| mnaser | just wondering if I should rip out the swift3 middleware or not | 13:33 |
| evrardjp | I didn't dig into the details. I think middleware exists but is now part of swift core code | 13:33 |
| evrardjp | so depends on what you mean by "rip out the swift3 middleware" | 13:33 |
| *** CobHead has joined #openstack-ansible | 13:33 | |
| mnaser | evrardjp: as in just remove it from constraints, or also remove the settings related to it (and the swift configs for swift3) | 13:34 |
| evrardjp | I would remove it from constraints for now, and if it's broken, someone should fix it | 13:34 |
| evrardjp | https://github.com/openstack/swift/tree/master/swift/common/middleware/s3api | 13:34 |
| evrardjp | maybe it's not named swift3 anymore | 13:35 |
| mnaser | https://github.com/openstack/swift3/commit/90db5d1510b2a770387961e7bf0fbeae8101ba45 | 13:35 |
| mnaser | yeah it seems like the concept of swift3 is just non-existant | 13:35 |
| mnaser | so someone has to implement s3api eventually | 13:35 |
| mnaser | ok | 13:35 |
| evrardjp | don't rip it out | 13:36 |
| evrardjp | it seems it's close to what was done before | 13:36 |
| evrardjp | maybe a sed would do | 13:36 |
| * mnaser has no time for anything but ripping it out :P | 13:37 | |
| mnaser | (for now) | 13:37 |
| *** priteau has quit IRC | 13:37 | |
| evrardjp | if we don't have tests, it makes sense to rip it out then. | 13:37 |
| mnaser | cloudnull: just replied saying they're okay with EOLing stable/ocata, at the time it was a rax interest but I don't know if it still is.. | 13:37 |
| evrardjp | we can't release on that dead branch | 13:38 |
| evrardjp | can we? | 13:38 |
| mnaser | well I was thinking just tag ocata-eol and remove the branch | 13:38 |
| cloudnull | nope we're focused on queens / rocky. drop away good sir. | 13:38 |
| evrardjp | i thought EM meant we had to do a last tag, then we could do extra patches but not delete it | 13:38 |
| evrardjp | mnaser: not sure we can | 13:38 |
| mnaser | well the problem is any automated changes are hitting stable/ocata | 13:39 |
| evrardjp | and? | 13:39 |
| evrardjp | EM means no warranty whatsoever | 13:39 |
| evrardjp | IIRC | 13:39 |
| evrardjp | it's kinda a silly move, but let's not go back there today | 13:39 |
| evrardjp | as a deploy project, we are in a different place though, so I would be fine with killing the branch. | 13:40 |
| *** d3n14l has quit IRC | 13:40 | |
| cloudnull | personally (no corp motivated) I would love to see us keep old branches around | 13:41 |
| evrardjp | should we do this when the next branch is created, so that we don't overfill releases with new work? | 13:41 |
| cloudnull | just remove all teh gates around them when they're EOL | 13:41 |
| cloudnull | IDK if thats possible though | 13:41 |
| evrardjp | cloudnull: the intent was to keep the gates "as long as you could" | 13:41 |
| evrardjp | iirc | 13:41 |
| evrardjp | "All bugfixes (that meet the criteria described below) are appropriate. No Releases produced, reduced CI commitment." | 13:41 |
| evrardjp | https://docs.openstack.org/project-team-guide/stable-branches.html#extended-maintenance | 13:42 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_swift master: swift3: remove support https://review.openstack.org/648689 | 13:42 |
| evrardjp | cloudnull: I proposed initially to get rid of that old branch, and I regret not having done so | 13:43 |
| evrardjp | I understand the appeal of keeping the branch open though | 13:43 |
| cloudnull | that would be my personal preference | 13:44 |
| evrardjp | it sends a wrong message | 13:44 |
| evrardjp | imo | 13:44 |
| cloudnull | however I 100% understand the issues | 13:44 |
| cloudnull | that keeping it around will / can cause | 13:44 |
| mnaser | yeah I mean we can just be much more aggressive on backports | 13:44 |
| evrardjp | anyway, the branch is open now :) | 13:44 |
| mnaser | like those automated patches.. we can just abandon them, not a big deal | 13:44 |
| *** hamzaachi has joined #openstack-ansible | 13:44 | |
| mnaser | and the back ported ones, we can ask the person to make a review to remote all jobs as a dependency | 13:45 |
| evrardjp | we can have noop ci too | 13:45 |
| mnaser | if they really want to land their code | 13:45 |
| evrardjp | to fix the jobs would be better :p | 13:45 |
| evrardjp | hahaha | 13:45 |
| mnaser | in some ideal world with more than 24 hours a day :P | 13:46 |
| mnaser | evrardjp: mind upgrading your +1 to +2 https://review.openstack.org/#/c/648465/4 and maybe another core to push it through to the gate? | 13:47 |
| evrardjp | mnaser: totally | 13:47 |
| evrardjp | done | 13:48 |
| cloudnull | done | 13:51 |
| mnaser | also, appreciate more reviews on https://review.openstack.org/#/c/648477/11 -- it will help speed the gates up a bit again by avoiding running a repo_build *then* building inside each gate | 13:56 |
| mnaser | I think we need to bump requirements in the integrated repo inside openstack_services | 14:04 |
| mnaser | the croniter issue was fixed but we are pinned in integrated tests so its failing | 14:04 |
| *** joabdearaujo has joined #openstack-ansible | 14:05 | |
| *** cmart has joined #openstack-ansible | 14:07 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: requriements: bump to tip of stable/stein https://review.openstack.org/648697 | 14:11 |
| mnaser | evrardjp, odyssey4me: ^ id like your opinions on this. https://review.openstack.org/#/c/647851/ is broken and would be fixed by that | 14:11 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Clean up testing https://review.openstack.org/648183 | 14:13 |
| *** Pandry has quit IRC | 14:14 | |
| evrardjp | modifying the comments don't break the tooling anymore mnaser | 14:18 |
| evrardjp | croniter just bit me a few days ago indeed | 14:18 |
| evrardjp | I am fine with this | 14:18 |
| evrardjp | the question is | 14:19 |
| evrardjp | do you want us to branch now? | 14:19 |
| evrardjp | :) | 14:19 |
| odyssey4me | not just yet, almost :) | 14:19 |
| odyssey4me | we're trying to get all these changes in before hand, otherwise it'll be backport hell | 14:20 |
| evrardjp | yeah ok | 14:20 |
| evrardjp | so we keep tracking master. | 14:20 |
| evrardjp | else we can already track stable/stein | 14:20 |
| evrardjp | and update master later to track master | 14:20 |
| odyssey4me | well, I expect that once we branch you'll run your bump scripts again anyway before we release | 14:20 |
| odyssey4me | so merging that requirements change should be fine | 14:21 |
| odyssey4me | assuming it works | 14:21 |
| evrardjp | yeah, but they would still need updating | 14:21 |
| mnaser | yeah, I don't want us to do that just yet, just landing few last things | 14:21 |
| mnaser | so for now just need to get this bump so we can land some things of the telemetry services integrated jobs to land | 14:21 |
| evrardjp | I mean either we update things now to point to stein now and update master later, or we update later to point to stein, and keep master as track branch in master | 14:22 |
| mnaser | oh I see what you mean | 14:22 |
| evrardjp | I would be more confortable if I did a patch to track stein nowadays | 14:22 |
| evrardjp | the line just below | 14:22 |
| mnaser | evrardjp: for now though I worry that MAYBE if we do that, some magical unicorn came and broke a lot of things | 14:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Test using integrated build https://review.openstack.org/647842 | 14:22 |
| evrardjp | mnaser: the longer we wait, the more we'll pull from a wrong branch :) | 14:23 |
| mnaser | evrardjp: right, but I'm planning to cut things off by today so I don't think we have that much more potential waiting around | 14:23 |
| evrardjp | I can stop my bump horde though | 14:23 |
| mnaser | my goal was to get all this done before Saturday / eod today | 14:23 |
| evrardjp | great | 14:23 |
| mnaser | so I figured the delta is probably not much | 14:24 |
| odyssey4me | hmm, I see what you mean evrardjp - yep, perhaps we should switch the track branch to stable/stein on master until we branch | 14:24 |
| evrardjp | odyssey4me: that's what I would do today | 14:24 |
| evrardjp | if everybody is okay | 14:24 |
| evrardjp | we can keep that change | 14:24 |
| mnaser | btw, the REALLY neat thing from the work that we just introduced is | 14:24 |
| mnaser | now the integrated repo tests a common set of validated services always instead of master and then different commit in integrated | 14:24 |
| evrardjp | we won't need this bumping , we'll need another one :) | 14:24 |
| evrardjp | oh yeah another thing | 14:25 |
| evrardjp | we did that in the past mnaser | 14:25 |
| evrardjp | developers complained :p | 14:25 |
| mnaser | boo :P | 14:25 |
| evrardjp | but now we are a different set of devs :) | 14:25 |
| mnaser | evrardjp: ok I think we are probably safe if you made a mass bump tracking stable/stein in master right now | 14:25 |
| cloudnull | now with different complaints :D | 14:25 |
| mnaser | and we can land that.. then just push up a patch afterwards to master resetting tracking to latest master with a procedural -2 till we branch | 14:26 |
| evrardjp | looking for more stable not features | 14:26 |
| mnaser | that sounds much cleaner, right? | 14:26 |
| evrardjp | mnaser: totally | 14:26 |
| mnaser | I can abandon my change if you can push that up (or if you're busy, feel free to point me to it) | 14:26 |
| evrardjp | nope, keep that one in | 14:26 |
| evrardjp | it's fine | 14:26 |
| mnaser | Nono lets just have 1 big bump patch | 14:27 |
| evrardjp | things don't break anymore when you change unexpected comments :p | 14:27 |
| cloudnull | odyssey4me have you run into https://pasted.tech/pastes/4df73ce14cd4bcd259d48036c32c4e7fc1fd2718.raw ? | 14:27 |
| evrardjp | that's the power of using a yaml lib :p | 14:27 |
| mnaser | evrardjp: lets just make one bump patch rather than 2, our ci is already slow as is -_- | 14:27 |
| evrardjp | wait, am I using it now? Can't remember. | 14:27 |
| odyssey4me | cloudnull I have - that's due to missing facts that should have been gathered. | 14:28 |
| evrardjp | mnaser: ok let me do a patch then | 14:28 |
| mnaser | evrardjp: merci | 14:29 |
| cloudnull | hum. maybe we should have that run setup in the venv build role for the "venv_build_host" ? | 14:29 |
| cloudnull | if you pull new roles and run os-*.yml it dies without those facts, if the build target is not in the scoped hosts, | 14:30 |
| odyssey4me | cloudnull if we could make it gather the minimal facts required so that it doesn't waste too much time, then perhaps... | 14:31 |
| odyssey4me | given that it is a runtime dependency, it seems appropriate | 14:31 |
| cloudnull | ++ | 14:31 |
| * cloudnull goes to see what can be done there. | 14:31 | |
| odyssey4me | you gonna push a patch? | 14:31 |
| *** shyamb has joined #openstack-ansible | 14:31 | |
| odyssey4me | roger that | 14:32 |
| mnaser | odyssey4me: I've got a broken VM with the pip stuff from infra | 14:33 |
| mnaser | I'll try to get an infra patch to fix it | 14:33 |
| odyssey4me | mnaser tyvm - if you need a hand with that let me know | 14:33 |
| mnaser | seems like ill have to do.. rewrite rules *gag* | 14:33 |
| odyssey4me | oh, good times - I'll leave you to it then | 14:34 |
| * odyssey4me runs away | 14:34 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add panko to integrated build https://review.openstack.org/647898 | 14:35 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add telemetry service bundle to AIO tests https://review.openstack.org/647921 | 14:35 |
| *** shyamb has quit IRC | 14:38 | |
| noonedeadpunk | ^ just rebase, hope won't broke anything | 14:38 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Add the manila service to OSA https://review.openstack.org/645653 | 14:42 |
| odyssey4me | ^ rebased on top of the panko patch to resolve conflicts | 14:42 |
| evrardjp | ok so | 14:46 |
| evrardjp | I have a series of questions already for the branches to track | 14:46 |
| evrardjp | it seems we went too far for gnocchi | 14:47 |
| evrardjp | should I move it to stable/4.3 vs master, and rollback time? | 14:47 |
| evrardjp | that's what I would do | 14:47 |
| evrardjp | let me propose a patch, and we can discuss what to do. | 14:48 |
| odyssey4me | evrardjp yeah, I think so - hopefully noone is running it with stein just yet | 14:48 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/ansible-role-python_venv_build master: Add fact gathing for hardware when building venvs https://review.openstack.org/648708 | 14:48 |
| odyssey4me | cloudnull That makes sense. I take it that you tested it? | 14:51 |
| mnaser | GAH | 14:53 |
| mnaser | #openstack-infra for fun | 14:53 |
| *** cmart has quit IRC | 14:54 | |
| *** CobHead has quit IRC | 14:55 | |
| cloudnull | yup running now | 14:55 |
| cloudnull | odyssey4me next question: `/var/www/repo/os-releases/18.0.0.0rc2.dev448/keystone-18.0.0.0rc2.dev448-constraints.txt` should that be `/var/www/repo/os-releases/18.0.0.0rc2.dev448/keystone-18.0.0.0rc2.dev448-source-constraints.txt` ? | 14:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: Test using integrated build https://review.openstack.org/648512 | 14:56 |
| cloudnull | https://pasted.tech/pastes/2019417e588559c8c0f524b770a7a4e8a253a07f.raw ? | 14:56 |
| *** CobHead has joined #openstack-ansible | 14:57 | |
| cloudnull | - /etc/ansible/roles/python_venv_build/tasks/python_venv_wheel_build.yml:157 | 14:57 |
| cloudnull | https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_wheel_build.yml#L119 | 14:59 |
| cloudnull | vs https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_wheel_build.yml#L148 | 14:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: missing document start "---" https://review.openstack.org/614107 | 14:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/639533 | 14:59 |
| cloudnull | or maybe that lower block just needs to be delegated ? | 15:02 |
| *** dxiri has joined #openstack-ansible | 15:03 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: Update role for new source build process https://review.openstack.org/648463 | 15:06 |
| *** shyamb has joined #openstack-ansible | 15:06 | |
| evrardjp | mnaser: I am removing swift3 from the integrated too then | 15:07 |
| *** izake has quit IRC | 15:07 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: Update role for new source build process https://review.openstack.org/648463 | 15:07 |
| odyssey4me | cloudnull the -source-constraints has the git URL reference, and is only used for the wheel build, the -constraints.txt has the resulting package name and version in it, rather than the git source | 15:09 |
| *** gillesMo has joined #openstack-ansible | 15:09 | |
| odyssey4me | think of -constraints as the old requirements_absolute_requirements.txt file, but in this case only for the one venv | 15:09 |
| *** cshen has quit IRC | 15:11 | |
| odyssey4me | the last task then slurps up the constraints file from the build host so that we can write it into the install host venv and use it when installing | 15:12 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Track Stein when we can https://review.openstack.org/648710 | 15:13 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Track master branch for Train cycle https://review.openstack.org/648711 | 15:13 |
| cloudnull | odyssey4me looks like that lower block just needs to be delegated to the build node | 15:14 |
| odyssey4me | cloudnull it is | 15:14 |
| odyssey4me | cloudnull the bit above that task is a block within a block | 15:14 |
| cloudnull | oh. hum . | 15:15 |
| cloudnull | maybe i was just caught in a bad state. | 15:15 |
| evrardjp | odyssey4me: look at that wonderful branchname. I had to: https://review.openstack.org/#/c/648711/1 | 15:15 |
| * cloudnull goes to wipe all that out and try again | 15:15 | |
| *** luksky has quit IRC | 15:16 | |
| evrardjp | mnaser: https://review.openstack.org/#/c/648710/1 | 15:16 |
| * cloudnull is applying all this to my home cluster now | 15:16 | |
| odyssey4me | evrardjp lol | 15:16 |
| odyssey4me | choochoo! | 15:17 |
| evrardjp | odyssey4me: could you vote on https://review.openstack.org/#/c/648710/1 ? | 15:18 |
| evrardjp | odyssey4me: :) | 15:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Update role for new source build process https://review.openstack.org/648398 | 15:18 |
| cloudnull | mnaser odyssey4me https://review.openstack.org/#/c/648477 - works for me :) nice work on all that btw | 15:19 |
| cloudnull | now through cinder . | 15:19 |
| evrardjp | cloudnull: I guess I can upgrade my vote then | 15:19 |
| cloudnull | on the cinder w/ stien front, did we ever figure out why online migrations are failing ? | 15:20 |
| mnaser | cloudnull: yays | 15:20 |
| evrardjp | if it's already +w I won't have much more time to think about what else to remove :) | 15:20 |
| mnaser | cloudnull: I think that was just a one time thing afaik | 15:20 |
| cloudnull | https://pasted.tech/pastes/beb1a5bb2754303c38a267d5a5b6063ff899f54f.raw | 15:21 |
| cloudnull | I still get that ' | 15:21 |
| evrardjp | odyssey4me: I want to say... amazing job on that thing. It's been a while we have that problem, and now we are kinda free. | 15:21 |
| * cloudnull getting a better formatted output | 15:22 | |
| noonedeadpunk | yep +1 really great work! | 15:23 |
| *** shananigans has joined #openstack-ansible | 15:24 | |
| cloudnull | https://pasted.tech/pastes/fe946329d193d1ad32099246aac83085887bdb22 | 15:24 |
| cloudnull | ^ thats what os-cinder is running but failing | 15:25 |
| cloudnull | logs have nothing in them. | 15:25 |
| mnaser | cloudnull: I think this is a by-product of the systemd logging that we use in cinder | 15:25 |
| mnaser | can you disable them and run it again and see as it logs on disk | 15:25 |
| cloudnull | Oh I mean I have logs just nothing related to an online database migration | 15:26 |
| cloudnull | unless we think that their would be other logs ? | 15:26 |
| * cloudnull trying to run the same with a static log file now w/ cinder-manage | 15:27 | |
| cloudnull | https://pasted.tech/pastes/282e32435b7ee44ad53e865d0b65d67da32a1ab3 - interesting | 15:30 |
| cloudnull | so it looks like cinder-manage has its own log things. | 15:30 |
| *** chandankumar is now known as raukadah | 15:30 | |
| cloudnull | also wierd that this is an rpc timeout given cinder is working fine. | 15:30 |
| cloudnull | likely something off in my env | 15:30 |
| evrardjp | cloudnull: you should seriously link to the github page of pasted | 15:30 |
| evrardjp | I can't do my PR to say what's wrong | 15:30 |
| cloudnull | as a gist ? | 15:31 |
| mnaser | cloudnull: cinder-manage does log things yeah | 15:31 |
| * cloudnull goes to add a comment system to pasted .... | 15:31 | |
| evrardjp | nope I want to see code | 15:31 |
| evrardjp | :p | 15:31 |
| mnaser | cloudnull: are we running online migrations before we start up cinder? | 15:31 |
| mnaser | looks like maybe cinder-vol or cinder-sch aren't up and it's timing out | 15:31 |
| mnaser | it is.. online.. data.. migrations after all :P | 15:31 |
| noonedeadpunk | oh, there were some problem with cinder online migrations for a while | 15:32 |
| evrardjp | cloudnull: I had to go through pypi | 15:32 |
| cloudnull | evrardjp pasted is on my github you can send PRs if you want :) | 15:32 |
| evrardjp | to find links | 15:32 |
| evrardjp | yeah not on pasted.tech | 15:32 |
| evrardjp | bad UX! | 15:32 |
| noonedeadpunk | like https://bugs.launchpad.net/cinder/+bug/1806156 | 15:32 |
| openstack | Launchpad bug 1806156 in Cinder "shared_targets_online_data_migration fails when cinder-volume service not running" [Undecided,Confirmed] | 15:32 |
| evrardjp | :) | 15:32 |
| mnaser | I mean it is _online_ data migrations :P | 15:32 |
| *** kopecmartin is now known as kopecmartin|off | 15:32 | |
| cloudnull | evrardjp oh you want to see the project URLs on pasted.tech ? | 15:32 |
| mnaser | perhaps we need to change the order of playbooks | 15:33 |
| * cloudnull done | 15:33 | |
| cloudnull | mnaser cinder is running in my env , so this would be an upgrade of sorts | 15:33 |
| cloudnull | early stein to later stein | 15:33 |
| evrardjp | cloudnull: yet pointing to https://github.com/cloudnull/pasted and https://github.com/cloudnull/pasted-client | 15:34 |
| mnaser | cloudnull: could the service be down by any chance? | 15:34 |
| cloudnull | nope, its all running and happy | 15:34 |
| cloudnull | evrardjp - https://pasted.tech/info/cli_client but I'll add a link to the main project too | 15:36 |
| mbuil | has anyone seen an error when cinder runs the task: "Perform online data migrations"? I get a "Timed out waiting for a reply to message ID 0873d4e3b24146c9ae334144c0694f49" coming from the amqp driver in the logs | 15:36 |
| mnaser | LOL | 15:41 |
| mnaser | mbuil: scroll LITERALLY a few lines above | 15:42 |
| mnaser | :p | 15:42 |
| cloudnull | evrardjp done. https://pasted.tech/ added to the FAQ :) | 15:45 |
| mbuil | mnaser: haha ok :) | 15:46 |
| mbuil | mnaser: I should search before writing but I was really thinking that the chances to have somebody hitting the same problem were extremely low | 15:47 |
| mbuil | mnaser: I am seeing it in rocky though | 15:48 |
| noonedeadpunk | mbuil +1 | 15:48 |
| noonedeadpunk | It's since https://review.openstack.org/#/c/614617/ | 15:48 |
| mbuil | noonedeadpunk thanks for the info! | 15:50 |
| mnaser | odyssey4me: as a side, what do we risk dealing with by adding a get-pip.py and calling it a day? :\ | 15:51 |
| mnaser | and then we can stop using distro virtualenv and not worry about a non common platform | 15:52 |
| odyssey4me | mnaser that would mean that we dirty the host with pip again, which is took considerable effort to move away from | 15:56 |
| mnaser | odyssey4me: yeah im not a fan | 15:56 |
| mnaser | im trying to think what we can do as a gate-only hack | 15:57 |
| mnaser | like how to disable that behavior for one run only | 15:57 |
| odyssey4me | we'd have to restore using get-pip in bootstrap-ansible, and would have to go back to using the pip_install role on the hosts | 15:57 |
| mnaser | yeah no, we just need to get around it for the pip upgrade somehow | 15:57 |
| odyssey4me | mnaser lemme look at options - perhaps we can do something to isolate just that part, although we're now implementing a workaround for CI into something that works fine in production | 15:58 |
| mnaser | odyssey4me: yeah, that's where im a bit torn | 15:58 |
| mnaser | cause it seems like it would be a role level workaround | 15:58 |
| mnaser | where it should live in our CI stuff | 15:58 |
| mnaser | so trying to see maybe we can provide an extra option to disable it | 15:58 |
| evrardjp | what's the pain here? | 15:58 |
| odyssey4me | evrardjp the extra index isn't handled properly by the version of pip provided by python-virtualenv on ubuntu 18 | 15:59 |
| evrardjp | oh shiiit | 15:59 |
| odyssey4me | it seems that there may be a local patch in the package which breaks it | 15:59 |
| evrardjp | you can use a different name | 15:59 |
| evrardjp | that sounds like a nice bug | 16:00 |
| evrardjp | can I see? :D | 16:00 |
| odyssey4me | reported in 2016: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837764 | 16:00 |
| openstack | Debian bug 837764 in src:python-pip "python-pip: Using `--extra-index-url` results in `HTTPError: 404 Client Error: NOT FOUND`" [Important,Open] | 16:00 |
| evrardjp | but surely the one in python-virtualenv should not have that, right? | 16:01 |
| odyssey4me | this is what we're seeing: http://paste.openstack.org/show/748588/ | 16:01 |
| *** nurdie has joined #openstack-ansible | 16:01 | |
| evrardjp | so it adds '/pip' basically ? | 16:02 |
| nurdie | https://docs.openstack.org/glance/pike/admin/rollingupgrades.html says that rolling upgrades aren't supported | 16:02 |
| nurdie | Does anyone have experience with OSA upgrading a controller stack, then one of several compute nodes at a time and live migrating from old compute hosts to upgraded compute hosts? | 16:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Test using integrated build https://review.openstack.org/648506 | 16:03 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Update role for new source build process https://review.openstack.org/648465 | 16:03 |
| evrardjp | odyssey4me: my question is it where does http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-18.04-x86_64/pip/ come from | 16:04 |
| evrardjp | with poorly written english, as usual | 16:04 |
| tosky | less important issue: two backports of "Replace openstack.org git:// URLs with https://" for stable/queens and stable/pike on os_sahara failed because of the percona-xtrabackup package (old issue) | 16:05 |
| tosky | do those branches require some backport? | 16:05 |
| tosky | (see https://review.openstack.org/#/c/646803/ and https://review.openstack.org/#/c/646802/ ) | 16:05 |
| mnaser | evrardjp: infra pip.conf which adds extra-index-url | 16:06 |
| evrardjp | is that normal that link contains pip/ at the end for the wheels ? | 16:07 |
| odyssey4me | evrardjp for an index, yes | 16:07 |
| evrardjp | mmm why? | 16:07 |
| evrardjp | #learning | 16:07 |
| odyssey4me | /pip is not on that index because infra aren't building pip for ubuntu 18 | 16:07 |
| evrardjp | oh that's what you meant! | 16:08 |
| evrardjp | ok | 16:08 |
| evrardjp | can't those task run as isolated? | 16:08 |
| odyssey4me | nurdie we have a minor and major upgrade guide in the docs, and it seems to work quite well for all OSA users, and it does a rolling upgrade where the services support it | 16:08 |
| mnaser | evrardjp: it looks like isolated does not ignore options in /etc/pip.conf | 16:09 |
| odyssey4me | evrardjp of course we could make them do that, but then we're putting a workaround in the role just for the sake of CI, which isn't ideal | 16:09 |
| evrardjp | where do we copy the pip config file? | 16:09 |
| evrardjp | because I think isolated ignores user configuration but non /etc/pip.conf | 16:09 |
| odyssey4me | but we may have to do that - it seems that infra isn't keen to switch to using --find-links instead, which would solve the problem | 16:10 |
| evrardjp | that could be a way | 16:10 |
| mnaser | I HAVE AN IDEA | 16:10 |
| mnaser | PIP_CONFIG_FILE=/dev/null | 16:10 |
| * mnaser tries | 16:10 | |
| evrardjp | that would work too | 16:10 |
| evrardjp | oh no | 16:10 |
| mnaser | holy | 16:10 |
| mnaser | it worked | 16:10 |
| odyssey4me | lol mnaser - that's terrible | 16:10 |
| mnaser | odyssey4me: budget isolated | 16:10 |
| odyssey4me | if we want to do that, then we just don't copy the config file | 16:10 |
| mnaser | odyssey4me: well don't we want to avoid this ONLY for pip upgrade? | 16:10 |
| *** ansmith has quit IRC | 16:11 | |
| mnaser | cause once it's upgraded... ideally it shouldn't be an issue | 16:11 |
| evrardjp | mnaser: budget isolated :D | 16:11 |
| mnaser | odyssey4me: once we upgrade, /openstack/venvs/wheel-builder/bin/pip2 install -U pip is fine. | 16:11 |
| nurdie | odyssey4me: https://docs.openstack.org/openstack-ansible/pike/user/script-upgrade.html that will do rolling upgrades? o_0 | 16:11 |
| mnaser | which means if we pip2 install other things down the line, they'll be fine | 16:11 |
| odyssey4me | yeah, we can add a special var that's only used for the pip upgrade and make opensack ci use it | 16:11 |
| mnaser | like | 16:11 |
| mnaser | pip_isolated_upgrade or something | 16:12 |
| mnaser | cause --isolated doesn't do the job | 16:12 |
| odyssey4me | nurdie yessir, it will try its best - I encourage you to use a test environment and be familiar with everything it does | 16:12 |
| evrardjp | noconf or something | 16:12 |
| evrardjp | that's clearer | 16:12 |
| odyssey4me | mnaser --isolated did not work? | 16:12 |
| evrardjp | mnaser: as I said, isolated would work if it's placed in user vs /etc/ | 16:12 |
| mnaser | odyssey4me: no, --isolated doesn't override /etc/pip.conf ^ | 16:12 |
| odyssey4me | oh I see | 16:12 |
| odyssey4me | fun times | 16:12 |
| mnaser | copying to user might make it complicated | 16:13 |
| evrardjp | odyssey4me: iirc, isolated now only works for runtime env + user | 16:13 |
| mnaser | depending on what it would run blabla | 16:13 |
| evrardjp | yeah | 16:13 |
| evrardjp | I think it's fine though | 16:13 |
| evrardjp | but I can't remember where we can define pip.conf in the user interface of pip | 16:13 |
| evrardjp | so I think your hack is totally a hack | 16:13 |
| mnaser | we don't anymore :D | 16:13 |
| odyssey4me | ok mnaser - wanna push up a patch to ansible-role-python_venv_build to make a boolean flag to enable that? | 16:13 |
| mnaser | we just copy /etc/pip.conf from the host into containers now | 16:13 |
| evrardjp | nope that's not what I meant | 16:13 |
| evrardjp | I meant pip itself | 16:14 |
| mnaser | sure, anyone has ideas on what to use as a var | 16:14 |
| evrardjp | it doesn't expose -f <conf file> | 16:14 |
| mnaser | evrardjp: ooooh I see what you mean | 16:14 |
| evrardjp | or something like that | 16:14 |
| mnaser | instead of an envvar | 16:14 |
| evrardjp | env vars and cli args are quite the same | 16:14 |
| evrardjp | for pip i mean | 16:14 |
| evrardjp | but using an env var for something that's not exposed... seems weird and prone to breaking | 16:14 |
| evrardjp | let me check the docs if that's a stable interface | 16:15 |
| mnaser | any ideas on var names | 16:15 |
| mnaser | lol | 16:15 |
| evrardjp | noconf_upgrade ? | 16:15 |
| nurdie | If I upgrade a pike controller stack to queens, will my pike compute nodes communicate just fine with them while I upgrade them one at a time? | 16:15 |
| odyssey4me | venv_pip_upgrade_noconf ? | 16:15 |
| evrardjp | something like that | 16:16 |
| evrardjp | when true, uses devnull | 16:16 |
| odyssey4me | nurdie yep, the playbooks will do exactly that for you - it upgrades things in the right order and serialises the upgrade appropriately | 16:16 |
| nurdie | odyssey4me: Thank you! | 16:17 |
| mnaser | hmm | 16:17 |
| *** shyamb has quit IRC | 16:17 | |
| mnaser | unless I duplicate the task | 16:17 |
| mnaser | id have to hardcode /etc/pip.conf | 16:17 |
| mnaser | I don't think omit works in environment: {} | 16:18 |
| odyssey4me | mnaser not really - just do something like {{ (venv_pip_upgrade_noconf | bool) | ternary('-c /dev/null', '') | 16:18 |
| mnaser | I wonder if PIP_CONFIG_FILE='' pip .. would actually mean it never uses it | 16:18 |
| mnaser | ok I just tested it | 16:19 |
| mnaser | ='' will ignore it | 16:19 |
| odyssey4me | well, I was thinking of what evrardjp said - use the CLI arg, not the env var, to tell pip to use /dev/null as a config file | 16:19 |
| mnaser | it won't actually go for "nothing" | 16:19 |
| mnaser | ok fair let me see | 16:19 |
| evrardjp | or we go for a pip config list ; pip config unset; pip config set | 16:19 |
| mnaser | it doesn't exist | 16:21 |
| mnaser | I have the code here in front of me | 16:21 |
| evrardjp | ok | 16:21 |
| mnaser | https://www.irccloud.com/pastebin/ZRpTQ6b8/ | 16:21 |
| evrardjp | well PIP_CONFIG_FILE is not documented, so I am afraid of a fragile thing, but it seems to be there :) | 16:22 |
| evrardjp | so why not | 16:22 |
| mnaser | only the gate will break if this does change | 16:22 |
| mnaser | and we know it exists in the crappy version of pip that we need to upgrade | 16:23 |
| odyssey4me | this assumes that the ansible pip module is able to use an environment argument | 16:23 |
| evrardjp | OH WAIT | 16:23 |
| evrardjp | it is documented | 16:23 |
| evrardjp | go ahead! | 16:23 |
| evrardjp | :D | 16:23 |
| *** tosky has quit IRC | 16:23 | |
| evrardjp | "You can set a custom path location for this config file using the environment variable PIP_CONFIG_FILE" | 16:23 |
| evrardjp | lgtm shipit | 16:23 |
| odyssey4me | there is another way to do this which may be a little less dirty | 16:24 |
| evrardjp | odyssey4me: it should be working fine with environment | 16:24 |
| evrardjp | that would be a non module problem | 16:24 |
| evrardjp | but still an ansible problem :) | 16:24 |
| odyssey4me | currently there are two vars - venv_pip_build_args and venv_pip_install_args, we could add another one for build & install which is specific to the pip upgrade part, not the wheel build/pip install part | 16:25 |
| evrardjp | odyssey4me: I think it's quite clean to use PIP_CONFIG_FILE or to run commands of pip config | 16:25 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-python_venv_build master: Add option to upgrade PyPI/etc without using site config https://review.openstack.org/648724 | 16:25 |
| odyssey4me | mnaser so if PIP_CONFIG_FILE='' then it uses the site-wide conf file? | 16:26 |
| mnaser | odyssey4me: yes, I confirmed | 16:27 |
| mnaser | https://www.irccloud.com/pastebin/fHbQfpQJ/ | 16:27 |
| mnaser | https://www.irccloud.com/pastebin/Ae9nJ4rg/ | 16:27 |
| odyssey4me | brilliant, I think this is reasonably clean then | 16:28 |
| evrardjp | I think too | 16:28 |
| odyssey4me | mnaser so now we set that lag as part of https://review.openstack.org/648566 and make it depend on that patch, and happy days | 16:29 |
| odyssey4me | *flag | 16:29 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: gate: cache /etc/pip.conf inside containers https://review.openstack.org/648566 | 16:29 |
| evrardjp | removing my vote, add the flag, and will re-add my vote | 16:29 |
| mnaser | evrardjp: bring it back :> | 16:29 |
| evrardjp | done already | 16:30 |
| evrardjp | I was checking the depends on, don't be such in a hurry! | 16:30 |
| odyssey4me | orsm | 16:30 |
| mnaser | :O | 16:30 |
| mnaser | did I just see mhayden +1 a patch | 16:30 |
| evrardjp | yes | 16:30 |
| odyssey4me | wuuuut | 16:30 |
| evrardjp | hahaha :) | 16:30 |
| mnaser | odyssey4me: https://review.openstack.org/#/c/648710/ | 16:30 |
| mnaser | also would be nice to get your +w because its pretty major | 16:31 |
| evrardjp | LOL | 16:31 |
| mnaser | evrardjp: https://review.openstack.org/#/c/648512/ ready for +W | 16:31 |
| evrardjp | let's get that bad boy in | 16:32 |
| * mhayden is a helper | 16:32 | |
| mnaser | mhayden: :D | 16:33 |
| mhayden | mnaser: also evrardjp promised me a Stella Artois for a +1 | 16:33 |
| * mhayden pokes d34dh0r53 ^^ | 16:34 | |
| mnaser | does that mean you'll be in denver? ;) | 16:34 |
| odyssey4me | evrardjp does the tooling you're using check is stable/stein exists in all those repositories? wrt https://review.openstack.org/#/c/648710/ | 16:34 |
| mnaser | odyssey4me: I assumed it did cause masakari is still tracking master =P | 16:34 |
| evrardjp | yes | 16:34 |
| evrardjp | odyssey4me: ^ | 16:34 |
| mnaser | or masakari-dashboard | 16:34 |
| mnaser | cause looks like they haven't branched | 16:34 |
| mhayden | mnaser: i wish i could be there | 16:34 |
| odyssey4me | evrardjp nice, ok awesome | 16:34 |
| evrardjp | well, it throws an exception if the thing is not what it should be | 16:35 |
| mnaser | mhayden: its OPEN INFRA after all | 16:35 |
| mnaser | you're not going to an openstack event :-P | 16:35 |
| nurdie | Will setup-hosts.yml interrupt any instances running on compute nodes? | 16:35 |
| evrardjp | odyssey4me: so I just had to edit for those who don't work | 16:35 |
| nurdie | Doesn't look like it | 16:35 |
| evrardjp | explicit is better than implicit | 16:35 |
| odyssey4me | nurdie it should not | 16:35 |
| evrardjp | mhayden: I sure will bring a stella next time I see you. | 16:35 |
| nurdie | odyssey4me: Thanks you da person! | 16:36 |
| evrardjp | like these mega cans of 1L | 16:36 |
| odyssey4me | hahaha, a mini keg of stella | 16:36 |
| evrardjp | that's the 3L one | 16:36 |
| odyssey4me | man, that's enough to make anyone go blind | 16:36 |
| evrardjp | oops -- | 16:36 |
| evrardjp | I don't know this | 16:36 |
| evrardjp | never heard of mini kegs of stella | 16:36 |
| evrardjp | odyssey4me: there are russians in this channel, pretty sure more alcohol can you make you blind there than in a keg of stella | 16:37 |
| noonedeadpunk | +1 | 16:38 |
| evrardjp | pretty sure stella has more negative effects though. | 16:38 |
| noonedeadpunk | :D | 16:38 |
| odyssey4me | evrardjp http://www.beerminikeg.com | 16:38 |
| mhayden | well darn, y'all were being all productive and now i ruined with beer talk | 16:38 |
| evrardjp | that's productive | 16:38 |
| noonedeadpunk | but at least now I know whom I mask ask about nice bar with stella in Belgium | 16:38 |
| evrardjp | :D | 16:38 |
| evrardjp | odyssey4me: Yes there are the ones I am talking about | 16:39 |
| evrardjp | 3L and 5L | 16:39 |
| evrardjp | "perfect draft" | 16:39 |
| mhayden | hey, it's almost 5PM GMT anyway | 16:39 |
| noonedeadpunk | s/mask/may/ | 16:39 |
| evrardjp | bollocks! | 16:39 |
| evrardjp | it's already almost 6 where I am | 16:39 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_ceilometer master: Test using integrated build https://review.openstack.org/648513 | 16:39 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_panko master: Test using integrated build https://review.openstack.org/648514 | 16:39 |
| admin0 | hi all .. which playbook to have logs go to a new logs host | 16:39 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_gnocchi master: Test using integrated build https://review.openstack.org/648515 | 16:39 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_aodh master: Test using integrated build https://review.openstack.org/647851 | 16:40 |
| evrardjp | odyssey4me: https://www.beerhawk.co.uk/perfectdraft-leffe-spring-6l-keg-2019 | 16:40 |
| noonedeadpunk | admin0 rsyslog_client? | 16:40 |
| evrardjp | odyssey4me: with multi buy offer | 16:40 |
| evrardjp | click click! | 16:40 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible-rsyslog_client/latest/ | 16:40 |
| admin0 | noonedeadpunk, there is no such playbook :D | 16:40 |
| admin0 | at least in the base | 16:40 |
| odyssey4me | admin0 depends which release, but I think there's a journal playbook thingy somewhere | 16:40 |
| admin0 | 18.1.5 | 16:41 |
| mnaser | I think this is the last step for us to get telemetry stuff to merge ^ | 16:41 |
| admin0 | i am trying to keep these clouds updated | 16:41 |
| d34dh0r53 | I only +1 for La Moneuse | 16:41 |
| noonedeadpunk | This role is included during other roles execution | 16:41 |
| mnaser | so now the _only_ broken thing we have is | 16:41 |
| mnaser | https://review.openstack.org/#/c/648231/ | 16:41 |
| mnaser | we need to land the integrated role change | 16:41 |
| admin0 | i setup a new node as logs .. want to try using magnanimous-turbo-chainsaw .. so that it can do ELK and some graphs | 16:41 |
| odyssey4me | admin0 this one I think https://github.com/openstack/openstack-ansible/blob/stable/rocky/playbooks/infra-journal-remote.yml | 16:41 |
| mnaser | and then we can land that and be done with python_venv_build | 16:41 |
| noonedeadpunk | so you may just configure staff and run setup-openstack | 16:42 |
| evrardjp | OMG those perfect draft kegs, on top of being bad, they are over priced | 16:42 |
| nurdie | odyssey4me: Do you feel that a safe/fair bet for run-upgrade.sh is just updating setup-openstack.yml with --limit '!compute_hosts' to allow me to upgrade them one at a time while live migrating in between compute upgrades? | 16:42 |
| mnaser | {"msg": "'dict object' has no attribute u'lbaas'"} | 16:42 |
| evrardjp | noonedeadpunk: I said I would the release this week, it's still the week btw | 16:43 |
| admin0 | odyssey4me, thanks | 16:43 |
| nurdie | odyssey4me: That looks safe to me. Everything besides setup-openstack.yml is mostly controller changes | 16:43 |
| mnaser | I've been iterating here -- https://review.openstack.org/#/c/648509/6 | 16:43 |
| evrardjp | I am not too late! | 16:43 |
| odyssey4me | nurdie the openstack services being upgraded on each compute host won't impact the instances on the host | 16:43 |
| nurdie | o_0 | 16:43 |
| nurdie | Wat | 16:43 |
| nurdie | odyssey4me: So, run-upgrade.sh, in theory, should not interrupt a single running instance? | 16:43 |
| mnaser | I think octavia_container_network_name is kinda broken in there | 16:43 |
| evrardjp | it's not kvm you're upgrading | 16:43 |
| odyssey4me | nurdie but hey, if you'd prefer to then yeah - use the limit | 16:44 |
| nurdie | Ohhhhhh I get it! | 16:44 |
| evrardjp | the network control plane might be impacted though | 16:44 |
| noonedeadpunk | evrardjp: You're not:) I've holded upgrade till monday anyway, so you have some time on your hands :D | 16:44 |
| mnaser | do we log our inventory somewhere? | 16:44 |
| mnaser | in our jobs | 16:44 |
| nurdie | I saw that. Especially with setup-hosts looks like it might touch iptables rules, etc | 16:44 |
| odyssey4me | nurdie the only interruption you may find would be if you're using L3 routers with neutron and the neutron agents are inside containers | 16:44 |
| evrardjp | mnaser: we used to at least | 16:44 |
| nurdie | odyssey4me: That is how I am set up | 16:44 |
| admin0 | nurdie, even if the openstack controllers are dead and openstack database is dead, your kvm instances will be running - from experience :) | 16:44 |
| evrardjp | check at host parts of logs | 16:45 |
| nurdie | Well I'm starting to feel a tiny less TERRIFIED about this script :) | 16:45 |
| odyssey4me | mnaser is that perhaps a missing network that bootstrap-aio is not setting up? | 16:45 |
| mnaser | odyssey4me: I do see that br-lbaas is setup though | 16:45 |
| mnaser | im just not sure if the logic of all of that makes sense | 16:46 |
| mnaser | its a lot of introspection | 16:46 |
| odyssey4me | mnaser I don't think the octavia scenario for the AIO ever worked. I don't think I've seen it work, ever. | 16:47 |
| mnaser | odyssey4me: indeed, yeah, im just trying to fix it while other stuff land | 16:47 |
| mnaser | what does container_networks look like hm | 16:47 |
| odyssey4me | So yeah, there might be tons that can be cleaned up. To figure that issue out may require building a test host and taking the template apart bit by bit to isolate the part that's breaking | 16:48 |
| * mnaser opens up ssh | 16:48 | |
| mnaser | ok so | 16:48 |
| mnaser | container_networks['lbaas_address'] was actually correct, I 'broke' that by removing it | 16:49 |
| mnaser | container_networks['lbaas_address']['address'] should indeed be right | 16:49 |
| mnaser | now why would it complain that {"msg": "'dict object' has no attribute 'address'"} | 16:50 |
| mnaser | could a container network be listed .. and not have assigned addresses? | 16:50 |
| mnaser | oh would you look at that | 16:53 |
| *** gillesMo has quit IRC | 16:53 | |
| mnaser | I wonder if it's because we're using Baremetal and that inventory has the wrong thing™ | 16:53 |
| mnaser | I guess I will really have to fire up a vm at this point | 16:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: swift3: remove support https://review.openstack.org/648689 | 16:57 |
| admin0 | the folder /var/log/log-storage not appearing in the rsyslog container | 17:00 |
| *** ansmith has joined #openstack-ansible | 17:00 | |
| *** cshen has joined #openstack-ansible | 17:05 | |
| admin0 | cloudnull, does following the steps on https://github.com/rcbops/magnanimous-turbo-chainsaw in a deploy host already used to setup 18.1.5 break the environment ? | 17:06 |
| nurdie | Thanks for answering my questions all, specially you odyssey4me | 17:06 |
| cloudnull | admin0? | 17:06 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Test using integrated build https://review.openstack.org/648595 | 17:07 |
| admin0 | trying to setup https://github.com/rcbops/magnanimous-turbo-chainsaw .. i did the first 2 steps in the deploy node .. . wanted to confirm its the way it should be before proceeding | 17:07 |
| admin0 | with curl https://raw.githubusercontent.com/rcbops/magnanimous-turbo-chainsaw/master/scripts/setup.sh | bash | 17:07 |
| admin0 | 17:07 | |
| *** spatel has joined #openstack-ansible | 17:07 | |
| *** cshen has quit IRC | 17:09 | |
| admin0 | cloudnull, never mind .. i have no idea what i am doing | 17:13 |
| cloudnull | that first step will do all the setup | 17:13 |
| cloudnull | but it shouldn't have any effect on the existing installation | 17:14 |
| cloudnull | or the tools on that host | 17:14 |
| cloudnull | the deploy scripts will used embedded ansible and a cached inventory to run the deployment of things | 17:14 |
| mnaser | https://review.openstack.org/#/c/648256/ small review | 17:15 |
| admin0 | step: source setup-workspace.sh | 17:15 |
| admin0 | .. but it does not say from which directory | 17:15 |
| *** DanyC has quit IRC | 17:17 | |
| admin0 | cloudnull, the probe is skipping all the containers saying dns name resolution error .. is that normal ? | 17:18 |
| admin0 | my deploy also does not have any hosts in its /etc/hosts | 17:19 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: rally: add empty rally.yml.aio file https://review.openstack.org/648741 | 17:20 |
| admin0 | odyssey4me, which playbook is responsible to add all container entries to /etc/hosts ? | 17:21 |
| *** SmearedBeard has quit IRC | 17:21 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_rally master: Test using integrated build https://review.openstack.org/648598 | 17:22 |
| *** cmart has joined #openstack-ansible | 17:23 | |
| mnaser | oh damnit | 17:24 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-python_venv_build master: Add option to upgrade PyPI/etc without using site config https://review.openstack.org/648724 | 17:25 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Test using integrated build https://review.openstack.org/647842 | 17:25 |
| mnaser | odyssey4me, evrardjp ^ I messed up with a missing `}}` | 17:26 |
| admin0 | can anyone let me know how/which playbook adds the contaiers and ips to /etc/hosts | 17:26 |
| admin0 | i was unable to grep -ri | 17:26 |
| mnaser | openstack_hosts admin0 | 17:26 |
| admin0 | strange ..it was never there | 17:27 |
| mnaser | it was always there =P | 17:27 |
| openstackgerrit | Michael Johnson proposed openstack/openstack-ansible-os_octavia master: Set octavia_amp_image_owner_id by default https://review.openstack.org/648744 | 17:29 |
| admin0 | mnaser, i meant there were no entries in my /etc/hosts of deploy | 17:29 |
| mnaser | oh I dunno then | 17:29 |
| mnaser | ok while some jobs run, I'm going to go grab some food | 17:30 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer stable/rocky: gnocchi_resources override fixed https://review.openstack.org/625213 | 17:30 |
| admin0 | cloudnull, the | 17:30 |
| admin0 | cloudnull, the openstack_hosts add /etc/hosts entries to every blades .. except the deploy node | 17:30 |
| admin0 | and mtc is run from deploy server | 17:31 |
| admin0 | the probe fails to find containers but the entries are not in /etc/hosts of deploy | 17:31 |
| admin0 | is that normal ? the probe is only supposed to find hosts but not containers ? | 17:31 |
| *** hamzaachi has quit IRC | 17:31 | |
| admin0 | or do I need to manually copy the /etc/hosts to deploy and re-run the probe ? | 17:32 |
| guilhermesp | uoow very active day | 17:33 |
| openstackgerrit | Merged openstack/openstack-ansible master: barbican: run without SSL inside gate for metal jobs https://review.openstack.org/648234 | 17:33 |
| openstackgerrit | Merged openstack/openstack-ansible master: repo: stop using repo_build + pip_install https://review.openstack.org/648477 | 17:33 |
| guilhermesp | haven't scrolled up but do we have a wip for this? http://logs.openstack.org/31/648231/4/check/openstack-ansible-deploy-aio_metal_octavia-ubuntu-bionic/26ae770/logs/ara-report/result/367dc087-106b-42df-961c-213342fe07fe/ | 17:37 |
| guilhermesp | mnaser ^ | 17:37 |
| mnaser | I’m working on it in a local vm to see what’s going on | 17:38 |
| guilhermesp | all right | 17:38 |
| *** cmart has quit IRC | 17:41 | |
| admin0 | cloudnull, i will wait for you when you are free | 17:46 |
| *** cmart has joined #openstack-ansible | 17:49 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Test using integrated build https://review.openstack.org/648512 | 17:50 |
| *** yolanda has quit IRC | 17:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.openstack.org/642843 | 17:58 |
| odyssey4me | nurdie happy to help - good luck! | 18:04 |
| odyssey4me | admin0 I think it's openstack-hosts? | 18:04 |
| admin0 | it added odyssey4me .. but not on the deploy server | 18:04 |
| admin0 | it added everywhere else | 18:04 |
| admin0 | but this repo https://github.com/rcbops/magnanimous-turbo-chainsaw .. probe systems .. it probes from the deploy host .. and fails to resolve all deployment hosts | 18:05 |
| odyssey4me | admin0 yeah, that is true - I guess pulling /etc/hosts from one of the infra hosts would be useful | 18:06 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: missing document start "---" https://review.openstack.org/614107 | 18:06 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/639533 | 18:06 |
| admin0 | ok | 18:06 |
| guilhermesp | not sure why is complaining now http://logs.openstack.org/83/648583/3/check/openstack-ansible-deploy-aio_metal_trove-centos-7/7ea8c6e/job-output.txt.gz#_2019-03-29_06_15_44_653811 but I think would be a solution to add trove_service_net_setup = true in os-trove-overrides? | 18:08 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_horizon master: Update the horizon help link https://review.openstack.org/582793 | 18:09 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_trove master: Test using integrated build https://review.openstack.org/648583 | 18:10 |
| openstackgerrit | Merged openstack/openstack-ansible master: gate: switch all public URLs to use HTTP for metal https://review.openstack.org/648579 | 18:13 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_horizon master: Update the horizon help link https://review.openstack.org/582793 | 18:14 |
| *** CobHead has quit IRC | 18:14 | |
| admin0 | before i run it, what is fleet+osquery ? | 18:15 |
| admin0 | i only know elk | 18:15 |
| admin0 | not sure what skydive is | 18:15 |
| mnaser | guilhermesp: the os-trove-overrides no longer works in our new aio scenarios | 18:18 |
| mnaser | so im wondering if we should throw that stuff into global_overrides somewhere | 18:19 |
| *** CobHead has joined #openstack-ansible | 18:19 | |
| guilhermesp | yes you're right | 18:19 |
| raukadah | odyssey4me: still around need some help on this review https://review.openstack.org/#/c/648678/2/playbooks/tempest.yml | 18:20 |
| raukadah | and testing here https://review.openstack.org/#/c/648680/3/zuul.d/standalone-jobs.yaml | 18:20 |
| admin0 | it fails on ERROR! Unable to open /ansible-requirements.yml: [Errno 2] No such file or directory: u'/ansible-requirements.yml' | 18:21 |
| raukadah | I need to set var key dynamically | 18:21 |
| *** nurdie has quit IRC | 18:22 | |
| guilhermesp | mnaser that means, overrides from role tests would be placed in the integrated repo? | 18:22 |
| mnaser | guilhermesp: yeah | 18:24 |
| mnaser | gnocchi finally passing - https://review.openstack.org/#/c/648515/ | 18:24 |
| guilhermesp | ok so let me see which is the best way, coz seems that trove is requiring that var | 18:25 |
| guilhermesp | mnaser: +w for gnocchi :) | 18:25 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server master: pypiserver: drop pypi server https://review.openstack.org/648556 | 18:28 |
| *** electrofelix has quit IRC | 18:29 | |
| *** hamzaachi has joined #openstack-ansible | 18:30 | |
| *** DanyC has joined #openstack-ansible | 18:30 | |
| mnaser | good bye pypiserver | 18:31 |
| guilhermesp | seems that not we just have octavia and ironic to be merged https://review.openstack.org/#/q/topic:bp/python-build-simplification+(status:open+OR+status:merged) <3 | 18:32 |
| admin0 | source setup-workspace.sh | 18:32 |
| admin0 | -- this is where it gives "ERROR! Unable to open /ansible-requirements.yml: [Errno 2] No such file or directory: u'/ansible-requirements.yml'" "/embedded-ansible-setup.yml" --- maybe wrong directory | 18:32 |
| admin0 | i will try to keep diggign | 18:32 |
| cloudnull | admin0 sorry was afk | 18:35 |
| cloudnull | is that running in the /opt/magnanimous-turbo-chainsaw directory ? | 18:36 |
| admin0 | yep | 18:36 |
| cloudnull | and /opt/openstack-ansible-ops is there? | 18:37 |
| admin0 | yep | 18:37 |
| *** luksky has joined #openstack-ansible | 18:37 | |
| admin0 | the setup script cloned that | 18:37 |
| cloudnull | from the mtc directory i just run `. scripts/setup-workspace.sh` | 18:38 |
| cloudnull | https://pasted.tech/pastes/40618498c110256e0dd42e65ed9b1ed730c00d81 | 18:38 |
| *** pcaruana has quit IRC | 18:38 | |
| *** DanyC has quit IRC | 18:38 | |
| *** DanyC has joined #openstack-ansible | 18:42 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Update role for new source build process https://review.openstack.org/648463 | 18:42 |
| *** DanyC has quit IRC | 18:43 | |
| *** DanyC has joined #openstack-ansible | 18:43 | |
| *** raukadah has quit IRC | 18:45 | |
| *** CobHead has quit IRC | 18:54 | |
| *** cshen has joined #openstack-ansible | 19:06 | |
| *** cshen has quit IRC | 19:10 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.openstack.org/645937 | 19:13 |
| guilhermesp | hum wondering why it is complaining http://logs.openstack.org/66/648566/5/check/openstack-ansible-deploy-aio_lxc-centos-7/6a6a8b5/logs/ara-report/result/bedaf39d-cd6a-4cce-a79c-fd136f0c23a6/ | 19:14 |
| guilhermesp | seems ok here https://review.openstack.org/#/c/648724/2/tasks/python_venv_wheel_build.yml | 19:14 |
| *** CobHead has joined #openstack-ansible | 19:16 | |
| guilhermesp | seems that that lack of }} was one of the patch set edits, let me recheck | 19:22 |
| *** noonedeadpunk has quit IRC | 19:25 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server master: Test using integrated build https://review.openstack.org/648563 | 19:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Test using integrated build https://review.openstack.org/648104 | 19:28 |
| *** hamzaachi has quit IRC | 19:29 | |
| *** hamzaachi has joined #openstack-ansible | 19:29 | |
| mnaser | guilhermesp: that was already rechecked :) | 19:30 |
| guilhermesp | but have failed right? maybe I haven't subscribed to the PR before to receive updates | 19:31 |
| guilhermesp | sorry for that :P | 19:31 |
| mnaser | nope | 19:32 |
| mnaser | if you click Toggle CI | 19:32 |
| mnaser | you see that Zuul hasn't answered since recheck | 19:32 |
| mnaser | and you can find it in http://zuul.openstack.org | 19:32 |
| guilhermesp | huum right, so at least my recheck didn't reset the one before | 19:33 |
| mnaser | you've got to be kidding me | 19:33 |
| mnaser | ceph jobs all seem to be failing now?! | 19:33 |
| mnaser | or seem to be failing often anyways | 19:34 |
| *** tosky has joined #openstack-ansible | 19:48 | |
| guilhermesp | mnaser +1 https://review.openstack.org/#/c/648566/ <3 | 19:52 |
| *** yolanda has joined #openstack-ansible | 19:58 | |
| *** DanyC has quit IRC | 20:04 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Track Stein when we can https://review.openstack.org/648710 | 20:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646824 | 20:09 |
| *** nurdie has joined #openstack-ansible | 20:19 | |
| *** nurdie has quit IRC | 20:19 | |
| *** nurdie has joined #openstack-ansible | 20:20 | |
| mnaser | yay finally | 20:24 |
| *** mrhillsman is now known as mrhillsman_away4 | 20:40 | |
| *** mrhillsman_away4 is now known as mrhillsman_wkend | 20:41 | |
| *** shananigans has quit IRC | 20:42 | |
| *** spatel has quit IRC | 20:51 | |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Add option to upgrade PyPI/etc without using site config https://review.openstack.org/648724 | 20:55 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: fix tox python3 overrides https://review.openstack.org/606796 | 21:05 |
| *** cshen has joined #openstack-ansible | 21:06 | |
| *** cshen has quit IRC | 21:10 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: missing document start "---" https://review.openstack.org/614119 | 21:21 |
| fnpanic | hi | 21:21 |
| fnpanic | does anyone know when the service specfic policy.json in horizon are updated? | 21:21 |
| fnpanic | i change the policy.json of the glance service | 21:22 |
| fnpanic | with ovverides | 21:22 |
| fnpanic | and it installed perfect | 21:22 |
| fnpanic | i also did a os-horizon-install | 21:22 |
| *** phasespace has joined #openstack-ansible | 21:22 | |
| fnpanic | but the policy.json does not match | 21:22 |
| fnpanic | - /openstack/venvs/horizon-18.1.4/lib/python2.7/site-packages/openstack_dashboard/conf/glance_policy.json | 21:23 |
| fnpanic | it is me or what else do i need to do? | 21:23 |
| admin0 | cloudnull, https://pasted.tech/pastes/36014404037e73e3c9569c9bd3ef2d863c483cda | 21:24 |
| cloudnull | admin0 running bash? | 21:25 |
| cloudnull | from the output it looks like readlink has different options? | 21:26 |
| admin0 | i am a read docs, copy/paste guy :) | 21:33 |
| *** nurdie_ has joined #openstack-ansible | 21:43 | |
| mnaser | hmm | 21:43 |
| mnaser | I think I found a reason why our jobs are maybe not so fast | 21:43 |
| mnaser | it looks like we're never caching wheels when we download them, is that on purpose? | 21:44 |
| mnaser | id assume in a scenario where we deploy in metal it would cache some of the wheels | 21:44 |
| *** nurdie has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Fixed variable in example playbook https://review.openstack.org/648644 | 21:50 |
| cloudnull | admin0 it should work with copy/paste but ive only ever tested it with bash | 21:51 |
| logan- | mnaser: i wouldnt be surprised if we have some pip configuration to disable caching due to the expectation of a repo server having all of that content readily available | 21:55 |
| admin0 | cloudnull, when starting from scratch, on the bash bash /opt/mtc-setup.sh | 21:55 |
| admin0 | ERROR! Unable to open /ansible-requirements.yml: [Errno 2] No such file or directory: u'/ansible-requirements.yml' | 21:55 |
| admin0 | i mean why would any script put file in / | 21:56 |
| admin0 | cloudnull, how do I paste files to pasted.tech ? | 21:58 |
| admin0 | like curl and post the paste | 21:58 |
| cloudnull | you can copy / paste here https://pasted.tech/pastes | 21:58 |
| cloudnull | or i have a python client | 21:58 |
| cloudnull | `pip install pasted-client` | 21:59 |
| logan- | mnaser: probably not though, no hits in osa repos for no-cache-dir, so nevermind | 21:59 |
| mnaser | logan-: I think I ripped out all of that | 21:59 |
| cloudnull | admin0 those tools are not installing anything on / | 21:59 |
| cloudnull | it should all be in /opt | 21:59 |
| admin0 | how do I use pasted-client :D ? | 22:00 |
| logan- | echo 'hello pastes' | pasted | 22:01 |
| admin0 | cloudnull, https://pasted.tech/pastes/7db287a4de3d9dbe9383960659df4a20f5825734 | 22:01 |
| admin0 | that is the output of 2nd command which is after the wget https://github.com/rcbops/magnanimous-turbo-chainsaw | 22:03 |
| admin0 | bash /opt/mtc-setup.sh | 22:03 |
| cloudnull | admin0 - https://pasted.tech/pastes/49b626746b149defdd41215d4a69faf23d1f8461 - works for me ? | 22:04 |
| cloudnull | ^ running ubuntu 18.04. what OS is that maybe something I need to address there? | 22:05 |
| cloudnull | ++ thanks logan- :) | 22:06 |
| fnpanic | just seen this doc | 22:06 |
| fnpanic | https://docs.openstack.org/horizon/rocky/contributor/topics/policy.html | 22:06 |
| cloudnull | maybe an os specific variable is not set ? | 22:07 |
| cloudnull | admin0 ^ | 22:07 |
| fnpanic | how to update the poilcy files in horizon? | 22:07 |
| admin0 | cloudnull, no idea :) i do not know | 22:07 |
| fnpanic | i guess this is the olso stuff | 22:07 |
| *** fghaas has quit IRC | 22:08 | |
| cloudnull | admin0 whats the os its running on ? | 22:09 |
| mnaser | also | 22:10 |
| mnaser | the ceph job failed again in the gate | 22:10 |
| *** hamzaachi has quit IRC | 22:13 | |
| mnaser | http://logs.openstack.org/66/648566/5/gate/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/83b8870/ if anyone can dig please | 22:13 |
| mnaser | bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (13) Permission denied | 22:16 |
| mnaser | 2019-03-29 21:47:44.363 7f4f5caa6e00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied | 22:16 |
| mnaser | jrosser, logan-: ever seen something like that ^? | 22:17 |
| admin0 | ubuntu 18.04 | 22:20 |
| logan- | mnaser: nope, found https://github.com/ceph/ceph-ansible/issues/2354 though | 22:22 |
| mnaser | yikes, it's non deterministic too | 22:23 |
| *** cmart has quit IRC | 22:24 | |
| logan- | this sounds vaguely familiar though | 22:24 |
| logan- | i want to say https://github.com/openstack/openstack-ansible/commit/dcf448ec811dc53523b03752aa9a52c53ea15ae7#diff-139f124fbf5bb2851ae02494011e5805 "Set permissions on loopback devices" was meant to address this at one point | 22:25 |
| logan- | if only the gate logs for PS3 on https://review.openstack.org/#/c/629317 still existed :) | 22:26 |
| logan- | that was the PS right before jrosser added that task | 22:27 |
| mnaser | hmm | 22:27 |
| mnaser | But then I can’t imagine something resetting the permissions on it after | 22:27 |
| mnaser | logan-: I’m tempted to move the job to metal for speed and really no one runs OSDs on control plane anyways. Thoughts. ? | 22:30 |
| logan- | i don't think it will help, the osds already run on the host | 22:32 |
| logan- | only the mons and rgw are containerized | 22:32 |
| logan- | it could help with speed, but not this error imo | 22:33 |
| logan- | not sure if we gain much having the mons/rgw containerized other than a test path following the default deployment scenario | 22:34 |
| mnaser | logan-: bleh, yeah i figured it would be faster | 22:38 |
| mnaser | our deploys have become so much slower | 22:38 |
| mnaser | especially centos and I don't know why | 22:39 |
| logan- | :/ | 22:39 |
| logan- | just ceph? or all centos jobs? | 22:39 |
| mnaser | centos in general | 22:39 |
| mnaser | https://review.openstack.org/#/c/648566/ | 22:39 |
| logan- | well nvm i don't think we have a centos ceph job :) | 22:39 |
| *** cmart has joined #openstack-ansible | 22:40 | |
| mnaser | 2h17m with using infra wheel mirrors | 22:40 |
| mnaser | bionic takes 1h56m (which is still long imho) | 22:40 |
| mnaser | now, metal takes a whole HOUR less | 22:40 |
| logan- | yeah | 22:40 |
| mnaser | which makes me feel like we have some sort of lxc related thing | 22:40 |
| mnaser | even bionic metal vs lxc is a big hour difference | 22:41 |
| mnaser | looking in art nothing stands out though | 22:41 |
| logan- | setup-openstack.yml takes 25 minutes longer on lxc vs metal there. but we also skip some services on metal iirc, like horizon, not sure how many others we skip | 22:42 |
| mnaser | I think horizon is the only openstack service | 22:42 |
| mnaser | I see a lot of NetworkManager stuff | 22:43 |
| logan- | 638 additional tasks run in the setup-openstack.yml on lxc job, so about .42 seconds per additional task. avg on the metal setup-openstack.yml is 0.4s per task. so ie. its not the tasks that are taking longer on lxc, its the number of tasks increasing that eats up the extra time there | 22:46 |
| logan- | just comparing http://logs.openstack.org/66/648566/5/check/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/352c8c9/logs/ara-report/ vs http://logs.openstack.org/66/648566/5/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/1100fe2/logs/ara-report/ | 22:47 |
| mnaser | im thinking we have some sort of performance issue in how we execute inside containers | 22:47 |
| mnaser | 6 minutes => 19 minutes for setup infra | 22:47 |
| mnaser | only 131 more tasks | 22:48 |
| mnaser | wow | 22:48 |
| mnaser | creating a container takes 2m44s? | 22:48 |
| mnaser | (2 gigabytes) / (2 minutes 44 seconds) = 12.195122 MBps | 22:49 |
| mnaser | I doubt our base images are 2gb? | 22:49 |
| logan- | install galera server: metal=1m16s lxc=4m51s | 22:49 |
| logan- | container create should be very fast (like a few seconds or less) if we are using overlay containers in the gate, which we used to do | 22:50 |
| logan- | dir backed containers do require a full rootfs copy for each container | 22:50 |
| mnaser | but overlay had that nasty issue that we kept hitting right? | 22:50 |
| mnaser | about out of disk in ceph jobs | 22:50 |
| logan- | i think that was the machinectl/btrfs stuff. we used 'overlay' backing for years without issue prior to the machinectl/btrfs stuff afaik. | 22:51 |
| logan- | it uses overlayfs instead of btrfs cow subvolumes | 22:51 |
| logan- | yeah, using dir right now: https://github.com/openstack/openstack-ansible/blob/ae014318d2567fef677957c0d3f7ce1ad1ebce93/tests/roles/bootstrap-host/defaults/main.yml#L191-L192 | 22:52 |
| logan- | so we'll see a lot of job timing variance depending on which nodepool provider the job hits | 22:52 |
| mnaser | logan-: https://review.openstack.org/#/c/576624/ 1h26m for denial at the time | 22:54 |
| mnaser | but I mean things like installing Galera taking 5 times more time inside containers | 22:54 |
| mnaser | that .. is odd | 22:54 |
| mnaser | I worry that we're shipping with a huge regression and we're messing up performance by changing something | 22:55 |
| mnaser | meh, just saw the numbers for rocky... seems to be the same times | 22:56 |
| mnaser | https://review.openstack.org/#/c/647479/ | 22:56 |
| logan- | https://github.com/openstack/openstack-ansible/blob/fdc69308066be00dbc8845ebf7b352d88d1e8cf4/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L169-L171 was used from newton -> pike, then we switched to machinectl for queens, then reverted to 'dir' for q->s | 22:58 |
| logan- | https://github.com/openstack/openstack-ansible/commit/55101030f244eb1c1c868b429607d57784a18f7f | 22:58 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: rally: add empty rally.yml.aio file https://review.openstack.org/648741 | 22:58 |
| mnaser | so | 23:01 |
| mnaser | this is the only last issue with python_venv_build: https://review.openstack.org/#/c/648231/ | 23:01 |
| mnaser | the inventory inside container_networks somehow doesn't include the ip for the host for the lab's network | 23:01 |
| mnaser | so it complains about lbaas_address being missing | 23:01 |
| mnaser | I ran it locally and this is the inventory I have http://paste.openstack.org/show/748616/ | 23:03 |
| mnaser | specifically "lbaas_address" is missing address in there, so its unable to grab it from there in order to build list of health managers | 23:04 |
| mnaser | "If the host is found to be "is_metal" it will be marked as "on_metal" and will not have an additionally assigned IP address." | 23:05 |
| mnaser | wellllll that'll do it | 23:05 |
| *** cshen has joined #openstack-ansible | 23:06 | |
| mnaser | so this probably will work in containers, but it'll be sloOooOooOw | 23:08 |
| admin0 | cloudnull, what do i need to do to make this work ? | 23:09 |
| *** cshen has quit IRC | 23:11 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: octavia: use br-lbaas IP for health managers https://review.openstack.org/648793 | 23:15 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Test using integrated build https://review.openstack.org/648509 | 23:16 |
| *** SmearedBeard has joined #openstack-ansible | 23:28 | |
| *** SmearedBeard has quit IRC | 23:40 | |
| cloudnull | admin0 im really not sure what the delta is | 23:54 |
| cloudnull | maybe we could do a screen share? | 23:54 |
| cloudnull | maybe not this evening, but soon. | 23:54 |
| admin0 | sure .. tomorrow or sunday .any time | 23:58 |
| admin0 | its 1AM for me here now | 23:59 |
| cloudnull | yea. I might be around this weekend but monday for sure. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!