| *** mrda has quit IRC | 00:28 | |
| *** jcoufal has joined #tripleo | 00:31 | |
| *** mmethot_ has quit IRC | 00:40 | |
| *** mmethot has joined #tripleo | 00:40 | |
| *** mmethot has quit IRC | 00:45 | |
| *** mmethot has joined #tripleo | 00:45 | |
| *** mmethot has quit IRC | 00:46 | |
| *** mmethot has joined #tripleo | 00:47 | |
| *** mmethot has quit IRC | 00:55 | |
| *** mmethot_ has joined #tripleo | 00:55 | |
| *** jaganathan has quit IRC | 01:01 | |
| *** mschuppert has quit IRC | 01:19 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements stable/rocky: Add proper permissions on script https://review.opendev.org/682535 | 01:29 |
|---|---|---|
| *** mrda has joined #tripleo | 01:35 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements stable/stein: Add proper permissions on script https://review.opendev.org/682534 | 01:40 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/stein: Explicitly make a list https://review.opendev.org/682674 | 01:40 |
| openstackgerrit | Merged openstack/tripleo-common master: mistral-executor image install novajoin package https://review.opendev.org/682527 | 01:52 |
| *** pierrepr1netti has joined #tripleo | 01:53 | |
| *** mcornea has joined #tripleo | 01:55 | |
| *** ricolin has joined #tripleo | 01:55 | |
| *** pierreprinetti has quit IRC | 01:56 | |
| *** jcoufal has quit IRC | 01:56 | |
| openstackgerrit | Merged openstack/paunch stable/stein: Add --cpuset-cpus support for both Docker and Podman https://review.opendev.org/682484 | 01:58 |
| openstackgerrit | Merged openstack/ansible-role-tripleo-modify-image master: Update README examples on yum update/install https://review.opendev.org/682370 | 01:58 |
| *** aakarsh has joined #tripleo | 02:00 | |
| *** mcornea has quit IRC | 02:04 | |
| *** dsneddon has quit IRC | 02:04 | |
| openstackgerrit | Merged openstack/paunch master: Cleanup useless warnings for podman https://review.opendev.org/682651 | 02:14 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add basic unit tests for tripleo::profile::base::keystone https://review.opendev.org/682519 | 02:14 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Use Python3 client for OSP15 https://review.opendev.org/680970 | 02:14 |
| openstackgerrit | Merged openstack/paunch master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/673763 | 02:14 |
| openstackgerrit | Merged openstack/paunch master: Blacklist sphinx 2.1.0 (autodoc bug) https://review.opendev.org/673762 | 02:14 |
| *** dsneddon has joined #tripleo | 02:17 | |
| *** dsneddon has quit IRC | 02:22 | |
| *** rlandy|bbl is now known as rlandy | 02:37 | |
| *** HardCase has quit IRC | 02:41 | |
| *** dsneddon has joined #tripleo | 02:45 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move nested bash script into a real file https://review.opendev.org/683270 | 02:51 |
| *** ricolin_ has joined #tripleo | 03:04 | |
| *** rfolco has quit IRC | 03:05 | |
| *** ricolin has quit IRC | 03:06 | |
| mwhahaha | cloudnull, EmilienM: so I think one issue with dropping the pool numbers is that we communicate with too many services with the pool so it's constantly thrashing new connections | 03:08 |
| mwhahaha | cloudnull, EmilienM: Looking in my logs we talk to docker.io, the local registry, cloudflare. so each time we talk to a different location, it's closing the pool to the previous place | 03:08 |
| mwhahaha | I think we need a large connection pool but smaller threads | 03:09 |
| weshay | everyone should have a large pool, beer, bbq | 03:14 |
| * mwhahaha is currently consuming canadian beer | 03:14 | |
| * mwhahaha raises one for EmilienM | 03:14 | |
| weshay | the nicest beer in the world | 03:14 |
| mwhahaha | it appologizes for getting you hungover | 03:14 |
| weshay | lolz | 03:14 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Ensure machine-id is not included in images https://review.opendev.org/683272 | 03:15 |
| cloudnull | mwhahaha that makes a lot of sense | 03:16 |
| cloudnull | up the pool to something like 32 (maybe 64?), but lower worker thread max ? | 03:16 |
| mwhahaha | we could try that | 03:17 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Ensure machine-id is not included in images https://review.opendev.org/683272 | 03:17 |
| mwhahaha | i'm currently running a prep to gather logs, i'll run that tomorrow | 03:17 |
| mwhahaha | is too late to be dealing with such shenanigans | 03:17 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Ensure machine-id is not included in images https://review.opendev.org/683272 | 03:17 |
| cloudnull | ++ | 03:18 |
| cloudnull | we had https://review.opendev.org/#/c/682731/9/tripleo_common/image/image_uploader.py going | 03:18 |
| cloudnull | maybe a good start ? | 03:18 |
| cloudnull | its still running, only had to wait ~14 hours, so should be done soon | 03:19 |
| weshay | ya man.. it gets ugly when this stuff happens | 03:19 |
| cloudnull | indeed | 03:19 |
| cloudnull | but good seeing some things merge | 03:19 |
| weshay | we had a good run :) | 03:19 |
| mwhahaha | cloudnull: yea that's the patch i pulled actually | 03:19 |
| weshay | ya.. | 03:19 |
| cloudnull | there was a bit of a merge party earlier | 03:20 |
| cloudnull | which was good | 03:20 |
| mwhahaha | cloudnull: oh did we ever revert the session close stuff? | 03:22 |
| mwhahaha | that might also be contributing | 03:22 |
| mwhahaha | no we did not | 03:23 |
| mwhahaha | will need to try that as well | 03:23 |
| cloudnull | we did not | 03:23 |
| cloudnull | https://review.opendev.org/#/c/682719/ | 03:24 |
| cloudnull | we have that one here. we could propose that on its own | 03:24 |
| cloudnull | right now its stacked | 03:24 |
| mwhahaha | yikes it took 31 mins for my run w/ the lower counts | 03:31 |
| openstackgerrit | Pranali Deore proposed openstack/puppet-tripleo master: WIP, Add glance multiple backend support https://review.opendev.org/683275 | 03:32 |
| *** psachin has joined #tripleo | 03:36 | |
| *** rlandy has quit IRC | 03:38 | |
| *** ykarel|away has joined #tripleo | 03:39 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Use async to ensure faster task execution https://review.opendev.org/683277 | 03:45 |
| *** ramishra has joined #tripleo | 03:47 | |
| *** gkadam has joined #tripleo | 03:58 | |
| *** gkadam has quit IRC | 03:58 | |
| *** skramaja has joined #tripleo | 03:59 | |
| *** dsneddon has quit IRC | 03:59 | |
| *** dsneddon has joined #tripleo | 04:01 | |
| *** dsneddon has quit IRC | 04:05 | |
| *** ykarel|away is now known as ykarel | 04:11 | |
| *** karthiks has joined #tripleo | 04:12 | |
| *** soniya29 has joined #tripleo | 04:24 | |
| *** rh-jelabarre has quit IRC | 04:32 | |
| *** karthiks has quit IRC | 04:33 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: DNM - test CI against quay.io https://review.opendev.org/683212 | 04:37 |
| *** karthiks has joined #tripleo | 04:39 | |
| *** ratailor has joined #tripleo | 04:40 | |
| *** _mmethot_ has joined #tripleo | 04:41 | |
| *** mmethot_ has quit IRC | 04:41 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move nested bash script into a real file https://review.opendev.org/683270 | 04:42 |
| *** surpatil has joined #tripleo | 04:43 | |
| *** weshay has quit IRC | 04:52 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Use async to ensure faster task execution https://review.opendev.org/683277 | 04:56 |
| *** ade_lee_ has joined #tripleo | 04:57 | |
| *** ade_lee__ has quit IRC | 04:57 | |
| *** ade_lee_ has quit IRC | 05:04 | |
| *** ade_lee_ has joined #tripleo | 05:04 | |
| *** holser has joined #tripleo | 05:08 | |
| *** holser has quit IRC | 05:12 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move basic global functions into ansible tasks https://review.opendev.org/683286 | 05:15 |
| *** pcaruana has joined #tripleo | 05:28 | |
| *** dsneddon has joined #tripleo | 05:34 | |
| *** dsneddon has quit IRC | 05:40 | |
| *** ykarel has quit IRC | 05:43 | |
| *** pcaruana has quit IRC | 05:53 | |
| *** ricolin_ is now known as ricolin | 05:54 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: Don't use pacemaker in ironic sa job https://review.opendev.org/682410 | 05:55 |
| *** ksambor has joined #tripleo | 05:57 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: Remove docker resource from scen004 https://review.opendev.org/683289 | 05:57 |
| *** lmiccini has joined #tripleo | 05:59 | |
| *** ykarel has joined #tripleo | 06:03 | |
| *** gfidente has joined #tripleo | 06:06 | |
| openstackgerrit | Merged openstack/ansible-role-collect-logs master: Don't gzip ARA when generating HTML https://review.opendev.org/683039 | 06:07 |
| *** apetrich has quit IRC | 06:08 | |
| Tengu | EmilienM: hey! I didn't follow all the thing about the CI issue with inflight validations, but there's already a way to deactivate them in the CLI directly, at least in master. iirc you even reviewed the change, at least mwhahaha did. So maybe it's "just" a matter of changing the defaults in there instead of adding a new param in t-h-t? | 06:10 |
| Tengu | EmilienM: also..... the spec emphasis multiple times the inflight should NOT be invasive nor too long nor too expensive in term of resources. Maybe the real thing is to understand why the nova inflight takes so long, and put it back on the bench instead of dropping all the existing inflight checks? | 06:11 |
| Tengu | anyway. we can talk about that on Monday once I'm really back ;). | 06:11 |
| openstackgerrit | Merged openstack/paunch master: Improve volume validation https://review.opendev.org/682141 | 06:11 |
| *** Tengu has quit IRC | 06:16 | |
| *** Tengu has joined #tripleo | 06:17 | |
| *** jfrancoa has joined #tripleo | 06:19 | |
| *** slaweq has joined #tripleo | 06:19 | |
| *** Tengu has quit IRC | 06:20 | |
| *** Tengu has joined #tripleo | 06:21 | |
| *** jfrancoa has quit IRC | 06:23 | |
| *** psachin has quit IRC | 06:23 | |
| *** Tengu has quit IRC | 06:28 | |
| *** mschuppert has joined #tripleo | 06:28 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Add ovn_dbs_vip to export_data https://review.opendev.org/682294 | 06:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Revert "Move nova cell v2 discovery to deploy_steps_tasks" https://review.opendev.org/682644 | 06:32 |
| *** kopecmartin|off is now known as kopecmartin | 06:36 | |
| openstackgerrit | Natal Ngétal proposed openstack/python-tripleoclient master: Add test for the function check_deprecated_parameters. https://review.opendev.org/674531 | 06:36 |
| *** xek has joined #tripleo | 06:36 | |
| *** psachin has joined #tripleo | 06:39 | |
| *** dciabrin has joined #tripleo | 06:40 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Add posibilities to set ovn_openflow_probe_interval for controller https://review.opendev.org/683294 | 06:40 |
| *** holser has joined #tripleo | 06:41 | |
| *** Tengu has joined #tripleo | 06:42 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: Remove docker resource from scen004 https://review.opendev.org/683289 | 06:43 |
| *** Tengu has quit IRC | 06:44 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install Python 3 libselinux packages for Fedora https://review.opendev.org/683295 | 06:45 |
| *** zbr is now known as zbr|ruck | 06:48 | |
| *** Tengu has joined #tripleo | 06:50 | |
| *** bogdando has joined #tripleo | 06:57 | |
| *** hberaud|gone is now known as hberaud | 06:59 | |
| *** gfidente has quit IRC | 07:04 | |
| *** ccamacho has joined #tripleo | 07:13 | |
| *** marios has joined #tripleo | 07:15 | |
| *** tosky has joined #tripleo | 07:18 | |
| *** udesale has joined #tripleo | 07:20 | |
| *** yolanda has quit IRC | 07:21 | |
| *** amoralej|off is now known as amoralej | 07:23 | |
| *** yolanda has joined #tripleo | 07:23 | |
| *** rcernin has quit IRC | 07:29 | |
| *** rpittau|afk is now known as rpittau | 07:31 | |
| *** yolanda has quit IRC | 07:32 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install Python 3 libselinux packages for Fedora https://review.opendev.org/683295 | 07:33 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: [WIP] Queens-only: Run cellv2 host discovery via workflow or deploy_steps_tasks https://review.opendev.org/683069 | 07:35 |
| *** dsneddon has joined #tripleo | 07:36 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 07:41 |
| *** apetrich has joined #tripleo | 07:41 | |
| *** jpena|off is now known as jpena | 07:42 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 07:44 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 07:46 |
| *** gfidente has joined #tripleo | 07:47 | |
| *** yolanda has joined #tripleo | 07:47 | |
| *** iurygregory has joined #tripleo | 07:51 | |
| *** ykarel is now known as ykarel|lunch | 07:52 | |
| *** soniya29 has quit IRC | 07:54 | |
| *** jpich has joined #tripleo | 07:54 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 07:55 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 07:59 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: DNM test CI containers update with yum caching https://review.opendev.org/683101 | 07:59 |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart-extras master: Move from ocata to queens for the initial packages installation https://review.opendev.org/683313 | 08:02 |
| *** paramite|clone has joined #tripleo | 08:05 | |
| *** paramite has quit IRC | 08:07 | |
| *** dsneddon has quit IRC | 08:07 | |
| *** apetrich has quit IRC | 08:07 | |
| *** tkajinam has quit IRC | 08:09 | |
| *** alexmcleod has joined #tripleo | 08:14 | |
| *** suuuper has joined #tripleo | 08:32 | |
| *** dsneddon has joined #tripleo | 08:34 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Add posibilities to set ovn_openflow_probe_interval for controller https://review.opendev.org/683294 | 08:39 |
| *** cylopez has joined #tripleo | 08:39 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Depreace EC2 API https://review.opendev.org/683319 | 08:43 |
| odyssey4me | bogdando: Any chance for a re-review for https://review.opendev.org/682855 now that the pep8 issues are fixed? | 08:47 |
| *** ykarel|lunch is now known as ykarel | 08:50 | |
| *** rcernin has joined #tripleo | 08:56 | |
| *** florianf has joined #tripleo | 08:59 | |
| *** dsneddon has quit IRC | 09:01 | |
| *** dsneddon has joined #tripleo | 09:06 | |
| *** dsneddon has quit IRC | 09:10 | |
| *** pcaruana has joined #tripleo | 09:12 | |
| bogdando | odyssey4me: done, thanks for this improvement! | 09:14 |
| bogdando | odyssey4me: please take a look in return https://review.opendev.org/#/c/683100/ :) | 09:15 |
| bogdando | doing local benchmarking | 09:15 |
| bogdando | 10 images , now 36 (all I got for some env deployed) | 09:15 |
| *** paramite|clone has quit IRC | 09:22 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: DNM test CI containers update with yum caching https://review.opendev.org/683101 | 09:23 |
| *** derekh has joined #tripleo | 09:27 | |
| *** dtantsur|afk is now known as dtantsur | 09:27 | |
| *** paramite has joined #tripleo | 09:29 | |
| zbr|ruck | anyone that is also core on kolla, i want to merge https://review.opendev.org/#/c/683200/ - thanks. | 09:30 |
| *** apetrich has joined #tripleo | 09:41 | |
| *** apetrich has quit IRC | 09:45 | |
| openstackgerrit | Sandeep Yadav proposed openstack/tripleo-quickstart-extras master: Replace StandaloneLocalMtu with InterfaceLocalMtu https://review.opendev.org/683343 | 09:46 |
| odyssey4me | bogdando: review done, some comments/questions there | 09:48 |
| *** rcernin has quit IRC | 09:49 | |
| *** hberaud is now known as hberaud|lunch | 09:54 | |
| *** paramite has quit IRC | 09:56 | |
| *** apetrich has joined #tripleo | 09:59 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Correct wrong name in certmonger_certificate for ovn_controller https://review.opendev.org/682586 | 10:01 |
| *** jaosorior has quit IRC | 10:03 | |
| *** gfidente has quit IRC | 10:05 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Adding tests to skip list in Rocky https://review.opendev.org/683350 | 10:08 |
| Damjanek | Hmm. My RHOSP13 deployment dies ~10min after running openstack overcloud deploy. It did not yet managed to start stack creation. The message I'm seeing is: "Timed out waiting for messages from Execution (ID: f9852847-e5a3-4ecc-af0b-ec96a7737f0f, State: RUNNING). The WebSocket timed out before the Workflow completed." I can't find anything suspicious in mistral logs or anywhere. | 10:08 |
| Damjanek | Any pointers, where I should look for issues? | 10:08 |
| slaweq | arxcruz|rover: chandankumar: hi, do You remember my patch for tobiko to add there tripleo based job? | 10:09 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Adding tests to skip list in Rocky https://review.opendev.org/683350 | 10:09 |
| arxcruz|rover | slaweq: yes... | 10:09 |
| slaweq | arxcruz|rover: chandankumar: it was https://review.opendev.org/#/c/651828/ but it seems that I will need multinode environment to run tobiko | 10:09 |
| chandankumar | slaweq: ok | 10:09 |
| slaweq | mostly because tobiko spawns resources for tests and don't clean them - it's intentional | 10:09 |
| arxcruz|rover | hmmmmm | 10:10 |
| chandankumar | slaweq: then you need to switch to fs010 | 10:10 |
| slaweq | and on single node we can be out of e.g. memory | 10:10 |
| openstackgerrit | Sandeep Yadav proposed openstack/tripleo-docs master: Replace StandaloneLocalMtu with InterfaceLocalMtu https://review.opendev.org/683351 | 10:10 |
| slaweq | chandankumar: arxcruz|rover so I started new patch https://review.opendev.org/#/c/683126/ | 10:10 |
| chandankumar | slaweq: tripleo-ci-centos-7-containers-multinode | 10:10 |
| openstackgerrit | Merged openstack/tripleo-ansible master: Let ReaR uses different backup strategy https://review.opendev.org/682583 | 10:10 |
| slaweq | but it fails for me for now on multinode-overcloud.yaml | 10:10 |
| slaweq | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cc9/683126/2/check/tobiko-tripleo-containers-update/cc9e3ca/job-output.txt | 10:11 |
| slaweq | can You take a look on it? Maybe You will quickly know what I'm missing again | 10:11 |
| chandankumar | zbr|ruck: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cc9/683126/2/check/tobiko-tripleo-containers-update/cc9e3ca/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 10:12 |
| slaweq | chandankumar: fs010 will have heat? I don't remember now why by I choose fs016 - will it be good also or not? | 10:12 |
| chandankumar | zbr|ruck: resources.ServiceChain: Property error: resources[59].properties: Property DockerRsyslogConfigImage not assigned issue known? | 10:12 |
| arxcruz|rover | slaweq: 2019-09-19 22:58:22 | resources.ServiceChain: Property error: resources[59].properties: Property DockerRsyslogConfigImage not assigned | 10:12 |
| arxcruz|rover | chandankumar: nope, not known | 10:12 |
| chandankumar | slaweq: fs016 is triggered in rdo thirdparty or periodic job in rdo side | 10:12 |
| slaweq | chandankumar: ok, so featureset '010' than would be better, right? | 10:14 |
| chandankumar | slaweq: yes, | 10:15 |
| slaweq | chandankumar: now, another question, should I add ci/environments/scenario002-multinode-containers.yaml to be used on this job? to have e.g. heat installed | 10:15 |
| slaweq | chandankumar: and if yes, how I should add it to job definition? | 10:16 |
| slaweq | chandankumar: should it be something like: | 10:16 |
| slaweq | composable_scenario: scenario002-multinode-containers.yaml | 10:16 |
| Damjanek | Ok. Seems that this is an issue: | 10:16 |
| Damjanek | https://bugzilla.redhat.com/show_bug.cgi?id=1664055 | 10:16 |
| openstack | bugzilla.redhat.com bug 1664055 in openstack-tripleo-common "Message collection size is too large for Zaqar" [High,Post] - Assigned to apetrich | 10:16 |
| slaweq | in feature_override section? | 10:17 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-common stable/queens: [WIP] Add workflow to do cellv2 host discovery https://review.opendev.org/683067 | 10:18 |
| chandankumar | slaweq: let me check that | 10:18 |
| slaweq | chandankumar: thx a lot | 10:18 |
| chandankumar | slaweq: Here is the job defined https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L183 | 10:21 |
| chandankumar | slaweq: here is the multinode.yaml https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode.yml | 10:25 |
| chandankumar | slaweq: you can reorder the playbook in same and it will work | 10:25 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Use memcached for token caching in barbican authtoken https://review.opendev.org/681371 | 10:27 |
| *** artom has quit IRC | 10:32 | |
| slaweq | chandankumar: thx a lot | 10:34 |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Enable "port_forwarding" feature in neutron ML2 ovs environment https://review.opendev.org/683357 | 10:35 |
| EmilienM | Tengu: they were enabled by default | 10:36 |
| slaweq | chandankumar: but in my job I have those playbooks in same order | 10:40 |
| slaweq | except multinode-overcloud-upgrade.yml but job is failing before that even | 10:40 |
| *** paramite has joined #tripleo | 10:41 | |
| *** paramite|clone has joined #tripleo | 10:45 | |
| *** paramite has quit IRC | 10:46 | |
| *** brault has joined #tripleo | 10:47 | |
| chandankumar | EmilienM: need some help on this failure http://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cc9/683126/2/check/tobiko-tripleo-containers-update/cc9e3ca/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 10:52 |
| chandankumar | EmilienM: resources.ServiceChain: Property error: resources[59].properties: Property DockerRsyslogConfigImage not assigned | 10:52 |
| openstackgerrit | Merged openstack/tripleo-ci master: Run build containers on rocky branch https://review.opendev.org/682335 | 10:53 |
| *** pbandark has joined #tripleo | 10:54 | |
| *** pcaruana has quit IRC | 10:56 | |
| *** cylopez has quit IRC | 10:56 | |
| EmilienM | chandankumar: I'm on pto today | 10:56 |
| chandankumar | EmilienM: ok | 10:56 |
| *** jpena is now known as jpena|lunch | 11:01 | |
| *** panda is now known as panda|lunch | 11:03 | |
| *** artom has joined #tripleo | 11:05 | |
| *** dsneddon has joined #tripleo | 11:06 | |
| *** apetrich has quit IRC | 11:07 | |
| *** raildo has joined #tripleo | 11:07 | |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/stein: Fix image validation when AP services get several images. https://review.opendev.org/683066 | 11:08 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Fix passing tags through infrared to the tripleo-upgrade role. https://review.opendev.org/681268 | 11:17 |
| *** lucasagomes has joined #tripleo | 11:19 | |
| *** ccamacho has quit IRC | 11:21 | |
| *** hberaud|lunch is now known as hberaud | 11:26 | |
| *** jaosorior has joined #tripleo | 11:27 | |
| *** pcaruana has joined #tripleo | 11:29 | |
| *** dsneddon has quit IRC | 11:31 | |
| *** rh-jelabarre has joined #tripleo | 11:32 | |
| *** chandankumar is now known as raukadah | 11:33 | |
| *** pcaruana has quit IRC | 11:42 | |
| *** pcaruana has joined #tripleo | 11:42 | |
| *** dsneddon has joined #tripleo | 11:42 | |
| *** dmellado has quit IRC | 11:45 | |
| Tengu | EmilienM: I know. not sure if it's a good thing to deactivate, since a feature won't be tested as it should :/. | 11:47 |
| *** jaosorior has quit IRC | 11:47 | |
| Tengu | i.e. if a validation takes too long, maybe it's an issue to correct in order to make it faster, instead of dropping them all. | 11:47 |
| *** dsneddon has quit IRC | 11:47 | |
| *** dmellado has joined #tripleo | 11:48 | |
| *** apetrich has joined #tripleo | 11:51 | |
| *** udesale has quit IRC | 11:52 | |
| *** two_oes has joined #tripleo | 11:52 | |
| *** jpena|lunch is now known as jpena | 11:53 | |
| *** udesale has joined #tripleo | 11:54 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/stein: Safeguard against 'Undef' length() checks. https://review.opendev.org/682385 | 11:55 |
| apetrich | Damjanek, where is it the message too large issue happening? Can I help with something? | 11:55 |
| *** morazi has joined #tripleo | 11:57 | |
| *** weshay has joined #tripleo | 12:00 | |
| *** dsneddon has joined #tripleo | 12:01 | |
| Damjanek | apetrich: I'm observing that in mistral engine log. When applied a workaround with bumping sizes for both zaqar and mistral and then rebooted undercloud (because for some reason zaqar and mistral services restart was not enough), it went thru | 12:02 |
| Damjanek | Btw, is this normal behavour when using idrac pm driver, that every single time node is being rebooted by ironic, it reconfigures bios? | 12:03 |
| Damjanek | Or I have some wrong config or something? | 12:03 |
| *** artom has quit IRC | 12:04 | |
| apetrich | Damjanek, weird just rekicking the services is not enough. but good that it passed | 12:04 |
| apetrich | Damjanek, anyway feel free to ping if you have another mistral issue :) | 12:04 |
| Damjanek | apetrich: Will keep that in mind. Thanks! | 12:05 |
| *** dsneddon has quit IRC | 12:06 | |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates master: Workaround to avoid puppet managing internal files. https://review.opendev.org/654407 | 12:07 |
| *** panda|lunch is now known as panda | 12:16 | |
| *** amoralej is now known as amoralej|lunch | 12:19 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add missing allocations for active nodes https://review.opendev.org/678698 | 12:23 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Ensure pacemaker bootstrap controller is upgraded first. https://review.opendev.org/680595 | 12:23 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-quickstart master: Add support for running tempest against Octavia featuresets https://review.opendev.org/669666 | 12:23 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-quickstart master: Set the amphora image to stein branch https://review.opendev.org/678899 | 12:23 |
| *** ratailor has quit IRC | 12:25 | |
| *** psachin has quit IRC | 12:28 | |
| *** skramaja has quit IRC | 12:37 | |
| *** jaosorior has joined #tripleo | 12:38 | |
| *** rlandy has joined #tripleo | 12:38 | |
| *** cylopez has joined #tripleo | 12:39 | |
| *** apetrich has quit IRC | 12:42 | |
| *** rfolco has joined #tripleo | 12:46 | |
| *** aakarsh has quit IRC | 12:47 | |
| *** mcornea has joined #tripleo | 12:50 | |
| *** rcernin has joined #tripleo | 12:57 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Enable "port_forwarding" feature in neutron ML2 ovs environment https://review.opendev.org/683357 | 13:01 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: WIP: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 13:01 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade stable/stein: Fix passing tags through infrared to the tripleo-upgrade role. https://review.opendev.org/683379 | 13:05 |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Remove leftovers after NeutronExternalNetworkBridge option https://review.opendev.org/683380 | 13:05 |
| openstackgerrit | Merged openstack/os-net-config master: Add linkdelay option https://review.opendev.org/680569 | 13:06 |
| *** dsneddon has joined #tripleo | 13:08 | |
| *** paramite|clone has quit IRC | 13:13 | |
| *** brault has quit IRC | 13:13 | |
| *** dsneddon has quit IRC | 13:13 | |
| *** aakarsh has joined #tripleo | 13:15 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Add Octavia driver agent service https://review.opendev.org/658118 | 13:22 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-common stable/queens: [Queens] Add workflow to do cellv2 host discovery https://review.opendev.org/683067 | 13:25 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: [Queens] Run cellv2 host discovery via workflow or deploy_steps_tasks https://review.opendev.org/683069 | 13:25 |
| *** munimeha1 has joined #tripleo | 13:26 | |
| *** amoralej|lunch is now known as amoralej | 13:27 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add new healthcheck-service-status validation https://review.opendev.org/682377 | 13:30 |
| *** gfidente has joined #tripleo | 13:31 | |
| *** gfidente has quit IRC | 13:33 | |
| *** devep has joined #tripleo | 13:35 | |
| ykarel | slaweq, regarding the issue u faced, DockerRsyslogConfigImage, chandankumar asked me to look | 13:37 |
| ykarel | slaweq, https://review.opendev.org/#/c/624330/ missed changing docker image names to container like https://review.opendev.org/#/c/651550/13 | 13:37 |
| ykarel | https://review.opendev.org/#/c/624330/14/deployment/logging/rsyslog-container-puppet.yaml | 13:37 |
| ykarel | DockerRsyslogImage and DockerRsyslogConfigImage are outdated | 13:38 |
| ykarel | correct names are https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/overcloud_containers.yaml.j2#L924-L925 | 13:38 |
| *** apetrich has joined #tripleo | 13:39 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-specs master: Add a spec for K8s Metal3 baremetal operator https://review.opendev.org/659288 | 13:39 |
| slaweq | ykarel: thx for looking but how I can/should fix it? | 13:40 |
| slaweq | I'm not playing with those variables in my patch at all | 13:40 |
| ykarel | slaweq, so u using fs016 multinode | 13:41 |
| openstackgerrit | Alex Schultz proposed openstack/paunch master: Add --cpuset-cpus support for both Docker and Podman https://review.opendev.org/681257 | 13:41 |
| ykarel | slaweq, https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario001-multinode-containers.yaml#L17 | 13:41 |
| *** devep has quit IRC | 13:41 | |
| ykarel | and deployment/logging/rsyslog-container-puppet.yaml has wrong parameters | 13:41 |
| ykarel | so you hit it | 13:41 |
| mwhahaha | wait does that mean scenario001 doesn't have rsyslog but the multinode job did? | 13:42 |
| slaweq | ykarel: ok, so if I changed for fs 010 I should not have this issue anymore, right? | 13:43 |
| ykarel | slaweq, yes atlesat this issue will not be hit | 13:43 |
| ykarel | mwhahaha, you comparing standalone vs multinode? | 13:44 |
| mwhahaha | yea that's what i mean | 13:44 |
| ykarel | if yes multinode have rsyslog but standalone don't | 13:44 |
| mwhahaha | because if you're saying multinode is broken, then we should have found this via standalone | 13:45 |
| mwhahaha | the fix is to fix the Docker* vars on rsyslog tho | 13:45 |
| mwhahaha | and we should have the service enabled for a scenario | 13:45 |
| ykarel | mwhahaha, yes right | 13:45 |
| ykarel | 001-multinode are not running in master | 13:45 |
| ykarel | so standalone should cover it | 13:46 |
| openstackgerrit | Alex Schultz proposed openstack/paunch stable/stein: Improve volume validation https://review.opendev.org/683386 | 13:46 |
| *** surpatil has quit IRC | 13:47 | |
| *** weshay is now known as weshay|ruck | 13:47 | |
| ykarel | cloudnull, mwhahaha can u check https://review.opendev.org/#/c/681876/ | 13:50 |
| ykarel | dumb-init has broken jobs in queens/rocky | 13:50 |
| mwhahaha | 2019-09-20 05:56:13 | "/usr/bin/mysqld_safe: line 755: ulimit: -1: invalid option", | 13:50 |
| ykarel | nope | 13:50 |
| mwhahaha | ? | 13:50 |
| ykarel | dumb-init not found | 13:50 |
| mwhahaha | 2019-09-20 05:56:13 | "stderr: container_linux.go:235: starting container process caused \"exec: \\\"dumb-init\\\": executable file not found in $PATH\"", | 13:50 |
| ykarel | yes | 13:51 |
| openstackgerrit | Luca Miccini proposed openstack/puppet-tripleo stable/stein: Enable deep_compare for fencing resource https://review.opendev.org/683388 | 13:51 |
| mwhahaha | so dumb init is missing from haproxy | 13:51 |
| openstackgerrit | Luca Miccini proposed openstack/puppet-tripleo stable/rocky: Enable deep_compare for fencing resource https://review.opendev.org/683390 | 13:51 |
| *** apetrich has quit IRC | 13:51 | |
| mwhahaha | haproxy/rabbit/mysql | 13:51 |
| openstackgerrit | Luca Miccini proposed openstack/puppet-tripleo stable/queens: Enable deep_compare for fencing resource https://review.opendev.org/683391 | 13:51 |
| mwhahaha | bandini: did we have to do something to fix dumb-init for haproxy/rabbit/mysql containers | 13:51 |
| ykarel | mwhahaha, issue is dumb-init is installed there, but /usr/bin is bind mounted in container | 13:51 |
| slaweq | ykarel: thx | 13:51 |
| mwhahaha | lol | 13:51 |
| ykarel | so binaries in container are lost | 13:52 |
| mwhahaha | brilliant | 13:52 |
| ykarel | :D | 13:52 |
| * mwhahaha blames bandini for good measure | 13:52 | |
| mwhahaha | meh i'll look into the logs for stein to figure out what we did later | 13:52 |
| * bandini totally guilty as usual | 13:52 | |
| bandini | what's broken exactly? | 13:53 |
| mwhahaha | we switched to dumb-init | 13:54 |
| mwhahaha | which is loaded from /usr/bin | 13:54 |
| mwhahaha | it seems the pacemaker containers bindmoutn host /usr/bin | 13:54 |
| mwhahaha | making dumb-init unavailable | 13:54 |
| *** artom has joined #tripleo | 13:54 | |
| bandini | hohum | 13:54 |
| mwhahaha | can we drop the host bind mount? | 13:55 |
| mwhahaha | seems odd that we do that | 13:55 |
| mwhahaha | i would assume it's for the pacemaker binaries maybe? | 13:55 |
| *** dsneddon has joined #tripleo | 13:55 | |
| bandini | do you have a log of the failure and which bind mount are we talking about exactly? | 13:55 |
| mwhahaha | https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens/50b2520/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-09-20_05_56_13 | 13:56 |
| mwhahaha | 2019-09-20 05:56:13 | "stderr: container_linux.go:235: starting container process caused \"exec: \\\"dumb-init\\\": executable file not found in $PATH\"", | 13:56 |
| bandini | ok let me see | 13:56 |
| mwhahaha | \"/usr/bin:/usr/bin:ro\" | 13:56 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: Don't use pacemaker in ironic sa job https://review.opendev.org/682410 | 13:57 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: Use podman for scenario012 https://review.opendev.org/682341 | 13:57 |
| ykarel | mwhahaha, bind mount is there for running docker tag:- https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/pacemaker/haproxy.yaml#L213-L218 | 13:59 |
| mwhahaha | can we bind mount docker then | 13:59 |
| ykarel | and it's not there in train/stein as tripleo-container-tag role is used | 14:00 |
| mwhahaha | and not all of it | 14:00 |
| mwhahaha | or should we just install docker in those containers | 14:00 |
| ykarel | and that role directly run docker tag, instead running from containers | 14:00 |
| ykarel | or install dumb-init on host? | 14:00 |
| mwhahaha | actually i thought we installed docker in the containers | 14:00 |
| *** dsneddon has quit IRC | 14:00 | |
| ykarel | that will be a workaround though | 14:01 |
| *** gfidente has joined #tripleo | 14:01 | |
| mwhahaha | well for minor updates we need to limit it to containers | 14:01 |
| mwhahaha | it seems like installing docker in those containers is a better idea | 14:01 |
| mwhahaha | we did that for neutron and stuff | 14:01 |
| bogdando | actually bind-mount docker binary would be better | 14:03 |
| *** ayoung has joined #tripleo | 14:03 | |
| bogdando | a statically compiled binary, why installing the whole thing?.. | 14:03 |
| mwhahaha | ok let's switch to that then | 14:03 |
| bandini | if that will work, then i think it is the better approach | 14:03 |
| bandini | although we need to check if docker binary exist otherwise a podman deploy will fail due to missing bindmount? | 14:04 |
| bogdando | yes | 14:04 |
| mwhahaha | this is queens/rocky | 14:04 |
| mwhahaha | no podman | 14:04 |
| mwhahaha | we don't have this problem stein+ | 14:04 |
| mwhahaha | as mentioned we switched to something else | 14:04 |
| bandini | ok so let's tweak just in pre stein to /usr/bin/docker and see if CI does not barf on us | 14:04 |
| mwhahaha | this showed up because we're backporting dumb-init | 14:04 |
| bandini | right | 14:05 |
| bogdando | may be we could also backport tagging it on host :) | 14:05 |
| bogdando | not in container | 14:05 |
| mwhahaha | let's not | 14:05 |
| bogdando | :D | 14:05 |
| bandini | seems a bit risky to backport | 14:05 |
| bandini | although could work with some solid testing | 14:06 |
| mwhahaha | let's start with the docker bind mount :D then we can work on switching if we want | 14:06 |
| *** openstackgerrit has quit IRC | 14:06 | |
| bandini | agree, want me to propose a rocky review ? | 14:07 |
| bogdando | cautious cautiosness right | 14:07 |
| mwhahaha | yes plz | 14:07 |
| bandini | ack | 14:07 |
| *** openstackgerrit has joined #tripleo | 14:07 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Statically import CI reproducer roles https://review.opendev.org/681171 | 14:07 |
| *** rascasoft has quit IRC | 14:10 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add new healthcheck-service-status validation https://review.opendev.org/682377 | 14:10 |
| weshay|ruck | cloudnull, check this out | 14:11 |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-heat-templates stable/stein: Enable deep_compare by default for stonith resources https://review.opendev.org/683395 | 14:11 |
| weshay|ruck | 2019-09-20 06:42:33 | RUNNING HANDLER [tripleo-container-image-prepare : Delete param file] ********** | 14:11 |
| weshay|ruck | 2019-09-20 06:42:33 | task path: /usr/share/ansible/roles/tripleo-container-image-prepare/handlers/main.yml:18 | 14:11 |
| weshay|ruck | 2019-09-20 06:42:33 | Friday 20 September 2019 06:41:06 +0000 (0:15:06.780) 0:20:53.993 ****** | 14:11 |
| weshay|ruck | 2019-09-20 06:42:33 | changed: [undercloud] => { | 14:11 |
| weshay|ruck | 2019-09-20 06:42:33 | "changed": true, | 14:11 |
| weshay|ruck | 2019-09-20 06:42:33 | "path": "/tmp/ansible.xiUhDl-prepare-param", | 14:11 |
| weshay|ruck | 2019-09-20 06:42:33 | "state": "absent" | 14:11 |
| weshay|ruck | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9c8/678698/2/check/tripleo-ci-centos-7-containers-multinode/9c86690/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 14:11 |
| weshay|ruck | 15min | 14:11 |
| mwhahaha | we didn't do anything? | 14:11 |
| cloudnull | mornings | 14:12 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates stable/rocky: Stop bindmounting /usr/bin https://review.opendev.org/683396 | 14:12 |
| bandini | mwhahaha, dciabrin: ^ | 14:12 |
| bandini | let's see if ci passes | 14:12 |
| mwhahaha | k | 14:12 |
| weshay|ruck | cloudnull, we're going to put up a change that removes the handlers here https://review.opendev.org/#/c/674414/24/tripleo_ansible/roles/tripleo-container-image-prepare/handlers/main.yml | 14:13 |
| weshay|ruck | 2019-09-19 21:13:31 | RUNNING HANDLER [tripleo-container-image-prepare : Delete param file] ********** | 14:13 |
| weshay|ruck | 2019-09-19 21:13:31 | Thursday 19 September 2019 21:12:46 +0000 (0:28:32.293) 0:36:39.510 **** | 14:13 |
| weshay|ruck | 2019-09-19 21:13:31 | changed: [undercloud] => { | 14:13 |
| weshay|ruck | 2019-09-19 21:13:31 | "changed": true, | 14:13 |
| weshay|ruck | 2019-09-19 21:13:31 | "path": "/tmp/ansible.Q4cd6h-prepare-param", | 14:13 |
| weshay|ruck | 2019-09-19 21:13:31 | "state": "absent" | 14:13 |
| cloudnull | weshay|ruck is it the handler or just that the delete takes too long ? | 14:13 |
| weshay|ruck | delete takes too long | 14:14 |
| dciabrin | bandini, sec | 14:14 |
| weshay|ruck | arxcruz|rover, is working it ^ | 14:14 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: New validation: detect failed containers https://review.opendev.org/656785 | 14:15 |
| cloudnull | maybe we change those to simply command: rm https://pasted.tech/pastes/50c863cb2f3c45a3511e4366c9393072a154cc31 | 14:15 |
| cloudnull | see if that's in fact faster? | 14:15 |
| weshay|ruck | cloudnull, sounds fine | 14:16 |
| cloudnull | arxcruz|rover ^ | 14:16 |
| weshay|ruck | cloudnull, what's being removed there... the containers or yml file? | 14:16 |
| cloudnull | temp files | 14:16 |
| cloudnull | I suspect the ansible file module is walking the tree, and I guess there's lots of files ? | 14:17 |
| mwhahaha | likely | 14:17 |
| mwhahaha | we can't drop the cleanup, but we can rm -rf | 14:17 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-ansible master: Remove notify handler to delete the prepare parameter temp file https://review.opendev.org/683402 | 14:17 |
| ykarel | bandini, we need to run that patch with promotion pipeline where container exists with new entrypoint | 14:17 |
| *** dsneddon has joined #tripleo | 14:17 | |
| weshay|ruck | arxcruz|rover, ^ | 14:17 |
| ykarel | s/promotion pipeline/promotion jobs | 14:17 |
| arxcruz|rover | weshay|ruck: so rm -rf then ? | 14:17 |
| weshay|ruck | arxcruz|rover, ya | 14:17 |
| arxcruz|rover | okay, abandoning this one | 14:18 |
| weshay|ruck | update that review to use shell: rm | 14:18 |
| arxcruz|rover | okay | 14:18 |
| mwhahaha | ykarel: no we want to make sur eit doesn't break the current stuff | 14:18 |
| ykarel | mwhahaha, i meant additionally we need to check with promotion jobs | 14:18 |
| mwhahaha | ykarel: then we can land it and run it in promotion | 14:18 |
| mwhahaha | yea | 14:18 |
| bandini | yeah let's first see that it passes | 14:18 |
| ykarel | bandini, i can check that locally | 14:19 |
| cloudnull | make sure to add the ansible lint skip tag there - https://pasted.tech/pastes/58188095b175122dd58f7c69bd8839ad0db1be40 - but other than that the rm method should be faster. | 14:19 |
| arxcruz|rover | cloudnull: ack | 14:20 |
| ykarel | bandini, i got You don't have either docker-client or docker-client-latest installed. Please install either one and retry | 14:20 |
| ykarel | with /usr/bin/docker bind mount | 14:20 |
| mwhahaha | yea that was my concern | 14:20 |
| mwhahaha | is that there's more stuff | 14:20 |
| *** iurygregory_ has joined #tripleo | 14:21 | |
| bandini | let me try here | 14:21 |
| cloudnull | arxcruz|rover you might need to use shell, and make an test to see if the directory exists before running `rm`, | 14:21 |
| weshay|ruck | arxcruz++ | 14:21 |
| cloudnull | let me know how it goes | 14:21 |
| *** iurygregory has quit IRC | 14:21 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-ansible master: Switching the delete param handler to use rm https://review.opendev.org/683402 | 14:22 |
| *** rcernin has quit IRC | 14:23 | |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-heat-templates stable/rocky: Enable deep_compare by default for stonith resources https://review.opendev.org/683404 | 14:23 |
| weshay|ruck | arxcruz|rover, doesn't rm .. require shell: and not command? | 14:23 |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-heat-templates stable/queens: Enable deep_compare by default for stonith resources https://review.opendev.org/683405 | 14:23 |
| weshay|ruck | I think lint may fail on that | 14:23 |
| weshay|ruck | cloudnull, does that matter? ^ | 14:23 |
| arxcruz|rover | weshay|ruck: checking | 14:23 |
| weshay|ruck | this in theory should bring containers-multinode back to under 3hrs | 14:24 |
| cloudnull | weshay|ruck lint will be ok, arxcruz|rover used the skip_ansible_lint tag so it shouldn't complain | 14:24 |
| *** hberaud is now known as hberaud|school-r | 14:25 | |
| weshay|ruck | ah right | 14:25 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-ansible master: Switching the delete param handler to use rm https://review.opendev.org/683402 | 14:25 |
| arxcruz|rover | cloudnull: added || true in case the directory doesn't exist | 14:25 |
| arxcruz|rover | so it will not fail the task | 14:26 |
| *** cylopez has quit IRC | 14:26 | |
| * weshay|ruck notes standalone jobs are ALSO over 3 hours tripleo-ci-centos-7-scenario001-standaloneSUCCESS in 3h 05m 56s | 14:26 | |
| weshay|ruck | :( | 14:26 |
| *** jaosorior has quit IRC | 14:30 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move basic global functions into ansible tasks https://review.opendev.org/683286 | 14:30 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates stable/rocky: Stop bindmounting /usr/bin https://review.opendev.org/683396 | 14:30 |
| bandini | dciabrin, ykarel, mwhahaha: ^ this approach works for me locally | 14:30 |
| mwhahaha | cloudnull: -^ | 14:31 |
| * cloudnull running that review myself | 14:31 | |
| openstackgerrit | Alex Schultz proposed openstack/paunch master: Add --cpuset-cpus support for both Docker and Podman https://review.opendev.org/681257 | 14:32 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-ansible master: Switching the delete param handler to use rm https://review.opendev.org/683402 | 14:32 |
| ykarel | bandini, yes working for me too | 14:33 |
| *** hberaud|school-r is now known as hberaud | 14:33 | |
| ykarel | arxcruz|rover, fyi ^^ | 14:33 |
| *** dtantsur is now known as dtantsur|afk | 14:33 | |
| arxcruz|rover | ykarel: sorry man, i was working on another issue, i'm all ears now :) | 14:34 |
| ykarel | arxcruz|rover, np | 14:34 |
| arxcruz|rover | ykarel: i'll submit a patch to revert it ok ? | 14:35 |
| ykarel | arxcruz|rover, no | 14:35 |
| ykarel | arxcruz|rover, bandini has pushed a fix | 14:36 |
| ykarel | so let's observe results there | 14:36 |
| arxcruz|rover | ykarel: ack | 14:36 |
| ykarel | arxcruz|rover, and regarding master, i reran failed job with https://review.rdoproject.org/r/22163 | 14:37 |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-heat-templates stable/stein: Enable deep_compare by default for stonith resources https://review.opendev.org/683409 | 14:37 |
| ykarel | master promotion | 14:37 |
| ykarel | 001,050 passed | 14:37 |
| ykarel | 020 still running, 039 failed | 14:37 |
| ykarel | so i think we can consider promotion | 14:38 |
| arxcruz|rover | ykarel: okay, let's see | 14:38 |
| *** rcernin has joined #tripleo | 14:38 | |
| arxcruz|rover | abishop: I'll let the job finishe, because we need to ensure that's the issue that is making overcloud timeout | 14:39 |
| openstackgerrit | Alex Schultz proposed openstack/paunch master: Add --cpuset-cpus support for both Docker and Podman https://review.opendev.org/681257 | 14:40 |
| abishop | arxcruz|rover: np, good to verify the approach before refining it :) | 14:41 |
| openstackgerrit | Alex Schultz proposed openstack/paunch stable/stein: Fixing upper stable url https://review.opendev.org/683413 | 14:47 |
| *** udesale has quit IRC | 14:50 | |
| *** udesale has joined #tripleo | 14:51 | |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-heat-templates stable/stein: Enable deep_compare by default for stonith resources https://review.opendev.org/683409 | 14:53 |
| *** iurygregory_ is now known as iurygregory | 14:53 | |
| arxcruz|rover | weshay|ruck: I've update https://bugs.launchpad.net/tripleo/+bug/1844778 with the logs from featureset020 and the Delete param | 14:54 |
| openstack | Launchpad bug 1844778 in tripleo "Overcloud deployment is timeing out on scenario001 ovb" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 14:54 |
| *** Vorrtex has joined #tripleo | 14:56 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Enable "port_forwarding" feature in neutron ML2 ovs environment https://review.opendev.org/683357 | 14:59 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade stable/stein: Change docker references to podman. https://review.opendev.org/683418 | 14:59 |
| *** dsneddon has quit IRC | 15:00 | |
| *** _mmethot_ has quit IRC | 15:04 | |
| *** _mmethot_ has joined #tripleo | 15:05 | |
| *** bnemec is now known as beekneemech | 15:06 | |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Change docker references to podman. https://review.opendev.org/683081 | 15:09 |
| weshay|ruck | cloudnull, so that handler runs on the undercloud as well | 15:17 |
| weshay|ruck | but we can't see how long it's running for due to verbosity of ansible-playbook | 15:17 |
| * weshay|ruck going to put up a change for that | 15:17 | |
| weshay|ruck | we may save 30-50min | 15:18 |
| mwhahaha | i've got a change to reduce some of that | 15:18 |
| mwhahaha | https://review.opendev.org/#/c/682988/ | 15:18 |
| * weshay|ruck looks | 15:18 | |
| weshay|ruck | mwhahaha, so.. what I need to help re: perf is a more verbose ansible-playbook call for the undercloud | 15:19 |
| cloudnull | weshay|ruck that would be spectacular | 15:19 |
| mwhahaha | weshay|ruck: you don't need verbose, you need the timing outputs right? | 15:20 |
| mwhahaha | because there's not really more verbosity to be had | 15:20 |
| mwhahaha | so we can identify wtf is taking forever consistently? | 15:20 |
| * weshay|ruck will paste for clarity | 15:20 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-quickstart-extras master: Add playbook and role to run tobiko https://review.opendev.org/655423 | 15:21 |
| weshay|ruck | mwhahaha, http://paste.openstack.org/show/778064/ | 15:21 |
| weshay|ruck | there is no way to know how long that handler took in the undercloud atm | 15:21 |
| weshay|ruck | because ansible logging sucks | 15:21 |
| mwhahaha | don't really need that | 15:22 |
| mwhahaha | what you need is the Thursday 19 September 2019 21:12:46 +0000 (0:28:32.293) 0:36:39.510 **** | 15:22 |
| mwhahaha | that's not a verbosity thing | 15:22 |
| weshay|ruck | ya | 15:22 |
| mwhahaha | and that's what i was talking about | 15:22 |
| weshay|ruck | agree | 15:22 |
| mwhahaha | verbosity would be all the changed details (which we don't want) | 15:22 |
| weshay|ruck | ok.. so I'm not groking your patch then yet | 15:22 |
| mwhahaha | i agree we do want the timing | 15:22 |
| weshay|ruck | k.. ya | 15:22 |
| mwhahaha | my patch clears up all the excessive var details | 15:22 |
| mwhahaha | becuase import_var dumps the vars being imported | 15:23 |
| mwhahaha | which clutters logs | 15:23 |
| weshay|ruck | ya.. getting rid of logs we don't need is ++ | 15:23 |
| weshay|ruck | but I also want the real elapsed time | 15:23 |
| weshay|ruck | s/want/need | 15:24 |
| mwhahaha | that's an output plugin thing if i recal | 15:24 |
| weshay|ruck | aye | 15:24 |
| weshay|ruck | we could update a plugin for sure | 15:24 |
| mwhahaha | profile_tasks or whatever | 15:25 |
| weshay|ruck | callback_whitelist = profile_tasks | 15:25 |
| weshay|ruck | https://docs.ansible.com/ansible/latest/plugins/callback/profile_tasks.html | 15:25 |
| *** Vorrtex has quit IRC | 15:25 | |
| mwhahaha | yea that | 15:25 |
| mwhahaha | i don't know where we're setting that in the overcloud | 15:26 |
| mwhahaha | are we hacking that in via ci? | 15:26 |
| weshay|ruck | I'd rather not | 15:26 |
| mwhahaha | no i mean is that what we're doing now | 15:26 |
| mwhahaha | cause i'm not finding a reference to that in tripleo-common or tripleoclient | 15:26 |
| weshay|ruck | you are actively working on doing that now? just to be clear? | 15:26 |
| mwhahaha | i can, i'm just looking | 15:27 |
| mwhahaha | yes we're hacking it in via quickstart atm | 15:27 |
| mwhahaha | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/ansible.cfg | 15:27 |
| * mwhahaha throws things are people | 15:27 | |
| mwhahaha | s/are/at | 15:27 |
| weshay|ruck | 3 years ago | 15:28 |
| weshay|ruck | that is not related to the overcloud / undercloud deployment | 15:29 |
| * weshay|ruck not clear what you are trying to say atm.. | 15:29 | |
| weshay|ruck | we need that set in tripleo-common | 15:30 |
| mwhahaha | right | 15:30 |
| weshay|ruck | it can be in both dude | 15:30 |
| mwhahaha | i think it's getting inherieted | 15:30 |
| mwhahaha | anyway | 15:30 |
| weshay|ruck | doubt it | 15:30 |
| mwhahaha | wonder if we should enable full_skip too | 15:32 |
| mwhahaha | will check that out later | 15:32 |
| mwhahaha | https://docs.ansible.com/ansible/latest/plugins/callback/full_skip.html | 15:32 |
| weshay|ruck | k | 15:33 |
| mwhahaha | sending a tripleo-common patch in a sec | 15:34 |
| *** dpeacock has quit IRC | 15:34 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: Fix the non-HA case where OVN has its own VIP https://review.opendev.org/682968 | 15:34 |
| mwhahaha | so i think what's happening is that we shell out to ansible for image modify which loads https://opendev.org/openstack/ansible-role-tripleo-modify-image/src/branch/master/ansible.cfg | 15:34 |
| mwhahaha | so thats why timings exist for those | 15:34 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Add profile_tasks to ansible runs https://review.opendev.org/683425 | 15:35 |
| *** kopecmartin is now known as kopecmartin|off | 15:35 | |
| *** apetrich has joined #tripleo | 15:37 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: WIP: Add full_skip to default ansible.cfg https://review.opendev.org/683428 | 15:40 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert subprocess to a reusable method https://review.opendev.org/683429 | 15:40 |
| *** dpeacock has joined #tripleo | 15:41 | |
| weshay|ruck | mwhahaha, /me will add the same to tq | 15:42 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Set display_skipped_host to no https://review.opendev.org/683428 | 15:42 |
| mwhahaha | actually full skip is deprecated | 15:42 |
| mwhahaha | there's a way to do it built in now | 15:43 |
| mwhahaha | want to see what it looks like first | 15:43 |
| *** rcernin has quit IRC | 15:44 | |
| mwhahaha | did gerrit just die? | 15:47 |
| mwhahaha | or is it just me | 15:47 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add new parameter options to Octavia service https://review.opendev.org/682641 | 15:47 |
| mwhahaha | weird it's back | 15:47 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Use shell to ensure faster task execution https://review.opendev.org/683277 | 15:48 |
| weshay|ruck | arxcruz|rover, and look at https://review.opendev.org/#/c/683428/ | 15:50 |
| weshay|ruck | arxcruz|rover, I'm going to add the same for tq | 15:50 |
| *** iurygregory has quit IRC | 15:52 | |
| ykarel | weshay|ruck, arxcruz|rover master promoting:- promoter Promoting the container images for dlrn hash 41408527ae88ac42de11ca2818ff405320795a8b on master to current-tripleo | 15:54 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Switching the delete param handler to use rm https://review.opendev.org/683402 | 15:54 |
| cloudnull | arxcruz|rover updated your review based on some feedback - just to keep that moving - sorry. | 15:54 |
| weshay|ruck | ykarel, ? | 15:54 |
| weshay|ruck | ykarel, did you change criteria? | 15:54 |
| ykarel | weshay|ruck, nope | 15:55 |
| *** suuuper has quit IRC | 15:55 | |
| weshay|ruck | ykarel, why is there a promotion? | 15:55 |
| ykarel | weshay|ruck, https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=41408527ae88ac42de11ca2818ff405320795a8b&distro_hash=9e5988125e88f803ba20743be7aa99079dd275f2 | 15:55 |
| ykarel | all required job passed | 15:55 |
| *** cylopez has joined #tripleo | 15:56 | |
| weshay|ruck | really? | 15:56 |
| mwhahaha | pffft don't be shocked :D | 15:56 |
| mwhahaha | we write really awesome code! | 15:56 |
| weshay|ruck | lolz | 15:56 |
| ykarel | weshay|ruck, run fail jobed in check, those were failed at overcloud deploy timeout | 15:56 |
| ykarel | https://review.rdoproject.org/r/#/c/22163/ | 15:57 |
| *** rpittau is now known as rpittau|afk | 15:57 | |
| *** lmiccini has quit IRC | 15:58 | |
| weshay|ruck | ok.. thanks ykarel | 15:59 |
| *** ykarel is now known as ykarel|away | 16:00 | |
| *** jpich has quit IRC | 16:02 | |
| *** fultonj has quit IRC | 16:04 | |
| arxcruz|rover | cloudnull: no problem as far as the promotion pipeline move forward :D | 16:04 |
| arxcruz|rover | I don't want to have nightmares on friday :P | 16:04 |
| arxcruz|rover | brb, upgrade firmware | 16:05 |
| *** two_oes has quit IRC | 16:05 | |
| *** two_oes has joined #tripleo | 16:06 | |
| bogdando | as far as we have here a little optimization party, PTAL https://review.opendev.org/#/c/683100/ | 16:08 |
| bogdando | each saved minute saves a kitten | 16:08 |
| *** Vorrtex has joined #tripleo | 16:09 | |
| *** cylopez has quit IRC | 16:13 | |
| weshay|ruck | bogdando, ah.. that is a sweet way to express that | 16:18 |
| *** pcaruana has quit IRC | 16:19 | |
| bogdando | whatever encourages you folks merge that | 16:19 |
| bogdando | :D | 16:19 |
| bogdando | will fix pep issues after the testing CI job passes | 16:19 |
| bogdando | *completes | 16:20 |
| odyssey4me | bogdando: LGTM other than that pesky pep8 issue | 16:27 |
| bogdando | odyssey4me: thanks! I'll push an update shortly. Don't want to spoil https://review.opendev.org/#/c/683101/ that checks that patch but with caching ON | 16:28 |
| *** holser__ has joined #tripleo | 16:29 | |
| *** onmywaytosunset has joined #tripleo | 16:30 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: remove unused ansible.cfg file https://review.opendev.org/683503 | 16:30 |
| *** jpena is now known as jpena|off | 16:31 | |
| arxcruz|rover | bogdando: would be faster instead of a cat, save one beer | 16:31 |
| arxcruz|rover | for me | 16:31 |
| bogdando | heh | 16:31 |
| *** holser has quit IRC | 16:31 | |
| *** onmywaytosunset has left #tripleo | 16:32 | |
| *** cylopez has joined #tripleo | 16:32 | |
| *** karthiks has quit IRC | 16:35 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: do not log skipped tasks https://review.opendev.org/683530 | 16:35 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert subprocess to a reusable method https://review.opendev.org/683429 | 16:35 |
| *** bogdando has quit IRC | 16:36 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move nested bash script into a real file https://review.opendev.org/683270 | 16:37 |
| *** onmywaytosunset has joined #tripleo | 16:37 | |
| onmywaytosunset | getting permssion denied /etc/ceilometer/gnocchi-resources inside the container | 16:38 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move basic global functions into ansible tasks https://review.opendev.org/683286 | 16:38 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert subprocess to a reusable method https://review.opendev.org/683429 | 16:39 |
| *** marios has quit IRC | 16:41 | |
| *** ykarel|away has quit IRC | 16:41 | |
| *** derekh has quit IRC | 16:42 | |
| *** hberaud is now known as hberaud|gone | 16:43 | |
| *** fultonj has joined #tripleo | 16:45 | |
| *** fultonj has joined #tripleo | 16:46 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ansible master: increase coverage of tripleo-ansible https://review.opendev.org/683587 | 16:48 |
| weshay|ruck | mwhahaha, you cool w/ ^ | 16:48 |
| mwhahaha | not really | 16:48 |
| mwhahaha | but whatever | 16:48 |
| mwhahaha | weshay|ruck: why multinode? | 16:49 |
| mwhahaha | we already run scenario004 | 16:49 |
| mwhahaha | what are we missing | 16:49 |
| weshay|ruck | would help ensure that changes there don't mess w/ perf | 16:49 |
| weshay|ruck | we're adding a lot of stuff | 16:49 |
| weshay|ruck | for some amount of time... seems like a good idea to me | 16:50 |
| mwhahaha | not sure if that's really the solution but ok | 16:50 |
| mwhahaha | we can give it a shot | 16:50 |
| weshay|ruck | ya.. this is not a have to | 16:50 |
| weshay|ruck | but I think it makes sense for now | 16:50 |
| weshay|ruck | the coverage is light | 16:50 |
| weshay|ruck | imho | 16:50 |
| weshay|ruck | cloudnull, thinking it would be a good idea to turn on container-multinode on all tripleo-ansible roles while were in heavy devel | 16:52 |
| weshay|ruck | pita I know.. sorry :) | 16:53 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move globals into main level 0 case https://review.opendev.org/683590 | 16:54 |
| *** cylopez has quit IRC | 16:55 | |
| *** ramishra has quit IRC | 16:55 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: Run Manila in scenario004 without pacemaker https://review.opendev.org/682853 | 16:55 |
| *** udesale has quit IRC | 16:57 | |
| *** gbarros has joined #tripleo | 16:58 | |
| *** poopcat has quit IRC | 16:58 | |
| *** holser__ has quit IRC | 16:58 | |
| *** ykarel|away has joined #tripleo | 16:58 | |
| cloudnull | weshay|ruck ++ | 16:58 |
| cloudnull | one comment but otherwise lgtm | 16:59 |
| *** florianf has quit IRC | 17:00 | |
| *** holser has joined #tripleo | 17:01 | |
| *** mcornea has quit IRC | 17:02 | |
| *** mcornea has joined #tripleo | 17:03 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ansible master: increase coverage of tripleo-ansible https://review.opendev.org/683587 | 17:03 |
| weshay|ruck | cloudnull, good point.. updated | 17:03 |
| *** onmywaytosunset has quit IRC | 17:05 | |
| *** holser has quit IRC | 17:05 | |
| mwhahaha | cloudnull: oh so lookin in the image uploader code, i think we have some excessive session creation | 17:09 |
| mwhahaha | cloudnull: we have a bunch of authenticate calls that don't have session passed with them | 17:09 |
| cloudnull | that does sound about right :) | 17:10 |
| mwhahaha | cloudnull: things like discover_image_tag, filter_images_with_labels, discover_tag_from_inspect, tags_for_image | 17:10 |
| mwhahaha | though it's likely because of the stupid scope stuff | 17:11 |
| cloudnull | ++ | 17:11 |
| mwhahaha | but i don't think we should be creating a sesssion for each one of these calls | 17:11 |
| mwhahaha | seems like we need to seperate the pooling aspect from the token bits | 17:12 |
| tbarron | weshay|ruck: i'm being dense about what CI we are waiting for on 682853, it keeps passing scenario004 and gets +1 vote from zuul | 17:12 |
| cloudnull | the way a lot of that code works is a little odd. its authenticating on the scoped image, which we should be able to track with the sha1 ids, but its a bit all over the place | 17:12 |
| weshay|ruck | tbarron, we can't do a depends on from upstream to third party.. has to work the other way.. so | 17:13 |
| cloudnull | its all using the same auth regardless of the image, so it makes a lot of sense to try and separate those functions, if we can | 17:14 |
| weshay|ruck | tbarron, to see the results upstream we had to get that zuul change merged | 17:14 |
| weshay|ruck | which it now is | 17:14 |
| tbarron | weshay|ruck: ah, that's why the depends-on seemed reversed to me. So where do I look to see the results that we're waiting on? | 17:15 |
| weshay|ruck | tbarron, I rebased the change.. the job is running, you'll see results in https://review.opendev.org/#/c/682853/ | 17:15 |
| weshay|ruck | tbarron, job is running here https://review.rdoproject.org/zuul/status | 17:16 |
| weshay|ruck | filter on 682853 | 17:16 |
| weshay|ruck | https://review.rdoproject.org/zuul/stream/986c0953040b4601ab20b6a92b440e52?logfile=console.log | 17:16 |
| weshay|ruck | hopefully the right container_cli command will be passed | 17:16 |
| tbarron | weshay|ruck: ok, cool, and thanks. I can do the zuul ongoing change thing, just wasn't getting that we're looking at this change set | 17:16 |
| weshay|ruck | should be | 17:16 |
| tbarron | makes sense since you rebased though | 17:17 |
| *** suuuper has joined #tripleo | 17:23 | |
| ykarel|away | mmm Delete param taking 25-40 minutes, really? | 17:25 |
| ykarel|away | can't digest | 17:25 |
| ykarel|away | me looking at https://review.opendev.org/#/c/683402/5 | 17:25 |
| ykarel|away | mwhahaha, arxcruz|rover any link for this ^^ | 17:26 |
| ykarel|away | log link | 17:26 |
| arxcruz|rover | ykarel|away: https://bugs.launchpad.net/tripleo/+bug/1844778 | 17:27 |
| openstack | Launchpad bug 1844778 in tripleo "Overcloud deployment is timeing out on scenario001 ovb" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 17:27 |
| * ykarel|away looks | 17:27 | |
| ykarel|away | arxcruz|rover, you mean 36:15.383 taken by for that delete task? | 17:28 |
| ykarel|away | that time is for tripleo-container-image prepare task i think | 17:28 |
| arxcruz|rover | ykarel|away: don't think so... | 17:29 |
| arxcruz|rover | ykarel|away: did you check the log ? | 17:29 |
| ykarel|away | arxcruz|rover, yes | 17:29 |
| ykarel|away | https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/8e0ca89/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-09-19_22_11_13 | 17:29 |
| arxcruz|rover | ok, first of all, how can you get the link for the time? | 17:30 |
| ykarel|away | so in ^^, Thursday 19 September 2019 22:10:28 +0000 (0:36:15.383) 0:43:02.648 **** | 17:31 |
| ykarel|away | i say, 36:15 is for tripleo-container-image prepare task | 17:31 |
| ykarel|away | not handler | 17:31 |
| ykarel|away | arxcruz|rover, see https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/8e0ca89/logs/ara_oooq_overcloud/ara-report/ for more clearity | 17:32 |
| ykarel|away | sort it based on Duration | 17:32 |
| ykarel|away | and u will see 36:15 is for tripleo-container-image preapare command | 17:32 |
| arxcruz|rover | hmmmm | 17:35 |
| arxcruz|rover | weshay|ruck: ^ | 17:35 |
| weshay|ruck | ya.. just amazing how awful ansible logging is at times | 17:35 |
| *** amoralej is now known as amoralej|off | 17:35 | |
| weshay|ruck | good way to sell tower | 17:36 |
| arxcruz|rover | still, i want to see the logs from the patch | 17:38 |
| arxcruz|rover | how much time the image prepare was before ? | 17:39 |
| *** tosky_ has joined #tripleo | 17:48 | |
| *** holser has joined #tripleo | 17:49 | |
| ykarel|away | arxcruz|rover, befor which patch | 17:51 |
| *** tosky is now known as Guest87259 | 17:51 | |
| *** tosky_ is now known as tosky | 17:51 | |
| *** Guest87259 has quit IRC | 17:52 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move config processing into a single method https://review.opendev.org/683670 | 17:52 |
| *** holser has quit IRC | 17:54 | |
| *** Tengu has quit IRC | 18:07 | |
| *** Tengu has joined #tripleo | 18:07 | |
| *** gbarros has quit IRC | 18:09 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Consolodate mp_puppet_config https://review.opendev.org/683683 | 18:17 |
| *** paramite|clone has joined #tripleo | 18:17 | |
| *** dsneddon has joined #tripleo | 18:22 | |
| *** morazi has quit IRC | 18:26 | |
| *** ricolin has quit IRC | 18:27 | |
| *** ricolin has joined #tripleo | 18:27 | |
| *** dsneddon has quit IRC | 18:27 | |
| *** jistr has quit IRC | 18:28 | |
| *** redrobot has quit IRC | 18:28 | |
| *** elGamal has quit IRC | 18:28 | |
| *** jistr has joined #tripleo | 18:28 | |
| *** elGamal has joined #tripleo | 18:29 | |
| *** dansmith has quit IRC | 18:29 | |
| *** dansmith has joined #tripleo | 18:29 | |
| *** Vorrtex has quit IRC | 18:34 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: Run Manila in scenario004 without pacemaker https://review.opendev.org/682853 | 18:36 |
| *** holser has joined #tripleo | 18:37 | |
| weshay|ruck | cloudnull, didn't EmilienM say he did NOT see resets using quay Resetting dropped connection: quay.io | 18:40 |
| weshay|ruck | https://88c78217ea88b68a79a3-c409a01604f6f3a7ce618899a7a4cd0d.ssl.cf1.rackcdn.com/683212/2/check/tripleo-ci-centos-7-containers-multinode/2f599e9/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 18:40 |
| cloudnull | correct. | 18:40 |
| cloudnull | no resets no retries | 18:40 |
| cloudnull | err , maybe it was no retries, but resets. | 18:40 |
| weshay|ruck | hrm.. there are a bunch in that log | 18:41 |
| * cloudnull doesn't remember. | 18:41 | |
| *** holser has quit IRC | 18:41 | |
| *** poopcat has joined #tripleo | 18:41 | |
| mwhahaha | it was retries | 18:41 |
| mwhahaha | resets happen | 18:41 |
| mwhahaha | not sure why | 18:41 |
| mwhahaha | was wondering if they have connection-close or something | 18:41 |
| cloudnull | looking at that log I dont see any "re-auth" | 18:41 |
| weshay|ruck | ok.. so still worth trying? | 18:42 |
| mwhahaha | i'm looking at the session code and i'm not sure we can move it because auth'd sessions are scoped to a specific image | 18:42 |
| mwhahaha | yes it is | 18:42 |
| mwhahaha | it's way less calls | 18:42 |
| mwhahaha | the problem is that docker.io auth calls are expensive | 18:42 |
| mwhahaha | 2x | 18:42 |
| weshay|ruck | 24min task | 18:42 |
| mwhahaha | 24 no mirroring right? | 18:42 |
| weshay|ruck | not really improving time, but we can still move | 18:42 |
| weshay|ruck | ya.. I don't think it was mirrored yet | 18:43 |
| * weshay|ruck checks cloudnull's patch | 18:43 | |
| weshay|ruck | https://review.opendev.org/#/c/683240/ | 18:43 |
| cloudnull | we need our infra core :) | 18:43 |
| * weshay|ruck just asked in channel | 18:45 | |
| weshay|ruck | cloudnull, saw something on reddit.. re: everyone gets to merge what ever they want for 12 hours once a year | 18:45 |
| weshay|ruck | 12 hour window | 18:45 |
| mwhahaha | sounds like a terrible idea | 18:46 |
| cloudnull | the purge | 18:46 |
| weshay|ruck | the name was funny | 18:46 |
| cloudnull | 's/purge/merge/' | 18:46 |
| weshay|ruck | https://www.reddit.com/r/ProgrammerHumor/comments/d6d947/may_the_merge_begin/ | 18:46 |
| weshay|ruck | ha ha.. | 18:46 |
| weshay|ruck | the sequal... "the rebase" | 18:46 |
| mwhahaha | https://www.youtube.com/watch?v=VU0GYSA1POs | 18:47 |
| weshay|ruck | tru | 18:47 |
| weshay|ruck | that is how you get ants | 18:47 |
| weshay|ruck | the ovb job was quick | 18:49 |
| weshay|ruck | http://logs.rdoproject.org/12/683212/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/1725a7a/logs/undercloud/home/zuul/install-undercloud.log.txt.gz | 18:49 |
| weshay|ruck | 14min | 18:49 |
| weshay|ruck | tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001SUCCESS in 2h 59m 10s | 18:49 |
| weshay|ruck | kind of points back to the infra | 18:49 |
| weshay|ruck | when ovb is 41 min faster than container-multinode | 18:50 |
| *** Vorrtex has joined #tripleo | 18:50 | |
| weshay|ruck | totally need a an infra core | 18:50 |
| mwhahaha | the wide range of times totally points to environmental things | 18:51 |
| weshay|ruck | ya | 18:51 |
| weshay|ruck | \0/ | 18:51 |
| weshay|ruck | too much C02 in the atmosphere for containers to work | 18:52 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move config processing into consolodated methods https://review.opendev.org/683670 | 19:06 |
| *** morazi has joined #tripleo | 19:07 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move config processing into consolodated methods https://review.opendev.org/683670 | 19:10 |
| *** alexmcleod has quit IRC | 19:12 | |
| *** ricolin has quit IRC | 19:16 | |
| *** ykarel|away has quit IRC | 19:17 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Fix container configuration file path https://review.opendev.org/682424 | 19:18 |
| *** lucasagomes has quit IRC | 19:25 | |
| *** dsneddon has joined #tripleo | 19:31 | |
| *** slaweq has quit IRC | 19:32 | |
| weshay|ruck | tbarron, fyi.. got results back | 19:32 |
| tbarron | weshay|ruck: you sound like an oncologist | 19:33 |
| tbarron | weshay|ruck: i'll look and see what's happening with ceph now that wasn't happening before you rebased | 19:34 |
| tbarron | weshay|ruck: i bet running *with* pacemaker would fail with your rebase too :) | 19:34 |
| weshay|ruck | tbarron, please call the hotline at the ci center for confidential results | 19:34 |
| weshay|ruck | there will be survey at the end of the call for quality assurance | 19:34 |
| tbarron | weshay|ruck: that's why i'm confused about why we hold up the normal tripleo fix for dependencies elsewhere even though I'm happy to work on thodse too | 19:35 |
| tbarron | those | 19:35 |
| tbarron | but I'm always confused so don't worry | 19:35 |
| weshay|ruck | tbarron, this one is confusing.. because of pacemaker | 19:35 |
| tbarron | i'm gonna want to backport this fix to stable though, maybe as is? it worked just fine ... | 19:36 |
| weshay|ruck | I think scen01/02 are using podman on both centos and rhel | 19:36 |
| weshay|ruck | tbarron, backport to what exactly? | 19:36 |
| tbarron | all the way to stable/queens, the non pcs path should work even if we didn't support it downstream | 19:37 |
| tbarron | and it never did | 19:37 |
| weshay|ruck | ah | 19:37 |
| tbarron | so that's what i was fixing, that's what the bug and the @#$#@$ nag on irc and the trello card say | 19:38 |
| *** morazi has quit IRC | 19:38 | |
| tbarron | now I understand it's for a higher cause :) | 19:38 |
| weshay|ruck | we are saving lives here | 19:38 |
| tbarron | and am perfectly willing to help get done what needs to be done, but i'm foggy on our goal | 19:38 |
| tbarron | but that doesn't matter, i'm just a soldier | 19:38 |
| tbarron | anyways, i'll look at the new logs, something went wrong deploying ceph cause those pools didn't get created | 19:39 |
| *** suuuper has quit IRC | 19:45 | |
| *** morazi has joined #tripleo | 19:52 | |
| *** morazi has quit IRC | 20:01 | |
| *** morazi has joined #tripleo | 20:02 | |
| *** pbandark has quit IRC | 20:05 | |
| *** pbandark has joined #tripleo | 20:06 | |
| *** artom has quit IRC | 20:10 | |
| cloudnull | mwhahaha any chance you could give https://review.opendev.org/#/c/683277/ a nudge | 20:15 |
| mwhahaha | k | 20:16 |
| openstackgerrit | Martin Mágr proposed openstack/puppet-tripleo master: Add collectd-sensubility configuration https://review.opendev.org/678863 | 20:18 |
| *** two_oes has quit IRC | 20:19 | |
| *** two_oes has joined #tripleo | 20:19 | |
| cloudnull | ty | 20:20 |
| *** Vorrtex has quit IRC | 20:29 | |
| *** jtomasek has quit IRC | 20:32 | |
| *** redrobot has joined #tripleo | 20:34 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: WIP HA: reorder init_bundle and restart_bundle for improved updates https://review.opendev.org/675993 | 20:36 |
| openstackgerrit | Michael Johnson proposed openstack/tripleo-heat-templates master: Added the connection_logging parameter for Octavia https://review.opendev.org/683728 | 20:44 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Cleanup override file https://review.opendev.org/682455 | 21:02 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove tacker container code https://review.opendev.org/682457 | 21:03 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove Trove password constant https://review.opendev.org/682479 | 21:03 |
| *** _mmethot_ has quit IRC | 21:05 | |
| *** mmethot_ has joined #tripleo | 21:05 | |
| *** rfolco has quit IRC | 21:08 | |
| openstackgerrit | Michael Johnson proposed openstack/tripleo-heat-templates master: Add the Octavia log offload parameters https://review.opendev.org/683731 | 21:12 |
| *** gfidente has quit IRC | 21:16 | |
| *** paramite|clone has quit IRC | 21:17 | |
| *** apetrich has quit IRC | 21:19 | |
| *** aakarsh has quit IRC | 21:20 | |
| *** rlandy has quit IRC | 21:25 | |
| *** pierrepr1netti is now known as pierreprinetti | 21:26 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Consolodate file operations https://review.opendev.org/683734 | 21:35 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Remove python executable print https://review.opendev.org/683735 | 21:35 |
| mwhahaha | cloudnull: for blocks like https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo-ceph-common/tasks/main.yml#L21-L33 should we no_log it like we did for https://review.opendev.org/#/c/682988/ | 21:41 |
| mwhahaha | i'm looking at the deploy logs and we have a bunch of extra logs/fake fatals because of those env var lookup things | 21:42 |
| mwhahaha | https://logs.rdoproject.org/25/683425/1/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/0015380/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-09-20_16_16_50 | 21:42 |
| *** rh-jelabarre has quit IRC | 21:43 | |
| cloudnull | I think some of those load blocks can just be removed. | 21:43 |
| mwhahaha | we call these tripleo-ceph things a lot | 21:43 |
| cloudnull | the role template adds that block by default but I suspect some of our roles done have any vars files so there no reason to have it | 21:44 |
| mwhahaha | with https://review.opendev.org/#/c/683428/ they'll stop showing up (at least the skipped things) though it seems excessive | 21:44 |
| *** nkinder has quit IRC | 21:44 | |
| *** raildo has quit IRC | 21:44 | |
| *** rh-jelabarre has joined #tripleo | 21:44 | |
| mwhahaha | https://logs.rdoproject.org/28/683428/2/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/cbe88d0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-09-20_16_32_01 | 21:45 |
| mwhahaha | still a bit chatty | 21:45 |
| openstackgerrit | Merged openstack/tripleo-ansible master: Switching the delete param handler to use rm https://review.opendev.org/683402 | 21:47 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: remove unessisary blocks https://review.opendev.org/683736 | 21:48 |
| cloudnull | ^ mwhahaha quick scan of the roles, those were the two we had that were using that block without any vars , so i remvoed it | 21:49 |
| mwhahaha | i mean we can probably leave it if we ever want them, but include_vars dumps the contents to the script when we run with -vv | 21:49 |
| mwhahaha | would be nice if this was just inherient to ansible | 21:50 |
| mwhahaha | perhaps a module? | 21:50 |
| cloudnull | we might be able to write a vars module | 21:50 |
| cloudnull | but no, ansible doesn't have a built in var file loader, other than for the file `vars/main.yml` | 21:51 |
| mwhahaha | the logic we have for the files seems backwards from other examples i've seen anyway | 21:51 |
| mwhahaha | so maybe just a tripleo_load_os_vars | 21:51 |
| cloudnull | most folks include those types of vars in the playbook, sadly we cant run that at the playbook level, so we're using that task to scan for and load OS specific var files when they're encountered. | 21:54 |
| mwhahaha | yea that's why i was wondering if a module could work | 21:54 |
| * mwhahaha shrugs | 21:54 | |
| cloudnull | not every role in tripleo-ansible has that block , so i think it makes sense to remove it where its not needed. | 21:54 |
| cloudnull | howevere, I can look into making a module . | 21:54 |
| cloudnull | that's something we could probably do? | 21:55 |
| * cloudnull reading https://docs.ansible.com/ansible/latest/plugins/vars.html | 21:55 | |
| mwhahaha | seems like for repeated tasks it would be beneficial | 21:55 |
| cloudnull | ++ | 21:55 |
| *** mcornea has quit IRC | 21:59 | |
| mwhahaha | we seem to be including tripleo-ceph stuff multiple times int eh same step | 22:01 |
| mwhahaha | weird | 22:01 |
| mwhahaha | cloudnull: does the meta/main.yaml do anythhing at run time or is it just a galaxy thing | 22:02 |
| cloudnull | it load dependencies if there are any. | 22:03 |
| cloudnull | in most cases its just a galaxy thing | 22:03 |
| mwhahaha | so we run tripleo-ceph-common repeatedly then | 22:03 |
| mwhahaha | because of teh dependencies | 22:03 |
| cloudnull | probably | 22:03 |
| mwhahaha | i wonder if we need to drop those | 22:03 |
| mwhahaha | https://logs.rdoproject.org/28/683428/2/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/cbe88d0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-09-20_16_32_01 | 22:04 |
| mwhahaha | check all the repeats of tripleo-ceph-common | 22:04 |
| mwhahaha | actually https://logs.rdoproject.org/25/683425/1/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/0015380/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-09-20_16_16_51 is a better example | 22:04 |
| cloudnull | good question for fultonj - maybe we can figure out a way to run once ? | 22:04 |
| mwhahaha | we're including it in our playbook itself | 22:04 |
| mwhahaha | so can we just not have that? | 22:04 |
| * mwhahaha doesn't like dependencies in ansible because of this | 22:05 | |
| cloudnull | if we can control the playbook, we can control the load ordering | 22:05 |
| mwhahaha | we do | 22:06 |
| mwhahaha | cause these are controlled via deploy tasks | 22:06 |
| cloudnull | we can also set run_once on https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo-ceph-common/tasks/main.yml#L35-L49 | 22:07 |
| cloudnull | which will make it a little more quite | 22:07 |
| mwhahaha | that works too | 22:07 |
| mwhahaha | ceph-{work-dir,fetch-dir,common} run repeatedly | 22:08 |
| *** xek has quit IRC | 22:12 | |
| mwhahaha | actually i don't think we want run_once | 22:15 |
| mwhahaha | because that shares it across the deploy | 22:15 |
| mwhahaha | if i'm reading https://docs.ansible.com/ansible/latest/user_guide/playbooks_delegation.html#run-once correctly | 22:15 |
| mwhahaha | run_once is one host, not one instance per host | 22:16 |
| *** stevebaker has joined #tripleo | 22:20 | |
| *** rh-jelabarre has quit IRC | 22:22 | |
| mwhahaha | fultonj: https://bugs.launchpad.net/tripleo/+bug/1844837 | 22:25 |
| openstack | Launchpad bug 1844837 in tripleo "ceph runs the tripleo-ceph-{work-dir,fetch-dir,common} roles more than once in the deployment steps" [High,Triaged] | 22:25 |
| mwhahaha | cloudnull: -^ | 22:25 |
| *** gbarros has joined #tripleo | 22:29 | |
| * mwhahaha gives up for the week | 22:29 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add conditions to ceph roles for dependencies https://review.opendev.org/683739 | 22:30 |
| cloudnull | mwhahaha ^ | 22:31 |
| cloudnull | I think i worked out whats needed, but we'll see in the gate | 22:31 |
| mwhahaha | k | 22:31 |
| mwhahaha | will check on monday when the results might be ready :D | 22:31 |
| cloudnull | ++ | 22:31 |
| * cloudnull was going to break away myself | 22:31 | |
| cloudnull | talk to you monday | 22:32 |
| cloudnull | have a good weekend | 22:32 |
| *** gbarros has quit IRC | 22:40 | |
| *** nkinder has joined #tripleo | 23:01 | |
| *** gbarros has joined #tripleo | 23:08 | |
| *** nkinder has quit IRC | 23:26 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: Use shell to ensure faster task execution https://review.opendev.org/683277 | 23:27 |
| *** tosky has quit IRC | 23:28 | |
| *** dsneddon has quit IRC | 23:29 | |
| *** munimeha1 has quit IRC | 23:35 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Disable keystone token_flush by default https://review.opendev.org/682512 | 23:56 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Don't run keystone_cron container if fernet token is used https://review.opendev.org/682265 | 23:56 |
| *** rfolco has joined #tripleo | 23:57 | |
| *** gbarros has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!