| *** holser has joined #tripleo | 00:09 | |
| *** bnemec has joined #tripleo | 00:16 | |
| *** pkopec has quit IRC | 00:20 | |
| *** holser has quit IRC | 00:23 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 00:24 |
|---|---|---|
| *** bnemec has quit IRC | 00:35 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 00:48 |
| *** cdearborn has quit IRC | 00:58 | |
| *** spsurya has joined #tripleo | 01:00 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Revert "Point InternalTLSVncCAFile to /etc/ipa/ca.crt" https://review.opendev.org/677549 | 01:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/stein: Revert "Point InternalTLSVncCAFile to /etc/ipa/ca.crt" https://review.opendev.org/677548 | 01:03 |
| *** rlandy|ruck has quit IRC | 01:06 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Revert "Point InternalTLSVncCAFile to /etc/ipa/ca.crt" https://review.opendev.org/677550 | 01:11 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Add *_domain_name in authtoken configuration in Octavia https://review.opendev.org/677587 | 01:20 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Add *_domain_name in authtoken configuration in Sahara https://review.opendev.org/677588 | 01:28 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 01:29 |
| openstackgerrit | Merged openstack/diskimage-builder master: dracut-regenerate: catch failures and exit code https://review.opendev.org/676032 | 01:48 |
| *** mschuppert has quit IRC | 01:53 | |
| openstackgerrit | Merged openstack/diskimage-builder master: block-device-efi : expand disk size calculation https://review.opendev.org/676354 | 02:30 |
| *** rh-jelabarre has quit IRC | 02:37 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add "rhel_containers" variable to skip containers for RHEL https://review.opendev.org/676497 | 02:56 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Define var undercloud_enable_nova https://review.opendev.org/664168 | 02:56 |
| *** cjloader has joined #tripleo | 03:09 | |
| openstackgerrit | Merged openstack/python-tripleoclient stable/stein: Use reset to fix cmdline https://review.opendev.org/677569 | 03:33 |
| *** psachin has joined #tripleo | 03:34 | |
| *** janki has joined #tripleo | 03:41 | |
| *** surpatil has joined #tripleo | 03:50 | |
| *** gkadam has joined #tripleo | 03:54 | |
| *** gkadam has quit IRC | 03:54 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Use the special user role 'service' as service token role https://review.opendev.org/674516 | 04:05 |
| *** dsneddon has quit IRC | 04:15 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Use the special user role 'service' as service token role https://review.opendev.org/674516 | 04:16 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Strictly require service token roles https://review.opendev.org/677599 | 04:17 |
| *** dsneddon has joined #tripleo | 04:18 | |
| *** skramaja has joined #tripleo | 04:22 | |
| *** dsneddon has quit IRC | 04:24 | |
| *** jaosorior has quit IRC | 04:34 | |
| *** ade_lee has quit IRC | 04:35 | |
| *** ade_lee has joined #tripleo | 04:37 | |
| *** ramishra has joined #tripleo | 04:41 | |
| *** dsneddon has joined #tripleo | 04:42 | |
| *** soniya29 has joined #tripleo | 04:44 | |
| *** dsneddon has quit IRC | 04:46 | |
| *** ykarel|away has joined #tripleo | 04:48 | |
| *** ratailor has joined #tripleo | 05:02 | |
| *** shyamb has joined #tripleo | 05:06 | |
| *** kopecmartin|off is now known as kopecmartin | 05:08 | |
| *** udesale has joined #tripleo | 05:10 | |
| *** udesale has quit IRC | 05:14 | |
| *** raukadah is now known as chkumar|rover | 05:17 | |
| *** dsneddon has joined #tripleo | 05:22 | |
| *** soniya29 has quit IRC | 05:24 | |
| *** waleedm has joined #tripleo | 05:25 | |
| Tengu | hello there :) | 05:25 |
| openstackgerrit | Merged openstack/tripleo-common master: Close the http sessions of registry on image prepare https://review.opendev.org/676387 | 05:27 |
| *** dmacpher has quit IRC | 05:31 | |
| *** dmacpher_ has joined #tripleo | 05:31 | |
| *** ykarel|away is now known as ykarel | 05:35 | |
| *** holser has joined #tripleo | 05:38 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-common stable/stein: Close the http sessions of registry on image prepare https://review.opendev.org/677609 | 05:43 |
| *** yprokule has joined #tripleo | 05:44 | |
| *** jaosorior has joined #tripleo | 05:49 | |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/python-tripleoclient stable/stein: Run Validations with ThreadPoolExecutor https://review.opendev.org/677170 | 06:02 |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Use tripleo-validations-package role instead of puppet https://review.opendev.org/677429 | 06:06 |
| *** florianf has joined #tripleo | 06:06 | |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: Add Molecule tests for check-network-gateway https://review.opendev.org/672745 | 06:07 |
| *** hjensas has quit IRC | 06:12 | |
| *** sanjayu__ has joined #tripleo | 06:13 | |
| *** mschuppert has joined #tripleo | 06:18 | |
| *** jaosorior has quit IRC | 06:21 | |
| *** dsneddon has quit IRC | 06:24 | |
| *** dciabrin has joined #tripleo | 06:30 | |
| *** holser has quit IRC | 06:32 | |
| *** nawar has joined #tripleo | 06:32 | |
| nawar | morning | 06:33 |
| nawar | is there a problem with the docs on openstack site | 06:33 |
| ramishra | nawar: It seems I can access those ex. https://docs.openstack.org/tripleo-docs/latest/install/index.html, what're you looking for? | 06:35 |
| nawar | ramishra everything is deprecated and the installation of undercould is gone | 06:37 |
| ramishra | nawar: I think it's due to https://github.com/openstack/tripleo-docs/commit/c6918e5da60a26ca9e64855778efe128f2ce6de2 | 06:46 |
| ramishra | moved to deploy guide.. I don't see the deploy guide published yet though | 06:46 |
| *** waleedm_ has joined #tripleo | 06:51 | |
| *** waleedm__ has joined #tripleo | 06:52 | |
| *** mkisielewski has joined #tripleo | 06:53 | |
| *** waleedm has quit IRC | 06:55 | |
| *** dsneddon has joined #tripleo | 06:56 | |
| *** waleedm_ has quit IRC | 06:56 | |
| *** waleedm has joined #tripleo | 06:59 | |
| *** waleedm_ has joined #tripleo | 07:00 | |
| *** shyamb has quit IRC | 07:01 | |
| *** waleedm__ has quit IRC | 07:02 | |
| *** jaosorior has joined #tripleo | 07:04 | |
| *** waleedm has quit IRC | 07:05 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-docs master: Enhancement to cell v2 doc with split for stein/train https://review.opendev.org/672744 | 07:07 |
| *** jpich has joined #tripleo | 07:12 | |
| *** rcernin has quit IRC | 07:14 | |
| ramishra | nawar: https://0b2142c78b3d718d4667-da9fdb6441bd81af0c9914e787df88dd.ssl.cf5.rackcdn.com/676720/3/check/openstack-tox-docs/4c88b1c/html/install/index.html is the new install guide picked from a doc check job, if need it in the time being | 07:14 |
| *** dsneddon has quit IRC | 07:17 | |
| *** shyamb has joined #tripleo | 07:20 | |
| *** udesale has joined #tripleo | 07:24 | |
| *** amoralej|off is now known as amoralej | 07:25 | |
| skramaja | EmilienM slagle could you take a look at https://review.opendev.org/#/c/667792/? | 07:27 |
| *** xek has joined #tripleo | 07:28 | |
| *** triple-oh-noob has joined #tripleo | 07:34 | |
| triple-oh-noob | Hi. Um, has someone moved the openstack tripleo installer docs recently? | 07:34 |
| triple-oh-noob | getting 404s for a few pages I was on only yesterday | 07:35 |
| triple-oh-noob | https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html#tuning-ceph-osd-cpu-and-memory | 07:35 |
| triple-oh-noob | eg | 07:35 |
| ramishra | triple-oh-noob: https://0b2142c78b3d718d4667-da9fdb6441bd81af0c9914e787df88dd.ssl.cf5.rackcdn.com/676720/3/check/openstack-tox-docs/4c88b1c/html/install/advanced_deployment/ceph_config.html#tuning-ceph-osd-cpu-and-memory, still in the process of being published.. | 07:39 |
| *** rpittau|afk is now known as rpittau | 07:40 | |
| Tengu | hmmm.... does CI have some issue? https://e1f254eb8f81e7924f24-889c896421b71b87d6e141679c8ef033.ssl.cf5.rackcdn.com/672745/12/check/tripleo-ci-centos-7-containers-multinode/b195cd9/job-output.txt "/etc/ci/mirror_info.sh: line 54: NODEPOOLMIRROR_HOST: unbound variable" | 07:40 |
| Tengu | (apparently the new logging infra doesn't provide the direct link for lines anymore ;_;) | 07:40 |
| triple-oh-noob | That was just one example page. | 07:40 |
| Tengu | chkumar|rover: any idea for that "/etc/ci/mirror_info.sh: line 54: NODEPOOLMIRROR_HOST: unbound variable" ? | 07:41 |
| ramishra | triple-oh-noob: I pasted the full install guide earlier for another query | 07:41 |
| ramishra | here it goes again https://0b2142c78b3d718d4667-da9fdb6441bd81af0c9914e787df88dd.ssl.cf5.rackcdn.com/676720/3/check/openstack-tox-docs/4c88b1c/html/install/index.html | 07:41 |
| Tengu | meh... same issue there: https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_45/672745/12/check/tripleo-ci-centos-7-undercloud-containers/eae9aa1/job-output.txt | 07:42 |
| ykarel | ramishra, yup https://review.opendev.org/#/c/677029/ moved the doc to deploy-guide and seems publish-deploy-guide job is not running, which resulting in docs not available | 07:42 |
| ramishra | There were a few patches for all the reorganization of the docs and the publishing takes a little time.. | 07:42 |
| ykarel | ramishra, let me fire a patch to run that job and see if it does something | 07:43 |
| ramishra | ykarel: the job is not running? what's the frequency? | 07:43 |
| ykarel | ramishra, we need to include that job in project config of tripleo-docs project | 07:43 |
| ramishra | ykarel: Isn't it already there? | 07:44 |
| ykarel | ramishra, i don't see it https://github.com/openstack/tripleo-docs/blob/master/.zuul.yaml | 07:44 |
| *** dsneddon has joined #tripleo | 07:45 | |
| ramishra | ykarel: ok, I thought it has some frequency and takes some time..I could be wrong though | 07:45 |
| *** jpena|off is now known as jpena | 07:47 | |
| openstackgerrit | yatin proposed openstack/tripleo-docs master: Run deploy guide jobs https://review.opendev.org/677661 | 07:48 |
| ykarel | ramishra, le't see now ^^ | 07:49 |
| triple-oh-noob | ramishra: Many thanks. Was disconnected last night otherwise i'd have had a look over the chat :-) | 07:49 |
| *** bhagyashris has joined #tripleo | 07:50 | |
| *** jtomasek has joined #tripleo | 07:51 | |
| *** shyamb has quit IRC | 07:54 | |
| *** zbr is now known as zbr|ooo | 07:56 | |
| *** jaosorior has quit IRC | 08:00 | |
| *** lucasagomes has joined #tripleo | 08:01 | |
| *** ksambor has quit IRC | 08:04 | |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/python-tripleoclient stable/stein: Run Validations with ThreadPoolExecutor https://review.opendev.org/677170 | 08:04 |
| *** ksambor has joined #tripleo | 08:06 | |
| *** cylopez has joined #tripleo | 08:09 | |
| *** jtomasek has quit IRC | 08:09 | |
| *** dtantsur|afk is now known as dtantsur | 08:11 | |
| *** pkopec has joined #tripleo | 08:12 | |
| Tengu | ksambor: sorry for that -^ - trying to find the right line(s) for the doc requirement.... | 08:14 |
| *** jtomasek has joined #tripleo | 08:14 | |
| *** gfidente|afk is now known as gfidente | 08:15 | |
| *** ykarel is now known as ykarel|lunch | 08:15 | |
| openstackgerrit | yatin proposed openstack/tripleo-docs master: Run deploy guide jobs https://review.opendev.org/677661 | 08:15 |
| chkumar|rover | Tengu: checking | 08:16 |
| ramishra | ykarel: I think promotion is part of publish-openstack-docs-pti https://github.com/openstack/openstack-zuul-jobs/blob/master/zuul.d/project-templates.yaml#L103 | 08:16 |
| ramishra | https://github.com/openstack/project-config/blob/master/zuul.d/jobs.yaml#L303 | 08:17 |
| ramishra | there have been some changes around that recently, so may be something is broken | 08:17 |
| ksambor | Tengu if you will put sphinx!=1.6.6,!=1.6.7,!=2.1.0,>=1.6.2; python_version>='3.4' # BSD like is in tripleo-common it should work | 08:19 |
| *** holser has joined #tripleo | 08:19 | |
| Tengu | ksambor: it was complaining about the 2.1.0 in fact :) | 08:25 |
| chkumar|rover | Tengu: https://review.opendev.org/#/c/677669/ | 08:26 |
| chkumar|rover | will fix the issue | 08:26 |
| Tengu | chkumar|rover: ah, great! | 08:26 |
| *** suuuper has joined #tripleo | 08:27 | |
| *** tkajinam has quit IRC | 08:29 | |
| *** hjensas has joined #tripleo | 08:30 | |
| *** shyamb has joined #tripleo | 08:32 | |
| triple-oh-noob | If I had deployed previously with masquerade = true for the undercloud, can I "undo" this option simply? | 08:36 |
| *** shyam89 has joined #tripleo | 08:45 | |
| *** shyamb has quit IRC | 08:46 | |
| *** pierreprinetti has joined #tripleo | 08:53 | |
| *** gkadam has joined #tripleo | 08:54 | |
| Tengu | wow, that was a fast merge, chkumar|rover :). thanks! | 08:56 |
| *** gkadam is now known as gkadam-brb | 08:57 | |
| shyam89 | Hi | 08:58 |
| shyam89 | We are doing minor updat on osp13 cloud | 08:58 |
| shyam89 | Step - "openstack overcloud update run --nodes Controller" has failed | 08:58 |
| shyam89 | no ansible.log found in /var/lib/mistral/*/ | 08:59 |
| shyam89 | What other log files/debugging steps we should start with? | 08:59 |
| *** gkadam-brb is now known as gkadam | 09:00 | |
| shyam89 | Tengu: Any thoughts ^^^ | 09:04 |
| *** surpatil has quit IRC | 09:09 | |
| *** jtomasek has quit IRC | 09:11 | |
| chkumar|rover | Tengu: it broke everywhere so fast :-) | 09:17 |
| *** jaosorior has joined #tripleo | 09:19 | |
| ykarel|lunch | ramishra, no, publish-deploy-guide will publish deploy-guide | 09:25 |
| *** ykarel|lunch is now known as ykarel | 09:25 | |
| ykarel | ramishra, https://review.opendev.org/#/c/677661 is green now, once this merges, guide should be published with job in post pipeline | 09:25 |
| *** derekh has joined #tripleo | 09:26 | |
| ramishra | ykarel: I don't know..., AFAIU publish-deploy-guide is an old job before PTI and the cotents changed/missing are part of the install guide https://0b2142c78b3d718d4667-da9fdb6441bd81af0c9914e787df88dd.ssl.cf5.rackcdn.com/676720/3/check/openstack-tox-docs/4c88b1c/html/install/index.html | 09:29 |
| ykarel | ramishra, okk then need to check, ramishra is https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_61/677661/2/check/build-openstack-deploy-guide/7c544e1/docs/ not complete? | 09:31 |
| ramishra | ykarel: I think it's incomplete... | 09:33 |
| ykarel | ramishra, then it means migration is incomplete, but anyway job need to run and publish | 09:37 |
| ykarel | ramishra, i just checked publish-deploy-guide only pushes deploy-guide | 09:37 |
| ykarel | some projects are running it http://zuul.openstack.org/builds?job_name=publish-deploy-guide | 09:37 |
| ykarel | and from logs and corresponding playbooks it's publishing to project-deploy-guide | 09:38 |
| ykarel | https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L858-L866 | 09:39 |
| ramishra | ykarel: As I mentioned publishing docs.openstack.org happens in the promote pipeline, but I'm not too famililar with PTI.. about the projects using publish-deploy-guide, may be those guys have not migrated to PTI.. good to check with infra... | 09:40 |
| ramishra | better to check with infra | 09:40 |
| *** pierreprinetti has quit IRC | 09:40 | |
| ykarel | ramishra, i checked again and publish-openstack-docs-pti is publishing only docs, but will confirm from infra | 09:47 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/python-tripleoclient stable/stein: Suppress output for ssh-keygen https://review.opendev.org/677577 | 09:52 |
| ykarel | chkumar|rover, is scenario001 failure known | 10:02 |
| ykarel | chkumar|rover, http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario001-standalone&branch=master | 10:05 |
| ykarel | chkumar|rover, caused with https://review.opendev.org/#/c/677427/1 | 10:05 |
| *** shyam89 has quit IRC | 10:05 | |
| ykarel | used here, https://opendev.org/openstack/puppet-tripleo/src/branch/master/manifests/profile/base/ceilometer.pp#L157 needs cleanup | 10:10 |
| ykarel | omg it's not just scenario001, other scenarios are also affected | 10:11 |
| ykarel | reporting bug | 10:12 |
| *** ksambor is now known as ksambor|lunch | 10:13 | |
| *** rpittau is now known as rpittau|bbl | 10:14 | |
| ykarel | chkumar|rover, https://bugs.launchpad.net/tripleo/+bug/1840901 | 10:16 |
| openstack | Launchpad bug 1840901 in tripleo "Jobs failing with:- Could not find class ::ceilometer::dispatcher::gnocchi" [Undecided,New] | 10:16 |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/puppet-tripleo master: Remove deprecated ceilometer::dispatcher::gnocchi https://review.opendev.org/677690 | 10:23 |
| *** sshnaidm|afk is now known as sshnaidm | 10:23 | |
| *** shyamb has joined #tripleo | 10:30 | |
| *** triple-oh-noob has quit IRC | 10:31 | |
| *** bhagyashris has quit IRC | 10:31 | |
| *** gfidente has quit IRC | 10:35 | |
| *** gfidente has joined #tripleo | 10:41 | |
| *** ksambor|lunch is now known as ksambor | 10:46 | |
| *** jchhatbar has joined #tripleo | 10:47 | |
| *** janki has quit IRC | 10:49 | |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: [DNM] - Testing ceph_dashboard haproxy endpoint config https://review.opendev.org/677699 | 11:02 |
| *** owalsh is now known as owalsh|away | 11:08 | |
| *** florianf has quit IRC | 11:11 | |
| *** sanjayu__ has quit IRC | 11:17 | |
| *** shyamb has quit IRC | 11:20 | |
| *** saneax has joined #tripleo | 11:20 | |
| *** raildo has joined #tripleo | 11:21 | |
| *** mcornea has joined #tripleo | 11:24 | |
| *** florianf has joined #tripleo | 11:24 | |
| *** paramite has joined #tripleo | 11:25 | |
| *** udesale has quit IRC | 11:26 | |
| *** udesale has joined #tripleo | 11:27 | |
| *** suuuper has quit IRC | 11:30 | |
| *** holser has quit IRC | 11:32 | |
| *** holser__ has joined #tripleo | 11:32 | |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates master: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677703 | 11:33 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates stable/stein: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677704 | 11:36 |
| *** rh-jelabarre has joined #tripleo | 11:37 | |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates stable/stein: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677704 | 11:38 |
| *** morazi has joined #tripleo | 11:38 | |
| *** jpena is now known as jpena|lunch | 11:39 | |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates master: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677703 | 11:42 |
| *** shyamb has joined #tripleo | 11:42 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Allow combining system_upgrade_prepare and system_upgrade_run into system_upgrade https://review.opendev.org/676187 | 11:42 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-common master: Change ceph dashboard service name to meet puppet requirements https://review.opendev.org/677705 | 11:42 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_grafana composable service https://review.opendev.org/674556 | 11:43 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates stable/stein: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677704 | 11:43 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates stable/stein: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677704 | 11:44 |
| *** florianf has quit IRC | 11:45 | |
| openstackgerrit | Francesco Pantano proposed openstack/puppet-tripleo master: Replace ceph_grafana-server with ceph_grafana https://review.opendev.org/677708 | 11:47 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: [DNM] - Testing ceph_dashboard haproxy endpoint config https://review.opendev.org/677699 | 11:48 |
| *** rlandy has joined #tripleo | 11:50 | |
| *** rlandy is now known as rlandy|ruck | 11:50 | |
| ykarel | chkumar|rover, Playbook run of multinode-standalone.yml passed successfully for ceilometer | 11:51 |
| chkumar|rover | ykarel: awesome | 11:51 |
| *** florianf has joined #tripleo | 11:51 | |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates master: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677703 | 11:55 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates master: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677703 | 11:58 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 11:58 | |
| *** nawar has left #tripleo | 11:59 | |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates stable/stein: Use default value for NovaLiveMigrationWaitForVIFPlug https://review.opendev.org/677704 | 11:59 |
| *** udesale has quit IRC | 12:02 | |
| *** ekultails has joined #tripleo | 12:10 | |
| *** rpittau|bbl is now known as rpittau | 12:10 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-ansible master: WIP: Create a role for OvS-DPDK host configuration https://review.opendev.org/677712 | 12:15 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 12:24 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 12:26 |
| *** udesale has joined #tripleo | 12:32 | |
| *** rfolco has quit IRC | 12:33 | |
| *** pbandark has joined #tripleo | 12:34 | |
| *** shyam89 has joined #tripleo | 12:37 | |
| *** shyamb has quit IRC | 12:37 | |
| *** jchhatba_ has joined #tripleo | 12:39 | |
| *** rfolco has joined #tripleo | 12:39 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| *** gkadam has quit IRC | 12:42 | |
| *** jchhatbar has quit IRC | 12:42 | |
| *** jchhatba_ has quit IRC | 12:43 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 12:43 | |
| *** amoralej is now known as amoralej|lunch | 12:50 | |
| *** shyam89 has quit IRC | 12:55 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 12:58 |
| *** pierreprinetti has joined #tripleo | 13:04 | |
| *** cdearborn has joined #tripleo | 13:05 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-common master: Add filter plugins path to the ansible.cfg https://review.opendev.org/677726 | 13:17 |
| *** tesseract has joined #tripleo | 13:22 | |
| *** tesseract has quit IRC | 13:22 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-docs master: Enhancement to cell v2 doc with split for stein/train https://review.opendev.org/672744 | 13:22 |
| chkumar|rover | EmilienM: mwhahaha https://review.opendev.org/#/c/677690/ needs +2 and +w on this to clear check queue | 13:24 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Move OvS-DPDK deployment to ansible role https://review.opendev.org/677728 | 13:30 |
| *** bfournie has joined #tripleo | 13:31 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Move OvS-DPDK deployment to ansible role https://review.opendev.org/677728 | 13:31 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-ansible master: WIP: Create a role for OvS-DPDK host configuration https://review.opendev.org/677712 | 13:32 |
| *** ratailor has quit IRC | 13:36 | |
| *** hjensas has quit IRC | 13:37 | |
| *** Goneri has joined #tripleo | 13:42 | |
| ykarel | EmilienM, mwhahaha can u check https://review.opendev.org/#/c/677661/ | 13:44 |
| ykarel | deploy-guide | 13:44 |
| mwhahaha | thx | 13:44 |
| ade_lee | bandini, hey | 13:46 |
| *** zbr|ooo is now known as zbr | 13:47 | |
| *** udesale has quit IRC | 13:47 | |
| *** ykarel is now known as ykarel|afk | 13:53 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Run deploy guide jobs https://review.opendev.org/677661 | 13:53 |
| *** bnemec has joined #tripleo | 13:55 | |
| bandini | mwhahaha: https://review.opendev.org/#/c/677405/ ok to merge? | 13:57 |
| mwhahaha | for you? maybe | 13:57 |
| bandini | eheheh | 13:57 |
| Tengu | wooohooo! CI back on track - care to make that one merge? https://review.opendev.org/#/c/672745/ it's locking a bunch of other changes already approved :) | 13:59 |
| cloudnull | tripleo-transformation meeting time - sshnaidm cloudnull mnaser ekultails owalsh mwhahaha Tengu ykarel sshnaidm Vorrtex | 14:00 |
| cloudnull | #startmeeting tripleo | 14:00 |
| openstack | Meeting started Wed Aug 21 14:00:18 2019 UTC and is due to finish in 60 minutes. The chair is cloudnull. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| cloudnull | #topic rollcall | 14:00 |
| *** openstack changes topic to "rollcall (Meeting topic: tripleo)" | 14:00 | |
| cloudnull | o/ | 14:00 |
| Tengu | «o/ | 14:00 |
| ekultails | o/ | 14:00 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Adapt os_tempest in FS001 https://review.opendev.org/673400 | 14:01 |
| *** amoralej|lunch is now known as amoralej | 14:01 | |
| cloudnull | as a reminder, our etherpad can be found here | 14:01 |
| mwhahaha | hi2u | 14:01 |
| cloudnull | #link https://etherpad.openstack.org/p/tripleo-ansible-agenda | 14:01 |
| *** ykarel|afk has quit IRC | 14:02 | |
| cloudnull | please update the meeting details if you want to talk about something specifically | 14:02 |
| cloudnull | ok so lets just jump into it. | 14:05 |
| cloudnull | #topic recap | 14:06 |
| *** openstack changes topic to "recap (Meeting topic: tripleo)" | 14:06 | |
| cloudnull | last week mnaser got started on the ansible-sig | 14:06 |
| mnaser | yes, im emailing something out soon to setup sometime we can start meeting at :) | 14:06 |
| cloudnull | there's a new IRC channel for folks to join if they're interested. | 14:06 |
| cloudnull | #openstack-ansible-sig | 14:07 |
| cloudnull | mnaser are there pending reviews that folks should be aware of? | 14:07 |
| mnaser | uh i need to unlazy and clean up the patch to add the ssh connection plugin under 'sig' | 14:08 |
| mnaser | i will get around that soon i hope | 14:08 |
| cloudnull | on that same note - https://review.opendev.org/#/c/676421 | 14:08 |
| mnaser | (also i'll be at ansiblefest) | 14:08 |
| cloudnull | oh cool! that'll be fun :) | 14:08 |
| cloudnull | owalsh|away ekultails mnaser and I worked on getting the connection plugin into a stand-alone repo, that's done and should be imported into the openstack namespace just as soon as the sig bits get ironed out. | 14:09 |
| cloudnull | anything else folks want to talk about with regard to some of the things we covered here/last week ? | 14:10 |
| cloudnull | ok moving on. | 14:12 |
| cloudnull | #topic Open Discussion | 14:12 |
| *** openstack changes topic to "Open Discussion (Meeting topic: tripleo)" | 14:12 | |
| cloudnull | we'll start with | 14:13 |
| cloudnull | TripleO-Ansible roadmap seeking feedback - https://etherpad.openstack.org/p/tripleo-ansible-roadmap | 14:13 |
| cloudnull | ^ ekultails ? | 14:13 |
| *** aakarsh|2 has joined #tripleo | 14:13 | |
| *** rfolco is now known as not_rlandy | 14:13 | |
| *** not_rlandy is now known as folco | 14:14 | |
| ekultails | That is something cloudnull and I were talking about last week. The Etherpad explains the general direction we're looking to take with TripleO-Ansible and TripleO in Ansible. | 14:14 |
| *** folco is now known as rfolco | 14:14 | |
| *** gregwork has quit IRC | 14:14 | |
| cloudnull | i believe this also ties into the spec from EmilienM -https://review.opendev.org/#/c/671563/ | 14:15 |
| EmilienM | this one is on hold for now | 14:15 |
| EmilienM | (I haven't convinced myself about costs vs benefits of this one) | 14:15 |
| *** cdearborn has quit IRC | 14:15 | |
| *** mgagne has quit IRC | 14:16 | |
| *** csatari has quit IRC | 14:16 | |
| *** aakarsh has quit IRC | 14:16 | |
| *** irclogbot_1 has quit IRC | 14:17 | |
| cloudnull | EmilienM anything we can help with in terms of ironing that out ? | 14:17 |
| *** mgagne has joined #tripleo | 14:17 | |
| *** cdearborn has joined #tripleo | 14:18 | |
| EmilienM | not that I can think now | 14:18 |
| cloudnull | ok | 14:18 |
| *** gregwork has joined #tripleo | 14:19 | |
| *** irclogbot_1 has joined #tripleo | 14:19 | |
| *** csatari has joined #tripleo | 14:19 | |
| cloudnull | if folks can add to etherpad ekultails started it'd be very helpful in general. | 14:19 |
| cloudnull | last of the tripleo-common role import work is ready for review - https://review.opendev.org/#/q/topic:hieraroles+(status:open) | 14:19 |
| cloudnull | if folks can review that one it'd be greatly appreciated. | 14:20 |
| cloudnull | Tripleo-Ansible also a few open-reviews that are now passing - https://review.opendev.org/#/q/project:%255Eopenstack/tripleo-ansible+status:open+label:verified%253D%252B1%252Cuser%253Dzuul - if folks have time to review these, that would also be greatly appreciated. | 14:21 |
| cloudnull | is there anything else folks want to talk about while we're here? | 14:21 |
| *** mkisielewski has quit IRC | 14:25 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-ci master: DMN: testing for periodic jobs https://review.opendev.org/629157 | 14:26 |
| cloudnull | ok well if there's nothing else, I think we can call it. | 14:26 |
| cloudnull | thanks everyone! | 14:26 |
| cloudnull | #endmeeting | 14:26 |
| *** openstack changes topic to "CI Status: GREENish RDOCloud Status: MEHish | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs/latest/" | 14:26 | |
| openstack | Meeting ended Wed Aug 21 14:26:56 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:26 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-08-21-14.00.html | 14:26 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-08-21-14.00.txt | 14:27 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-08-21-14.00.log.html | 14:27 |
| *** ykarel|afk has joined #tripleo | 14:27 | |
| dmsimard | cloudnull: I have two nodes for the standalone job, one from rax and one from ovh | 14:28 |
| cloudnull | dmsimard cool! | 14:28 |
| *** ykarel|afk is now known as ykarel | 14:28 | |
| cloudnull | have you been able to dig into the failures ? | 14:28 |
| ykarel | ramishra, deploy-guide published, https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/ | 14:29 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 14:29 | |
| ykarel | ramishra, can u share what u find missing or report bug so it can be fixed | 14:29 |
| *** pierreprinetti has quit IRC | 14:30 | |
| openstackgerrit | Takashi Kajinami proposed openstack/python-tripleoclient master: DNM: just to show possible improvement for the parent patch https://review.opendev.org/677741 | 14:31 |
| dmsimard | cloudnull: I have not, I'll have time a bit later but I've added your keys in the meantime: root@162.242.237.97 and root@158.69.67.231 | 14:31 |
| dmsimard | s/keys/key/ | 14:32 |
| *** waleedm_ has quit IRC | 14:32 | |
| ramishra | ykarel: I would have liked us to use htaccess redirects from install guide to deploy guide for users to still be able to access the old pages, but yeah I'll check later | 14:32 |
| cloudnull | dmsimard thanks! | 14:34 |
| cloudnull | will take a look shortly :) | 14:34 |
| ykarel | ramishra, ack | 14:34 |
| *** mcornea has quit IRC | 14:36 | |
| *** mcornea has joined #tripleo | 14:37 | |
| *** chkumar|rover is now known as raukadah | 14:40 | |
| *** jaosorior has quit IRC | 14:41 | |
| *** Vorrtex has joined #tripleo | 14:41 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 14:45 | |
| *** cfontain_ has joined #tripleo | 14:46 | |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/stein: Revert "Reduce the usage of "podman inspect" command" https://review.opendev.org/677754 | 14:48 |
| *** ratailor has joined #tripleo | 14:49 | |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Revert "Fix mismatching fixed vs unique container names" https://review.opendev.org/677756 | 14:53 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Revert "Fix mismatching fixed vs unique container names" https://review.opendev.org/677756 | 14:53 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_grafana composable service https://review.opendev.org/674556 | 14:54 |
| *** PagliaccisCloud has joined #tripleo | 14:55 | |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Revert "Reduce the usage of "podman inspect" command" https://review.opendev.org/677757 | 14:57 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Revert "Optimize container CLI for getting unique names" https://review.opendev.org/677758 | 14:58 |
| *** pierreprinetti has joined #tripleo | 14:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Revert "Optimize container CLI for getting unique names" https://review.opendev.org/677758 | 14:59 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Revert "Optimize container CLI for getting unique names" https://review.opendev.org/677758 | 14:59 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Revert "Reduce the usage of "podman inspect" command" https://review.opendev.org/677757 | 14:59 |
| *** yprokule has quit IRC | 15:03 | |
| *** jtomasek has joined #tripleo | 15:06 | |
| dmsimard | cloudnull: again rsyslog failing -- can we nuke that image or something ? | 15:07 |
| dmsimard | weshay, rlandy|ruck: ^ | 15:07 |
| dmsimard | https://18e51e15a34f17ffbc81-ffc80d196410a18186442d9badd30b78.ssl.cf2.rackcdn.com/674919/29/check/tripleo-ci-centos-7-standalone/79c7da9/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 15:07 |
| weshay | cloudnull dmsimard how about we put something in that randomizes which container is pulled first :) | 15:08 |
| weshay | and then test | 15:08 |
| rlandy|ruck | should update | 15:08 |
| dmsimard | weshay: there's other containers that are working right before | 15:08 |
| * weshay looks | 15:08 | |
| dmsimard | centos-binary-cinder-api, centos-binary-cinder-scheduler etc | 15:08 |
| dmsimard | I wonder if that tagged image would still be on the rdo registry /me looks | 15:09 |
| weshay | dmsimard we can also exclude it from a deployment I guess | 15:09 |
| weshay | ur right.. in other containers are pulling ok | 15:09 |
| *** suuuper has joined #tripleo | 15:12 | |
| dmsimard | so it's trying to pull centos-binary-rsyslog:a447a10b12efed2e989ed61de5d0d1562a2919ea_d0e11ceb | 15:12 |
| dmsimard | which we still have in rdo registry | 15:13 |
| weshay | ah cool | 15:13 |
| dmsimard | there's some differences between the image in dockerhub and the one from rdo registry | 15:15 |
| weshay | what are you seeing? | 15:15 |
| dmsimard | ah, actually, there aren't | 15:17 |
| dmsimard | https://www.diffchecker.com/lMQYR8pP | 15:17 |
| dmsimard | the layer hashes are the same | 15:17 |
| dmsimard | it's the repo tags that are different, I don't think that matters | 15:17 |
| dmsimard | ¯\_(ツ)_/¯ | 15:18 |
| weshay | we can try to repush a rsyslog container if you think it may help... the containers are getting refreshed fairly often | 15:18 |
| weshay | already.. if there was a bad push.. anything about the docker files stand out? | 15:19 |
| weshay | if that one container was bad.. we would see it other providers too.. so we're still chasing something odd here afaict | 15:20 |
| dmsimard | weshay: this is most definitely occurring in other places than ovh | 15:20 |
| dmsimard | the one I linked yesterday came from limestone, I've seen failures in rax too | 15:21 |
| rlandy|ruck | abishop: hi - after changing the size of CinderLVMLoopDeviceSize - we've seem some instances of this failure: tempest.api.volume.admin.test_snapshots_actions.SnapshotsActionsTest.test_snapshot_force_delete_when_snapshot_is_error as in logs: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/9c5f8af/logs/undercloud/ | 15:21 |
| rlandy|ruck | home/zuul/tempest.log.txt.gz and https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/9c5f8af/logs/overcloud-controller-0/var/log/containers/cinder/cinder-scheduler.log.txt.gz?level=WARNING#_2019-08-21_04_27_15_820 | 15:21 |
| rlandy|ruck | abishop: I am rerunning those tests to conform repeatability - but any other suggestions to modify? | 15:22 |
| rlandy|ruck | confirm | 15:22 |
| weshay | dmsimard ah ur right.. this is new.. every time I've looked at this it was 99% ovh | 15:23 |
| weshay | but not now | 15:23 |
| weshay | node_provider | 15:23 |
| weshay | 23% limestone-regionone | 15:23 |
| weshay | 15% rax-ord | 15:23 |
| weshay | 14% ovh-bhs1 | 15:23 |
| weshay | 13% fortnebula-regionone | 15:23 |
| weshay | 12% rax-dfw | 15:23 |
| weshay | rlandy|ruck this is new to me | 15:23 |
| weshay | news | 15:23 |
| *** cylopez has quit IRC | 15:24 | |
| rlandy|ruck | weshay: not really fornebula has been a problem for a while as well | 15:24 |
| weshay | wait.. ha | 15:24 |
| weshay | sorry wrong query | 15:24 |
| dmsimard | weshay: the rsyslog error is always the same on the same hash with the same hash mismatch -- I would at least try to push another one in case there's a bad blob somewhere | 15:24 |
| weshay | dang it | 15:24 |
| rlandy|ruck | limestone we fixed somewhat | 15:24 |
| *** pierreprinetti has quit IRC | 15:24 | |
| mwhahaha | weshay: dmsimard: do we ever see these container failures in stein? or is it only in master? | 15:25 |
| abishop | rlandy|ruck: checking... | 15:25 |
| weshay | they may have fixed it | 15:25 |
| weshay | 0 fails in 24 hrs / 168 fails in 10 days | 15:25 |
| weshay | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Action%5C%5C%5C%22%3A%5C%5C%5C%22pull%5C%22%20AND%20message%3A%5C%22ERROR%20%2Fvar%2Flog%2Ftripleo-container-image-prepare.log%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1 | 15:25 |
| dmsimard | mwhahaha: I haven't witnessed it occurring in stein | 15:26 |
| mwhahaha | so if it hasn't happened in stein, it would point to some code change in master | 15:26 |
| weshay | dmsimard | 15:29 |
| weshay | node_provider | 15:29 |
| weshay | 83% ovh-bhs1 | 15:29 |
| weshay | 9% ovh-gra1 | 15:29 |
| weshay | 1% rax-dfw | 15:29 |
| weshay | 1% vexxhost-sjc1 | 15:29 |
| weshay | 1% rax-iad | 15:29 |
| rlandy|ruck | ha - ok | 15:29 |
| weshay | dmsimard it's still ovh :) | 15:29 |
| weshay | and it's not happened in the last 24hr | 15:30 |
| dmsimard | weshay: I think there are two different issues | 15:30 |
| weshay | k.. k | 15:30 |
| dmsimard | 1) we need to do something about rsyslog | 15:30 |
| weshay | k | 15:30 |
| dmsimard | 2) there are /sometimes/ authentication errors leading to failed pulls (of which rsyslog is also victim) | 15:31 |
| weshay | k | 15:31 |
| dmsimard | 1) is a digest mismatch, i.e, this: | 15:32 |
| dmsimard | msg="error copying src image [\"docker://192.168.24.1:8787/tripleomaster/centos-binary-rsyslog:bb7bc41730bc5be97fe27a9445337152c45f08f6_a47d70e7\"] to dest image [\"192.168.24.1:8787/tripleomaster/centos-binary-rsyslog:bb7bc41730bc5be97fe27a9445337152c45f08f6_a47d70e7\"] err: Error reading blob | 15:32 |
| dmsimard | sha256:eeed6d25dc3e17ce4389e478dbc4b5dc2f8148f9d22d7fb37991359feb98b31c: Digest did not match, expected sha256:eeed6d25dc3e17ce4389e478dbc4b5dc2f8148f9d22d7fb37991359feb98b31c, got sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | 15:32 |
| dmsimard | 2) is something else | 15:32 |
| dmsimard | #1 is hurting us and happening all over, we need to fix it | 15:32 |
| dmsimard | the work that cloudnull is doing will hopefully help us figure out #2 | 15:33 |
| weshay | dmsimard the logging confuses me at times.. so just to make sure I'm understanding it... docker://192 is the local registry.. is that error pulling from docker.io/proxy or pushing into the local reg | 15:33 |
| dmsimard | weshay: that particular line came from https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_21/677521/3/check/tripleo-ci-centos-7-standalone/6406cc9/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 15:34 |
| dmsimard | I'm not the most knowledgeable around these bits of code but if I were to take a guess, that's image_uploader downloading the images off of docker hub to push them to the local registry, yes | 15:34 |
| weshay | dmsimard that looks like a push to the local reg error | 15:34 |
| weshay | cloudnull am I reading that incorrectly? | 15:35 |
| weshay | I guess the digest didn't match | 15:35 |
| weshay | total hits: 231 | 15:37 |
| weshay | build_branch | 15:37 |
| weshay | 100% master | 15:37 |
| weshay | mwhahaha ^ | 15:37 |
| weshay | so 100% on master 99% on ovh, and either logstash had a hiccup or it hasn't happened in the last 24 hr | 15:37 |
| weshay | the main change from stein -> master would be buildah built containers vs.. docker | 15:38 |
| dmsimard | weshay: what is your query and how are you figuring out 99% ovh ? | 15:38 |
| * weshay gets | 15:38 | |
| weshay | dmsimard https://opendev.org/opendev/elastic-recheck/commit/1d1121f111748ef90538318719b09cb3c51226e1 | 15:39 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_grafana composable service https://review.opendev.org/674556 | 15:44 |
| ykarel | weshay, since when it's happening, buildah got updated in master on 12th August | 15:44 |
| ykarel | as per logstash query i see it's around that day only | 15:44 |
| weshay | interesting | 15:45 |
| ykarel | but why only ovh, may be performance related | 15:45 |
| weshay | thanks ykarel /me looks | 15:45 |
| dmsimard | weshay: that's #2 | 15:45 |
| dmsimard | I want to fix rsyslog :D | 15:45 |
| dmsimard | not dismissing that there isn't a problem, though | 15:45 |
| dmsimard | I was not able to find anything that would be specific to ovh yesterday | 15:45 |
| weshay | aye.. thanks for looking... fwiw it may be due to something we can't see | 15:46 |
| weshay | dmsimard ykarel may be on to something ( as usual ) I don't see hits prior to 8/12 | 15:47 |
| dmsimard | as far as I can tell the rsyslog image in the rdo registry is no different than the one in docker hub -- pulled both individually and everything between them matches | 15:47 |
| ykarel | weshay, dmsimard if node is hold where issue is reproduced, good to confirm by downgrading buildah(to buildah-1.8.2-1.gite23314b.el7), and then see what's issue with new buildah | 15:49 |
| dmsimard | ykarel: we have two held nodes | 15:49 |
| dmsimard | I can add your key to them | 15:49 |
| ykarel | https://trunk.rdoproject.org/centos7-master/deps/latest/x86_64/buildah-1.8.2-1.gite23314b.el7.x86_64.rpm | 15:50 |
| *** pierreprinetti has joined #tripleo | 15:50 | |
| dmsimard | # rpm -qa |grep buildah | 15:50 |
| dmsimard | buildah-1.10.1-2.git8c1c2c5.el7.x86_64 | 15:50 |
| ykarel | yes ^^ the updated one | 15:50 |
| ykarel | https://review.rdoproject.org/r/#/c/21791/1/buildsys-tags/cloud7-openstack-train-testing.yml | 15:50 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 15:51 |
| ykarel | dmsimard, https://ykarel.fedorapeople.org/mydata/pupkey.txt , but i can check only after some time(going for dinner) | 15:52 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/stein: Suppress output for ssh-keygen https://review.opendev.org/677577 | 15:52 |
| weshay | https://paste.pics/377e5067c36b5c648250e0167808d8af | 15:52 |
| dmsimard | ykarel: root@162.242.237.97 and root@158.69.67.231 | 15:53 |
| dmsimard | weshay: yeah ... | 15:53 |
| ykarel | dmsimard, ack, both acccessible | 15:54 |
| *** ykarel is now known as ykarel|afk | 15:54 | |
| *** mgagne has quit IRC | 15:54 | |
| *** mgagne has joined #tripleo | 15:55 | |
| *** jtomasek has quit IRC | 15:56 | |
| dmsimard | weshay: when/where is buildah installed ? can we send a dummy patch to test with the previous package and see what happens ? | 15:56 |
| dmsimard | I see https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/pre.yaml#L48 | 15:59 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: runner: return True to check if image/container exists https://review.opendev.org/677774 | 15:59 |
| * weshay gets | 15:59 | |
| weshay | dmsimard https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push and http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah | 16:02 |
| *** skramaja has quit IRC | 16:02 | |
| *** suuuper has quit IRC | 16:04 | |
| *** paramite has quit IRC | 16:04 | |
| *** rpittau is now known as rpittau|afk | 16:05 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-validations stable/stein: Set undercloud-connection to local by default https://review.opendev.org/677776 | 16:06 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-validations stable/stein: Add undercloud_key_file option in tripleo-ansible-inventory. https://review.opendev.org/677777 | 16:06 |
| dmsimard | ok buildah is a dependency of openstack-tripleo-common | 16:07 |
| dmsimard | --> Processing Dependency: buildah for package: openstack-tripleo-common-11.1.1-0.20190821053435.80013b0.el7.noarch | 16:08 |
| weshay | the version of buildah in centos7 may be older than what we want/need | 16:09 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Remove deprecated ceilometer::dispatcher::gnocchi https://review.opendev.org/677690 | 16:09 |
| rlandy|ruck | yay | 16:10 |
| dmsimard | weshay: latest upstream version is 1.10.1 which is what is currently provided | 16:11 |
| weshay | k | 16:11 |
| dmsimard | we had 1.8.2 prior | 16:11 |
| *** ratailor has quit IRC | 16:12 | |
| dmsimard | 225 commits between 1.8.2 and 1.10.1, would be a fun bisect | 16:15 |
| dmsimard | virt-sig has a build for 1.9.2 | 16:15 |
| weshay | dmsimard I wonder if we should put this on hold for another 24hrs? | 16:18 |
| mwhahaha | so | 16:18 |
| dmsimard | so buildah is picked up from openstack-tripleo-common and openstack-tripleo-common is actually picked up from tripleoclient --> Processing Dependency: openstack-tripleo-common >= 10.7.0 for package: python2-tripleoclient-12.1.1-0.20190821060145.f160f47.el7.noarch | 16:18 |
| mwhahaha | we're theorizing that it's failing on cloudflare | 16:18 |
| mwhahaha | we query docker hub for the laywers which redirects us to cloudflare | 16:18 |
| mwhahaha | we're wondering if cloudflare is returning 401 for ovh | 16:19 |
| dmsimard | mwhahaha: I have a held node from ovh | 16:19 |
| *** ykarel|afk is now known as ykarel | 16:20 | |
| weshay | fak... I'm only right on my second attempts today.. the centos7 master job is still using docker I think.. rlandy|ruck can we look at that? | 16:20 |
| *** xarlos has joined #tripleo | 16:20 | |
| weshay | rhel8 is using buildah | 16:21 |
| weshay | which is not related to this issue | 16:21 |
| *** lucasagomes has quit IRC | 16:21 | |
| weshay | might have taken you down a false path | 16:21 |
| ykarel | dmsimard, weshay so https://bugs.launchpad.net/tripleo/+bug/1839532 is reported on 9th august, so issue is happening even before 12th august buildah got updated | 16:21 |
| openstack | Launchpad bug 1839532 in tripleo "tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error" [Critical,Triaged] | 16:21 |
| ykarel | also matt reported in a comment it's happening from 5th August | 16:22 |
| weshay | ykarel ok.. makes more sense then | 16:22 |
| ykarel | weshay, looking further i found buildah is update to 1.9.0 in centos extras on 5th August | 16:22 |
| weshay | do we have jobs failing today.. perhaps logstash is faking me out today too.. | 16:22 |
| * weshay looks | 16:22 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 16:23 |
| ykarel | dmsimard, yes we can try installing different version of buildah in job | 16:25 |
| ykarel | after tripleo packages are installed | 16:25 |
| ykarel | in install-packages.sh | 16:25 |
| ykarel | or we can blacklist doubted buildah version | 16:25 |
| dmsimard | ykarel: do you want to try that ? I was going to hack something around here https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-7/consistent-master.yml#L100 | 16:26 |
| dmsimard | (or master.yml, or both) | 16:26 |
| ykarel | dmsimard, in ci master.yml is used | 16:26 |
| dmsimard | ykarel: ok, it is my understanding that https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-7/consistent-master.yml#L116 picks up openstack-tripleo-common which picks up buildah | 16:26 |
| ykarel | dmsimard, yes | 16:27 |
| ykarel | dmsimard, you can try similar to https://review.opendev.org/#/c/636860/ | 16:27 |
| dmsimard | I would naively add a "yum install https://trunk.rdoproject.org/centos7-master/deps/latest/x86_64/buildah-1.8.2-1.gite23314b.el7.x86_64.rpm" but not sure | 16:27 |
| ykarel | dmsimard, we need yum downgrade https://trunk.rdoproject.org/centos7-master/deps/latest/x86_64/buildah-1.8.2-1.gite23314b.el7.x86_64.rpm | 16:27 |
| ykarel | if doing after buildah is installed latest version | 16:27 |
| ykarel | or if we change repo to exclude buggy version, 1.8.2 will auto install | 16:28 |
| ykarel | so we need to exclude 1.9 from extras, and 1.10 from rdo deps | 16:29 |
| dmsimard | ok let me see | 16:30 |
| *** amoralej is now known as amoralej|off | 16:30 | |
| ykarel | ack | 16:30 |
| *** jpena is now known as jpena|off | 16:30 | |
| weshay | ykarel dmsimard I don't think buildah is in play | 16:31 |
| weshay | ykarel dmsimard for centos job I think we're still using docker | 16:31 |
| ykarel | weshay, why? as per dates atleast it seems related | 16:31 |
| dmsimard | weshay: it's great if it's not, it's not expensive to double check | 16:31 |
| weshay | dmsimard k | 16:32 |
| weshay | ykarel so I've had my head in rhel8 too much.. we're using buildah there to build the containers in the periodic jobs, but centos-7 train is using docker | 16:32 |
| weshay | I think I lead you down the wrong road | 16:33 |
| ykarel | weshay, from job logs, container_build_tool': 'buildah' | 16:33 |
| ykarel | https://logs.opendev.org/63/674363/1/gate/tripleo-ci-centos-7-undercloud-containers/9285c4c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 16:33 |
| ykarel | for -push job, it's buildah rhel8, and docker centos7 | 16:34 |
| weshay | ykarel ya.. after checking that is what I'm seeing yes | 16:34 |
| *** florianf has quit IRC | 16:35 | |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-quickstart master: Do not merge: Test master with buildah 1.8 https://review.opendev.org/677785 | 16:36 |
| dmsimard | ykarel: ^ like this ? | 16:36 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/stein: Add tags always into external update tasks. https://review.opendev.org/677786 | 16:36 |
| ykarel | dmsimard, /me looks | 16:36 |
| dmsimard | ah wait | 16:36 |
| dmsimard | mistake | 16:36 |
| *** dtantsur is now known as dtantsur|afk | 16:37 | |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-quickstart master: Do not merge: Test master with buildah 1.8 https://review.opendev.org/677785 | 16:37 |
| ykarel | dmsimard, looks fine | 16:38 |
| *** cfontain_ has quit IRC | 16:38 | |
| ykarel | hopefuuly some jobs will land on ovh | 16:38 |
| ykarel | weshay, so on ovh it always fails, or it passes sometimes | 16:38 |
| dmsimard | mwhahaha: this hunch with ovh and cloudflare, would we be able to reproduce it on a held now ? | 16:38 |
| dmsimard | s/now/node/ | 16:38 |
| *** pierreprinetti has quit IRC | 16:41 | |
| *** spsurya has quit IRC | 16:43 | |
| * rlandy|ruck chceks | 16:43 | |
| *** cfontain_ has joined #tripleo | 16:45 | |
| ykarel | dmsimard, i am running deploy on the holded node root@158.69.67.231 | 16:46 |
| ykarel | after downgrading buildah | 16:46 |
| dmsimard | ykarel: cool, how long for feedback ? | 16:47 |
| ykarel | dmsimard, i think 10-20 minutes | 16:47 |
| *** xek has quit IRC | 16:47 | |
| dmsimard | ++ | 16:47 |
| mwhahaha | dmsimard: maybe | 16:49 |
| *** Garyx has joined #tripleo | 16:49 | |
| weshay | dmsimard ykarel it's not related | 16:53 |
| *** cfontain_ has quit IRC | 16:53 | |
| dmsimard | weshay: what isn't ? | 16:54 |
| weshay | buildah | 16:54 |
| dmsimard | weshay: I'm out of ideas :) | 16:54 |
| weshay | I think mwhahaha is on to something w/ cloudflare | 16:55 |
| rlandy|ruck | weshay: we can change that to use buildah if you wish | 16:55 |
| rlandy|ruck | one setting | 16:55 |
| weshay | rlandy|ruck heh.. no.. I think we should pause all this work for another 24hrs | 16:55 |
| weshay | let's see if they fixed something finally as this hasn't hit our radar today... | 16:56 |
| weshay | afaict | 16:56 |
| rlandy|ruck | ok | 16:56 |
| weshay | I think the interwebs are afraid of dmsimard | 16:56 |
| weshay | as soon as he was invoked... it stopped | 16:57 |
| dmsimard | weshay: there's still #1 that needs to be taken care of :p | 16:58 |
| weshay | agree | 16:59 |
| weshay | <dmsimard> 1) is a digest mismatch, i.e, this: | 16:59 |
| weshay | 09:32 <dmsimard> msg="error copying src image [\"docker://192.168.24.1:8787/tripleomaster/centos-binary-rsyslog:bb7bc41730bc5be97fe27a9445337152c45f08f6_a47d70e7\"] to dest image [\"192.168.24.1:8787/tripleomaster/centos-binary-rsyslog:bb7bc41730bc5be97fe27a9445337152c45f08f6_a47d70e7\"] err: Error reading blob | 16:59 |
| weshay | 09:32 <dmsimard> sha256:eeed6d25dc3e17ce4389e478dbc4b5dc2f8148f9d22d7fb37991359feb98b31c: Digest did not match, expected sha256:eeed6d25dc3e17ce4389e478dbc4b5dc2f8148f9d22d7fb37991359feb98b31c, got sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | 16:59 |
| dmsimard | maybe that one could be buildah related ? who knows | 16:59 |
| weshay | mwhahaha the above snippet is uploading to the local reg right? | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack/tripleo-ansible master: Switch to promote docs job https://review.opendev.org/677791 | 17:03 |
| *** psachin has quit IRC | 17:04 | |
| *** Chaserjim has joined #tripleo | 17:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack/tripleo-validations master: Switch to promote docs job https://review.opendev.org/677792 | 17:06 |
| ykarel | weshay, ack, /me still doubt though | 17:07 |
| ykarel | as buildah in push job and in deploy job playing differently | 17:07 |
| weshay | playing differently? | 17:10 |
| weshay | ykarel don't understand yet what you mean | 17:11 |
| *** PagliaccisCloud has quit IRC | 17:12 | |
| ykarel | weshay, so in push job, rhel8 is buildah, centos7 is docker, in that job it just build and push, but in deploy container-image prepare, buildah pulls from docker.io and push to local registry, and then doing buildah pull from local registry and at that point it's failing(iiuc correctly the bug) | 17:15 |
| ykarel | and also iirc promoter uses docker to push to docker.io | 17:16 |
| ykarel | i am seeing DEBUG chardet.charsetprober [ ] IBM866 confidence = 0.0413828258123, below negative shortcut threshhold 0.05 in tripleo-container-image-prepare log in the reproducer, is that seen before? | 17:18 |
| openstackgerrit | Merged openstack/tripleo-common stable/stein: Add python3 file for hardened images https://review.opendev.org/677564 | 17:22 |
| *** fultonj has joined #tripleo | 17:23 | |
| xarlos | fultonj: There's definately a connectivity problem. | 17:23 |
| xarlos | oh, hahaha. Noticed you joined just as I started to type that. Sorry. Welcome :-D | 17:23 |
| xarlos | before ceph even has issues, it seems that pacemaker isn't able to sync up with the other nodes either. | 17:25 |
| xarlos | Seems that it may be trying to talk over the wrong network for these services. | 17:26 |
| openstackgerrit | Merged openstack/tripleo-common stable/stein: Add support for RHEL 8 and start using versionless element https://review.opendev.org/677565 | 17:26 |
| openstackgerrit | Merged openstack/tripleo-common stable/rocky: Enable staging-ovirt (fence_rhevm) fencing agent. https://review.opendev.org/677531 | 17:27 |
| openstackgerrit | Merged openstack/tripleo-common stable/queens: Enable staging-ovirt (fence_rhevm) fencing agent. https://review.opendev.org/676420 | 17:27 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/stein: Run Validations with ThreadPoolExecutor https://review.opendev.org/677170 | 17:27 |
| *** jpich has quit IRC | 17:27 | |
| ykarel | dmsimard, on the reproducer node it failed with 1.8.2 also, not sure what's going on then | 17:43 |
| dmsimard | ykarel: where is the failure ? | 17:44 |
| ykarel | dmsimard, same place, rsyslog pull | 17:44 |
| ykarel | the first image it tries, it fails and not move forward | 17:45 |
| ykarel | only 1 image in registry, curl 192.168.24.1:8787/v2/_catalog | 17:45 |
| dmsimard | ykarel: I've opened a tmux, can you join and show me ? | 17:48 |
| ykarel | dmsimard, in | 17:51 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: runner: return True to check if image/container exists https://review.opendev.org/677774 | 17:53 |
| *** dprince has joined #tripleo | 17:57 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: zypper-minimal: Don't get confused by etc/resolv.conf symlink https://review.opendev.org/677796 | 17:57 |
| *** gfidente is now known as gfidente|afk | 18:06 | |
| *** abishop is now known as abishop|afk | 18:11 | |
| *** pkopec has quit IRC | 18:14 | |
| *** morazi has quit IRC | 18:21 | |
| *** ramishra has quit IRC | 18:24 | |
| dmsimard | weshay, mwhahaha: ykarel and I have been able to reproduce the rsyslog error but it's still mysterious | 18:26 |
| mwhahaha | the sha generation one or the 401 issue | 18:26 |
| dmsimard | the digest mismatch | 18:28 |
| *** Chaserjim has quit IRC | 18:28 | |
| dmsimard | see http://paste.openstack.org/raw/761107/ | 18:32 |
| *** jtomasek has joined #tripleo | 18:32 | |
| *** kopecmartin is now known as kopecmartin|off | 18:33 | |
| weshay | k.. will look in a minute :) in 1-1 | 18:33 |
| weshay | dmsimard thank you!! | 18:33 |
| mwhahaha | wait so you reproduced it but the source is the local registry | 18:34 |
| mwhahaha | meaning when we wrote it out, it was bad | 18:34 |
| mwhahaha | or | 18:34 |
| mwhahaha | ovh has odd disk issues? | 18:34 |
| dmsimard | mwhahaha: I would like to re-push it to the local registry but I'm not sure how | 18:35 |
| mwhahaha | you have to use this code that is currently giving us fits :D | 18:35 |
| mwhahaha | cause that's the only way to do it | 18:35 |
| mwhahaha | currently we are: (ﾉಥ益ಥ）ﾉ sɹǝuıɐʇuoɔ | 18:36 |
| dmsimard | mwhahaha: yeah doing a manual buildah push gives a http 405 | 18:36 |
| mwhahaha | yea the local iamge is a read only implementation | 18:36 |
| mwhahaha | we have python code that fork lifts the layers on to disk | 18:36 |
| mwhahaha | so from that paste, did you just run those 3 commands so it failed teh first time but was successful later? | 18:37 |
| * mwhahaha is trying to understand context | 18:37 | |
| *** holser__ has quit IRC | 18:37 | |
| dmsimard | right, the issue reproduces only if the image isn't pulled already | 18:37 |
| mwhahaha | but we dont' use buildah pull | 18:38 |
| dmsimard | pulling it from dockerhub before trying to pull it from the local registry doesn't reproduce the digest mismatch | 18:38 |
| mwhahaha | oh yes we do | 18:39 |
| dmsimard | the layer that the error is complaining about is apparently there: /var/lib/image-serve/v2/tripleomaster/centos-binary-rsyslog/blobs/sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df.gz | 18:40 |
| dmsimard | it says "expected sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df, got e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | 18:40 |
| dmsimard | I can't find e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 anywhere | 18:40 |
| mwhahaha | well it's a generated hash based on the file content i think | 18:41 |
| mwhahaha | it's supposed to match the manifest | 18:41 |
| dmsimard | oh wait | 18:41 |
| mwhahaha | was your reproducer on the ovh node or elsewhere? | 18:41 |
| dmsimard | ovh but I don't think there's a correlation between ovh and the rsyslog issue | 18:42 |
| mwhahaha | did we see that elsewhere? | 18:42 |
| dmsimard | yes | 18:42 |
| mwhahaha | k | 18:42 |
| dmsimard | so | 18:44 |
| dmsimard | in an interesting turn of events | 18:44 |
| dmsimard | it turns out that the sha256sum of /var/lib/image-serve/v2/tripleomaster/centos-binary-rsyslog/blobs/sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df.gz is e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | 18:44 |
| dmsimard | which is what buildah complains about | 18:44 |
| mwhahaha | right | 18:44 |
| mwhahaha | that likely doesn't match the expected output from the blob call | 18:45 |
| dmsimard | I sha256sum'd some other blobs and they all match | 18:45 |
| mjturek | does anyone know Gabriele Cerami's nick? He asked me to ping him but I can't find it | 18:46 |
| mwhahaha | mjturek: panda | 18:47 |
| mjturek | thanks mwhahaha ! | 18:47 |
| mjturek | panda you around? | 18:47 |
| mwhahaha | dmsimard: so odd thing is that i don't know why that layer should be fetched, i don't see it in the manifest for rsyslog | 18:48 |
| weshay | mjturek probably too late for him | 18:49 |
| dmsimard | mwhahaha: where are you seeing that ? I see 8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df in both dockerhub and rdo registry https://www.diffchecker.com/lMQYR8pP | 18:49 |
| mjturek | fair enough! I'll try tomorrow morning weshay | 18:50 |
| weshay | :) | 18:50 |
| mwhahaha | dmsimard: maybe i'm looking at the wrong tag, let me see | 18:50 |
| dmsimard | also, for the record, the blob sum mismatch: http://paste.openstack.org/raw/761109/ | 18:50 |
| weshay | hrm | 18:52 |
| dmsimard | let me check the other node that we had held out of curiosity | 18:52 |
| *** ksambor has quit IRC | 18:52 | |
| weshay | when ur pulling from the held nodepool node.. is it going through ovh proxy/cdn ( whatever ) | 18:53 |
| dmsimard | weshay: doesn't matter where I'm pulling from if the sha256 sum doesn't match on disk | 18:53 |
| * weshay wonders if it's a round-robin style cdn.. and one server has an older copy of the container | 18:53 | |
| weshay | tru | 18:54 |
| dmsimard | it's the push to the local registry that fails | 18:54 |
| weshay | ya.. in that case... we need a diff logstash query to see how often that happens though | 18:54 |
| dmsimard | this other node has a matching sha256 for that particular layer | 18:56 |
| dmsimard | but it uses the tag a447a10b12efed2e989ed61de5d0d1562a2919ea_d0e11ceb instead of bb7bc41730bc5be97fe27a9445337152c45f08f6_a47d70e7 | 18:57 |
| dmsimard | ah, there was a promotion, a447 is newer | 18:59 |
| dmsimard | bb7bc41730bc5be97fe27a9445337152c45f08f6_a47d70e7 is from 2019-08-18 while a447a10b12efed2e989ed61de5d0d1562a2919ea_d0e11ceb is from today | 19:00 |
| dmsimard | oh so it does still mismatch but on different layers now.. | 19:02 |
| mwhahaha | when you do a buildah pull from the local registry? | 19:02 |
| dmsimard | still the same e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 culprit though | 19:02 |
| dmsimard | Error reading blob sha256:ea492b89a5eefe62721d9f4b129961d48b48e8a01b61ac53b42b24d1dc87cdd0: Digest did not match, expected sha256:ea492b89a5eefe62721d9f4b129961d48b48e8a01b61ac53b42b24d1dc87cdd0, got sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | 19:02 |
| dmsimard | mwhahaha: yeah "# buildah pull 192.168.24.1:8787/tripleomaster/centos-binary-rsyslog:a447a10b12efed2e989ed61de5d0d1562a2919ea_d0e11ceb" | 19:02 |
| mwhahaha | so that would point to the code that writes it out to /var/www/img-serve being bad | 19:03 |
| mwhahaha | either we're cutting off a bit or something | 19:03 |
| mwhahaha | seems odd that it's the same layer repeatedly unless the cache is bad? | 19:03 |
| *** morazi has joined #tripleo | 19:04 | |
| dmsimard | it is suspicious that the same sha256sum is returned by two different layers from two different tags | 19:04 |
| mwhahaha | unless that's like the base centos layer | 19:04 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 19:05 |
| weshay | mwhahaha dmsimard that is happening but not that often http://paste.openstack.org/show/761110/ | 19:05 |
| mwhahaha | cloudnull: fun fact if you request blobs from the registry and it's not found you actually do get a 404 not a 401 | 19:08 |
| dmsimard | weshay: I'm not convinced | 19:09 |
| dmsimard | weshay: those results are from a zuul-operator job | 19:09 |
| weshay | some of them | 19:09 |
| weshay | some are from 7% tripleo-ci-fedora-28-standalone | 19:09 |
| dmsimard | weshay: that's one hit in the last 7 days | 19:09 |
| dmsimard | weshay: I don't think that query is getting the stuff we're interested in or perhaps the file it's located in isn't indexed | 19:10 |
| weshay | let me try a diff query but the file is indexed | 19:10 |
| dmsimard | I would expect at least a few hits even for just "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" but logstash doesn't return anything | 19:10 |
| weshay | tbh it may not be indexing today and that's why we're not seeing ovh issues | 19:10 |
| dmsimard | weshay: logstash queue is a bit high http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 | 19:11 |
| weshay | ah see that | 19:12 |
| weshay | so nice to have an infra guy here!!!1 | 19:12 |
| dmsimard | oh | 19:12 |
| dmsimard | turns out that "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" is the sha256sum of an empty file :/ http://paste.openstack.org/show/761112/ | 19:14 |
| mwhahaha | empty gzip or empty | 19:14 |
| *** jtomasek has quit IRC | 19:15 | |
| dmsimard | empty file | 19:15 |
| dmsimard | sha256sum'd an empty file created by touch | 19:16 |
| * dmsimard sighs | 19:16 | |
| weshay | that's good to know though.. /me notes it | 19:16 |
| dmsimard | ok so then the blob is empty on disk, now what ? something failed during the upload to the local registry ? | 19:17 |
| *** saneax has quit IRC | 19:17 | |
| *** pierreprinetti has joined #tripleo | 19:18 | |
| *** jtomasek has joined #tripleo | 19:18 | |
| mwhahaha | yea trying to track down where that is | 19:18 |
| weshay | this is all local on the node right? no upstream log? | 19:18 |
| mwhahaha | i don't suppose the disk is full | 19:18 |
| dmsimard | it's not | 19:20 |
| dmsimard | that was for mwhahaha ^ | 19:20 |
| dmsimard | weshay: the two nodes that we have the issue on are held from the tripleo-ci-centos-7-standalone job | 19:21 |
| weshay | k k | 19:21 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1175-L1194 | 19:21 |
| mwhahaha | i think that's teh fetching the content | 19:21 |
| mwhahaha | thought maybe it's a buildah thing where buildah pull results in a 0 byte laywers | 19:23 |
| mwhahaha | this is the export code https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1429-L1458 | 19:24 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1504-L1550 | 19:25 |
| weshay | I don't see that sha in anything but some f28 jobs | 19:25 |
| mwhahaha | dmsimard: do you still have the node with the 0 file? | 19:25 |
| dmsimard | mwhahaha: I have two of them | 19:25 |
| mwhahaha | dmsimard: can you look in the manifest file to see if size = 0? | 19:25 |
| mwhahaha | or gimme teh ssh | 19:25 |
| mwhahaha | https://launchpad.net/~alex-schultz/+sshkeys | 19:26 |
| dmsimard | mwhahaha: root@162.242.237.97 = rax, root@158.69.67.231 = ovh | 19:26 |
| *** morazi has quit IRC | 19:27 | |
| dmsimard | from the rax node: http://paste.openstack.org/raw/761114/ | 19:28 |
| weshay | rsyslog just updated again fyi | 19:30 |
| weshay | 4 hrs ago | 19:30 |
| weshay | 0eff77ac8d8e39c967d275f15fb633a6cc97ecc4_5016c5e0 | 19:30 |
| dmsimard | ohhhh | 19:31 |
| weshay | what's up | 19:32 |
| mwhahaha | probably nothing, he just likes to get our hopes up | 19:32 |
| weshay | lolz | 19:32 |
| dmsimard | paste from rax with a little bit more meat: http://paste.openstack.org/show/761116/ | 19:33 |
| mwhahaha | are you ohhhh'ing the dropped conneciton thing? | 19:34 |
| mwhahaha | so it tries to fetch, conneciton gets drop, doesn't refetch because files already exists | 19:35 |
| *** morazi has joined #tripleo | 19:35 | |
| mwhahaha | we need to HEAD but check file size | 19:35 |
| mwhahaha | let me see if we properly return filesize | 19:37 |
| dmsimard | mwhahaha: the "Uploading layer" and "export layer to" parts that occur before "Fetching layer" are likely resulting in an empty file | 19:37 |
| dmsimard | I'm not sure why we would attempt to upload or export anything before fetching it though | 19:37 |
| mwhahaha | i think this is the awkward part of our having to manually output the laywers to the fs | 19:38 |
| mwhahaha | my layers needs lawyers | 19:38 |
| *** pierreprinetti has quit IRC | 19:40 | |
| *** abishop|afk is now known as abishop | 19:40 | |
| mwhahaha | we don't provide a filesize | 19:41 |
| mwhahaha | i found it | 19:42 |
| mwhahaha | hmm maybe not | 19:43 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_export.py#L78-L123 | 19:44 |
| mwhahaha | so that's the function that handles this bits | 19:44 |
| mwhahaha | now if it throws an exception, it's supposed to remove teh file | 19:44 |
| mwhahaha | that's called from https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1512 | 19:45 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1205-L1221 is the function where teh Uploading layer comes in | 19:46 |
| mwhahaha | which if that fails, gets retried and so the 2nd time thorugh, the file already exists | 19:47 |
| mwhahaha | so it seems liek https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_export.py#L117-L118 isn't working correctly | 19:48 |
| mwhahaha | but i don't see any "Error while" in the logs | 19:49 |
| *** pbandark has quit IRC | 19:50 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DNM: Test adding compute_feature_enabled.config_drive to fs035 https://review.opendev.org/677807 | 19:52 |
| mwhahaha | it looks like https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_export.py#L101-L105 doesn't get caught by the exception | 19:54 |
| mwhahaha | because I don't see the "Error while writing" bits | 19:54 |
| * mwhahaha flips tables | 19:54 | |
| mwhahaha | i don't think IOError inherits from Exception | 19:55 |
| dmsimard | I see plenty of "Calculated layer digest", no "Expected digest" or "Error while writing blob" | 19:55 |
| mwhahaha | right i think we need to catch the IOError | 19:57 |
| mwhahaha | or at least add cleanup code in there | 19:57 |
| xarlos | should I specify the ha environment template AFTER my network isolation template, or does it not matter? :-/ | 20:00 |
| *** morazi has quit IRC | 20:01 | |
| dmsimard | mwhahaha: cloudnull has been working on https://review.opendev.org/#/c/674919/31/tripleo_common/image/image_export.py | 20:01 |
| mwhahaha | yea but i'm going to just target this one bit, we can rewrite it later | 20:02 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Adapt os_tempest in FS001 https://review.opendev.org/673400 | 20:02 |
| mwhahaha | this part should likely be backported beyond the 401 bits | 20:02 |
| dmsimard | mwhahaha: fwiw the rax node has the cloudnull image_export (bunch of "Provided layer digest" in logs) and it doesn't trip the removal of the blob either | 20:03 |
| dmsimard | so digest == layer_digest | 20:03 |
| dmsimard | http://paste.openstack.org/show/761157/ | 20:04 |
| *** morazi has joined #tripleo | 20:05 | |
| cloudnull | dmsimard it should only trip the removal when the layer's are different - https://review.opendev.org/#/c/674919/31/tripleo_common/image/image_export.py@104 | 20:09 |
| *** holser has joined #tripleo | 20:09 | |
| cloudnull | and verify_digest is true | 20:10 |
| dmsimard | cloudnull: we found out that the rsyslog digest mismatches are actually because some blobs are empty in /var/lib/image-serve | 20:10 |
| cloudnull | so there's actually something wrong with that image? | 20:10 |
| dmsimard | cloudnull: no, something goes wrong during the upload of the image to the local registry -- you can see it happen here: http://paste.openstack.org/show/761116/ | 20:11 |
| cloudnull | mwhahaha and I were looking at how that manifest / blob arrays are built this morning | 20:11 |
| mwhahaha | cloudnull: so it seems like mid-export, it's getting an issue which is triggering reauth/retry | 20:11 |
| mwhahaha | cloudnull: but it's not cleaning up the empty file before it does that | 20:11 |
| mwhahaha | cloudnull: so when it retries, it skips redownloading it because it thinks it's there | 20:12 |
| cloudnull | i see | 20:12 |
| mwhahaha | cloudnull: http://paste.openstack.org/show/761116/ | 20:12 |
| mwhahaha | that past show its | 20:12 |
| mwhahaha | uploading... export .. fetching .... reauth ... and later Layer already exists | 20:12 |
| mwhahaha | so somewhere iun the export code sadness is occuring | 20:13 |
| *** dprince has quit IRC | 20:13 | |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_export.py#L90-L119 | 20:13 |
| *** ykarel has quit IRC | 20:13 | |
| mwhahaha | i think if f.write throws an IOerror, it doesn't trigger the Exception | 20:13 |
| dmsimard | mwhahaha: so "layer_stream" is actually empty ? | 20:14 |
| mwhahaha | also possible | 20:14 |
| cloudnull | so the connection is severed mid stream | 20:15 |
| mwhahaha | if length is 0, we need to nuke it as well | 20:15 |
| mwhahaha | though i'm trying ot figure out how the "Error while writing" isn't triggered | 20:15 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1495 is the uploading log line | 20:15 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1512 should be the export | 20:16 |
| dmsimard | mwhahaha: https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1205-L1221 ? | 20:16 |
| *** ykarel has joined #tripleo | 20:16 | |
| mwhahaha | i think that's the registry to registry | 20:17 |
| mwhahaha | thi sis local to registry? | 20:17 |
| mwhahaha | would be nice if these lines were different :D | 20:17 |
| cloudnull | when we see the log lines - "Calculated layer digest" and "Provided layer digest" the write should already be complete. | 20:17 |
| dmsimard | yeah .. | 20:17 |
| mwhahaha | The other issue is that this might be multithread related | 20:18 |
| mwhahaha | cause this is all shoved in to the threadpool | 20:18 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1256-L1264 | 20:19 |
| *** morazi has quit IRC | 20:19 | |
| dmsimard | mwhahaha: it's kinda hard to tell if it's local to registry or registry to registry | 20:20 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1175-L1194 is the fetching laywer bits | 20:20 |
| mwhahaha | so if that session.get triggers a 401, that'd be a problem | 20:20 |
| mwhahaha | now the questio nis where does that get used from | 20:20 |
| cloudnull | looking at https://review.opendev.org/#/c/674919/31/tripleo_common/image/image_export.py@91 we should see a write exception (IOError) if the stream is interrupted or is broken for some reason? | 20:20 |
| cloudnull | that's used in _copy_layer_registry_to_registry | 20:21 |
| *** jtomasek has quit IRC | 20:21 | |
| mwhahaha | i wonder if https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1166-L1174 is throwing it off | 20:22 |
| mwhahaha | it really does seems like we ahve too many retries everywhere | 20:23 |
| mwhahaha | and that we should only retry the entire thing | 20:23 |
| cloudnull | mwhahaha +1 | 20:23 |
| cloudnull | the use of tenacity is rampant | 20:24 |
| dmsimard | one could say it's .... tenacious | 20:24 |
| cloudnull | looking at https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/logs_19/674919/31/check/tripleo-ci-centos-7-standalone/18a23f1/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz we can now see the manifests, search for Current image manifest | 20:24 |
| cloudnull | the rsyslog manifest json is there. | 20:24 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: add an about doc section https://review.opendev.org/677814 | 20:26 |
| mwhahaha | so that one is the same problem | 20:26 |
| mwhahaha | uploading layer... export.. fetch... 401...layer already exists | 20:26 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: add an about doc section https://review.opendev.org/677814 | 20:26 |
| mwhahaha | and then the subsequent issues are buildah trying to pull an empty layer | 20:26 |
| mwhahaha | ctrl+f the first ea492b89a5eefe62721d9f4b129961d48b48e8a01b61ac53b42b24d1dc87cdd0 | 20:27 |
| cloudnull | right after "Resetting dropped connection: mirror.ord.rax.opendev.org" | 20:27 |
| mwhahaha | so the connections went stale because it was doing stuff for a while | 20:29 |
| mwhahaha | took like 18 mins to download sha256:209971f216a96e86b32611cc51fb4d60289b328fda8149dac99cf93b12567d7a | 20:29 |
| *** morazi has joined #tripleo | 20:31 | |
| dmsimard | need to be off for a bit, if there's anything infra related you'd like to check, feel free to ping me and I'll check in later | 20:33 |
| mwhahaha | thanks dmsimard | 20:33 |
| weshay | thanks! dmsimard :) | 20:33 |
| mwhahaha | cloudnull: so i think the entry point for this i _copy_layer_registry_to_registry | 20:33 |
| dmsimard | btw we should probably update https://bugs.launchpad.net/tripleo/+bug/1839532 with the extent of what we know | 20:34 |
| openstack | Launchpad bug 1839532 in tripleo "tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error" [Critical,Triaged] | 20:35 |
| openstackgerrit | Merged openstack/tripleo-common master: Allow mixing of count and instances https://review.opendev.org/670437 | 20:35 |
| mwhahaha | i filed https://bugs.launchpad.net/tripleo/+bug/1840973 for this specific 0 byte issue | 20:35 |
| openstack | Launchpad bug 1840973 in tripleo "container image prepare doesn't cleanup files on filesystem connection issues occur mid export" [High,In progress] - Assigned to Alex Schultz (alex-schultz) | 20:35 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 20:37 |
| cloudnull | added https://review.opendev.org/#/c/674919/32/tripleo_common/image/image_export.py@99 | 20:37 |
| mwhahaha | cloudnull: so i think https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1469 is the retry issue. So when the connection drops, that function is retried even though it's already gone into cls._copy_stream_to_registry | 20:37 |
| cloudnull | should there be a write error in the stream it will raise an IOError which _copy_layer_registry_to_registry seems to expect | 20:38 |
| mwhahaha | it seems to be going _copy_layer_local_to_registry -> _copy_stream_to_registry -> image_export.export_stream -> _layer_stream_local -> sadness | 20:39 |
| mwhahaha | actually not _layer_stream_loca, it's trying to fetch it from teh remote which would be _layer_stream_registry | 20:40 |
| mwhahaha | so that would mean it's _copy_layer_registry_to_registry actually | 20:41 |
| * mwhahaha cries a bit more | 20:41 | |
| mwhahaha | wonder if that means that _copy_registry_to_registry is the retry since that'll retry on RequestException | 20:42 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1238 | 20:42 |
| mwhahaha | _copy_registry_to_registry should only trigger on IOError | 20:42 |
| openstackgerrit | Merged openstack/diskimage-builder master: zypper-minimal: Don't get confused by etc/resolv.conf symlink https://review.opendev.org/677796 | 20:43 |
| *** ykarel has quit IRC | 20:48 | |
| *** fultonj has quit IRC | 20:54 | |
| cloudnull | with the new change, and seemingly endless layers of abstractions it "should" force an IOError when it fails. i hope? i think? i hope... | 20:56 |
| mwhahaha | no idea | 20:57 |
| mwhahaha | the other way to attack it is to do a content length check when seeing if the layer already exists but that might not work for real registries | 20:58 |
| mwhahaha | it'd prevent this for our local registry but that doesn't seem to be right | 20:58 |
| mwhahaha | it seems like tenacity is taking precedence over our exception handling in this case | 20:58 |
| mwhahaha | you'd think that https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_export.py#L89-L119 would be sufficient but it doesn't seem to be triggering in this case | 20:59 |
| cloudnull | (_copy_layer_registry_to_registry || _copy_layer_local_to_registry) > _copy_stream_to_registry > image_export.export_stream > sadness seems to be the ticket. | 21:00 |
| mwhahaha | i think we should remove the trtry from _layer_stream_registry | 21:01 |
| mwhahaha | because that should get caught elsewhere | 21:01 |
| cloudnull | but _copy_layer_local_to_registry has tenacity only checking for `requests.exceptions.RequestException` | 21:01 |
| cloudnull | +1 | 21:01 |
| mwhahaha | so in this case i don't think it's copy_layer_local_to_rgistry | 21:01 |
| mwhahaha | because that assumes it's available on disk (in this case it isn't yet) | 21:01 |
| mwhahaha | so this is a registry -> registry copy | 21:02 |
| mwhahaha | so the offending entry point is _copy_layer_registry_to_registry | 21:02 |
| * cloudnull will push another review to remove tenacity on _layer_stream_registry | 21:02 | |
| mwhahaha | _copy_layer_registry_to_registry should only retry on IOError which would be a digest problem | 21:02 |
| cloudnull | done. | 21:03 |
| mwhahaha | so if we take tenacity off _layer_stream_registry, i wonder if it'll trigger the exception handling in export_stream | 21:03 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 21:03 |
| cloudnull | ^ about to find out | 21:03 |
| mwhahaha | soooo come back tomorrow to find out ? :D | 21:03 |
| cloudnull | same bat time, same bat channel | 21:03 |
| * cloudnull will sit here and watch it for the next hour or two | 21:04 | |
| cloudnull | so while I have you , | 21:04 |
| cloudnull | what is the best way to troubleshoot "TemplateOutputError: Error in 46 output role_data: Incorrect arguments: Items to merge must be maps."? | 21:05 |
| mwhahaha | cloudnull: the items you're trying to merge together aren't dictionaries | 21:05 |
| mwhahaha | so data type problem | 21:05 |
| mwhahaha | do you have a review in reference to that error? | 21:06 |
| cloudnull | https://review.opendev.org/#/c/677237/ | 21:06 |
| cloudnull | at least this one I can reproduce locally. | 21:06 |
| mwhahaha | i'm assuming it's https://review.opendev.org/#/c/677237/10/deployment/tripleo-firewall/tripleo-firewall-baremetal-ansible.yaml@60 | 21:08 |
| mwhahaha | i'd send a note to tabi maybe he can give you a hint tomorrow | 21:09 |
| cloudnull | will do | 21:10 |
| cloudnull | thanks! | 21:10 |
| *** Goneri has quit IRC | 21:12 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add an undeploy_roles workflow https://review.opendev.org/672848 | 21:16 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add ExtraKernelPackages https://review.opendev.org/676504 | 21:16 |
| *** altlogbot_2 has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Use python docstring. https://review.opendev.org/677418 | 21:17 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: keystone: drop duplicate -DFOREGROUND https://review.opendev.org/677430 | 21:17 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates stable/stein: Configure nova_compute for vendordata https://review.opendev.org/677836 | 21:20 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates stable/rocky: Configure nova_compute for vendordata https://review.opendev.org/677838 | 21:21 |
| *** rcernin has joined #tripleo | 21:27 | |
| openstackgerrit | Tom Barron proposed openstack/tripleo-heat-templates master: Unescape IPv6 addresses for ceph_nfs_bind_addr https://review.opendev.org/677841 | 21:35 |
| *** raildo has quit IRC | 21:36 | |
| *** altlogbot_3 has joined #tripleo | 21:38 | |
| *** altlogbot_3 has quit IRC | 21:38 | |
| *** sshnaidm is now known as sshnaidm|afk | 21:40 | |
| *** altlogbot_2 has joined #tripleo | 21:42 | |
| *** altlogbot_2 has quit IRC | 21:42 | |
| xarlos | mwhahaha: If you get a moment, could you have a look at this and see if you can point me into the direction of fixing my HA config? :-/ http://pastebin.com/57wwhrHk | 21:52 |
| xarlos | I cannot quite tell if some of the things there are expected erronious, or a key indicator to my new deployment | 21:54 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 21:57 | |
| *** bnemec has quit IRC | 21:59 | |
| mwhahaha | I don't have any suggestions | 21:59 |
| xarlos | okay, thanks. | 21:59 |
| mwhahaha | Does pcs status return ok? | 22:00 |
| xarlos | No. Says cluster is not currently running. | 22:00 |
| xarlos | I can telnet to the port. | 22:01 |
| *** morazi has quit IRC | 22:07 | |
| mwhahaha | Sounds like pacemaker startup issues | 22:08 |
| mwhahaha | Maybe ntp/hostname/networking | 22:08 |
| xarlos | yeah. Could it be something I missed out configuring? Typically as long as I define the number of nodes, there's not a lot else to do :-/ | 22:09 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 22:09 |
| *** morazi has joined #tripleo | 22:09 | |
| xarlos | ntp seems fine | 22:14 |
| xarlos | hostname is correct in /etc/hosts | 22:15 |
| openstackgerrit | Takashi Kajinami proposed openstack/python-tripleoclient master: DNM: just to show possible improvement for the parent patch https://review.opendev.org/677741 | 22:15 |
| xarlos | networking could be an issue, but in what sense? :-/ | 22:15 |
| openstackgerrit | Merged openstack/tripleo-ansible master: Switch to promote docs job https://review.opendev.org/677791 | 22:18 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: Support direct provisioning of bare metal https://review.opendev.org/664456 | 22:18 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: DNM: undercloud_enable_nova: false by default https://review.opendev.org/664170 | 22:18 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: DNM: baremetal_provision:true by default https://review.opendev.org/677849 | 22:18 |
| *** holser has quit IRC | 22:20 | |
| *** rh-jelabarre has quit IRC | 22:21 | |
| *** threestrands has joined #tripleo | 22:34 | |
| *** rcernin has quit IRC | 22:40 | |
| *** rcernin has joined #tripleo | 22:43 | |
| xarlos | wjere can I get the pcsd password from? | 22:45 |
| *** ekultails has quit IRC | 22:50 | |
| *** tkajinam has joined #tripleo | 22:56 | |
| openstackgerrit | Takashi Kajinami proposed openstack/python-tripleoclient master: DNM: just to show possible improvement for the parent patch https://review.opendev.org/677741 | 22:59 |
| *** morazi has quit IRC | 23:01 | |
| *** Vorrtex has quit IRC | 23:01 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 23:04 |
| xarlos | mwhahaha: What would be stopping me from sshing from one controller to another? | 23:09 |
| xarlos | (iptables is not runnig) | 23:09 |
| *** morazi has joined #tripleo | 23:10 | |
| xarlos | I can talk over the provisioning network over ssh, but not over any of the bridged connections. | 23:12 |
| xarlos | (different nics) | 23:12 |
| xarlos | Having masquerade on the undercloud has o bearing on the overcloud, does it? | 23:33 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Add unique names support for cont_exec_args method https://review.opendev.org/677852 | 23:35 |
| *** dsneddon has quit IRC | 23:36 | |
| *** dsneddon has joined #tripleo | 23:37 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add arm64 based functional test https://review.opendev.org/676111 | 23:39 |
| *** dsneddon has quit IRC | 23:44 | |
| *** cdearborn has quit IRC | 23:45 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Add KOLLA_BOOTSTRAP=True to 2 bootstrap containers https://review.opendev.org/677855 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!