| *** henriqueof has quit IRC | 00:00 | |
| *** izza_ has joined #openstack-kolla | 00:13 | |
| izza_ | Hi,can anybody here help me pls...i'm testing rally heat(scenario: create_stack_scale) and i'm encountering below error: ***heat-engine***** 2019-06-13 18:27:19,809.809 1 ERROR heat.engine.service ResourceFailure: resources.asg: HEAT-E99001 Service nova is not available for resource type OS::Nova::Server, reason: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped | 00:21 |
|---|---|---|
| izza_ | ***Rally Logs*** TimeoutException: Rally tired waiting 3600.00 seconds for Stack <Stack {u'parent': None, u'disable_rollback': True, u'description': u'No description', u'parameters': {u'OS::project_id': u'53ae40faef3d47938019d7eded6dc4d2', u'OS::stack_id': u'ba39558d-63f7-43de-9a7f-aee7e325f623', u'OS::stack_name': u's_rally_c10f9daf_7eSBDfas', u'image': u'cirros-0.4.0-x86_64-disk', u'scaling_adjustment': u'1', u'flavor': u'm | 00:21 |
| izza_ | i badly needed ur help guys...i have deadline Today | 00:23 |
| *** dciabrin has joined #openstack-kolla | 00:35 | |
| *** dciabrin_ has quit IRC | 00:35 | |
| *** chrizl has quit IRC | 01:30 | |
| *** guozijn has joined #openstack-kolla | 01:33 | |
| *** igordc has quit IRC | 01:35 | |
| *** Sravan has joined #openstack-kolla | 02:00 | |
| *** Sravan has quit IRC | 02:05 | |
| *** zhanglong has joined #openstack-kolla | 02:12 | |
| *** zhanglong has quit IRC | 02:28 | |
| *** zhanglong has joined #openstack-kolla | 02:28 | |
| *** ykarel has joined #openstack-kolla | 02:44 | |
| *** ykarel has quit IRC | 02:51 | |
| *** guozijn has quit IRC | 02:52 | |
| *** guozijn has joined #openstack-kolla | 02:56 | |
| *** gregoryo has joined #openstack-kolla | 02:57 | |
| *** guozijn has quit IRC | 03:05 | |
| *** guozijn has joined #openstack-kolla | 03:06 | |
| *** adrianreza_ has quit IRC | 03:07 | |
| *** Wasaac has joined #openstack-kolla | 03:07 | |
| *** dmellado_ has joined #openstack-kolla | 03:08 | |
| *** dmellado has quit IRC | 03:08 | |
| *** dmellado_ is now known as dmellado | 03:08 | |
| *** Wasaac has quit IRC | 03:12 | |
| *** dosaboy has quit IRC | 03:22 | |
| *** jschlueter has quit IRC | 03:22 | |
| *** EmilienM has quit IRC | 03:22 | |
| *** ab-a has quit IRC | 03:22 | |
| *** egonzalez has quit IRC | 03:22 | |
| *** jbadiapa has quit IRC | 03:22 | |
| *** kklimonda has quit IRC | 03:22 | |
| *** Reepicheep has quit IRC | 03:22 | |
| *** rouk has quit IRC | 03:22 | |
| *** dasp has quit IRC | 03:22 | |
| *** hamzy has quit IRC | 03:22 | |
| *** sean-k-mooney has quit IRC | 03:22 | |
| *** VickyG has quit IRC | 03:22 | |
| *** jenglisch has quit IRC | 03:25 | |
| *** jenglisch has joined #openstack-kolla | 03:27 | |
| *** guozijn has quit IRC | 03:28 | |
| *** EmilienM has joined #openstack-kolla | 03:38 | |
| *** dosaboy has joined #openstack-kolla | 03:38 | |
| *** jschlueter has joined #openstack-kolla | 03:38 | |
| *** ab-a has joined #openstack-kolla | 03:38 | |
| *** egonzalez has joined #openstack-kolla | 03:38 | |
| *** jbadiapa has joined #openstack-kolla | 03:38 | |
| *** kklimonda has joined #openstack-kolla | 03:38 | |
| *** rouk has joined #openstack-kolla | 03:38 | |
| *** Reepicheep has joined #openstack-kolla | 03:38 | |
| *** dasp has joined #openstack-kolla | 03:38 | |
| *** hamzy has joined #openstack-kolla | 03:38 | |
| *** sean-k-mooney has joined #openstack-kolla | 03:38 | |
| *** VickyG has joined #openstack-kolla | 03:38 | |
| *** irclogbot_1 has quit IRC | 03:41 | |
| *** irclogbot_3 has joined #openstack-kolla | 03:42 | |
| openstackgerrit | Taeha Kim proposed openstack/kolla master: Add RHEL subscription registration https://review.opendev.org/495148 | 03:50 |
| *** zhanglong has quit IRC | 03:54 | |
| *** dasp has quit IRC | 03:58 | |
| *** Sravan has joined #openstack-kolla | 03:59 | |
| *** ykarel has joined #openstack-kolla | 04:10 | |
| *** igordc has joined #openstack-kolla | 04:12 | |
| *** guozijn has joined #openstack-kolla | 04:15 | |
| *** igordc has quit IRC | 04:19 | |
| *** dasp has joined #openstack-kolla | 04:21 | |
| *** guozijn has quit IRC | 04:25 | |
| *** skramaja has joined #openstack-kolla | 04:25 | |
| *** ykarel has quit IRC | 04:29 | |
| *** Sravan has quit IRC | 04:31 | |
| *** Sravan has joined #openstack-kolla | 04:31 | |
| *** Sravan has quit IRC | 04:36 | |
| *** janki has joined #openstack-kolla | 04:47 | |
| *** whoami-rajat has joined #openstack-kolla | 04:47 | |
| *** ykarel has joined #openstack-kolla | 04:48 | |
| *** shyamb has joined #openstack-kolla | 04:58 | |
| *** Sravan has joined #openstack-kolla | 05:03 | |
| *** Sravan has quit IRC | 05:07 | |
| *** pcaruana has joined #openstack-kolla | 05:09 | |
| *** shyamb has quit IRC | 05:12 | |
| *** shyamb has joined #openstack-kolla | 05:12 | |
| *** luksky has joined #openstack-kolla | 05:26 | |
| *** guozijn has joined #openstack-kolla | 05:40 | |
| *** guozijn has quit IRC | 05:48 | |
| *** jbadiapa has quit IRC | 05:49 | |
| *** guozijn has joined #openstack-kolla | 05:50 | |
| *** Sravan has joined #openstack-kolla | 05:58 | |
| *** Sravan has quit IRC | 06:03 | |
| *** dpawlik has joined #openstack-kolla | 06:04 | |
| *** ivve has joined #openstack-kolla | 06:06 | |
| gregoryo | Right, after months of lower level work, I now have a kolla-ansible OpenStack deployment working! \o/ | 06:07 |
| yoctozepto | gregoryo: congrats! | 06:11 |
| *** shyamb has quit IRC | 06:24 | |
| *** gregoryo has quit IRC | 06:35 | |
| *** rpittau|afk is now known as rpittau | 06:35 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Blacklist bandit 1.6.0 https://review.opendev.org/665312 | 06:38 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Fix lower-constraints test fail due to failing dep build https://review.opendev.org/665188 | 06:41 |
| *** guozijn has quit IRC | 06:44 | |
| *** shyamb has joined #openstack-kolla | 06:52 | |
| *** guozijn has joined #openstack-kolla | 06:58 | |
| *** shyamb has quit IRC | 07:00 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla-ansible master: Fixes dhcp-hostsdir permission for ironic-inspector https://review.opendev.org/665316 | 07:01 |
| *** shyamb has joined #openstack-kolla | 07:03 | |
| *** zhangfei has joined #openstack-kolla | 07:08 | |
| *** Sravan has joined #openstack-kolla | 07:19 | |
| *** cah_link has joined #openstack-kolla | 07:20 | |
| * yoctozepto curious why RDO CI check succeeded (https://review.opendev.org/#/c/495148/78) | 07:21 | |
| *** Sravan has quit IRC | 07:24 | |
| *** rgogunskiy has joined #openstack-kolla | 07:25 | |
| *** jbadiapa has joined #openstack-kolla | 07:27 | |
| *** shyamb has quit IRC | 07:28 | |
| *** shyamb has joined #openstack-kolla | 07:29 | |
| *** ktibi has joined #openstack-kolla | 07:42 | |
| *** gfidente has joined #openstack-kolla | 07:47 | |
| *** shyamb has quit IRC | 07:47 | |
| *** izza_ has quit IRC | 07:49 | |
| *** ykarel is now known as ykarel|afk | 07:52 | |
| *** Wasaac has joined #openstack-kolla | 07:53 | |
| *** ykarel|afk is now known as ykarel|lunch | 07:53 | |
| *** Wasaac has quit IRC | 07:54 | |
| *** Wasaac has joined #openstack-kolla | 07:54 | |
| *** Sravan has joined #openstack-kolla | 07:59 | |
| *** zbr|ooo is now known as zbr|ruck | 08:01 | |
| *** Sravan has quit IRC | 08:03 | |
| *** k_mouza has joined #openstack-kolla | 08:08 | |
| *** gfidente has quit IRC | 08:12 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Remove ``hnas_iscsi`` from the supported storage backends list of Cinder https://review.opendev.org/665325 | 08:14 |
| *** dougsz has joined #openstack-kolla | 08:15 | |
| *** gfidente has joined #openstack-kolla | 08:15 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Remove ``hnas_iscsi`` from the supported storage backends list of Cinder https://review.opendev.org/665325 | 08:16 |
| *** priteau has joined #openstack-kolla | 08:23 | |
| openstackgerrit | Maciej Kucia proposed openstack/kolla-ansible master: keystone: Prevent race in Fernet initialization https://review.opendev.org/665326 | 08:23 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Fix lower-constraints test fail due to failing dep build https://review.opendev.org/665188 | 08:27 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Debian/Ubuntu: allow to use unsigned repositories https://review.opendev.org/665199 | 08:30 |
| *** k_mouza has quit IRC | 08:32 | |
| *** k_mouza has joined #openstack-kolla | 08:33 | |
| *** ktibi has quit IRC | 08:35 | |
| harsha | hi, anyone there? | 08:40 |
| harsha | i want to set up docker registry, what images would i need to put in there? | 08:42 |
| harsha | ping yoctozepto, i think if docker image is locally available , the bridge would successfully be made and then deploy would continue. have to try | 08:44 |
| *** ktibi has joined #openstack-kolla | 08:45 | |
| *** shyamb has joined #openstack-kolla | 08:49 | |
| *** calbers has quit IRC | 08:53 | |
| *** Wasaac has quit IRC | 08:56 | |
| *** Wasaac has joined #openstack-kolla | 08:58 | |
| *** ykarel|lunch is now known as ykarel | 08:59 | |
| *** Wasaac has quit IRC | 08:59 | |
| *** Wasaac has joined #openstack-kolla | 08:59 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: doc: Start using openstackdoctheme's extlink extension https://review.opendev.org/665330 | 09:01 |
| *** guozijn has quit IRC | 09:02 | |
| *** Wasaac has quit IRC | 09:04 | |
| yoctozepto | harsha: hi, I doubt it would help | 09:08 |
| yoctozepto | local registry only optimizes deployment | 09:08 |
| yoctozepto | does not fix problems with connectivity in general | 09:08 |
| yoctozepto | you said that even from the local subnet there was not connectivity | 09:09 |
| harsha | yoctozepto: like the step its failing is when its trying download some image so i thought if its available locally it might solve the issue | 09:09 |
| harsha | yoctozepto: if i let the docker containers be as it is and restart , there is still no connectivity. | 09:10 |
| harsha | yoctozepto: but, i were to stop all the dockers and then restart the control node then the internet is back and i ping the machine too | 09:11 |
| harsha | yoctozepto: or i could just download that one container now itself and check | 09:15 |
| harsha | yoctozepto: the error: http://paste.openstack.org/show/752870/, is it trying to download any image though? | 09:17 |
| yoctozepto | harsha: yeah, it's trying to pull an image | 09:23 |
| harsha | yoctozepto: any idea which image? | 09:23 |
| *** Luzi has joined #openstack-kolla | 09:29 | |
| *** shyamb has quit IRC | 09:32 | |
| yoctozepto | it does not say it | 09:32 |
| mnasiadka | yoctozepto: RDO 3rd party CI just templates out all Dockerfiles and builts those tripleo uses ;) | 09:34 |
| yoctozepto | mnasiadka: but the Zuul's tripleo variant failed too :O | 09:41 |
| yoctozepto | looks as if they do not track the change in question but the branch | 09:42 |
| yoctozepto | the templating had to fail :P | 09:43 |
| yoctozepto | the base was b0rken | 09:43 |
| yoctozepto | then it does not make sense they run it for each proposal | 09:45 |
| *** Wasaac has joined #openstack-kolla | 09:46 | |
| yoctozepto | oh, they build fedora and we do not support fedora, they probably replace the base template | 09:53 |
| yoctozepto | for it | 09:53 |
| *** sm806 has quit IRC | 09:53 | |
| *** sm806 has joined #openstack-kolla | 09:53 | |
| *** sm806 has quit IRC | 09:54 | |
| *** sm806 has joined #openstack-kolla | 09:55 | |
| *** k_mouza_ has joined #openstack-kolla | 09:57 | |
| *** k_mouza has quit IRC | 10:00 | |
| *** Wasaac has quit IRC | 10:15 | |
| *** Wasaac_ has joined #openstack-kolla | 10:15 | |
| *** shyamb has joined #openstack-kolla | 10:20 | |
| harsha | yoctozepto: should i download all the neutron images listed in https://github.com/openstack/kolla/tree/master/docker/neutron ? | 10:21 |
| *** luksky has quit IRC | 10:26 | |
| *** guozijn has joined #openstack-kolla | 10:26 | |
| yoctozepto | harsha: you can't be sure it was neutron to be downloaded in the first place | 10:27 |
| yoctozepto | best set up a pull-through cache registry | 10:28 |
| yoctozepto | then it is local | 10:28 |
| yoctozepto | and downloads the image for you on demand | 10:28 |
| *** ktibi has quit IRC | 10:47 | |
| *** guozijn has quit IRC | 10:48 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Fix issues with Percona in Zuul and clean up a little bit https://review.opendev.org/665355 | 10:51 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Fix issues with Percona in Zuul and clean up a little bit https://review.opendev.org/665355 | 10:53 |
| *** jbadiapa has quit IRC | 10:57 | |
| *** k_mouza_ has quit IRC | 11:01 | |
| *** k_mouza has joined #openstack-kolla | 11:02 | |
| *** luksky has joined #openstack-kolla | 11:04 | |
| *** kplant has joined #openstack-kolla | 11:04 | |
| *** Sravan has joined #openstack-kolla | 11:07 | |
| *** schwicht has joined #openstack-kolla | 11:11 | |
| openstackgerrit | Chason Chan proposed openstack/kolla master: doc: Start using openstackdoctheme's extlink extension https://review.opendev.org/665361 | 11:11 |
| *** Sravan has quit IRC | 11:11 | |
| *** ktibi has joined #openstack-kolla | 11:26 | |
| *** Wasaac_ has quit IRC | 11:28 | |
| *** shyamb has quit IRC | 11:37 | |
| *** goldyfruit has quit IRC | 11:40 | |
| openstackgerrit | Chason Chan proposed openstack/kolla master: doc: Start using openstackdoctheme's extlink extension https://review.opendev.org/665361 | 11:43 |
| *** shyamb has joined #openstack-kolla | 11:43 | |
| *** iclon_ has joined #openstack-kolla | 11:45 | |
| *** EmilienM is now known as EvilienM | 11:45 | |
| *** iclon__ has quit IRC | 11:47 | |
| *** janki has quit IRC | 11:49 | |
| *** Luzi has quit IRC | 11:49 | |
| yoctozepto | guys, is there a way to make kolla-ansible copy a file/directory to a container - need it to copy my CA certificate | 11:53 |
| *** schwicht has quit IRC | 11:56 | |
| *** Wasaac has joined #openstack-kolla | 12:02 | |
| *** Wasaac has quit IRC | 12:07 | |
| *** shyamb has quit IRC | 12:16 | |
| *** shyam89 has joined #openstack-kolla | 12:16 | |
| *** schwicht has joined #openstack-kolla | 12:18 | |
| *** Wasaac has joined #openstack-kolla | 12:19 | |
| *** factor has joined #openstack-kolla | 12:23 | |
| *** Wasaac has quit IRC | 12:23 | |
| *** hamzy has quit IRC | 12:27 | |
| *** ktibi has quit IRC | 12:27 | |
| mnasiadka | yoctozepto: native one? not really | 12:27 |
| *** dave-mccowan has joined #openstack-kolla | 12:27 | |
| *** shyam89 has quit IRC | 12:30 | |
| *** dave-mccowan has quit IRC | 12:35 | |
| *** Wasaac has joined #openstack-kolla | 12:42 | |
| *** schwicht has quit IRC | 12:47 | |
| *** skramaja has quit IRC | 12:52 | |
| hjensas | yoctozepto: if you are using a distro, package the CA cert in an RPM, DEB package and use the interface to include packages? | 12:52 |
| mgoddard | yoctozepto: normally people build certs into their images | 12:52 |
| mgoddard | yoctozepto: typically by curling it from a local server during build | 12:53 |
| *** priteau has quit IRC | 12:59 | |
| *** Wasaac has quit IRC | 12:59 | |
| *** Wasaac has joined #openstack-kolla | 12:59 | |
| *** hamzy has joined #openstack-kolla | 13:02 | |
| *** Sravan has joined #openstack-kolla | 13:07 | |
| *** goldyfruit has joined #openstack-kolla | 13:08 | |
| *** Sravan has quit IRC | 13:11 | |
| *** ktibi has joined #openstack-kolla | 13:12 | |
| *** hamzy has quit IRC | 13:13 | |
| *** hamzy has joined #openstack-kolla | 13:17 | |
| *** ykarel is now known as ykarel|afk | 13:19 | |
| *** hamzy has quit IRC | 13:19 | |
| *** hamzy has joined #openstack-kolla | 13:20 | |
| *** goldyfruit has quit IRC | 13:23 | |
| openstackgerrit | Chason Chan proposed openstack/kolla master: doc: Start using openstackdoctheme's extlink extension https://review.opendev.org/665361 | 13:23 |
| *** goldyfruit has joined #openstack-kolla | 13:24 | |
| *** priteau has joined #openstack-kolla | 13:27 | |
| *** zhanglong has joined #openstack-kolla | 13:29 | |
| *** mchlumsky has joined #openstack-kolla | 13:31 | |
| *** schwicht has joined #openstack-kolla | 13:40 | |
| *** goldyfruit has quit IRC | 13:47 | |
| *** jbadiapa has joined #openstack-kolla | 13:50 | |
| *** goldyfruit has joined #openstack-kolla | 13:53 | |
| *** whoami-rajat has quit IRC | 13:57 | |
| *** Wasaac has quit IRC | 14:01 | |
| *** Wasaac_ has joined #openstack-kolla | 14:01 | |
| *** calbers has joined #openstack-kolla | 14:08 | |
| *** JamesBenson has joined #openstack-kolla | 14:08 | |
| yoctozepto | mnasiadka, hjensas, mgoddard: thanks for info | 14:13 |
| yoctozepto | mgoddard, mnasiadka, hrw: may we get the CI patching triptych rolling for stable/stein? https://review.opendev.org/665188 https://review.opendev.org/665199 https://review.opendev.org/665355 | 14:16 |
| yoctozepto | if we are lucky, no recheck will be needed :D | 14:16 |
| *** dpawlik has quit IRC | 14:17 | |
| yoctozepto | hate it already when gating fails due to random things | 14:17 |
| *** ykarel|afk is now known as ykarel|away | 14:27 | |
| *** priteau has quit IRC | 14:30 | |
| *** ykarel_ has joined #openstack-kolla | 14:30 | |
| *** ykarel_ has quit IRC | 14:30 | |
| *** ykarel|away has quit IRC | 14:32 | |
| *** hamzy has quit IRC | 14:36 | |
| *** hamzy has joined #openstack-kolla | 14:37 | |
| *** calbers has quit IRC | 14:40 | |
| *** calbers has joined #openstack-kolla | 14:40 | |
| *** zhanglong has quit IRC | 14:41 | |
| *** lemko has joined #openstack-kolla | 14:50 | |
| *** Wasaac_ has quit IRC | 14:59 | |
| *** hamzy has quit IRC | 15:01 | |
| *** hamzy has joined #openstack-kolla | 15:02 | |
| *** yoctozepto has quit IRC | 15:03 | |
| *** yoctozepto has joined #openstack-kolla | 15:04 | |
| mgoddard | yoctozepto: do they need to go back to other branches? | 15:04 |
| *** Wasaac has joined #openstack-kolla | 15:04 | |
| yoctozepto | mgoddard: nope, only stein: xtrabackup (Percona) is stein, ubuntu bionic is also only stein | 15:07 |
| yoctozepto | hence stein and done | 15:07 |
| yoctozepto | oh, well, the sphinx one could be relevant but I am not sure :/ | 15:08 |
| *** luksky has quit IRC | 15:08 | |
| mgoddard | yoctozepto: what about cryptography? | 15:11 |
| mgoddard | rocky should support bionic | 15:11 |
| yoctozepto | mgoddard: this is bionic | 15:13 |
| yoctozepto | oh, well | 15:13 |
| yoctozepto | but CI applies xenial for rocky | 15:13 |
| yoctozepto | if rocky is supposed to really support bionic as build host | 15:14 |
| yoctozepto | then yeah, let's backport | 15:14 |
| yoctozepto | mgoddard: how many releases behind do we support in kolla? | 15:14 |
| *** calbers has quit IRC | 15:17 | |
| *** calbers has joined #openstack-kolla | 15:19 | |
| *** gfidente has quit IRC | 15:21 | |
| *** ivve has quit IRC | 15:29 | |
| *** ktibi has quit IRC | 15:31 | |
| mgoddard | yoctozepto: releases of kolla? currently queens, rocky, stein (when released) | 15:36 |
| mgoddard | pike is in extended maintenance | 15:36 |
| mgoddard | i.e. no new releases, but CI still in place for backports as required | 15:36 |
| *** rgogunskiy has quit IRC | 15:39 | |
| *** zhangfei has quit IRC | 15:43 | |
| yoctozepto | most issues are coming from queens and rocky, so let's ignore pike then and backport to the other two | 15:50 |
| *** Wasaac has quit IRC | 15:56 | |
| *** hjensas is now known as hjensas|afk | 16:01 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Do not create Keystone domain-specific config directory if config not supplied https://review.opendev.org/665423 | 16:07 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Do not create Keystone domain-specific config directory if useless https://review.opendev.org/665423 | 16:08 |
| *** rpittau is now known as rpittau|afk | 16:11 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/rocky: Fix lower-constraints test fail due to failing dep build https://review.opendev.org/665427 | 16:21 |
| *** dcapone2004 has joined #openstack-kolla | 16:21 | |
| *** goldyfruit has quit IRC | 16:24 | |
| yoctozepto | eh, queens does not want it cherry-picked via gerrit | 16:25 |
| *** guozijn has joined #openstack-kolla | 16:30 | |
| *** Wasaac has joined #openstack-kolla | 16:31 | |
| *** goldyfruit has joined #openstack-kolla | 16:32 | |
| *** Wasaac has quit IRC | 16:36 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/queens: Fix lower-constraints test fail due to failing dep build https://review.opendev.org/665432 | 16:38 |
| yoctozepto | the title is false but I did not want to change it to avoid some confusion (introducing other) | 16:39 |
| *** k_mouza has quit IRC | 16:44 | |
| rouk | how would i target a reconfigure of just kolla_toolbox? | 16:49 |
| rouk | its not a role, just a task within common | 16:49 |
| *** Sravan has joined #openstack-kolla | 16:52 | |
| *** guozijn has quit IRC | 16:52 | |
| yoctozepto | rouk: I don't know the answer and too busy to check but what is your usecase? | 16:56 |
| *** Sravan has quit IRC | 16:56 | |
| rouk | i need to deploy new kolla-toolbox containers, and its part of the common role, but limiting to common tag doesnt touch it. so i guess im forced to do a full node reconfigure? | 16:57 |
| *** dougsz has quit IRC | 16:57 | |
| kplant | my cinder_backup container is restarting indefinitely on stable/stein, seems it's missing ceph client? http://paste.openstack.org/show/752941/ | 16:57 |
| kplant | is there a flag i'm missing in globals.yml? | 16:57 |
| *** Sravan has joined #openstack-kolla | 16:57 | |
| rouk | kplant: Update backup_driver from "cinder.backup.drivers.ceph" to "cinder.backup.drivers.ceph.CephBackupDriver" | 16:58 |
| *** guozijn has joined #openstack-kolla | 17:00 | |
| *** lemko has quit IRC | 17:00 | |
| yoctozepto | rouk: I guess noone thought of it since kolla-toolbox is internal to kolla-ansible | 17:01 |
| kplant | rouk: thank you, giving that a shot | 17:01 |
| yoctozepto | so it's deployed when kolla-ansible needs it | 17:01 |
| kplant | https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html#cinder needs to be fixed :-) | 17:02 |
| yoctozepto | kplant: yeah, they changed it in the meantime | 17:03 |
| yoctozepto | I guess we are better off pointing to projects docs | 17:03 |
| yoctozepto | especially in this case | 17:04 |
| yoctozepto | see https://docs.openstack.org/cinder/latest/configuration/block-storage/backup/ceph-backup-driver.html | 17:04 |
| yoctozepto | you should really read project docs for ini config details | 17:04 |
| yoctozepto | as kolla does not parse it | 17:04 |
| kplant | fair point | 17:04 |
| yoctozepto | well, it does for merging them | 17:04 |
| yoctozepto | but not for interpretation | 17:05 |
| yoctozepto | kplant: may you please report a bug on kolla-ansible for these docs? | 17:05 |
| yoctozepto | so that i don't forget | 17:05 |
| kplant | sure | 17:06 |
| kplant | done | 17:10 |
| *** Sravan has quit IRC | 17:17 | |
| *** rouk has quit IRC | 17:17 | |
| *** Sravan has joined #openstack-kolla | 17:17 | |
| kplant | i have another stupid question.. if i want to override policies for glance, would it be best to generate the policy.json with oslo or should i try to grab it from git if i can? | 17:17 |
| *** Sravan has quit IRC | 17:18 | |
| *** igordc has joined #openstack-kolla | 17:22 | |
| *** hamzy has quit IRC | 17:23 | |
| *** priteau has joined #openstack-kolla | 17:23 | |
| *** hamzy has joined #openstack-kolla | 17:24 | |
| *** Sravan has joined #openstack-kolla | 17:35 | |
| klindgren | Anyone have some idea's on how to make kolla-ansible more performant and less of a memory hog? I am seeing bug reports around include_tasks as well and include_role. But with kolla-ansible we have ~400 hosts in one deployment. The deployment now takes ~4.5 hours to complete. Each ansible worker towards the end consumes ~2.6GB of ram, and All we are deploying is: Nova, placement, keystone, neutron, glance, common, chrony. | 17:42 |
| klindgren | Nova, neutron and common take the most time to run. ~45-50 minutes each. This is on a redeployment where nothing is actually changed. IE no config updates, no new image act ect. So basically everything should be noop-operations. | 17:43 |
| klindgren | I do sometimes see restarts of services due to config changes, which I believe is do to the fact that people are using loops through hosts in groups. But because under the hood ansible doesn't use ordered dicts, the ordering of hosts is somewhat random and we see unneeded config updates and service restarts due to that. | 17:45 |
| *** hamzy has quit IRC | 17:47 | |
| *** hamzy has joined #openstack-kolla | 17:56 | |
| *** JamesBenson has quit IRC | 18:04 | |
| *** JamesBenson has joined #openstack-kolla | 18:13 | |
| *** graysonh has joined #openstack-kolla | 18:15 | |
| *** JamesBenson has quit IRC | 18:18 | |
| openstackgerrit | Kris Lindgren proposed openstack/kolla-ansible master: Add a job that *only* deploys updated containers https://review.opendev.org/645966 | 18:26 |
| yoctozepto | kplant: I would take it from the relevant git branch | 18:26 |
| *** JamesBenson has joined #openstack-kolla | 18:31 | |
| *** dpawlik has joined #openstack-kolla | 18:34 | |
| *** hamzaachi has joined #openstack-kolla | 18:35 | |
| *** JamesBenson has quit IRC | 18:35 | |
| *** k_mouza has joined #openstack-kolla | 18:41 | |
| *** JamesBenson has joined #openstack-kolla | 18:46 | |
| *** k_mouza has quit IRC | 18:46 | |
| *** JamesBenson has quit IRC | 18:47 | |
| *** Sravan has quit IRC | 18:47 | |
| *** JamesBenson has joined #openstack-kolla | 18:47 | |
| *** Sravan has joined #openstack-kolla | 18:47 | |
| kplant | yoctozepto: ty | 18:56 |
| *** dims has quit IRC | 18:57 | |
| *** hamzy has quit IRC | 19:05 | |
| *** dims has joined #openstack-kolla | 19:05 | |
| *** schwicht has quit IRC | 19:05 | |
| *** Sravan has quit IRC | 19:05 | |
| *** Sravan has joined #openstack-kolla | 19:06 | |
| *** Sravan has quit IRC | 19:12 | |
| *** graysonh has quit IRC | 19:16 | |
| *** goldyfruit has quit IRC | 19:17 | |
| *** goldyfruit has joined #openstack-kolla | 19:17 | |
| *** goldyfruit has quit IRC | 19:22 | |
| *** priteau has quit IRC | 19:26 | |
| *** hamzaachi has quit IRC | 19:35 | |
| *** hamzaachi has joined #openstack-kolla | 19:37 | |
| *** schwicht has joined #openstack-kolla | 19:37 | |
| *** pcaruana has quit IRC | 19:39 | |
| *** jbadiapa has quit IRC | 19:40 | |
| *** iclon__ has joined #openstack-kolla | 19:45 | |
| *** hamzy has joined #openstack-kolla | 19:45 | |
| *** iclon_ has quit IRC | 19:47 | |
| *** luksky has joined #openstack-kolla | 19:56 | |
| *** factor has quit IRC | 20:05 | |
| *** hamzaachi has quit IRC | 20:14 | |
| *** hamzaachi has joined #openstack-kolla | 20:15 | |
| *** kplant has quit IRC | 20:22 | |
| *** hamzaachi has quit IRC | 20:23 | |
| *** hamzaachi has joined #openstack-kolla | 20:24 | |
| yoctozepto | klindgren: maybe too much info is collected, kolla-ansible only calls ansible with its playbooks, so it's either the ansible itself or the way the playbooks are written | 20:25 |
| *** schwicht has quit IRC | 20:25 | |
| yoctozepto | > <klindgren> I do sometimes see restarts of services due to config changes, which I believe is do to the fact that people are using loops through hosts in groups. | 20:26 |
| yoctozepto | probably lacking in idempotence | 20:26 |
| yoctozepto | it's easy to make a check always "change" | 20:26 |
| yoctozepto | does your proposed method help with any of these issues? | 20:27 |
| *** jonaspaulo has quit IRC | 20:28 | |
| klindgren | When I looked at this on the services that are disruptive. IE rabbitmq: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/rabbitmq.conf.j2#L12-L14 | 20:28 |
| *** goldyfruit has joined #openstack-kolla | 20:28 | |
| klindgren | Loops through hosts that are in a group, the issue especially if you use a yam based inventory, is that hosts are in a dict, which is not ordered. So the ordering of hosts changes, which causes config to change. When infact they are effectively the same config, just host ordering is now different. | 20:29 |
| klindgren | So that triggers a restart of rabbitmq, for no good reason | 20:29 |
| yoctozepto | klindgren: have you confirmed the order changes? as in you inspected the file? | 20:29 |
| klindgren | Yes, locally I had a patch to fix that, which stopped most of the restarts. | 20:30 |
| yoctozepto | you are more than welcome to share it, I will look at my deployment to see if I experience this | 20:30 |
| yoctozepto | we should strive to make it as reproducible and idempotent as possible | 20:31 |
| yoctozepto | random things are the worst :P | 20:31 |
| klindgren | Under the way that rabbitmq clustering use to be configured had to do something like: | 20:37 |
| JamesBenson | hey | 20:37 |
| klindgren | https://www.paste.org/99112 | 20:37 |
| JamesBenson | We had a dirty shutdown in our datacenter | 20:38 |
| klindgren | To sort the hosts in the groups, set a var to handle the first host in that group, where its used later on. | 20:38 |
| JamesBenson | but it seems like our VM's are having issues with the disk? | 20:38 |
| JamesBenson | and one is stuck in a: task: "Powering Off", status: active, power state: Shut Down | 20:39 |
| JamesBenson | any idea's on how to repair? | 20:39 |
| klindgren | JamesBenson, can you be more specific? Issues with disk is pretty unspecific | 20:39 |
| JamesBenson | yes, sorry | 20:39 |
| klindgren | re: vm's with actions in them that are stuck, use reset state | 20:40 |
| klindgren | with --active, which will face them to active, then you can do shutdown/start actions | 20:40 |
| klindgren | S/face/force/ | 20:41 |
| JamesBenson | @klindgren https://paste.ubuntu.com/p/KbBC9tTMcC/ | 20:41 |
| klindgren | Ouch, looks like something terrible happened with the vm's filesystem | 20:45 |
| JamesBenson | yeah, I think it's ceph related.... | 20:46 |
| JamesBenson | they are all like that. | 20:46 |
| JamesBenson | I found this: https://bugs.launchpad.net/nova/+bug/1781878 | 20:46 |
| openstack | Launchpad bug 1781878 in OpenStack Compute (nova) "VM fails to boot after evacuation when it uses ceph disk" [Undecided,Invalid] | 20:46 |
| JamesBenson | I should issue that through ceph_mon docker image... | 20:47 |
| JamesBenson | ? | 20:47 |
| klindgren | I don't use ceph, so I can't help here. | 20:48 |
| JamesBenson | okay | 20:48 |
| JamesBenson | on a different note, my other cluster, mariadb isn't recovering? | 20:48 |
| JamesBenson | any ideas on how to remedy? | 20:48 |
| JamesBenson | It's failing on: TASK [mariadb : Waiting for MariaDB service to be ready] | 20:48 |
| JamesBenson | the mariadb docker containers dont like to start... | 20:50 |
| klindgren | Docker logs of the marinade container can be helpful | 20:50 |
| klindgren | When they don't start | 20:50 |
| klindgren | look like that's just running: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/tasks/check.yml#L4 | 20:51 |
| klindgren | A docker exec command for show databases; | 20:51 |
| * klindgren unchecks "correct spelling automatically..... | 20:51 | |
| klindgren | docker logs mariadb (I think thats the container name) | 20:52 |
| JamesBenson | it is | 20:52 |
| klindgren | we use external mysql as well. | 20:52 |
| kklimonda | @JamesBenson for your disk issue, check if there are locks on rbds | 20:52 |
| JamesBenson | how? | 20:53 |
| kklimonda | @JamesBenson if so, make sure that your ceph users for nova and cinder both have capability to blacklist clients | 20:53 |
| JamesBenson | https://paste.ubuntu.com/p/pTNY5mmThx/ | 20:53 |
| JamesBenson | oh boy: sudo: unknown uid 42434: who are you? | 20:54 |
| kklimonda | @JamesBenson `rbd lock list <pool name>/<image name>` | 20:54 |
| *** goldyfruit_ has joined #openstack-kolla | 20:54 | |
| kklimonda | you can list images from pool with `rbd list <pool name>` | 20:54 |
| kklimonda | iirc | 20:54 |
| JamesBenson | https://paste.ubuntu.com/p/qQQhFT2Ndc/ | 20:54 |
| kklimonda | also, you can check if your ceph users have correct caps: it should be either `profile rbd` or `allow command 'osd blacklist'` on mon | 20:55 |
| *** goldyfruit has quit IRC | 20:56 | |
| *** mchlumsky has quit IRC | 20:56 | |
| JamesBenson | https://paste.ubuntu.com/p/DSRbkfzCyh/ | 20:56 |
| JamesBenson | those 3 pastes are mariadb logs from 3 servers. | 20:57 |
| kklimonda | well, that's unusual | 20:57 |
| JamesBenson | the last one was too long so I truncated it. | 20:57 |
| *** hamzaachi has quit IRC | 20:59 | |
| kklimonda | @JamesBenson I assume the container is not running, so you can't exec into it to debug it further? | 20:59 |
| kklimonda | @JamesBenson when it happens to me, I edit config.json for the container in question, changing command to "/bin/sleep infinity" and the lets me exec into the container and see what's wrong | 21:00 |
| *** goldyfruit_ has quit IRC | 21:00 | |
| JamesBenson | the mariadb containers are not running | 21:00 |
| *** goldyfruit_ has joined #openstack-kolla | 21:01 | |
| kklimonda | it kinda sounds like you have some corrupted files in the container | 21:01 |
| kklimonda | the "who are you?" sudo message happens when you can't match uid to the user, which I believe is done by going through /etc/passwd | 21:02 |
| kklimonda | the other one is even weirder | 21:02 |
| JamesBenson | yeah, it mentioned it didn't know nova in the /etc/passwd file | 21:07 |
| JamesBenson | which I didn't think it put anyone there.... | 21:08 |
| *** Sravan has joined #openstack-kolla | 21:12 | |
| klindgren | did you control plane servers also have filesystem corruption? IE did stuff get placed into /lost+found? | 21:12 |
| *** zbr|ruck has quit IRC | 21:13 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add kolla-build-debian-source-arm64 experimental job https://review.opendev.org/557659 | 21:14 |
| *** schwicht has joined #openstack-kolla | 21:21 | |
| *** zbr has joined #openstack-kolla | 21:30 | |
| *** schwicht has quit IRC | 21:56 | |
| *** goldyfruit___ has joined #openstack-kolla | 21:57 | |
| *** goldyfruit_ has quit IRC | 22:00 | |
| *** Sravan has quit IRC | 22:00 | |
| *** EvilienM is now known as EmilienM | 22:03 | |
| *** luksky has quit IRC | 22:08 | |
| *** Sravan has joined #openstack-kolla | 22:11 | |
| *** dpawlik has quit IRC | 22:29 | |
| *** Sravan has quit IRC | 22:37 | |
| *** cah_link1 has joined #openstack-kolla | 22:41 | |
| *** cah_link has quit IRC | 22:42 | |
| *** cah_link1 is now known as cah_link | 22:42 | |
| *** Sravan has joined #openstack-kolla | 22:44 | |
| *** igordc has quit IRC | 22:45 | |
| *** dpawlik has joined #openstack-kolla | 22:45 | |
| *** Sravan has quit IRC | 22:49 | |
| *** dpawlik has quit IRC | 22:50 | |
| *** JamesBenson has quit IRC | 22:50 | |
| *** igordc has joined #openstack-kolla | 23:13 | |
| *** goldyfruit___ has quit IRC | 23:19 | |
| *** cah_link1 has joined #openstack-kolla | 23:24 | |
| *** cah_link has quit IRC | 23:26 | |
| *** cah_link1 is now known as cah_link | 23:26 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!