| opendevreview | Piotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication https://review.opendev.org/c/openstack/kolla-ansible/+/805954 | 05:41 |
|---|---|---|
| ignaziocassano | yoctozepto: I'have just tested the patch you sent me | 06:29 |
| ignaziocassano | yoctozepto: PS Good morning | 06:29 |
| ignaziocassano | yoctozepto: https://paste.openstack.org/show/808331/ | 06:34 |
| yoctozepto | ignaziocassano: good morning; ok, so it behaved like I imagined regarding machined; but it seems your problem is unrelated to machined; thus, the search for fix continues... | 06:59 |
| yoctozepto | I wonder if it's some docker/cgroups interaction that shuts them all down on container exit | 07:00 |
| parallax | morning | 07:02 |
| ignaziocassano | yoctozepto: would you like I forward the email received from Sean Mooney ....he talked about cgroups | 07:02 |
| *** amoralej|off is now known as amoralej | 07:02 | |
| yoctozepto | ignaziocassano: I saw their mail but I did not dive deep in there; you can ping Sean on IRC but they are in some US timezone ;-) | 07:03 |
| ignaziocassano | yoctozepto: I've already sent an email to him but no answer. It sent me an opened red hat bug | 07:05 |
| ignaziocassano | yoctozepto: I did not understand the purpose of your patch. Anycase if the libvirt restar causes the vm shutdown I think we should mark it as a bug . I do not know how the community work . I wonder kolla coomunity should work with nova/cinder ans so on developers | 07:12 |
| yoctozepto | ignaziocassano: yeah, you can report a bug as it should not be happening (and I can assure you it does not happen to me ;-) ) | 07:14 |
| yoctozepto | https://bugs.launchpad.net/kolla-ansible | 07:14 |
| yoctozepto | perhaps the very latest docker broke or something | 07:14 |
| ignaziocassano | yoctozepto: I am going to open the bug | 07:15 |
| yoctozepto | ok | 07:20 |
| *** rpittau|afk is now known as rpittau | 07:24 | |
| ignaziocassano | yoctozepto: https://bugs.launchpad.net/kolla-ansible/+bug/1941706 | 07:28 |
| ignaziocassano | yoctozepto: Can I deplpy again without your patch to have a standard deplpoyment ? | 07:28 |
| opendevreview | Piotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication https://review.opendev.org/c/openstack/kolla-ansible/+/805954 | 07:38 |
| ignaziocassano | yoctozepto: I asked if I can redeploy without your patch because I wonder if you need I make further tests | 07:44 |
| mgoddard | yoctozepto: the lowlights joke was that we might advertise lowlights in PR | 07:57 |
| *** ykarel is now known as ykarel|lunch | 08:28 | |
| hrw | morning | 08:47 |
| ignaziocassano | Hello All, I discovered another issue with cinder netapp nfs driver. If snapshot visibility is enabled on netapp, when I restart docker container it stops cinder services because it tries to mount snapshots and change its owner. Since snapshots are read only, cinder volume does not restart. The workaround is disabling snapshot visibility on netapp. | 09:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage https://review.opendev.org/c/openstack/kolla-ansible/+/797005 | 09:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage https://review.opendev.org/c/openstack/kolla-ansible/+/797005 | 09:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs https://review.opendev.org/c/openstack/kolla-ansible/+/797017 | 09:14 |
| yoctozepto | ignaziocassano: yes, you can redeploy without the patch now | 09:35 |
| yoctozepto | mgoddard: in Puerto Rico? :D | 09:37 |
| ignaziocassano | yoctozepto: thanks | 09:48 |
| *** ykarel|lunch is now known as ykarel | 10:18 | |
| yoctozepto | dtantsur: the kolla-bifrost is green now; please continue to monitor; thank you! | 11:03 |
| dtantsur | great :) | 11:11 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage https://review.opendev.org/c/openstack/kolla-ansible/+/797005 | 11:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage https://review.opendev.org/c/openstack/kolla-ansible/+/797005 | 11:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x https://review.opendev.org/c/openstack/kolla/+/804513 | 11:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs https://review.opendev.org/c/openstack/kolla-ansible/+/797017 | 12:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Bump up Ansible to 2.11 https://review.opendev.org/c/openstack/kolla-ansible/+/796758 | 12:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Bump up Ansible to 2.11 https://review.opendev.org/c/openstack/kolla-ansible/+/796758 | 12:06 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x https://review.opendev.org/c/openstack/kolla/+/804513 | 12:15 |
| mihail | is this the correct place to ask for help with kayobe? | 12:55 |
| parallax | mihail: yes | 12:56 |
| mihail | i am trying to deploy an all in one kayobe and use ceph as the storage backend for cinder glance and nova, but when i run kayobe overcloud services deploy the ceph.conf is deleted from kayobe-config/etc/kolla/config/glance, cinder and nova. why is that? | 12:57 |
| mihail | i followed the kolla-ansible external ceph guide | 12:58 |
| mgoddard | mihail: you shouldn't touch files in etc/kolla/ | 12:59 |
| yoctozepto | headsup: monasca no longer leaderless | 12:59 |
| mgoddard | mihail: instead use etc/kayobe/kolla/ | 12:59 |
| *** amoralej is now known as amoralej|lunch | 12:59 | |
| mihail | mgoddard: so i should create the etc/kayobe/kolla/config/ directory and add only the ceph.conf in glance/ cinder/ and nova/? | 13:00 |
| mgoddard | mihail: yes | 13:00 |
| mihail | thank you | 13:01 |
| yoctozepto | only sahara and adjutant remain entirely leaderless; zun with odd situation (ptl candidate not contributing) | 13:02 |
| mgoddard | yoctozepto: adrian is always late with adjutant | 13:10 |
| yoctozepto | mgoddard: the question is whether openstack tc wants adjutant in its current state as it has done almost nothing during xena | 13:10 |
| yoctozepto | well, "almost nothing" here meaning really nothing | 13:11 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage https://review.opendev.org/c/openstack/kolla-ansible/+/797005 | 13:11 |
| yoctozepto | mgoddard: https://www.stackalytics.io/?module=adjutant-group hyper sad | 13:12 |
| mgoddard | yoctozepto: yeah, was just looking | 13:12 |
| yoctozepto | mgoddard: https://etherpad.opendev.org/p/yoga-leaderless for an overview | 13:12 |
| mephmanx_ | All, I am having an issue with an openstack wallaby install of openstack using kolla. I am seeing an error with elastic search and no logs are being added. here is the error: | 13:17 |
| mephmanx_ | {"level":"ERROR","loggerName":"logstash.outputs.elasticsearch","timeMillis":1629936010246,"thread":"Ruby-0-Thread-9: :1","logEvent":{"message":"Unable to get license information","url":"http://10.0.200.21:9200/","exception":{"metaClass":{"metaClass":{"exception":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError","message":"Got response code '400' contacting Elasticsearch at URL 'http://10.0.200. | 13:17 |
| mephmanx_ | From what I have found it sounds like a licensing issue with version of elsasticsearch above 7.10 and it looks like kolla openstack wallaby is deploying 7.10.2. | 13:17 |
| yoctozepto | license information? oh my | 13:17 |
| mephmanx_ | My version output: | 13:17 |
| mephmanx_ | { "name" : "10.0.200.20", "cluster_name" : "kolla_logging", "cluster_uuid" : "MAinUYsrS-evlWA35ro3dg", "version" : { "number" : "7.10.2", "build_flavor" : "oss", "build_type" : "rpm", "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9", "build_date" : "2021-01-13T00:42:12.435326Z", "build_snapshot" : false, "lucene_version" : "8.7.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_ve | 13:17 |
| yoctozepto | poor elasticsearch :-( | 13:18 |
| yoctozepto | parallax: you might know more ^ | 13:18 |
| mnasiadka | yoctozepto: we noticed yesterday that elasticsearch python client is too new to support what we are using :D | 13:18 |
| yoctozepto | mnasiadka: :D | 13:19 |
| mnasiadka | parallax: did you raise a bug about that? | 13:19 |
| yoctozepto | you mean too old likely | 13:19 |
| mnasiadka | yoctozepto: no, it only supports Elastic non-OSS :) | 13:19 |
| yoctozepto | but here I understand the server broke at all | 13:19 |
| yoctozepto | mnasiadka: oh my | 13:19 |
| mnasiadka | yoctozepto: a.k.a. Elastic chose to screw with OSS people | 13:19 |
| yoctozepto | argh | 13:19 |
| yoctozepto | all hope in amazon's fork | 13:20 |
| parallax | 7.10.2 is the correct version | 13:20 |
| parallax | working fine in master, haven't checked wallaby | 13:20 |
| mnasiadka | parallax: but curator doesn't work? | 13:24 |
| parallax | mephmanx_: downgrade logstash to 7.10.x, it will work then | 13:25 |
| parallax | curator's fine with elasticsearch<7.14 just like in the GitHub issue mentioned in https://bugs.launchpad.net/kolla-ansible/+bug/1941073 | 13:27 |
| mephmanx_ | Im on 7.10.2.... I guess down to 7.10.0? I deploy using kolla... Im not hugely familiar with the internals.... ma I able to set something inside the globals.yml to force that version? | 13:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage https://review.opendev.org/c/openstack/kolla-ansible/+/797005 | 13:27 |
| parallax | mephmanx_: can you double check? in kolla master, centos build gets logstash-oss-1:7.14.0-1.x86_64 | 13:37 |
| parallax | that's incompatible with oss elasticsearch 7.10.2 also from master - basically, that's a Kolla bug | 13:38 |
| parallax | as both should stay on 7.10.x | 13:39 |
| mephmanx_ | So its not me? | 13:39 |
| parallax | mephmanx_: you could revert to using fluentd's elasticsearch output as well | 13:40 |
| mephmanx_ | Im not sure how to do that with the kolla install / deploy... everything is in docker containers and Im not a docker expert either... | 13:41 |
| parallax | correct me if I'm wrong... isn't logstash part of Monasca which we..... are dropping? | 13:46 |
| yoctozepto | are we dropping Monasca? | 13:47 |
| mephmanx_ | hmm...ok. I do have monasca enable in my globals.yml. I should turn that off? | 13:47 |
| yoctozepto | logstash is part of the "default" ELK stack | 13:47 |
| yoctozepto | we use EFK | 13:47 |
| yoctozepto | replacing logstash with fluentd | 13:47 |
| mnasiadka | yoctozepto: we as the company are starting to not use it, we might consider thinking if we really want to support it - or at least deprecate it. | 13:48 |
| mnasiadka | the second we is the Kolla community | 13:48 |
| yoctozepto | mnasiadka: ++, lovely decision | 13:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs https://review.opendev.org/c/openstack/kolla-ansible/+/797017 | 13:48 |
| yoctozepto | mnasiadka: I'm all in | 13:48 |
| parallax | mephmanx_: you'd better disabling, could replace that with enable_central_logging: "yes" - get's you EFK stack working as yoctozepto suggests | 13:49 |
| mephmanx_ | if I turn that off and redeploy my env, might that resolve it? | 13:49 |
| mephmanx_ | ok, that works for me. | 13:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs https://review.opendev.org/c/openstack/kolla-ansible/+/797017 | 13:49 |
| yoctozepto | mephmanx_: could you also report the bug to https://bugs.launchpad.net/kolla ? | 13:49 |
| yoctozepto | mostly to track it | 13:50 |
| mephmanx_ | I can. Thank you. Ill rebuild my env later today and report how it goes. | 13:50 |
| *** amoralej|lunch is now known as amoralej | 13:50 | |
| yoctozepto | :-) | 13:50 |
| mnasiadka | yoctozepto: Ansible is a mess - Ansible 3.0 includes ansible-base 2.10 (so no change really for us), Ansible 4.0 includes ansible-base 2.11 (and requires python 3.8 more or less) | 14:00 |
| mnasiadka | so what do we bump to? :) | 14:01 |
| yoctozepto | mnasiadka: first of all - can we use base/core without the bloat? | 14:01 |
| mnasiadka | yoctozepto: we can, but I think in X we should just bump to Ansible 4.0 (base 2.11), and in Y get rid of ansible package, just install required collections and ansible-base | 14:02 |
| mnasiadka | and we agreed to do the collections+base in kolla-toolbox in X | 14:02 |
| mnasiadka | 3.* doesn't receive any minor updates anymore - so moving from unsupported to unsupported is something weird to do | 14:03 |
| mgoddard | mnasiadka: remember we have max and min | 14:05 |
| mgoddard | py3.8 is quite a strict requirement | 14:05 |
| mnasiadka | mgoddard: yes, I know - but in reality we're just bumping ansible-core 2.10 to ansible-core 2.11 | 14:05 |
| mnasiadka | (and the associated collections) | 14:06 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Bump up Ansible to 4.x / ansible-core 2.11 https://review.opendev.org/c/openstack/kolla-ansible/+/796758 | 14:06 |
| mgoddard | centos is py3.6 by default | 14:06 |
| mnasiadka | let's see where this fails, and we'll decide what to do | 14:07 |
| yoctozepto | mnasiadka: is it hard to drop collections layer in k-a? (just asking) | 14:24 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage https://review.opendev.org/c/openstack/kolla-ansible/+/797005 | 14:36 |
| opendevreview | Merged openstack/kayobe stable/wallaby: Add twentyFiveGigE to port descriptions in inspector rules https://review.opendev.org/c/openstack/kayobe/+/805763 | 14:45 |
| yoctozepto | does stackhpc support sahara? | 14:54 |
| * yoctozepto wears tc hat and queries the community about it | 14:54 | |
| yoctozepto | mnasiadka / mgoddard: ^ question re sahara | 14:55 |
| mgoddard | yoctozepto: We spent some time with it early on, but not much since | 14:56 |
| yoctozepto | mgoddard; thanks! | 14:57 |
| mgoddard | added a new priority to the list: Bump Ceph packages to Octopus | 15:02 |
| mgoddard | Nautilus soon to be EOL... | 15:02 |
| mgoddard | also added some "reviewers" fields to the priorities | 15:04 |
| mgoddard | mnasiadka, yoctozepto, hrw, osmanlicilegi: please sign up to review those patches you are interested in | 15:06 |
| yoctozepto | ok | 15:10 |
| yoctozepto | remind me if I forget | 15:10 |
| *** amoralej is now known as amoralej|off | 15:10 | |
| hrw | mgoddard: Debian has Nautilus only so far | 15:16 |
| yoctozepto | duh | 15:18 |
| hrw | centos has even Pacific ;D | 15:20 |
| yoctozepto | yeah, that's why nautilus is going buh-bye I guess | 15:24 |
| *** rpittau is now known as rpittau|afk | 16:02 | |
| opendevreview | Merged openstack/kayobe stable/victoria: Add twentyFiveGigE to port descriptions in inspector rules https://review.opendev.org/c/openstack/kayobe/+/805764 | 16:17 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Overcloud introspection data save broken https://review.opendev.org/c/openstack/kayobe/+/805263 | 16:18 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/791304 | 16:18 |
| opendevreview | Merged openstack/kayobe master: Fix documentation links for routed control plane networks https://review.opendev.org/c/openstack/kayobe/+/804398 | 16:18 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 17:48 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 17:49 |
| mephmanx | To all just an update. I rebuilt the env without monasca and kibana DOES work now. Thank you very much! | 18:15 |
| mephmanx | update. Kibana works fine but grafana is now gone. I looked into kolla and see this: enable_grafana: "{{ enable_monasca | bool }}" | 18:20 |
| mephmanx | So it looks like by default grafana is not enabled if monasca is turned off. If I enable grafana without monasca I assume things will still work? The 2 arent tied together, correct? Also, is there something better than grafana or is it required for monasca but otherwise its if desired? | 18:21 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!