Thursday, 2021-08-26

opendevreviewPiotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication  https://review.opendev.org/c/openstack/kolla-ansible/+/80595405:41
ignaziocassano yoctozepto: I'have just tested the patch you sent me06:29
ignaziocassano yoctozepto: PS Good morning06:29
ignaziocassano yoctozepto: https://paste.openstack.org/show/808331/06:34
yoctozeptoignaziocassano: good morning; ok, so it behaved like I imagined regarding machined; but it seems your problem is unrelated to machined; thus, the search for fix continues...06:59
yoctozeptoI wonder if it's some docker/cgroups interaction that shuts them all down on container exit07:00
parallaxmorning07:02
ignaziocassanoyoctozepto: would you like I forward the email received from Sean Mooney ....he talked about cgroups07:02
*** amoralej|off is now known as amoralej07:02
yoctozeptoignaziocassano: I saw their mail but I did not dive deep in there; you can ping Sean on IRC but they are in some US timezone ;-)07:03
ignaziocassanoyoctozepto: I've already sent an email to him but no answer. It sent me an opened red hat bug07:05
ignaziocassanoyoctozepto: I did not understand the purpose of your patch. Anycase  if the libvirt restar causes the vm shutdown I think  we should mark it as a bug . I do not know how the community work . I wonder kolla coomunity should work with nova/cinder ans so on developers07:12
yoctozeptoignaziocassano: yeah, you can report a bug as it should not be happening (and I can assure you it does not happen to me ;-) )07:14
yoctozeptohttps://bugs.launchpad.net/kolla-ansible07:14
yoctozeptoperhaps the very latest docker broke or something07:14
ignaziocassanoyoctozepto: I am going to open the bug07:15
yoctozeptook07:20
*** rpittau|afk is now known as rpittau07:24
ignaziocassanoyoctozepto: https://bugs.launchpad.net/kolla-ansible/+bug/194170607:28
ignaziocassanoyoctozepto: Can I deplpy again without your patch to have a standard deplpoyment ?07:28
opendevreviewPiotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication  https://review.opendev.org/c/openstack/kolla-ansible/+/80595407:38
ignaziocassanoyoctozepto: I asked if I can redeploy without your patch because I wonder if you need I make further tests07:44
mgoddardyoctozepto: the lowlights joke was that we might advertise lowlights in PR07:57
*** ykarel is now known as ykarel|lunch08:28
hrwmorning08:47
ignaziocassanoHello All, I discovered another issue with cinder netapp nfs driver. If snapshot visibility is enabled on netapp, when I restart docker container it stops cinder services because it tries to mount snapshots and change its owner. Since snapshots are read only, cinder volume does not restart. The workaround is disabling snapshot visibility on netapp.09:02
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage  https://review.opendev.org/c/openstack/kolla-ansible/+/79700509:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage  https://review.opendev.org/c/openstack/kolla-ansible/+/79700509:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/79701709:14
yoctozeptoignaziocassano: yes, you can redeploy without the patch now09:35
yoctozeptomgoddard: in Puerto Rico? :D09:37
ignaziocassanoyoctozepto: thanks09:48
*** ykarel|lunch is now known as ykarel10:18
yoctozeptodtantsur: the kolla-bifrost is green now; please continue to monitor; thank you!11:03
dtantsurgreat :)11:11
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage  https://review.opendev.org/c/openstack/kolla-ansible/+/79700511:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage  https://review.opendev.org/c/openstack/kolla-ansible/+/79700511:14
opendevreviewMichal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x  https://review.opendev.org/c/openstack/kolla/+/80451311:37
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/79701712:02
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Bump up Ansible to 2.11  https://review.opendev.org/c/openstack/kolla-ansible/+/79675812:05
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Bump up Ansible to 2.11  https://review.opendev.org/c/openstack/kolla-ansible/+/79675812:06
opendevreviewMichal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x  https://review.opendev.org/c/openstack/kolla/+/80451312:15
mihailis this the correct place to ask for help with kayobe?12:55
parallaxmihail: yes12:56
mihaili am trying to deploy an all in one kayobe and use ceph as the storage backend for cinder glance and nova, but when i run kayobe overcloud services deploy the ceph.conf is deleted from kayobe-config/etc/kolla/config/glance, cinder and nova. why is that?12:57
mihaili followed the kolla-ansible external ceph guide12:58
mgoddardmihail: you shouldn't touch files in etc/kolla/12:59
yoctozeptoheadsup: monasca no longer leaderless12:59
mgoddardmihail: instead use etc/kayobe/kolla/12:59
*** amoralej is now known as amoralej|lunch12:59
mihailmgoddard: so i should create the etc/kayobe/kolla/config/ directory and add only the ceph.conf in glance/ cinder/ and nova/?13:00
mgoddardmihail: yes13:00
mihailthank you13:01
yoctozeptoonly sahara and adjutant remain entirely leaderless; zun with odd situation (ptl candidate not contributing)13:02
mgoddardyoctozepto: adrian is always late with adjutant13:10
yoctozeptomgoddard: the question is whether openstack tc wants adjutant in its current state as it has done almost nothing during xena13:10
yoctozeptowell, "almost nothing" here meaning really nothing13:11
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage  https://review.opendev.org/c/openstack/kolla-ansible/+/79700513:11
yoctozeptomgoddard: https://www.stackalytics.io/?module=adjutant-group hyper sad13:12
mgoddardyoctozepto: yeah, was just looking13:12
yoctozeptomgoddard: https://etherpad.opendev.org/p/yoga-leaderless for an overview13:12
mephmanx_All, I am having an issue with an openstack wallaby install of openstack using kolla.  I am seeing an error with elastic search and no logs are being added.  here is the error:  13:17
mephmanx_{"level":"ERROR","loggerName":"logstash.outputs.elasticsearch","timeMillis":1629936010246,"thread":"Ruby-0-Thread-9: :1","logEvent":{"message":"Unable to get license information","url":"http://10.0.200.21:9200/","exception":{"metaClass":{"metaClass":{"exception":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError","message":"Got response code '400' contacting Elasticsearch at URL 'http://10.0.200.13:17
mephmanx_From what I have found it sounds like a licensing issue with version of elsasticsearch above 7.10 and it looks like kolla openstack wallaby is deploying 7.10.2.  13:17
yoctozeptolicense information? oh my13:17
mephmanx_My version output:13:17
mephmanx_{ "name" : "10.0.200.20", "cluster_name" : "kolla_logging", "cluster_uuid" : "MAinUYsrS-evlWA35ro3dg", "version" : { "number" : "7.10.2", "build_flavor" : "oss", "build_type" : "rpm", "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9", "build_date" : "2021-01-13T00:42:12.435326Z", "build_snapshot" : false, "lucene_version" : "8.7.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_ve13:17
yoctozeptopoor elasticsearch :-(13:18
yoctozeptoparallax: you might know more ^13:18
mnasiadkayoctozepto: we noticed yesterday that elasticsearch python client is too new to support what we are using :D13:18
yoctozeptomnasiadka: :D13:19
mnasiadkaparallax: did you raise a bug about that?13:19
yoctozeptoyou mean too old likely13:19
mnasiadkayoctozepto: no, it only supports Elastic non-OSS :)13:19
yoctozeptobut here I understand the server broke at all13:19
yoctozeptomnasiadka: oh my13:19
mnasiadkayoctozepto: a.k.a. Elastic chose to screw with OSS people13:19
yoctozeptoargh13:19
yoctozeptoall hope in amazon's fork13:20
parallax7.10.2 is the correct version13:20
parallaxworking fine in master, haven't checked wallaby13:20
mnasiadkaparallax: but curator doesn't work?13:24
parallaxmephmanx_: downgrade logstash to 7.10.x, it will work then13:25
parallaxcurator's fine with elasticsearch<7.14 just like in the GitHub issue mentioned in https://bugs.launchpad.net/kolla-ansible/+bug/194107313:27
mephmanx_Im on 7.10.2....  I guess down to 7.10.0?  I deploy using kolla...  Im not hugely familiar with the internals.... ma I able to set something inside the globals.yml to force that version?13:27
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage  https://review.opendev.org/c/openstack/kolla-ansible/+/79700513:27
parallaxmephmanx_: can you double check? in kolla master, centos build gets logstash-oss-1:7.14.0-1.x86_6413:37
parallaxthat's incompatible with oss elasticsearch 7.10.2 also from master - basically, that's a Kolla bug13:38
parallaxas both should stay on 7.10.x13:39
mephmanx_So its not me?  13:39
parallaxmephmanx_: you could revert to using fluentd's elasticsearch output as well 13:40
mephmanx_Im not sure how to do that with the kolla install / deploy...  everything is in docker containers and Im not a docker expert either...13:41
parallaxcorrect me if I'm wrong... isn't logstash part of Monasca which we..... are dropping?13:46
yoctozeptoare we dropping Monasca?13:47
mephmanx_hmm...ok.  I do have monasca enable in my globals.yml.  I should turn that off?13:47
yoctozeptologstash is part of the "default" ELK stack13:47
yoctozeptowe use EFK13:47
yoctozeptoreplacing logstash with fluentd13:47
mnasiadkayoctozepto: we as the company are starting to not use it, we might consider thinking if we really want to support it - or at least deprecate it.13:48
mnasiadkathe second we is the Kolla community13:48
yoctozeptomnasiadka: ++, lovely decision13:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/79701713:48
yoctozeptomnasiadka: I'm all in13:48
parallaxmephmanx_: you'd better disabling, could replace that with enable_central_logging: "yes" - get's you EFK stack working as yoctozepto suggests13:49
mephmanx_if I turn that off and redeploy my env, might that resolve it?13:49
mephmanx_ok, that works for me.13:49
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Add centos-8-stream jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/79701713:49
yoctozeptomephmanx_: could you also report the bug to https://bugs.launchpad.net/kolla ?13:49
yoctozeptomostly to track it13:50
mephmanx_I can.  Thank you.  Ill rebuild my env later today and report how it goes.13:50
*** amoralej|lunch is now known as amoralej13:50
yoctozepto:-)13:50
mnasiadkayoctozepto: Ansible is a mess - Ansible 3.0 includes ansible-base 2.10 (so no change really for us), Ansible 4.0 includes ansible-base 2.11 (and requires python 3.8 more or less)14:00
mnasiadkaso what do we bump to? :)14:01
yoctozeptomnasiadka: first of all - can we use base/core without the bloat?14:01
mnasiadkayoctozepto: we can, but I think in X we should just bump to Ansible 4.0 (base 2.11), and in Y get rid of ansible package, just install required collections and ansible-base14:02
mnasiadkaand we agreed to do the collections+base in kolla-toolbox in X14:02
mnasiadka3.* doesn't receive any minor updates anymore - so moving from unsupported to unsupported is something weird to do14:03
mgoddardmnasiadka: remember we have max and min14:05
mgoddardpy3.8 is quite a strict requirement14:05
mnasiadkamgoddard: yes, I know - but in reality we're just bumping ansible-core 2.10 to ansible-core 2.1114:05
mnasiadka(and the associated collections)14:06
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Bump up Ansible to 4.x / ansible-core 2.11  https://review.opendev.org/c/openstack/kolla-ansible/+/79675814:06
mgoddardcentos is py3.6 by default14:06
mnasiadkalet's see where this fails, and we'll decide what to do14:07
yoctozeptomnasiadka: is it hard to drop collections layer in k-a? (just asking)14:24
opendevreviewRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/victoria: docs: Document CentOS 8 Stream usage  https://review.opendev.org/c/openstack/kolla-ansible/+/79700514:36
opendevreviewMerged openstack/kayobe stable/wallaby: Add twentyFiveGigE to port descriptions in inspector rules  https://review.opendev.org/c/openstack/kayobe/+/80576314:45
yoctozeptodoes stackhpc support sahara?14:54
* yoctozepto wears tc hat and queries the community about it14:54
yoctozeptomnasiadka / mgoddard: ^ question re sahara14:55
mgoddardyoctozepto: We spent some time with it early on, but not much since14:56
yoctozeptomgoddard; thanks!14:57
mgoddardadded a new priority to the list: Bump Ceph packages to Octopus15:02
mgoddardNautilus soon to be EOL...15:02
mgoddardalso added some "reviewers" fields to the priorities15:04
mgoddardmnasiadka, yoctozepto, hrw, osmanlicilegi: please sign up to review those patches you are interested in15:06
yoctozeptook15:10
yoctozeptoremind me if I forget15:10
*** amoralej is now known as amoralej|off15:10
hrwmgoddard: Debian has Nautilus only so far15:16
yoctozeptoduh15:18
hrwcentos has even Pacific ;D15:20
yoctozeptoyeah, that's why nautilus is going buh-bye I guess15:24
*** rpittau is now known as rpittau|afk16:02
opendevreviewMerged openstack/kayobe stable/victoria: Add twentyFiveGigE to port descriptions in inspector rules  https://review.opendev.org/c/openstack/kayobe/+/80576416:17
opendevreviewVerification of a change to openstack/kayobe master failed: Overcloud introspection data save broken  https://review.opendev.org/c/openstack/kayobe/+/80526316:18
opendevreviewVerification of a change to openstack/kayobe master failed: Use ansible_facts to reference facts  https://review.opendev.org/c/openstack/kayobe/+/79130416:18
opendevreviewMerged openstack/kayobe master: Fix documentation links for routed control plane networks  https://review.opendev.org/c/openstack/kayobe/+/80439816:18
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization  https://review.opendev.org/c/openstack/kolla-ansible/+/69217917:48
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization  https://review.opendev.org/c/openstack/kolla-ansible/+/69217917:49
mephmanxTo all just an update.  I rebuilt the env without monasca and kibana DOES work now.  Thank you very much!18:15
mephmanxupdate.  Kibana works fine but grafana is now gone.  I looked into kolla and see this: enable_grafana: "{{ enable_monasca | bool }}"18:20
mephmanxSo it looks like by default grafana is not enabled if monasca is turned off.  If I enable grafana without monasca I assume things will still work?  The 2 arent tied together, correct?  Also, is there something better than grafana or is it required for monasca but otherwise its if desired?18:21

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!