Tuesday, 2019-08-13

*** gregoryo has joined #openstack-kolla00:47
*** dave-mccowan has quit IRC00:58
*** factor has joined #openstack-kolla01:39
*** kplant has quit IRC01:39
*** spsurya has joined #openstack-kolla02:12
*** igordc has quit IRC02:17
*** igordc has joined #openstack-kolla02:39
*** igordc has quit IRC02:55
*** hongbin has joined #openstack-kolla03:00
openstackgerritChason Chan proposed openstack/kolla-ansible master: update horizon configuration for python3 migrating  https://review.opendev.org/67424103:02
openstackgerritChason Chan proposed openstack/kolla-ansible master: Update python_path for "source" install type  https://review.opendev.org/67582203:12
*** gkadam has joined #openstack-kolla03:39
*** gkadam has quit IRC03:40
*** skramaja has joined #openstack-kolla03:42
openstackgerritMerged openstack/kolla-ansible master: Set 'distro_python_version' variable  https://review.opendev.org/67558104:00
*** goldyfruit_ has quit IRC04:11
*** goldyfruit_ has joined #openstack-kolla04:11
*** hongbin has quit IRC04:17
*** whoami-rajat has joined #openstack-kolla04:18
*** dave-mccowan has joined #openstack-kolla04:56
*** absubram has joined #openstack-kolla04:58
*** dave-mccowan has quit IRC05:01
*** dpawlik has joined #openstack-kolla05:20
*** igordc has joined #openstack-kolla05:28
*** gkadam has joined #openstack-kolla05:30
*** absubram has quit IRC05:34
*** cah_link has joined #openstack-kolla05:34
*** pcaruana has joined #openstack-kolla05:40
*** pcaruana has quit IRC05:49
*** igordc has quit IRC06:02
openstackgerritwu.chunyang proposed openstack/kolla-cli master: Do not always copy files  https://review.opendev.org/67581306:25
*** ivve has joined #openstack-kolla06:43
mnasiadkamorning06:50
*** dpawlik has quit IRC06:52
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561406:53
yoctozeptomorning06:57
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561406:58
mnasiadkabrtknr: bump the version of magnum as in minor release? or use e.g. stein version of magnum on queens?06:58
yoctozeptobrtknr: rdo? try tripleo or centos-devel channel06:58
yoctozeptothey could help with such bump06:58
yoctozeptoI assume you are using centos-binary images?06:58
mnasiadkayoctozepto: no, he's not using tripleo for sure :)06:59
*** chrizl has joined #openstack-kolla06:59
yoctozeptomnasiadka: but tripleo has stronger bonds with rdo :-)06:59
mnasiadkayoctozepto: well, I'm surprised magnum even made it to RDO :)06:59
*** luksky has joined #openstack-kolla07:01
*** lemko has joined #openstack-kolla07:02
openstackgerritMerged openstack/kolla-cli master: Do not always copy files  https://review.opendev.org/67581307:11
*** chrizl has quit IRC07:12
*** chrizl has joined #openstack-kolla07:13
*** chrizl has quit IRC07:16
BlinkizHello. How do I troubleshoot a mariadb cluster that will not start? I solved it by shutting down all docker mariadb containers, changed wsrep_cluster_address to gcomm:// and safe_to_bootstrap: 1 in grastate.dat file. Am guessing there is a better way with better overview of the cluster so am recovering the right on?07:19
Blinkizone07:19
BlinkizNot something kolla will handle? Upstream problem? :)07:24
mnasiadkathere is the kolla-ansible mariadb_recovery script07:33
mnasiadkait might help, but basically all the same techniques as you would use on a non-containerized galera cluster apply :)07:33
*** rpittau|afk is now known as rpittau07:34
*** pcaruana has joined #openstack-kolla07:45
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Build images with healthcheck scripts  https://review.opendev.org/67612808:00
*** dpawlik has joined #openstack-kolla08:00
*** priteau has joined #openstack-kolla08:04
*** dougsz has joined #openstack-kolla08:13
*** osmanlicilegi has quit IRC08:32
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Build images with healthcheck scripts  https://review.opendev.org/67612808:32
*** osmanlicilegi has joined #openstack-kolla08:33
*** dpawlik has quit IRC08:37
hrwelo elo08:37
hrwyoctozepto: to discuss rdo you go to #rdo ;D08:38
hrwor #rdo-devel08:38
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67613108:38
mnasiadkageez, circular dependency... hmm08:39
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Fixes Monasca log transformer UTC offset exception  https://review.opendev.org/67566008:46
mnasiadkayoctozepto: so, it seems that all additional plugins that we installed, are depending on fluentd, so we have both fluentd 0.12 and td-agent 3 in the image :D08:57
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561409:07
mgoddardstackedsax: thanks for the update. kklimonda put in quite a lot of work to write up the approach we all agreed on in a spec, then proposed 3 patches (https://review.opendev.org/#/c/663555, https://review.opendev.org/#/c/664516, https://review.opendev.org/#/c/664517) based on the design. Please do collaborate with him on this09:07
*** gregoryo has quit IRC09:07
mgoddardstackedsax: it really doesn't make sense to be duplicating effort09:08
stackedsaxthanks, and we agree.  I mentioned kklimonda earlier in the hopes I would catch his attention09:08
*** rpittau is now known as rpittau|bbl09:09
mgoddardI think he's based in Poland, so should be active09:09
stackedsaxmost of the time spent so far was on setting up a useful testing environment for ourselves and making sure we understood what was going on.  Now that we're at that point, we're reaching back out to collaborate.09:10
kklimondahmm, it seems my IRC client's been acting up - I've not received previous notifications :/09:10
mgoddardstackedsax: ok, great :)09:11
stackedsaxheya, no worries kklimonda -- I was hoping to connect  with you about the SSL everywhere stuff09:11
*** dpawlik has joined #openstack-kolla09:11
stackedsaxbut most useful would be to get generalfuzz in on the convo09:11
stackedsaxwill you still be online at 9 PDT?09:12
mgoddardkklimonda: are you happy for stackedsax and generalfuzz to update your SSL patches?09:12
kklimondastackedsax: that's 4PM UTC, right? I'll be around09:13
mgoddardstackedsax: I should also be around if you ping me09:14
stackedsaxkklimonda: that's right.  I'll let generalfuzz know to hop on here.  I may have a meeting that will run over, but I'll try and prevent that.09:14
stackedsaxI think we'll have lots of questions first, just to make sure we're following the right path.09:14
stackedsaxit's late here at the moment -- will chat when I awake.  Thanks!09:15
kklimondastackedsax, mgoddard: I'd definitely appreciate some help with the patch - it's still something I need to be working on, but I've been pulled into couple of emergencies over the last month or so, and couldn't really focus on it.09:15
stackedsaxkklimonda: happy to help.  generalfuzz is pretty much dedicated to this issue at the moment, so hopefully we can be useful to the cause.09:17
*** priteau has quit IRC09:20
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561409:47
brtknrmnasiadka: yoctozepto: I ended up bumping the minor version in the end and it kind of works, still well behind the tip of stable/stein branch though10:12
brtknrlast updated back in April10:12
mnasiadkabrtknr: stable branch vs a release... well, it's always the project maintainer fault mainly, if there is no newer release :)10:13
mnasiadkaafter they release - we can bump it up here10:13
hrwmnasiadka: td-agent 4 will be also arm64.10:13
brtknrmnasiadka: ah okay that makes sense10:13
mnasiadkahrw: awesome10:14
hrwmnasiadka: at least that's what they announced10:14
hrwmnasiadka: sep/oct10:14
*** dciabrin has quit IRC10:15
brtknrmnasiadka: ah okay that makes sense10:17
*** ktibi has joined #openstack-kolla10:17
*** ktibi has quit IRC10:29
*** gfidente has joined #openstack-kolla10:36
mgoddard"10:37
mnasiadkamgoddard: testing keyboard? :)10:50
mgoddardmgoddard: unclosed quotes making you feel uneasy?10:51
mnasiadkaexactly10:57
mnasiadka:)10:57
yoctozepto<mnasiadka> yoctozepto: so, it seems that all additional plugins that we installed, are depending on fluentd, so we have both fluentd 0.12 and td-agent 3 in the image :D11:07
yoctozeptosounds bad11:07
yoctozepto<mgoddard> mgoddard: unclosed quotes making you feel uneasy?11:07
yoctozeptoI imagine it's a pair of single quotes11:08
mnasiadkayeah, I had to move those to plugin installations11:08
yoctozeptobut have a look at this: '''11:08
yoctozepto<hrw> yoctozepto: to discuss rdo you go to #rdo ;D11:09
yoctozeptomissed that channel11:09
hrwI spent lot of time there in past11:09
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Build images with healthcheck scripts  https://review.opendev.org/67612811:11
*** goldyfruit_ has quit IRC11:13
openstackgerritMerged openstack/kolla master: Ensures monasca-thresh data and worker dirs exist and are empty on start up  https://review.opendev.org/67477911:13
*** kplant has joined #openstack-kolla11:17
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Make installation of epel-release optional  https://review.opendev.org/67617412:05
*** ivve has quit IRC12:09
*** rpittau|bbl is now known as rpittau12:11
openstackgerritKeith Plant proposed openstack/kolla-ansible master: Configure Telegraf to monitor Docker containers  https://review.opendev.org/67542112:17
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Support enabling the Octavia panel  https://review.opendev.org/67617612:18
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Fix swift log level configuration  https://review.opendev.org/67617712:29
*** henriqueof has joined #openstack-kolla12:32
*** henriqueof has quit IRC12:37
*** priteau has joined #openstack-kolla12:41
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Add support for Swift S3 API  https://review.opendev.org/67618112:47
*** factor has quit IRC12:51
*** factor has joined #openstack-kolla12:52
openstackgerritIsaac Prior proposed openstack/kolla master: Install Monasca plugin for Grafana by default  https://review.opendev.org/67618512:59
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561413:04
*** watersj has joined #openstack-kolla13:08
watersjhi, I've deployed kolla, release=stein, binary, getting a timeout starting a VM. Any pointers on to where to look as to why vm will not start.13:10
*** watersj has left #openstack-kolla13:13
*** goldyfruit has joined #openstack-kolla13:13
kplanttail -f /var/log/kolla/nova/nova-*13:14
kplantthat would be a good shotgun approach13:14
*** gkadam has quit IRC13:17
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Build images with healthcheck scripts  https://review.opendev.org/67612813:18
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Build images with healthcheck scripts  https://review.opendev.org/67612813:19
*** priteau has quit IRC13:20
*** pcaruana has quit IRC13:23
*** BjoernT has joined #openstack-kolla13:36
*** BjoernT_ has joined #openstack-kolla13:39
*** BjoernT has quit IRC13:39
*** BjoernT has joined #openstack-kolla13:44
*** BjoernT_ has quit IRC13:46
*** whoami-rajat has quit IRC13:59
*** pcaruana has joined #openstack-kolla14:00
*** dpawlik has quit IRC14:06
openstackgerritwu.chunyang proposed openstack/kolla-cli master: Add group list command  https://review.opendev.org/67620114:10
*** cah_link has quit IRC14:13
*** iclon__ has quit IRC14:16
*** iclon has joined #openstack-kolla14:18
openstackgerritChandan Kumar (raukadah) proposed openstack/kolla master: [DNM] RHEL8 container build check  https://review.opendev.org/67556414:24
*** whoami-rajat has joined #openstack-kolla14:25
*** pomac has quit IRC14:32
*** dcapone2004 has joined #openstack-kolla14:33
*** skramaja has quit IRC14:48
*** dave-mccowan has joined #openstack-kolla14:57
*** dave-mccowan has quit IRC15:04
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Fix idempotency of fluentd customisations  https://review.opendev.org/67621615:23
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Wait for MariaDB to be accessible via HAProxy  https://review.opendev.org/67621915:28
*** jschluet is now known as jschlueter15:30
openstackgerritMark Goddard proposed x/kayobe master: Only run ncclient installation task once  https://review.opendev.org/67622215:33
*** jbadiapa has joined #openstack-kolla15:35
*** KeithMnemonic has quit IRC15:52
*** shyamb has joined #openstack-kolla15:57
shyambHi15:57
shyambMy process in container mounts a NFS share inside container15:57
shyambI want this share to be available on nova_compute container too15:58
*** vesper11 has quit IRC15:58
shyambWe mount all volumes that nova_compute container mounts15:58
shyambAny way to achieve my goal here?15:59
*** vesper11 has joined #openstack-kolla16:00
shyambHere is my docker run command: https://github.com/shyam-biradar/triliovault-cfg-scripts/blob/master/kolla-ansible/trilio-datamover/deploy.sh#L65-#L72\16:01
shyamb"https://github.com/shyam-biradar/triliovault-cfg-scripts/blob/master/kolla-ansible/trilio-datamover/deploy.sh#L65-#L72"16:01
shyambhere is my dockerfile: https://github.com/shyam-biradar/triliovault-cfg-scripts/blob/master/kolla-ansible/trilio-datamover/Dockerfile_queens_centos16:01
shyambAny ideas are welcome16:02
*** spsurya has quit IRC16:04
hrwno ideas16:08
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Build images with healthcheck scripts  https://review.opendev.org/67612816:08
shyambokay16:09
shyambhrw: nova --version in rocky nova_compute container is 11.x16:09
shyambroot@rockydevkolla:~/triliovault-cfg-scripts# docker exec nova_compute nova --version16:09
shyamb11.0.016:09
*** absubram has joined #openstack-kolla16:09
shyambis it correct?16:09
shyambI see in nova_releases 18.x nova blongs to rocky16:10
kplantshyamb: would it be possible it mount NFS on the host and then bind to containers?16:10
shyambAre we using old nova?16:10
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: HAProxy backend connection limits  https://review.opendev.org/67623216:10
shyambkplant: Yes, that is the docker way16:10
mnasiadkashyamb: you are running python-novaclient, not nova server :-)16:11
mnasiadkashyamb: and yes, it's version might be 11.016:11
shyambmnsiadka: got it16:12
shyambserver package is 18.216:12
shyambclient is 1116:12
*** absubram_ has joined #openstack-kolla16:13
mnasiadkaoh geez, now I need to fix the config in td-agent change... uhh16:14
*** shyamb has quit IRC16:15
*** absubram has quit IRC16:15
*** absubram_ is now known as absubram16:15
*** BjoernT has quit IRC16:17
yoctozeptomnasiadka: https://review.opendev.org/66174716:23
yoctozeptoprotected your interest16:23
mnasiadkayoctozepto: great, thanks16:24
*** rpittau is now known as rpittau|afk16:32
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67613116:46
*** gfidente has quit IRC16:51
*** dougsz has quit IRC16:59
openstackgerritMerged openstack/kolla-ansible master: Fix swift log level configuration  https://review.opendev.org/67617717:03
*** jbadiapa has quit IRC17:12
kklimondamgoddard: you wanted a ping when generalfuzz is up and ready to talk :)17:12
generalfuzzHi. I'm all about collaborating with kklimonda on the end to end SSL communication17:13
kklimondageneralfuzz: have you looked at the existing patches and the spec?17:15
generalfuzzI have done a pass through the patches and spec (though need to do another pass through the patches)17:16
generalfuzzIt was suggested that as a first pass implementation that when HAProxy is enabled  to create a "local proxy" on all nodes to do the SSL termination and have HAProxy do a tcp pass through to the "local proxy"17:19
stackedsaxthat was Adam Harwell's initial suggestion when I chatted to him, at least17:20
stackedsaxhe suggested to start there, and then continue with the eventual plan to pick off as many services with wsgi as possible17:22
stackedsaxI think that was the direction you were going, kklimonda?17:22
kklimondastackedsax: that was the initial patch, yes - I took the existing code and got it to work on stein/master branch...17:24
kklimondastackedsax: however during a kolla meeting it was suggested that instead we should skip the "local proxy" and just go updating services to support TLS natively17:25
stackedsaxkklimonda: yeah, and we can do that, but that's an all-or-nothing proposition, isn't it?17:26
stackedsaxperhaps we can find the notes from the meeting -- would love to know the thinking17:27
kklimondastackedsax: my idea was to make backend encryption optional on a per-service basis until it's implemented everywhere - https://review.opendev.org/#/c/664516/2/ansible/roles/keystone/defaults/main.yml17:30
stackedsaxyep, that's what we'd like to do as a phase 2, but we can jump straight there if there's a reason the local proxy is a bad idea17:31
generalfuzzI did a POC implementation of the "local proxy" idea (tested and working, only for a few services) - which was helpful to get me familiarized with parts of Kolla and open stack ecosystem. I'm now ready to re-review your patches, and to see/inquire how I can be of assistance. I can also flush out the "local proxy" implementation if it a helpful intermediary step.17:31
stackedsaxkklimonda: would it be useful to have a call with the three of us?17:32
stackedsaxThe history of this subject is pretty long, it might be quicker to talk it through.  Either way.17:33
kklimondaI think mgoddard is going to be probably also interested in discussing it, especially if we want to consider moving back to local proxy as an interim step17:33
*** jonaspaulo has joined #openstack-kolla17:34
kklimondawe can jump on a call, if it's not too late for me - I'll be online for the next ~1.5h17:34
kklimondaas for the local proxy idea, we have some initial implementation here: https://review.opendev.org/#/c/548407/17:34
stackedsaxhmm, I thought that initial implementation just turned off the internal network altogether didn't it?17:35
kklimondano, it already had the local proxy: https://review.opendev.org/#/c/548407/44/ansible/roles/haproxy-config/templates/haproxy_single_service_listen.cfg.j217:36
stackedsaxbut it was listening on the external interface, or something?  I recall that there was some moral objection I had to it...17:38
stackedsaxheh, yeah, let's talk :D  I'll dm you with some connection info17:38
kklimondayeah, the implementation was basically removing internal network as part of enabling TLS17:39
*** hamzy has quit IRC17:54
*** henriqueof has joined #openstack-kolla18:04
*** BjoernT has joined #openstack-kolla18:15
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible master: Allow cinder coordination backend to be configured  https://review.opendev.org/67626118:28
*** hamzy has joined #openstack-kolla18:30
*** dcapone2004 has quit IRC18:35
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67613118:35
*** nde has joined #openstack-kolla18:36
*** nde has quit IRC18:51
*** hamzy has quit IRC18:51
*** hamzy has joined #openstack-kolla18:51
*** ivve has joined #openstack-kolla19:14
*** iniazi has joined #openstack-kolla19:21
*** whoami-rajat has quit IRC19:33
*** kplant has quit IRC19:48
*** dpawlik has joined #openstack-kolla20:19
*** dpawlik has quit IRC21:02
stackedsaxmgoddard: had a good chat with kklimonda and generalfuzz earlier.  We plan to be at the meeting tomorrow morning and can update then, unless you want a brief now.21:08
*** dpawlik has joined #openstack-kolla21:11
*** absubram has quit IRC21:26
*** absubram has joined #openstack-kolla21:41
*** igordc has joined #openstack-kolla21:52
*** hamzy has quit IRC21:58
*** henriqueof has quit IRC22:07
*** dpawlik has quit IRC22:10
*** jonaspaulo has quit IRC22:12
*** absubram has quit IRC22:14
*** igordc has quit IRC22:28
*** igordc has joined #openstack-kolla22:28
*** absubram has joined #openstack-kolla22:29
*** igordc has quit IRC22:33
*** ivve has quit IRC22:36
*** luksky has quit IRC22:37
*** lemko has quit IRC23:22

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!