Wednesday, 2019-08-14

« Tuesday, 2019-08-13 Index Thursday, 2019-08-15 »
*** whoami-rajat has joined #openstack-kolla00:09
*** igordc has joined #openstack-kolla00:09
*** KeithMnemonic has joined #openstack-kolla00:20
*** KeithMnemonic has quit IRC00:47
*** KeithMnemonic has joined #openstack-kolla00:49
*** spsurya has joined #openstack-kolla01:05
*** hamzy has joined #openstack-kolla01:17
openstackgerritMerged openstack/kolla-cli master: Add group list command  https://review.opendev.org/67620101:20
*** BjoernT_ has joined #openstack-kolla01:44
*** BjoernT_ has quit IRC01:46
*** BjoernT has quit IRC01:47
*** KeithMnemonic has quit IRC02:17
*** BjoernT has joined #openstack-kolla02:22
*** whoami-rajat has quit IRC02:28
*** BjoernT has quit IRC02:46
*** strigazi has quit IRC03:09
*** strigazi has joined #openstack-kolla03:10
*** whoami-rajat has joined #openstack-kolla03:20
*** gkadam has joined #openstack-kolla03:41
*** gkadam has quit IRC03:41
*** gkadam has joined #openstack-kolla03:43
*** gkadam has quit IRC04:00
*** absubram has quit IRC04:11
*** absubram has joined #openstack-kolla04:30
*** igordc has quit IRC05:02
*** haria has quit IRC05:42
*** cah_link has joined #openstack-kolla05:43
*** absubram has quit IRC05:48
mnasiadkamorning05:50
mnasiadkacloudnull: would you have some time to look at https://review.opendev.org/#/c/675614/? Basically we are trying to bump up fluentd using TreasureData repo, but tripleo is removing all non-RedHat controlled repos. I guess it needs some another override on tripleo side?05:56
mnasiadkaand now I need to find a solution for cyclic dependency...06:00
yoctozeptomornin06:05
yoctozeptomnasiadka: yeah, for Kien's haproxy change too ;/06:06
*** dpawlik has joined #openstack-kolla06:11
*** luksky has joined #openstack-kolla06:18
*** skramaja has joined #openstack-kolla06:18
*** dpawlik has quit IRC06:20
*** dpawlik has joined #openstack-kolla06:23
*** gfidente has joined #openstack-kolla06:24
*** cah_link has quit IRC06:26
*** gfidente has quit IRC06:30
*** cah_link has joined #openstack-kolla06:31
mnasiadkayoctozepto: unless we can gather version of fluentd and base k-a templating on this - I don't think there is a way ;)06:39
mnasiadkaand then there is still fluentd vs td-agent06:40
*** lemko has joined #openstack-kolla06:43
*** egon^ has joined #openstack-kolla06:56
*** ivve has joined #openstack-kolla06:59
*** cah_link has quit IRC07:13
*** shyamb has joined #openstack-kolla07:16
*** cah_link has joined #openstack-kolla07:16
*** janki has joined #openstack-kolla07:23
shyambHi07:23
shyambI need to mount a nfs share on nova_compute container07:24
shyambduring openstack deployment07:24
shyambany ideas are welcome07:24
*** shyamb has quit IRC07:41
*** hamzaachi has joined #openstack-kolla07:41
*** gfidente has joined #openstack-kolla07:53
*** shyamb has joined #openstack-kolla07:53
openstackgerritIsaac Prior proposed openstack/kolla master: Install Monasca plugin for Grafana by default  https://review.opendev.org/67618507:58
*** rgogunskiy has joined #openstack-kolla08:00
*** dougsz has joined #openstack-kolla08:00
mnasiadkashyamb: mount the nfs on host level and add it to extra volumes?08:01
shyambmnasiadka: Hi08:09
shyambif you have any document on extra volumes, please forward to me08:10
shyambThanks08:10
shyambI am new to this08:10
*** jbadiapa has joined #openstack-kolla08:12
*** rpittau|afk is now known as rpittau08:18
*** hamzaachi has quit IRC08:20
shyambHi08:21
shyambHow to apply minor updates on kolla containers?08:21
shyamblike if some bug fixes came and we got new docker image on the dockerhub for kolla08:22
shyambWhat it takes to deploy new containers on existing openstack cloud?08:22
yoctozeptomgoddard, mnasiadka: ok, I studied the problem, it seems we can attack the cyclic dependency problem using little Zuul reconfig08:27
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: move to new config format of fluent-plugin-rewrite-tag-filter  https://review.opendev.org/67613108:27
yoctozeptomeaning we want to actually have non-voting but gating jobs :-)08:27
*** dougsz has quit IRC08:27
yoctozeptoboth k and k-a share the same gate queue08:27
yoctozeptok-a does all the tests, we w+1 that change because it's all green08:28
yoctozeptothen w+1 the kolla change08:28
yoctozeptogating starts08:28
yoctozeptoprofit08:28
yoctozeptootherwise we have to play circles with any interdependency lock08:29
mnasiadkayoctozepto: if ^^ this passes, we might solve this with simple symbolic link, and then remove it afterwards :)08:29
mnasiadkaif centos provided fluent-plugin-rewrite-tag-filter supports new format08:30
mnasiadka(configuration file format)08:30
yoctozeptomnasiadka: I don't like that because you have to remember to undo some wokaround08:31
yoctozeptoit also makes backporting tedious08:31
yoctozeptoand it can happen that workaround itself is tedious08:31
yoctozeptothen it's tedious squared :-)08:31
mnasiadkaif we can move to new config format in old plugin - that's at least one variable that we can rule out, and it's easy to backport it :)08:32
mnasiadkaand then we can think what to do next :D08:32
mnasiadkasecond thing - we need to do it in a way that doesn't break anything08:32
yoctozeptomnasiadka: this is obvious, I was re: the changing paths case08:32
mnasiadkayeah, true08:33
*** luksky has quit IRC08:39
*** dougsz has joined #openstack-kolla08:40
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Enable the Octavia panel in Horizon  https://review.opendev.org/67617608:43
*** janki has quit IRC08:44
*** hamzaachi has joined #openstack-kolla08:45
*** shyamb has quit IRC08:52
*** factor has quit IRC08:58
*** icarusfactor has joined #openstack-kolla08:58
*** gfidente has quit IRC09:00
*** jbadiapa has quit IRC09:03
*** janki has joined #openstack-kolla09:06
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:13
*** luksky has joined #openstack-kolla09:13
*** lemko has quit IRC09:13
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Gate on kolla-ansible jobs but make them not vote in check  https://review.opendev.org/67637709:16
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:18
hrwelo09:18
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:19
yoctozeptomnasiadka: it's not a Zuul project09:21
yoctozeptoclone as you would in  real life09:21
yoctozepto;-)09:21
hrwaarch64--09:22
mnasiadkayoctozepto: you can destroy any playground, right? ;)09:23
*** hamzaachi has quit IRC09:23
yoctozeptomnasiadka: no, I can save your sanity09:24
mnasiadkayoctozepto: just laughing09:25
mnasiadkayoctozepto: there was something like zuul-cloner in the past, right? or should I just do old school git clone? :D09:25
yoctozeptomnasiadka: I believe they got rid of it09:26
yoctozeptoor at least deprecated it09:26
yoctozeptoyou need to check09:26
*** ktibi has joined #openstack-kolla09:31
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:34
mnasiadkayoctozepto, mgoddard: so the new config format for fluent-plugin-rewrite-tag-filter is not supported for the version available from CentOS opstools, so we're back to square one09:35
yoctozeptomnasiadka: :-( ;-(09:37
*** janki has quit IRC09:40
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:41
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: move to new config format of fluent-plugin-rewrite-tag-filter  https://review.opendev.org/67613109:43
hrwI am going to do something not popular09:43
hrwaarch64 ci jobs need to be limited to some set of images09:43
hrwI do not plan to make enemies in infra by saying 'we will bump linaro-london infra flavour to 24-48 vcpu'09:44
*** hamzaachi has joined #openstack-kolla09:45
*** icarusfactor has quit IRC09:51
*** icarusfactor has joined #openstack-kolla09:51
yoctozeptohrw: lol, is it that slow? are you able to pinpoint the slowness pain points maybe?09:51
hrwyoctozepto: when I have whole machine for kolla build then (24 threads on 48 cores) I build all 1:3709:52
hrwyoctozepto: will dig in logs09:52
*** shyamb has joined #openstack-kolla09:52
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Add support for Swift S3 API  https://review.opendev.org/67618109:53
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Add support for Swift S3 API  https://review.opendev.org/67618109:55
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Bump up fluentd  https://review.opendev.org/67613109:55
*** hamzaachi has quit IRC09:58
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:59
hrw(took 1:54:21.350122)openstack-base09:59
hrw(took 1:30:53.922842)haproxy09:59
hrw(took 1:25:51.163077)nova-libvirt09:59
hrw(took 1:11:07.828314)ceph-base09:59
hrw(took 0:54:00.665875)helm-repository09:59
hrwwow.10:00
hrwlooks like completely i/o starvation10:01
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Fix idempotency of fluentd customisations  https://review.opendev.org/67621610:05
hrwmailed linaro team10:10
hrw(took 0:23:42.756531)    openstack-base                                                                                                                                       [0/1893]10:11
hrw(took 0:22:35.261580)    gnocchi-base10:11
hrw(took 0:20:12.330347)    neutron-base10:12
hrw(took 0:19:54.850329)    gnocchi-base10:12
hrwops10:12
hrw(took 0:23:42.756531)    openstack-base10:12
hrw(took 0:22:35.261580)    gnocchi-base10:12
hrw(took 0:20:12.330347)    neutron-base10:12
hrw(took 0:19:54.850329)    gnocchi-base10:12
hrw(took 0:19:14.864279)    dragonflow-base10:12
hrwthose are results from other machine.10:12
mnasiadkayou've got an SSA disk? :)10:12
*** ktibi has quit IRC10:12
hrwmnasiadka: ?10:13
mnasiadkahrw: just laughing, it's been some time since SSDs are a commodity, so that difference is weird :)10:14
hrwmnasiadka: those shorter ones are from machine with just HDD10:14
mnasiadkahrw: yeah, so maybe your disk is some kilometers away in 1st case :)10:15
hrwnormal, plain sata drive10:15
hrw(took 0:24:03.951729)   gnocchi-base10:16
hrw(took 0:10:18.461530)   neutron-base10:16
hrw(took 0:09:40.747166)   nova-base10:16
hrw(took 0:09:23.770550)   horizon10:16
hrwother HDD machine10:17
hrwbut cache did a lot in both situations10:17
hrwhave to do build with clean docker10:17
*** gfidente has joined #openstack-kolla10:27
*** dougsz has quit IRC10:28
shyambHi10:28
shyambDo we have document to apply container updates?10:28
shyambnot release upgrade10:29
shyambWe just want to get latest containers from dockerhub and deploy on existing cloud10:30
shyambIt's rocky ubuntu10:30
*** hamzaachi has joined #openstack-kolla10:32
hrwfresh build started on one machine with clear docker to compare10:33
hrwshyamb: you reminds me to finally play with such10:34
*** shyamb has quit IRC10:35
*** dpawlik has quit IRC10:41
*** shyamb has joined #openstack-kolla10:42
*** dougsz has joined #openstack-kolla10:44
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: HAProxy backend connection limits  https://review.opendev.org/67623210:44
hrwand build on another10:45
shyambextra_volumes feature is not working10:46
*** hamzaachi has quit IRC10:46
shyambI wanted to add extra mount to nova_compute10:46
shyambI added "nova_compute_extra_volumes:10:46
shyamb  - "/var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0:/var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0"10:46
shyamb"10:46
shyambto globals.yaml10:46
shyambcleaned nova_compute container10:47
shyamband re-run kolla-ansible deploy command10:47
shyambI see nova_compute container does not have this extra mount10:47
shyambAm I missing anything?10:47
hrwshyamb: nova_extra_volumes10:50
hrwnova_compute_extra_volumes: "{{ nova_extra_volumes }}"10:50
*** priteau has joined #openstack-kolla10:50
hrwit will be added to each nova container10:50
shyambhrw: We just want to add this mount to nova_compute10:51
shyambnot possible?10:51
hrwshyamb: change ansible roles for nova then10:52
yoctozepto<shyamb> Do we have document to apply container updates?10:52
yoctozeptoI don't remember if we do10:52
hrwshyamb: as by default you should use 'nova_extra_volumes'10:52
yoctozeptobut it's simple as10:53
yoctozeptokolla-ansible pull10:53
yoctozeptokolla-ansible deploy10:53
hrwshyamb: ansible/roles/nova/defaults/main.yml10:53
shyambhrw: okay10:53
shyambI will try nova_extra_volumes10:53
shyambthis will work without changing nova role, right?10:54
shyambyoctozepto: Got it10:54
shyambthanks10:54
*** hamzaachi has joined #openstack-kolla10:55
shyambhrw: In this case, if cloud is already deployed, do we need to clean the nova containers?10:55
shyambor just deploy with new globals.yaml will add the mount to existing containers?10:55
hrwno idea, sorry10:56
shyambhrw: thanks10:59
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637611:02
mnasiadkashyamb: no, kolla-ansible will compare running containers, and restart those that need it11:03
shyambmnasiadka: okay, great11:05
shyambI am testing it11:05
shyambthanks11:05
mnasiadkayoctozepto: so, come again - what do we do with those fluentd changes? :)11:05
yoctozeptoshyamb, mnasiadka: in case of pull that would be all containers though :-)11:06
yoctozeptosince all images WILL change11:06
yoctozepto(unless you run pull right after pull)11:07
yoctozepto(they don't change THAT fast)11:07
yoctozeptomnasiadka: no idea, mgoddard blocked me a bit11:07
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67613111:07
yoctozeptowe need some discussion around that11:07
mnasiadkaso let's wait for mgoddard :)11:07
*** dpawlik has joined #openstack-kolla11:09
shyambmnasiadka: extra volumes not getting reflected if I do not clean existing containers11:10
shyambnova_extra_volumes:11:10
shyamb  - "/mnt/test-dir:/mnt/test-dir"11:10
shyamb  - "/var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0:/var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0"11:10
shyambcloud was already there11:11
shyambI just added extra volumes to global.yaml and re-run deploy command11:11
shyambthis is not working11:11
*** kplant has joined #openstack-kolla11:15
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Use Docker healthchecks  https://review.opendev.org/67638911:16
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: CI: Zun jobs  https://review.opendev.org/67639011:21
hrwmnasiadka, yoctozepto, mgoddard: https://review.opendev.org/#/c/672700/ (docker daemon.json) sits and waits11:26
hrwops, got comments.11:27
BlinkizAm not that familiar with OpenStack so am going to ask. I have the physical interface ens3f1 connected to all three control nodes. Neutron_external_interface is ens3f1.11:30
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Modernize a way of configuring Docker daemon  https://review.opendev.org/67270011:31
BlinkizI now have configured on my switches so VLAN 1534 is delivered tagged to this interface ens3f1 on all control nodes.11:31
BlinkizWhat do I need to do now to get this as a network inside OpenStack?11:32
BlinkizIt seems like ovs has configured the flat network physnet1. bridge_mappings is physnet1:br-ex11:32
BlinkizShould I discard the flat_networks = physnet1 and replace with network_vlan_ranges = physnet1:1000:4000?11:33
BlinkizOr am totally wrong for even looking at the ml2_conf.ini file?11:34
kplantyou could do something like: [ml2_type_vlan] network_vlan_ranges = physnet111:34
kplantthat would allow you to use any tag on physnet111:35
Blinkizkplant, Thank you for the answer.11:35
Blinkizkplant, So am at the right place anyway, ml2_conf.ini.. :)11:35
kplantthen you just add the provider network with tag 1534 in neutron11:35
*** skramaja has quit IRC11:36
kplantopenstack network create some-netwrk --external --provider-physical-network physnet1 --provider-network-type vlan --provider-segment 153411:36
kplantas long as: physnet1 -> br-ex -> ens3f111:36
kplantthat should work11:36
Blinkizkplant, great!11:37
yoctozeptohrw: blessed you with a ton of comments11:37
hrwyoctozepto: blessed you will be11:38
Blinkizkplant, Is there any reason to avoid defining all VLANs (leaving the range empty) because VLANs like network_interface, api_interface and such exist here also.11:38
yoctozeptoBlinkiz: security11:39
kplant^11:39
BlinkizI guess not because I choose with --provider-segment but I ask anyway :)11:39
kplantif you don't trust your operators11:39
yoctozeptokplant: administrators*11:39
yoctozeptoyou are operator :D11:39
Blinkizokay. Thanks for the answer :)11:39
kplantand if someone gets access that shouldn't, they could snipe layer 2 traffic11:39
yoctozeptoin the official terminology11:39
kplantyoctozepto: :-)11:39
Blinkizkplant, Thank you for the help.11:40
kplantyw11:40
*** factor__ has joined #openstack-kolla11:41
*** icarusfactor has quit IRC11:43
*** shyamb has quit IRC11:45
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Use Docker healthchecks  https://review.opendev.org/67638911:51
openstackgerritMichal Nasiadka proposed openstack/kolla master: Add OPENSTACK_RELEASE to before-rc1  https://review.opendev.org/67558811:56
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Add missing Octavia policy file to Horizon  https://review.opendev.org/67617612:01
*** shyamb has joined #openstack-kolla12:02
*** factor__ has quit IRC12:02
shyambextra_volumes feature is not working at all12:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Modernize a way of configuring Docker daemon  https://review.opendev.org/67270012:06
hrwyoctozepto: your comments addressed or replied12:08
cloudnullmnasiadka on it12:10
mnasiadkacloudnull: cool12:12
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561412:12
mnasiadkagood I just realised I would break fluentd on CentOS arm64 completely :D12:12
shyambLogged this bug: https://bugs.launchpad.net/kolla/+bug/184014212:19
openstackLaunchpad bug 1840142 in kolla "kolla-ansible : service level extra_volumes feature not working" [Undecided,New]12:19
shyambfor extra_volumes feature12:19
kplanti occassionally get "Error response from daemon: No such container: mariadb" when trying to deploy an aio on centos7 from stable/stein12:22
kplantanybody else get that?12:22
hrw(took 0:56:20.985358)   openstack-base12:23
hrwthat's look wrong but still better than 1:5612:23
*** luksky has quit IRC12:24
openstackgerritMerged openstack/kolla-ansible master: Add support for Swift S3 API  https://review.opendev.org/67618112:28
*** shyamb has quit IRC12:36
mnasiadkayoctozepto: https://review.opendev.org/676402 - it will work ;)12:40
yoctozeptomnasiadka: you prefer this way?12:43
yoctozeptoyou really like the play :D12:43
mnasiadkayoctozepto: I prefer zuul retrying cloning than me doing it12:43
yoctozeptomnasiadka: good point12:44
yoctozeptothough does it retry?12:44
yoctozeptonever checked12:44
mnasiadkayoctozepto: zuul guys claim it does :)12:44
yoctozeptohrw: cool, will see later :-)12:44
yoctozeptomnasiadka: gr8 then12:44
yoctozeptomnasiadka: ok, read that discussion, sounds promising to use12:45
*** KeithMnemonic has joined #openstack-kolla12:46
mnasiadkayoctozepto: yeah, especially depends-on12:46
*** priteau has quit IRC12:48
*** luksky has joined #openstack-kolla12:52
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Use Docker healthchecks  https://review.opendev.org/67638913:03
*** Blinkiz has quit IRC13:07
yoctozeptomnasiadka: exactly ;D13:09
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add docker inspect output to docker_info logs  https://review.opendev.org/67640813:10
mgoddardshyamb: extra volumes support was added in stein, not rocky13:17
*** BjoernT has joined #openstack-kolla13:19
*** hamzaachi has quit IRC13:22
*** BjoernT has quit IRC13:23
*** BjoernT has joined #openstack-kolla13:23
mgoddardyoctozepto: is it this one I blocked you on: https://review.opendev.org/#/c/676377 ?13:24
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561413:33
mnasiadkamgoddard: well, I agree that nonvoting and failing on gates probably is not the best idea - but I think we need to come up with some approach/logic towards those changes, that can't be made in a proper way (like the fluent-plugin-rewrite-tag-filter...)13:37
mnasiadkamgoddard: I was thinking about adding some steps that would extract the plugin version and base the logic on that13:38
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Wait for MariaDB to be accessible via HAProxy  https://review.opendev.org/67621913:38
mnasiadkamgoddard: like add a LABEL to the docker image with version of the plugin and extract it using docker_container_facts (or similar) - probably the most straightforward option13:39
mgoddardmnasiadka: I was just about to say that13:39
mgoddardhttps://docs.docker.com/engine/reference/commandline/inspect/13:39
mgoddardyou can get labels for an image13:39
mnasiadkamgoddard: actually it would be awesome to have a LABEL with installed packages, but we would need to build the containers twice ;)13:40
mgoddardultimately, cyclic dependencies just show us a compatibility issue that our users will hit - we need to provide a smooth transition13:40
mnasiadkasomewhat true :)13:41
mgoddarddefinitely - each time we break compat between images and ansible we get people asking questions in here for weeks or months13:41
mgoddardand possibly lose some users13:41
mgoddardI agree that having a standard pattern for handling this would be nice. Labels seem like an answer13:41
mgoddardyou probably want docker_image_info https://docs.ansible.com/ansible/latest/modules/docker_image_info_module.html#docker-image-info-module13:42
mgoddardif we do find a way that works, let's write it up in the contributor guide, even if briefly13:43
mnasiadkayeah, let me add the labels and adjust kolla-ansible in a set of changes13:45
mnasiadkaand then we will be able to merge the bump up (I hope)13:45
openstackgerritMichal Nasiadka proposed openstack/kolla master: Add fluentd_binary and fluentd_version labels  https://review.opendev.org/67641113:47
mnasiadkamgoddard: yeah, docker_image_info :)13:48
mnasiadkamgoddard: well actually _facts, because _info showed up in 2.813:50
mgoddardmnasiadka: ok13:50
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Test accessing dashboard  https://review.opendev.org/67641213:54
openstackgerritMerged openstack/kolla-ansible master: Configure Telegraf to monitor Docker containers  https://review.opendev.org/67542114:00
kplant^ is there a process to backport that to stable/stein ?14:01
mgoddardkplant: we normally only backport bug fixes14:02
kplantah okay - that makes sense14:03
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Use fluentd image labels  https://review.opendev.org/67641314:07
*** hamzaachi has joined #openstack-kolla14:07
*** cah_link has quit IRC14:09
stingrayzaanybody here used application credentials?14:13
*** absubram has joined #openstack-kolla14:15
*** absubram has quit IRC14:19
*** dpawlik has quit IRC14:20
*** absubram has joined #openstack-kolla14:25
*** michaelbarkdoll has joined #openstack-kolla14:31
roukstingrayza: yeah, quite a bit.14:31
rouknot sure this is the right place to talk about application creds though.14:31
*** chason has joined #openstack-kolla14:34
*** BjoernT_ has joined #openstack-kolla14:35
*** BjoernT has quit IRC14:36
stingrayzaI suppose #openstack-keystone would be better :) or mind if I go direct?14:37
rouk#openstack is the place, the project channels are generally for dev/bugs, not support.14:37
stingrayzaah, right - didn't know the base channel was there, thanks. will try that14:40
michaelbarkdollWould kolla-ansible be alright with using a bond0 trunked nic for the following settings: network_interface = "bond0.10" and kolla_external_vip_interface = "bond.20" in /etc/kolla/globals.yml?14:42
michaelbarkdollWould kolla-ansible be alright with using a bond0 trunked nic for the following settings: network_interface = "bond0.10" and kolla_external_vip_interface = "bond0.20" in /etc/kolla/globals.yml?14:42
roukmichaelbarkdoll: we use bonds for everything, it works fine.14:42
michaelbarkdollThanks14:43
roukwe bond, then make subinterfaces per vlan, and tie it to that.14:43
yoctozepto<mgoddard> yoctozepto: is it this one I blocked you on: https://review.opendev.org/#/c/676377 ?14:46
*** kevinz has joined #openstack-kolla14:46
yoctozeptoyeah but 'blocked' in positive sense14:46
*** priteau has joined #openstack-kolla14:50
*** luksky has quit IRC14:51
mgoddardMeeting in 6 minutes14:54
mgoddard^ mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne14:54
michaelbarkdollOk, odd question.  If I put a subinterface say bond0.10 as my kolla_external_vip_interface but I also want to use that same vlan (10) for external vm traffic on a trunk defined in neutron_external_interface = "bond1.10" (seperate bond) would there be conflicts?14:57
yoctozeptomeeting in 1 minute14:59
yoctozeptomichaelbarkdoll: yes due to bridge br-ex capturing all traffic14:59
yoctozeptosince you most likely deploy controller with network on the same node14:59
*** scottsol has joined #openstack-kolla14:59
michaelbarkdollThanks, I'll have to avoid.  Have a good meeting.14:59
yoctozeptoyou could try using br-ex for vip_interface but I don't remember if the order of operations would allow that, probably not15:00
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Aug 14 15:00:19 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
mgoddard#topic rollcall15:00
openstackThe meeting name has been set to 'kolla'15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
yoctozeptoo/15:00
mgoddard\o15:00
yoctozeptohrw, mnasiadka15:00
scottsolo/15:00
chasono/15:01
generalfuzzo/15:01
priteau\o15:01
dougszo/ (sort of - in another meeting)15:01
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
mgoddard* Roll-call15:03
mgoddard* Announcements15:03
mgoddard* Review action items from last meeting15:03
mgoddard* Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:03
mgoddard* Kayobe Stein release status15:03
mgoddard* Train release planning15:03
mgoddard* Ceph ansible migration15:03
mgoddard* Kolla Ansible TLS Internal API15:03
mgoddard#topic announcements15:03
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:03
mgoddardNone from me. Anyone else?15:03
yoctozeptono major15:04
mgoddard#topic Review action items from last meeting15:04
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:04
mnasiadkao/ (sorry for being late)15:04
mgoddardmgoddard to ask infra about restarting gerrit15:04
mgoddardmgoddard or someone else to check stable backports15:04
mgoddardthis week seemed to go very quickly, didn't do either15:05
mgoddard#action mgoddard to ask infra about restarting gerrit15:05
mgoddard#action mgoddard or someone else to check stable backports15:05
mgoddard#topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:05
*** openstack changes topic to "Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard (Meeting topic: kolla)"15:05
mgoddardHow is CI looking15:06
yoctozeptomgoddard: never been greener when it comes to master/stein15:07
yoctozeptodid not check others15:07
mgoddardthat's what I like to hear15:07
mgoddardIs everyone keeping their feature status up to date in the whiteboard?15:08
mgoddard(in the priority list)15:08
yoctozeptomine yeah, no progress since last meeting15:09
*** absubram has quit IRC15:09
mgoddard#topic Train release planning15:10
*** openstack changes topic to "Train release planning (Meeting topic: kolla)"15:10
mgoddardWe covered this in detail last time, no need to go over it again IMO15:10
mgoddardPlease all just keep in mind the train release schedule15:10
mgoddard#link https://releases.openstack.org/train/schedule.html15:11
mgoddard(we will lag by 2+ weeks)15:11
mgoddard#topic Kayobe Stein release status15:11
*** openstack changes topic to "Kayobe Stein release status (Meeting topic: kolla)"15:11
mgoddardpriteau, dougsz: how are we looking?15:11
mgoddardI'd like to cut a stable branch soon15:12
priteauThere are many patches that we could land soon15:12
mgoddardquite a few patches with 1x +2 we could land before15:12
priteauhttps://review.opendev.org/#/q/project:x/kayobe+is:open+branch:master+label:Code-Review%252B215:13
mgoddardok, let's get those patches in this week15:14
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Decrypt keyring files using ansible-vault  https://review.opendev.org/67644115:14
priteauAnd also should we proceed with the rename before branching, so there is not too much to change in both master and stable/stein?15:14
mgoddardif there are others without +2 we want in, please add review priority15:14
dougszsounds good, will try and spend some time on them later today15:15
mgoddardpriteau: https://review.opendev.org/#/c/674505/ ?15:15
mgoddardI think we need to wait for infra to do the rename15:16
mgoddardshould only need to cherry pick that patch to stable/stein15:16
*** rgogunskiy has quit IRC15:16
*** rgogunskiy has joined #openstack-kolla15:17
priteauOK15:17
mgoddardthanks dougsz15:18
mgoddard#topic Ceph ansible migration15:18
*** openstack changes topic to "Ceph ansible migration (Meeting topic: kolla)"15:18
mgoddardmnasiadka: you're up15:18
*** dave-mccowan has joined #openstack-kolla15:18
mnasiadkaWell, I have started CI work and some docs work15:18
mnasiadkaWill continue next week (I’m off tomorrow and on Fri)15:19
mnasiadkaSo I’ll have more to say/show on Wed15:19
mgoddardDid you want to discuss the overall approach?15:20
mnasiadkaYeah15:20
mnasiadkaWe have two options:15:20
mnasiadka1) role with importong ceph-ansible roles like OSA does15:20
mnasiadka2) totally separate flow of ceph deployment15:21
goldyfruito/15:21
mnasiadkamgoddard: currently I’m leaning towards 1), but would like to hear comments15:21
kplantOSA's approach is better imo, it's nice to have the familiar interface. vars being the same as ceph-ansible, etc.15:22
kplantimporting what already works and people may be familiar with15:22
yoctozepto(2) seems more like us though15:22
yoctozepto"outsource what we can"15:22
yoctozeptomaybe we could get something inbetween15:23
mnasiadkayoctozepto: it’s still outsourcing, just we need to run preparation tasks (which we need to do in 2) as well)15:23
yoctozeptolike orchestrating the basic set of services15:23
mgoddardhow much of ceph-ansible would we need to reimplement in order to just use the roles?15:23
mnasiadkaAnd then importing role by role from ceph-ansible15:23
yoctozeptook, this way...15:23
mnasiadkamgoddard: generate vars, reuse our inventory and import roles in correct order15:24
yoctozeptoseems reasonable15:24
yoctozeptoyou have my blessing with (1)15:24
mgoddardwould we execute these roles as part of the normal deploy?15:25
yoctozeptounless it gets super clumsy, I will +2 it once done15:25
mgoddardone change in the workflow I like about 2 is that it cleanly separates ceph and openstack15:26
mnasiadkamgoddard: we can, or we make it separate command15:26
*** dave-mccowan has quit IRC15:26
mnasiadkaOr we can do both...15:26
yoctozeptomgoddard: yeah, I originally thought this way15:26
mgoddardyou can't accidentally modify your ceph cluster during an openstack upgrade15:26
yoctozeptoexactly ^15:26
yoctozeptothe very reason I have my ceph cluster deployed externally - more control, less worries15:27
mgoddardI'm open to 1), but would like to see a PoC before committing15:27
yoctozeptothough people MAY like the simplicity15:27
openstackgerritMerged openstack/kolla-ansible master: Add missing Octavia policy file to Horizon  https://review.opendev.org/67617615:27
mgoddardmnasiadka: do you have code you could throw at gerrit?15:27
yoctozeptomnasiadka: ^ even not working15:28
priteauHow complex is each approach to maintain as Ceph-Ansible evolves?15:28
mgoddardthat's a good question15:28
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Test accessing dashboard  https://review.opendev.org/67641215:28
mnasiadkamgoddard: I have some, I can push but that’s halfway done15:29
mgoddard2 is very simple to maintain - we just point to ceph-ansible docs15:29
mgoddardhow will we access ceph-ansible code? git clone from within kolla-ansible?15:31
mnasiadkaAnd make a writeup on using generated keys, basically updating external ceph docs15:31
mnasiadkamgoddard: in CI I’m cloning, because CentOS rpms are uninstallable :)15:31
mnasiadkaAt least 4.0 for now15:32
yoctozepto<mgoddard> 2 is very simple to maintain - we just point to ceph-ansible docs15:32
yoctozeptolol15:32
yoctozeptowe need to make docs of our own anyway15:32
yoctozeptowe had some on-channel report about ceph-ansible being incompatible with our manual for external ceph15:32
*** kevinz has quit IRC15:33
yoctozeptosomething cinder/nova key not being separate related15:33
mgoddardI think there was a bug too15:33
mnasiadkaNot really incompatible, just default config is incompatible :)15:33
michaelbarkdollI clone ceph-ansible currently.  Interesting the 4.0 branch started to require a grafana-server section to be defined which resulted in docker version mismatch if a ceph node is also used as a os node.15:33
*** rgogunskiy has quit IRC15:34
mnasiadkamichaelbarkdoll: true, but you can override grafana container name15:34
yoctozeptomodern ops ("infrastructure as code" blah) advocates cloning everything ;-)15:34
yoctozeptoas in "git everywhere"15:34
yoctozeptomnasiadka: any idea on ceph-ansible dynamics?15:35
yoctozeptore: supporting the (1) option ^15:35
mnasiadkayoctozepto: I doubt they will change all role names next release15:36
mnasiadkaIt’s rather small now, but what do I know15:36
yoctozeptoand the options we want/have to use is probably a small set?15:36
mnasiadkaRDO surprises us every release15:36
yoctozeptoheh15:37
mnasiadkayoctozepto: yeah15:37
mnasiadkaSo I’m still thinking about improving docs so it matches today state15:37
yoctozeptosince (2) is about adapting docs15:37
yoctozeptowe should do at least (2) and most probably (1) as well15:37
mnasiadkaAnd then we can think if we do (1)15:37
mgoddardthat's a fair point15:38
yoctozeptook, then all agreed15:38
yoctozeptomove on, mgoddard15:39
mgoddard#topic Kolla Ansible TLS Internal API15:39
*** openstack changes topic to "Kolla Ansible TLS Internal API (Meeting topic: kolla)"15:39
mgoddardscottsol, stackedsax, kklimonda, generalfuzz: you're up15:39
generalfuzzwe met yesterday for a bit15:39
generalfuzzwe discussed the "local-proxy" approach, which we all felt like be a good first step, and then as a second step we would switch services to use native tls via wgsi one at a time.15:40
mgoddardif we add an intermediate step, we would need to support it15:41
mgoddardwhich could complicate things long term15:41
yoctozeptogeneralfuzz: can you elaborate on "local-proxy"15:42
priteauDoes every single openstack service support tls?15:42
*** ivve has quit IRC15:42
yoctozeptopriteau: all APIs because of HTTP webserver in front15:42
mgoddardpriteau: rephrase: does every single openstack service support WSGI?15:42
yoctozeptorabbitmq for communication also15:42
yoctozeptomariadb checked15:43
mgoddardI think they're just looking at internal API for now15:43
yoctozeptomgoddard: yeah but answering priteau's doubts :-)15:43
scottsolno this is why the intermediate step is being looked at15:43
scottsolas a "catch all"15:43
mgoddardgeneralfuzz: does the implementation of TLS on the frontend for the internal API offer an improvement in your view?15:44
scottsolit may take some time to convince all projects to support wsgi15:44
yoctozeptoscottsol: please give an example for those not blessed with the knowledge :-)15:44
mgoddardi.e. only https://review.opendev.org/#/c/66355515:44
*** hamzaachi has quit IRC15:45
generalfuzzthe "local-proxy" apporach Is to have a proxy server on every node that terminates TLS communication before communicating with the service. Currently HAProxy does the termination, and then communicates using http interally to services15:45
generalfuzzmgoddard - I don't thinkso15:46
*** kevinz has joined #openstack-kolla15:46
yoctozeptogeneralfuzz: understood, thanks15:46
*** hamzaachi has joined #openstack-kolla15:47
yoctozeptoI like the approach, it seems enough for me15:47
yoctozeptono need to encrypt loopback traffic anyway15:47
*** absubram has joined #openstack-kolla15:48
mgoddardhow do we prevent a port conflict between haproxy and the backend?15:48
mgoddardbackend listens on localhost?15:48
mgoddardhaproxy would need to listen both on the VIP and api_interface_address15:49
generalfuzzports would be defined as global variables15:49
generalfuzzwhy the api_interface_address?15:50
mgoddardif you need to access a particular haproxy instance, you can't use the VIP15:50
generalfuzzI imagine the "local-proxy" to be a separate container from HAProxy, listening on a defined intermediary port to do SSL termination15:52
yoctozepto^ exactly what I understood too15:53
yoctozeptoyou want to abuse haproxy for ssl termination15:53
mgoddardit could be. that's not how the similar original design worked - it reused a singe haproxy15:53
mgoddardwhat about using apache within the continer to do the same thing?15:54
mgoddardthen we could use wsgi where available, and just proxy otherwise15:54
generalfuzzThe advantage to having separate proxy containers is that It is simple to reroute haproxy Directly to service when they support  SSL termination natively15:54
*** rpittau is now known as rpittau|afk15:55
mgoddardusing apache to do this gives us a few things15:56
generalfuzzusing apache instead of HAProxy - Does it make any difference?15:56
mgoddardwe don't need to deploy haproxy on every node15:56
mgoddardWSGI becomes an implementation detail15:56
yoctozepto<generalfuzz> using apache instead of HAProxy - Does it make any difference?15:57
yoctozeptoyeah, we have extra apaches :-)15:57
mgoddardso the architecture doesn't change when you switch15:57
yoctozeptoI like mgoddard's idea more15:57
mgoddardand we don't need to add an intermediate design which we then have to support or deprecate and remove15:57
yoctozeptoespecially if most are wsgi anyway15:57
mnasiadkaIf most are wsgi, then it’s a simple fix?15:58
mnasiadkaAnd then let’s worry about the rest?15:58
yoctozeptois there some wsgi/non-wsgi listing anywhere?15:58
generalfuzzI believe kklimonda did a first pass at that list15:58
mgoddardif people think the local-haproxy architecture is better, we could do that path instead. I'm just not so keen on doing both :)15:59
mgoddardI thought the TCP passthrough idea was interesting. I assume we could still do it with apache? Would there be any certificate weirdness?16:00
scottsolmgoddard - has there been any work done towards putting apache inside the container where wsgi does not already exist?16:00
mgoddardscottsol: nope16:00
scottsolI think that could be a good solution16:00
mgoddardbut a proxy config should be easy enough16:00
mgoddardjust configure the service to listen on localhost, then you don't get a port conflict16:01
mgoddardwe are out of meeting time I'm afraid16:01
mnasiadkaI don’t like that path to be honest... sounds like obfuscation and double work :)16:01
mgoddardlet's carry on in this channel though if people are available - it's good discussion16:01
mgoddard#endmeeting16:02
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:02
openstackMeeting ended Wed Aug 14 16:02:04 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:02
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-14-15.00.html16:02
scottsolI'm happy to stick around16:02
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-14-15.00.txt16:02
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-14-15.00.log.html16:02
generalfuzzme too16:02
mgoddardmnasiadka: which path don't you like?16:02
yoctozeptoI prefer the apache path too16:02
mnasiadkaAny internediate point of proxying the traffic once more :)16:02
mnasiadka*intermediate16:03
*** michaelbarkdoll has quit IRC16:03
yoctozeptomnasiadka: did not notice typo until you corrected16:03
mgoddardmnasiadka: I see your point16:04
mgoddardwould like to see that list of non-wsgi services16:04
generalfuzzit would only be if the openstack deployment Is configured to run with TLS everywhere, and only for services that did not  natively support TLS termination16:04
mgoddardhttps://etherpad.openstack.org/p/kolla-internal-tls16:04
yoctozeptohorizon is wsgi 100%16:05
yoctozeptowhy it has an x16:05
yoctozeptoah, misread16:06
openstackgerritKevin Carter (cloudnull) proposed openstack/kolla master: [DNM] Testing dependent review  https://review.opendev.org/67644916:07
mgoddardlooks like the major unknowns are designate, neutron, octavia16:11
yoctozeptonetworking ;D16:12
*** gfidente has quit IRC16:13
mgoddardneutron and octavia are good16:14
generalfuzzgood meaning they have wsgi support?16:14
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: TrivialFix: remove unused template var  https://review.opendev.org/67645216:15
mgoddardyes16:15
*** dave-mccowan has joined #openstack-kolla16:15
mgoddardI've updated the list16:15
yoctozeptogr816:16
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Fix idempotency of fluentd customisations  https://review.opendev.org/67621616:16
mgoddardat this point it might be easier to add support for wsgi to designate?16:17
mgoddardor at least speak to the team and see what is involved16:17
yoctozeptomgoddard: suddenly other services are not supported? :-)16:18
mgoddardyoctozepto: they're more in the best effort category :)16:18
yoctozeptomgoddard: so they can break after enabling tls? ;-)16:21
mgoddardyoctozepto: no, they will just be clear text to the backend16:21
yoctozeptothey better be :D16:21
yoctozeptogoldyfruit: masakari, qinling - your turf I guess?16:22
yoctozeptore: https://etherpad.openstack.org/p/kolla-internal-tls16:22
mgoddardplan was to make backend tls optional per-service anyway16:22
yoctozeptoall righty16:22
goldyfruityoctozepto, yes16:22
yoctozeptoblazar, cyborg, dragonflow, senlin - these really look abandoned16:23
yoctozeptogoldyfruit: please help with their mod_wsgi status16:23
goldyfruitqinling and masakari are fully wsgi supported16:23
mgoddardscottsol, generalfuzz, stingrayza: wdyt to the direction discussed?16:23
yoctozeptokuryr, zun - me interested, will check16:23
mgoddardyoctozepto: priteau would not like to hear you say that about blazar16:23
scottsolI was under the impression it was more than just designate that were going to cause issues16:24
yoctozeptosorry priteau, don't read it please16:24
priteauyoctozepto: I am hurt :(16:24
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Add Masakari Ansible role  https://review.opendev.org/61571516:24
scottsolbut if thats the case maybe we should see what the designate team say about adding the functionality. Although, it would be good to have something in place for less mature services if we want to add them to KA in the future16:25
mnasiadkakuryr libnetwork uses wsgi - maybe just not in Kolla?16:26
generalfuzzI prefer the local proxy approach so we can immediately have secure internal communication, and then allows us to then to support services with WSGI piece meal16:26
mgoddardgeneralfuzz: notd16:27
mgoddard*noted16:28
yoctozeptoscottsol: apache is always there for http ssl termination16:28
mgoddardat least both approaches share the frontend design, so that work need not be blocked16:30
mgoddardscottsol: what are your thoughts?16:30
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Retry Elasticsearch synced flush during upgrade  https://review.opendev.org/67645616:30
*** kevinz has quit IRC16:32
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605016:32
scottsolI'm happy to go with either approach as long as everyone is on the same page16:32
mnasiadkayeah, that's for sure important :)16:33
yoctozeptowe can run voting if implementers don't mind :-)16:34
generalfuzztotally - I will work on whatever approach is decided.16:35
yoctozeptomgoddard, priteau: for my defense I shall add I meant they look abandoned in terms of k and k-a commits in recent weeks ;-)16:37
scottsolnot having the intermediate container there will be less hassle in the long run we just need to make sure we have the bases covered as its best to have everything covered out of the box with KA16:37
*** hamzaachi has quit IRC16:37
yoctozeptonot as upstream projects16:37
yoctozeptowho crossed out designate and what it was meant to mean16:38
yoctozeptosuspecting mgoddard16:38
scottsolgeneralfuzz - is going to the Designate team something you and stackedsax could help with16:38
mgoddardI didn't cross out designate16:38
yoctozeptolooking for other suspects16:39
mnasiadkawell, we can always put some blame on the suspects on openstack-discuss ML :)16:39
stackedsaxsorry I'm late -- just caught up with the discussion.  thanks for updating the list.16:41
yoctozeptoah-ha, the suspect has come16:41
stackedsaxscottsol: sure16:42
stackedsaxyoctozepto: it wasn't me!16:42
*** kevinz has joined #openstack-kolla16:42
stackedsaxI mean, as far as crossing out designate16:42
stackedsaxI'm sure everything else was my fault16:42
yoctozeptostackedsax: but how did you know I meant you and not mnasiadka? that's suspicious at least! :D16:43
yoctozeptohaha, just kidding, though it bugs me why nobody admitted to committing the crime16:43
yoctozepto;p16:43
* stackedsax starts whistling and tries to shuffle away slowly16:44
goldyfruithttps://etherpad.openstack.org/p/kolla-internal-tls/timeslider#952 and https://etherpad.openstack.org/p/kolla-internal-tls/timeslider#95316:47
goldyfruitThe crime has been done not very long ago16:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Use fluentd image labels  https://review.opendev.org/67641316:47
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: CI: Zun jobs  https://review.opendev.org/67639016:50
yoctozeptogoldyfruit: yeah, unfortunately etherpad does not blame for style changes16:50
*** spiette has quit IRC16:52
openstackgerritMark Goddard proposed x/kayobe master: Generate openrc files during control host bootstrap  https://review.opendev.org/66745116:53
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Add missing when condition for swift config files  https://review.opendev.org/67646516:54
yoctozeptomgoddard, mnasiadka : we got the meeting past the planned time but I would like to reraise the point of support matrix16:55
yoctozeptocurrent owner does not seem to actively maintain it16:55
yoctozeptomaybe we have someone to volunteer16:55
mnasiadkayoctozepto: yeah, are you volunteering to start?16:56
mnasiadkaI think somebody just needs to start the ball rolling, and we can iterate16:56
yoctozeptomnasiadka: yeah, I feel so too, unfortunately need to find a better tool to handle that than whiteboard16:56
mnasiadkayoctozepto: the tool is Gerrit and kolla docs :)16:57
mnasiadkaor kolla-ansible, because that's where we test it16:57
yoctozeptoyeah, git and docs is a fine choice16:57
mnasiadkamgoddard: kolla or k-a docs for that bloody support matrix?16:57
yoctozeptokolla I guess16:57
yoctozeptoit is umbrella as well16:58
mgoddardboth IMO16:58
mgoddardkolla for images, kolla-ansible for ansible16:58
yoctozeptomgoddard: you mean we could have separate support matrices?16:58
mgoddardyes - they're separate projects16:59
mgoddardI'd expect them to be mostly the same16:59
mgoddardbut images would include things like X not on ARM16:59
mgoddardonly on centos16:59
mgoddardansible would be more about features17:00
mgoddardmake sense?17:00
mgoddardputting all that in one place would be very busy17:00
yoctozeptokind of17:00
mgoddard:)17:00
mnasiadka+117:00
mgoddardopen to alternative suggestions17:00
*** scottsol has quit IRC17:00
yoctozeptoopen to volunteers :-)17:00
mnasiadkaso, I can take the kolla one - but can start mid next week17:00
yoctozeptogr817:00
mnasiadkaso that leaves k-a one for yoctozepto :D17:01
yoctozeptoindeed, will feel inspired by your work on k17:01
yoctozepto;-)17:01
mnasiadkahaha17:01
mnasiadkaok, gotta pack - leaving for extended weekend tomorrow morning17:02
yoctozeptoseriously, I can take it on myself after pushing out IPv6 proposal for you to review17:02
yoctozeptowhich I planned for this week because I now have more time for this hobby side of work :-)17:03
*** goldyfruit has quit IRC17:03
yoctozepto    mgoddard tried using IUS to get python 3.6 on CentOS 7. Was able to get a kolla-ansible-centos-source job to pass17:04
yoctozeptocongrats ^17:04
mgoddardyoctozepto: thanks. Most changes will actually be discarded - just changing package names for python36u17:05
mgoddardstill, it's encouraging17:05
*** goldyfruit has joined #openstack-kolla17:06
yoctozeptomgoddard: yeah, it definitely is17:07
*** spiette has joined #openstack-kolla17:08
yoctozeptoon similar note - basic zun seems to work on centos too17:08
yoctozeptohttps://review.opendev.org/67639017:08
stackedsaxif we could go back to the tls convo, I'm not sure what the consensus was in the end.17:08
yoctozeptoI'm adopting zun in our systems17:08
yoctozeptostackedsax: to vote17:08
yoctozeptobut we did not vote finally17:08
yoctozeptore: zun - now trying multinode17:09
mgoddardstackedsax: I don't think we had consensus.17:09
stackedsaxfair enough.  I also didn't quite understand the apache option (although I'm fine using whatever as an ssl-terminating-proxy)17:09
mgoddardI thought there was general support for the apache approach, but generalfuzz preferred local-proxy17:09
yoctozeptobut agreed to accept vote results17:10
stackedsax'local-proxy' was our way of saying 'whatever-technology-as-a-proxy'17:10
yoctozeptostackedsax: apache approach is simple17:10
mgoddardstackedsax: the apache option is as described in the spec - backend TLS terminated by apache17:10
yoctozeptoadd apache where it is not already there17:10
yoctozeptoand it is in most anyway17:11
stackedsaxso, going through each and every service, not setting up a proxy17:11
mgoddardright17:11
yoctozeptoby the looks of the whiteboard17:11
yoctozeptoyeah17:11
mgoddardit's certainly more work, but results in one less hop17:11
yoctozeptono per-node-proxy17:11
yoctozeptoper-container-terminator17:11
yoctozepto;D17:11
yoctozeptoapache terminator17:12
yoctozeptosounds like a movie title17:12
yoctozeptoapache tls terminator17:12
yoctozeptowho wants to be the producer17:12
stackedsaxpart of our thinking came from Adam Harwell, who was of the opinion that having the option of a local proxy would help with adding future services17:12
stackedsaxfuture services that might not have added mod_wsgi support yet17:13
mgoddardmy comments in https://review.opendev.org/#/c/664516 were about making the apache config more general such that we can just import a role17:13
mgoddardthat is true. it's quite expensive to add an extra network hop to do it though17:13
yoctozepto^ +117:13
mgoddardand we could make that hop using apache, if necessary17:14
stackedsaxour plan was a phased approach: start with the panacea, then start picking off all of the mod_wsgi-enabled services, then address the services that don't have mod_wsgi17:14
*** spsurya has quit IRC17:14
stackedsaxwe certainly don't want the extra hop if we don't have to17:15
mgoddardmy concern is that it's an architectural side step then back again. With the apache approach the topology remains the same - it's just a config change17:16
mgoddardI hear what you're saying about getting to a solution quickly17:16
mgoddardperhaps your downstream requirements pull you in that direction regardless?17:17
mgoddardI'd really just like to end up with one approach for this in kolla-ansible17:20
stackedsaxno requirements driving our desires, just my preference to be able to get TLS working without having to go through every service17:20
openstackgerritMerged openstack/kolla-ansible master: CI: Sanity check that nodepool.private_ipv4 is assigned  https://review.opendev.org/67489817:21
mgoddardstackedsax: there is always the option that you go for the intermediate approach downstream, then migrate to the other approach when complete17:23
stackedsaxI suppose we can still roll out TLS on the internal network service by service with the apache approach, but I think I would find that also confusing17:23
mgoddardthe frontend could be TLS everywhere17:24
mgoddardit's just the backend that would be piecemeal17:24
stackedsaxmaybe I'm still misunderstanding the suggestion, then17:24
mgoddardmaybe :)17:25
stackedsaxwhich spec were you referring to?17:25
mgoddardhttps://review.opendev.org/#/c/663865/17:25
mgoddardthis patch is all that's required for the frontends: https://review.opendev.org/#/c/663555/617:26
mgoddardsimples17:26
stackedsaxdoesn't that use haproxy to terminate ssl?17:28
mgoddardyes17:29
mgoddardit's terminated twice17:29
mgoddardat the backend also17:29
stackedsaxwhich, until the backend enables termination would still send over the internal network http?17:31
mgoddardright17:31
stackedsaxah, I see.  that happens to be the one requirement which was driving our thinking.  we would like even that traffic to be https, if possible.17:32
stackedsaxat least I understand the disconnect now, thanks17:33
mgoddardstackedsax: yeah, understood. And that local-proxy gets you to fully encrypted faster17:35
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: [WIP] Factor out nova-cell role  https://review.opendev.org/67565917:37
mgoddardI think it's just a question of weighing up upstream vs internal requirements17:37
stackedsaxagreed.17:37
stackedsaxas for terminating twice, won't there be a penalty for that?17:37
*** priteau has quit IRC17:37
mgoddardeither way, frontend only is an improvement - it pushes all clear text to the controllers17:38
mgoddardthere will be a penalty for double termination17:38
mgoddardthe tcp mode wasn't proposed when we discussed before, but I guess it could work with this approach too17:39
stackedsaxwhen we terminate at the backend, is there any reason not to remove the termination from the frontend?17:39
stackedsaxsome logging capability or something that I'm not thinking about?17:40
mgoddardit may depend on your certificates - it's basically a man in the middle, right?17:40
mgoddardit could work for wildcard certs17:40
mgoddardbut if you connect to the API FQDN/VIP and get a cert that matches the backend, your client would (should) not be happy17:41
stackedsaxyeah.  it feels like it might make cert updates trickier to have to do them in two places17:41
*** dougsz has quit IRC17:41
mgoddardI think we'd need to support double termination in either case17:42
stackedsaxhowso?17:42
mgoddardPerhaps there are setups where wildcard certs would allow for TCP passthrough option17:42
mgoddardbut mandating TCP passthrough would mandate wildcard certs, right?17:43
*** dpawlik has joined #openstack-kolla17:43
yoctozeptomgoddard: tcp only invalidates sni, hostname-based certs should still work though17:44
yoctozepto(in theory)17:44
mgoddardyoctozepto: really? user connects to api.example.org, which resolves to a VIP. The front haproxy passes through to local-haproxy or apache on a different host, which could have a cert matching its hostname or API interface IP address17:46
yoctozeptoyeah, the tls client (haproxy) in this case can do hostname-based verification as long as you configure backend using hostname and not IP address - but you can have certs on IP addresses17:49
yoctozeptogeez, I missed the "passthrough"17:49
yoctozeptonow I know why you are so astonished ;p17:50
mgoddardwe definitely need more diagrams of this stuff17:50
yoctozeptopassthrough is a no-no in this case17:50
*** chason has quit IRC17:50
mgoddardvery easy to get confused with all the hops17:50
stackedsaxmgoddard: totally agreed :D17:50
yoctozeptoyeah, I missed one word and changed the meaning completely17:51
*** ivve has joined #openstack-kolla17:52
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: CI: Zun jobs  https://review.opendev.org/67639017:56
openstackgerritMark Goddard proposed openstack/kolla-ansible master: update horizon configuration for python3 migrating  https://review.opendev.org/67424118:01
yoctozeptogoldyfruit: qinling zun support is gone?18:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Test accessing dashboard  https://review.opendev.org/67641218:07
*** dpawlik has quit IRC18:15
goldyfruityoctozepto, ?18:17
goldyfruitZun backend for Qinling ?18:18
goldyfruitNever been there!18:18
yoctozeptogoldyfruit: yeah, I could not find other mention than https://medium.com/@n.neerja28/qinling-in-a-nutshell-6d9cf353734618:25
goldyfruityoctozepto, yeah :/18:28
goldyfruitIn some videos too18:28
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: CI: Zun jobs  https://review.opendev.org/67639018:41
openstackgerritMerged x/kayobe master: Only run ncclient installation task once  https://review.opendev.org/67622218:56
openstackgerritMerged x/kayobe master: Sort overcloud inventory hosts and groups  https://review.opendev.org/67599418:56
*** scottsol has joined #openstack-kolla19:02
openstackgerritMerged x/kayobe master: Add support for custom CloudKitty configuration  https://review.opendev.org/67313919:12
openstackgerritMerged x/kayobe master: Stop allocating network and broadcast addresses  https://review.opendev.org/67231719:12
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Gate on jobs kolla-ansible already gates on  https://review.opendev.org/67637719:13
openstackgerritMerged x/kayobe master: Remove storage management network from controllers  https://review.opendev.org/67248219:15
openstackgerritKevin Carter (cloudnull) proposed openstack/kolla master: [DNM] Testing dependent review  https://review.opendev.org/67644919:28
*** scottsol has quit IRC19:29
*** kplant has quit IRC19:34
openstackgerritMerged x/kayobe master: Restrict ncclient to 0.6.2 to avoid unknown host key issue  https://review.opendev.org/67533619:35
openstackgerritMerged x/kayobe master: Update links to docs and IRC for kolla governance  https://review.opendev.org/67451219:35
*** goldyfruit has quit IRC19:46
*** goldyfruit has joined #openstack-kolla19:46
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Use fluentd image labels  https://review.opendev.org/67641319:48
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Use fluentd image labels  https://review.opendev.org/67641319:50
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Use fluentd image labels  https://review.opendev.org/67641319:54
openstackgerritMichal Nasiadka proposed openstack/kolla master: Add fluentd_binary and fluentd_version labels  https://review.opendev.org/67641119:59
openstackgerritMerged openstack/kolla-ansible master: Add missing when condition for swift config files  https://review.opendev.org/67646520:07
*** luksky has joined #openstack-kolla20:07
*** dpawlik has joined #openstack-kolla20:45
*** dpawlik has quit IRC21:35
*** luksky has quit IRC22:03
*** ivve has quit IRC22:33
*** BjoernT_ has quit IRC22:40
*** absubram has quit IRC23:29
« Tuesday, 2019-08-13 Index Thursday, 2019-08-15 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!