Monday, 2020-02-17

*** dave-mccowan has joined #openstack-kolla00:23
*** dave-mccowan has quit IRC00:48
*** ricolin has joined #openstack-kolla01:16
*** zhanglong has joined #openstack-kolla01:18
*** jcmdln has joined #openstack-kolla03:02
*** Tony31 has joined #openstack-kolla03:36
*** goldyfruit_ has quit IRC03:42
*** goldyfruit_ has joined #openstack-kolla03:42
openstackgerritChason Chan proposed openstack/kolla-ansible master: [Docs] Pin kolla-anisble to the same version of quickstart guide  https://review.opendev.org/70789604:19
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic  https://review.opendev.org/66451604:33
*** skramaja has joined #openstack-kolla04:50
generalfuzzIt would be really awesome to have a couple reviews on the current approach we are taking for enabling backend TLS through HAProxy - https://review.opendev.org/664516.04:52
patchbotpatch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 15 patch sets04:52
generalfuzzNow that we have a couple people starting to tackle it service by service, it would be good to know that we are taking an approach the community approves of.04:52
*** shyamb has joined #openstack-kolla05:07
*** shyamb has quit IRC05:08
Tony31good morning05:29
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-kolla05:34
cosmicsoundyoctozepto , i am possitive05:46
Tony31hi cosmicsound05:57
Tony31I was thinking about your tls problem - I was wondering if you have the cert applied on the "internal" ansible variable and it enabled on external or something like that.05:58
*** oyrogerg has joined #openstack-kolla06:06
cosmicsoundHi Tony31 at first I used self generated SSL yet I had not FQDN set on globals.yml06:09
cosmicsoundAfter I have explicitly setup TLS and FQDN for cloud.uhlhost.net domain and I added the right .PEM cert chain06:10
cosmicsoundInternal VIP ip does not have FQDN in place06:10
cosmicsoundhttps://mdb.uhlhost.net/uploads/e15c6e181f5415ce/cloud_uhlhost_net.pem this is the right SSL I put in place. I did not get any errors that the SSL is wrong or the chain. And I am positive this is the wildcard TLS i have in globals.yml06:12
cosmicsoundhttps://mdb.uhlhost.net/uploads/e5657707f57704d2/image.png no other settings I have changed beside external TLS and FQDN06:13
cosmicsoundhttps://mdb.uhlhost.net/uploads/850b3a78847ee23a/image.png and this is clearly showing that the PEM is right domain06:17
cosmicsoundhttps://mdb.uhlhost.net/uploads/b316327bbc027ff4/image.png this is on server side, got really happy to see the FQDN work, yet TLS fails it made me wonder why I have used kolla certificates to generate these self made certs06:19
Tony31I also have a wildcard SSL cert06:26
Tony31brb06:26
sorin-mihaicouple of days ago i tried to use the certs i have generated with certbot, but didn't manage to get it working06:27
Tony31Hi Sorin-mihai - I think I have a solution for you also06:31
Tony31Where did you see the error which you had seen? Was it during deploy or somewhere else?06:31
Tony31sorin-mihai are you using kayobe or kolla-ansible ?06:31
Tony31cosmicsound - I am using kayobe and I tried the "node_config" variable and I got an error (probably because I'm using kayobe, not kolla...). So I have `kolla_external_fqdn_cacert: "/etc/kolla/config/certificates/cacert-internal.pem"` and all that is in there is the CA bundle06:33
yoctozeptomorning06:34
Tony31morning ! :)06:35
Tony31Did you have a good weekend yoctozepto06:35
yoctozeptoTony31: so-so but thank you for asking, and you?06:35
Tony31Was great thank you :)06:36
yoctozepto:-)06:36
Tony31Could you help us with certificates? Bit confused on it :)  With Kayobe, I have the server cert and private key located directly into kolla.yml into `kolla_external_tls_cert:` and the CA cert goes into `kolla_external_fqdn_cacert: "/etc/kolla/config/certificates/cacert-internal.pem"` which I see has configured the kolla `globals.yml` variable06:39
Tony31`kolla_external_fqdn_cacert:` But when checking the kolla docs for the same, I see `kolla_external_fqdn_cacert:` is explained as containing the server cert and I am wondering if this could be the cause of the problems for cosmicsound and sorin-mihai ?06:39
*** sri_ has joined #openstack-kolla06:40
Tony31the server cert and private key goes into `/etc/kolla/certificates/haproxy.crt`06:45
Tony31cosmicsound - for your external cert problem, check your server cert and private key (server cert pem first, then paste the private key underneath) within `06:49
*** riuzen has joined #openstack-kolla06:50
Tony31I would also like some help installing zun/kuryr. I get error `pull access denied for kolla/centos-binary-kuryr-libnetwork` so I tried setting `kuryr_install_type: "source"` but I still get the error when i try to deploy06:51
*** rgogunskiy has joined #openstack-kolla06:53
cosmicsoundwell i was thinking there is no mention of any private key inside globals.yml06:56
cosmicsoundi did not see any variable to do with private key06:57
cosmicsoundguess my private key for the wildcard is missing06:57
Tony31cosmicsound no you are right about that, but there is text that says the server cert must contain the private key - it's a bit different but it works. I guess this is your problem.06:57
cosmicsoundcan you send me that link?06:58
cosmicsoundi did not seen yet such details06:58
Tony31https://docs.openstack.org/kolla-ansible/train/admin/advanced-configuration.html#tls-configuration06:58
cosmicsoundand its maybe even a dsecurity breach06:58
cosmicsoundin my knowledge priv keys are private for a reason06:58
Tony31na - no breach unless the private key is used or given out06:59
cosmicsoundadding them in the .pem chain will loose the prv key06:59
cosmicsoundif an aatacker can get access to it06:59
Tony31from what I can see, we're just presenting the info to kolla so that kolla can manage it06:59
cosmicsoundok06:59
Tony31what kolla will need to do on the backend is install these certs in the right place to set up the chain. Kolla is just abstracting this config06:59
*** shyamb has joined #openstack-kolla07:01
Tony31I understand your feeling though - like I said, a bit different to how I was expecting to approach this. Usually, the server cert and root / CA cert goes into one "bundle" and the private key is stored somewhere else.07:01
cosmicsound These two files are the server certificate with private key and the CA certificate with any intermediate certificates.07:01
Tony31yes thats as I understood it.07:01
cosmicsoundthis is only line that mentions a private key07:01
Tony31the CA cert file I have is specified with `kolla_external_fqdn_cacert:`07:02
Tony31this line: `These two files are the server certificate with private key and the CA certificate with any intermediate certificates.`07:02
cosmicsoundi added the ca bundle and made a right chsain07:02
cosmicsoundthat tls docs are confusing07:02
cosmicsound...07:02
cosmicsoundin end are 3 files07:02
cosmicsoundand its totally wrong07:02
Tony31no - 2 files.07:03
Tony31Which 3 do you have?07:03
cosmicsoundthe private key does not come from a provider07:03
cosmicsoundread a07:03
cosmicsoundagain07:03
cosmicsound1 crt 1 ca bundle 1 private key07:03
cosmicsoundget 3 files07:03
Tony31I completely understand where you are coming from, but in this kolla context it does not really matter07:03
Tony31OK do this:07:03
cosmicsoundall i said i added my right ssl chain of trust07:04
Tony31make 1 file in notepad++ called `server-private.crt` and paste in your server PEM. Then immediately underneath, paste in the private key07:04
cosmicsoundand the pem is ignored07:04
cosmicsoundno where i could deduct that the private key needs to be pasted inside the cert07:04
Tony31Then make another file and call it CA bundle and paste in your CA bundle certs (i am not sure which way around they are meant to go. I usually do intermediate then root07:04
cosmicsoundor i might not be ablee to read that english07:04
Tony31you're not really making a cert for kolla, here. You're just placing the PEM text in the right location for kolla to pull it out and do the necessary things07:05
*** jbadiapa has joined #openstack-kolla07:05
Tony31well, this is as I understand it and I am happy to be corrected07:05
cosmicsoundi get this when ordering an ssl from sectigo/comodo07:06
cosmicsoundhttps://mdb.uhlhost.net/uploads/467a8d4857286f82/image.png07:06
cosmicsoundnothing more nothing less07:06
Tony31thats fine07:06
cosmicsoundi had added my ca-bundle into my crt07:06
Tony31we can make this work with that07:06
Tony31you just need your private key to hand07:06
cosmicsoundto have the right chain of trust07:06
cosmicsoundok i ca. get the prv key07:07
Tony31pull your CA bundle out of your cert because we just need within 1 file: `server cert / private key`07:07
Tony31:)07:07
*** thanhba has joined #openstack-kolla07:10
Tony31cosmicsound - I see that this is incorrect documentation on the link which I had sent to you: `The server certificate needs to be installed with the kolla deployment and is configured with the kolla_external_fqdn_cert or kolla_internal_fqdn_cert parameter.`07:11
Tony31The variable described is for the CA bundle. Not server cert.07:11
Tony31no wait, ignore that. I was mistaking these: `kolla_external_fqdn_cacert:` and `kolla_external_fqdn_cert:`07:12
thanhbaI have problem when use kolla to deploy openstack rocky.07:14
*** cah_link has joined #openstack-kolla07:15
thanhbawhen running nova instance for a while. it lost directory '/sys/fs/cgroup/devices/machine.slice/' then i can't add volume to this instance. this is log http://paste.openstack.org/show/789628/07:17
thanhbaI have problem when use kolla to deploy openstack rocky. when running nova instance for a while. it lost directory '/sys/fs/cgroup/devices/machine.slice/' then i can't add volume to this instance. this is log http://paste.openstack.org/show/789628/07:17
thanhbait is a bug? and how can i solve this problem?07:18
Tony31thanhba - which `kolla` are you using and which version?07:21
thanhbai use kolla 7.x07:22
Tony31I am not sure, do you need to use the matching openstack <-> kolla version ?07:22
thanhbahttp://paste.openstack.org/show/789629/07:23
thanhbayes, i know that. This is how i install kolla. I install kolla with repo git.07:24
Tony31ok - looks ok07:25
*** shyamb has quit IRC07:31
*** Jeffrey4l has quit IRC07:41
*** Jeffrey4l has joined #openstack-kolla07:41
*** hrw has quit IRC07:47
*** hu_berlin_kalle has quit IRC07:48
Tony31is it possible to put compute nodes on different layer 3 networks when using kolla or kayobe ?07:57
*** zhanglong has quit IRC08:01
*** zhanglong has joined #openstack-kolla08:02
yoctozeptoTony31: zun&kuryr are source-only, yup08:04
yoctozeptoTony31: pure computes should be able to be on different subnets08:04
*** bengates has joined #openstack-kolla08:04
yoctozeptoTony31: only controllers/network nodes depend on having single subnet afair08:05
yoctozeptomostly due to keepalived08:05
Tony31yoctozepto thank you - Do I need to configure routes for each interface? For example, the internal API interface, it will have a different subnet, so do I need to route out of this interface for the internal API? Does it make sense what I am trying to explain?08:06
Tony31or, should I just keep it simple and have a single interface for everything08:06
Tony31(on the remote compute node)08:06
*** rpittau|afk is now known as rpittau08:10
*** gbatir_ has joined #openstack-kolla08:11
*** thanhba has quit IRC08:12
mnasiadkamorning08:12
Tony31morning :D08:12
yoctozeptoTony31: uhm, all subnets should be routable08:13
yoctozeptoTony31: interfaces can be different08:13
Tony31they are routable but I have one default gateway08:13
Tony31(per system)08:13
yoctozeptoTony31: this is general networking, not really kolla-specific08:13
yoctozeptoif that one gateway handles it well08:14
yoctozeptothen it's all right ;-)08:14
Tony31I think what I need to find out is if internal API traffic needs to leave and return to the same internal API interface between compute and controller on both sides. Hence, creating "routes" for the individual interfaces.08:15
yoctozeptowell, they need to reach the required destination08:23
yoctozeptowhich is called routability08:23
yoctozeptoI'm answering based on my k-a knowledge, kayobe might be forcing more, better ask mnasiadka/mgoddard08:23
*** pbing19 has joined #openstack-kolla08:24
Tony31no - I'm fine with networks and routing :)  My question is specifically about kolla/kolla-ansible/kayobe and how the separate interfaces are working when we have separate interfaces for the services such as `internal API`08:24
Tony31Talking about the api interface for example and communication between compute node and controller node - they have API interfaces in different subnets.08:25
Tony31So would I need host routes for both controller and compute node, which tells each host to use the API interface to reach the remote systems api interface in this respect?08:26
mnasiadkawell, if you won't set the static routing to go through another gateway on the dedicated interface - it will go through the interface that reaches default gateway, right?08:27
Tony31that's my point08:27
mnasiadkait all depends what you want to achieve, how your networks are configured, do you have security zones and security restrictions and so on08:27
Tony31is it still a working scenario in that case?08:27
*** tonythomas has joined #openstack-kolla08:28
Tony31looking at the controller node which is deployed with kayobe, I dont see any firewall configured there.08:28
yoctozeptoI believe mnasiadka had in mind any external constraints08:29
mnasiadkaTony31: kayobe does not configure firewall08:29
yoctozeptok-a has nothing by design to prevent separating computes to different subnets08:29
Tony31OK this is good - this is what I am looking for. Thank you08:29
yoctozeptoand together we clear all doubts08:29
Tony31So in this case, using separate subnets for example internal API, do I MUST configure routes ? or is it fine to allow sourcing of packets from another interface? For example, default gateway is on eth0 while internal API is on eth2. Since API of controller is on a separate layer 3 routed network, this system would have to send via it's gateway on08:31
Tony31eth0 with a source address of the packet which is eth2.08:31
Tony31I guess that would be poor design but would it work is the question :)08:32
mnasiadkayoctozepto: the single node wsrep errors are by date/time correlated with haproxy killing the connection - but question why neutron stalls for such a long time :)08:32
yoctozeptomnasiadka: yeah, it's a question to your friend08:32
yoctozeptomnasiadka: this looks like an internal process08:32
yoctozeptomnasiadka: despite having req number08:32
mnasiadkawell, it's an upgrade job, so I guess mariadb comes down at some point, and this crappy neutron is not able to cope with that I guess - and get's stuck in some weird state08:33
yoctozeptoTony31: this really is not a kolla question, but networking in general / linux networking08:33
yoctozeptoTony31: if A can reach B on its designated IP address08:34
yoctozeptoTony31 and vice versa08:34
yoctozeptoTony31: then  it is fine08:34
yoctozepto;-)08:34
yoctozeptomnasiadka: yeah, that's what happens there08:34
yoctozeptomnasiadka: though not very precise08:34
Tony31yoctozepto fantastic thank you for your help on this one to help me understand.08:35
yoctozeptoyw,08:36
mnasiadkayoctozepto: well, it's interesting that the message "WSREP has not yet prepared node..." is an internal error, but lost connection to mysql during query is a connection error handled at oslo db layer08:36
mnasiadkayoctozepto: maybe that's the key08:36
yoctozeptomnasiadka: could be08:36
mnasiadkabecause from my perspective it should be handled in the same way08:36
mnasiadkalet me raise a bug to Oslo, let's see :)08:37
yoctozeptomnasiadka: there was one already08:38
yoctozeptomnasiadka: from me08:38
mnasiadkayoctozepto: got a link?08:38
yoctozeptomnasiadka: looking now08:39
yoctozeptomnasiadka: not found it in lp08:40
yoctozeptomnasiadka: was it sb?08:40
mnasiadkayoctozepto: well, oslo wiki says oslo.db bugs are managed in launchpad08:41
yoctozeptomnasiadka: then can't find08:41
yoctozeptomaybe it was not mine after all hmm08:41
*** hu_berlin_kalle has joined #openstack-kolla08:41
mgoddardmorning all08:42
mgoddardI'm out today and tomorrow08:42
mgoddardplease address all requests to mnasiadka :p08:42
yoctozeptomgoddard: will do08:42
yoctozeptomgoddard: have fun08:42
yoctozeptomnasiadka: ah, it was keystone then08:42
yoctozeptomnasiadka: and keystoneauth08:42
yoctozeptomnasiadka: so no, wsrep did not get being reported08:43
mnasiadkayoctozepto: I think we need 1047 or regexp in InternalError filters - https://github.com/openstack/oslo.db/blob/fe74320e8b2c0656ca4f1090a70c28d8294142ac/oslo_db/sqlalchemy/exc_filters.py#L39108:43
yoctozeptomnasiadka: I guess we understand this all better now08:43
yoctozeptomnasiadka: most likely08:44
yoctozeptomnasiadka: oslo folks probably know it better08:45
yoctozeptomnasiadka: this is clearly a "retry" kind of message08:45
yoctozeptomnasiadka: and not fail critically08:45
yoctozeptomnasiadka: btw, did you get manila to work? you seem pretty content closing ceph bugs08:47
*** shyamb has joined #openstack-kolla08:47
mnasiadkaI did get manila to work, I'll add manila to ceph-ansible CI - I'm only closing those that related to kolla-ceph - unless you want to invest cycles in fixing kolla-ceph in Train and earlier :)08:48
*** rgogunskiy_ has joined #openstack-kolla08:48
mnasiadkayoctozepto: this one is still a bit weird: https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron/neutron-server.txt#436108:50
mnasiadkabut after this - we restart neutron-server and it gladly connects08:50
yoctozeptomnasiadka: what's weird about it though?08:51
yoctozeptomnasiadka: https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron/neutron-server.txt#260 it started early08:52
Tony31hi mgoddard hope you had a good weekend08:52
mgoddardhi Tony31 thanks, and you08:52
*** rgogunskiy has quit IRC08:52
mgoddardhow's the deployment going?08:52
Tony31great thank you :)   - deployment is good, unable to install swift at the moment. Michael logged that one. So I'm going to look at kubernetes (magnum) and try zun/kuryr. I have to set some variables to allow it to install from source. I think I missed one as I only listed kuryr. But otherwise it's great and I think what you guys have done here with08:54
Tony31this is fantastic - more people should get onboard with this08:54
yoctozeptoTony31: I think I logged that08:59
yoctozeptoTony31: thanks for the good word08:59
*** k_mouza has joined #openstack-kolla09:04
Tony31my bad. it was yoctozepto who was helpful over the weekend09:06
*** ivve has joined #openstack-kolla09:07
*** k_mouza has quit IRC09:08
mnasiadkawell, 8 minutes after mariadb start - in theory we still get an error about node not ready for application use - but neutron-server restart does the job - funny, right?09:09
mnasiadkayoctozepto: https://bugs.launchpad.net/neutron/+bug/186357909:12
openstackLaunchpad bug 1863579 in oslo.db "Unhandled error - WSREP has not yet prepared node for application use" [Undecided,New]09:12
mnasiadkalet's see what happens09:12
yoctozeptomnasiadka: https://zuul.opendev.org/t/openstack/build/bd8c3d336030431686071cb26f2793d2/log/primary/logs/kolla/neutron/neutron-server.txt#417509:13
yoctozeptomnasiadka: it was critical09:13
yoctozeptoand later kolla saves the day09:13
*** dougsz has joined #openstack-kolla09:13
yoctozeptomnasiadka: ok, I added myself as victim, subscribed, commented and changed to "confirmed"09:17
yoctozeptomnasiadka: and now we wait09:17
mnasiadkayup09:17
*** FlorianFa has joined #openstack-kolla09:18
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70737509:21
yoctozeptomnasiadka: any idea I could not reproduce this stuff around dbus leakage?09:22
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CentOS 8  https://review.opendev.org/70769009:22
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Remove activate-virtualenv and deactivate-virtualenv roles  https://review.opendev.org/70807209:22
yoctozeptomnasiadka: I agree to limit the scope but prefer to understand09:22
mnasiadkayoctozepto: have no clue, we see it on some envs with high activity of neutron-rootwrap, but on some not really09:23
yoctozeptomnasiadka: very odd09:23
yoctozeptomnasiadka: and this fixes it?09:24
mnasiadkayoctozepto: yeah, Stig confirmed on an environment that it fixes it for him - and we don't break anything else09:24
mnasiadkawonder if that's not some... centos weirdness with systemd09:26
mnasiadkawe usually see it on envs with sriov-agent enabled09:26
mnasiadkamaybe that's also the key :)09:26
yoctozeptomnasiadka: could be, I have no sriov to play with in the first place :D09:28
yoctozeptomnasiadka: commented09:33
yoctozeptomnasiadka: a separate issue - don't other services need similar measures of isolation?09:34
mnasiadkayoctozepto: well, I guess they do - it's ironic, cinder and sahara I think - but I focused on this being the top talker :)09:35
yoctozeptomnasiadka: well, now I wonder if manila's bug you closed today was not really caused by this09:35
Tony31Can I re-assign these variables on a per-host basis? `admin_oc_net_name`09:35
yoctozeptomnasiadka: as it has /run09:35
yoctozeptomnasiadka: and could guess it wants to call some systemd stuff09:35
mnasiadkayoctozepto: manila has /run only for iscsi I think09:36
yoctozeptomnasiadka: thus failing miserably09:36
yoctozeptomnasiadka: well, it somehow figured out to look for systemd ;D09:36
yoctozeptomnasiadka: could be manila though09:36
mnasiadkayoctozepto: I guess we don't need /run/openvswitch in linuxbridge agent? :D09:40
yoctozeptomnasiadka: yeah, and probably none other than ovs agent09:41
mnasiadkagod knows, I can use the CI to test :D09:41
yoctozeptomnasiadka: yeah, I saw mgoddard adding more networking testing to kayobe09:42
yoctozeptomnasiadka: could help in k-a as well09:42
mnasiadkadhcp_agent has interface_driver = openvswitch setting09:42
yoctozeptomnasiadka: then just verify those and check any logs and we are clear09:42
mnasiadkaso I guess it could use a connection to ovs socket09:42
yoctozeptomnasiadka: yeah, but it was supposed to be choosing agent09:43
yoctozeptomnasiadka: well, best test09:43
yoctozeptomnasiadka: add tests and test09:43
yoctozeptomnasiadka: let the ussuri cycle be the neutron cycle for kolla09:43
mnasiadkawell, it has ovsdb connection using tcp, so it shouldn't need the mount09:43
yoctozeptomnasiadka: well, maybe none of these require this mount and we are discussing dump :-)09:44
mnasiadkalet's see09:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70737509:49
mnasiadkaso let's test09:49
mnasiadkafirst the standard ovs stuff09:50
mnasiadkayoctozepto: might be a linuxbridge based CI would be useful09:50
yoctozeptomnasiadka: might be09:50
mnasiadkait seems some people are using it - I don't know if in k-a, but still09:51
yoctozeptomnasiadka: not sure if it passes now that you made openvswitch require this path to exist earlier09:51
*** pbing19 has quit IRC09:51
mnasiadkawell, openvswitch creates this path09:52
yoctozeptomnasiadka: but not docker09:52
mnasiadkaif it would fail for this reason - it would fail earlier as well09:52
yoctozeptoI expect a failure to mount09:52
yoctozeptomnasiadka: earlier you mounted /run09:53
yoctozeptowhich exists09:53
mnasiadkai patchset 1 I mounted /run?09:53
mnasiadkawhere?09:53
yoctozeptomnasiadka: in ovs09:53
yoctozeptoI don't mean neutron now, but ovs09:53
mnasiadkaYeah, and I didn't even edit ovs role in patchset 2, it was as it was in patchset 109:53
mnasiadka:)09:53
yoctozeptomnasiadka: ah, sorry, right09:54
yoctozeptomnasiadka: then it work09:54
*** shyamb has quit IRC09:54
yoctozeptoworks*09:54
*** gfidente has joined #openstack-kolla09:54
yoctozeptothough docker usually fails if path does not exist previously09:54
yoctozeptodo we touch host for this?09:54
mnasiadkanope, only extend_start in kolla I think creates that path if it doesn't exist09:55
*** pbing19 has joined #openstack-kolla09:55
yoctozeptomnasiadka: yeah, but that runs afterwards, weird09:55
yoctozeptomnasiadka: maybe it has something to do with ":shared"09:56
mnasiadkanon-existent bind mounts from the docker engine will get initialized to an empty directory owned by root09:56
yoctozeptomnasiadka: our our module does the creation09:56
mnasiadkait only fails on :ro09:57
yoctozeptomnasiadka: mhm, possible09:57
yoctozeptomnasiadka: question for 100 points - why are we requiring dbus as kolla, and how is it used in deployed projects? (other than causing them memleaks)10:03
yoctozepto(I know systemd depends on dbus, but we make it explicit req in debian)10:03
mnasiadkayoctozepto: we required dbus for ceph-nfs on Ubuntu only10:03
mnasiadkabecause it used old ganesha, which had problems when dbus was missing10:03
mnasiadkanow we could remove it10:03
yoctozeptomnasiadka: ah, this very old10:04
yoctozeptomnasiadka: because later I removed that dep10:05
yoctozeptomnasiadka: rocky: https://review.opendev.org/#/c/674889/1/ansible/roles/ceph/tasks/start_nfss.yml10:05
patchbotpatch 674889 - kolla-ansible (stable/rocky) - ceph: fixes to deployment and upgrade (MERGED) - 1 patch set10:05
mnasiadkahmm, we have dbus in bindep?10:05
yoctozeptomnasiadka: exactly - why?10:05
yoctozeptomnasiadka: kayobe too for that matter ;-)10:05
yoctozeptomnasiadka: and also in zuul playbook10:06
mnasiadkaseems ubuntu gate has problems with missing dbus10:06
mnasiadkabut it was 3 years ago, change made by Steven Dake :D10:06
yoctozeptomnasiadka: from cisco10:06
mnasiadkayoctozepto: father of Kolla ;)10:07
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Remove dbus from bindep  https://review.opendev.org/70808210:07
mnasiadkalet's see ^^10:07
*** shyamb has joined #openstack-kolla10:09
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: Remove dbus from bindep and playbooks  https://review.opendev.org/70808210:12
yoctozeptomnasiadka: ^ to actually have an effect10:12
yoctozeptothough it's most likely installed anyway due to systemd nowadays10:13
mnasiadkawe installed it twice? lol :)10:13
yoctozeptomnasiadka: yeah, that's what I am talking about10:13
Tony31need some help :)  could you point me in the right direction? I would like to add a compute node but it's networks have been redefined with `host_vars` such as `admin_oc_net_name:` the problem is they are not being taken and the host build is assigning IPs from the wrong networks.10:13
Tony31Can I use host_vars for this? If yes, then what name do I need to call the files there?10:14
Tony31the goal is to install a compute node that is in a different layer 3 network10:14
mnasiadkayou can call the files as you like, just put the vars inside10:14
Tony31hmm thats what I thought10:15
ivvehello, i have a problem with the neutron-l3-agent container (possible keepalived issue), im looking for how to enable debuglogs for keepalived as its default is syslog and there is no syslog in the container.. how can i proceed? any pointers welcome10:16
ivveneutron debug already enabled but i can't seem to see much other than transition to backup/master10:16
ivvewhich i can see even if debug is not enabled10:17
*** sri_ has quit IRC10:18
mnasiadkaivve: tried looking in /var/lib/neutron/ha_confs/<router id>/ ?10:21
ivvemnasiadka: i have been looking how it creates debug log in keepalived.py in neutron and if logging.is_debug_enabled(cfg.CONF):10:21
ivve                cmd.append('-D')10:21
ivveand it does that correctly10:22
mnasiadkaivve: yeah, but keepalived has two options - either log to console, or to syslog - syslog is not available inside Kolla docker image10:22
ivveso im just kinda wondering, am i seeing everything in /var/log/neutron/neutron-l3-agent.log?10:22
mnasiadkaivve: so, in theory neutron should store those logs in the directory I mentioned10:22
ivvei know, so the confs as by your suggestion (i have checked these before too) have nothing regarding logging in them10:23
ivveso im guessing it just goes into /dev/null then10:23
mnasiadkawell, seems neutron enforces use_syslog on keepalived in l3 agent10:24
ivveah okay10:25
mnasiadkaivve: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/keepalived_state_change.py#L15910:25
ivvei found the log and since -D flag is on all of them due to keepalived.py they should also be full logs with debug messages10:26
ivvethanks mnasiadka10:26
mnasiadkaivve: now you can go and complain to neutron guys :)10:26
ivveyes, i think it is their problem, just need to prove it. puh :(10:27
ivvejust gonna figure out if this arping is failing first10:27
ivvei get two masters sometimes causing "splitbrain" on namespaces10:27
ivveand the best part, sometimes10:27
*** riuzen has quit IRC10:28
Tony31I think this is a bug10:28
ivvecould be keepalived bug/problem too i guess10:28
ivvebut have to figure out what actually fails and promotes10:28
Tony31mnasiadka I added a new network called `sanb` in kayobe networks.yml. And in the `group_vars/compute` I added the mapping `sanb_interface: eth4` Now I am trying to add a new compute node in another location and it is getting assigned an IP in `sanb` network when it's not configured for it10:30
mnasiadkaivve: well, as I said - keepalived can log to console or to syslog - kind of it's a keepalived weakness :)10:30
hu_berlin_kallehi , I get the following error on kayobe overcloud host configure10:30
hu_berlin_kalleERROR: Cannot uninstall 'ipaddress'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.10:30
hu_berlin_kalleDid I mess omething up in the config somehow? (Or is this a bug?)10:31
hu_berlin_kalle(the error happens on all controllers and all compute node10:31
Tony31mnasiadka - I am doing a grep10:31
mnasiadkaTony31: I guess this compute is still in the compute group?10:32
mnasiadkaTony31: group_vars will set a var for ALL hosts in a group10:32
mnasiadkaso either create a group in inventory for those computes in another location10:32
mnasiadkaor do host_vars instead (for each host)10:33
Tony31mnasiadka thank you for your guidance! I think this is the issue10:33
Tony31so it looks like etc/kayobe/compute.yml is taking precendence over the host_vars. Once I commented out the configuration in the compute.yml I now get the correct IPs allocated10:38
*** shyamb has quit IRC10:46
Tony31What is this scanning for ssh keys and where is it scanning? https://pastebin.com/f7MK30mE10:55
Tony31this is during the host configure.10:56
Tony31I dont know how it was resolved before10:56
Tony31I can ssh from the control host, no issue but the `configure` still fails at this step10:59
*** rpittau is now known as rpittau|bbl11:03
*** Tony31 has quit IRC11:04
*** todin has joined #openstack-kolla11:18
cosmicsoundHow do you do folks?11:18
cosmicsoundyoctozepto , @?11:19
*** shyamb has joined #openstack-kolla11:33
*** gfidente has quit IRC11:47
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting nova/heat api  https://review.opendev.org/70713111:48
mnasiadkayoctozepto: seems removing /run doesn't break ml2/ovs11:53
*** pbing19 has quit IRC11:55
*** rlljorge has joined #openstack-kolla12:01
*** dougsz has quit IRC12:01
*** pbing19 has joined #openstack-kolla12:08
rlljorgeHello, Someone there any idea about this problem https://bugs.launchpad.net/kolla-ansible/+bug/186324912:10
openstackLaunchpad bug 1863249 in kolla-ansible "Cloudkitty not working in Stein" [Undecided,New]12:10
*** Tony31 has joined #openstack-kolla12:10
*** jcmdln has quit IRC12:14
*** factor has joined #openstack-kolla12:15
*** kplant has joined #openstack-kolla12:18
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/70811412:21
*** sri_ has joined #openstack-kolla12:21
*** kplant has quit IRC12:22
*** kplant has joined #openstack-kolla12:22
Tony31Does anyone know about this ssh key scanning issue that I have?12:23
Tony31`TASK [ssh-known-host : Scan for SSH keys] ************************************************************************************************************************************************************12:23
Tony31"changed": false, "cmd": ["ssh-keyscan"], "delta": "0:00:00.022516", "end": "2020-02-17 18:53:46.918114", "item": "", "msg": "non-zero return code", "rc": 1, "start": "2020-02-17 18:53:46.895598", "stderr": "usage: ssh-keyscan [-46cHv] [-f file] [-p port] [-T timeout] [-t type]\n\t\t   [host | addrlist namelist] ...", "stderr_lines": ["usage:12:23
Tony31ssh-keyscan [-46cHv] [-f file] [-p port] [-T timeout] [-t type]", "\t\t   [host | addrlist namelist] ..."], "stdout": "", "stdout_lines": []}`12:23
Tony31Where would it be trying to scan?12:23
Tony31the playbook is located `kayobe/venvs/kayobe/share/kayobe/ansible/roles/ssh-known-host/tasks`12:28
*** shyamb has quit IRC12:36
*** rpittau|bbl is now known as rpittau12:37
yoctozeptomnasiadka: do we test instance connectivity though?12:38
yoctozeptomnasiadka: and are there no errors/warnings in neutron services logs?12:39
yoctozeptothen I am very +2 on it12:39
*** Luzi has joined #openstack-kolla12:44
mnasiadkayoctozepto: I’m just thinking a stupid ssh to instance would be nice12:45
yoctozeptomnasiadka: would be lovely; as previously said, I think mgoddard did it in kayobe just recently12:46
*** pbing19 has quit IRC12:49
*** dougsz has joined #openstack-kolla12:51
yoctozeptomnasiadka: this looks pretty bad12:57
yoctozepto2020-02-17 10:34:31.895 6 ERROR neutron.agent.linux.ip_lib [req-2a0b9de6-9a5b-4f25-b34f-a01c49f4dc16 - - - - -] Device tap28a1a217-3a cannot be used as it has no MAC address12:57
yoctozeptohttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_654/707375/3/check/kolla-ansible-ubuntu-source-multinode-ipv6/654000c/secondary1/logs/kolla/12:57
yoctozeptomnasiadka: but not sure if it's CI in general or this new change12:58
yoctozeptomnasiadka: CI in general12:58
mnasiadkaThat’s why we need better network tests in CI :)12:58
yoctozeptomnasiadka: so nothing to worry about before further testing12:59
yoctozeptomnasiadka: yeah, indeed12:59
yoctozeptomnasiadka: only debian&ubuntu get this13:00
yoctozeptomnasiadka: so must be their misconfig13:01
yoctozeptomnasiadka: or it's just superfluous13:01
yoctozeptomnasiadka: odd, I checked several logs results and in non-upgrade success jobs it's the only error we are getting13:04
yoctozeptomnasiadka: and only ubuntu/debian13:04
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/70713113:05
mnasiadkaWonder what is this :)13:07
yoctozeptomnasiadka: only hrw is currently ubuntu/debian core folk13:11
yoctozeptomnasiadka: so not enough love13:11
mnasiadkayoctozepto: and he is not really into deploying :)13:12
yoctozeptomnasiadka: HE IS DA BUILDA13:12
yoctozepto:D13:12
mnasiadkaLike Noe and the ark?13:12
yoctozeptokinda13:12
yoctozeptojust no animal fetish13:12
yoctozeptoI *think*13:12
*** Tony31 has quit IRC13:18
*** gfidente has joined #openstack-kolla13:25
*** spiette_ has quit IRC14:04
*** spiette has joined #openstack-kolla14:07
*** dave-mccowan has joined #openstack-kolla14:11
*** bengates has quit IRC14:14
*** pbing19 has joined #openstack-kolla14:26
*** rgogunskiy_ has quit IRC14:28
*** rgogunskiy has joined #openstack-kolla14:29
*** Luzi has quit IRC14:30
*** rgogunskiy has quit IRC14:31
*** riuzen has joined #openstack-kolla14:33
*** zhanglong has quit IRC14:36
*** sri_ has quit IRC14:37
riuzenHi, i wanna ask about magnum on openstack-kolla. So create k8s cluster but stuck when creating master node. I setup my openstack to do this: request between service using HTTP API (Internal API) and for external access using HTTPS API (External API).How to make my magnum service request from internal API? From magnum log I got this error: Failed to contact the endpoint at https://10.30.30.101:5000 for discovery. Fallback to using that endpoint as14:41
riuzenthe base url.: SSLError: SSL exception connecting to https://10.30.30.101:5000: HTTPSConnectionPool(host='10.30.30.101', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))14:41
*** pbing19 has quit IRC14:47
*** pbing19 has joined #openstack-kolla14:47
*** pbing19 has quit IRC14:49
*** pbing19 has joined #openstack-kolla14:49
*** hu_berlin_kalle has quit IRC14:49
*** cah_link has quit IRC14:50
todinHi, is ovs-dpdk still suposed to work right now? I get this error Interface name undefined for network 'dpdk_tunnel' (set 'dpdk_tunnel_interface'). And the codes was introduced with bc053c0 an there it says "No idea whether ovs-dpdk works at all atm."14:55
*** TrevorV has joined #openstack-kolla14:58
*** bengates has joined #openstack-kolla15:01
*** bengates has quit IRC15:02
*** bengates has joined #openstack-kolla15:02
*** gfidente has quit IRC15:06
*** pbing19 has quit IRC15:08
*** pbing19 has joined #openstack-kolla15:08
sorin-mihaiwhat is the default location for kvm/qemu VMs, /var/lib/libvirt or somewhere else?15:15
kplantif you're using libvirt, the default image directory is /var/lib/libvirt/images15:17
kplantthe metadata is in /var/lib/libvirt/qemu15:17
*** gfidente has joined #openstack-kolla15:19
todinkplant: but that dir is only accesible within of the container, from the point of view of the hostsystem it is a docker-volume15:20
kplantthey didn't ask about kolla nova_libvirt, just kvm/qemu... so that's what i assumed15:20
sorin-mihaii'm only interested from the host point of view, planning to separate the docker stuff from libvirt so that i can backup with lvm. i see that the kolla_logs is also set somewhere in the docker folder, is there a point to not keep it in the real /var/log?15:21
todinkplant: fair enough15:21
sorin-mihailet me rephrase then, when i use 'openstack server create' where are the 'disk' files stored?15:23
kplantwhat's your storage backend?15:24
sorin-mihailvm15:24
kplantright, so cinder volumes would likely be in the "cinder-volumes" vg15:25
kplantand nova volumes would likely be in /var/lig/nova/instances/15:25
kplants/lig/lib15:26
sorin-mihaiso nova instances would be separated from /var/lib/libvirt/images or is some sort of relation between the two?15:28
sorin-mihaias an extreme case, i could create instances in the host with virsh along those created with nova?15:29
kplantin the interest of avoiding an XY problem15:30
kplantwhat are you trying to do?15:31
sorin-mihaiplaning partitioning for a aio host. some of my previous backup plans assumed that /var/lib/libvirt contains (almost) everything needed to 'restore' kvm/qemu images managed through libvirt, at least the metadata and the raw qcow files. i'm sure is not the best aproach...15:33
*** skramaja has quit IRC15:33
sorin-mihaii have a 4tb raid1 for the OS and a 20tb raid50 for hot backups. both 'drives' will be managed with lvm, i'm thinking of separating some things in different volumes, like /var/log, /var/lib/libvirt/, /var/lib/docker and some others15:36
kplantare you thinking of doing an aio for production use?15:37
*** bengates has quit IRC15:37
*** bengates has joined #openstack-kolla15:38
sorin-mihaiit's a dev shop, just 1 server for now, if it proves it's use in the development cycle we get more servers and move to a decent multi node setup. initially i planned the server only for libvirt instances, but by the time the server arrived i changed my mind. lol15:38
*** gbatir_ has quit IRC15:40
*** bengates has quit IRC15:46
*** bengates has joined #openstack-kolla15:48
kplantgot it15:50
kplantso as todin pointed out, libvirt is ran in a docker container with k-a15:51
kplantso all of the libvirt metadata lives within the volume15:51
kplantand i don't think there are any binds to pull guest images out15:51
*** riuzen has quit IRC15:52
sorin-mihaifor me, this is strange now. why run libvirt inside a docker and force the hypervisor to downgrade to qemu, instead of running libvirt along docker in the host?15:53
sorin-mihaior this is only in the aio case?15:54
kplantjust because libvirtd is running in a container doesn't mean the host can't use qemu-kvm15:55
sorin-mihaiyeah, i know. just trying to understand what would be the benefit(s)15:56
noxoid_greater control over the libvirt/qemu versions and upgrade process16:00
sorin-mihaiso, would the disk files used by nova instances be raw qcow on top of the filesystem, or that would happen only if using cinder?16:00
sorin-mihaior if using cinder they will actually be lv and not qcow files? /me goes rtfm about cinder16:05
noxoid_do not confuse cinder volume storage and nova storage. they can be configured to use separate backends16:05
noxoid_if you have nova configured to use lvm via https://docs.openstack.org/nova/stein/configuration/config.html#libvirt.images_type then the data will be on an LV with the metadata in the aforementioned /var location16:09
noxoid_iirc the default, if you made no changes, is qcow2 files in the aforementioned /var location16:10
sorin-mihaiyeah, i got that. though, with the minimal configuration needed to get things running cinder is not enabled, so i didn't think about that too much during the tests i've run so far (bear with me please, i had to also adjust to centos after more than 10 years of "anything else but rpm")16:11
*** bengates has quit IRC16:13
*** bengates has joined #openstack-kolla16:27
*** bengates has quit IRC16:29
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/70811416:31
*** bengates has joined #openstack-kolla16:31
*** bengates has quit IRC16:34
*** bengates has joined #openstack-kolla16:35
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node  https://review.opendev.org/70821316:46
*** bengates has quit IRC16:48
*** bengates has joined #openstack-kolla16:48
*** rpittau is now known as rpittau|afk16:54
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node  https://review.opendev.org/70821316:55
yoctozeptotodin: hi, it is surely supposed to by general means; the commit is about IPv6 and it most likely does not work with ovsdpdk atm due to aforementioned oddity16:57
yoctozeptotodin: please raise a bug about ovsdpdk if you think it's broken, we likely have shortage of its users16:57
*** ivve has quit IRC16:58
*** bengates has quit IRC16:59
*** gfidente has quit IRC17:18
openstackgerritJason Anderson proposed openstack/kolla-ansible master: [gnocchi] Disable statsd daemon by default  https://review.opendev.org/67161817:28
*** factor has quit IRC17:30
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-kolla17:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: CI: Refactor VXLAN overlay setup  https://review.opendev.org/70821717:46
*** cz3 is now known as tequilasunset17:55
*** dougsz has quit IRC17:56
*** tequilasunset is now known as cz317:56
openstackgerritwes hayutin proposed openstack/kolla master: change tripleo centos-7 build containers to centos-8  https://review.opendev.org/70821817:59
openstackgerritRadosław Piliszek proposed openstack/kolla master: Get rid of Python 2 support  https://review.opendev.org/69131618:01
weshay|ruckFYI.. folks https://review.opendev.org/708218  remove the blocking centos-7 tripleo build containers job18:03
patchbotpatch 708218 - kolla - change tripleo centos-7 build containers to centos-8 - 1 patch set18:03
*** aleccoder has quit IRC18:06
*** klippo has quit IRC18:07
r3ap3rI have two questions for the Dev team when anyone gets a second. Have yall moved to all Python 3 in your Gitlab Senarios? If so, do you have any "hey, look out for this" if I try a deployment using only Python 3 on CentOS 7?18:09
r3ap3rI meant "all" of your Gitlab Senarios.18:09
yoctozeptoweshay|ruck: thanks, already +218:10
yoctozeptor3ap3r: c7 py3 is probably still missing selinux bindings18:10
weshay|rucknp :) thanks to you folks ( in general )18:10
weshay|ruckthink that is the case re: selinux18:10
yoctozeptor3ap3r: and we never moved to py3 on c718:11
yoctozeptor3ap3r: we are moving to c8 :-)18:11
r3ap3ryoctozepto: yeah, I saw that when looking up the deps manually first.18:11
yoctozeptor3ap3r: btw, we don't have gitlab but zuul18:11
yoctozeptor3ap3r: nice piece of software18:11
yoctozeptor3ap3r: yeah, it's breaking for ansible18:11
yoctozeptor3ap3r: not much to do for us unless you want to hack ansible for yourself18:12
yoctozeptor3ap3r: or provide selinux bindings18:12
*** dannins has quit IRC18:13
r3ap3ryoctozepto: yeah, I'm aware of the move to C8, just didn't know if it was ready or not, was going to run through a fresh deploy today. I have not messed with zuul, I will have to take a peak. I'm working on my "DevOps" skillset but not quite there with being comfortable enough to contribute to a project yet. ;-)18:14
yoctozeptor3ap3r: I see, zuul is openstack (well, actually opendev atm) -wide solution for CI/CD18:16
yoctozeptor3ap3r: https://zuul-ci.org/docs/zuul/index.html18:16
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic  https://review.opendev.org/66451618:17
r3ap3ryoctozepto: Cool, will check it out. Thanks. :-)18:18
*** tonythomas has quit IRC18:20
*** aleccoder has joined #openstack-kolla18:26
openstackgerritJason Anderson proposed openstack/kolla-ansible master: [haproxy] Support interfaces with '-' chars  https://review.opendev.org/66332518:36
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/70811418:42
*** klippo has joined #openstack-kolla18:47
*** Tengu has quit IRC19:07
*** Tengu has joined #openstack-kolla19:09
*** Tengu has quit IRC19:14
*** Tengu has joined #openstack-kolla19:15
*** pbing19 has quit IRC19:19
*** Tengu has quit IRC19:21
*** Tengu has joined #openstack-kolla19:22
openstackgerritJason Anderson proposed openstack/kolla-ansible master: [haproxy] Support interfaces with '-' chars  https://review.opendev.org/66332519:32
*** rlljorge has quit IRC19:38
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Refactor VXLAN overlay setup  https://review.opendev.org/70821719:43
*** Tengu has quit IRC19:44
*** Tengu has joined #openstack-kolla19:46
*** klippo has quit IRC19:46
*** aleccoder has quit IRC19:46
*** openstackstatus has quit IRC19:49
*** Tengu has quit IRC19:51
*** Tengu has joined #openstack-kolla19:55
*** adeberg has quit IRC20:12
*** aleccoder has joined #openstack-kolla20:17
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: test-core-openstack: add floating ip and ssh to the instance  https://review.opendev.org/70825020:21
*** kplant has quit IRC20:23
todinyoctozepto: Thanks for your response, right now I am trying to figure out if it its broken or just badly documented, as soon as I know more I will let you know20:29
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: CI: Refactor VXLAN overlay setup  https://review.opendev.org/70821720:30
*** klippo has joined #openstack-kolla20:32
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic  https://review.opendev.org/66451620:33
yoctozeptotodin: could be both!20:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Refactor VXLAN overlay setup  https://review.opendev.org/70821720:40
todinyoctozepto: maybe, but the kolla projekt is still awsome, thank to you all for that20:41
yoctozeptotodin: thanks!20:42
*** TrevorV has quit IRC21:33
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node  https://review.opendev.org/70821321:37
*** lennyb has quit IRC22:17
*** lennyb has joined #openstack-kolla22:20
*** openstackstatus has joined #openstack-kolla23:02
*** ChanServ sets mode: +v openstackstatus23:02
*** igordc has joined #openstack-kolla23:03
r3ap3r`kolla-ansible deploy` and `kolla-ansible reconfigure` seem to be looking for `haproxy-internel.pem` but it doesn't appear to be generated by the `certificates` playbook? Am I missing something? Notes: AIO Deployment on CentOS 7 from Source: any other questions are welcome. Please see below pastebin for further details about the playbook errors.23:06
r3ap3rhttps://pastebin.com/3kNYD3i123:06
openstackgerritHongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image  https://review.opendev.org/70827323:07
openstackgerritHongbin Lu proposed openstack/kolla master: [WIP] Zun: add zun-cni-daemon image  https://review.opendev.org/70827323:07
r3ap3rI also looked in the `/etc/kolla/certificates/` directory and searched for the `haproxy-internal.pem` throughout the entire drive and it doesn't exists.23:07
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node  https://review.opendev.org/70821323:08
r3ap3rI'm also looking at the `generate.yml` on the Kolla Github project and it also doesn't reference creating the `haproxy-internal.pem` either. Maybe it is supposed to be somewhere else?23:12
*** goldyfruit_ has quit IRC23:51
*** negronjl has quit IRC23:55
*** negronjl has joined #openstack-kolla23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!