*** wolverineav has quit IRC | 00:02 | |
*** wolverineav has joined #openstack-kolla | 00:03 | |
*** wolverineav has quit IRC | 00:07 | |
*** andrein has quit IRC | 00:13 | |
*** wolverineav has joined #openstack-kolla | 00:20 | |
*** wolverineav has quit IRC | 00:27 | |
*** tolisbar1 has joined #openstack-kolla | 00:30 | |
*** tolisbar has quit IRC | 00:34 | |
*** tolisbar1 is now known as tolisbar | 00:34 | |
*** wolverineav has joined #openstack-kolla | 00:49 | |
*** wolverineav has quit IRC | 00:53 | |
*** igordc has quit IRC | 01:06 | |
*** hongbin has joined #openstack-kolla | 01:33 | |
*** unicell has joined #openstack-kolla | 02:34 | |
*** unicell has quit IRC | 02:34 | |
*** hongbin has quit IRC | 02:35 | |
*** hongbin has joined #openstack-kolla | 02:38 | |
*** igordc has joined #openstack-kolla | 02:40 | |
*** hongbin has quit IRC | 02:49 | |
*** hongbin has joined #openstack-kolla | 02:51 | |
*** baha has joined #openstack-kolla | 02:51 | |
*** baha has quit IRC | 02:51 | |
*** unicell has joined #openstack-kolla | 03:11 | |
*** unicell has quit IRC | 03:11 | |
*** tolisbar has quit IRC | 03:57 | |
*** hongbin has quit IRC | 04:05 | |
*** Sravan has joined #openstack-kolla | 04:47 | |
*** wolverineav has joined #openstack-kolla | 04:49 | |
*** wolverineav has quit IRC | 04:54 | |
*** igordc has quit IRC | 05:11 | |
*** skramaja has joined #openstack-kolla | 05:19 | |
*** andrein has joined #openstack-kolla | 05:30 | |
*** andrein has quit IRC | 05:41 | |
*** Sravan has quit IRC | 05:52 | |
*** Sravan has joined #openstack-kolla | 05:55 | |
*** Sravan has quit IRC | 05:56 | |
*** jbadiapa has joined #openstack-kolla | 06:01 | |
*** radeks has joined #openstack-kolla | 06:04 | |
*** unicell has joined #openstack-kolla | 06:24 | |
*** radeks has quit IRC | 06:28 | |
*** radeks has joined #openstack-kolla | 06:29 | |
*** pcaruana has joined #openstack-kolla | 06:30 | |
*** unicell has quit IRC | 06:34 | |
*** radeks has quit IRC | 06:55 | |
*** k3nny0ne has joined #openstack-kolla | 06:58 | |
*** JangwonLee_ has quit IRC | 07:00 | |
openstackgerrit | Kien Nguyen proposed openstack/kolla-ansible master: Add Ansible Role Zaqar https://review.openstack.org/407760 | 07:02 |
---|---|---|
hrw | moin | 07:04 |
hrw | mgoddard: release of Stein will make Ocata EOL, right? | 07:04 |
*** gfidente has joined #openstack-kolla | 07:05 | |
*** ivve has joined #openstack-kolla | 07:06 | |
mgoddard | morning | 07:10 |
mgoddard | hrw: nope, ocata stays in extended maintenance until it has been unmaintained for 6 months | 07:10 |
hrw | so https://review.openstack.org/#/c/648312/ will be needed. will push ocata version for review in a moment | 07:11 |
*** gfidente has quit IRC | 07:11 | |
mgoddard | hrw: its needed if we want to maintain it. At the moment it's been broken for 2+ months, I suggest we leave it broken unless someone says they need it | 07:12 |
mgoddard | hrw: since we currently have 5 stable branches | 07:12 |
hrw | mgoddard: it blocks two other ocata backports | 07:12 |
hrw | https://review.openstack.org/646439 https://review.openstack.org/631723 | 07:13 |
mgoddard | hrw: yeah, I'm suggesting we stop backporting | 07:13 |
mgoddard | the branch only stays alive if we keep it alive. Do you want it to stay alive? | 07:13 |
*** dciabrin_ has joined #openstack-kolla | 07:13 | |
hrw | mgoddard: I want to get backports queue clean | 07:14 |
mgoddard | hrw: then lets abandon those backports. No one has come forward saying they want ocata | 07:14 |
*** yankcrime has quit IRC | 07:15 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/ocata: kafka: fix URL to tarball https://review.openstack.org/651112 | 07:15 |
mgoddard | hrw: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004480.html | 07:15 |
mgoddard | hrw: if we fix ocata, we have to go back to 6 months before EOL. If we leave it we can EOL on 2019-07-07 | 07:16 |
hrw | ok | 07:16 |
hrw | abandoned 651112 then | 07:17 |
mgoddard | hrw: thanks | 07:17 |
*** dciabrin has quit IRC | 07:17 | |
mgoddard | hrw: how was your vacation? | 07:17 |
hrw | mgoddard: it was Linaro Connect. In Bangkok. in 34°C and above | 07:18 |
hrw | mgoddard: some say that it was conference. I call it gathering. | 07:18 |
mgoddard | hrw: oh yeah, forgot. Sounds fun | 07:18 |
hrw | lot of fun, got sunburnt, oversweated far too many times | 07:18 |
*** hamzaachi has joined #openstack-kolla | 07:18 | |
mgoddard | see any more of Thailand? | 07:19 |
*** tosky has joined #openstack-kolla | 07:20 | |
hrw | just Bangkok | 07:20 |
mgoddard | it's an interesting city, good food | 07:20 |
hrw | temperature and humidity makes it hard to live (for me) | 07:21 |
*** hamzaachi has quit IRC | 07:21 | |
*** JqckB has joined #openstack-kolla | 07:22 | |
*** sshnaidm|off is now known as sshnaidm|pto | 07:23 | |
*** mrunge has quit IRC | 07:32 | |
*** tosky has quit IRC | 07:33 | |
*** tolisbar has joined #openstack-kolla | 07:33 | |
*** mrunge has joined #openstack-kolla | 07:33 | |
*** tosky has joined #openstack-kolla | 07:33 | |
*** JangwonLee has joined #openstack-kolla | 07:34 | |
*** andrein has joined #openstack-kolla | 07:44 | |
*** JangwonLee has quit IRC | 07:48 | |
*** happyhemant has joined #openstack-kolla | 07:48 | |
*** JangwonLee has joined #openstack-kolla | 07:49 | |
*** tolisbar has quit IRC | 07:49 | |
mnasiadka | morning | 07:54 |
*** hamzaachi has joined #openstack-kolla | 07:56 | |
*** gfidente has joined #openstack-kolla | 08:04 | |
*** dougsz has joined #openstack-kolla | 08:08 | |
*** Sravan has joined #openstack-kolla | 08:12 | |
mgoddard | morning mnasiadka | 08:14 |
*** priteau has joined #openstack-kolla | 08:15 | |
*** wolverineav has joined #openstack-kolla | 08:15 | |
*** luksky has joined #openstack-kolla | 08:15 | |
*** Sravan has quit IRC | 08:16 | |
*** andrein has quit IRC | 08:17 | |
*** JangwonLee_ has joined #openstack-kolla | 08:19 | |
*** wolverineav has quit IRC | 08:20 | |
*** JangwonLee has quit IRC | 08:22 | |
*** zbr has quit IRC | 08:28 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Don't use easy_install on Ubuntu 18+ https://review.openstack.org/651136 | 08:29 |
*** yankcrime has joined #openstack-kolla | 08:33 | |
*** chrizl has joined #openstack-kolla | 08:42 | |
*** andrein has joined #openstack-kolla | 08:46 | |
*** Luzi has joined #openstack-kolla | 08:56 | |
openstackgerrit | Lucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes https://review.openstack.org/651143 | 09:00 |
openstackgerrit | Lucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes https://review.openstack.org/651143 | 09:04 |
*** k_mouza has joined #openstack-kolla | 09:10 | |
*** hamzaachi has quit IRC | 09:12 | |
openstackgerrit | Lucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes https://review.openstack.org/651143 | 09:12 |
*** luksky has quit IRC | 09:15 | |
*** andrein has quit IRC | 09:15 | |
*** hamzaachi has joined #openstack-kolla | 09:20 | |
*** hamzaachi_ has joined #openstack-kolla | 09:28 | |
*** hamzaachi has quit IRC | 09:28 | |
*** hamzaachi__ has joined #openstack-kolla | 09:29 | |
*** hamzaachi_ has quit IRC | 09:32 | |
*** livelace has joined #openstack-kolla | 09:33 | |
*** hamzaachi_ has joined #openstack-kolla | 09:35 | |
*** hamzaachi__ has quit IRC | 09:35 | |
*** hamzaachi_ has quit IRC | 09:40 | |
*** k_mouza has quit IRC | 09:47 | |
*** andrein has joined #openstack-kolla | 09:51 | |
*** kklimonda has joined #openstack-kolla | 09:51 | |
*** k_mouza has joined #openstack-kolla | 09:57 | |
*** hamzaachi has joined #openstack-kolla | 10:00 | |
*** skramaja has quit IRC | 10:03 | |
*** luksky has joined #openstack-kolla | 10:08 | |
*** tolisbar has joined #openstack-kolla | 10:11 | |
*** k_mouza has quit IRC | 10:11 | |
*** k_mouza_ has joined #openstack-kolla | 10:11 | |
*** gary_perkins_ has quit IRC | 10:11 | |
*** gary_perkins has joined #openstack-kolla | 10:11 | |
*** JangwonLee_ has quit IRC | 10:38 | |
*** k_mouza_ has quit IRC | 10:44 | |
*** owalsh has quit IRC | 10:53 | |
*** owalsh_ has joined #openstack-kolla | 10:53 | |
*** k_mouza has joined #openstack-kolla | 10:53 | |
*** livelace has quit IRC | 10:56 | |
*** k_mouza has quit IRC | 10:58 | |
hrw | mgoddard: nsfw patch? 18+ :D | 11:01 |
mgoddard | hrw: yeah I wouldn't open it with kids around :p | 11:01 |
hrw | and it reminded me that k-a needs py3 work too ;( | 11:03 |
*** k3nny0ne has quit IRC | 11:07 | |
*** zbr has joined #openstack-kolla | 11:09 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions https://review.openstack.org/651200 | 11:32 |
*** dvx76 has joined #openstack-kolla | 11:53 | |
*** dvx76 has quit IRC | 11:53 | |
*** dvx76 has joined #openstack-kolla | 11:55 | |
*** andrein has quit IRC | 11:55 | |
dvx76 | Hi folks. Can someone confirm fluentd is currently not being set up to handle openvswitch logs? I found https://review.openstack.org/#/c/607138/ which allows fluentd to actually access the relevant logs but that's it. | 11:58 |
mgoddard | hi dvx76, that's correct. It seemed to work locally for openvswitch vswitchd but failed the check in the CI test | 12:02 |
mgoddard | dvx76: if you want to try that patch, feel free. If it works for you we could merge it | 12:02 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Use become for all docker tasks https://review.openstack.org/651210 | 12:05 |
kklimonda | hmm, can I make kolla use an existing cert on the server for HAProxy instead of copying or generating one? | 12:11 |
*** gfidente has quit IRC | 12:13 | |
*** gfidente has joined #openstack-kolla | 12:13 | |
*** JangwonLee has joined #openstack-kolla | 12:16 | |
*** wolverineav has joined #openstack-kolla | 12:16 | |
*** andrein has joined #openstack-kolla | 12:18 | |
*** wolverineav has quit IRC | 12:20 | |
*** livelace has joined #openstack-kolla | 12:20 | |
dvx76 | mgoddard, thanks! We might look into it. | 12:24 |
openstackgerrit | Merged openstack/kolla-ansible master: Set previous release to Stein https://review.openstack.org/650854 | 12:32 |
openstackgerrit | Merged openstack/kolla-ansible master: Remove shutdown of MariaDB https://review.openstack.org/650855 | 12:32 |
*** livelace has quit IRC | 12:32 | |
*** priteau has quit IRC | 12:42 | |
*** openstackgerrit has quit IRC | 12:44 | |
*** Luzi_ has joined #openstack-kolla | 12:46 | |
mgoddard | kklimonda: I don't think that's supported right now | 12:47 |
*** Luzi has quit IRC | 12:49 | |
*** openstackgerrit has joined #openstack-kolla | 12:50 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Check for 'become' in tasks that use Docker in pep8 https://review.openstack.org/651226 | 12:50 |
*** jroll has quit IRC | 12:55 | |
*** JangwonLee has quit IRC | 12:55 | |
*** baha has joined #openstack-kolla | 12:57 | |
*** owalsh_ is now known as owalsh_afk | 12:58 | |
*** jroll has joined #openstack-kolla | 12:59 | |
*** mchlumsky has joined #openstack-kolla | 13:00 | |
*** mjturek has joined #openstack-kolla | 13:07 | |
*** andrein has quit IRC | 13:17 | |
*** Luzi_ has quit IRC | 13:17 | |
*** priteau has joined #openstack-kolla | 13:27 | |
*** mjturek has quit IRC | 13:35 | |
*** mjturek has joined #openstack-kolla | 13:36 | |
*** andrein has joined #openstack-kolla | 13:47 | |
*** baha has quit IRC | 13:53 | |
*** Sravan has joined #openstack-kolla | 13:56 | |
*** Sravan has quit IRC | 13:58 | |
*** Sravan has joined #openstack-kolla | 14:00 | |
*** priteau has quit IRC | 14:04 | |
*** baha has joined #openstack-kolla | 14:05 | |
openstackgerrit | Lucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes https://review.openstack.org/651143 | 14:10 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Check for 'become' in tasks that use Docker in pep8 https://review.openstack.org/651226 | 14:19 |
*** hrw has quit IRC | 14:24 | |
*** hrw has joined #openstack-kolla | 14:26 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Remove RabbitMQ support from Bifrost https://review.openstack.org/651254 | 14:31 |
*** devep has joined #openstack-kolla | 14:38 | |
*** owalsh_afk is now known as owalsh | 14:41 | |
*** livelace has joined #openstack-kolla | 14:42 | |
*** igordc has joined #openstack-kolla | 14:42 | |
*** devep has quit IRC | 14:45 | |
*** lemko has joined #openstack-kolla | 14:46 | |
*** jistr is now known as jistr|call | 14:51 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change heat bootstrap to use internal API interface https://review.openstack.org/651261 | 14:54 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/queens: Change heat bootstrap to use internal API interface https://review.openstack.org/651262 | 14:54 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/pike: Change heat bootstrap to use internal API interface https://review.openstack.org/651263 | 14:54 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions https://review.openstack.org/651200 | 15:02 |
*** JamesBenson has joined #openstack-kolla | 15:30 | |
JamesBenson | morning all | 15:31 |
*** ivve has quit IRC | 15:32 | |
mgoddard | morning JamesBenson | 15:36 |
JamesBenson | mgoddard, rocky 7.0.1 has been hell to me.... never had issues like this before. Hoping for a deploy today, but not holding my breath. I've been trying for over a month now. | 15:37 |
klindgren | mgoddard, question on: https://review.openstack.org/#/c/650448/6/etc/kolla/passwords.yml | 15:37 |
mgoddard | JamesBenson: what issues have you been hitting? | 15:37 |
mgoddard | klindgren: go for it | 15:38 |
JamesBenson | mgoddard: some hardware, some kolla, some no idea at all - things I've never seen. | 15:39 |
JamesBenson | mgoddard: one example is ceph_mon, deploy fails, I need to kill a process, container reboots, and then the deploy passes | 15:39 |
JamesBenson | mgoddard: that's probably the only error that is consistent in it all. | 15:40 |
*** dave-mccowan has joined #openstack-kolla | 15:40 | |
mgoddard | JamesBenson: hmm, I don't tend to use kolla to deploy ceph, so probably not the best person to help there | 15:40 |
klindgren | So when I was looking at the nova_ssh stuff. It seemed to me I came across a document that said if people wanted per server ssl certs. They should create an /etc/kolla/group_vars/<node name> and then they can define a per server key there. I don't get why a similar approach wouldn't work? Or if you go with the global, you can generate a wild card cert. that's valid for the fqdn of your hosts? | 15:40 |
klindgren | Though, I kinda wonder if we want to go down the path of stepping up an offline CA and signing a bunch of server/client certs. | 15:42 |
mgoddard | klindgren: ah I see, each host might need its own set of certs. I was comparing with the API, but typically only one cert is required there, and no client cert | 15:42 |
JamesBenson | mgoddard: I deployed ceph manually once, didn't particularly go well... compute having issues accessing the vm's running. | 15:42 |
*** luksky has quit IRC | 15:42 | |
JamesBenson | mgoddard: Deploying now, mariadb slaves aren't coming up.... constant issue after issue... another, everything seems to work on our 1g backbone, but I want to use a 10g backbone and it all fails. Any idea's with that? | 15:43 |
mgoddard | JamesBenson: yeah, wouldn't recommend that. We use ceph-ansible, although it has its own issues | 15:43 |
JamesBenson | mgoddard: I can SSH using 10G, what other tests are there to debug? | 15:44 |
mgoddard | JamesBenson: are you using 7.0.1 rather than stable/rocky for kolla-ansible? Sometimes there are unreleased fixes on the stable branch | 15:44 |
JamesBenson | I tried that too.... | 15:45 |
klindgren | mgoddard, wild cards, might allow us to do a single cert for the whole deployment. But without disabling certificate validation, you would need a cert that at least matches the fqdn of the HV. Otherwise you would get a cert mismatch error (if say trying to hit the ip vs's the fqdn). I have never checked (tbh) to see if we can do a global client cert. | 15:45 |
JamesBenson | mgoddard: I've deployed 4.0.2, 6.1.0, 7.0.0 on 3 separate racks no issues... trying to do 7.0.1 for a new project/new rack and having issues. | 15:46 |
mgoddard | JamesBenson: I'm not sure why changing the network would break things. Have the network interface names changed, updated the variables to match? | 15:46 |
klindgren | But at a minimum this should at least make it where we have some level of authentication between compute nodes for libvirt. As right now kolla configures libvirt with TCP and no auth. So the compute nodes libvirt's are basically wide open. If you are doing live migration, you need to allow libvirt to talk to each other. | 15:46 |
JamesBenson | so the interfaces are all different, but I've relabeled them in maas, so they all have the same name. | 15:46 |
JamesBenson | I tried modifying the labels in multinode but deploy didn't work... | 15:47 |
JamesBenson | maybe I was doing it wrong? | 15:47 |
JamesBenson | mgoddard: Do you need all interfaces in the multinode file or just the one that has different names? | 15:47 |
JamesBenson | mgoddard: http://paste.ubuntu.com/p/w5QsZXyTz7/ | 15:48 |
mgoddard | JamesBenson: any variables in globals.yml will override those in the inventory, so you need to remove them from globals.yml if you put them in the inventory | 15:50 |
klindgren | I thought you would specify via network_interface, for each host, which interface to use? Similar to what you have there, but because you have the storage_interface commented out, network_interface also ends up being commented out? | 15:52 |
*** andrein has quit IRC | 15:56 | |
*** andrein has joined #openstack-kolla | 15:57 | |
mgoddard | JamesBenson: klindgren is correct - those variables are commented out | 15:57 |
mgoddard | klindgren: would this blueprint help you: https://blueprints.launchpad.net/kolla-ansible/+spec/support-extra-volumes | 15:58 |
klindgren | It would only for on particular instance of where we run sensu | 15:59 |
klindgren | But need to mount into the container the same version of the dell OMSA tools, as what's running on the host | 16:00 |
mgoddard | klindgren: https://review.openstack.org/651143 | 16:00 |
klindgren | So that we can monitor hosts for HW failures | 16:00 |
*** jistr|call is now known as jistr | 16:00 | |
klindgren | We worked around that for now with a local patch, to the PR that we have in upstream kolla | 16:00 |
mgoddard | klindgren: I was thinking you could mount the certs in from a host directory | 16:00 |
klindgren | But being ablate define it in some var's or something would be nice | 16:00 |
*** Sravan has quit IRC | 16:01 | |
klindgren | Like just mount in /etc/pki/? | 16:01 |
mgoddard | yeah | 16:01 |
klindgren | (That was a route I had though about. I had also thought about just hard-linking the certs in /etc/kolla/nova-(compute|libvirt) to /etc/pki | 16:02 |
klindgren | Or just making one of our existing server bootstrap scripts, move the needed certs into the config directory | 16:02 |
klindgren | Imho - all of those ideas are viable | 16:04 |
mgoddard | I think passwords.yml is probably the wrong place for the certs anyhow - we're not going to generate them via kolla-genpwd | 16:04 |
*** mjturek has quit IRC | 16:05 | |
*** livelace has quit IRC | 16:06 | |
mgoddard | how about just adding nova role default vars? | 16:06 |
klindgren | Really, for the libvirt+tls stuff, I am just looking for at this point a general path forward. As I have some internal timelines that are pushing this to be rolled out faster than what I know I can get into kolla-ansible. Which is fine. I just want to make sure that I am in a generally "ok" position with how things are done. IE thing might change slightly, but it's not a total re-write. | 16:06 |
klindgren | I was thinking about that, I can move it, there as well. I was just kinda following what was done for nova_ssh | 16:07 |
mgoddard | ok, that's fine. I think the difference is it's a lot easier to generate SSH keys than TLS certs | 16:08 |
klindgren | With the assumption that people would handle the cert generation on their own for libvirt tls. And create the correct sensible inventory var's for that stuff to work correctly? Be it wildcard certs, or per server ssl? | 16:08 |
*** Sravan has joined #openstack-kolla | 16:08 | |
JamesBenson | mgoddard klindgren: I commented them out because it was failing, but if I uncomment, that's the process? | 16:08 |
mgoddard | klindgren: yeah. Either that, or have a with_first_found loop that can use either per-host or shared cert files on localhost | 16:10 |
klindgren | Yea, we have a shell script that we use for doing it in our internal zuul gate, I guess that I can include that in the tools directory, and I can add a blurb about it in documentation under advanced config? | 16:10 |
mgoddard | klindgren: e.g /etc/kolla/config/nova/libvirt-pki/[<hostname>/]client.pem | 16:11 |
mgoddard | I don't mind too much either way, just not sure about the logistics of pasting dozens of cert blobs into YAML | 16:12 |
*** Sravan has quit IRC | 16:13 | |
klindgren | mgoddard, - from experience even with tooling.... It sucks :-D | 16:13 |
*** priteau has joined #openstack-kolla | 16:13 | |
mgoddard | klindgren: yeah, hence my suggestion to just use the files directly | 16:13 |
mgoddard | only downside is you don't get ansible-vault for free | 16:14 |
JamesBenson | mgoddard: globals, see anything weird?: http://paste.ubuntu.com/p/jrMXZMMGCx/ | 16:14 |
mgoddard | although there is a 'decrypt' argument to the copy module | 16:14 |
klindgren | I can re-work the nova-compute and nova-libvirt containers to just map in /etc/pki/ volume, based upon a conditional | 16:15 |
klindgren | Like if libvirt_tls | bool | 16:16 |
mgoddard | JamesBenson: yeah, if you have network_interface in globals.yml it will override the per-host settings in the inventory | 16:16 |
JamesBenson | mgoddard: Like I said, I've been working on trying to get at least one decent deploy for the past month, originally I tried the multinode labeling, but it failed, so I went back and changed the names of all of the 10G interfaces, so they were the same. | 16:17 |
mgoddard | klindgren: I think that use case is up to you | 16:18 |
mgoddard | JamesBenson: I can't see anything unexpected | 16:19 |
JamesBenson | mgoddard : are your interfaces all labeled the same? or do you modify your multinode file? | 16:20 |
mgoddard | JamesBenson: they're often the same. I use kayobe, which sets this up automatically with host_vars | 16:20 |
JamesBenson | mgoddard : kayobe? was that previously kolla-k8s? | 16:22 |
mgoddard | JamesBenson: no, https://kayobe.readthedocs.io | 16:22 |
JamesBenson | how stable is it? | 16:22 |
mgoddard | JamesBenson: it's based on kolla-ansible, so won't necessarily fix all your problems | 16:23 |
mgoddard | JamesBenson: its pretty stable now, although much newer than kolla | 16:23 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Use become for all docker tasks https://review.openstack.org/651210 | 16:24 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Fix ubuntu binary deploys https://review.openstack.org/651317 | 16:24 |
JamesBenson | mgoddard: Xan it do xen hypervisor? | 16:25 |
JamesBenson | Can* | 16:25 |
mgoddard | JamesBenson: I haven't tried | 16:25 |
mgoddard | JamesBenson: it would probably need some changes to make it work | 16:25 |
JamesBenson | okay, I might try for the next version without xen, right now we desperately need a xen up and running, trying to duplicate efforts from aws locally. | 16:26 |
klindgren | I thought was was moving to kvm? | 16:28 |
JamesBenson | mgoddard: nice picture: https://superuser.openstack.org/articles/kayobe-5-0-release/ | 16:28 |
mgoddard | JamesBenson: hehe thanks | 16:29 |
JamesBenson | klindgren: yes, but another company deployed a xen layer over aws and is transferring the software to us. | 16:29 |
JamesBenson | mgoddard: to add to my troubles, ansible for some reason unknown to me, decided to no longer want to do file transfers.... sftp or scp... | 16:33 |
mgoddard | JamesBenson: that's a pain. Are you using an ansible module? | 16:35 |
JamesBenson | mgoddard : just pip install ansible, nothing fancy. | 16:35 |
mgoddard | JamesBenson: more network issues? | 16:36 |
mgoddard | JamesBenson: ansible ping module might be useful for testing the network? | 16:37 |
JamesBenson | yeah, | 16:38 |
JamesBenson | things baffle my mind. | 16:38 |
JamesBenson | ... I hope I can figure out this 10g issues, just bought a whole bunch of them to outfit all of our servers. :-( | 16:44 |
JamesBenson | I think I'll skip it for now and continue with the 1g. | 16:44 |
mgoddard | JamesBenson: does connectivity look good over the network? | 16:47 |
openstackgerrit | Mark Goddard proposed openstack/kolla master: Fix ubuntu binary deploys https://review.openstack.org/651327 | 16:47 |
JamesBenson | mgoddard : I can do anything manually over the 10g, just nothing with kolla/ansible. | 16:47 |
JamesBenson | mgoddard: correction, not nothing, just very minimial things. | 16:48 |
*** mjturek has joined #openstack-kolla | 16:50 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix ubuntu binary deploys https://review.openstack.org/651317 | 16:51 |
*** emccormick has joined #openstack-kolla | 16:52 | |
JamesBenson | mgoddard: http://paste.ubuntu.com/p/PP5nBzH6tB/. from the ceph_mon container, I have to kill PID7, deploy fails, redeploy - success. | 16:59 |
JamesBenson | task: "ceph : Getting ceph mgr keyring" | 16:59 |
JamesBenson | "/usr/local/share/kolla-ansible/ansible/roles/ceph/tasks/start_mgrs.yml" | 16:59 |
mgoddard | JamesBenson: have you looked for a bug in LP? | 16:59 |
JamesBenson | LP? | 17:00 |
mgoddard | launchpad | 17:00 |
*** dougsz has quit IRC | 17:00 | |
JamesBenson | :smacks head: sorry, | 17:00 |
JamesBenson | yes, didn't find anything regarding this | 17:00 |
mgoddard | JamesBenson: anything in the logs to suggest why the mon is failing? | 17:01 |
JamesBenson | nothing, logs show no errors | 17:01 |
mgoddard | JamesBenson: I notice that start_mgrs is directly after start_mons. Is it just that the mon isn't up yet? Do we need a retry? | 17:02 |
*** JqckB has quit IRC | 17:03 | |
JamesBenson | ansible times out... | 17:03 |
JamesBenson | http://paste.ubuntu.com/p/9sYg99Sx25/ from ceph-client admin log | 17:03 |
JamesBenson | http://paste.ubuntu.com/p/xyNzn8jF6V/ from ceph-mon log | 17:03 |
JamesBenson | auth times out... | 17:04 |
*** priteau has quit IRC | 17:04 | |
mgoddard | JamesBenson: just wondering if you were to add a sleep (for testing) before that task, perhaps it would allow the mons to come up? | 17:04 |
*** andrein has quit IRC | 17:05 | |
klindgren | mgoddard, hrm... I guess my question is what would you rather see? I am open to attempting to do it a few different ways: 1.) If files on the control node that are generated via an bash script via an offline self-created CA (that will be copied in to the HV's config dir). 2.) just assuming the tls_certs are in the kolla_config dir on the hosts and using those. I can make both of those happen, if thats an acceptable direction? | 17:06 |
mgoddard | klindgren: would you be able to attend the IRC meeting tomorrow to discuss? | 17:07 |
mgoddard | 1500UTC, #openstack-meeting-4 | 17:07 |
klindgren | yea - I can make it. | 17:08 |
mgoddard | klindgren: I'm just wary of offering an opinion without agreement from at least another core | 17:08 |
*** Sravan has joined #openstack-kolla | 17:08 | |
mgoddard | klindgren: I'll put it on the agenda | 17:08 |
*** Sravan_ has joined #openstack-kolla | 17:09 | |
emccormick | klindgren Octavia kind of needs to deal with this. Maybe look at how it does it? | 17:09 |
emccormick | not per-node, but that's easy enough to deal with using host-vars | 17:09 |
emccormick | It's basically 1) in your list. It'd be nice if it were consistent between the two projects I think | 17:11 |
emccormick | generic certificate handling logic would be spiffy | 17:11 |
*** hamzaachi has quit IRC | 17:12 | |
*** Sravan has quit IRC | 17:12 | |
emccormick | jamesbenson Another thing that can bork your interface choices is system hostname vs dns ip. I have a subdomain per interface type on mine and need to make sure the hostname of my system is equivalent to the one I"m using for internal api access | 17:19 |
emccormick | if it's not, rabbit won't deploy right just for starters | 17:20 |
JamesBenson | yeah, we don't have the hostnames on any dns here. so that's easier in one way... | 17:21 |
emccormick | and you'll need to use that name as your inventory hostname also | 17:21 |
emccormick | You've pushed a static hosts file to every node? | 17:21 |
JamesBenson | yep, but that's part of the bootstrap, so no worries. | 17:21 |
emccormick | ok, so have you changed the hostnames to match the IPs on your 10 gig interfaces? | 17:22 |
emccormick | honestly the speed of the interface means squat, so you've simply got to get around however you're specifying things in /etc/hosts vs inventory vs globals | 17:22 |
*** Sravan_ has quit IRC | 17:23 | |
*** gfidente is now known as gfidente|afk | 17:23 | |
JamesBenson | hence why my multinode file is based off of IP's not hostnames | 17:23 |
JamesBenson | emccormick: "speed of the interface means squat" I figured for ceph it would be critical... we never had speed sensitive things previously, but on AWS we are running on t2/t3's with the 5g backbone because it was too slow previously. | 17:25 |
JamesBenson | that's the big push to go to the 10g now. | 17:25 |
emccormick | I don't mean it isn't important to you personally | 17:26 |
emccormick | I mean it doesn't affect Kolla | 17:26 |
emccormick | Kolla does not care how fast your NIC is | 17:26 |
JamesBenson | yeah, understood on the kolla side | 17:26 |
JamesBenson | I was deploying on the 1g indicating to use the 10g on everything | 17:26 |
JamesBenson | but honestly I've tried like every combo and permutation, nothing seems to work besides 1g ... :-( | 17:27 |
*** Sravan has joined #openstack-kolla | 17:27 | |
emccormick | doing things by IP only can't get dicey, but if you've done it before I guess carry on :) | 17:27 |
emccormick | maybe an obvious things, but any system firewalls in the way? | 17:27 |
JamesBenson | nah, nothing set on ufw or iptables. | 17:28 |
*** Sravan has quit IRC | 17:28 | |
JamesBenson | emccormick: not saying ip's are fun, but that's been our process... | 17:28 |
emccormick | so when you run the deployment now, what's the first place it dies | 17:28 |
JamesBenson | emccormick: let me get back to you, just started the deployment again... realized I still had 10g stuff in the globals. | 17:29 |
JamesBenson | now everything, etc/hosts, global, multinode file, all says 1g. | 17:30 |
emccormick | And it works with that? | 17:30 |
JamesBenson | running... let you know. | 17:30 |
emccormick | k | 17:30 |
emccormick | I literally finished a Rocky deployment 10 minutes ago with 2 x 10g bonds, so I know it works ;). Ceph done with ceph-ansible though. | 17:32 |
*** livelace has joined #openstack-kolla | 17:32 | |
emccormick | everything pulled from git, not pypi and built my own containers | 17:32 |
JamesBenson | emccormick: good to know | 17:32 |
JamesBenson | pulling from docker hub | 17:33 |
*** Sravan has joined #openstack-kolla | 17:35 | |
emccormick | shouldn't matter really. I had to override a couple things for reasons, but otherwise built just like those | 17:35 |
*** Sravan has quit IRC | 17:36 | |
JamesBenson | prechecks passed. | 17:39 |
JamesBenson | deploying. | 17:39 |
*** hamzaachi has joined #openstack-kolla | 17:40 | |
*** gmann is now known as gmann_afk | 17:40 | |
*** devep has joined #openstack-kolla | 17:41 | |
JamesBenson | emccormick: has ceph-ansible gotten easier? I looked at it early on and had some issues figuring it out so I gave up and did the ceph-deploy method. | 17:45 |
*** devep has quit IRC | 17:46 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions https://review.openstack.org/651200 | 17:46 |
*** wolverineav has joined #openstack-kolla | 17:48 | |
emccormick | jamesbenson it's fine once you get used to it. This was my first with it. Took me a couple days of tinkering to get it right | 17:48 |
JamesBenson | ok | 17:48 |
emccormick | I previously used ceph-deploy for everything also | 17:48 |
JamesBenson | looking at the latest docs | 17:48 |
JamesBenson | whew, good to know | 17:48 |
emccormick | I wanted to go to Mimic though, and Kolla wouldn't let me do that | 17:49 |
JamesBenson | yeah | 17:49 |
JamesBenson | I'm sure after stein it'll get upgraded | 17:49 |
JamesBenson | is mimic LTS? | 17:49 |
emccormick | that's also why I built cusotm kolla containers | 17:49 |
emccormick | all cephs are LTS now | 17:49 |
JamesBenson | ooOOooo | 17:49 |
emccormick | no more interim dev releases. Luminous will EOL any day now | 17:49 |
emccormick | Nautilus came out and so n-2 goes bye bye | 17:50 |
JamesBenson | yeah once Nautilus is released | 17:52 |
JamesBenson | yeah | 17:52 |
*** wolverineav has quit IRC | 17:52 | |
*** wolverineav has joined #openstack-kolla | 17:53 | |
openstackgerrit | Mark Goddard proposed openstack/kolla master: Fix ubuntu binary deploys https://review.openstack.org/651327 | 17:54 |
JamesBenson | emccormick: 1G deployed no issues. | 18:07 |
*** Sravan has joined #openstack-kolla | 18:08 | |
JamesBenson | not even that annoying ceph_mon issue where I have to kill a process. | 18:08 |
emccormick | super | 18:09 |
emccormick | JamesBenson so switch back to the 10g interfaces and let's see where stuff dies. I have to step away for the next 90 minutes or so, but if you end up with errors to share, pastebin away and let me know | 18:10 |
*** Sravan has quit IRC | 18:10 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix ubuntu binary deploys https://review.openstack.org/651317 | 18:11 |
emccormick | and toss your inventory file, globals.yml, maybe an 'ip addr show' from one of each type of box, whatever else you can think of and I'll see if I see anything | 18:11 |
*** Sravan has joined #openstack-kolla | 18:12 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions https://review.openstack.org/651200 | 18:13 |
*** gmann_afk is now known as gmann | 18:21 | |
*** luksky has joined #openstack-kolla | 18:22 | |
*** andrein has joined #openstack-kolla | 18:26 | |
*** Sravan has quit IRC | 18:27 | |
*** Sravan has joined #openstack-kolla | 18:27 | |
*** andrein has quit IRC | 18:36 | |
*** andrein has joined #openstack-kolla | 18:37 | |
*** happyhemant has quit IRC | 18:38 | |
*** JqckB has joined #openstack-kolla | 18:38 | |
*** hamzaachi has quit IRC | 18:45 | |
*** wolverineav has quit IRC | 18:46 | |
*** wolverineav has joined #openstack-kolla | 18:47 | |
*** wolverineav has quit IRC | 18:52 | |
*** Sravan has quit IRC | 18:54 | |
*** Sravan has joined #openstack-kolla | 19:07 | |
*** Sravan has quit IRC | 19:14 | |
*** Sravan has joined #openstack-kolla | 19:15 | |
*** wolverineav has joined #openstack-kolla | 19:23 | |
*** wolverineav has quit IRC | 19:23 | |
*** wolverineav has joined #openstack-kolla | 19:24 | |
*** devep has joined #openstack-kolla | 19:35 | |
*** Chaserjim has joined #openstack-kolla | 19:41 | |
*** hamzaachi has joined #openstack-kolla | 19:45 | |
*** Sravan has quit IRC | 19:48 | |
*** Chaserjim has quit IRC | 19:57 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/kolla stable/rocky: Add support to deploy manila-api with Apache/wsgi https://review.openstack.org/651368 | 19:59 |
*** tolisbar has quit IRC | 20:00 | |
*** sean-k-mooney has quit IRC | 20:05 | |
*** sean-k-mooney has joined #openstack-kolla | 20:09 | |
*** baha has quit IRC | 20:09 | |
*** devep has quit IRC | 20:17 | |
*** mjturek has quit IRC | 20:18 | |
*** wolverineav has quit IRC | 20:27 | |
*** pcaruana has quit IRC | 20:31 | |
*** wolverineav has joined #openstack-kolla | 20:32 | |
*** pcaruana has joined #openstack-kolla | 20:33 | |
*** pcaruana has quit IRC | 20:36 | |
*** wolverineav has quit IRC | 20:37 | |
*** pcaruana has joined #openstack-kolla | 20:39 | |
*** lemko has quit IRC | 20:44 | |
*** pcaruana has quit IRC | 20:47 | |
*** wolverineav has joined #openstack-kolla | 20:50 | |
*** Sravan has joined #openstack-kolla | 20:50 | |
*** hamzaachi has quit IRC | 20:53 | |
*** hamzaachi has joined #openstack-kolla | 20:54 | |
*** wolverineav has quit IRC | 20:57 | |
*** wolverineav has joined #openstack-kolla | 20:59 | |
*** wolverineav has quit IRC | 21:00 | |
*** wolverineav has joined #openstack-kolla | 21:00 | |
*** tolisbar has joined #openstack-kolla | 21:02 | |
*** hamzaachi has quit IRC | 21:07 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/kolla stable/queens: Add support to deploy manila-api with Apache/wsgi https://review.openstack.org/651383 | 21:11 |
JamesBenson | emccormick: sorry for the massive delay there. | 21:11 |
JamesBenson | emccormick: I deployed and tried to create a VM, and I've had this issue in the past, "Error: Failed to perform requested operation on instance "419b35dc-44d5-4f8c-bb29-7f47502b891d", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aa1fce99-2932-4a91-99d7-75288553faa2.]." | 21:12 |
JamesBenson | I was able to create the admin networking, upload the ubuntu image, create flavors, security groups, etc. | 21:13 |
JamesBenson | emccormick: Had this issue in the past, but never able to understand why it happened. | 21:13 |
*** wolverineav has quit IRC | 21:15 | |
*** wolverineav has joined #openstack-kolla | 21:16 | |
JamesBenson | ~. | 21:18 |
JamesBenson | ~. | 21:18 |
JamesBenson | sorry. | 21:18 |
*** wolverineav has quit IRC | 21:21 | |
*** wolverineav has joined #openstack-kolla | 21:23 | |
*** igordc has quit IRC | 21:25 | |
*** wolverineav has quit IRC | 21:27 | |
*** wolverineav has joined #openstack-kolla | 21:28 | |
*** wolverineav has quit IRC | 21:28 | |
*** wolverin_ has joined #openstack-kolla | 21:29 | |
*** wolverin_ has quit IRC | 21:31 | |
*** wolverineav has joined #openstack-kolla | 21:33 | |
*** wolverineav has quit IRC | 21:38 | |
*** mchlumsky has quit IRC | 21:44 | |
JamesBenson | emccormick: If you are free tomorrow I'd love to debug some more. | 21:46 |
JamesBenson | Headed out for the day. | 21:46 |
*** luksky has quit IRC | 21:50 | |
*** JamesBenson has quit IRC | 21:54 | |
*** gfidente|afk has quit IRC | 21:54 | |
*** wolverineav has joined #openstack-kolla | 21:58 | |
*** dvx76 has quit IRC | 22:00 | |
*** cah_link has joined #openstack-kolla | 22:08 | |
*** andrein has quit IRC | 22:10 | |
*** cah_link has quit IRC | 22:12 | |
*** JqckB has quit IRC | 22:13 | |
*** tolisbar1 has joined #openstack-kolla | 22:27 | |
*** tolisbar has quit IRC | 22:29 | |
*** tolisbar1 is now known as tolisbar | 22:29 | |
*** cah_link has joined #openstack-kolla | 22:37 | |
*** cah_link has quit IRC | 22:43 | |
*** k_mouza has joined #openstack-kolla | 22:53 | |
*** tolisbar1 has joined #openstack-kolla | 23:11 | |
*** devep has joined #openstack-kolla | 23:13 | |
*** livelace has quit IRC | 23:14 | |
*** tolisbar has quit IRC | 23:14 | |
*** tolisbar1 is now known as tolisbar | 23:14 | |
*** devep has quit IRC | 23:18 | |
*** JamesBenson has joined #openstack-kolla | 23:30 | |
*** wolverineav has quit IRC | 23:31 | |
*** wolverineav has joined #openstack-kolla | 23:33 | |
*** JamesBenson has quit IRC | 23:34 | |
*** tosky has quit IRC | 23:36 | |
*** wolverineav has quit IRC | 23:41 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!