Tuesday, 2019-04-09

« Monday, 2019-04-08 Index Wednesday, 2019-04-10 »
*** wolverineav has quit IRC00:02
*** wolverineav has joined #openstack-kolla00:03
*** wolverineav has quit IRC00:07
*** andrein has quit IRC00:13
*** wolverineav has joined #openstack-kolla00:20
*** wolverineav has quit IRC00:27
*** tolisbar1 has joined #openstack-kolla00:30
*** tolisbar has quit IRC00:34
*** tolisbar1 is now known as tolisbar00:34
*** wolverineav has joined #openstack-kolla00:49
*** wolverineav has quit IRC00:53
*** igordc has quit IRC01:06
*** hongbin has joined #openstack-kolla01:33
*** unicell has joined #openstack-kolla02:34
*** unicell has quit IRC02:34
*** hongbin has quit IRC02:35
*** hongbin has joined #openstack-kolla02:38
*** igordc has joined #openstack-kolla02:40
*** hongbin has quit IRC02:49
*** hongbin has joined #openstack-kolla02:51
*** baha has joined #openstack-kolla02:51
*** baha has quit IRC02:51
*** unicell has joined #openstack-kolla03:11
*** unicell has quit IRC03:11
*** tolisbar has quit IRC03:57
*** hongbin has quit IRC04:05
*** Sravan has joined #openstack-kolla04:47
*** wolverineav has joined #openstack-kolla04:49
*** wolverineav has quit IRC04:54
*** igordc has quit IRC05:11
*** skramaja has joined #openstack-kolla05:19
*** andrein has joined #openstack-kolla05:30
*** andrein has quit IRC05:41
*** Sravan has quit IRC05:52
*** Sravan has joined #openstack-kolla05:55
*** Sravan has quit IRC05:56
*** jbadiapa has joined #openstack-kolla06:01
*** radeks has joined #openstack-kolla06:04
*** unicell has joined #openstack-kolla06:24
*** radeks has quit IRC06:28
*** radeks has joined #openstack-kolla06:29
*** pcaruana has joined #openstack-kolla06:30
*** unicell has quit IRC06:34
*** radeks has quit IRC06:55
*** k3nny0ne has joined #openstack-kolla06:58
*** JangwonLee_ has quit IRC07:00
openstackgerritKien Nguyen proposed openstack/kolla-ansible master: Add Ansible Role Zaqar  https://review.openstack.org/40776007:02
hrwmoin07:04
hrwmgoddard: release of Stein will make Ocata EOL, right?07:04
*** gfidente has joined #openstack-kolla07:05
*** ivve has joined #openstack-kolla07:06
mgoddardmorning07:10
mgoddardhrw: nope, ocata stays in extended maintenance until it has been unmaintained for 6 months07:10
hrwso https://review.openstack.org/#/c/648312/ will be needed. will push ocata version for review in a moment07:11
*** gfidente has quit IRC07:11
mgoddardhrw: its needed if we want to maintain it. At the moment it's been broken for 2+ months, I suggest we leave it broken unless someone says they need it07:12
mgoddardhrw: since we currently have 5 stable branches07:12
hrwmgoddard: it blocks two other ocata backports07:12
hrwhttps://review.openstack.org/646439 https://review.openstack.org/63172307:13
mgoddardhrw: yeah, I'm suggesting we stop backporting07:13
mgoddardthe branch only stays alive if we keep it alive. Do you want it to stay alive?07:13
*** dciabrin_ has joined #openstack-kolla07:13
hrwmgoddard: I want to get backports queue clean07:14
mgoddardhrw: then lets abandon those backports. No one has come forward saying they want ocata07:14
*** yankcrime has quit IRC07:15
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/ocata: kafka: fix URL to tarball  https://review.openstack.org/65111207:15
mgoddardhrw: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004480.html07:15
mgoddardhrw: if we fix ocata, we have to go back to 6 months before EOL. If we leave it we can EOL on 2019-07-0707:16
hrwok07:16
hrwabandoned 651112 then07:17
mgoddardhrw: thanks07:17
*** dciabrin has quit IRC07:17
mgoddardhrw: how was your vacation?07:17
hrwmgoddard: it was Linaro Connect. In Bangkok. in 34°C and above07:18
hrwmgoddard: some say that it was conference. I call it gathering.07:18
mgoddardhrw: oh yeah, forgot. Sounds fun07:18
hrwlot of fun, got sunburnt, oversweated far too many times07:18
*** hamzaachi has joined #openstack-kolla07:18
mgoddardsee any more of Thailand?07:19
*** tosky has joined #openstack-kolla07:20
hrwjust Bangkok07:20
mgoddardit's an interesting city, good food07:20
hrwtemperature and humidity makes it hard to live (for me)07:21
*** hamzaachi has quit IRC07:21
*** JqckB has joined #openstack-kolla07:22
*** sshnaidm|off is now known as sshnaidm|pto07:23
*** mrunge has quit IRC07:32
*** tosky has quit IRC07:33
*** tolisbar has joined #openstack-kolla07:33
*** mrunge has joined #openstack-kolla07:33
*** tosky has joined #openstack-kolla07:33
*** JangwonLee has joined #openstack-kolla07:34
*** andrein has joined #openstack-kolla07:44
*** JangwonLee has quit IRC07:48
*** happyhemant has joined #openstack-kolla07:48
*** JangwonLee has joined #openstack-kolla07:49
*** tolisbar has quit IRC07:49
mnasiadkamorning07:54
*** hamzaachi has joined #openstack-kolla07:56
*** gfidente has joined #openstack-kolla08:04
*** dougsz has joined #openstack-kolla08:08
*** Sravan has joined #openstack-kolla08:12
mgoddardmorning mnasiadka08:14
*** priteau has joined #openstack-kolla08:15
*** wolverineav has joined #openstack-kolla08:15
*** luksky has joined #openstack-kolla08:15
*** Sravan has quit IRC08:16
*** andrein has quit IRC08:17
*** JangwonLee_ has joined #openstack-kolla08:19
*** wolverineav has quit IRC08:20
*** JangwonLee has quit IRC08:22
*** zbr has quit IRC08:28
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Don't use easy_install on Ubuntu 18+  https://review.openstack.org/65113608:29
*** yankcrime has joined #openstack-kolla08:33
*** chrizl has joined #openstack-kolla08:42
*** andrein has joined #openstack-kolla08:46
*** Luzi has joined #openstack-kolla08:56
openstackgerritLucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes  https://review.openstack.org/65114309:00
openstackgerritLucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes  https://review.openstack.org/65114309:04
*** k_mouza has joined #openstack-kolla09:10
*** hamzaachi has quit IRC09:12
openstackgerritLucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes  https://review.openstack.org/65114309:12
*** luksky has quit IRC09:15
*** andrein has quit IRC09:15
*** hamzaachi has joined #openstack-kolla09:20
*** hamzaachi_ has joined #openstack-kolla09:28
*** hamzaachi has quit IRC09:28
*** hamzaachi__ has joined #openstack-kolla09:29
*** hamzaachi_ has quit IRC09:32
*** livelace has joined #openstack-kolla09:33
*** hamzaachi_ has joined #openstack-kolla09:35
*** hamzaachi__ has quit IRC09:35
*** hamzaachi_ has quit IRC09:40
*** k_mouza has quit IRC09:47
*** andrein has joined #openstack-kolla09:51
*** kklimonda has joined #openstack-kolla09:51
*** k_mouza has joined #openstack-kolla09:57
*** hamzaachi has joined #openstack-kolla10:00
*** skramaja has quit IRC10:03
*** luksky has joined #openstack-kolla10:08
*** tolisbar has joined #openstack-kolla10:11
*** k_mouza has quit IRC10:11
*** k_mouza_ has joined #openstack-kolla10:11
*** gary_perkins_ has quit IRC10:11
*** gary_perkins has joined #openstack-kolla10:11
*** JangwonLee_ has quit IRC10:38
*** k_mouza_ has quit IRC10:44
*** owalsh has quit IRC10:53
*** owalsh_ has joined #openstack-kolla10:53
*** k_mouza has joined #openstack-kolla10:53
*** livelace has quit IRC10:56
*** k_mouza has quit IRC10:58
hrwmgoddard: nsfw patch? 18+ :D11:01
mgoddardhrw: yeah I wouldn't open it with kids around :p11:01
hrwand it reminded me that k-a needs py3 work too ;(11:03
*** k3nny0ne has quit IRC11:07
*** zbr has joined #openstack-kolla11:09
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions  https://review.openstack.org/65120011:32
*** dvx76 has joined #openstack-kolla11:53
*** dvx76 has quit IRC11:53
*** dvx76 has joined #openstack-kolla11:55
*** andrein has quit IRC11:55
dvx76Hi folks. Can someone confirm fluentd is currently not being set up to handle openvswitch logs? I found https://review.openstack.org/#/c/607138/ which allows fluentd to actually access the relevant logs but that's it.11:58
mgoddardhi dvx76, that's correct. It seemed to work locally for openvswitch vswitchd but failed the check in the CI test12:02
mgoddarddvx76: if you want to try that patch, feel free. If it works for you we could merge it12:02
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use become for all docker tasks  https://review.openstack.org/65121012:05
kklimondahmm, can I make kolla use an existing cert on the server for HAProxy instead of copying or generating one?12:11
*** gfidente has quit IRC12:13
*** gfidente has joined #openstack-kolla12:13
*** JangwonLee has joined #openstack-kolla12:16
*** wolverineav has joined #openstack-kolla12:16
*** andrein has joined #openstack-kolla12:18
*** wolverineav has quit IRC12:20
*** livelace has joined #openstack-kolla12:20
dvx76mgoddard, thanks! We might look into it.12:24
openstackgerritMerged openstack/kolla-ansible master: Set previous release to Stein  https://review.openstack.org/65085412:32
openstackgerritMerged openstack/kolla-ansible master: Remove shutdown of MariaDB  https://review.openstack.org/65085512:32
*** livelace has quit IRC12:32
*** priteau has quit IRC12:42
*** openstackgerrit has quit IRC12:44
*** Luzi_ has joined #openstack-kolla12:46
mgoddardkklimonda: I don't think that's supported right now12:47
*** Luzi has quit IRC12:49
*** openstackgerrit has joined #openstack-kolla12:50
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check for 'become' in tasks that use Docker in pep8  https://review.openstack.org/65122612:50
*** jroll has quit IRC12:55
*** JangwonLee has quit IRC12:55
*** baha has joined #openstack-kolla12:57
*** owalsh_ is now known as owalsh_afk12:58
*** jroll has joined #openstack-kolla12:59
*** mchlumsky has joined #openstack-kolla13:00
*** mjturek has joined #openstack-kolla13:07
*** andrein has quit IRC13:17
*** Luzi_ has quit IRC13:17
*** priteau has joined #openstack-kolla13:27
*** mjturek has quit IRC13:35
*** mjturek has joined #openstack-kolla13:36
*** andrein has joined #openstack-kolla13:47
*** baha has quit IRC13:53
*** Sravan has joined #openstack-kolla13:56
*** Sravan has quit IRC13:58
*** Sravan has joined #openstack-kolla14:00
*** priteau has quit IRC14:04
*** baha has joined #openstack-kolla14:05
openstackgerritLucas.hua proposed openstack/kolla-ansible master: Make kolla-ansible support extra volumes  https://review.openstack.org/65114314:10
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check for 'become' in tasks that use Docker in pep8  https://review.openstack.org/65122614:19
*** hrw has quit IRC14:24
*** hrw has joined #openstack-kolla14:26
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove RabbitMQ support from Bifrost  https://review.openstack.org/65125414:31
*** devep has joined #openstack-kolla14:38
*** owalsh_afk is now known as owalsh14:41
*** livelace has joined #openstack-kolla14:42
*** igordc has joined #openstack-kolla14:42
*** devep has quit IRC14:45
*** lemko has joined #openstack-kolla14:46
*** jistr is now known as jistr|call14:51
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change heat bootstrap to use internal API interface  https://review.openstack.org/65126114:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/queens: Change heat bootstrap to use internal API interface  https://review.openstack.org/65126214:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/pike: Change heat bootstrap to use internal API interface  https://review.openstack.org/65126314:54
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions  https://review.openstack.org/65120015:02
*** JamesBenson has joined #openstack-kolla15:30
JamesBensonmorning all15:31
*** ivve has quit IRC15:32
mgoddardmorning JamesBenson15:36
JamesBensonmgoddard, rocky 7.0.1 has been hell to me.... never had issues like this before.  Hoping for a deploy today, but not holding my breath.  I've been trying for over a month now.15:37
klindgrenmgoddard, question on: https://review.openstack.org/#/c/650448/6/etc/kolla/passwords.yml15:37
mgoddardJamesBenson: what issues have you been hitting?15:37
mgoddardklindgren: go for it15:38
JamesBensonmgoddard: some hardware, some kolla, some no idea at all - things I've never seen.15:39
JamesBensonmgoddard: one example is ceph_mon, deploy fails, I need to kill a process, container reboots, and then the deploy passes15:39
JamesBensonmgoddard: that's probably the only error that is consistent in it all.15:40
*** dave-mccowan has joined #openstack-kolla15:40
mgoddardJamesBenson: hmm, I don't tend to use kolla to deploy ceph, so probably not the best person to help there15:40
klindgrenSo when I was looking at the nova_ssh stuff.  It seemed to me I came across a document that said if people wanted per server ssl certs. They should create an /etc/kolla/group_vars/<node name> and then they can define a per server key there.  I don't get why a similar approach wouldn't work?  Or if you go with the global, you can generate a wild card cert. that's valid for the fqdn of your hosts?15:40
klindgrenThough, I kinda wonder if we want to go down the path of stepping up an offline CA and signing a bunch of server/client certs.15:42
mgoddardklindgren: ah I see, each host might need its own set of certs. I was comparing with the API, but typically only one cert is required there, and no client cert15:42
JamesBensonmgoddard: I deployed ceph manually once, didn't particularly go well... compute having issues accessing the vm's running.15:42
*** luksky has quit IRC15:42
JamesBensonmgoddard: Deploying now, mariadb slaves aren't coming up.... constant issue after issue... another, everything seems to work on our 1g backbone, but I want to use a 10g backbone and it all fails.  Any idea's with that?15:43
mgoddardJamesBenson: yeah, wouldn't recommend that. We use ceph-ansible, although it has its own issues15:43
JamesBensonmgoddard: I can SSH using 10G, what other tests are there to debug?15:44
mgoddardJamesBenson: are you using 7.0.1 rather than stable/rocky for kolla-ansible? Sometimes there are unreleased fixes on the stable branch15:44
JamesBensonI tried that too....15:45
klindgrenmgoddard, wild cards, might allow us to do a single cert for the whole deployment.  But without disabling certificate validation, you would need a cert that at least matches the fqdn of the HV.  Otherwise you would get a cert mismatch error (if say trying to hit the ip vs's the fqdn).  I have never checked (tbh) to see if we can do a global client cert.15:45
JamesBensonmgoddard: I've deployed 4.0.2, 6.1.0, 7.0.0 on 3 separate racks no issues... trying to do 7.0.1 for a new project/new rack and having issues.15:46
mgoddardJamesBenson: I'm not sure why changing the network would break things. Have the network interface names changed, updated the variables to match?15:46
klindgrenBut at a minimum this should at least make it where we have some level of authentication between compute nodes for libvirt.  As right now kolla configures libvirt with TCP and no auth.  So the compute nodes libvirt's are basically wide open.  If you are doing live migration, you need to allow libvirt to talk to each other.15:46
JamesBensonso the interfaces are all different, but I've relabeled them in maas, so they all have the same name.15:46
JamesBensonI tried modifying the labels in multinode but deploy didn't work...15:47
JamesBensonmaybe I was doing it wrong?15:47
JamesBensonmgoddard: Do you need all interfaces in the multinode file or just the one that has different names?15:47
JamesBensonmgoddard: http://paste.ubuntu.com/p/w5QsZXyTz7/15:48
mgoddardJamesBenson: any variables in globals.yml will override those in the inventory, so you need to remove them from globals.yml if you put them in the inventory15:50
klindgrenI thought you would specify via network_interface, for each host, which interface to use?  Similar to what you have there, but because you have the storage_interface commented out, network_interface also ends up being commented out?15:52
*** andrein has quit IRC15:56
*** andrein has joined #openstack-kolla15:57
mgoddardJamesBenson: klindgren is correct - those variables are commented out15:57
mgoddardklindgren: would this blueprint help you: https://blueprints.launchpad.net/kolla-ansible/+spec/support-extra-volumes15:58
klindgrenIt would only for on particular instance of where we run sensu15:59
klindgrenBut need to mount into the container the same version of the dell OMSA tools, as what's running on the host16:00
mgoddardklindgren: https://review.openstack.org/65114316:00
klindgrenSo that we can monitor hosts for HW failures16:00
*** jistr|call is now known as jistr16:00
klindgrenWe worked around that for now with a local patch, to the PR that we have in upstream kolla16:00
mgoddardklindgren: I was thinking you could mount the certs in from a host directory16:00
klindgrenBut being ablate define it in some var's or something would be nice16:00
*** Sravan has quit IRC16:01
klindgrenLike just mount in /etc/pki/?16:01
mgoddardyeah16:01
klindgren(That was a route I had though about.  I had also thought about just hard-linking the certs in /etc/kolla/nova-(compute|libvirt) to /etc/pki16:02
klindgrenOr just making one of our existing server bootstrap scripts, move the needed certs into the config directory16:02
klindgrenImho - all of those ideas are viable16:04
mgoddardI think passwords.yml is probably the wrong place for the certs anyhow - we're not going to generate them via kolla-genpwd16:04
*** mjturek has quit IRC16:05
*** livelace has quit IRC16:06
mgoddardhow about just adding nova role default vars?16:06
klindgrenReally, for the libvirt+tls stuff, I am just looking for at this point a general path forward.  As I have some internal timelines that are pushing this to be rolled out faster than what I know I can get into kolla-ansible.  Which is fine.  I just want to make sure that I am in a generally "ok" position with how things are done.  IE thing might change slightly, but it's not a total re-write.16:06
klindgrenI was thinking about that, I can move it, there as well.  I was just kinda following what was done for nova_ssh16:07
mgoddardok, that's fine. I think the difference is it's a lot easier to generate SSH keys than TLS certs16:08
klindgrenWith the assumption that people would handle the cert generation on their own for libvirt tls.  And create the correct sensible inventory var's for that stuff to work correctly?  Be it wildcard certs, or per server ssl?16:08
*** Sravan has joined #openstack-kolla16:08
JamesBensonmgoddard klindgren: I commented them out because it was failing, but if I uncomment, that's the process?16:08
mgoddardklindgren: yeah. Either that, or have a with_first_found loop that can use either per-host or shared cert files on localhost16:10
klindgrenYea, we have a shell script that we use for doing it in our internal zuul gate, I guess that I can include that in the tools directory, and I can add a blurb about it in documentation under advanced config?16:10
mgoddardklindgren: e.g /etc/kolla/config/nova/libvirt-pki/[<hostname>/]client.pem16:11
mgoddardI don't mind too much either way, just not sure about the logistics of pasting dozens of cert blobs into YAML16:12
*** Sravan has quit IRC16:13
klindgrenmgoddard, - from experience even with tooling.... It sucks :-D16:13
*** priteau has joined #openstack-kolla16:13
mgoddardklindgren: yeah, hence my suggestion to just use the files directly16:13
mgoddardonly downside is you don't get ansible-vault for free16:14
JamesBensonmgoddard: globals, see anything weird?: http://paste.ubuntu.com/p/jrMXZMMGCx/16:14
mgoddardalthough there is a 'decrypt' argument to the copy module16:14
klindgrenI can re-work the nova-compute and nova-libvirt containers to just map in /etc/pki/ volume, based upon a conditional16:15
klindgrenLike if libvirt_tls | bool16:16
mgoddardJamesBenson: yeah, if you have network_interface in globals.yml it will override the per-host settings in the inventory16:16
JamesBensonmgoddard: Like I said, I've been working on trying to get at least one decent deploy for the past month, originally I tried the multinode labeling, but it failed, so I went back and changed the names of all of the 10G interfaces, so they were the same.16:17
mgoddardklindgren: I think that use case is up to you16:18
mgoddardJamesBenson: I can't see anything unexpected16:19
JamesBensonmgoddard : are your interfaces all labeled the same?  or do you modify your multinode file?16:20
mgoddardJamesBenson: they're often the same. I use kayobe, which sets this up automatically with host_vars16:20
JamesBensonmgoddard : kayobe?  was that previously kolla-k8s?16:22
mgoddardJamesBenson: no, https://kayobe.readthedocs.io16:22
JamesBensonhow stable is it?16:22
mgoddardJamesBenson: it's based on kolla-ansible, so won't necessarily fix all your problems16:23
mgoddardJamesBenson: its pretty stable now, although much newer than kolla16:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use become for all docker tasks  https://review.openstack.org/65121016:24
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Fix ubuntu binary deploys  https://review.openstack.org/65131716:24
JamesBensonmgoddard: Xan it do xen hypervisor?16:25
JamesBensonCan*16:25
mgoddardJamesBenson: I haven't tried16:25
mgoddardJamesBenson: it would probably need some changes to make it work16:25
JamesBensonokay, I might try for the next version without xen, right now we desperately need a xen up and running, trying to duplicate efforts from aws locally.16:26
klindgrenI thought was was moving to kvm?16:28
JamesBensonmgoddard:  nice picture: https://superuser.openstack.org/articles/kayobe-5-0-release/16:28
mgoddardJamesBenson: hehe thanks16:29
JamesBensonklindgren: yes, but another company deployed a xen layer over aws and is transferring the software to us.16:29
JamesBensonmgoddard: to add to my troubles, ansible for some reason unknown to me, decided to no longer want to do file transfers.... sftp or scp...16:33
mgoddardJamesBenson: that's a pain. Are you using an ansible module?16:35
JamesBensonmgoddard : just pip install ansible, nothing fancy.16:35
mgoddardJamesBenson: more network issues?16:36
mgoddardJamesBenson: ansible ping module might be useful for testing the network?16:37
JamesBensonyeah,16:38
JamesBensonthings baffle my mind.16:38
JamesBenson... I hope I can figure out this 10g issues, just bought a whole bunch of them to outfit all of our servers.  :-(16:44
JamesBensonI think I'll skip it for now and continue with the 1g.16:44
mgoddardJamesBenson: does connectivity look good over the network?16:47
openstackgerritMark Goddard proposed openstack/kolla master: Fix ubuntu binary deploys  https://review.openstack.org/65132716:47
JamesBensonmgoddard : I can do anything manually over the 10g, just nothing with kolla/ansible.16:47
JamesBensonmgoddard: correction, not nothing, just very minimial things.16:48
*** mjturek has joined #openstack-kolla16:50
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix ubuntu binary deploys  https://review.openstack.org/65131716:51
*** emccormick has joined #openstack-kolla16:52
JamesBensonmgoddard: http://paste.ubuntu.com/p/PP5nBzH6tB/. from the ceph_mon container, I have to kill PID7, deploy fails, redeploy - success.16:59
JamesBensontask: "ceph : Getting ceph mgr keyring"16:59
JamesBenson"/usr/local/share/kolla-ansible/ansible/roles/ceph/tasks/start_mgrs.yml"16:59
mgoddardJamesBenson: have you looked for a bug in LP?16:59
JamesBensonLP?17:00
mgoddardlaunchpad17:00
*** dougsz has quit IRC17:00
JamesBenson:smacks head: sorry,17:00
JamesBensonyes, didn't find anything regarding this17:00
mgoddardJamesBenson: anything in the logs to suggest why the mon is failing?17:01
JamesBensonnothing, logs show no errors17:01
mgoddardJamesBenson: I notice that start_mgrs is directly after start_mons. Is it just that the mon isn't up yet? Do we need a retry?17:02
*** JqckB has quit IRC17:03
JamesBensonansible times out...17:03
JamesBensonhttp://paste.ubuntu.com/p/9sYg99Sx25/ from ceph-client admin log17:03
JamesBensonhttp://paste.ubuntu.com/p/xyNzn8jF6V/ from ceph-mon log17:03
JamesBensonauth times out...17:04
*** priteau has quit IRC17:04
mgoddardJamesBenson: just wondering if you were to add a sleep (for testing) before that task, perhaps it would allow the mons to come up?17:04
*** andrein has quit IRC17:05
klindgrenmgoddard, hrm... I guess my question is what would you rather see? I am open to attempting to do it a few different ways:  1.) If files on the control node that are generated via an bash script via an offline self-created CA (that will be copied in to the HV's config dir). 2.) just assuming the tls_certs are in the kolla_config dir on the hosts and using those.   I can make both of those happen, if thats an acceptable direction?17:06
mgoddardklindgren: would you be able to attend the IRC meeting tomorrow to discuss?17:07
mgoddard1500UTC, #openstack-meeting-417:07
klindgrenyea - I can make it.17:08
mgoddardklindgren: I'm just wary of offering an opinion without agreement from at least another core17:08
*** Sravan has joined #openstack-kolla17:08
mgoddardklindgren: I'll put it on the agenda17:08
*** Sravan_ has joined #openstack-kolla17:09
emccormickklindgren Octavia kind of needs to deal with this. Maybe look at how it does it?17:09
emccormicknot per-node, but that's easy enough to deal with using host-vars17:09
emccormickIt's basically 1) in your list. It'd be nice if it were consistent between the two projects I think17:11
emccormickgeneric certificate handling logic would be spiffy17:11
*** hamzaachi has quit IRC17:12
*** Sravan has quit IRC17:12
emccormickjamesbenson Another thing that can bork your interface choices is system hostname vs dns ip. I have a subdomain per interface type on mine and need to make sure the hostname of my system is equivalent to the one I"m using for internal api access17:19
emccormickif it's not, rabbit won't deploy right just for starters17:20
JamesBensonyeah, we don't have the hostnames on any dns here. so that's easier in one way...17:21
emccormickand you'll need to use that name as your inventory hostname also17:21
emccormickYou've pushed a static hosts file to every node?17:21
JamesBensonyep, but that's part of the bootstrap, so no worries.17:21
emccormickok, so have you changed the hostnames to match the IPs on your 10 gig interfaces?17:22
emccormickhonestly the speed of the interface means squat, so you've simply got to get around however you're specifying things in /etc/hosts vs inventory vs globals17:22
*** Sravan_ has quit IRC17:23
*** gfidente is now known as gfidente|afk17:23
JamesBensonhence why my multinode file is based off of IP's not hostnames17:23
JamesBensonemccormick: "speed of the interface means squat" I figured for ceph it would be critical... we never had speed sensitive things previously, but on AWS we are running on t2/t3's with the 5g backbone because it was too slow previously.17:25
JamesBensonthat's the big push to go to the 10g now.17:25
emccormickI don't mean it isn't important to you personally17:26
emccormickI mean it doesn't affect Kolla17:26
emccormickKolla does not care how fast your NIC is17:26
JamesBensonyeah, understood on the kolla side17:26
JamesBensonI was deploying on the 1g indicating to use the 10g on everything17:26
JamesBensonbut honestly I've tried like every combo and permutation, nothing seems to work besides 1g ... :-(17:27
*** Sravan has joined #openstack-kolla17:27
emccormickdoing things by IP only can't get dicey, but if you've done it before I guess carry on :)17:27
emccormickmaybe an obvious things, but any system firewalls in the way?17:27
JamesBensonnah, nothing set on ufw or iptables.17:28
*** Sravan has quit IRC17:28
JamesBensonemccormick: not saying ip's are fun, but that's been our process...17:28
emccormickso when you run the deployment now, what's the first place it dies17:28
JamesBensonemccormick: let me get back to you, just started the deployment again... realized I still had 10g stuff in the globals.17:29
JamesBensonnow everything, etc/hosts, global, multinode file, all says 1g.17:30
emccormickAnd it works with that?17:30
JamesBensonrunning... let you know.17:30
emccormickk17:30
emccormickI literally finished a Rocky deployment 10 minutes ago with 2 x 10g bonds, so I know it works ;). Ceph done with ceph-ansible though.17:32
*** livelace has joined #openstack-kolla17:32
emccormickeverything pulled from git, not pypi and built my own containers17:32
JamesBensonemccormick: good to know17:32
JamesBensonpulling from docker hub17:33
*** Sravan has joined #openstack-kolla17:35
emccormickshouldn't matter really. I had to override a couple things for reasons, but otherwise built just like those17:35
*** Sravan has quit IRC17:36
JamesBensonprechecks passed.17:39
JamesBensondeploying.17:39
*** hamzaachi has joined #openstack-kolla17:40
*** gmann is now known as gmann_afk17:40
*** devep has joined #openstack-kolla17:41
JamesBensonemccormick: has ceph-ansible gotten easier?  I looked at it early on and had some issues figuring it out so I gave up and did the ceph-deploy method.17:45
*** devep has quit IRC17:46
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions  https://review.openstack.org/65120017:46
*** wolverineav has joined #openstack-kolla17:48
emccormickjamesbenson it's fine once you get used to it. This was my first with it. Took me a couple days of tinkering to get it right17:48
JamesBensonok17:48
emccormickI previously used ceph-deploy for everything also17:48
JamesBensonlooking at the latest docs17:48
JamesBensonwhew, good to know17:48
emccormickI wanted to go to Mimic though, and Kolla wouldn't let me do that17:49
JamesBensonyeah17:49
JamesBensonI'm sure after stein it'll get upgraded17:49
JamesBensonis mimic LTS?17:49
emccormickthat's also why I built cusotm kolla containers17:49
emccormickall cephs are LTS now17:49
JamesBensonooOOooo17:49
emccormickno more interim dev releases. Luminous will EOL any day now17:49
emccormickNautilus came out and so n-2 goes bye bye17:50
JamesBensonyeah once Nautilus is released17:52
JamesBensonyeah17:52
*** wolverineav has quit IRC17:52
*** wolverineav has joined #openstack-kolla17:53
openstackgerritMark Goddard proposed openstack/kolla master: Fix ubuntu binary deploys  https://review.openstack.org/65132717:54
JamesBensonemccormick: 1G deployed no issues.18:07
*** Sravan has joined #openstack-kolla18:08
JamesBensonnot even that annoying ceph_mon issue where I have to kill a process.18:08
emccormicksuper18:09
emccormickJamesBenson so switch back to the 10g interfaces and let's see where stuff dies. I have to step away for the next 90 minutes or so, but if you end up with errors to share, pastebin away and let me know18:10
*** Sravan has quit IRC18:10
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix ubuntu binary deploys  https://review.openstack.org/65131718:11
emccormickand toss your inventory file, globals.yml, maybe an 'ip addr show' from one of each type of box, whatever else you can think of and I'll see if I see anything18:11
*** Sravan has joined #openstack-kolla18:12
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Update quickstart instructions  https://review.openstack.org/65120018:13
*** gmann_afk is now known as gmann18:21
*** luksky has joined #openstack-kolla18:22
*** andrein has joined #openstack-kolla18:26
*** Sravan has quit IRC18:27
*** Sravan has joined #openstack-kolla18:27
*** andrein has quit IRC18:36
*** andrein has joined #openstack-kolla18:37
*** happyhemant has quit IRC18:38
*** JqckB has joined #openstack-kolla18:38
*** hamzaachi has quit IRC18:45
*** wolverineav has quit IRC18:46
*** wolverineav has joined #openstack-kolla18:47
*** wolverineav has quit IRC18:52
*** Sravan has quit IRC18:54
*** Sravan has joined #openstack-kolla19:07
*** Sravan has quit IRC19:14
*** Sravan has joined #openstack-kolla19:15
*** wolverineav has joined #openstack-kolla19:23
*** wolverineav has quit IRC19:23
*** wolverineav has joined #openstack-kolla19:24
*** devep has joined #openstack-kolla19:35
*** Chaserjim has joined #openstack-kolla19:41
*** hamzaachi has joined #openstack-kolla19:45
*** Sravan has quit IRC19:48
*** Chaserjim has quit IRC19:57
openstackgerritGoutham Pacha Ravi proposed openstack/kolla stable/rocky: Add support to deploy manila-api with Apache/wsgi  https://review.openstack.org/65136819:59
*** tolisbar has quit IRC20:00
*** sean-k-mooney has quit IRC20:05
*** sean-k-mooney has joined #openstack-kolla20:09
*** baha has quit IRC20:09
*** devep has quit IRC20:17
*** mjturek has quit IRC20:18
*** wolverineav has quit IRC20:27
*** pcaruana has quit IRC20:31
*** wolverineav has joined #openstack-kolla20:32
*** pcaruana has joined #openstack-kolla20:33
*** pcaruana has quit IRC20:36
*** wolverineav has quit IRC20:37
*** pcaruana has joined #openstack-kolla20:39
*** lemko has quit IRC20:44
*** pcaruana has quit IRC20:47
*** wolverineav has joined #openstack-kolla20:50
*** Sravan has joined #openstack-kolla20:50
*** hamzaachi has quit IRC20:53
*** hamzaachi has joined #openstack-kolla20:54
*** wolverineav has quit IRC20:57
*** wolverineav has joined #openstack-kolla20:59
*** wolverineav has quit IRC21:00
*** wolverineav has joined #openstack-kolla21:00
*** tolisbar has joined #openstack-kolla21:02
*** hamzaachi has quit IRC21:07
openstackgerritGoutham Pacha Ravi proposed openstack/kolla stable/queens: Add support to deploy manila-api with Apache/wsgi  https://review.openstack.org/65138321:11
JamesBensonemccormick: sorry for the massive delay there.21:11
JamesBensonemccormick: I deployed and tried to create a VM, and I've had this issue in the past, "Error: Failed to perform requested operation on instance "419b35dc-44d5-4f8c-bb29-7f47502b891d", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aa1fce99-2932-4a91-99d7-75288553faa2.]."21:12
JamesBensonI was able to create the admin networking, upload the ubuntu image, create flavors, security groups, etc.21:13
JamesBensonemccormick: Had this issue in the past, but never able to understand why it happened.21:13
*** wolverineav has quit IRC21:15
*** wolverineav has joined #openstack-kolla21:16
JamesBenson~.21:18
JamesBenson~.21:18
JamesBensonsorry.21:18
*** wolverineav has quit IRC21:21
*** wolverineav has joined #openstack-kolla21:23
*** igordc has quit IRC21:25
*** wolverineav has quit IRC21:27
*** wolverineav has joined #openstack-kolla21:28
*** wolverineav has quit IRC21:28
*** wolverin_ has joined #openstack-kolla21:29
*** wolverin_ has quit IRC21:31
*** wolverineav has joined #openstack-kolla21:33
*** wolverineav has quit IRC21:38
*** mchlumsky has quit IRC21:44
JamesBensonemccormick: If you are free tomorrow I'd love to debug some more.21:46
JamesBensonHeaded out for the day.21:46
*** luksky has quit IRC21:50
*** JamesBenson has quit IRC21:54
*** gfidente|afk has quit IRC21:54
*** wolverineav has joined #openstack-kolla21:58
*** dvx76 has quit IRC22:00
*** cah_link has joined #openstack-kolla22:08
*** andrein has quit IRC22:10
*** cah_link has quit IRC22:12
*** JqckB has quit IRC22:13
*** tolisbar1 has joined #openstack-kolla22:27
*** tolisbar has quit IRC22:29
*** tolisbar1 is now known as tolisbar22:29
*** cah_link has joined #openstack-kolla22:37
*** cah_link has quit IRC22:43
*** k_mouza has joined #openstack-kolla22:53
*** tolisbar1 has joined #openstack-kolla23:11
*** devep has joined #openstack-kolla23:13
*** livelace has quit IRC23:14
*** tolisbar has quit IRC23:14
*** tolisbar1 is now known as tolisbar23:14
*** devep has quit IRC23:18
*** JamesBenson has joined #openstack-kolla23:30
*** wolverineav has quit IRC23:31
*** wolverineav has joined #openstack-kolla23:33
*** JamesBenson has quit IRC23:34
*** tosky has quit IRC23:36
*** wolverineav has quit IRC23:41
« Monday, 2019-04-08 Index Wednesday, 2019-04-10 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!