| *** mpjetta has quit IRC | 00:14 | |
| *** marst has joined #openstack-ansible | 00:19 | |
| *** cmart has quit IRC | 00:31 | |
| *** gyee has quit IRC | 00:39 | |
| *** marst has quit IRC | 00:42 | |
| *** cshen has joined #openstack-ansible | 00:57 | |
| *** cshen has quit IRC | 01:02 | |
| cloudnull | evenings all | 01:14 |
|---|---|---|
| *** cyberpear has quit IRC | 01:59 | |
| *** cmart has joined #openstack-ansible | 02:23 | |
| *** BjoernT has joined #openstack-ansible | 02:40 | |
| *** BjoernT_ has joined #openstack-ansible | 02:45 | |
| *** BjoernT has quit IRC | 02:48 | |
| *** dave-mccowan has joined #openstack-ansible | 02:50 | |
| *** dave-mccowan has quit IRC | 02:54 | |
| *** cshen has joined #openstack-ansible | 02:58 | |
| *** kmadac3 has joined #openstack-ansible | 03:01 | |
| *** cshen has quit IRC | 03:02 | |
| *** kmadac2 has quit IRC | 03:04 | |
| *** cmart has quit IRC | 04:21 | |
| *** udesale has joined #openstack-ansible | 04:22 | |
| *** raukadah is now known as chandankumar | 05:50 | |
| *** jbadiapa has quit IRC | 06:03 | |
| *** jbadiapa has joined #openstack-ansible | 06:04 | |
| *** ivve has joined #openstack-ansible | 06:37 | |
| *** DanyC has joined #openstack-ansible | 06:44 | |
| *** shardy has quit IRC | 06:44 | |
| *** shardy has joined #openstack-ansible | 06:52 | |
| *** DanyC has quit IRC | 06:54 | |
| *** rgogunskiy has joined #openstack-ansible | 06:57 | |
| *** BjoernT has joined #openstack-ansible | 07:00 | |
| *** kopecmartin|off is now known as kopecmartin | 07:00 | |
| *** BjoernT_ has quit IRC | 07:02 | |
| *** maxbab has joined #openstack-ansible | 07:12 | |
| *** markvoelker has quit IRC | 07:19 | |
| *** tosky has joined #openstack-ansible | 07:23 | |
| *** cshen has joined #openstack-ansible | 07:39 | |
| *** aedc has quit IRC | 07:43 | |
| *** gkadam_ has joined #openstack-ansible | 07:50 | |
| *** luksky has joined #openstack-ansible | 07:54 | |
| *** hamzaachi has joined #openstack-ansible | 07:55 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-tests master: Set tempest_image_dir to /opt/cache/files in openstack CI https://review.openstack.org/641304 | 07:55 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Fix tempest_image_dir default var to user's home tempest-image dir https://review.openstack.org/640742 | 07:55 |
| *** hamzaachi has quit IRC | 07:56 | |
| *** hamzaachi has joined #openstack-ansible | 07:57 | |
| *** flaviosr has quit IRC | 08:01 | |
| *** hamzaachi has quit IRC | 08:02 | |
| *** fnpanic has joined #openstack-ansible | 08:06 | |
| fnpanic | good morning | 08:06 |
| *** pcaruana has joined #openstack-ansible | 08:07 | |
| *** pcaruana has quit IRC | 08:11 | |
| *** BjoernT has quit IRC | 08:23 | |
| *** miloa has joined #openstack-ansible | 08:23 | |
| *** pcaruana has joined #openstack-ansible | 08:23 | |
| *** hamzaachi has joined #openstack-ansible | 08:24 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Fix bashate xargs run https://review.openstack.org/643104 | 08:25 |
| *** hamzaachi has quit IRC | 08:25 | |
| *** hamzaachi has joined #openstack-ansible | 08:26 | |
| *** fnpanic has quit IRC | 08:26 | |
| *** fnpanic has joined #openstack-ansible | 08:41 | |
| *** CeeMac has joined #openstack-ansible | 08:58 | |
| CeeMac | morning | 09:00 |
| *** simon-AS559 has joined #openstack-ansible | 09:02 | |
| *** Dantalion has joined #openstack-ansible | 09:09 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-tests master: Set tempest_image_dir to /opt/cache/files in openstack CI https://review.openstack.org/641304 | 09:11 |
| miloa | morning | 09:14 |
| *** aedc has joined #openstack-ansible | 09:18 | |
| *** DanyC has joined #openstack-ansible | 09:22 | |
| fnpanic | /UNIGNORE noonedeadpunk | 09:23 |
| fnpanic | UNIGNORE noonedeadpunk | 09:23 |
| fnpanic | argh | 09:23 |
| *** pcaruana has quit IRC | 09:48 | |
| chandankumar | jrosser: Hello | 10:00 |
| chandankumar | jrosser: I need some help here https://review.openstack.org/#/c/641304/ How Can I pass default value while using ternary? | 10:01 |
| jrosser | chandankumar: you can’t access role defaults from outside the role | 10:12 |
| jrosser | You could conditionally include and extra tasks file that sets nodepool specific vars, that may be a better pattern | 10:14 |
| chandankumar | jrosser: I think we donot need http://git.openstack.org/cgit/openstack/openstack-ansible/tree/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#n189 | 10:17 |
| chandankumar | in openstack-ansible-tests | 10:17 |
| chandankumar | as it is already taken care by openstack-ansible | 10:18 |
| jrosser | All of the osa role tests currently use Openstack-ansible-tests though | 10:20 |
| jrosser | It depends what you are tying to fix | 10:20 |
| jrosser | Right, I’m away for the rest of today..... | 10:20 |
| chandankumar | abandoning the ansible-tests patch | 10:20 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Fix tempest_image_dir default var to user's home tempest-image dir https://review.openstack.org/640742 | 10:21 |
| jamesdenton | admin0 It is not abandoned, but does need to be rebased and revisited. | 10:41 |
| *** udesale has quit IRC | 10:56 | |
| *** udesale has joined #openstack-ansible | 10:56 | |
| *** nicolasbock has joined #openstack-ansible | 10:57 | |
| *** yolanda has quit IRC | 11:02 | |
| *** yolanda has joined #openstack-ansible | 11:03 | |
| *** shardy has quit IRC | 11:11 | |
| *** udesale has quit IRC | 11:15 | |
| *** udesale has joined #openstack-ansible | 11:15 | |
| *** udesale has quit IRC | 11:27 | |
| *** aedc has quit IRC | 11:27 | |
| *** udesale has joined #openstack-ansible | 11:28 | |
| *** nicolasbock has quit IRC | 11:29 | |
| *** nicolasbock has joined #openstack-ansible | 11:30 | |
| *** rgogunskiy has quit IRC | 11:31 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Fix bashate https://review.openstack.org/643300 | 11:32 |
| *** pcaruana has joined #openstack-ansible | 11:32 | |
| *** rgogunskiy has joined #openstack-ansible | 11:33 | |
| odyssey4me | guilhermesp mnaser I think it may be best to just branch stable/rocky from the current head of master for os_mistral, then use that branch for the integrated build. Any fixes that go in can then be specific to the branch. At the very least, the way the python builds happen will be different between Stein (new method) and Rocky (old method). | 11:39 |
| *** dave-mccowan has joined #openstack-ansible | 11:46 | |
| *** luksky has quit IRC | 11:50 | |
| *** rgogunskiy has quit IRC | 11:55 | |
| *** rgogunskiy has joined #openstack-ansible | 11:56 | |
| *** ansmith has quit IRC | 11:58 | |
| chandankumar | odyssey4me: Hello | 12:14 |
| chandankumar | odyssey4me: please have a look at this one https://review.openstack.org/#/c/640742/ when free! | 12:14 |
| *** luksky has joined #openstack-ansible | 12:18 | |
| ostackz | evrardjp jrosser Hi, how exactly should "real" SSL certificates work? I mean I have specified cert related variables like haproxy_user_ssl_cert in user_variables.yml, run playbooks and horizon URL is "green" now, but "list stacks" still fails. Seems this is due to IP used in URL, not hostname https://pastebin.com/raw/7yfE4H1B | 12:23 |
| evrardjp | https://docs.openstack.org/openstack-ansible/latest/user/security/index.html ? | 12:33 |
| chandankumar | odyssey4me: thanks! | 12:34 |
| ostackz | evrardjp, thanks, must be I missed that I need to define ssl cert file for all services not only haproxy | 12:36 |
| ostackz | odyssey4me jrosser still just using "real" certs for all services does not seem enough, because SSL lib complains that it does not like IP to be used in URL: ERROR urllib3.connection <...> CertificateError: hostname '10.x.x.x' doesn't match either of '*.zzzzzzzzz.zz', 'zzzzzzzzz.zz' | 12:43 |
| ostackz | not clear how to make heat to work. Is there any variable I can force heat to use URLs instead of IP? Because I cannot "real" cert for internal IP | 12:47 |
| *** ansmith has joined #openstack-ansible | 12:48 | |
| *** ansmith_ has joined #openstack-ansible | 12:50 | |
| *** ansmith has quit IRC | 12:53 | |
| *** hwoarang has quit IRC | 12:56 | |
| *** hwoarang has joined #openstack-ansible | 12:57 | |
| *** maxbab has quit IRC | 13:00 | |
| evrardjp | sorry I am in meetings for the whole day, I can't really help you now | 13:03 |
| ioni | ostackz, i for one have the endpoints configured with domain | 13:08 |
| ioni | adding the ssl to haproxy makes all services that use public url(endpoint) to have a valid ssl | 13:09 |
| ioni | external_lb_vip_address: yourdomain | 13:09 |
| ioni | in /etc/openstack_deploy/openstack_user_config.yml | 13:09 |
| ostackz | ioni I did put external_lb_vip_address ad domain name, but then Keepalived fails saying domain is not IP address | 13:10 |
| ostackz | ioni do you have Rocky with working heat stacks? | 13:15 |
| *** fnpanic has quit IRC | 13:15 | |
| ioni | i don't really use heat | 13:15 |
| ostackz | ioni in horizon, if you click on Orchestration->Stacks - do you see empty list(this is ok) or "Error: Unable to retrieve stack list."? | 13:17 |
| ioni | i don't have horizon | 13:20 |
| ioni | let me try something | 13:20 |
| ostackz | ioni from cli that would be "openstack stack list" | 13:20 |
| ioni | yeah, let me switch to public url | 13:20 |
| ioni | no error | 13:21 |
| ioni | works fine | 13:21 |
| ostackz | and do you have Rocky release? | 13:21 |
| ioni | yes | 13:21 |
| ioni | haproxy_keepalived_external_vip_cidr: "{{external_lb_vip_address}}/32" | 13:21 |
| ioni | haproxy_keepalived_internal_vip_cidr: "{{internal_lb_vip_address}}/32" | 13:21 |
| ioni | in user_variables.yml | 13:21 |
| ioni | that's all i did | 13:22 |
| ostackz | ok, sounds promissing! Need to understand why specifying external_lb_vip_address: my.hostname.here does not work for me | 13:23 |
| ioni | can you ping it? | 13:23 |
| ioni | i had an issue when the hostname of the controller was the same as external_lb_vip_address and /etc/hosts was messed up | 13:24 |
| ioni | had the wrong ip mapped | 13:24 |
| ostackz | network wise it is ok - can ping and open ports. The issue is that ssl libs do not like IP in URL of site, but I cannot get "real" cert for internal IP. https://pastebin.com/raw/7yfE4H1B | 13:28 |
| ioni | openstack endoint list | 13:29 |
| *** marst has joined #openstack-ansible | 13:31 | |
| *** rgogunskiy has quit IRC | 13:32 | |
| ostackz | ioni all endpoints are listed as IP https://x.x.x.x:port | 13:42 |
| ioni | even for private? | 13:43 |
| ioni | *internal | 13:43 |
| ostackz | yes, internal, public, admin | 13:44 |
| ostackz | IP of coures differs, but there are no URLs | 13:44 |
| ostackz | in your case you have name based URLs, right? For public endpoints | 13:45 |
| *** marst has quit IRC | 13:45 | |
| guilhermesp | mornings | 13:48 |
| guilhermesp | odyssey4me: yeah that'll be the idea. mnaser is going to branch it today and then I'm gonna make the fixes here https://review.openstack.org/#/c/642614/1 | 13:48 |
| guilhermesp | we are about to deploy it in a rocky env today hopefully | 13:49 |
| *** hwoarang has quit IRC | 13:53 | |
| *** hwoarang has joined #openstack-ansible | 13:55 | |
| odyssey4me | guilhermesp I don't think the python_venv_build role for stable/rocky is at the same level as that for master, so using os_mistral as-is may present some challenges | 14:05 |
| jrosser | ostackz: if you change the external vip from an ip to an fqdn you will need to run all the playbooks again, to update the service catalog. Have you done that? | 14:06 |
| ostackz | jrosser yes, but afterwards keepalived complained that url is not valid IP address | 14:07 |
| ostackz | still wondering if it is ok to set external_lb_vip_address as fqdn not IP | 14:08 |
| *** aludwar has joined #openstack-ansible | 14:08 | |
| *** cmart has joined #openstack-ansible | 14:08 | |
| jrosser | Yes it is ok | 14:09 |
| ostackz | ioni jrosser and I have "haproxy_keepalived_external_vip_cidr: "{{external_lb_vip_address}}/24"", may be netmask is not ok? Should it be /32 in all cases? | 14:09 |
| jrosser | But it must resolve properly with a dns entry sonewhere | 14:09 |
| *** marst has joined #openstack-ansible | 14:10 | |
| ostackz | ioni, jrosser, thanks! allright I need to get endpoints to URL then. Also will have to check if putting /32 in mask resolves keepalived issue. | 14:12 |
| cloudnull | mornings | 14:13 |
| jrosser | ostackz: yes one step at a time, get the service catalog straight or things won’t be right later | 14:14 |
| *** cmart has quit IRC | 14:14 | |
| jrosser | ostackz: I have haproxy_keepalived_external_vip_cidr set to “<external-ip>/32” | 14:16 |
| *** cmart has joined #openstack-ansible | 14:17 | |
| jrosser | And external_lb_vip_address to <fqdn> | 14:17 |
| jrosser | And your dns setup must agree about those two things | 14:18 |
| *** fnpanic has joined #openstack-ansible | 14:22 | |
| jrosser | ceph repos look brok somehow http://logs.openstack.org/00/643300/1/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/6798a07/job-output.txt.gz#_2019-03-14_12_36_42_170380 | 14:25 |
| mnaser | i think we have to readapt things to not use python_venv_build in a stable/rocky change | 14:32 |
| mnaser | cc guilhermesp ^ | 14:32 |
| *** hwoarang has quit IRC | 14:35 | |
| odyssey4me | mnaser yep, it'll need to align to doing what the stable/rocky roles do | 14:35 |
| *** hwoarang has joined #openstack-ansible | 14:36 | |
| guilhermesp | once we have a branch for rocky we can adapt I think. I'd assume I can get another rocky roles as reference? | 14:36 |
| guilhermesp | mnaser odyssey4me | 14:36 |
| cloudnull | jrosser that could be the repos are sync'ing which would cause the md5sum mismatch | 14:36 |
| jrosser | it's been like that for ~24 hours now | 14:37 |
| mnaser | guilhermesp: yeah you can look at any role inside stable/rocky and itll have the 'older' format which doesnt use venv | 14:37 |
| guilhermesp | cool then mnaser | 14:38 |
| fnpanic | jrosser: we noticed this also. Looks like the error happend on the 12. in the ceph repo | 14:39 |
| fnpanic | it is bricked for now | 14:39 |
| * jrosser asks in #ceph-devel | 14:42 | |
| jrosser | also, is there a good reason why all the playbooks in the tests repo are connection: local? Does something bad happen if it is not like that? | 14:44 |
| *** hwoarang has quit IRC | 14:46 | |
| odyssey4me | jrosser it's complicated... they're not all like that, but they're supposed to be - the role tests do not run as root, so accessing localhost things gets more complicated | 14:48 |
| odyssey4me | we've hit plenty of inventory and fact gathering issues along the way, which is one of the many nails in the role test coffin | 14:49 |
| jrosser | odyssey4me: i am stuck with this http://logs.openstack.org/45/641445/16/check/openstack-ansible-functional-ubuntu-bionic/093e1ed/job-output.txt.gz#_2019-03-12_18_43_16_691020 | 14:49 |
| jrosser | i have installed pyopenssl on the host but i think that it isnt in the tox env | 14:49 |
| jrosser | so the ansible openssl_* modules can't see it | 14:49 |
| jrosser | so i was wondering if to move that part to be *not* connection: local to be in the regular host environment | 14:50 |
| odyssey4me | jrosser yep, that's exactly it - so either put the modules in the tox env, or change the python_venv_interpreter for that task | 14:51 |
| jrosser | which would be best - i spend 10x more time on the test than the functionality by now :( | 14:51 |
| *** hwoarang has joined #openstack-ansible | 14:52 | |
| odyssey4me | I think the latter is likely easier | 14:52 |
| odyssey4me | something like ansible_python_interpreter: "/usr/bin/python2" | 14:53 |
| *** fnpanic has quit IRC | 14:54 | |
| *** yolanda has quit IRC | 14:57 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Install user supplied CA certificates into system trust store https://review.openstack.org/641445 | 14:59 |
| jrosser | odyssey4me: thanks for the tip! | 14:59 |
| *** BjoernT has joined #openstack-ansible | 15:02 | |
| *** cmart has quit IRC | 15:34 | |
| *** hamzy has quit IRC | 15:43 | |
| *** yolanda has joined #openstack-ansible | 15:45 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Install user supplied CA certificates into system trust store https://review.openstack.org/641445 | 15:46 |
| spotz | mnaser do we have a plan for PTG yet? | 16:05 |
| *** ostackz has quit IRC | 16:11 | |
| *** ivve has quit IRC | 16:25 | |
| *** hamzaachi has quit IRC | 16:25 | |
| *** kopecmartin is now known as kopecmartin|off | 16:26 | |
| *** cshen has quit IRC | 16:27 | |
| *** hamzaachi has joined #openstack-ansible | 16:27 | |
| *** hamzaachi_ has joined #openstack-ansible | 16:30 | |
| *** hamzaachi has quit IRC | 16:32 | |
| *** luksky has quit IRC | 16:41 | |
| chandankumar | mnaser: cloudnull https://review.openstack.org/#/c/640742/ please have a look, when free! | 16:42 |
| spotz | chandankumar: done | 16:45 |
| chandankumar | spotz: thanks! | 16:45 |
| guilhermesp | chandankumar: have you seen jobs failing like this ? http://logs.openstack.org/42/642842/2/check/openstack-ansible-deploy-aio_distro_lxc-opensuse-423/a10a1b5/logs/ara-report/result/92126a64-3f78-4462-a62c-f5117329e569/ | 16:45 |
| *** emine__ has quit IRC | 16:46 | |
| chandankumar | guilhermesp: checking! | 16:46 |
| spotz | chandankumar: NP | 16:46 |
| guilhermesp | chandankumar: thx! | 16:48 |
| *** cmart has joined #openstack-ansible | 16:49 | |
| chandankumar | guilhermesp: it appears to be something new , need to take a look! | 16:50 |
| *** gkadam_ has quit IRC | 16:50 | |
| guilhermesp | all right! I've been seeing this since last Tuesday I think... rechecked yesterday thinking that was something intermittent | 16:52 |
| chandankumar | do you know where we dump /var/log/tempest.log in logs folder? | 16:53 |
| guilhermesp | hum not sure, digging into the reports to see if I find | 16:55 |
| *** udesale has quit IRC | 17:06 | |
| *** gyee has joined #openstack-ansible | 17:07 | |
| *** hamzy has joined #openstack-ansible | 17:09 | |
| chandankumar | I think I reproduced the issue | 17:14 |
| guilhermesp | nice chandankumar ? something that needs a PR? you don't have time I can submit one | 17:22 |
| guilhermesp | (exclude ? mark :P ) | 17:22 |
| chandankumar | http://paste.openstack.org/show/747805/ | 17:23 |
| chandankumar | i have done pip install tempest, tempest init foobar then temepst run | 17:23 |
| chandankumar | I have asked gmann to look into it | 17:23 |
| guilhermesp | that's nice, thanks chandankumar ! | 17:24 |
| guilhermesp | odyssey4me mnaser so I can branch it the os_mistral role now for stable/rocky. Just wanted to know the best way to do it and then I can change the python_venv in rocky to fit the pattern in rocky | 17:27 |
| *** miloa has quit IRC | 17:27 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Install tempest from git https://review.openstack.org/643404 | 17:31 |
| *** chandankumar is now known as chkumar246 | 17:34 | |
| *** DanyC has quit IRC | 17:39 | |
| *** DanyC has joined #openstack-ansible | 17:40 | |
| *** DanyC has quit IRC | 17:41 | |
| *** priteau has joined #openstack-ansible | 17:45 | |
| *** goldenfri has quit IRC | 17:48 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Install user supplied CA certificates into system trust store https://review.openstack.org/641445 | 18:04 |
| *** cshen has joined #openstack-ansible | 18:05 | |
| *** simon-AS559 has quit IRC | 18:07 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Fix tempest_image_dir default var to user's home tempest-image dir https://review.openstack.org/640742 | 18:07 |
| *** DanyC has joined #openstack-ansible | 18:15 | |
| *** hamzaachi_ has quit IRC | 18:16 | |
| *** ivve has joined #openstack-ansible | 18:17 | |
| *** DanyC has quit IRC | 18:19 | |
| *** cshen has quit IRC | 18:21 | |
| *** cmart has quit IRC | 18:33 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Set tempest concurrency to int only https://review.openstack.org/643423 | 18:35 |
| chkumar246 | guilhermesp: ^^ with above review that warning/error will go away if we backport above patch for rocky also! | 18:36 |
| * chkumar246 will do some more tests tomorrow, see ya! | 18:42 | |
| *** cmart has joined #openstack-ansible | 18:43 | |
| guilhermesp | thank you for that chkumar246 !!! | 18:46 |
| *** priteau has quit IRC | 18:56 | |
| *** aedc has joined #openstack-ansible | 18:58 | |
| openstackgerrit | OpenStack Release Bot proposed openstack/openstack-ansible-os_mistral stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/643432 | 19:06 |
| openstackgerrit | OpenStack Release Bot proposed openstack/openstack-ansible-os_mistral stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/643433 | 19:07 |
| openstackgerrit | OpenStack Release Bot proposed openstack/openstack-ansible-os_mistral master: Update master for stable/rocky https://review.openstack.org/643434 | 19:07 |
| *** priteau has joined #openstack-ansible | 19:16 | |
| *** priteau has quit IRC | 19:21 | |
| *** priteau has joined #openstack-ansible | 19:23 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_mistral master: Port venv build of os_mistral to stable/rocky https://review.openstack.org/643442 | 19:28 |
| *** priteau has quit IRC | 19:28 | |
| guilhermesp | mnaser odyssey4me and any other core, reviews would be really appreciated ^ | 19:28 |
| *** luksky has joined #openstack-ansible | 19:43 | |
| jrosser | guilhermesp: thats a patch to master - i'm not sure i understand | 19:44 |
| guilhermesp | yeah | 19:44 |
| guilhermesp | just noticed that | 19:44 |
| guilhermesp | funny coz I checked out stable/rocky locally | 19:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/643432 | 19:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral master: Update master for stable/rocky https://review.openstack.org/643434 | 19:46 |
| *** cshen has joined #openstack-ansible | 19:46 | |
| *** yolanda has quit IRC | 19:46 | |
| *** cshen has quit IRC | 19:50 | |
| guilhermesp | is there a way to edit the PR to stable/rocky or would I need to abandon and create a new one? | 20:03 |
| admin0 | is this normal: message": "Exceeded maximum number of retries. Exceeded max scheduling attempts 5 for instance xxx-xxx- . Last exception: internal error: cannot load AppArmor profile | 20:04 |
| admin0 | i didn't missed any playbook runs | 20:04 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_mistral master: Port venv build of os_mistral to stable/rocky https://review.openstack.org/643442 | 20:07 |
| *** DanyC has joined #openstack-ansible | 20:08 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_mistral stable/rocky: Port venv build of os_mistral to stable/rocky https://review.openstack.org/643449 | 20:14 |
| guilhermesp | that's the correct one jrosser mnaser cloudnull odyssey4me ^ | 20:15 |
| guilhermesp | I will abandon the wrong one | 20:15 |
| *** priteau has joined #openstack-ansible | 20:24 | |
| *** marst has quit IRC | 20:34 | |
| *** ansmith_ has quit IRC | 20:35 | |
| *** priteau has quit IRC | 20:43 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_mistral stable/rocky: Port venv build of os_mistral to stable/rocky https://review.openstack.org/643449 | 20:44 |
| *** hamzy has quit IRC | 20:45 | |
| *** marst has joined #openstack-ansible | 21:01 | |
| *** BjoernT has quit IRC | 21:34 | |
| admin0 | if all rabbitmq containers were nuked/recreated, does the cluster fix itself or is there a need to run the setup-openstack playbook again ? | 21:41 |
| *** cshen has joined #openstack-ansible | 21:46 | |
| admin0 | i guess i will find out and will update here :D | 21:47 |
| *** cshen has quit IRC | 21:50 | |
| admin0 | have to run setup openstack :) | 21:54 |
| cloudnull | hey admin0 | 22:04 |
| cloudnull | yes you have to rerun the setup-openstakc | 22:04 |
| cloudnull | that will recreate all the messaging users and vhosts | 22:04 |
| admin0 | \o | 22:04 |
| admin0 | yep .. running it now | 22:05 |
| admin0 | btw, now i am a proud (runner) of 11 different openstack platforms .. all based on OSA :) | 22:05 |
| admin0 | 5 are in 5 different countries - internal cloud to the company .. 4 = specific use case based to test things like NUMA etc | 22:06 |
| *** DanyC has quit IRC | 22:08 | |
| cloudnull | very nice! | 22:10 |
| cloudnull | all running the same release? | 22:10 |
| admin0 | all 5 are on 18.1.2 | 22:10 |
| cloudnull | how have your NUMA testing been going? | 22:10 |
| admin0 | the 4 are on diff versions | 22:10 |
| admin0 | 2 is used to test with sr-iov .. | 22:10 |
| cloudnull | sweet! | 22:10 |
| admin0 | 1 i am trying to get jamesdenton patch and see if i can get dpdk running/working | 22:11 |
| admin0 | 1 is kind of used for DTAP | 22:11 |
| admin0 | before i affect 1000+ users, i test update in one before i schedule in the rest of the 5 | 22:11 |
| admin0 | 2 are in ovs, 3 are in lb , 1 has ceph, rest 4 are local storage | 22:11 |
| admin0 | all kinds of | 22:11 |
| admin0 | 1 has linux for cinder, 1 has ceph, one has qnap, one has equalogic :D | 22:12 |
| admin0 | what would help me now is a global monitoring dashboard, and a global ELK dashboard | 22:12 |
| admin0 | i want to provide ELK to the company, so that people can grep their UUID/requestID themselves and figure out what went wrong before opening ticket | 22:13 |
| admin0 | 4 have AD backend .. | 22:13 |
| cloudnull | thats how we use elk | 22:14 |
| *** simon-AS559 has joined #openstack-ansible | 22:14 | |
| cloudnull | you can also now rollup multiple elk clusters into a crentralized thing | 22:14 |
| admin0 | us, canada, uk, sweden, india .. 5 different latency zones -- so i keep it separate | 22:15 |
| admin0 | i also found servers geting their / full because somehow /var/log/libvirt/libvirt.log is not properly rotated | 22:15 |
| admin0 | so it reached 40-50+ gb in size | 22:16 |
| cloudnull | uh, thats massive | 22:16 |
| * cloudnull have not had that issue before | 22:16 | |
| admin0 | maybe because the cloud is used a lot | 22:17 |
| admin0 | some day it crosses 1000+ instance launches .. | 22:18 |
| admin0 | but its also deleted as the products is being tested, demoed , all versions etc | 22:18 |
| cloudnull | could be the high vm churn rate | 22:21 |
| *** pcaruana has quit IRC | 22:21 | |
| admin0 | since the number of serves are growing, i am thinking maybe a centralized saltstack to manage all | 22:21 |
| cloudnull | ive not given salt a try in years. | 22:22 |
| cloudnull | id be curious what you think of it | 22:22 |
| admin0 | i used it in my last job which had mirantis openstack | 22:22 |
| admin0 | i found it better than ceph/puppet | 22:23 |
| admin0 | chef/puppet | 22:23 |
| admin0 | the job before that was openstack via puppet and the job before that was via ceph | 22:23 |
| admin0 | chef :D | 22:23 |
| admin0 | chef => puppet => saltstack => OSA | 22:23 |
| admin0 | that is my journey.. | 22:23 |
| admin0 | i gave osa-ops a try once .. but elasticsearch gave me 100% cpu usage on all 56 cores .. very afraid to try now | 22:24 |
| admin0 | if it can have something like nagios/icinga/check_mk as well,it would be awesome :) | 22:25 |
| *** marst has quit IRC | 22:25 | |
| cloudnull | prometheanfire might have some icinga2 recommendations | 22:30 |
| cloudnull | the osp elk tools have evolved quite a bit recently, specifically the collectors have been tuned quite a bit | 22:31 |
| cloudnull | I run elk on dedicated logging nodes within a cluster. | 22:31 |
| admin0 | how to do that ? | 22:31 |
| cloudnull | I'd probably not recommend putting it on the infra hosts | 22:31 |
| admin0 | can't forget this: https://ibb.co/sFhjzp6 | 22:32 |
| admin0 | elk on infra node :) | 22:33 |
| cloudnull | This is our deployment tool chain, https://github.com/rcbops/magnanimous-turbo-chainsaw | 22:33 |
| cloudnull | we piggyback on the log_hosts defined within the openstack_user_config.yml | 22:33 |
| cloudnull | we also use an overlay inventory file which allows is to deploy our ops tools without having to extend the osa inventory | 22:34 |
| cloudnull | https://github.com/openstack/openstack-ansible-ops/tree/master/overlay-inventories | 22:34 |
| cloudnull | in that way we can deploy elk, skydive, grafana, and osquery without having to mess with files in env.d or regenerating osa inventory | 22:36 |
| cloudnull | which is important to us, maybe not everyone, mainly because we're supporting clouds as old as kilo | 22:36 |
| *** ivve has quit IRC | 22:36 | |
| * cloudnull hides his face in shame | 22:36 | |
| *** aedc has quit IRC | 22:36 | |
| admin0 | no .. should be proud that you are still supporting it :) | 22:37 |
| admin0 | others would run away | 22:37 |
| admin0 | i know of an icehouse cluster still running | 22:37 |
| * cloudnull knows of a grizzly cluster still running | 22:38 | |
| admin0 | :) | 22:38 |
| cloudnull | but yes, thats a good way to look at it | 22:38 |
| cloudnull | so our tools have all been made to support legacy things but also to let us do what we need without having to modify the running config within an osa deployment | 22:39 |
| admin0 | i will try magnanimous | 22:39 |
| cloudnull | maybe not super important for a lot of folks m but it was for us | 22:39 |
| cloudnull | yea , give it a spin. works pretty well | 22:40 |
| admin0 | it has to be run from the deploy node ? | 22:40 |
| cloudnull | its just a couple helper things surounding the osa-ops repo | 22:40 |
| cloudnull | yes | 22:40 |
| cloudnull | from the deployment host | 22:40 |
| cloudnull | it needs to be able to read the osa inventory json file | 22:40 |
| prometheanfire | ohai? | 22:42 |
| prometheanfire | I like the stuff in https://github.com/rcbops/magnanimous-turbo-chainsaw / osa-ops :P | 22:42 |
| admin0 | the elk/ server is not a part of openstack inventroy .. in which file do we specify it | 22:43 |
| admin0 | and what kind of connectivity does it need .. br-mgmt range ? | 22:43 |
| cloudnull | br-mgmt is all it needs | 22:45 |
| cloudnull | and to deploy it with the magnanimous-turbo-chainsaw all you need to have is "log_hosts" defined in your openstack_user_config.yml | 22:46 |
| admin0 | so log_hosts is where it will do all of this stuff ? | 22:46 |
| cloudnull | yes. | 22:46 |
| cloudnull | itll install the collectors within the hosts in your cluster, but elasticsearch, logstash, and kibana are all deployed on the log hosts. | 22:47 |
| admin0 | steps #1. prepare a new server for elk/<other cool stuff> #2 give it a br-mgmt range #3. git checkout mtc on deploy #4. add this server as log_host in the inventory #5. follow the rest of the stuff in the ReadMe | 22:47 |
| cloudnull | prometheanfire do you still icinga2 ? | 22:47 |
| cloudnull | admin0 yes | 22:47 |
| prometheanfire | somewhat, both that and netdata (easy to set up) | 22:48 |
| admin0 | do you have a script or something for the icinga part reading from the inventory ? | 22:49 |
| prometheanfire | I don't use icinga or netdata to monitor openstack, just other services | 22:52 |
| prometheanfire | sorry :| | 22:52 |
| admin0 | ok | 22:54 |
| admin0 | how do you guys monitor openstack ? | 23:03 |
| *** simon-AS559 has quit IRC | 23:04 | |
| guilhermesp | Any core to help me to merge it? :P https://review.openstack.org/643449 | 23:04 |
| *** luksky has quit IRC | 23:11 | |
| prometheanfire | admin0: grafana, elk, prometheus (depending on what you are monitoring) | 23:22 |
| prometheanfire | logs or metrics | 23:22 |
| prometheanfire | check out https://github.com/openstack/openstack-ansible-ops | 23:22 |
| admin0 | yep .. i will be setting that up tomororw | 23:23 |
| prometheanfire | that's more of a gathering place for tools, but take a look around :D | 23:24 |
| *** tosky has quit IRC | 23:26 | |
| *** gyee has quit IRC | 23:34 | |
| *** cshen has joined #openstack-ansible | 23:46 | |
| *** cshen has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!