| *** nurdie has joined #openstack-ansible | 00:02 | |
| *** nurdie has quit IRC | 00:06 | |
| *** rfolco has joined #openstack-ansible | 00:21 | |
| *** klamath_atx has joined #openstack-ansible | 00:26 | |
| *** nurdie has joined #openstack-ansible | 00:35 | |
| *** nurdie has quit IRC | 00:40 | |
| *** d34dh0r53 has quit IRC | 00:42 | |
| *** cshen has quit IRC | 00:42 | |
| *** klamath_atx has quit IRC | 00:46 | |
| *** jhesketh has joined #openstack-ansible | 00:49 | |
| *** nurdie has joined #openstack-ansible | 00:51 | |
| *** nurdie has quit IRC | 00:56 | |
| *** rfolco has quit IRC | 01:03 | |
| *** gyee has quit IRC | 01:12 | |
| *** macz_ has quit IRC | 01:15 | |
| *** dave-mccowan has quit IRC | 01:59 | |
| *** macz_ has joined #openstack-ansible | 02:36 | |
| *** cshen has joined #openstack-ansible | 02:39 | |
| *** macz_ has quit IRC | 02:41 | |
| *** cshen has quit IRC | 02:43 | |
| *** nurdie has joined #openstack-ansible | 02:52 | |
| *** nurdie has quit IRC | 02:57 | |
| *** nurdie has joined #openstack-ansible | 03:08 | |
| *** nurdie has quit IRC | 03:13 | |
| *** klamath_atx has joined #openstack-ansible | 03:34 | |
| *** klamath_atx has quit IRC | 03:45 | |
| *** macz_ has joined #openstack-ansible | 04:25 | |
| *** macz_ has quit IRC | 04:30 | |
| *** cshen has joined #openstack-ansible | 04:39 | |
| *** cshen has quit IRC | 04:44 | |
| *** nurdie has joined #openstack-ansible | 05:09 | |
| *** nurdie has quit IRC | 05:14 | |
| *** nurdie has joined #openstack-ansible | 05:25 | |
| *** nurdie has quit IRC | 05:30 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** cloudnull has quit IRC | 06:01 | |
| *** cloudnull has joined #openstack-ansible | 06:02 | |
| *** miloa has joined #openstack-ansible | 06:19 | |
| *** bverschueren has quit IRC | 06:30 | |
| *** bverschueren has joined #openstack-ansible | 06:31 | |
| *** cshen has joined #openstack-ansible | 06:39 | |
| *** macz_ has joined #openstack-ansible | 06:40 | |
| *** cshen has quit IRC | 06:44 | |
| *** macz_ has quit IRC | 06:44 | |
| *** cshen has joined #openstack-ansible | 07:00 | |
| noonedeadpunk | ThiagoCMC: yeah, you need to use ceph_client_ceph_conf_overrides or ceph_conf_overrides | 07:22 |
|---|---|---|
| *** nurdie has joined #openstack-ansible | 07:23 | |
| *** nurdie has quit IRC | 07:25 | |
| *** pcaruana has quit IRC | 07:59 | |
| *** andrewbonney has joined #openstack-ansible | 08:11 | |
| *** pcaruana has joined #openstack-ansible | 08:12 | |
| *** prometheanfire has quit IRC | 08:16 | |
| *** rpittau|afk is now known as rpittau | 08:19 | |
| *** prometheanfire has joined #openstack-ansible | 08:20 | |
| *** jbadiapa has joined #openstack-ansible | 08:37 | |
| jrosser | morning | 09:02 |
| noonedeadpunk | morning) | 09:05 |
| CeeMac | morning | 09:05 |
| noonedeadpunk | jrosser: also it feels that ELK needs dropping this? https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/elk_metrics_7x/roles/elastic_dependencies/defaults/main.yml#L32 | 09:16 |
| jrosser | noonedeadpunk: doh! well i guess we don't see that because ours is a mixture of metal and LXD, so the lxc conditional will skip all that | 09:18 |
| noonedeadpunk | I see:) will adjust it then) | 09:18 |
| jrosser | we've got a set of elasticsearch hosts on metal then logstash is in LXD, and the beats deploy on the OSA containers and hosts | 09:19 |
| jrosser | managing the inventory is challenging | 09:19 |
| noonedeadpunk | tbh I feel that graylog may be the way simpler way of doing log collection, but all depends on amount of hosts here... But checking options now:) | 09:20 |
| noonedeadpunk | tbh I really don't like the way how triggers are done in ELK... So you need to select already written logs to check if they're matching criteria... Comparing to graylog that puts incoming logs into buffer and checks for trigger criteria while writing buffer to elastic.... | 09:23 |
| noonedeadpunk | another problem is that buffer can be overflowed.... | 09:23 |
| *** yolanda has quit IRC | 09:29 | |
| *** yolanda has joined #openstack-ansible | 09:30 | |
| jrosser | noonedeadpunk: did you see on the ML about nova.conf changes so that the compute service will fail if the db credentials are present? | 09:30 |
| jrosser | oh yes - i see you reply | 09:31 |
| noonedeadpunk | yeah, was also going to ask the same question | 09:34 |
| noonedeadpunk | I think it really doesn't matter for us what approach will be chosen | 09:35 |
| jrosser | it will not be a big change to template out a much more minimal config file for compute | 09:35 |
| noonedeadpunk | yeah | 09:36 |
| noonedeadpunk | well, or another template in case they will do another include for nova-db.conf or smth like that | 09:36 |
| pto | I am trying to debug the Keystone federated identity problem. I have found out why the keystone_federation_sp_idp_setup.yml fail. Its because the keystone_db_sync.yml skips the migration tasks for some reason: http://paste.openstack.org/show/799945/ | 09:41 |
| pto | Could anyone point me in a direction where ansible_local['openstack_ansible']['keystone']['need_db_expand'] | bool set? | 09:42 |
| noonedeadpunk | I think jrosser did a patch for that? do you have it applied? | 09:42 |
| *** klamath_atx has joined #openstack-ansible | 09:43 | |
| jrosser | pto: those facts are only set to true when the keytstone venv install is 'changed' https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/tasks/keystone_install.yml#L113-L139 | 09:44 |
| noonedeadpunk | btw regarding these patches... I think we've find out some weird things being done there? | 09:44 |
| pto | jrosser: Is this the patch which shuld fix the problem? https://review.opendev.org/#/c/760431/ | 09:45 |
| jrosser | well i think we should be specific about exactly what we mean by "the problem" | 09:46 |
| pto | jrosser: sorry. The one where the external IDP check fails because the SQL table is locked in a migration, until the keystone-manage db_sync --contact is run | 09:47 |
| jrosser | do you mean that the db expand / contract is *never* run even on the first installation of keystone | 09:47 |
| jrosser | the failure to setup the IDP is really a symptom | 09:48 |
| jrosser | we should identify and fix the root cause | 09:48 |
| jrosser | as far as we could see before the issue was with the sequencing of the tasks | 09:49 |
| pto | jrosser: The IDP setup fails, because of this contact need to be applied before its configured: https://docs.openstack.org/keystone/ussuri/_modules/keystone/common/sql/expand_repo/versions/012_expand_add_domain_id_to_idp.html | 09:49 |
| jrosser | the db expand/contract was done in the playbook rather than the role, according to the commit history this is to ensure that things are done in the right order at upgrade time | 09:50 |
| jrosser | so my patch rather crudely moved the IDP setup to the last step in the playbook by running the os_keystone role again but only for the IDP setup tasks | 09:50 |
| jrosser | i broke the IDP tasks out in the role here https://review.opendev.org/#/c/760429/ | 09:51 |
| pto | jrosser: I think that will solve the problem. Alternative, the ansible_local['openstack_ansible']['keystone']['need_db_expand'] could be set to true, and the same jobs will be run in keystone_db_sync.yml which is executed before the idp setup | 09:51 |
| jrosser | and then called them late in the keystone install playbook here https://review.opendev.org/#/c/760431/ | 09:51 |
| pto | jrosser: Do you know why its skipped? | 09:51 |
| jrosser | well like i was asking before, is this the first deployment (i.e the keystone venv is being created), or is is a re-run of the playbooks when the keystone venv already exists? | 09:52 |
| jrosser | becasue those flags should only be set to be true the one time that the venv is created (or upgraded) | 09:53 |
| pto | jrosser: I just did a fresh install, and the db sync flag is not run initially when the venv is created. The task is skipped | 09:55 |
| noonedeadpunk | I think we stopped there that we have db_sync --contract both in the integrated repo and in https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/tasks/keystone_db_sync.yml#L82 where it's never launched | 09:57 |
| jrosser | looking in CI job results i see the flags set https://zuul.opendev.org/t/openstack/build/b4e6d72de54941dc96e51f40b9ed500a/log/job-output.txt#11452-11461 | 09:58 |
| jrosser | and then the contract is done here and the flag cleared https://zuul.opendev.org/t/openstack/build/b4e6d72de54941dc96e51f40b9ed500a/log/job-output.txt#12055-12063 | 10:01 |
| pto | jrosser: hmmm... So the flags is only set when the venv is build. If I undeploy all containers (lxc-containers-destroy.yml) and wipe /openstack - would that recreate the venvs? | 10:01 |
| jrosser | you can force the venv to be rebuilt | 10:01 |
| jrosser | openstack-ansible playbooks/os-keystone-install.yml -e venv_rebuild=yes | 10:02 |
| jrosser | that variable comes from here https://github.com/openstack/ansible-role-python_venv_build/blob/master/defaults/main.yml#L107 | 10:03 |
| jrosser | the flag is only set when the state of the task which builds the venv is 'changed' | 10:03 |
| pto | jrosser: I had the same problem when the server was fresh installed. I guess the venv should be build if it did not exists, and trigger the changed? | 10:03 |
| jrosser | that means that the db expand/contract should only happen on first deployment of keystone, or at each point that a new venv is built. That is normally when you do an upgrade as the venv is versioned | 10:04 |
| jrosser | pto: yes it should be | 10:04 |
| jrosser | comparing what is happening in your deployment to the CI logs that I gave above may be helpful | 10:04 |
| jrosser | you should have a similar log left in /openstack | 10:05 |
| pto | I copied my stage configs to the production servers (cleaned the inventories and artifacts) , and it failed the same place when i ran it first time | 10:05 |
| pto | jrosser: Just to be clear, you want the output from openstack-ansible os-keystone-install.yml - where setup-hosts.yml and setup-infrastructure.yml has been completed? | 10:07 |
| jrosser | i guess what i'm saying is that the tasks appear to do the right thing in the CI job | 10:08 |
| jrosser | so we should try to understand why the same is not happening for your real deployment | 10:08 |
| pto | jrosser: Agree with that. the CI jobs looks okay. What debug level do you want in the trace? | 10:08 |
| jrosser | well for a start just the default log level should show which tasks are run/skipped | 10:10 |
| pto | jrosser: play running, i will post output later | 10:16 |
| *** gokhani has joined #openstack-ansible | 10:31 | |
| pto | jrosser: Here is the output from a setup-everything.yml http://paste.openstack.org/show/799946/ - The machines has not beed reinstalled, but i have deleted all /etc/ansible/facts.d/*, /openstack and /root | 10:45 |
| pto | and ofc removed all containers | 10:46 |
| *** gshippey has joined #openstack-ansible | 10:49 | |
| *** kukacz has quit IRC | 10:51 | |
| *** gokhani has quit IRC | 10:53 | |
| *** kukacz has joined #openstack-ansible | 10:56 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Remove `elastic_lxc_template_config` mapping https://review.opendev.org/762501 | 11:01 |
| *** rh-jelabarre has quit IRC | 11:02 | |
| *** rh-jelabarre has joined #openstack-ansible | 11:02 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Fix elk 7 env.d sample https://review.opendev.org/762331 | 11:05 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Change ansible tests to prefer Python3 over Python2 in vitualenv https://review.opendev.org/751773 | 11:10 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Install virtualenv https://review.opendev.org/741997 | 11:17 |
| *** gokhani has joined #openstack-ansible | 11:31 | |
| pto | jrosser: Anything interesting in the logs? | 11:37 |
| *** rfolco has joined #openstack-ansible | 11:39 | |
| jrosser | pto: sorry i have some meetings | 11:51 |
| pto | jrosser: No worries :-) | 11:51 |
| openstackgerrit | Merged openstack/openstack-ansible master: Use parallel git clone https://review.opendev.org/588372 | 11:55 |
| *** yann-kaelig has joined #openstack-ansible | 12:01 | |
| noonedeadpunk | jrosser: mind looking on https://review.opendev.org/#/c/761937/ ? | 12:12 |
| *** tosky has joined #openstack-ansible | 12:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh master: Updated from OpenStack Ansible Tests https://review.opendev.org/758739 | 12:51 |
| jrosser | pto: i think your log is truncated in the paste and only the keystone playbook really should be needed | 12:55 |
| noonedeadpunk | jrosser: smth is wrong with https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/elk_metrics_7x/roles/elastic_data_hosts/vars/data-node-variables.yml#L138-L143 but really can't figure out what exactly | 13:18 |
| noonedeadpunk | I mean that `data_nodes | map('extract', hostvars, 'ansible_host') | map('regex_replace', '(.*)', '\\1:' ~ elastic_port) | list` results in `172.29.236.126:9200:9200` | 13:18 |
| noonedeadpunk | which feels like second map is applied twice | 13:18 |
| jrosser | hmm | 13:20 |
| noonedeadpunk | so `"{{ ['172.29.236.126'] | map('regex_replace', '(.*)', '\\1:9200') | list }}"` is actually the same result | 13:21 |
| noonedeadpunk | ok, needs ^ and $ | 13:23 |
| pto | jrosser: Here is it with the full paste: https://pastebin.com/tXn2bad8 | 13:24 |
| mgariepy | jrosser, any luck with netplan ? | 13:24 |
| mgariepy | i start to beleive that they do code it just to mess with me. | 13:24 |
| jrosser | mgariepy: no, trying really not to touch it | 13:24 |
| noonedeadpunk | haha | 13:25 |
| mgariepy | i do like to rename my network interface like: 25G-1 25G-2 etc.. | 13:25 |
| pto | jrosser: The flag is set correctly, 'need_db_contract', 'value': 'True' - but the task db_sync --contact has not been run. Triggers are still there | 13:25 |
| mgariepy | but, when you do add the match on mac addr, if you don't have a bond or a bridge. | 13:25 |
| mgariepy | the vlan interface will have the same mac and the generated interface link file will try to rename the vlan interface to 25G-1 or whatever. | 13:26 |
| mgariepy | but there is a fix in the code now.. https://github.com/CanonicalLtd/netplan/commit/83c1c1c08426beb8c395a809dd4c1aa788223e9b#diff-d9e1c24eef897e54a05d1eb355c0cfad12a9a078c741326a53161c081f58308f | 13:27 |
| mgariepy | tl;dr; systemd-networkd next time.. | 13:28 |
| *** macz_ has joined #openstack-ansible | 13:30 | |
| kleini | systemd-networkd is really very good, once you understand the files and their power | 13:32 |
| noonedeadpunk | maybe one day it will become usable as netplan format is neat tbh... but yeah... | 13:33 |
| mgariepy | yep netplan format integrated with the new installer is nice. | 13:33 |
| jrosser | pto: do you have my patches applied? | 13:33 |
| kleini | YAML is neat, sorry, I don't agree at all | 13:33 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Fix regex_replace mapping https://review.opendev.org/762519 | 13:33 |
| pto | jrosser: Nope. Its fresh from upstream ussuri/stable. | 13:33 |
| jrosser | to both the os_keystone role and to openstack-ansible | 13:34 |
| mgariepy | but it would needs a way to set arbitrary key in the generated file. | 13:34 |
| noonedeadpunk | we need to fix ops ci.... | 13:34 |
| jrosser | pto: well i think that we already know it is not working in the current code | 13:34 |
| mgariepy | instead or C code to add a key in the config file. | 13:34 |
| *** macz_ has quit IRC | 13:35 | |
| jrosser | mgariepy: i wonder really if the systemd_networkd role here is a good choice | 13:35 |
| pto | jrosser: I tried to add an extra dbsync in the top of the keystone_federation_sp_idp_setup.yml (which only runs when idp is active). This solves the problem | 13:35 |
| pto | jrosser: Should I submit a patch for this or does it need a more refactoring? | 13:36 |
| mgariepy | jrosser, the advantage of it is that it works no matther the OS. | 13:36 |
| jrosser | pto: i have submitted two patches here already which should fix this | 13:36 |
| mgariepy | which is not quite true for netplan. | 13:36 |
| jrosser | mgariepy: yeah, i think if i didnt have already reliable ifupdown probably the OSA ansible role would be the place i'd start | 13:36 |
| mgariepy | well my install was reliable until i tought of removing all the bridges and bonds from my computes..lol | 13:38 |
| jrosser | noonedeadpunk: i just checked our variables and i have this override elasticsearch_data_node_socket_addresses: '{{ elastic_endpoints | map("regex_replace", "(.*)" ,"\1:" ~ elastic_port) | list }}' | 13:39 |
| noonedeadpunk | hm..... | 13:39 |
| noonedeadpunk | does just debug: msg: gives predictable result for "{{ ['172.29.236.126'] | map('regex_replace', '(.*)', '\\1:9200') | list }}"? | 13:41 |
| jrosser | and elastic_endpoints: [ ip-address, another-ip-address] | 13:41 |
| noonedeadpunk | but I think it will since this override works... | 13:42 |
| jrosser | i get 172.29.236.126:9200 | 13:43 |
| noonedeadpunk | and I'm not...... | 13:43 |
| noonedeadpunk | I have ansible 2.9.9 | 13:43 |
| jrosser | http://paste.openstack.org/show/799955/ | 13:44 |
| jrosser | same 2.9.9 here | 13:44 |
| noonedeadpunk | http://paste.openstack.org/show/799956/ | 13:45 |
| noonedeadpunk | o_O | 13:45 |
| noonedeadpunk | wtf | 13:45 |
| jrosser | yes, that! | 13:45 |
| jrosser | i knew this was something we override so could be a bug we never excersie that codepath | 13:46 |
| pto | jrosser: Is this the patch? https://review.opendev.org/#/c/760429/? | 13:46 |
| jrosser | but turns out the override is exactly the same expression | 13:46 |
| jamesdenton | mornin | 13:47 |
| jrosser | o/ hello | 13:48 |
| jrosser | noonedeadpunk: what OS is yours? | 13:48 |
| noonedeadpunk | focal | 13:49 |
| noonedeadpunk | maybe jinja version.... | 13:49 |
| jrosser | mine is bionic | 13:49 |
| noonedeadpunk | Jinja2 2.11.2? | 13:49 |
| jrosser | is this distro jinja or the ansible venv? | 13:49 |
| noonedeadpunk | ansible venv | 13:49 |
| jrosser | Jinja2==2.11.1 | 13:50 |
| noonedeadpunk | downgrading it changed nothing for me.... | 13:51 |
| noonedeadpunk | uh | 13:51 |
| noonedeadpunk | lets put it another way... does ansible localhost -m debug -a msg="{{ ['172.29.236.126'] | map('regex_replace', '^(.*)$', '\\1:9200') | list }}" work for you as well? | 13:52 |
| jrosser | yes they both give the port just once | 13:53 |
| noonedeadpunk | py3,6 vs py38? | 13:53 |
| jrosser | that will be different yes | 13:53 |
| noonedeadpunk | I'm wondering if we have the same thing somehwere in regular roles... | 13:54 |
| jrosser | i have no idea what is happening, i just found an old 20.04 AIO and it gives just the one :9200 | 13:57 |
| jrosser | is yours a very new focal cloud image? | 13:58 |
| noonedeadpunk | dunno, it's reproducing on my desktop as well which is running focal.... | 13:58 |
| noonedeadpunk | as of image I think it's pretty old... | 13:59 |
| * jrosser upgrades packages on AIO | 13:59 | |
| *** gokhani has quit IRC | 14:03 | |
| jrosser | noonedeadpunk: just so that it's clear - your second regex with the ^ and $ is working on focal | 14:06 |
| *** macz_ has joined #openstack-ansible | 14:06 | |
| noonedeadpunk | yes | 14:07 |
| jrosser | the one which works for me on bionic without those is giving the :9200:9200 on focal just like you see | 14:07 |
| noonedeadpunk | ok, good means I'm not crazy | 14:08 |
| jrosser | i just read back and saw opportinity for confusion :) | 14:08 |
| noonedeadpunk | so I think we should patch things to have ^$ I guess... | 14:09 |
| noonedeadpunk | as it's optios that should work anyway? | 14:09 |
| jrosser | yes i just confirmed those behave properly on focal and bionic | 14:10 |
| jrosser | wierd though? | 14:11 |
| *** macz_ has quit IRC | 14:11 | |
| noonedeadpunk | I claim py version, as things are changed from 3.6... Like that you can't now reference dict as a method so in case tst = {'a': 'b'} you can't in 3.8 do tst.a - only tst['a'] | 14:12 |
| noonedeadpunk | not sure it's reflected in ansible as well.... | 14:12 |
| noonedeadpunk | it's not... probably folks did workaround for this... | 14:14 |
| *** d34dh0r53 has joined #openstack-ansible | 14:23 | |
| mgariepy | anyone had isseu with live migration on servers with the exact smae hw ? | 14:29 |
| mgariepy | libvirt complaining about cpu feature not compatible. | 14:29 |
| andrewbonney | Yeah, I've patched something in Nova for that. I'll try and find a link | 14:31 |
| andrewbonney | Assuming it's the same thing: https://bugs.launchpad.net/nova/+bug/1898715 and https://review.opendev.org/#/c/757577/ | 14:32 |
| openstack | Launchpad bug 1898715 in OpenStack Compute (nova) "Live migration fails despite matching CPUs" [Undecided,Fix released] - Assigned to Andrew Bonney (andrewbonney) | 14:32 |
| *** dave-mccowan has joined #openstack-ansible | 14:33 | |
| mgariepy | yep sounds like it | 14:37 |
| mgariepy | thanks\ | 14:37 |
| *** dave-mccowan has quit IRC | 14:37 | |
| *** spatel has joined #openstack-ansible | 14:38 | |
| mgariepy | not yet merged in U :( | 14:40 |
| andrewbonney | Yeah, needs to go into V first as well given how their CI works | 14:41 |
| *** nurdie has joined #openstack-ansible | 14:47 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Install virtualenv https://review.opendev.org/741997 | 14:49 |
| jrosser | pto: there are two patches, this https://review.opendev.org/#/c/760431/ and https://review.opendev.org/#/c/760429/ | 14:51 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Install virtualenv https://review.opendev.org/741997 | 14:51 |
| jrosser | pto: there may be some further refactoring of those as it was a pretty quick set of changes i did | 14:51 |
| mgariepy | andrewbonney, poking them to have some review.. | 15:02 |
| andrewbonney | Thanks! | 15:02 |
| mgariepy | not sure it will get it merge faster but well.. we shall see. | 15:03 |
| jrosser | mgariepy: do you see this also with cascade lake like we did? | 15:03 |
| mgariepy | yep | 15:04 |
| jrosser | actually it was cascade lake refresh x2xx cpu model | 15:04 |
| jrosser | actually it was cascade lake refresh x2xxR cpu model | 15:04 |
| mgariepy | i have gold 5218 cpus. | 15:08 |
| *** yolanda has quit IRC | 15:11 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Install virtualenv https://review.opendev.org/741997 | 15:35 |
| *** macz_ has joined #openstack-ansible | 16:03 | |
| *** macz_ has quit IRC | 16:08 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Install virtualenv https://review.opendev.org/741997 | 16:11 |
| noonedeadpunk | jrosser: any dea what could possible go wrong here? as see no issues in kibana log https://zuul.opendev.org/t/openstack/build/d84fa66d4b5d478fa66fef1744855b3c/log/job-output.txt#2134 | 16:12 |
| *** macz_ has joined #openstack-ansible | 16:12 | |
| *** macz_ has joined #openstack-ansible | 16:13 | |
| andrewbonney | noonedeadpunk: our colleague James had been looking at the previous error in https://zuul.opendev.org/t/openstack/build/d84fa66d4b5d478fa66fef1744855b3c/log/job-output.txt#2108 | 16:17 |
| andrewbonney | We found that restarting elasticsearch was necessary to fix a 502 gateway timeout | 16:17 |
| noonedeadpunk | ha, yes, looks the same | 16:18 |
| noonedeadpunk | I think his patch is https://review.opendev.org/#/c/751773/13 | 16:19 |
| noonedeadpunk | and yeah, it's passing.... | 16:19 |
| andrewbonney | Yeah, he's just joining IRC :) | 16:19 |
| * noonedeadpunk needs to look closer the diff | 16:19 | |
| *** JamesGibo has joined #openstack-ansible | 16:19 | |
| noonedeadpunk | preoblem there is that this patch was dependant on gshippey ones, which would never merge because of the broken gates | 16:20 |
| noonedeadpunk | so decided to edit first one | 16:20 |
| andrewbonney | noonedeadpunk: looks like he's on now. It's JamesGibo if you want to check on anything | 16:21 |
| noonedeadpunk | https://review.opendev.org/#/c/741997 seems to do exactly same changes that https://review.opendev.org/#/c/751773 but not passing lk7 test... | 16:21 |
| noonedeadpunk | aha, ok, you;ve commented out indexes creation | 16:24 |
| JamesGibo | Hi, Yeah i commented out for testing, since then i have been working on this locally | 16:25 |
| noonedeadpunk | should we probably just flush handlers? | 16:26 |
| JamesGibo | The task "Create kibana index patterns" completes successfully, but the indexes do not show up in Kibana UI until elasticsearch is restarted | 16:26 |
| noonedeadpunk | all elasticsearch services? As there're plenty of them... | 16:28 |
| JamesGibo | Bu this does not fix the error with "Create basic indexes" task, which sets the default index, this also returns a 502 error in the browser | 16:28 |
| noonedeadpunk | ah, elastic,sorry | 16:28 |
| JamesGibo | I just restarted the elasticsearch service, not all the beats or kibana | 16:29 |
| noonedeadpunk | so restart of elastic needs to be done between `Create kibana index patterns` and `Create basic indexes`? | 16:31 |
| noonedeadpunk | that's pretty strange... | 16:31 |
| jrosser | that really feels wrong to need that somehow | 16:32 |
| noonedeadpunk | as well as actually failing `Create kibana index patterns` for the first try... | 16:33 |
| JamesGibo | No, restarting the elaasticsearch service just makes the index's appear in the kibana ui, it does not fix `Create basic indexes` task. My gut feeling is the error is somewhere else in the configuration which is making it misbehave at the end of the playbook, but don't know where | 16:35 |
| noonedeadpunk | so even if we restart elasticsearch it won't fix role, right? | 16:36 |
| jrosser | the templates for the config files may have drifted out of date with the out-of-the-box ones | 16:36 |
| JamesGibo | yeah, there is still an issue | 16:36 |
| jrosser | it was always a bit of a fight keeping track of changes in the upstream config | 16:37 |
| JamesGibo | I thought that and rolled back to that last version that passed CI, put that didn't seem to help. Can't remeber which version of kibana, elasticserach and the beats i tried | 16:37 |
| *** klamath_atx has quit IRC | 16:54 | |
| *** JamesGibo has quit IRC | 16:58 | |
| *** jamesgibo has joined #openstack-ansible | 17:02 | |
| *** klamath_atx has joined #openstack-ansible | 17:07 | |
| *** jamesgibo has quit IRC | 17:08 | |
| *** klamath_atx has quit IRC | 17:12 | |
| *** klamath_atx has joined #openstack-ansible | 17:13 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ops master: Install virtualenv https://review.opendev.org/741997 | 17:18 |
| *** klamath_atx has quit IRC | 17:18 | |
| *** djhankb has quit IRC | 17:24 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_barbican master: Clean up barbican.conf https://review.opendev.org/759084 | 17:34 |
| *** klamath_atx has joined #openstack-ansible | 17:43 | |
| *** klamath_atx has quit IRC | 17:48 | |
| *** klamath_atx has joined #openstack-ansible | 17:49 | |
| *** rpittau is now known as rpittau|afk | 17:51 | |
| *** klamath_atx has quit IRC | 17:53 | |
| nsmeds | I've got another issue to throw into the wind: since updating to Ussuri _sometimes_ a large percent of new instances fail to reach the metadata service (and thus don't run user data). | 18:06 |
| nsmeds | My coworker spun up 100+ instances without issue earlier, and yet I'm getting about 50% failure rate on my recent instances. | 18:07 |
| nsmeds | And unrelated to that: anyone running OSA with memcached, what kind of hit ratios are you getting with memcached? | 18:10 |
| *** spatel has quit IRC | 18:11 | |
| *** pcaruana has quit IRC | 18:14 | |
| jrosser | nsmeds: the active router (if you're using neutron l3) may be on different hosts if you and your co-worker are using different projects | 18:27 |
| *** klamath_atx has joined #openstack-ansible | 18:27 | |
| jrosser | that could mean you are using different infra / network hosts for the metadata so it would be worth seeing if there is a pattern there | 18:28 |
| *** cshen has quit IRC | 18:31 | |
| kleini | openstack-ansible-ops contains deployment of ELK for collecting logs. Are there somewhere instructions, how to deploy elastic, logstash & kibana in lxc containers along with OSA? | 18:37 |
| *** klamath_atx has quit IRC | 18:47 | |
| *** cshen has joined #openstack-ansible | 18:58 | |
| *** miloa has quit IRC | 19:04 | |
| *** cshen has quit IRC | 19:10 | |
| *** gyee has joined #openstack-ansible | 19:19 | |
| nsmeds | okay, thanks @jrosser I'll dig into that. Yes we're using DVR / neutron l3 agents. | 19:34 |
| nsmeds | Tbh, we have a hunch its related to either HAProxy or memcached (we ran into other similar issues post-Ussuri upgrade), but definitely exploring all options | 19:35 |
| nsmeds | The other issue was https://bugs.launchpad.net/openstack-ansible/+bug/1903226 - where Keystone had issues when talking to memcached via HAProxy frontend. We still haven't fully explored what's going on there, since bypassing HAProxy works for now (and other issues to fix). | 19:37 |
| openstack | Launchpad bug 1903226 in openstack-ansible "Conflicting documentation for HA memcached" [Undecided,New] | 19:37 |
| *** cshen has joined #openstack-ansible | 20:00 | |
| *** andrewbonney has quit IRC | 20:00 | |
| *** MickyMan77 has quit IRC | 20:25 | |
| *** d34dh0r53 has quit IRC | 20:29 | |
| *** d34dh0r53 has joined #openstack-ansible | 20:31 | |
| *** mrda has quit IRC | 20:58 | |
| *** mrda has joined #openstack-ansible | 21:01 | |
| *** nurdie has quit IRC | 21:28 | |
| *** jbadiapa has quit IRC | 21:30 | |
| *** klamath_atx has joined #openstack-ansible | 21:52 | |
| *** nurdie has joined #openstack-ansible | 22:00 | |
| *** klamath_atx has quit IRC | 22:04 | |
| *** nurdie has quit IRC | 22:05 | |
| *** klamath_atx has joined #openstack-ansible | 22:12 | |
| *** yann-kaelig has quit IRC | 22:25 | |
| *** persia has quit IRC | 22:31 | |
| *** fyx has quit IRC | 22:31 | |
| *** persia has joined #openstack-ansible | 22:32 | |
| *** fyx has joined #openstack-ansible | 22:32 | |
| *** nurdie has joined #openstack-ansible | 22:35 | |
| *** klamath_atx has quit IRC | 22:36 | |
| *** nurdie has quit IRC | 22:41 | |
| *** tosky has quit IRC | 22:44 | |
| *** gshippey has quit IRC | 22:48 | |
| *** djhankb has joined #openstack-ansible | 23:27 | |
| *** djhankb has quit IRC | 23:32 | |
| *** djhankb has joined #openstack-ansible | 23:33 | |
| *** nurdie has joined #openstack-ansible | 23:39 | |
| *** CeeMac has quit IRC | 23:42 | |
| *** nurdie has quit IRC | 23:42 | |
| *** klamath_atx has joined #openstack-ansible | 23:43 | |
| *** cshen has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!