| opendevreview | Merged openstack/openstack-ansible stable/victoria: Bump SHA for openstack_hosts to pick up centos repos fix https://review.opendev.org/c/openstack/openstack-ansible/+/829169 | 01:02 |
|---|---|---|
| ygk_12345 | i am facing this issue with OSA wallaby | 08:36 |
| ygk_12345 | libvirt.libvirtError: unsupported configuration: interface fa:16:3e:42:20:3e - bandwidth settings are not supported for hostdev interfaces | 08:36 |
| ygk_12345 | for sr-iov vms | 08:37 |
| ygk_12345 | any workaround ? | 08:39 |
| ygk_12345 | libvirt.libvirtError: unsupported configuration: interface fa:16:3e:42:20:3e - bandwidth settings are not supported for hostdev interfaces | 08:49 |
| ygk_12345 | for wallaby OSA | 08:49 |
| ygk_12345 | anyone ? | 08:49 |
| ygk_12345 | is that a bug in wallaby ? | 08:56 |
| andrewbonney | I'm not sure which settings it's referring to, but we ran SRIOV VMs in wallaby and now xena without issue | 08:57 |
| ygk_12345 | andrewbonney: when I spin up a vm on sr-iov hosts, it is reporting the above error | 08:59 |
| andrewbonney | I guess I'd be taking a look in the SRIOV config to see if any variables were perhaps removed in the W release, assuming they worked in V | 09:00 |
| ygk_12345 | andrewbonney: which config file to be exact ? | 09:00 |
| andrewbonney | If your deploy is close to ours then I'd guess /etc/neutron/plugins/ml2/sriov_nic_agent.ini, but as the error is from libvirt it might be in a nova config file | 09:01 |
| ygk_12345 | andrewbonney: i find no setting in the sriov*.ini file | 09:04 |
| ygk_12345 | andrewbonney: i only see this in ova.conf : live_migration_bandwidth | 09:06 |
| ygk_12345 | andrewbonney: nova.conf | 09:06 |
| ygk_12345 | any workaround ? | 09:12 |
| ygk_12345 | andrewbonney: any workaround ? | 09:22 |
| ygk_12345 | libvirt.libvirtError: unsupported configuration: interface fa:16:3e:42:20:3e - bandwidth settings are not supported for hostdev interfaces | 09:22 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_glance master: Add support for TLS to Glance https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/821011 | 09:28 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_glance master: Add support for TLS to Glance https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/821011 | 09:41 |
| andrewbonney | ygk_12345: I haven't seen the error before so I'm not sure. Searching suggests it might be QoS related, but someone more familiar with libvirt might be able to suggest how to debug further | 09:42 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/xena: Use unix socket while granting access for the backup service https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/829259 | 10:10 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Fix behaviour of OSA_CONFIG_DIR https://review.opendev.org/c/openstack/openstack-ansible/+/829256 | 11:20 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Fix behaviour of OSA_CONFIG_DIR https://review.opendev.org/c/openstack/openstack-ansible/+/829257 | 11:20 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| opendevreview | Merged openstack/openstack-ansible-os_horizon stable/xena: Move Listen definition to VHosts https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/829254 | 12:20 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon stable/wallaby: Move Listen definition to VHosts https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/829255 | 12:20 |
| admin1 | hi guys .. so i am using gnocchi and redis incoming driver .. from inside the virtualenv, i did pip install redis and the driver is there .. but gnocchi is stuk on the logs saying there is no redis driver .. ... here is the logs .. | 12:26 |
| admin1 | https://gist.githubusercontent.com/a1git/14a2744c2995ffaf87b8e59cdd0bdcb9/raw/36f74dfda603fbb76cdc6dbb3db8bc102a3cb912/gistfile1.txt | 12:26 |
| admin1 | sorry .. correct one: https://gist.githubusercontent.com/a1git/9e4ea9b08ecc4a24f8cc6e583220a6f5/raw/ca4d370de7057cd4ccc53539515005e3b45a2214/gistfile1.txt | 12:27 |
| noonedeadpunk | and what about `from redis import sentinel`? | 12:36 |
| noonedeadpunk | as I see it also needs `hiredis` package installed | 12:37 |
| noonedeadpunk | https://github.com/gnocchixyz/gnocchi/blob/master/setup.cfg#L72-L73 | 12:38 |
| admin1 | from redis import sentinel works fine .... hiredis is also installed | 13:04 |
| chandankumar | jrosser: hello, please have a look at https://review.opendev.org/c/openstack/ansible-config_template/+/828295 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/828449 and https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/829164 when free, thanks! | 13:22 |
| jrosser | chandankumar: sure - I can look later today | 13:25 |
| lowercase | Hello, a req at work is having us use Fluentd in our openstack environment. After researching a bit I discovered kolla-ansible has fluentd compatibility and openstack-ansible does not. I checked the board and there are no fluentd. Has this ever been brought up before or requested? | 14:00 |
| admin1 | You are not building wheels while running role against multiple hosts. This might result in DOS-ing OpenDev infrustructure servers. In order to proceed, please ensure that you have repo servers for selected OS version and architecture. If you want to avoid building wheel on purpose, ensure that you run playbook in serial manner. In case of causing | 14:10 |
| admin1 | unreasonable load on the opendev.org git servers, your access may be blocked to protect other users and the OpenDev CI infrastructure which are reliant on this service. Found venv_build_targets: {'20.04': {'x86_64': 'r1c1_repo_container-87e33efa'}}" | 14:10 |
| admin1 | is this a remote message or a osa error message ? | 14:10 |
| admin1 | all i am doing is running setup-openstack.yml | 14:10 |
| lowercase | admin1: i think the channel you want to join is #opendev and chat with them | 14:15 |
| damiandabrowski[m] | lowercase: we have ELK stack in our openstack-ansible-ops repo, but it uses logstash instead of fluentd: https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/elk_metrics_7x | 14:19 |
| admin1 | they ask me to go back and ask in this channel | 14:20 |
| damiandabrowski[m] | @admin1 it's osa error message: https://opendev.org/openstack/ansible-role-python_venv_build/commit/de731e7f9e15a99e9587b42de175531bab3f401e | 14:20 |
| lowercase | admin1: sorry | 14:20 |
| lowercase | damiandabrowski[m]: i am using Loki | 14:21 |
| admin1 | so .. i am setting up a new cluster .. it worked till gnocchi .. where i had to do some extra packages for redis, then i # the rest of the setup and continuing only the failed playbooks | 14:21 |
| admin1 | but now this fails with this error message all the time | 14:22 |
| admin1 | i need to run the full setup-openstack starting with keystone to not have this error ? | 14:22 |
| fungi | lowercase: it's an error message added by the openstack-ansible maintainers to help prevent openstack-ansible users from accidentally launching a giant ddos attack against the opendev git server farm | 14:22 |
| lowercase | fungi: i know, i am the one who did the original ddos'ng | 14:23 |
| fungi | jrosser is familiar with the details and can probably explain more from the openstack-ansible side once he's around | 14:23 |
| admin1 | but now its blocking my playbook runs .. so how do i proceed ? | 14:23 |
| lowercase | admin1: one sec | 14:23 |
| fungi | the opendev sysadmins don't know (and shouldn't need to know) how to tell you how to properly configure openstack-ansible, but hopefully the maintainers of openstack-ansible can if their documentation isn't clear enough | 14:24 |
| admin1 | -e venv_wheel_build_skip_check=true | 14:24 |
| jrosser | admin1: only do that if you’re happy to bypass the giant warning | 14:26 |
| fungi | if that ends up causing you to perform a massive number of git clone operations against opendev.org, the server admins for opendev will block your ip addresses and thus your playbooks | 14:26 |
| jrosser | admin1: I think you need to take a moment to understand why you get that message | 14:27 |
| admin1 | jrosser, this is my setup-openstack now .. https://gist.githubusercontent.com/a1git/85fb28075866cedd04bf96c9ecbc97de/raw/1c919131199476c6dfedb0effc25a805573b4452/gistfile1.txt | 14:27 |
| admin1 | it should not clone it | 14:27 |
| jrosser | I’m not following, tbh | 14:28 |
| admin1 | ok .. i am setting up a new cluster .. it went till gnocchi .. and failed due to radius .. i control C .. install the radius stuff and gnocchi is done .. then i # the rest and re-run the setup-openstack playbook | 14:29 |
| jrosser | lowercase: regarding fluentd, we dont directly integrate any particular monitoring stack directly | 14:29 |
| admin1 | and now getting this error | 14:29 |
| lowercase | jrosser: would you be interested in having a monitoring stack in the playbooks? | 14:29 |
| admin1 | i brb in 30 min .. | 14:30 |
| jrosser | lowercase: you should look in the openstack-ansible-ops repo where we have a bunch of “3rd party” stuff like that | 14:30 |
| lowercase | looks like there is just elk and graylog in there. no fluentd sauce | 14:33 |
| lowercase | welp, guess ill continue copying all the code out of kolla-ansible and adapting it for my use case. | 14:34 |
| jrosser | lowercase: the openstack-ansible-ops repo is really a collection of useful stuff that operators have contributed that doesnt want to sit permanently in OSA | 14:38 |
| jrosser | it is almost certainly the case that not everyone wants fluentd, just like not everyone wants ELK so the maintenance overhead of having it all integrated would be too much | 14:38 |
| jrosser | for exmaple, my team has put most of the work into the elk stack in openstack-ansible-ops becasue thats what we use in our deployment | 14:39 |
| noonedeadpunk | lowercase: eventually indeed we do miss fluentd output and support for oslo. I was thinking about making journald usage onfigurable with variable. As it's basically matter of 1 setting, so patch should be trivial I guess? | 14:41 |
| lowercase | that would be helpful, I am going to investigate how to integrate the kolla-ansible curated filters against journal. Afaik its still possible, i just haven't figured it out. | 14:43 |
| lowercase | Cause at the end of the day, its just a source with a tag. Now is there a way i can discriminate which filter gets applied based on a key word element in the log? idk. | 14:45 |
| lowercase | Now truly this would be a copy paste job if the log files were written to a file. | 14:45 |
| spatel | kolla-ansible dump logs in flat file where we can easily tail log for issue etc.. that kind of feature would be good in OSA | 14:49 |
| spatel | currently if i don't have ELK then its little difficult to debig | 14:50 |
| spatel | debug* | 14:50 |
| noonedeadpunk | lowercase: well log files is poor thing imo | 14:50 |
| noonedeadpunk | it don't have metadata avaialble and should be managed hevaily | 14:51 |
| lowercase | absolutely agreed, but if there is centralized logging you're not looking at logs on a server anyway. | 14:51 |
| noonedeadpunk | at the same time if you want fluentd you still don't need files | 14:51 |
| mgariepy | jouirnactl -f -u unit-file.service or slice works ok | 14:51 |
| noonedeadpunk | you can't really parse plain text files. But if you forward or transofrm journald to syslog, you have metadata avialble | 14:52 |
| noonedeadpunk | As example - I used that to forward journal to graylog https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/graylog/graylog-forward-logs.yml | 14:52 |
| spatel | noonedeadpunk i am also doing journalbeat to push logs to graylog. (This is for big deployment) | 14:53 |
| noonedeadpunk | and for fluent you still want to use https://docs.openstack.org/oslo.log/latest/reference/formatters.html#oslo_log.formatters.FluentFormatter I bet | 14:53 |
| lowercase | noonedeadpunk: fluentd has one thing i haven't figured out. Journal has logs for everything, I need fluentd to be able to discriminate which service it is and apply different regex filters on it to get the awesome clean fluentd json format. | 14:53 |
| spatel | We have couple of small deployment like 20 or 30 node stack where we don't want to run ELK | 14:53 |
| noonedeadpunk | so openstack services can be configured to send logs natively to fluent | 14:54 |
| noonedeadpunk | I haven't used that in prod a lot though | 14:54 |
| lowercase | well how the eff do i use that | 14:54 |
| lowercase | thats amazing | 14:55 |
| noonedeadpunk | I _think_ that instead of `use_journal` `use_json` should be used with some other set of options... | 14:58 |
| noonedeadpunk | I can recall somebody using that tbh... | 14:59 |
| noonedeadpunk | oh, here's some config sample for oslo https://docs.openstack.org/oslo.log/latest/admin/example_nova.html | 15:00 |
| noonedeadpunk | lowercase: ^ | 15:01 |
| noonedeadpunk | I believe this should be applicable for any service | 15:01 |
| noonedeadpunk | But you would need to use config_overrides everywhere as of today to make it happen | 15:01 |
| noonedeadpunk | As no neat variable is present | 15:02 |
| jrosser | feels like we should have some global stuff for setting that | 15:02 |
| jrosser | as it looks super useful | 15:02 |
| noonedeadpunk | oh yes | 15:02 |
| noonedeadpunk | I'm thinking about it for a while indeed :) | 15:02 |
| jrosser | nice low-hanging fruit contribution there for someone :) | 15:02 |
| noonedeadpunk | but still it's doable as of today, just with more effort | 15:02 |
| lowercase | certainly, and this a better solution than laying out flat config files for fluentd. So much less transformation that needs to happen | 15:03 |
| jrosser | lowercase: you're familiar with our config overrrides? | 15:03 |
| lowercase | yea | 15:03 |
| jrosser | cool, like noonedeadpunk says you can make a whole bunch of those to add this in | 15:03 |
| jrosser | just beware that you can only define them once per service unless you are really knowing whats going on | 15:04 |
| lowercase | nice. Once i do figure it out. I'm not saying i will make a point to feature request and PR some sauce for you. | 15:04 |
| jrosser | so if you already override something for nova, you need to add to that, rather than make another | 15:05 |
| lowercase | oh yea no worries there | 15:05 |
| noonedeadpunk | also with that I dunno if you might want to set `use_journald = False` | 15:05 |
| lowercase | hmm.. good point | 15:06 |
| noonedeadpunk | still doable with same override | 15:06 |
| noonedeadpunk | doh.. .according to doc each venv would require `fluent-logger` to be installed... | 15:08 |
| noonedeadpunk | and that is more tricky thing to do tbh... | 15:08 |
| jrosser | i am having memories of osprofiler | 15:08 |
| jrosser | that was similar, and i think there are vars for extra pip packages? | 15:09 |
| noonedeadpunk | um, not really? | 15:10 |
| noonedeadpunk | or well, dunno.. | 15:10 |
| lowercase | pip_extra_packages ? | 15:11 |
| jrosser | https://github.com/openstack/openstack-ansible-os_keystone/blob/master/defaults/main.yml#L494 | 15:11 |
| jrosser | that sort of thing | 15:11 |
| noonedeadpunk | I'm not sure it's around for all roles tbh | 15:11 |
| noonedeadpunk | Or well, I;'m pretty sure it's not | 15:12 |
| lowercase | nova_user_pip_packages: [] | 15:12 |
| lowercase | yeah | 15:12 |
| noonedeadpunk | well, core roles at least does... | 15:12 |
| lowercase | i dont think so either because ive ran into it where i think it was cinder needed an additional ceph package and i couldn't add it | 15:12 |
| lowercase | hmm its in cinder | 15:13 |
| noonedeadpunk | cinder has it https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/defaults/main.yml#L311 | 15:13 |
| noonedeadpunk | all core roles does from what I see https://codesearch.opendev.org/?q=_user_pip_packages&i=nope&literal=nope&files=&excludeFiles=&repos= | 15:14 |
| lowercase | that's a handy tool | 15:15 |
| noonedeadpunk | it really is :) | 15:23 |
| noonedeadpunk | the missing thing is search for branches :( | 15:23 |
| noonedeadpunk | as you see only master | 15:23 |
| admin1 | jrosser, noonedeadpunk .. i am back .. ok ..so assuming something is wrong with the playbooks or deployment, how to not load opendev ? | 15:24 |
| admin1 | can we proxy the packages in a local repo ? | 15:24 |
| noonedeadpunk | replace opendev with github for openstack_services.yml ? :D | 15:27 |
| noonedeadpunk | But, well, IMO, it's worth fixing issue why wheels build fails imo | 15:27 |
| noonedeadpunk | it's kind of almost always easier thing to do | 15:28 |
| lowercase | what is the reason thats still the default. I know i have ddos'd you guys before cause thats the default lol | 15:28 |
| noonedeadpunk | `replace opendev with github` -> and we don't care about loading Microsoft lol | 15:28 |
| noonedeadpunk | lowercase: well, it's not really default tbh. It was us trying to recover things when they went unexpected which lead to such possibility of ddos | 15:30 |
| noonedeadpunk | now we added failure message that makes you aware that things going wrong | 15:30 |
| noonedeadpunk | as dos happens when you run things against all computes for instance | 15:31 |
| noonedeadpunk | while it could be valid behaviour for single controller during OS upgrade for example | 15:31 |
| *** dviroel is now known as dviroel|lunch | 15:31 | |
| noonedeadpunk | So we merged and backported that https://opendev.org/openstack/ansible-role-python_venv_build/commit/de731e7f9e15a99e9587b42de175531bab3f401e | 15:31 |
| noonedeadpunk | it was more that we haven't thought through that ppl will upgrade computes before getting controllers upgraded :) | 15:32 |
| fungi | yeah, the past incidents were mostly when there were dozens (or hundreds) of servers all being told to deploy openstack software at the same time and couldn't find a cache (pre-built wheels, i suppose?) in the local environment so every server independently pulled the source code over the internet. changing your git remotes to github may avoid creating a flood for opendev, but it | 15:36 |
| fungi | doesn't solve the problem that your entire environment just decided to download many redundant copies of the exact same things | 15:36 |
| fungi | and independently run identical compilation of the source repositories into wheel form, and so on | 15:37 |
| jrosser | admin1: best to look at the conditions on the task that has failed | 15:41 |
| jrosser | and then look ad the values you get in your deployment, and understand the reason why, then fix | 15:41 |
| spatel | admin1 are you sure that your playbook overload opendev | 15:46 |
| opendevreview | James Denton proposed openstack/openstack-ansible-ops master: Update MNAIO for Focal https://review.opendev.org/c/openstack/openstack-ansible-ops/+/824486 | 15:47 |
| fungi | spatel: it didn't, the error from the playbooks was prophylactic | 15:47 |
| spatel | +1 | 15:48 |
| jrosser | spatel: the error is triggered when the conditions in the deployment are not as expected | 15:48 |
| jrosser | sadly no-one who encouters this has debugged sufficiently to get to the root cause, other than doing "computes before infra" upgrades | 15:49 |
| jrosser | so ultimately it may be actually a documentation ambiguity, or we handle something wrong at upgrade, or something entirely else...... | 15:50 |
| fungi | and switching the git remote to github isn't really an ideal solution either. as anyone who's run a ci system for testing software on github can attest, even github will block you for performing excessive git operations | 15:51 |
| noonedeadpunk | well, basically there were 2 options - either to fully remove fallback when repo host couldn't be found, which is not great as we assume that the role to build vevn is re-usable outside of the osa, so having container with wheels is not always good assumption | 15:59 |
| noonedeadpunk | or to add check that we don't run without wheels agaist set of hosts | 15:59 |
| noonedeadpunk | so if it's ran against single host or with serial - it won't really cause load. Yes, traffic still will be insane, but end of the day this will take internity for deployer and likely motivate to fix things on their side | 16:00 |
| *** dviroel|lunch is now known as dviroel | 16:39 | |
| admin1 | spatel, i don't think my playbooks overload opendev .. i almost never tinker with the playbooks .. just use them as-is | 17:11 |
| admin1 | the max i change is the variables | 17:11 |
| jrosser | admin1: it would be really helpful to us to understand the conditions in your deployment which cause that error | 17:27 |
| admin1 | so variables ? | 17:30 |
| admin1 | and config | 17:30 |
| admin1 | jrosser, https://gist.githubusercontent.com/a1git/4c6376ede6b65993fd1e55c251fc0d00/raw/43322ef837ab7774f160151003a6186d9b81f8cc/gistfile1.txt | 17:37 |
| jrosser | I mean looking at the values you get for the conditionals in the task that failed | 17:38 |
| opendevreview | Merged openstack/ansible-config_template master: Move zuul jobs layout to centos9 only for master branch https://review.opendev.org/c/openstack/ansible-config_template/+/828295 | 17:38 |
| jrosser | and the working out why that is the case | 17:38 |
| jrosser | the config doesn’t really help | 17:38 |
| admin1 | setup-hosts =pass, setup-infra = pass .. | 17:39 |
| admin1 | let me try to get the logs for setup-openstack | 17:39 |
| admin1 | jrosser, not sure if this helps .. ( this much i could get from tmux ) .. https://gist.githubusercontent.com/a1git/bbdef3bc0c11a20f243b27f958b627b9/raw/90ca16b7a4a6d44770d56cf9c4ecceecc055c152/gistfile1.txt -- basically if i return, except utility and ceilometer .. everything else is good | 17:42 |
| admin1 | i am now running again openstack-ansible setup-openstack.yml -vvvv to a log file so that i can pass the logs | 17:43 |
| spatel | admin1 openstack-ansible default logs all output in /openstack/ansible/log file.. i don't correct path but it does put stuff somewhere there | 17:45 |
| jrosser | you can also delete/move the log file to get one only containing the latest run | 17:46 |
| admin1 | err.. in the deployment host ? | 17:53 |
| admin1 | got it .. got it | 17:54 |
| spatel | what is wrong with my command: openstack role list --user spatel --user-domain ostack2 --project production | 19:35 |
| spatel | openstack role list: error: unrecognized arguments: --user spatel --project production | 19:36 |
| spatel | getting error | 19:36 |
| jrosser | it's not an expected parameter? | 19:37 |
| jrosser | `usage: openstack role list [-h] [-f {csv,df-to-csv,json,table,value,yaml}] [-c COLUMN] [--format-config-file FORMAT_CONFIG] [--quote {all,minimal,none,nonnumeric}] [--noindent] [--max-width <integer>] [--fit-width] [--print-empty] [--sort-column SORT_COLUMN] [--sort-ascending | --sort-descending] [--domain <domain>]` | 19:37 |
| spatel | hmm, same command working with my other openstack running in stein | 19:38 |
| spatel | may be in recent openstack this thing changed | 19:39 |
| spatel | i want to see my role in production project | 19:39 |
| spatel | jrosser they replaced that command with openstack role assignment list :( | 19:47 |
| spatel | crap! | 19:47 |
| admin1 | ok .. i have the full log .. trying to upload it somewhere | 19:52 |
| admin1 | its over my head on why this is happening | 19:53 |
| admin1 | will try to arrange an url | 19:53 |
| admin1 | jrosser, spatel .. sorry for the full log . but here it is => http://admin0.nl/build.txt | 20:01 |
| spatel | omg! this is massive file. you should just port error section | 20:02 |
| spatel | in bottom its saying failed=0 so what is the wrong here? everything looks good to me | 20:03 |
| admin1 | search failed=1 | 20:05 |
| admin1 | there are 4 failed services .. 3x utility and 1x ceilometer | 20:05 |
| admin1 | and then just above is the DOS message | 20:05 |
| jrosser | you say it's the utlity container, yes that is the host with the failed task | 20:06 |
| jrosser | but which playbook are we in? | 20:06 |
| jrosser | rally? | 20:06 |
| admin1 | setup-openstack.yml | 20:06 |
| admin1 | its the whole output of setup-openstack wtih -vvvv | 20:07 |
| jrosser | sure, i see that | 20:07 |
| admin1 | sorry for the big file .. wget and opening via code/sublime might be faster | 20:07 |
| admin1 | my question is .. what can i do to not have this issue .. and also why exactly this issue is | 20:08 |
| jrosser | that is what i want to find | 20:08 |
| jrosser | your config will not say | 20:08 |
| jrosser | you know that setup-openstack.yml is just a playbook which includes many others? | 20:08 |
| admin1 | which is why i included -vvvv | 20:09 |
| jrosser | and you can run all of those others individually to try to narrow down | 20:09 |
| jrosser | -vvvv just adds a whole heap of extra info about each task SSH connection | 20:09 |
| jrosser | you can run openstack-ansible playbooks/<whatever-you-need>.yml | 20:10 |
| admin1 | as far as i have recalled, the error comes during ceilometer | 20:11 |
| admin1 | so just run ceilometer role and capture logs ? | 20:11 |
| jrosser | when i scroll back from the end of your file the last thing it was trying to do was install rally | 20:12 |
| jrosser | and then when i go look in the code, i see this https://github.com/openstack/openstack-ansible-os_rally/blob/master/tasks/main.yml#L60 | 20:12 |
| jrosser | so i understand why that DDOS warning is produced when installing rally | 20:12 |
| admin1 | i will comment out rally .. don't really use it .. | 20:13 |
| jrosser | if you have another error when installing ceilometer, then you can run just the ceilometer playbook on it's own | 20:13 |
| jrosser | and you can also rename the giant log file so that you get a fresh one | 20:13 |
| admin1 | it took a while to take out the log .. multiple ssh and a remote desktop :D | 20:14 |
| admin1 | so the rally thing ... that line .. what does it acutally mean ? | 20:15 |
| admin1 | since i made zero changes, does this mean everyone else doign the same playbooks as me will face that issue | 20:15 |
| admin1 | or i am facing issue coz i don't have rally, and thus its just triggering it since i am running the playbook but not using it ? | 20:15 |
| jrosser | "when installing rally into the virtualenv, do not use the repo server to build wheels" | 20:15 |
| admin1 | oh .. that means take it direct from opendev ? | 20:16 |
| jrosser | and that then makes this fail https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/main.yml#L57 | 20:17 |
| jrosser | it always takes it direct from opendev | 20:17 |
| jrosser | so this is a bug which we need to fix | 20:17 |
| jrosser | i would like to know what happens for ceilometer | 20:19 |
| admin1 | now i know. i am going to dig deeper | 20:19 |
| admin1 | many thanks | 20:19 |
| jrosser | i think that the condition that makes a failure is maybe too strict | 20:20 |
| admin1 | i have gnocchi with redis.. and ceilometer needs to initialize db .. so i am moving towards that | 20:20 |
| jrosser | if it were checking for >3 hosts rather than >2 hosts then you would not get this fail on rally | 20:20 |
| jrosser | noonedeadpunk: ^ | 20:20 |
| jrosser | what is is really trying to prevent is you running os_nova against 100s of compute nodes in parallel | 20:21 |
| jrosser | when wheel building is for some reason disabled or broken | 20:21 |
| admin1 | is there a way to mirror local repo so that no matter what happens, opendev is only used ones and then cached | 20:22 |
| admin1 | that way, it might be future proof | 20:22 |
| jrosser | well, in the same group of stuff we use for dhcp/tftp/pxe is a server running apache + gitweb and a bunch of cron jobs to keep all the repos OSA needs fresh | 20:24 |
| jrosser | and then we override the relevant variables to point to those instead of opendev | 20:24 |
| jrosser | but thats pretty much up to you if you want to do that / how you do it | 20:24 |
| admin1 | thank you for the hint .. | 20:25 |
| admin1 | i will look into it | 20:25 |
| admin1 | or maybe contribute to host a repo mirror ? | 20:26 |
| admin1 | is that also allowed ? | 20:26 |
| jrosser | the repo server used to do this | 20:26 |
| jrosser | but it was not helpful as deployments wanting an internal git mirrror generally had their own already | 20:27 |
| jrosser | and it's complicated with the loadbalancer and upgrades | 20:27 |
| jrosser | so imho this is something that best sits outside OSA, like whatever you have for host provisioning | 20:28 |
| opendevreview | Merged openstack/openstack-ansible-ops master: Update MNAIO for Focal https://review.opendev.org/c/openstack/openstack-ansible-ops/+/824486 | 21:24 |
| *** dviroel is now known as dviroel|out | 21:26 | |
| *** prometheanfire is now known as Guest2 | 22:59 | |
| *** Guest2 is now known as prometheanfire | 23:05 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!