| *** kleini has joined #openstack-ansible | 00:02 | |
| *** kleini_ has quit IRC | 00:02 | |
| *** frenzy_friday has quit IRC | 00:04 | |
| *** frenzy_friday has joined #openstack-ansible | 00:05 | |
| *** spatel has quit IRC | 00:26 | |
| *** spatel has joined #openstack-ansible | 01:08 | |
| *** spatel has quit IRC | 02:22 | |
| *** spatel has joined #openstack-ansible | 02:32 | |
| *** frenzy_friday has quit IRC | 02:53 | |
| *** frenzy_friday has joined #openstack-ansible | 02:54 | |
| *** furkance has joined #openstack-ansible | 03:07 | |
| furkance | Hi, after installing cloudkitty with openstack-ansible, I'm getting this error log http://paste.openstack.org/show/805863/ Any ideas about how I can fix it? Thanks in advance for your help! | 03:07 |
|---|---|---|
| *** spatel has quit IRC | 04:49 | |
| *** luksky has joined #openstack-ansible | 06:44 | |
| noonedeadpunk | furkance: hey | 07:12 |
| noonedeadpunk | jonher: iirc you was facing/seeing same issue at some point? | 07:13 |
| noonedeadpunk | furkance: can you paste /etc/uwsgi/cloudkitty-api.ini ? I'm thinking if that could be https://opendev.org/openstack/openstack-ansible-os_cloudkitty/commit/c35565f419e1630ba4c3090d062d4b533963ff3e | 07:16 |
| *** andrewbonney has joined #openstack-ansible | 07:19 | |
| noonedeadpunk | btw, great news about bullseye! https://zuul.opendev.org/t/openstack/build/3e252dc540a346289dc9dce31069b8c1 | 07:27 |
| noonedeadpunk | lxc is super close as well | 07:28 |
| *** mgariepy has quit IRC | 07:46 | |
| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/792719 | 07:49 |
| jonher | that should be fixed in https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/782837 | 07:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Install py3-dev when not building wheels https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/794158 | 07:59 |
| jonher | it's backported to victoria as well in 789267 | 07:59 |
| noonedeadpunk | furkance: ^ | 08:00 |
| noonedeadpunk | yeah, ok , then I spotted it right:) Thanks for chiming in! | 08:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Add Debian Bullseye support https://review.opendev.org/c/openstack/openstack-ansible/+/783606 | 08:02 |
| jonher | i need to get my other changes to the role pushed too, for example the shipped cloudkitty.conf now has 2.5 year old syntax with stuff not working so it just crashes indefinitely, but i can push that for review later today what i've used in my testing | 08:05 |
| noonedeadpunk | that would be awesome | 08:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Use appropriate osbpo repo for Bullseye https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/794161 | 08:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Add Debian Bullseye support https://review.opendev.org/c/openstack/openstack-ansible/+/783606 | 08:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Cleanup after service variables merged https://review.opendev.org/c/openstack/openstack-ansible/+/769974 | 08:17 |
| noonedeadpunk | jrosser: there's some bug in oslo for sure, but it's not where I was expecting it... http://paste.openstack.org/show/806259/ | 08:56 |
| noonedeadpunk | so when you just drop file and not restart service, it will result in 500 with following trace http://paste.openstack.org/show/806260/ | 08:56 |
| jrosser | i wonder if it can only cope with an atomic replacement of the policy file | 08:59 |
| jrosser | like mv the new file over the old one | 08:59 |
| noonedeadpunk | yeah. And if I re-add empty file it's recovering | 09:05 |
| noonedeadpunk | But the problem is https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/nova_post_install.yml#L98-L106 | 09:05 |
| noonedeadpunk | But probably worth pathing oslo to handle this scenario... | 09:07 |
| *** mgoddard_ has joined #openstack-ansible | 09:13 | |
| jrosser | i guess theres a lot of uncertainty about the inital state, what happens with whichever $tool during deployment and then the final state which oslo needs to be robust to | 09:16 |
| *** mgoddard_ is now known as mgoddard | 09:18 | |
| *** mgariepy has joined #openstack-ansible | 09:19 | |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/792719 | 09:20 |
| *** odyssey4me has joined #openstack-ansible | 09:25 | |
| *** jnamdar has joined #openstack-ansible | 10:00 | |
| jnamdar | hi o/ | 10:01 |
| jnamdar | I'm having a bit of a headache on my panko container, looks like something's messed up in the venv | 10:01 |
| jnamdar | getting this on the panko-api service: `Fatal Python error: initfsencoding: Unable to get the locale encoding` and `ModuleNotFoundError: No module named 'encodings'` | 10:01 |
| noonedeadpunk | sounds familiar to cloudkitty issue... | 10:02 |
| noonedeadpunk | what's in your /etc/uwsgi/panko-api.ini? | 10:02 |
| jnamdar | can't seem to find such a module with pip, so that's weird | 10:03 |
| jnamdar | https://pastebin.ubuntu.com/p/3dMpjyrSpt/ | 10:03 |
| noonedeadpunk | yeah.... | 10:03 |
| jnamdar | tried looking for a solution, I saw something like python2 and 3 binaries conflicts | 10:04 |
| jnamdar | but not sure what to do since this is in a venv | 10:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_panko master: Fix wsgi_venv path for panko-api https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/794195 | 10:05 |
| noonedeadpunk | jnamdar: this should fix it ^ | 10:05 |
| jnamdar | ok thanks | 10:08 |
| jnamdar | so the filter would just leave the last directory in the path? | 10:08 |
| jnamdar | like virtualenv = /bin ? | 10:09 |
| noonedeadpunk | nope, like `virtualenv = /openstack/venvs/panko-21.2.6.dev1` | 10:09 |
| jnamdar | oh | 10:09 |
| noonedeadpunk | jrosser: damn it - bullseye got maria 10.5.10 from native repos.... | 10:10 |
| jnamdar | noice, seems to be working now, thx noonedeadpunk | 10:13 |
| jnamdar | had to add a trailing /bin as well for the panko-api binary full path | 10:14 |
| noonedeadpunk | good catch | 10:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_panko master: Fix wsgi_venv path for panko-api https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/794195 | 10:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_panko master: Fix wsgi_venv path for panko-api https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/794195 | 10:17 |
| opendevreview | Jonathan Herlin proposed openstack/openstack-ansible-os_cloudkitty master: Cloudkitty role cleanup and config updates https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/794199 | 10:24 |
| jonher | ^ that is what i'm running in my test deployment, if i removed some playbook used for some test etc. i'm sure i'll be told so :) | 10:25 |
| jonher | cloudkitty config is fetched from devstack setup that i then moved to the review above and deployed in a AIO | 10:26 |
| jrosser | noonedeadpunk: this is why we get the native repo https://github.com/openstack/openstack-ansible-galera_server/commit/c2e17ae47f451bb2c4f62bab690ea5f2dbbc1f2d | 10:41 |
| noonedeadpunk | yeah I know why... But it also means we can't change it. And when we were merging this it was 10.5.9 :( | 10:44 |
| *** mgariepy has quit IRC | 11:04 | |
| jrosser | unfortunatley the old packages are not still in the mirror so we can't pin it back | 11:04 |
| jrosser | and 10.5.10 is broken isnt it? | 11:04 |
| noonedeadpunk | it is | 11:14 |
| noonedeadpunk | and lxc already started failing on cinder "randomly" | 11:14 |
| *** zigo has joined #openstack-ansible | 11:25 | |
| noonedeadpunk | zigo: any chance to influence mariadb version used for debian bullseye repo? | 11:25 |
| zigo | noonedeadpunk: At this point, there's no way to change the version, no. Though if you have the bug you referred to on freenode, you should open a bug against the package. | 11:26 |
| zigo | noonedeadpunk: I did setup Cinder on Bullseye and had no issue... | 11:26 |
| zigo | noonedeadpunk: Are you *sure* Bullseye is affected by the bug you referred? | 11:26 |
| noonedeadpunk | zigo: yeah, we see it in CI | 11:27 |
| noonedeadpunk | https://bf8b3683a4d2b36abe64-1877198ef7d2100f38eb37f99c1c7298.ssl.cf1.rackcdn.com/783606/20/check/openstack-ansible-deploy-aio_lxc-debian-bullseye/aa42a18/logs/ara-report/results/2512.html | 11:27 |
| zigo | noonedeadpunk: Please do open the bug in the Debian BTS. The MariaDB maintainer is a very reponsive and responsible person, he WILL address if it you open the bug. | 11:28 |
| noonedeadpunk | ok, gotcha | 11:28 |
| jrosser | there was a corresponding mariadb JIRA ticket i think? | 11:29 |
| noonedeadpunk | yep https://jira.mariadb.org/browse/MDEV-25673 | 11:29 |
| noonedeadpunk | zigo: another thing I was wondering is presence of api-paste's in deb pakages? | 11:30 |
| noonedeadpunk | It might be we just miss to install containing it package... | 11:31 |
| zigo | noonedeadpunk: What do you mean? | 11:31 |
| zigo | What package is that? | 11:31 |
| noonedeadpunk | yeah, jsut a sec... | 11:32 |
| noonedeadpunk | zigo: so for example, glance-common for ubuntu does contain this file https://opendev.org/openstack/glance/src/branch/master/etc/glance-api-paste.ini | 11:35 |
| noonedeadpunk | ie https://launchpad.net/ubuntu/impish/amd64/glance-common/2:22.0.0-0ubuntu1 | 11:35 |
| noonedeadpunk | zigo: so I was wondering if it's as designed not to include it for deb? | 11:36 |
| zigo | noonedeadpunk: The Debian package has it too, it is just installed in a different way. | 11:36 |
| *** luksky has quit IRC | 11:36 | |
| *** luksky has joined #openstack-ansible | 11:37 | |
| noonedeadpunk | I think I actually already asked this before :( | 11:37 |
| noonedeadpunk | Oh, it's in control or smth? | 11:37 |
| zigo | noonedeadpunk: In the Ubuntu package (I haven't checked, but it should be the case), the /etc/glance/glance-api-paste.ini is just a CONFFILE (note: CONFFILE != configuration file), meaning it's part of the package in /etc. | 11:37 |
| noonedeadpunk | yep it is | 11:37 |
| zigo | noonedeadpunk: In Debian, glance-api-paste.ini is copied from /usr/share/glance-common/glance-api-paste.ini *IN THE POSTINST* of the package, and set with the root:glance and 640 unix rights. | 11:38 |
| zigo | It's copied *ONLY IF* there's no such file already. | 11:38 |
| zigo | Meaning, during an upgrade, it's the admin's responsibility to copy it if it has changed. | 11:39 |
| zigo | The reason I'm doing this way is to avoid dpkg's prompt durring upgrade if some changes has been done in the api-paste.ini file *and* because I want root:glance 640 as unix rights (so the user must be created, then the file copied). I strongly believe it's safer and nicer this way. | 11:40 |
| zigo | In Ubuntu, I haven't checked, but I'm quite sure it should be root:root 664 as unix rights, no? | 11:40 |
| noonedeadpunk | zigo: oh! I found and the issue is different | 11:40 |
| noonedeadpunk | python3-glance does not depend on glance-common | 11:40 |
| zigo | Right, it's the opposite way ! :) | 11:41 |
| zigo | glance-common depends on python3-glance, and that's how it should be. | 11:41 |
| zigo | IMO, you should *never* attempt to install python3-glance or glance-common on the configuration management level. | 11:41 |
| zigo | What you should care about, are the final services. | 11:41 |
| zigo | ie: you should be installing glance-api, and nothing else... | 11:41 |
| zigo | The packaging should be doing the rest for you. | 11:42 |
| noonedeadpunk | well, for ubuntu it's `Depends: glance-common` | 11:42 |
| noonedeadpunk | so we basically do https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/master/vars/debian.yml#L32 | 11:42 |
| noonedeadpunk | yeah, ok, so glance-api for debian depends on glance-common and for ubuntu only on python3-glance | 11:45 |
| noonedeadpunk | so that is really fair point! thanks for sorting this out! | 11:45 |
| zigo | noonedeadpunk: I just had a look to what's done in Ubuntu, and indeed, they have stuff backwards. The dependency chain *SHOULD* be: | 11:51 |
| zigo | glance-api -> glance-common -> python3-glance | 11:51 |
| zigo | Just like it's done by all other packages. But they have things backward this way: | 11:51 |
| zigo | glance-api -> python3-glance -> glance-common | 11:51 |
| zigo | I see it as an Ubuntu defect rather than a feature... | 11:51 |
| zigo | Anyway, on your side, just apt-get install glance-api, and you're good to go on both distros. | 11:51 |
| opendevreview | Merged openstack/openstack-ansible master: Gather additional required facts to min https://review.opendev.org/c/openstack/openstack-ansible/+/789776 | 11:55 |
| noonedeadpunk | zigo: yeah, agree, that's looks like nice solution | 11:57 |
| *** mgariepy has joined #openstack-ansible | 12:26 | |
| *** spatel has joined #openstack-ansible | 12:58 | |
| opendevreview | Jonathan Herlin proposed openstack/openstack-ansible-os_cloudkitty master: Cloudkitty role cleanup and config updates https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/794199 | 13:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Install glance-api for distro scenario https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/794220 | 13:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Add Debian Bullseye support https://review.opendev.org/c/openstack/openstack-ansible/+/783606 | 13:20 |
| furkance | noonedeadpunk, thanks! that one resolved but now I'm getting this one; http://paste.openstack.org/show/806274/ | 13:34 |
| noonedeadpunk | and how did you apply that patch? Manually or...? | 13:36 |
| furkance | with using git | 13:37 |
| *** manti[m] has joined #openstack-ansible | 13:37 | |
| noonedeadpunk | well I mean you could cherry-pick/branch/pull/etc | 13:38 |
| noonedeadpunk | as in case of using smth other then cherry-pick you could get master branch of the role or smth like that | 13:38 |
| noonedeadpunk | (not sure) | 13:38 |
| furkance | first I checked out to stable/victoria branch and then I used git pull | 13:39 |
| *** manti[m] is now known as manti | 13:39 | |
| noonedeadpunk | aha | 13:39 |
| noonedeadpunk | gotcha | 13:40 |
| noonedeadpunk | so the issue here is that in migration it fails to "alembic.util.exc.CommandError: Can't locate revision identified by 'Ifbf5b2515c7'" | 13:40 |
| noonedeadpunk | and `Ifbf5b2515c7` looks like gerrit change-id rather then git sha... | 13:40 |
| noonedeadpunk | yeah looks like it's about this specific patch https://review.opendev.org/c/openstack/cloudkitty/+/755865 | 13:42 |
| noonedeadpunk | which is from Wallaby | 13:42 |
| noonedeadpunk | so you should not have it on V, but I suspect your database before checkout might be using this migration? | 13:47 |
| furkance | when I was running openstack-ansible installation for cloudkitty at first I forgot to specify branch and commit in defaults/repo_packages/openstack_services.yml, so in previous installation attempts of mine probably it used Wallaby (a side note, our environment is stable/victoria) | 13:54 |
| spatel | noonedeadpunk jrosser i am ready to submit OVN cluster create ansible playbook | 13:54 |
| *** tosky has joined #openstack-ansible | 13:54 | |
| spatel | Just running some last test to make sure it pass them, also please feel free to modify my playbook or comments because it may not meet your expectation :) | 13:55 |
| *** zbr has joined #openstack-ansible | 13:55 | |
| noonedeadpunk | furkance: if cloudkitty service is empty, you should try jsut dropping database and re-run role | 14:03 |
| noonedeadpunk | spatel: let's see:) I think it's also worth pinging James once path will be sent | 14:04 |
| spatel | he is not here, i did ping him in old channel | 14:04 |
| furkance | noonedeadpunk, thanks, I dropped it when u mentioned and waiting for the installation complete | 14:05 |
| jrosser | spatel: i would expecy you to be adding tasks to the os_neutron role to make the OVN cluster, rather than a new playbook? | 14:11 |
| spatel | jrosser its part of os_neutron/task/provider/ just 2 files | 14:12 |
| jrosser | oh ok, np | 14:12 |
| spatel | its only for ubuntu at present, hope i will adjust later for centos (i don't have centos lab at present) | 14:13 |
| spatel | i think it should work for all distro.. its very simple tasks. | 14:13 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Add debian bullseye support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/793895 | 14:16 |
| furkance | noonedeadpunk, it completed succesfully and both cloudkitty-api, cloudkitty-processor services working properly, thanks for the help | 14:17 |
| jonher | make sure the service is not crashing repeatedly, unless you also have the patched config from 794199 or another valid cloudkitty.conf | 14:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Install glance-api for distro scenario https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/794220 | 14:23 |
| noonedeadpunk | zigo: returning back to packages - I guess I know why we were trying to install just python3-glance. Because glance-api is providing all systemd scripts over which we want to have control. Ofc we can do just overrides, but it was easier to just not install them. As python3-glance provide all required binaries | 14:27 |
| noonedeadpunk | in the meanwhile in deb to get binaries you have to also install glance-api, which will bring in systemd scripts as well | 14:28 |
| zigo | noonedeadpunk: Well, in Debian, the packages are using uwsgi, so if you don't install glance-api, you'll get back to eventlet, which is kind of super bad ... | 14:28 |
| zigo | noonedeadpunk: Is ansible using uwsgi or apache, or what? | 14:29 |
| noonedeadpunk | well if we're talking about glance specifically, then you have glance-wsgi-app in glance-common | 14:29 |
| noonedeadpunk | we're using uwsgi | 14:29 |
| noonedeadpunk | but, for glance specifically it's not always best choice, as things known to be broken with uwsgi | 14:30 |
| noonedeadpunk | but if we talk about cinder, there're no binaries in cinder-common except wsgi | 14:32 |
| noonedeadpunk | I guess we will have to somehow workaround this in our roles then. And do just overrides of systemd when we need | 14:34 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Remove obsoleted packages https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/793974 | 14:35 |
| *** opendevreview has quit IRC | 14:38 | |
| *** opendevreview has joined #openstack-ansible | 14:48 | |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Replace py2 httplib2 package with py3 alternative https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794002 | 14:48 |
| Gue______ | spatel hello | 14:50 |
| *** Gue______ is now known as jamesdenton | 14:51 | |
| spatel | Hi | 14:51 |
| jamesdenton | didn't realize my nick had changed | 14:51 |
| spatel | hehe | 14:51 |
| spatel | good to see you here.. | 14:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add Debian Bullseye support https://review.opendev.org/c/openstack/openstack-ansible/+/783606 | 14:54 |
| noonedeadpunk | I dunno why, but I see this "Guest" pretty frequently on OFTC | 14:54 |
| spatel | jamesdenton doing final validation of my patch to make sure nothing missing. I don't know how to check with AIO so will see how CI will react on it | 14:54 |
| jamesdenton | you might want to see if you can leverage the MNAIO | 14:55 |
| spatel | never tired that before but sure good to have.. | 14:56 |
| jamesdenton | https://github.com/openstack/openstack-ansible-ops/tree/master/multi-node-aio | 14:59 |
| jamesdenton | export MNAIO_ANSIBLE_PARAMETERS="-e osa_enable_networking_ovn=true"; ./build.sh | 14:59 |
| jamesdenton | so you might deploy, and then apply your patches and run the playbook(s) again | 15:00 |
| jamesdenton | not sure if there's a way to bring a patch in there at deploy time or not | 15:00 |
| jrosser | fast disks for that will be a massive bonus | 15:02 |
| noonedeadpunk | is it still working? o_O | 15:02 |
| noonedeadpunk | btw, I think we should not implement Bullseye distro install atm | 15:03 |
| noonedeadpunk | maybe bring it later on or for X... | 15:03 |
| noonedeadpunk | as it will be really a mess with systemd | 15:04 |
| spatel | jamesdenton thanks!! will give it a try (i may need to get bigger disk and fast :) ) | 15:04 |
| jamesdenton | i recommend NVMe, it helps! | 15:08 |
| jamesdenton | noonedeadpunk i last tested 11 months ago. i think | 15:09 |
| *** luksky has quit IRC | 15:09 | |
| noonedeadpunk | oh, good to know:) | 15:09 |
| *** spatel has quit IRC | 15:17 | |
| *** spatel has joined #openstack-ansible | 15:22 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rsyslog_client master: Replace linters test with integarted one https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/794254 | 15:23 |
| opendevreview | Merged openstack/openstack-ansible master: Always use checkout for git_requirements library https://review.opendev.org/c/openstack/openstack-ansible/+/791011 | 15:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rsyslog_client master: Replace linters test with integarted one https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/794254 | 15:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Add upgrade jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/794255 | 15:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_container_create master: Add upgrade jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/794256 | 15:29 |
| noonedeadpunk | would be awesome to summon a vote on https://review.opendev.org/c/openstack/openstack-ansible/+/790042/8 | 15:32 |
| jnamdar | hey all, when I use SCENARIO='aio_lxc_telemetry', is ceilometer configured to communicate with panko? | 15:33 |
| jnamdar | i have nothing panko-related in /etc/ceilometer, although it may default to panko for some missing config files | 15:34 |
| noonedeadpunk | I'm not sure we were really integrating it as I'm not sure if anybody uses panko for real atm | 15:35 |
| jnamdar | also I think gnocchi and ceilometer are well-configured, but I see nothing bringing up in gnocchi | 15:35 |
| jnamdar | idk how I can make sure ceilometer is sending anything | 15:35 |
| noonedeadpunk | gnocchi and ceilometer integrtation should be working | 15:36 |
| jnamdar | I'm getting a lot of "skip pollster ****, no resources found this cycle" | 15:36 |
| noonedeadpunk | but there were some issues with gnocchi itself (or more wierdness in logs) I had no time to look into. | 15:36 |
| noonedeadpunk | eventually ceilometer also gathers a lot with notification agent | 15:37 |
| noonedeadpunk | I'm really don't use pollster much - maybe just for glance iamges... | 15:37 |
| noonedeadpunk | the most important stuff goes through notifications anyway | 15:38 |
| jrosser | i think panko has just been retired too? | 15:39 |
| noonedeadpunk | regarding panko - really never tried it out, so can't help anyhow :( | 15:39 |
| noonedeadpunk | oh, indeed | 15:39 |
| noonedeadpunk | wallaby is the last release | 15:40 |
| jrosser | so we almost certainly drop the role for X | 15:40 |
| jnamdar | oh damn yea | 15:40 |
| jnamdar | it just did lol | 15:40 |
| jnamdar | am I supposed to see notifications going out in ceilometer-agent-notification.service? | 15:42 |
| jnamdar | I just have a bunch of logs at startup but that's all | 15:42 |
| noonedeadpunk | nope, you're not. But you should see incoming api requests to gnocchi | 15:44 |
| noonedeadpunk | either in haproxy or in access log | 15:45 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 16:17 |
| spatel | jamesdenton here you go.. lets review comments - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 16:18 |
| spatel | noonedeadpunk jrosser ^^ | 16:18 |
| spatel | wait.. need to add setup file :) | 16:19 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266 | 16:19 |
| jamesdenton | i will try to kick the tires on this later today/tomorrow | 16:21 |
| jamesdenton | spatel any impact to traffic if the playbooks are run? | 16:32 |
| spatel | are you talking about VM traffic? | 16:33 |
| *** mgariepy has quit IRC | 16:33 | |
| spatel | there won't be any impact because you have other nodes in cluster and client using haproxy to talk to ovn central | 16:34 |
| spatel | jamesdenton ^^ | 16:35 |
| jrosser | does the leader ever change? | 16:37 |
| jrosser | i.e in a failover situation | 16:38 |
| *** jnamdar has quit IRC | 16:40 | |
| spatel | yes | 16:43 |
| spatel | when you build first time then node-1 is leader and node-2/3 join node-1 | 16:43 |
| spatel | but when cluster is running and you reboot node-1 then leader can be anyone after election node-2 or node-3 | 16:44 |
| spatel | so lets say node-1 die with some reason in that case we need to find who is new leader and use that leader to bring back node-1 | 16:45 |
| spatel | my patch will take care of that also | 16:45 |
| noonedeadpunk | spatel: commented | 16:46 |
| spatel | noonedeadpunk thanks some of code i steal from existing playbooks so may be need improvement.. i will review your comments and improve code | 16:47 |
| noonedeadpunk | I have no idea about functional aspect though | 16:48 |
| *** tosky has quit IRC | 16:48 | |
| spatel | codes are pretty straight forward so not sure need any logic change but yes lets see if someone find any bug | 16:50 |
| spatel | noonedeadpunk i may need your help in some of your comments like, i tried to use handle but it didn't work and find handler | 16:51 |
| spatel | i tried to add in handler/main.yml but it was fail to find | 16:51 |
| noonedeadpunk | um? I dunno how it might appear to not being foung | 16:53 |
| noonedeadpunk | *found | 16:53 |
| noonedeadpunk | it's also more about what should trigger it | 16:54 |
| spatel | i will try again and show you error logs | 16:54 |
| spatel | noonedeadpunk do you mean with_item look here? https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266/2/tasks/providers/ovn_cluster_setup.yml#31. | 16:56 |
| spatel | i didn't understand systemd loop that is why asking | 16:56 |
| spatel | jrosser here is the full detail of clustering in my words - https://satishdotpatel.github.io/openstack-ansible-ovn-clustering/ | 17:00 |
| jrosser | spatel: well i was just wondering what happen if you restart one of the services without using the playbooks, like reboot a node or something | 17:01 |
| jrosser | that the leader seems to be used in the /etc/defaults template, and thats maybe only valid at the time the template was made? | 17:01 |
| spatel | if you reboot node then other node will take leader role... | 17:02 |
| noonedeadpunk | spatel: http://paste.openstack.org/show/806287/ | 17:02 |
| noonedeadpunk | or smth like that | 17:02 |
| noonedeadpunk | with_items can be just replaced with loop | 17:02 |
| spatel | i didn't try to reboot all 3 node at same time.. (let me do it ) | 17:03 |
| *** furkance has quit IRC | 17:03 | |
| noonedeadpunk | https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#iterating-over-a-simple-lis | 17:03 |
| noonedeadpunk | * https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#iterating-over-a-simple-list | 17:03 |
| spatel | noonedeadpunk hmm interesting i will try that code and see | 17:03 |
| *** furkance has joined #openstack-ansible | 17:03 | |
| spatel | jrosser i didn't hardcode leader node anywhere, code will find new leader dynamically and use that to join cluster | 17:04 |
| spatel | leader is basically like donor | 17:05 |
| jrosser | oh, i was a bit confused with this then i think https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266/2/templates/ovn-central.j2 | 17:05 |
| spatel | oh okay, leader_node is fact generated dynamically | 17:05 |
| jrosser | so thats really my question, in that config file is it OK to template out what the leader node is at that time | 17:06 |
| jrosser | given that it may change | 17:06 |
| spatel | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/794266/2/tasks/providers/ovn_cluster_setup.yml#37 | 17:06 |
| jrosser | oh yes i see where it comes from :) but i am wondering about the case when you restart the service at some point in the future | 17:07 |
| jrosser | anyway - i've never had hands-on experience with any of this | 17:08 |
| spatel | If you have 3 node running cluster and want to add node-4 in that case you need to find leader and my code finding that leader and supply that value to template config. | 17:10 |
| spatel | but if you building fresh cluster then i hardcoded node-1 will be leader | 17:11 |
| noonedeadpunk | I think the question was - what will happen if you just just power cord to the current configured leader | 17:11 |
| spatel | OVN is really cool and you should be playing with :) its future | 17:11 |
| noonedeadpunk | *just cut | 17:11 |
| spatel | if you power cut current leader then it cluster automatically elect new leader | 17:12 |
| noonedeadpunk | will it be re-elected despite config file is providing it explicitly? | 17:12 |
| noonedeadpunk | ok) | 17:12 |
| spatel | yes.. despite config it will elect new leader | 17:12 |
| spatel | just like rabbitMQ | 17:13 |
| noonedeadpunk | and then when it will come back it will become leader again? | 17:13 |
| spatel | no it won't come back | 17:13 |
| spatel | leader can be anywhere in cluster no matter in running cluster | 17:14 |
| spatel | when you add new node in cluster that time you need to know which node is leader to just join cluster | 17:14 |
| spatel | jrosser noonedeadpunk check this out - https://ibb.co/mB8BDmf | 17:20 |
| spatel | node-1 was leader and i reboot | 17:21 |
| spatel | after reboot node-2 turn into leader and node-1 follower | 17:21 |
| noonedeadpunk | yep, gotcha) | 17:23 |
| noonedeadpunk | We really struggle so much with ovs atm... | 17:23 |
| spatel | we should make OVS default for OSA deployment :) | 17:23 |
| noonedeadpunk | you meant ovn?:) | 17:23 |
| spatel | Just OVS deployment instead of linuxbridge :) | 17:24 |
| noonedeadpunk | really I dunno. imo lxb is the way more reliable then ovs | 17:24 |
| spatel | Packstack using default OVN deployment, they are ahead of everyone | 17:24 |
| noonedeadpunk | devstack is using ovn by default | 17:25 |
| spatel | +1 for simple and reliable | 17:25 |
| noonedeadpunk | so, it's more like a standart now | 17:25 |
| spatel | Yeah! I am planning to run in production so that is why doing all these practices | 17:26 |
| noonedeadpunk | Like now we were facing issue with ovs being stuck because of glibc bug. And update glibc, restart ovs and not break network for half an hour is a real challednge with ovs... | 17:26 |
| noonedeadpunk | while I never saw nothing even close nasty with lxb | 17:27 |
| noonedeadpunk | and hope not seeing with ovn as well | 17:27 |
| noonedeadpunk | and things like, octavia can't spawn new amphoras because of stuck dhcp agent is insane thing... | 17:28 |
| spatel | ovs had lots of issue but now its much better with latest OS and kernel | 17:30 |
| noonedeadpunk | latest is... focal? | 17:30 |
| spatel | Yes focal and up.. | 17:30 |
| noonedeadpunk | I'm still on bionic | 17:30 |
| spatel | i haven't seen any issue again i am doing all test in lab | 17:30 |
| spatel | OVN has built in loadbalancer i didn't test yet but | 17:31 |
| noonedeadpunk | yeah, on small clouds it's okey, but when you have 7 net nodes 400 namespaces each - meh.... | 17:32 |
| noonedeadpunk | well, to be fair - lxb would just die | 17:32 |
| spatel | I love linuxbridge trust me.. i am running 800 node cloud with linuxbridge and not a single issue | 17:33 |
| spatel | 99.999 uptime since i built it almost 3 years now | 17:33 |
| spatel | OVN support hardware VTEP and can directly talk to your TOR switch if support openflow, that is really cool | 17:35 |
| spatel | you don't need fancy NIC card etc.. | 17:35 |
| *** mgariepy has joined #openstack-ansible | 17:37 | |
| *** andrewbonney has quit IRC | 17:49 | |
| *** ioni has quit IRC | 17:51 | |
| *** gshippey has quit IRC | 18:31 | |
| *** luksky has joined #openstack-ansible | 19:05 | |
| *** luksky has joined #openstack-ansible | 19:05 | |
| *** fungi has joined #openstack-ansible | 19:17 | |
| fungi | just a heads up, we've had now another openstack-ansible user end up causing a denial of service on our git servers by trying to perform a site upgrade, which flooded the opendev.org git server farm with requests. there's supposed to be a central git cache the servers all hit in an osa deployment, right? | 19:19 |
| *** tosky has joined #openstack-ansible | 19:23 | |
| jrosser | fungi: no, there isn't a cache of git repos | 19:30 |
| jrosser | in the distant past there used to be, there is however a cache of built wheels | 19:31 |
| jrosser | and the wheels should be built just the once | 19:31 |
| fungi | so if someone has a site with, say, 200 nova hosts... those will all clone nova independently from us? | 19:32 |
| fungi | when this came up before you (i think? maybe it was someone else) said that if it was doing that it was a sign the local cache was broken. i guess i assumed git cache, but maybe it was the wheel cache then? | 19:33 |
| jrosser | no, that should not happen | 19:33 |
| *** b1tsh1ft3r has joined #openstack-ansible | 19:33 | |
| jrosser | the first time the venv is built for a service/release combination the wheels should be built once | 19:33 |
| fungi | this is at least the third different osa user who has done this, so it would be great to figure out how it's getting triggered | 19:34 |
| jrosser | and then repeatedly installed from the local cache for all the remaining times they are needed | 19:34 |
| jrosser | yeah, well i did a whole series of this https://review.opendev.org/q/owner:jonathan.rosser%2540rd.bbc.co.uk+user-agent | 19:34 |
| jrosser | though that does rely on point releases getting out and someone trying to use one | 19:34 |
| fungi | i mean, in this case the challenge isn't identifying who/what software was doing it, thankfully they were the one to reach out to us about the outage | 19:35 |
| fungi | and said they were trying to do an operating system upgrade for an openstack-ansible site | 19:35 |
| jrosser | problem has been finding someone willing to do even basic investigation of this during the time they are in the middle of an upgrade | 19:36 |
| fungi | and corresponding openstack version upgrade | 19:36 |
| jrosser | first time turned into "we're just going to fight our way though rather than try to fix this" | 19:36 |
| fungi | yeah, i've urged people to come in here every time it crops up | 19:36 |
| fungi | but that seems not to happen | 19:36 |
| jrosser | sadly not | 19:37 |
| fungi | this user similarly worked around it by switching all hard-coded instances of opendev.org to github.com in osa playbooks/roles | 19:37 |
| b1tsh1ft3r | Looking for a little guidance on getting designate rolled out. Im running openstack-ansible train release. Looking at the docs here: https://docs.openstack.org/openstack-ansible-os_designate/train/ it would appear its just simply running the playbook and defining a pools.yml? However, im unsure if the dns backend definition is something that should reside in the sample playbook or perhaps the user_variab | 19:37 |
| b1tsh1ft3r | les.yml on the deploy host maybe? | 19:37 |
| jrosser | sad thing is it's making the deployment *massively* slower for them | 19:37 |
| spatel | b1tsh1ft3r i have tried my best to document designate here so give it a try and see - https://satishdotpatel.github.io/designate-integration-with-powerdns/ | 19:43 |
| spatel | PowerDNS is outside OSA infrastructure.. so if you have other DNS adjust according | 19:43 |
| b1tsh1ft3r | perfect! i'll give it a look | 19:43 |
| spatel | by the way is this your username or your password.. just kidding :) | 19:45 |
| opendevreview | Jeremy Stanley proposed openstack/openstack-ansible master: DNM: exercise base-test in ansible deploy jobs https://review.opendev.org/c/openstack/openstack-ansible/+/794332 | 19:49 |
| spatel | noonedeadpunk what is the advantage of config_template ? | 19:56 |
| *** lowercase has joined #openstack-ansible | 20:00 | |
| lowercase | hey whats going on. | 20:00 |
| lowercase | @fungi | 20:01 |
| lowercase | @jrosser | 20:02 |
| jrosser | hi - it would be really great to try to understand why your upgrade is causing so much load on the opendev server | 20:02 |
| jrosser | it's occurred a few times now for different people | 20:03 |
| jrosser | though it's getting kind of late here for me so ideally we could talk about it tomorrow? | 20:03 |
| lowercase | I'll be in the office tomorrow. I'm doing a couple chassis swaps to our openstack swift cluster. | 20:04 |
| lowercase | So, maybe friday? | 20:04 |
| lowercase | or is there a specific time? I could make myself be in front of a computer and out of the DC if the time is reasonable. | 20:06 |
| jrosser | I’m around tomorrow EU time but not Friday | 20:09 |
| jrosser | but noonedeadpunk is probably around then too | 20:10 |
| jrosser | it’s quite a big issue really because it breaks all the openstack CI jobs as well as your upgrades | 20:10 |
| lowercase | very much so | 20:11 |
| jrosser | potentially thousands of donated CPU hours of testing breaks | 20:11 |
| lowercase | How about 10:00AM UTC, thats 11:00AM London | 20:12 |
| lowercase | oh wait, thats stupid early for me | 20:13 |
| lowercase | 3:00 pm London? | 20:13 |
| *** b1tsh1ft3r has quit IRC | 20:14 | |
| *** spatel has quit IRC | 20:18 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix https://review.opendev.org/c/openstack/openstack-ansible/+/789376 | 20:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix https://review.opendev.org/c/openstack/openstack-ansible/+/789376 | 20:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add Debian Bullseye support https://review.opendev.org/c/openstack/openstack-ansible/+/783606 | 20:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix https://review.opendev.org/c/openstack/openstack-ansible/+/789376 | 20:26 |
| *** b1tsh1ft3r has joined #openstack-ansible | 20:33 | |
| *** b1tsh1ft3r has quit IRC | 20:51 | |
| *** b1tsh1ft3r has joined #openstack-ansible | 21:13 | |
| *** jamesdenton has quit IRC | 21:21 | |
| *** Gues_____ has joined #openstack-ansible | 21:23 | |
| *** b1tsh1ft3r has quit IRC | 21:30 | |
| *** furkance has quit IRC | 21:56 | |
| *** furkance has joined #openstack-ansible | 21:57 | |
| *** b1tsh1ft3r has joined #openstack-ansible | 22:00 | |
| *** b1tsh1ft3r has quit IRC | 22:17 | |
| *** furkance has quit IRC | 22:30 | |
| *** furkance has joined #openstack-ansible | 22:30 | |
| *** luksky has quit IRC | 22:35 | |
| *** b1tsh1ft3r has joined #openstack-ansible | 23:15 | |
| *** lowercase has quit IRC | 23:23 | |
| *** tosky has quit IRC | 23:25 | |
| *** b1tsh1ft3r has quit IRC | 23:30 | |
| *** furkance has quit IRC | 23:41 | |
| *** furkance has joined #openstack-ansible | 23:41 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!