Tuesday, 2022-12-27

moha7Hi there; I did a new deploy. But still have problem with up-ing the port 9696! Galera is active and running but with this error in the status output: [ERROR] Invalid (old?) table or database name '#tmp', --> http://ix.io/4jFA08:22
moha7neutron.slice had this error: ValueError: :6642: bad peer name format, but after restarting this service, it's now active and running08:24
moha7port 9696 is not up on ct1-neutron-server-container-8d8a1c6408:24
moha7I can not login to Horizon dashboard because this error: https://ibb.co/pv52pCT08:25
kleiniI delete those tmp files in /var/lib/mysql manually before starting MariaDB.09:05
noonedeadpunkmoha7: what are values of tmpdir and ignore_db_dirs you have in /etc/mysql/my.cnf ?10:13
moha7tmpdir = /var/lib/mysql/#tmp10:26
moha7ignore_db_dirs = '#tmp',lost+found10:27
moha7noonedeadpunk:  ^10:27
moha7openstack_user_config.yml: http://ix.io/4jB410:28
moha7user_variables.yml: http://ix.io/4jGa10:28
noonedeadpunkthat looks quite legit10:28
noonedeadpunkgive me couple of mins to setup multinode with galera to check that10:29
noonedeadpunkAs I'm quite sure that these #tmp shouldn't be an issue as they're ignored10:29
moha7how you do that in mins! It takes exactly 5 hours each time I go for fresh deploy10:30
kleiniI had issues with those tmp folders, too. Don't actually remember, if they are named "#tmp" or just "tmp.something".10:30
noonedeadpunkkleini: well, we've defined couple of vars by default to avoid such issues10:31
noonedeadpunkmoha7: `./scripts/gate-check-commit.sh aio_infra_lxc deploy source` :D10:32
noonedeadpunkbut it's all in one at the end10:32
moha7ct1-galera-container -->  systemctl status mariadb --> http://ix.io/4jGb 10:33
moha7ct3-galera-container --> systemctl status mariadb --> http://ix.io/4jGc10:33
noonedeadpunkwell, I would totally suggest cleaning up /var/lib/mysql from these `tmp.tyvhAZxFbU` or `#mysql50#tmp.tyvhAZxFbU` - the only "valid" thing there from temp stuff should be `#tmp` folder10:37
noonedeadpunkI don't think it's critical issue though that prevents you from having healthy cluster.10:37
noonedeadpunkSo. what does mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_%';" says?10:38
moha7>> http://ix.io/4jGe10:43
moha7after removing tmp.*, noe there's only this one in the status output: `[ERROR] Invalid (old?) table or database name '#tmp'`10:46
noonedeadpunkmoha7: based on that galera is healthy10:46
moha7now*10:46
noonedeadpunkI will check this #tmp thingy in a while... That sounds like some bug, but not sure when ignore_db_dirs stopped working....10:47
noonedeadpunkso back to the neutron thing?10:48
noonedeadpunkYou said it's active and running now, but it's not having port 9696?10:49
noonedeadpunk*not listens10:49
moha7yes11:00
moha7Oh, no!11:01
moha7w811:01
moha7neutron.slice is active, but not running with same error like yesterday: 11:02
moha7still having this error: ValueError: :6642: bad peer name format11:04
moha7==> http://ix.io/4jGm11:05
noonedeadpunkwell, ovn is not my strength (yet). I'd summon jamesdenton for that. But he said it's related to northd11:07
moha7seems something else other than network-northd_hosts and network-gateway_hosts is missing in `user_variables.yml` and `openstack_user_config.yml`11:07
moha7noonedeadpunk: Thanks for your help11:07
moha7jamesdenton: Should be there an neutron-ovn-northd container in `lxc-ls` output?11:41
noonedeadpunkyes, there should11:54
noonedeadpunkdon't you have that?11:54
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove ANSIBLE_ACTION_PLUGINS override  https://review.opendev.org/c/openstack/openstack-ansible/+/82459512:51
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add tempest repos to required-repos  https://review.opendev.org/c/openstack/openstack-ansible/+/83428912:52
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Respect haproxy_bind_internal_lb_vip_* variables  https://review.opendev.org/c/openstack/openstack-ansible/+/85224312:52
jamesdentongood morning13:28
jamesdentonmoha7 i suspect that at a minimum, you're missing the "network-northd_hosts" group from openstack_user_config.yml. And if you want gateway node functionality, you'll want the "network-gateway_hosts" group, too. I'll work on fixing  the northd group piece, as ideally it wouldn't be necessary IMO13:30
cloudnullMerry Christmas / happy holidays all - I hope you’re all doing wonderful. 14:17
jamesdentoncloudnull! same to you14:33
noonedeadpunkthanks cloudnull, merry christmas and happy new year for you too!14:39
noonedeadpunkhuh, I somehow generated desired inventory using env.vars during summer and now I can't reproduce....14:40
moha7noonedeadpunk: no, there's not ant neutron-ovn-northd! --> lxc-ls: http://ix.io/4jdC14:48
moha7penstack_user_config.yml: http://ix.io/4jB414:49
moha7user_variables.yml: http://ix.io/4jGa14:49
moha7jamesdenton: ^14:49
noonedeadpunkwell, network-northd_hosts is defined there14:50
noonedeadpunkisn't northd in `./scripts/inventory-manage.py -g` ?14:53
jamesdentonhmm14:58
jamesdentonmoha7 this is zed/master, correct?14:59
moha7git branch --> * master15:00
moha7I got it by:15:01
moha7`git clone -b master https://opendev.org/openstack/openstack-ansible /opt/openstack-ansible && cd $_ && scripts/bootstrap-ansible.sh`15:01
jamesdentonok, thanks15:01
jamesdentonand this was a fresh deployment? 15:02
jamesdentonIs there anything in these groups? neutron_ovn_northd or neutron_ovn_northd_container15:03
jamesdentonyou can prob use "./scripts/inventory-manage.py -g" like noonedeadpunk pointed out, or jq: cat /etc/openstack_deploy/openstack_inventory.json | jq .neutron_ovn_northd_container; cat /etc/openstack_deploy/openstack_inventory.json | jq .neutron_ovn_northd15:04
moha7Yup, a fresh deplyment. `inventory-manage.py -g`: http://ix.io/4jHx15:11
moha7"neutron_ovn_northd_container": { "hosts": [] }15:14
moha7jamesdenton: those groups are empty!15:14
jamesdentonyes, very odd15:14
jamesdentonone sec15:14
noonedeadpunkoh.... My issue wiht inventory caused by this https://opendev.org/openstack/openstack-ansible/commit/c76a4d5dfe6c22b17064a11d7c60aa5b629ab43915:19
noonedeadpunk(I guess)15:19
noonedeadpunkdamn...15:19
jamesdenton:|15:21
noonedeadpunkjrosser: maybe you do recall this... I do how we discussed that back then... So basically my problem is related to AZ implementation15:22
jamesdentonmoha7 i am attempting to recreate your issue, takes a little bit of time to stand it up15:22
moha7Thanks15:23
noonedeadpunkSo I had config like that https://paste.openstack.org/show/b6R6axfFPZMsrGgU7jo4/15:24
noonedeadpunkand in inventory had smth like https://paste.openstack.org/show/bl11c9SwQgDS2TNBp01F/15:26
noonedeadpunkBut now I'm not understanding how I can properly add all host containers15:27
noonedeadpunkhm...15:31
noonedeadpunkjsut reverting code doesn't help...15:52
noonedeadpunkdamn15:52
noonedeadpunkI'm soooo tired of figuring out the way... Probably I should have focused on implementing a key or smth that would allow do what I want15:52
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Update tox.ini  https://review.opendev.org/c/openstack/openstack-ansible/+/86868015:56
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Update tox.ini  https://review.opendev.org/c/openstack/openstack-ansible/+/86868015:57
jamesdentonmoha7 i'm not able to recreate your issue so far. Can you confirm the contents of /etc/openstack_deploy/env.d and conf.d?16:04
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Update tox.ini  https://review.opendev.org/c/openstack/openstack-ansible/+/86868016:18
jamesdentonnoonedeadpunk is centos still borked?16:18
noonedeadpunkjust fixed16:18
noonedeadpunklike hour ago or smth16:19
jamesdentonoh nice16:19
noonedeadpunkBut now tox 4 is borked :D16:19
jamesdentonoh no16:19
noonedeadpunk(like docs, renos, functional leftovers)16:19
noonedeadpunkI'm going to check through and propose patches16:19
jrosser i am pretty sure that the changes i did to the inventory code were right16:20
noonedeadpunkI'm not saying they're not... Just trying to realize what I did during summer that I cna't reproduce anymore16:24
noonedeadpunkBasically what I want - is when host added to another group defined in env.d - it's containers should appear there as well16:25
noonedeadpunkSo given I define az1_all / az1_hosts / az1_containers and define like comput01, control01, net01 to az1_hosts, I want all containers to appear in az1_containers this way or another16:26
noonedeadpunkAnd I somehow managed that, but control01_host-containers were children of az1_containers that I assume made it work...16:26
noonedeadpunk(though reverting code didn't help to be fair)16:27
noonedeadpunkBut I wonder if it's simply not possible with code we have today or I just forgot how to do that, despite have old inventory file along with env.d.....16:28
noonedeadpunkAnyway16:28
jrosserwell, something i think i have seen is that the output of inventory-manage might not be as complete as what ansible actually sees16:28
jrosserbut thats possibly something quite different again16:28
noonedeadpunkYeah, but I'm checking json directly16:28
noonedeadpunk(as I'm aware about inventory-manage already)16:29
noonedeadpunkSo I'm either brainfarting or it's simply not possible in an easy wya16:30
noonedeadpunkBut I can't get on with that as I see old generated inventory that had this working in the sandbox...16:31
jrosserwell - i have done this sort of thing only with compute nodes to make new groups16:34
jrosserlike "with GPU" or whatever16:34
jrosserso never with anything on the control plane16:35
*** dviroel_ is now known as dviroel16:38
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-uwsgi master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/86868316:53
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-pki master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-role-pki/+/86868416:53
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/86868516:53
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-qdrouterd master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-role-qdrouterd/+/86868616:53
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_mount master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/86868716:53
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/86868816:53
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/86868916:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-apt_package_pinning master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/86869016:53
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-hardening master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/ansible-hardening/+/86869116:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/86869216:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/86869316:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_container_create master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/86869416:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-memcached_server master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/86869516:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/86869616:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/86869716:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/86869816:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_openrc master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/86869916:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/86870016:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/86870216:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/86870116:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/86870316:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/86870416:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/86870516:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/86870616:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/86870716:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/86870816:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/86870916:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-repo_server master: Update tox.ini to work with 4.0  https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/86871016:53
BobZAnnapolisfolks, running some RefStack tests - getting this error msg for a handful of tests "Missing Identity Admin API credentials in configuration" - nothing in the RefStack documentation (that is obvious to this newbie to RefStack) that says where/how to set these up, tia16:56
jrosserBobZAnnapolis: i would start here https://codesearch.opendev.org/?q=Missing%20Identity%20Admin%20API%20credentials%20in%20configuration16:59
jrosserwhich takes you here https://opendev.org/openstack/tempest/src/branch/master/tempest/test.py#L282-L28817:00
noonedeadpunkyeah, GPU is easy17:01
noonedeadpunkEventually, Damian  has found that for GPU and stuff you even don't need to define custom env.d17:01
noonedeadpunkas adding anyting to openstack_user_config that ends with `-hosts` will result in creating such group in inventory17:02
noonedeadpunkIt won't create containers part - but for computes that's not needed17:02
jrosseryes we do that a lot with -hosts things17:02
jrosserit is not obvious17:02
noonedeadpunkI learned abut that literally last week17:03
noonedeadpunkAs was creating env.d for everything17:03
noonedeadpunkI didn't dare to try tbh 17:04
BobZAnnapolisjrosser: thumbs up, thanks - you beat me to it :-) i was headed into the code as well, there are 'admin' parameters in the etc/accounts.yaml file but looks like not enough - thank you17:07
moha7`ls /etc/openstack_deploy/env.d` --> aio_metal.yml.example  calico.yml.example  cinder-volume.yml.container.example  extra_container.yml.example  neutron.yml  nova.yml17:29
moha7conf.d/ --> http://ix.io/4jI8 --> all files have .aio or .example --> then there's not any actove config file in ../conf.d/.17:32
moha7jamesdenton: ^17:32
jamesdenton   can you show be env.d/neutron.yml?17:33
jamesdenton*me17:33
jamesdentonand nova17:33
moha7I copied contents for them from this page: https://satishdotpatel.github.io/openstack-ansible-multinode-ovn/17:34
jamesdentonahh ok, you won't need them anymore17:34
jamesdentonsince OVN is default for OSA now, those env.d overrides are not necessary and prob not compatible17:35
moha7Ah, noonedeadpunk mentioned yesterday, but I think he means that they are ignored. My bad17:35
jamesdentonso if you rename neutron.yml and nova.yml or move them, try regenerating the inventory with /opt/openstack-ansible/inventory/dynamic_inventory.py17:35
moha7jamesdenton: Now the group `neutron_ovn_northd_container` has values in that *json* file. Going to deploy from scratch: `/opt/openstack-ansible/playbooks/setup-everything.yml` after destroying current LXCes.18:01
jamesdentonno need to destory current containers if thats the only change18:01
jamesdentonas long as you didn't make changes to the inventory (other than addition of northd) you should be OK18:02
jamesdentonyou would want to rerun setup-hosts.yml and you can use --limit <northd container>, to get it setup, and then rerun neutron plays. Or, setup-everything is fine ,too18:02
moha7Thanks for this time-saving tip18:04
jamesdentonsure, just double check after running setup hosts that the container shows up. it should18:06
moha7Am I using the command in wrong way: `openstack-ansible /opt/openstack-ansible/playbooks/setup-hosts.yml --limit neutron_ovn_northd_container` because it's failed with: http://ix.io/4jIm -->  'dict object' has no attribute 'distribution'. The error appears to be in '/etc/ansible/roles/lxc_container_create/tasks/lxc_container_create_dir.yml'18:17
jrossermoha7: that looks like you are missing facts gathering for the physical hosts18:28
jrosserthey are only valid in the cache for (I think) 24 hours18:29
moha7One or two hours have passed since 24 hours. Should I install from the beginning, or is there a way to generate the facts twice?18:34
jrossernone of this ever really involves starting again from the beginning, only for massive mess18:35
jrosserperhaps expand the limit to include infra_all or some other group including the physical hosts18:37
moha7Well, I run it with `--limit network-infra_all`18:44
jamesdentonmoha7 is it running now? sorry, stepped away19:21
moha7No, by network-infra_all passed that previous error, but failed with another one: fatal: [ct1_neutron_server_container-56963a6e -> {{ _oslodb_setup_host }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}19:23
jamesdentonok, just try without the limit then19:23
jamesdentonopenstack-ansible setup-hosts.yml19:23
moha7+119:23
jamesdentonsorry19:24
jamesdentonsome of these need --limit localhost,<group or host>. who knows19:24
ElDuderinoanyone know of a good practical tuning guide for rabbitmq as it relates to openstack-ansible based HA installs? O19:55
ElDuderinoI'm watching https://www.openstack.org/videos/summits/austin-2016/tuning-rabbitmq-at-large-scale-cloud and am trying to determine how relevant it still is (2016 video) although we're still on Rocky so some of it might be relevant.19:56
*** dviroel is now known as dviroel|out20:17
ElDuderinoWe have a 3 node rabbit cluster and run it on the controllers in an lxc container. I'm trying to figure out if the OSA community has a 'canonical' way for any of us admins/installers that use OSA at scale to reference20:20
ElDuderinoIt appears that Mirantis has this approach, perhaps we do also? https://docs.mirantis.com/mcp/q4-18/mcp-deployment-guide/advanced-config/tune-rabbitmq-perf.html20:22
opendevreviewMerged openstack/openstack-ansible master: Update tox.ini  https://review.opendev.org/c/openstack/openstack-ansible/+/86868023:38

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!