| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Set ovn hostname using nodename facts https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803988 | 04:18 |
|---|---|---|
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Fix OVN metadata protocol to point https https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803989 | 04:18 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add support of OVN for CentOS-8-Stream https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803987 | 04:18 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Use list of cluster member for ovn ml2 agent to directly connect https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803990 | 04:18 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Set ovn hostname using nodename facts https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803988 | 04:34 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Fix OVN metadata protocol to point https https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803989 | 04:34 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Add support of OVN for CentOS-8-Stream https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803987 | 04:34 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Use list of cluster member for ovn ml2 agent to directly connect https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803990 | 04:34 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Replace centos-8.x jobs with centos-8-stream https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/804036 | 04:34 |
| *** rpittau|afk is now known as rpittau | 06:15 | |
| opendevreview | Marcus Klein proposed openstack/openstack-ansible stable/ussuri: Backported fixes for healthcheck playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/804044 | 07:22 |
| opendevreview | Marcus Klein proposed openstack/openstack-ansible stable/ussuri: Backported fixes for healthcheck playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/803926 | 07:25 |
| opendevreview | Marcus Klein proposed openstack/openstack-ansible stable/ussuri: Backported fixes for healthcheck playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/803926 | 07:28 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/wallaby: Fix shibboleth compatibility for ubuntu 18.04 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/803552 | 11:24 |
| kleini | ansible-hardening started to fail for me in production with U release. configuration for auditd.conf is somehow missing although it is defined in the vars directory of the role. Do you have any hints how to debug/resolve this? | 11:34 |
| spatel | jrosser morning! | 12:12 |
| spatel | This is what i did to stack them, hope this is what you were talking about yesterday - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/804036 | 12:12 |
| opendevreview | Andrew Bonney proposed openstack/ansible-role-python_venv_build master: Add distro/arch to requirements file path https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/801738 | 12:22 |
| opendevreview | Andrew Bonney proposed openstack/ansible-role-python_venv_build master: Add variables to set permissions for created files https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/804073 | 12:22 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Fix permissions for files created on repo server https://review.opendev.org/c/openstack/openstack-ansible/+/804077 | 12:28 |
| opendevreview | Merged openstack/openstack-ansible stable/victoria: haproxy: decrease check interval for letsencrypt back ends https://review.opendev.org/c/openstack/openstack-ansible/+/803956 | 12:54 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: haproxy: decrease check interval for letsencrypt back ends https://review.opendev.org/c/openstack/openstack-ansible/+/803955 | 12:54 |
| spatel | Does anyone upgrade from V -> W because i am seeing git clone repo task is failing at here - failed: [localhost] (item={'name': 'ceph-ansible', 'scm': 'git', 'src': 'https://github.com/ceph/ceph-ansible' | 13:10 |
| spatel | I am assuming we don't need this task because it always failed - TASK [Clone git repos (parallel)] | 13:10 |
| jrosser | spatel: https://github.com/openstack/openstack-ansible/commit/82592edf6ff3b8c7e2825aa2e5977791f4cb2832 | 13:21 |
| spatel | jrosser beauty! | 13:30 |
| spatel | so this is pending to merge ? | 13:30 |
| spatel | anything red on screen freak me out during upgrade :) | 13:31 |
| spatel | jrosser got encounter here and i don't have pki file structure in my /etc/openstack_deploy/pki | 13:42 |
| spatel | failed: [os-lab-infra-1] (item={'name': 'ExampleCorpRoot'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ExampleCorpRoot"}, "msg": "file not found: /etc/openstack_deploy/pki/roots/ExampleCorpRoot/certs/ExampleCorpRoot.crt"} | 13:42 |
| spatel | This is during setup-host playbook TASK [pki : Slurp up CA certificates from pki setup host (localhost)] | 13:44 |
| jrosser | spatel: those dirs should be on the deploy host | 13:49 |
| spatel | i am on deploy host and nothing there | 13:50 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-rabbitmq_server master: Fix erlang upgrades for rabbitmq external_repo installs https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/803711 | 13:50 |
| spatel | '/etc/openstack_deploy/pki': No such file or directory | 13:51 |
| spatel | I am following this document for my V -> W upgrade - https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-upgrades.html | 13:52 |
| jrosser | this has run? https://github.com/openstack/openstack-ansible/blob/master/playbooks/setup-hosts.yml#L16 | 13:55 |
| spatel | i have executed setup-hosts.yml and encounter that error so assuming it ran but let me run individual playbook | 13:57 |
| spatel | very interesting.. that playbook created directory | 13:58 |
| spatel | why not this playbook run? - openstack-ansible setup-hosts.yml --limit '!galera_all:!rabbitmq_all' | 13:58 |
| spatel | look like using --limit not allowing to run that playbook | 14:02 |
| spatel | we should add extra step in that case to just run that playbook for V -> W upgrade to setup pki | 14:03 |
| jrosser | you have excluded localhost with that limit | 14:19 |
| jrosser | oh | 14:20 |
| jrosser | spatel: https://zuul.opendev.org/t/openstack/build/10b59ed57bd14b848b11f94eded6805f/log/job-output.txt#14362-14381 | 14:43 |
| jrosser | you know you linked to the victoria upgrade instructions? | 14:44 |
| spatel | hold on in meeting | 14:54 |
| spatel | sorry i am back now | 15:01 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:01 |
| opendevmeet | Meeting started Tue Aug 10 15:01:51 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:01 |
| noonedeadpunk | #topic rollcall | 15:01 |
| noonedeadpunk | \o/ | 15:02 |
| spatel | noonedeadpunk is back that is big update :) | 15:02 |
| noonedeadpunk | well, unfortunatelly I'm not back for long :( | 15:02 |
| noonedeadpunk | have tons of family business this week as well. I really hoped I would be online more this days, but this doesn't worked out :( | 15:03 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/803097 looks really bad. I think it's because we haven't backported neutron patch... | 15:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Bump OpenStack-Ansible Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/803097 | 15:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Bump OpenStack-Ansible Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/803097 | 15:04 |
| noonedeadpunk | #topic office hours | 15:05 |
| noonedeadpunk | so, what's happening forum? | 15:07 |
| noonedeadpunk | I'll be fully back only next week( | 15:07 |
| jrosser | o/ hello | 15:07 |
| jrosser | i think we need this and it's backport https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/798960 | 15:08 |
| noonedeadpunk | but for this I guess we need neutron patch spatel working on? | 15:09 |
| jrosser | for master yes | 15:09 |
| jrosser | whatever unblocks os_neutron needs to go first | 15:09 |
| spatel | jrosser i have tried to stack my all neutron patches here, may need your input https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/804036/1 | 15:10 |
| noonedeadpunk | well, it's cherry-pick is not in a good shape either... https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/798881 | 15:10 |
| jrosser | no, there is so much to fix in os_neutron sadly | 15:10 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803987 looks like a light in the end | 15:11 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-rabbitmq_server master: Fix erlang upgrades for rabbitmq external_repo installs https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/803711 | 15:12 |
| noonedeadpunk | So I'd say we can set jobs as NV in https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/804036/1 ? | 15:12 |
| noonedeadpunk | and do bunch of rebases? | 15:12 |
| spatel | noonedeadpunk yes that patch is part os multiple stack and i need to understand from you guys why other patches failed and solve their inter dependency when they are in chain | 15:13 |
| jrosser | spatel: the first patch is tested on it's own | 15:13 |
| jrosser | and is not enough to make the tests pas | 15:13 |
| noonedeadpunk | yeah ^ | 15:13 |
| jrosser | the first and second then get tested together | 15:13 |
| jrosser | then first / second / third, and so on | 15:14 |
| jrosser | lets just do this now? | 15:14 |
| noonedeadpunk | so let's set failing jobs as NV, merge all that and revert NV patch at the end of the day | 15:14 |
| noonedeadpunk | But backporting that would be meh... | 15:14 |
| jrosser | then work out what to do with W too | 15:14 |
| jrosser | yeah, as there is centos and centos-stream there | 15:15 |
| jrosser | and we kind of need to decide what we do/dont support | 15:15 |
| noonedeadpunk | W is upgrade for Centos to Stream, right? | 15:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Replace centos-8.x jobs with centos-8-stream https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/804036 | 15:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Set ovn hostname using nodename facts https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803988 | 15:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Fix OVN metadata protocol to point https https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803989 | 15:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Add support of OVN for CentOS-8-Stream https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803987 | 15:17 |
| spatel | centos-8.4 has broken OVS job, question is should we fix that or drop CI job and continue working with stream ? | 15:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Use list of cluster member for ovn ml2 agent to directly connect https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803990 | 15:17 |
| noonedeadpunk | On W I'd say it would be great to fix that, but actually I kind of missed the reason at the first place. Smth related to some repos? | 15:18 |
| jrosser | its several things, we need to fix ubuntu OVN on W regardless | 15:18 |
| noonedeadpunk | yes, totally | 15:19 |
| jrosser | we can decide if we add centos-8-stream OVN to W or not - or we say that is a feature from X onward | 15:19 |
| spatel | Yes OVN repo is broken and this patch can fix it so i cut one patch to fix it but question is should we keep going with this kind of hack or drop it - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802402 | 15:19 |
| jrosser | and then there is the old centos-8 OVS jobs | 15:19 |
| jrosser | it feels like this kind of delays 23.1.0 | 15:20 |
| noonedeadpunk | spatel: I'd say this patch would be relevant only for W and V? | 15:20 |
| noonedeadpunk | No reason to merge that to master anyway | 15:21 |
| jrosser | i think this was from before we dropped the old centos jobs from master | 15:21 |
| spatel | i believe on W because i got issue with that, i didn't try on V yet | 15:21 |
| noonedeadpunk | yeah | 15:21 |
| spatel | CentOS community moved their repo to stream and that is why its failing to find latest OVS version for centos-8.4 | 15:22 |
| jrosser | lets deal with one thing at a time :/ | 15:23 |
| jrosser | too many moving parts | 15:23 |
| spatel | +1 | 15:23 |
| noonedeadpunk | Ok, another thing. I was wondering if you have any thoughts regarding pain point, that could be selected as next community goals? | 15:24 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/pain-point-elimination | 15:24 |
| spatel | Could you guy give review and bump to this patch, it will fix broken centos-8-stream distro deployment - https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/803475 | 15:27 |
| jrosser | spatel: did you see i left a comment on the depends-on? | 15:29 |
| spatel | Just replied - var directory doesn't exist on chroot environment so when i tried /etc/yum/vars it failed to create var directory. may be we need to create first /etc/yum/vars directory first and then copy contents | 15:29 |
| spatel | chroot doesn't have /etc/yum directory also | 15:30 |
| spatel | for look like first we need to create empty /etc/yum and /etc/yum/var | 15:30 |
| *** owalsh_ is now known as owalsh | 15:31 | |
| noonedeadpunk | I commented mentioned patch as well | 15:31 |
| noonedeadpunk | Also I'd love to ask for some reviews of https://review.opendev.org/q/topic:%22bump_osa%22+(status:open) | 15:31 |
| noonedeadpunk | As we didn't have a release for quite a while | 15:32 |
| spatel | noonedeadpunk - nova has dependency requirement of some foo packages and that is part of this repo, without this repo it won't let you install nova components | 15:34 |
| spatel | that is why distro installation failing | 15:34 |
| jrosser | oh yes this is all nasty | 15:34 |
| jrosser | its actually errors on the compute nodes that this fixe | 15:34 |
| noonedeadpunk | doh | 15:34 |
| noonedeadpunk | that is soooooo stupid.... | 15:34 |
| spatel | Yes.. | 15:34 |
| noonedeadpunk | good way to push ppl out of lxb... | 15:35 |
| spatel | RPMs stuff someone may be be rebuildrequired: foo in spec file and that is why we endup here | 15:35 |
| noonedeadpunk | ok, thanks for explanation | 15:35 |
| spatel | i think its totally safe for add repo its not going to hurt so far | 15:36 |
| noonedeadpunk | considering it's for distro setup only - I don't care _that_ much | 15:37 |
| spatel | totally! that patch exclusively adding repo, no matter source or distro | 15:39 |
| noonedeadpunk | uh, that's worse. | 15:39 |
| noonedeadpunk | as eventually we're trying to control packages that are to be installed from repos that not present by default | 15:40 |
| spatel | we can put condition if that is better way to handle it | 15:40 |
| noonedeadpunk | that's why we drop epel-release there iirc | 15:40 |
| noonedeadpunk | as well as centos-release-storage-common | 15:41 |
| spatel | hmm, pervasively i created this patch to solve distro issue but then abandon it - https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/803368 | 15:41 |
| jrosser | well nova role doesnt run on network nodes though? so we end up with wierd differences in OVS repo between hosts | 15:42 |
| spatel | if we are concern then lets put condition == distro and push that patch out after review | 15:44 |
| noonedeadpunk | yeah | 15:45 |
| noonedeadpunk | we can really do like (service_install_method | default('source') == 'distro') | ternary('present', 'absent') | 15:45 |
| spatel | ok | 15:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-vault master: Initial commit to Vault role https://review.opendev.org/c/openstack/ansible-role-vault/+/800792 | 15:48 |
| noonedeadpunk | btw, have any folks were coming with questions regarding ^ ? | 15:53 |
| noonedeadpunk | or they just PMing me regardless.... | 15:53 |
| noonedeadpunk | ok, whatever actually:) | 15:58 |
| noonedeadpunk | #endmeeting | 15:58 |
| opendevmeet | Meeting ended Tue Aug 10 15:58:15 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-10-15.01.html | 15:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-10-15.01.txt | 15:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-10-15.01.log.html | 15:58 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Implement RabbitMQ cluster rolling restart feature https://review.opendev.org/c/openstack/openstack-ansible/+/804109 | 16:19 |
| *** rpittau is now known as rpittau|afk | 16:31 | |
| spatel | quick question for disk cache mode you guys using for good disk performance, i am seeing this on my virsh dumpxml <instance> | 16:51 |
| spatel | <driver name='qemu' type='qcow2' cache='none' discard='ignore'/> | 16:51 |
| spatel | jrosser i got this error - https://paste.opendev.org/show/807985/ | 16:52 |
| spatel | during this playbook run - openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' | 16:53 |
| jrosser | restart galera by hand in the container | 16:53 |
| spatel | hmm | 16:54 |
| spatel | then run the playbook? | 16:55 |
| jrosser | could be this https://jira.mariadb.org/browse/MDEV-24829 | 16:55 |
| spatel | what if i don't want to upgrade galera if this version is buggy? | 16:56 |
| spatel | now it failed here - https://paste.opendev.org/show/807986/ should i use -e galera_ignore_cluster_state=true ? | 16:57 |
| spatel | why don't we have default instead of saying you don't have? | 16:58 |
| jrosser | ? | 16:59 |
| spatel | I am saying i never set wsrep_cluster_name as far as i know then why its saying its different now ? | 17:01 |
| spatel | i didn't check play just asking question, may be this is recent change which i am not aware | 17:01 |
| jrosser | no it's always been like this | 17:03 |
| jrosser | each database instance has a cluster name which is defined in the defaults | 17:03 |
| spatel | then why did i encounter this error in upgrade but not first installation | 17:04 |
| jrosser | the error is that the db you just installed doesnt have the right cluster name, which is normally set during bootstrapping the db cluster | 17:04 |
| jrosser | is the galera instance you just upgraded a member of the cluster? | 17:05 |
| spatel | no its my LAB which has single controller or infra node but multiple compute nodes | 17:06 |
| spatel | in short single infra node | 17:07 |
| spatel | i have added wsrep_cluster_name = "openstack_galera_cluster" in user_variables.yml and re-running same play.. | 17:11 |
| jrosser | i don't know - there are upgrade jobs running for all CI, so this does work | 17:14 |
| spatel | still getting same error let me understand what is going on and debug | 17:16 |
| spatel | jrosser issue was - SQL Error 1040: Too Many Connection | 17:23 |
| spatel | in my.cnf has max_connections = 200 | 17:24 |
| spatel | jrosser this is very odd now i don't have data in mysql - https://paste.opendev.org/show/807988/ | 17:31 |
| spatel | nevermind.. look like root account can't see other databases | 17:32 |
| spatel | root is powerless account now | 17:32 |
| spatel | i can see data after i use mysql -u nova -p | 17:33 |
| spatel | now i stuck here - https://paste.opendev.org/show/807989/ | 17:40 |
| spatel | debugging.. | 17:40 |
| jrosser | spatel: this https://jira.mariadb.org/browse/MDEV-25030 | 17:56 |
| jrosser | there is now an 'admin' user instead of root | 17:57 |
| opendevreview | Merged openstack/openstack-ansible stable/ussuri: Bump OpenStack-Ansible Ussuri https://review.opendev.org/c/openstack/openstack-ansible/+/803099 | 17:58 |
| spatel | jrosser what is the point to create admin instead or root? security? | 18:00 |
| jrosser | we previously made changes to the built-in root user in the db, which is kind of unsupported | 18:01 |
| jrosser | you're supposed to make another user and give that the rights you want | 18:01 |
| spatel | oh! | 18:01 |
| jrosser | from the release notes `Galera privileged username has changed from root to admin. Old ‘root’@’%’ user can be removed after upgrade process.` | 18:02 |
| spatel | any idea about this play - https://paste.opendev.org/show/807989/ going to turn on logging to see what is going on | 18:02 |
| jrosser | i don't know | 18:03 |
| spatel | jrosser thank you! didn't read release notes yet :) | 18:03 |
| jrosser | if your database was in trouble before the upgrade with too many connections, then i'm not sure whats happening now | 18:03 |
| spatel | i have bump connection to 1000 and now i am not seeing too many connection issue | 18:04 |
| opendevreview | Merged openstack/openstack-ansible stable/victoria: Bump OpenStack-Ansible Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/803098 | 18:05 |
| jrosser | spatel: right, but your upgrade did go strange? | 18:06 |
| spatel | only has issue with mysql but after restarting i didn't see issue | 18:06 |
| spatel | i am thinking to downgrade mysql version to original this one might has some unknown bugs | 18:07 |
| jrosser | i'll be sticking with the W released version | 18:09 |
| jrosser | these point upgrades of galera have happened on every OSA major releas | 18:09 |
| jrosser | we've just been very unlucky with a bug in both 10.5.8 and 10.5.9 | 18:10 |
| jrosser | if you stick with 10.5.8 you will stick with the "does not start up every so often" mutex error which is worse than the broken grant stuff, which there is a workaround for in the code | 18:10 |
| spatel | openstack doesn't care about mysql version right? no dependency at all | 18:11 |
| spatel | before upgrade i was running 10.5.6 and it was super stable | 18:12 |
| spatel | do you think 10.5.6 also has issue ? | 18:15 |
| spatel | jrosser very strange error here - https://paste.opendev.org/show/807991/ | 18:18 |
| spatel | did you see that before? | 18:18 |
| jrosser | i guess /openstack/venvs/utility-23.0.0/bin/python: not found | 18:20 |
| jrosser | ^ is that there? | 18:21 |
| spatel | '/openstack/venvs/utility-23.0.0/bin/python': No such file or directory | 18:21 |
| spatel | even i don't have this dir also /openstack/venvs/utility-23.0.0 | 18:22 |
| jrosser | so thats the problem, you're missing the utility venv, or the path is wrong..... i just read the errors :) | 18:22 |
| spatel | did i miss some steps? | 18:23 |
| spatel | i am just following our official doc for upgrade | 18:23 |
| jrosser | do you have anything in /openstack/venvs/ on 10.65.6.17 ? | 18:23 |
| spatel | https://paste.opendev.org/show/807992/ | 18:24 |
| spatel | let me check on that specific host | 18:25 |
| spatel | jrosser i have this - https://paste.opendev.org/show/807993/ | 18:25 |
| spatel | i don't have 23.0.0 | 18:26 |
| jrosser | right, so you have some kind of mixture of checking out 23.0.0 tag and then checking out stable/wallaby (or some other point) and not doing bootstrap-ansible | 18:26 |
| jrosser | or something like that | 18:27 |
| jrosser | you've done part of the deployment from 23.0.0 | 18:27 |
| jrosser | oh not 23.x | 18:28 |
| spatel | Let me tell you what i did | 18:28 |
| jrosser | buy you know what i mean | 18:28 |
| spatel | first i upgrade my V with minor version using this checkout ( git checkout tags/22.2.0 -b stable/victoria ) | 18:28 |
| spatel | once i upgraded to 22.2.0 then i did checkout | 18:28 |
| jrosser | right, and we see the utility venv for that utility-22.2.0 | 18:29 |
| spatel | git checkout tags/23.0.0 -b stable/wallaby | 18:29 |
| spatel | based on this doc i did run bootstrap - https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-upgrades.html | 18:29 |
| spatel | ${SCRIPTS_PATH}/bootstrap-ansible.sh | 18:29 |
| spatel | I am doing manual upgrade method instead of script | 18:30 |
| jrosser | one of the first things in setup-infrastructure.yml should have been to build the utility container and it's venv | 18:31 |
| jrosser | whats happened to that? | 18:31 |
| spatel | hmm | 18:31 |
| jrosser | oh | 18:31 |
| jrosser | well it's not first is it https://github.com/openstack/openstack-ansible/blob/master/playbooks/setup-infrastructure.yml#L23 | 18:31 |
| spatel | let me see.. i think i know what went wrong, when my mysql failed i did restart but then didn't run that playbook again | 18:32 |
| jrosser | ok, like i say the same steps are done in the upgrade CI jobs | 18:32 |
| spatel | re-running - openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' | 18:33 |
| spatel | so there is no other solution to fix this issue? TASK [galera_server : Create galera users] | 18:42 |
| spatel | playbook always throwing error here and as you said restart mysql | 18:43 |
| spatel | don't we need that play to create users etc..? | 18:45 |
| spatel | just curious why its not failing in CI job | 18:55 |
| admin1 | spatel you can do like vvvv and also edit the playbook to unhide the exact message | 18:56 |
| spatel | all i can see that play trying to create users using root@localhost account and that failing because we don't use root account | 18:57 |
| admin1 | it does set the root password to what is in the user_secrets | 18:58 |
| spatel | https://paste.opendev.org/show/807994/ | 18:58 |
| spatel | jrosser mentioned earlier osa now using admin account for mysql instead of root | 18:59 |
| spatel | jrosser should i leave this failure here and just move to next playbook, if this play doesn't matter for upgrade | 19:00 |
| admin1 | it looks like its trying to login as root and doing something, ( as in a newly installed mysql ) but failing due to that | 19:02 |
| admin1 | if its new, dropping galera containers and redoing it might help | 19:03 |
| spatel | This is V -> W Upgrade i am testing so it has data | 19:03 |
| admin1 | oh .. | 19:03 |
| admin1 | in that case, what i would do is check the secrets and add those accounts manually in the db | 19:04 |
| admin1 | so that when the playbooks try to run, they don't get an issue due to access not being proper | 19:04 |
| b1tsh1ft3r | Hey all, removed a compute node from the cluster because hardware failed. Looks as if after running the playbook to remove the node and then re-running the play to re-provision the hardware and bring it back into the cluster it seems launching instances on it or migrating to it doesnt work at all. I cant seem to get any instances to launch on the new node. Im also not seeing any sort of errors in the log | 19:05 |
| b1tsh1ft3r | s or ui! | 19:05 |
| admin1 | b1tsh1ft3r, check the nova and libvirt logs | 19:06 |
| admin1 | they will tell why it fails | 19:06 |
| admin1 | or the scheduler if you are not even getting it to schedule to that node | 19:06 |
| spatel | i can tell you problem is part of placement :) | 19:07 |
| spatel | b1tsh1ft3r ^ | 19:07 |
| spatel | you may need to delete node from resource and restart nova-compute which will re-generate new UUID | 19:07 |
| spatel | i had that kind of issue many time when i remove and add compute nodes | 19:08 |
| b1tsh1ft3r | so, ive got through the process and shut down nova-compute and neutron-linuxbridge-agent, run the node removal playbook (appears to remove it from the hypervisors list in the ui and the back end service list). Is that not the same as what you just mentioned?! | 19:11 |
| spatel | b1tsh1ft3r check this - openstack resource provider list | 19:13 |
| spatel | if you see your compute delete from resource provider | 19:14 |
| spatel | then go back to compute node and restart service which will re-register itself with new UUID | 19:14 |
| b1tsh1ft3r | gotcha. let me give it a try | 19:16 |
| admin1 | is deleting the lxc containers enough to start fresh ? | 19:17 |
| admin1 | do i need to delete anything else | 19:17 |
| admin1 | things like clearning out dnsmasq etc entries .. | 19:19 |
| spatel | cleaning up lxc is enough | 19:21 |
| b1tsh1ft3r | spatel: Looks like removing the resources provider and re-starting nova resulting in the same uuid being generated | 19:21 |
| admin1 | it will be in a new id i think | 19:21 |
| spatel | generation vs uuid | 19:22 |
| spatel | both are same? | 19:22 |
| b1tsh1ft3r | id is new it looks like. uuid is the same | 19:22 |
| spatel | that is good | 19:23 |
| spatel | just build your vm and you should be good | 19:23 |
| admin1 | does osa support ironic ? | 19:29 |
| admin1 | how is it checked in ci ? | 19:29 |
| spatel | i don't think we have CI job for ironic for OSA | 19:34 |
| spatel | it does has role but i never tested because it required special need like PXE vlan and ilo access etc.. | 19:35 |
| spatel | b1tsh1ft3r does that solve your problem? | 19:47 |
| admin1 | my lxc containes are not getting ip address . not on eth0 ( dnsmasq is running ) and also not on the eth1 | 19:49 |
| admin1 | the setup-hosts just passes all steps and fails when trying to download packages, coz there is no ip address | 19:49 |
| admin1 | if i go inside any container and do ip link, i dont see eth0 or eth1, though I see that on the lxc configs | 19:50 |
| b1tsh1ft3r | spatel: no, i think it might just be easier to give the compute node a new name and just run the plays again and bring it in as a new node tbh. Is there an easy way to regenerate the inventory ?! | 19:50 |
| admin1 | a reboot in one fixed it . not sure what had happened | 19:52 |
| spatel | b1tsh1ft3r what is your log saying | 19:53 |
| spatel | compute logs and then check placement logs | 19:53 |
| spatel | also check openstack compute service list and see if any duplication there | 19:55 |
| spatel | This mysql 10.5.9 is disaster :( | 20:02 |
| spatel | not letting me set permission causing not able to move forward | 20:02 |
| admin1 | spatel, you can put it in safe mode and play with permissions | 20:19 |
| spatel | i am going to upgrade 10.6.4 stable release.. i don't want hand fix in production | 20:21 |
| admin1 | i thought you were in staging | 20:23 |
| admin1 | where some stuff like this is allowed | 20:23 |
| spatel | my lab is close to production | 20:23 |
| admin1 | ok | 20:23 |
| spatel | good luck 10.6.4 also throwing same error :) | 20:27 |
| spatel | i gotta go i will look into later but look like upgrade isn't that easy (specially with mysql() | 20:27 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Bump OpenStack-Ansible Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/803097 | 20:39 |
| jrosser | spatel: your original galera playbook failed becasue you had too many connections error | 21:23 |
| jrosser | that means it did not run the handlers at the end of the play which fix up the 10.5.9 problems | 21:24 |
| jrosser | this https://github.com/openstack/openstack-ansible-galera_server/blob/master/handlers/galera_upgrade_hook.yml | 21:24 |
| opendevreview | Merged openstack/openstack-ansible-os_murano stable/train: Remove Debian jobs https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/803995 | 23:00 |
| opendevreview | Merged openstack/openstack-ansible-os_murano stable/ussuri: Remove Debian jobs https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/803996 | 23:00 |
| opendevreview | Merged openstack/openstack-ansible-os_murano stable/victoria: Remove Debian jobs https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/803997 | 23:01 |
| opendevreview | Merged openstack/openstack-ansible-os_murano stable/wallaby: Remove Debian jobs https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/803998 | 23:01 |
| opendevreview | Merged openstack/openstack-ansible-os_murano master: Remove Debian jobs https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/803999 | 23:02 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!