| karni | Hi | 05:39 |
|---|---|---|
| hamidlotfi_ | mornings | 06:23 |
| admin1 | mornings .. i am getting an error on missing haproxy_service_name variable .. which is strange .. its on the nova playbook -> https://gist.githubusercontent.com/a1git/b0674552c3cff56d4615e8f22291d125/raw/3d267a52c94e749f69eaf2015e7d84b9596523b5/gistfile1.txt | 07:27 |
| noonedeadpunk | admin1: is that antelope? | 07:51 |
| noonedeadpunk | as then https://review.opendev.org/c/openstack/openstack-ansible/+/889023 can be a fix | 07:52 |
| admin1 | noonedeadpunk it is Antelope .. funny thing is on the first install , there were no issues .. nothing failed .. so setup-openstack went on smooth .. .. but it started giving issues on 2nd run | 08:16 |
| admin1 | i will cherry-pick and retry | 08:17 |
| threelittlebirds | hey guys, trying to setup latest version 2023.01 in virtualized lab-env (3 infra, 5 computes) - throwing error an error while running setup-openstack playbook | 08:23 |
| threelittlebirds | https://paste.openstack.org/show/b0JutziwgLKvh199P5wn/ | 08:23 |
| noonedeadpunk | admin1: it's because it's facts caching related | 08:27 |
| noonedeadpunk | on first run you have facts gathered and valid, then next day they're invalidated and you heat issue | 08:27 |
| noonedeadpunk | threelittlebirds: in your case I would just re-run the playbook os-neutron-install.yml and see if issue preserves | 08:28 |
| noonedeadpunk | as that looks like some intermittent connection-related thing | 08:29 |
| noonedeadpunk | (or not intermittent, but still connection-related) | 08:29 |
| threelittlebirds | will do | 08:43 |
| threelittlebirds | erroring at the same step, i ran the playbook with -vvvv | 08:51 |
| threelittlebirds | https://paste.openstack.org/show/blJTK1Nmo8WerbKy61Pv/ | 08:51 |
| noonedeadpunk | aha | 08:56 |
| noonedeadpunk | threelittlebirds: sorry, I got distracted | 09:21 |
| * noonedeadpunk wonders how that worked at all | 09:25 | |
| threelittlebirds | nps. i guess the tasks is failing because the file (/tmp/api-paste.ini.original) is not inplace on compute1 but /openstack/venvs/neutron-27.0.2.dev17/etc/neutron/api-paste.ini.original exists | 09:26 |
| noonedeadpunk | oh, so this exists on compute? | 09:27 |
| noonedeadpunk | ok... | 09:27 |
| noonedeadpunk | For some reason I was thinking that there's no neutron venv at all on compute... | 09:27 |
| threelittlebirds | yes, exists | 09:29 |
| noonedeadpunk | IIRC that usually can be fixed with running the playbook with `-e venv_rebuild=true` | 09:32 |
| noonedeadpunk | but interesting to see output of `Fetch override files` task | 09:33 |
| noonedeadpunk | that is the one right before failure | 09:33 |
| noonedeadpunk | aha, ok, I missed condition in the beginning of the file, that won't deploy config if venv is absent | 09:37 |
| noonedeadpunk | But still interesting to see result of `Fetch override files` - as that should create /tmp/api-paste.ini.original. Btw, /tmp/api-paste.ini.original is the file on a localhost, not on the remote one | 09:38 |
| noonedeadpunk | threelittlebirds: ^ | 09:38 |
| admin1 | noonedeadpunk, so a workaround is to rm -rf the facts folder and then no patch needed ? | 09:47 |
| admin1 | which will have it gather the facts again ? | 09:47 |
| noonedeadpunk | nah, workaround would be to gather facts manually.... | 09:50 |
| noonedeadpunk | like doing ansible -m setup nova_all or smth | 09:50 |
| threelittlebirds | heres the result from `fetch override files`: https://paste.openstack.org/show/bLBjDQsiYuF7Gz2h7vFY/ | 09:55 |
| threelittlebirds | ill run the playbooks with your suggestion | 09:58 |
| threelittlebirds | ill run the playbook with your suggestion | 09:58 |
| noonedeadpunk | Hm, I kinda confused with `"skip_reason": "Conditional result was False"`. As the only condition which is present there is applied to whole block | 10:07 |
| noonedeadpunk | I just can imagine couple of corner cases.... | 10:10 |
| noonedeadpunk | but then again how it worked all this time... | 10:12 |
| noonedeadpunk | jrosser: are you around to hep out with that? | 10:12 |
| noonedeadpunk | as I'm confused | 10:12 |
| noonedeadpunk | specifically this part https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/neutron_post_install.yml#L108-L138 | 10:13 |
| noonedeadpunk | as this clean-up and run_once sounds like - you fetch files for neutron_server, then it's not executed during neutron_agents, but already removed for next batch | 10:15 |
| noonedeadpunk | or well, for serial run | 10:15 |
| noonedeadpunk | `execute on the first host available and afterwards apply any results and facts to all active hosts in the same batch` and we have different batches here I assume | 10:17 |
| threelittlebirds | ran the playbook with 'venv_rebuild=true' - but some issue: https://paste.openstack.org/show/bX6RAheioowEHnqQMFu4/ | 10:23 |
| threelittlebirds | same issue* | 10:23 |
| noonedeadpunk | threelittlebirds: I have a theory, but don't have env to reproduce that | 10:29 |
| noonedeadpunk | so can I ask you to re-run os-nova-install.yml but this time with -e 'neutron_serial=100%' | 10:30 |
| threelittlebirds | sure | 10:30 |
| noonedeadpunk | oh, sorry | 10:30 |
| noonedeadpunk | os-neutron-install.yml :) | 10:30 |
| threelittlebirds | here is the output: https://paste.openstack.org/show/bOkurwGZki6cgCtWaVek/ | 10:59 |
| noonedeadpunk | so it's same | 11:06 |
| noonedeadpunk | I'm really not getting `Conditional result was False` part | 11:06 |
| noonedeadpunk | it also fails for all computes while just being skipped for all others | 11:10 |
| noonedeadpunk | threelittlebirds: does previous tasks, like `Create plugins neutron dir` also got skipped for infra1? | 11:12 |
| noonedeadpunk | or Preserve original configuration file(s) ? | 11:13 |
| noonedeadpunk | if yes - can you share how you've defined groups in openstack_user_config.yml? | 11:14 |
| threelittlebirds | Create plugins neutron dir & Preserve original configuration file(s): https://paste.openstack.org/show/biTDqM86Bx1bPF0vD7Xg/ | 11:15 |
| threelittlebirds | groups in openstack_user_config.yml - https://paste.openstack.org/show/bqlqHYEby9VcFXTV5CgT/ | 11:16 |
| noonedeadpunk | threelittlebirds: and you're to use OVN? (or just default) | 11:17 |
| threelittlebirds | default | 11:18 |
| noonedeadpunk | ok, then I guess you're missing a thing there | 11:18 |
| noonedeadpunk | first - remove `network_hosts: *infrastructure_hosts` | 11:19 |
| noonedeadpunk | just a sec... | 11:20 |
| noonedeadpunk | I will double check :D | 11:20 |
| threelittlebirds | oko :D | 11:20 |
| noonedeadpunk | are you running metal deploy? | 11:21 |
| noonedeadpunk | or with lxc containers? | 11:21 |
| threelittlebirds | lxc | 11:23 |
| noonedeadpunk | Just to double-check - if you do `/opt/openstack-ansible/scripts/inventory-manage.py -G | grep neutron_server` - you get containers? | 11:25 |
| threelittlebirds | yes, all the way containers | 11:26 |
| noonedeadpunk | ok. then at very least you're missing `network-gateway_hosts` and `network-northd_hosts` | 11:28 |
| noonedeadpunk | are usually defined to `network-northd_hosts: *infrastructure_hosts` | 11:28 |
| noonedeadpunk | and `network-gateway_hosts` are defined to hosts thorugh which public traffic will be coming through | 11:29 |
| noonedeadpunk | so it's usually network-gateway_hosts: *compute_hosts | 11:29 |
| noonedeadpunk | or standalone net nodes | 11:29 |
| threelittlebirds | ok, ill edit my user_config | 11:32 |
| threelittlebirds | but those groups were not needed in yoga release, right? | 11:32 |
| noonedeadpunk | nope, they were not | 11:32 |
| noonedeadpunk | we've changed default network driver from LXB to OVN in Zed | 11:33 |
| threelittlebirds | so default is now ovn | 11:33 |
| noonedeadpunk | yup | 11:36 |
| noonedeadpunk | Eventually, you can choose a different network driver if needed | 11:36 |
| threelittlebirds | ok, so i guess i need to rerun the whole setup? | 11:37 |
| noonedeadpunk | for that you need to define: https://paste.openstack.org/show/b0ynurn6ZVZqHYiPWj2U/ | 11:38 |
| noonedeadpunk | not whole setup | 11:38 |
| noonedeadpunk | just lxc-containers-create.yml and then os-neutron-install.yml again | 11:38 |
| noonedeadpunk | though it smells like there is another bug present | 11:39 |
| noonedeadpunk | related to run_once | 12:02 |
| threelittlebirds | still same error message: https://paste.openstack.org/show/bc4RdEVbYjmnFOg0P5nt/ | 12:13 |
| threelittlebirds | but let me check my user_config again - since i was using it for yoga | 12:14 |
| noonedeadpunk | I kind of wonder if that's still result of some leftovers in inventory.... | 12:15 |
| noonedeadpunk | where infra hosts (bare metal ones) are part of some neutron group, but they don't have any service to isntall... | 12:16 |
| noonedeadpunk | As eventually these infra hosts should not be targeted during this run | 12:30 |
| noonedeadpunk | so I have an impressions that there're leftovers in inventory after defining network_hosts | 12:31 |
| Bico_Fino | To reinstall a nova_api container, this is enough -> openstack-ansible containers-lxc-create.yml os-nova-install.yml --limit "container" ? | 12:50 |
| noonedeadpunk | Bico_Fino: so first you need to drop the container | 12:58 |
| noonedeadpunk | openstack-ansible containers-lxc-destroy.yml --limit "container" | 12:58 |
| noonedeadpunk | then to create it again you indeed run openstack-ansible containers-lxc-create.yml --limit "container",lxc_hosts | 12:58 |
| noonedeadpunk | adding lxc_hosts is required, as some tasks are delegated to bare metal hosts | 12:59 |
| noonedeadpunk | you can use a specific host there instead of all lxc_hosts though | 12:59 |
| noonedeadpunk | and yes, then run openstack-ansible os-nova-install.yml --limit "container" | 12:59 |
| Bico_Fino | Thanks noonedeadpunk | 12:59 |
| *** NeilHanlon_ is now known as NeilHanlon | 13:11 | |
| * NeilHanlon should setup sasl | 13:11 | |
| Bico_Fino | noonedeadpunk tried that playbook from yesterday to update haproxy but failed, I will get the log. | 14:00 |
| noonedeadpunk | Bico_Fino: soo... you're stuck with removing things now, or? | 14:38 |
| Bico_Fino | noonedeadpunk yeah. :( | 14:39 |
| Bico_Fino | trying to setup a new infra node, but old stuff. | 14:39 |
| noonedeadpunk | New one has different ip/hostname, right? | 14:43 |
| Bico_Fino | yes. | 14:45 |
| noonedeadpunk | ok, so, then you need to remove old one from openstack_user_config.yml and run ./scripts/inventory-manage.py -r <node-name> | 14:47 |
| noonedeadpunk | IIRC - this should drop host and all containers from the inventory | 14:47 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Aug 29 15:00:30 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| mgariepy | hey ! | 15:00 |
| damiandabrowski | hi! | 15:01 |
| noonedeadpunk | #topic office hours | 15:04 |
| noonedeadpunk | I had quite short week, so obviously missed some things | 15:05 |
| NeilHanlon | heya folks | 15:05 |
| noonedeadpunk | I guess for me most valuable part is to finally get 27.1.0, and I guess main thing we're missing for "safe" upgrades is https://review.opendev.org/c/openstack/keystone/+/892864 | 15:05 |
| noonedeadpunk | Senlin, that has been broken upgrade path is now fixed | 15:06 |
| noonedeadpunk | For Adjutant - it's weird and I wasn't able to invest time there yet | 15:07 |
| noonedeadpunk | as it's broken differently for lxc and metal, so I'm really not sure wtf is going there | 15:07 |
| damiandabrowski | keeping fingers crossed to merge #892864 ASAP | 15:10 |
| noonedeadpunk | other then that, I've proposed to release ovs/ovn versions for stable branches | 15:12 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892563 | 15:12 |
| NeilHanlon | +1 from me on 892864 | 15:12 |
| noonedeadpunk | However, inside venv still old 2.17 buggy ovs will be installed due to constraints | 15:13 |
| noonedeadpunk | https://review.opendev.org/c/openstack/requirements/+/892891 | 15:14 |
| noonedeadpunk | doesn't look like it's getting attention from requirements team though :( | 15:14 |
| noonedeadpunk | Also, there's a patch to reduce amount of logs with journald | 15:18 |
| noonedeadpunk | which would also allow for users to provide their rotation rules | 15:18 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892151 | 15:18 |
| noonedeadpunk | This and releasing ovs version should help to solve disk_full for rocky | 15:18 |
| noonedeadpunk | Then also would be nice if you could review patches for stable branches: https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%255Estable/.*+status:open+ | 15:22 |
| noonedeadpunk | I was planning to issue role/service bumps for new releases | 15:22 |
| NeilHanlon | can do | 15:23 |
| noonedeadpunk | I've also somehow missed to sign us up for PTG, as was sure I did that. THough I've sent a follow-up email and we should be in schedule | 15:25 |
| NeilHanlon | oh, awesome. i meant to ask about that | 15:25 |
| noonedeadpunk | Though timeslots are not available yet, so too early to discuss date-time | 15:25 |
| NeilHanlon | I do see us on https://ptg2023.openinfra.dev/ now | 15:26 |
| noonedeadpunk | Yes, and please register :) | 15:28 |
| NeilHanlon | done! :) | 15:28 |
| noonedeadpunk | Not sure why - but that's needed | 15:28 |
| NeilHanlon | something something engagement metrics | 15:30 |
| noonedeadpunk | but then moderators also write back attendance... | 15:31 |
| noonedeadpunk | anyway | 15:31 |
| noonedeadpunk | for ongoing development | 15:31 |
| noonedeadpunk | and regarding quorum queues. | 15:32 |
| noonedeadpunk | I see damiandabrowski has reviewd changes and reportedly tested them | 15:33 |
| noonedeadpunk | There's also an ongoing improvement for that in oslo.messaging | 15:33 |
| noonedeadpunk | #link https://review.opendev.org/q/topic:bug-2031497 | 15:33 |
| damiandabrowski | yes, they look good from my perspective | 15:33 |
| noonedeadpunk | Though I have really no idea if that oslo.messaging has any chance to be in constraints of 2023.2 | 15:34 |
| noonedeadpunk | I guess no, as we're beyond feature freeze for libraries and projects.... | 15:34 |
| noonedeadpunk | As if they will - it would make sense to add couple of extra vars to control these new features | 15:34 |
| noonedeadpunk | though this can be done in follow-ups... | 15:35 |
| NeilHanlon | yep | 15:37 |
| noonedeadpunk | Once I will sort out current internal debts, I will try to look into Debian 12 support | 15:39 |
| noonedeadpunk | as that is another thing that would be nice to land | 15:40 |
| NeilHanlon | that'd be great | 15:44 |
| NeilHanlon | on my side i'm tracking what's going on in centos and such for EL10. that's supposedly coming at some point, so, yeah | 15:45 |
| noonedeadpunk | have you sorted out a way forward? | 15:46 |
| NeilHanlon | we have, yeah. we're more or less 'business as usual' just getting the sources from cloud instances for the most part. also working on the second generation of our build system https://github.com/peridotbuild/peridot/ | 15:46 |
| noonedeadpunk | ok, great, nice to hear:) | 15:47 |
| noonedeadpunk | #endmeeting | 15:58 |
| opendevmeet | Meeting ended Tue Aug 29 15:58:17 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-08-29-15.00.html | 15:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-08-29-15.00.txt | 15:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-08-29-15.00.log.html | 15:58 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Add ability to define a config for journald https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892151 | 18:28 |
| *** melwitt_ is now known as melwitt | 18:34 | |
| mgariepy | can someone review : https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/892314 and https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/892315 please, simple backport for ldap ssl stuff. | 18:56 |
| NeilHanlon | thanks for running the meeting btw noonedeadpunk ! | 18:56 |
| NeilHanlon | sure mgariepy | 18:56 |
| NeilHanlon | done | 18:57 |
| mgariepy | thanks a lot | 18:58 |
| NeilHanlon | any time :) | 18:58 |
| noonedeadpunk | thanks for taking time as usual NeilHanlon! | 18:58 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic stable/zed: Fix lack of constraints for ironic-inspector https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/892301 | 20:34 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2023.1: Allow Glance region to be set via variable https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/892319 | 20:44 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/892475 | 20:55 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/2023.1: Install libldap-common for keystone https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/892314 | 20:56 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/zed: Install libldap-common for keystone https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/892315 | 20:56 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/zed: Refactor LXC image expiration https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/892312 | 21:02 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/892357 | 21:15 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: nova/haproxy: fix typo in detection of 'serialconsole' https://review.opendev.org/c/openstack/openstack-ansible/+/889358 | 21:39 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Allow Glance region to be set via variable https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/892320 | 22:04 |
| opendevreview | Merged openstack/openstack-ansible-os_placement stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/892464 | 23:04 |
| opendevreview | Merged openstack/openstack-ansible-os_heat stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/892468 | 23:06 |
| opendevreview | Merged openstack/openstack-ansible-os_mistral stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/892472 | 23:06 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/zed: Release constraint for OVS/OVN versions for EL https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892563 | 23:07 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/892461 | 23:08 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/892470 | 23:10 |
| opendevreview | Merged openstack/openstack-ansible-os_designate stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/892471 | 23:11 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/892462 | 23:13 |
| opendevreview | Merged openstack/openstack-ansible-os_manila stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/892473 | 23:15 |
| opendevreview | Merged openstack/openstack-ansible-os_zun stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/892466 | 23:15 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/892467 | 23:26 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!