| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/xena: Do not use 'always' tag in inappropriate places https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/878017 | 08:10 |
|---|---|---|
| noonedeadpunk | Sooo... https://review.opendev.org/c/openstack/openstack-ansible/+/880150 seems passing... | 08:13 |
| noonedeadpunk | But I think we might need some more testing... | 08:13 |
| noonedeadpunk | looking at CI logs things work as expected - online migrations are executed where they should | 08:16 |
| noonedeadpunk | all services are obviously restarted as well | 08:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Install kata containers from source https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880288 | 11:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Define external_network_id in magnum_cluster_templates https://review.opendev.org/c/openstack/openstack-ansible/+/880047 | 12:03 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Fix haproxy_nova_api_metadata_service config https://review.opendev.org/c/openstack/openstack-ansible/+/880300 | 12:59 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Revert "Skip haproxy with setup-infrastructure for upgrades" https://review.opendev.org/c/openstack/openstack-ansible/+/880091 | 13:00 |
| opendevreview | Merged openstack/openstack-ansible master: Serve security.txt directly from haproxy https://review.opendev.org/c/openstack/openstack-ansible/+/880110 | 13:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/wallaby: Switch rabbitmq repo back to packagecloud https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/880324 | 13:47 |
| noonedeadpunk | we have really broken Zun as well | 15:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Ensure service is restarted on unit file changes https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/880339 | 16:11 |
| noonedeadpunk | btw. I've got exact same issue just today in aio https://zuul.opendev.org/t/openstack/build/b6cefddfd3fc4984bf7d4b5363511b99/log/job-output.txt#6123 | 16:21 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client master: Improve regexp for fetching nova secret from files https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/871819 | 16:22 |
| noonedeadpunk | where sahara venv install tried to use uwsgi packages | 16:24 |
| noonedeadpunk | and this thing we did not see in CI | 16:24 |
| noonedeadpunk | https://paste.openstack.org/show/bmHTk8Pua9tPiGpEyo4a/ | 16:24 |
| noonedeadpunk | smth sounds off with how variables are passed.... | 16:25 |
| noonedeadpunk | like scope or smth... | 16:25 |
| noonedeadpunk | BUT if I comment out uwsgi role - it's all fine | 16:26 |
| noonedeadpunk | I have close to no idea wtf is that | 16:27 |
| noonedeadpunk | and why we don't see that in CI | 16:28 |
| noonedeadpunk | well, with integrated jobs I mean | 16:28 |
| noonedeadpunk | replacing import_role with include_role does the trick | 16:33 |
| noonedeadpunk | hm, maybe these are 2 different issues... | 16:35 |
| noonedeadpunk | basically it breaks if both uwsgi and just python_venv_build role are improted. So looks like somehow, vars defined here https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/tasks/uwsgi_install.yml#L52 are also applied on import of python_venv_build in sahara role | 16:37 |
| noonedeadpunk | jrosser: don't you recall anything like that with messed up contexts? | 16:38 |
| noonedeadpunk | I think it's kinda close to our mess with include_vars that you've sorted out | 16:39 |
| noonedeadpunk | so looks like import_role does propogate and get all nested things imported as well along with their variables | 16:40 |
| noonedeadpunk | as there's also nothing in keystone role that would contain `openstack_hosts_package_state` | 16:44 |
| noonedeadpunk | I also wonder why I haven't seen anything like that before (as well as why we don't see that in CI) | 16:44 |
| jrosser | hello | 16:53 |
| jrosser | hmm yes | 16:56 |
| jrosser | well i am never sure if we are in include vs. import territory here | 16:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-uwsgi master: Replace imports with dynamic includes https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/880344 | 16:58 |
| noonedeadpunk | yeah, it's kinda messed a lot... | 16:58 |
| noonedeadpunk | btw, here another issue related I believe - https://zuul.opendev.org/t/openstack/build/608349326cda4c1cb8e87ec8c32c8ae3 | 17:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Use include instead of import for conditional tasks https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/874947 | 17:03 |
| noonedeadpunk | I feel like smth was "fixed" in 2.13 | 17:03 |
| noonedeadpunk | except facts. But potentially related to them | 17:04 |
| noonedeadpunk | nah, 2.12.10 is same | 17:06 |
| noonedeadpunk | aha, so you also can't refference handlers from other roles in case they're not imported | 17:09 |
| noonedeadpunk | (but included) | 17:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Use include instead of import for conditional tasks https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/874947 | 17:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Use include instead of import for conditional tasks https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/874949 | 17:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Use include instead of import for conditional tasks https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/874949 | 17:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Use include instead of import for conditional tasks https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/874949 | 17:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Use include instead of import for conditional tasks https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/874949 | 17:16 |
| noonedeadpunk | Or well, you can, but include should be processed first | 17:16 |
| noonedeadpunk | I can't imagine how to navigate or review imports vs includes.... | 17:17 |
| jrosser | this is becase at some point the variable namespace is kind of flat | 17:23 |
| jrosser | and then with the wrong kind of import/include you accidentally find vars from some other invocation of the role being present when not expected | 17:24 |
| jrosser | this makes my head hurt very very much :( | 17:24 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server stable/xena: Do not use 'always' tag in inappropriate places https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/878017 | 17:54 |
| noonedeadpunk | let's land these 2 patches and will do bump for final Xena release https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%255Estable/xena+status:open+ | 18:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance stable/xena: Disable uWSGI if ceph is used as a store https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/879699 | 18:29 |
| mgariepy | done | 18:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Drop apt_package_pinning from role requirements https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/880360 | 19:32 |
| noonedeadpunk | thanks mgariepy! | 19:33 |
| mgariepy | it helps my sanity to do something beside trying to debug network issue :D | 19:33 |
| damiandabrowski | ugh, i have weird error when running trove playbook on my AIO | 19:37 |
| damiandabrowski | https://paste.openstack.org/raw/b2JLnZkMbfnYqlXwHqQZ/ | 19:37 |
| damiandabrowski | i have no clue why it looks for uwsgi_build_distro_package_list | 19:38 |
| damiandabrowski | I was also able to pass through this stage after I commented out Import uwsgi role task in os_trove role O.o | 19:38 |
| jrosser | damiandabrowski: read the scollback a bit ... sounds similar to what noonedeadpunk was seeing earlier | 19:42 |
| noonedeadpunk | _exactly_ what I saw | 19:42 |
| noonedeadpunk | I wonder why it become issue now | 19:42 |
| noonedeadpunk | As I've _never_ seen that before | 19:43 |
| jrosser | what did we do to trigger this i wonder | 19:43 |
| noonedeadpunk | Hm, let me check in some older aio | 19:44 |
| noonedeadpunk | damiandabrowski: while https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/880344 covers the issue, it would be sweet to find the root cause | 19:46 |
| noonedeadpunk | eventually just this change would fix it https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/880344/1/tasks/uwsgi_install.yml | 19:47 |
| damiandabrowski | ahh, didn't see that patch, thanks | 19:47 |
| damiandabrowski | FWIW today i deployed AIO with most services included in SCENARIO and I only saw this issue for: trove, zun and sahara | 19:48 |
| noonedeadpunk | yup, eventually it raises when `venv_build_distro_package_list` is not passed to python_venv_build and both uwsgi and python_venv_build are imported and not included | 19:50 |
| noonedeadpunk | In case of any include somewhere - it's working | 19:50 |
| noonedeadpunk | so. older aio just failed the same way | 19:50 |
| noonedeadpunk | like 15do | 19:50 |
| noonedeadpunk | ok, I have one more that is 45do | 19:52 |
| damiandabrowski | just to confirm: #880344 helped me as well | 19:52 |
| noonedeadpunk | nah, that old aio is borked... | 19:57 |
| noonedeadpunk | I obviously runed it | 19:57 |
| jrosser | do we have other places with roles calling roles calling roles etc | 19:58 |
| noonedeadpunk | but unless it's smth with dependencies, like updated native python on destination or smth like then, then it's unlikely we broke behaviour with some latest change.... | 19:58 |
| noonedeadpunk | you know we have plenty of them... | 19:59 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/xena: fix indentation for condition https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/879480 | 19:59 |
| jrosser | tbh i have never really been completely comfortable with which should be used when | 20:00 |
| noonedeadpunk | but it's also "respecting" prescedence | 20:00 |
| noonedeadpunk | So despite uwsgi is "imported" after python_venv_build, but it will get role defaults overriden. | 20:01 |
| noonedeadpunk | python_venv_build defaults | 20:01 |
| noonedeadpunk | But then if you define the same var again - during your import - it will be respected | 20:02 |
| noonedeadpunk | yeah, same for me... Eventually I know only the cases where import can't be used at all... | 20:02 |
| noonedeadpunk | But now I also know that in case of using include, you can't reffer handlers from that role before include is done | 20:02 |
| noonedeadpunk | So first notify will fail | 20:03 |
| noonedeadpunk | I kinda wonder... couldn't it be, that some collection added `import_role`.... | 20:03 |
| noonedeadpunk | And we're using not builtin now, but smth else.... | 20:03 |
| noonedeadpunk | as we've merged collection bump not that long ago | 20:04 |
| noonedeadpunk | that would explain a lot of things | 20:04 |
| noonedeadpunk | but changing to ansible.builtin.import_role didn't help | 20:07 |
| noonedeadpunk | also grep through installed collections return nothing | 20:08 |
| noonedeadpunk | so... we can not import uwsgi role, as we're notifying handlers inside this role. But we can include rest of common ones, since we only listen for handlers they're notifying... | 20:12 |
| noonedeadpunk | ugh | 20:12 |
| noonedeadpunk | I'm starting hating ansible at such moments | 20:12 |
| noonedeadpunk | what is more ridiculous, is that in CI it's not failing on this step at all | 20:14 |
| noonedeadpunk | for my aio it fails right here https://zuul.opendev.org/t/openstack/build/6239b6d60acc4fc8a5ae9e90cf6fc447/log/job-output.txt#16119 | 20:16 |
| noonedeadpunk | damiandabrowski: you was using gate_check_commit? | 20:17 |
| jrosser | i wonder if that points to some built wheel like jinja2 or smth in ci | 20:18 |
| damiandabrowski | not this time, i manually executed bootstrap-ansible, bootstra-aio and setup-everything | 20:18 |
| noonedeadpunk | jrosser: yeah, I was thinking about that... but it kinda shouldn't.... | 20:19 |
| noonedeadpunk | last jinja2 was released year ago | 20:20 |
| jrosser | i guess you can point to the infra wheels in a local aio | 20:20 |
| jrosser | or even build also in a nodepool image | 20:20 |
| noonedeadpunk | can I? I thought their mirrors are in a private nodepool net or smth? | 20:22 |
| noonedeadpunk | maybe not.... | 20:22 |
| noonedeadpunk | I will try to spawn tomorrow using git-check-commit out of interest... | 20:25 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Drop `echo` from add-compute.sh script https://review.opendev.org/c/openstack/openstack-ansible/+/879477 | 21:15 |
| opendevreview | Merged openstack/openstack-ansible master: Install zuul-provided collections as directories https://review.opendev.org/c/openstack/openstack-ansible/+/880031 | 21:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_ironic master: Add TLS support to ironic backends https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/880394 | 22:46 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_aodh master: Add TLS support to aodh backends https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/880395 | 22:47 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_swift master: Add TLS support to swift backends https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/880396 | 22:49 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_designate master: Add TLS support to designate backends https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/880397 | 22:50 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_masakari master: Add TLS support to masakari backends https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/880398 | 22:50 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Remove security.txt parts https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/880111 | 23:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!