| *** d34dh0r53 has joined #openstack-ansible | 00:14 | |
| *** spatel has joined #openstack-ansible | 00:49 | |
| *** redrobot has quit IRC | 01:08 | |
| *** spatel has quit IRC | 01:10 | |
| *** cshen has joined #openstack-ansible | 01:40 | |
| *** cshen has quit IRC | 01:44 | |
| *** cshen has joined #openstack-ansible | 03:14 | |
| *** cshen has quit IRC | 03:18 | |
| *** dave-mccowan has quit IRC | 03:23 | |
| *** shyamb has joined #openstack-ansible | 04:21 | |
| *** MickyMan77 has quit IRC | 04:22 | |
| *** MickyMan77 has joined #openstack-ansible | 04:23 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-ansible | 04:33 | |
| *** shyamb has quit IRC | 05:03 | |
| *** shyamb has joined #openstack-ansible | 05:03 | |
| *** shyamb has quit IRC | 05:09 | |
| *** shyamb has joined #openstack-ansible | 05:10 | |
| *** jawad_axd has joined #openstack-ansible | 05:11 | |
| *** cshen has joined #openstack-ansible | 05:14 | |
| *** jawad_axd has quit IRC | 05:15 | |
| *** cshen has quit IRC | 05:19 | |
| snadge | so my employer is finally ready to move onto this train openstack-ansible deployment, but I'm seeing a bunch of timeout errors for neutron relating to rabbitmq.. i wonder if I have enough threads / processes allocated to various things etc | 05:30 |
|---|---|---|
| snadge | missed heartbeats and things like that.. it jams up.. errors.. then continues | 05:30 |
| *** shyamb has quit IRC | 05:45 | |
| *** mathlin has joined #openstack-ansible | 06:03 | |
| *** mathlin has quit IRC | 06:20 | |
| *** mathlin has joined #openstack-ansible | 06:22 | |
| *** cshen has joined #openstack-ansible | 06:25 | |
| *** cshen has quit IRC | 06:30 | |
| *** andrewbonney has joined #openstack-ansible | 06:50 | |
| jrosser | noonedeadpunk: did you see the train SHA bump failures, like on centos7 it just ignores the boostrap-host role totally https://zuul.opendev.org/t/openstack/build/1048979a2e204b80bf551baadca2fc40/log/job-output.txt#3913-3988 | 06:53 |
| noonedeadpunk | jrosser: not yet | 06:54 |
| noonedeadpunk | um.... | 07:00 |
| noonedeadpunk | that is weird indeed | 07:00 |
| noonedeadpunk | and tbh I thought that gate-check-commit is the one which launch bootstrap-ansible and bootstrap-aio | 07:02 |
| jrosser | i was looking here https://github.com/openstack/openstack-ansible/blob/stable/train/tests/bootstrap-aio.yml#L22 | 07:02 |
| jrosser | seems like the sshd role runs | 07:03 |
| noonedeadpunk | yeah | 07:03 |
| *** tosky has joined #openstack-ansible | 07:03 | |
| noonedeadpunk | and it's not failing as well | 07:03 |
| jrosser | then later it all just doesnt have inventory and deploys nothing, i guess becasue nothing got put in /etc/openstack_deploy | 07:04 |
| noonedeadpunk | yeah as we don't bootstrap corerectly, you're right about that | 07:04 |
| noonedeadpunk | *correctly | 07:05 |
| noonedeadpunk | oh, ok, we moved from bootstrap to pre step indeed to do it in more zuul native way, I can recall that | 07:06 |
| noonedeadpunk | I guess Logan was doing that | 07:06 |
| noonedeadpunk | Id say that might be just some gates flap unless that would be for all centos 7.... | 07:08 |
| jrosser | well, maybe just recheck and see if it comes good - 2 out of 3 centos7 jobs did that | 07:09 |
| noonedeadpunk | but.... sshd role seems not to be finished | 07:09 |
| noonedeadpunk | yeah, agree | 07:09 |
| *** pcaruana has joined #openstack-ansible | 07:14 | |
| *** pcaruana has quit IRC | 07:17 | |
| *** pcaruana has joined #openstack-ansible | 07:17 | |
| *** cshen has joined #openstack-ansible | 07:20 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Remove references to eth10/eth11 container interfaces https://review.opendev.org/751183 | 07:38 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Remove Centos-7 support https://review.opendev.org/742100 | 07:42 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Wait for aria2c to finish https://review.opendev.org/751724 | 07:44 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_container_create master: Remove support for Centos-7 https://review.opendev.org/742120 | 07:46 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_container_create master: Remove support for LXC2 configuration keys https://review.opendev.org/742121 | 07:46 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - test ansible 2.10 https://review.opendev.org/749484 | 07:51 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Bump ansible version to 2.9.13 https://review.opendev.org/737935 | 07:52 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Bump ansible version to 2.9.13 https://review.opendev.org/737935 | 07:52 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove Centos-7 support https://review.opendev.org/742103 | 07:52 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Remove Centos-7 support https://review.opendev.org/742104 | 07:54 |
| *** jawad_axd has joined #openstack-ansible | 08:00 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove opensuse-15 jobs https://review.opendev.org/737985 | 08:01 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove opensuse-15 jobs https://review.opendev.org/737985 | 08:02 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_magnum master: Only install devel packages during python_venv_build https://review.opendev.org/737844 | 08:07 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Remove apt proxy cleanup tasks https://review.opendev.org/736250 | 08:08 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Use ansible openstack collection https://review.opendev.org/718639 | 08:09 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Add os_senlin to required-projects https://review.opendev.org/749530 | 08:09 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] Bind services to mgmt network addresses https://review.opendev.org/670051 | 08:10 |
| *** jbadiapa has joined #openstack-ansible | 08:13 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_senlin master: Updated from OpenStack Ansible Tests https://review.opendev.org/752892 | 08:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Define condition for the first play host one time https://review.opendev.org/751248 | 08:34 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Define condition for the first play host one time https://review.opendev.org/751270 | 08:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Define condition for the first play host one time https://review.opendev.org/751783 | 08:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Define condition for the first play host one time https://review.opendev.org/751260 | 08:49 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Define condition for the first play host one time https://review.opendev.org/751274 | 08:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral master: Define condition for the first play host one time https://review.opendev.org/751784 | 08:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Use cinder_service_setup_host for qos and types creation https://review.opendev.org/750491 | 08:55 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: Define condition for the first play host one time https://review.opendev.org/751268 | 08:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh master: Define condition for the first play host one time https://review.opendev.org/751185 | 08:57 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Define condition for the first play host one time https://review.opendev.org/751257 | 08:58 |
| openstackgerrit | Merged openstack/openstack-ansible stable/ussuri: Bump SHAs for stable/ussuri https://review.opendev.org/752832 | 09:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Define condition for the first play host one time https://review.opendev.org/751790 | 09:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi master: Define condition for the first play host one time https://review.opendev.org/751272 | 09:35 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: Wait for aria2c to finish https://review.opendev.org/752905 | 09:38 |
| *** sshnaidm|afk is now known as sshnaidm | 09:52 | |
| *** SecOpsNinja has joined #openstack-ansible | 10:04 | |
| noonedeadpunk | jrosser: I'm really concerned about centos 8 job for galera... It fails for cluster, but spatel says it's working nicely for him... have no idea :( | 10:16 |
| snadge | yeah i have some strange networking stuff going on with amqp/rabbitmq | 10:17 |
| snadge | and neutron disconnecting | 10:17 |
| snadge | that's on centos 7.. im not sure if maybe its something dumb like systemd-networkd resetting interfaces when it shouldn't be | 10:18 |
| snadge | but getting things like connection resets | 10:18 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/5cf2d490f34846329a8aa45b83e38628/log/logs/openstack/container2/mariadb.service.journal.log.txt#542 | 10:19 |
| noonedeadpunk | snadge: do you get it for nova? | 10:19 |
| snadge | yes | 10:19 |
| noonedeadpunk | I mean it's for all services or for nova only?:) | 10:20 |
| noonedeadpunk | as I can recall some discussion that nova drops connections and it's more intended behaviour than an issue | 10:20 |
| snadge | nova.console.websocketproxy error: [Errno 32] Broken pipe | 10:21 |
| noonedeadpunk | do you see reall issues with nova except seeing this logged? | 10:22 |
| snadge | yeah.. amqp timeouts, and neutron errors in the nova log | 10:23 |
| snadge | for some reason the rabbit stuff is getting jammed up | 10:23 |
| snadge | i have 7 compute nodes, and one controller which runs all the containers | 10:25 |
| noonedeadpunk | does it have enough ram? | 10:25 |
| snadge | i think so | 10:25 |
| noonedeadpunk | (64gb might not be enough) | 10:25 |
| snadge | nova.console.websocketproxy error: [Errno 32] Broken pipe | 10:26 |
| snadge | wrong paste | 10:26 |
| noonedeadpunk | I think that might be okay | 10:26 |
| snadge | KiB Mem : 14010537+total, 28267148 free, 9387288 used, 10245094+buff/cache | 10:26 |
| snadge | so thats 16gb total? | 10:26 |
| noonedeadpunk | I mean iirc `Broken pipe` is caused with respawning of nova services, which have ttl or smth | 10:27 |
| noonedeadpunk | so it's smth that should not cause issues. but again, if I'm not mixing things up | 10:27 |
| snadge | 140gb rather, and 28gb free | 10:28 |
| noonedeadpunk | yah, more than 64 gb is enough for sure | 10:28 |
| noonedeadpunk | and what neutron errors are? | 10:29 |
| snadge | just heaps of rabbit timeouts | 10:29 |
| snadge | in the nuetron logs | 10:29 |
| snadge | so maybe i should look at simple things.. like how many threads/processes should be allocated to rabbit/neutron | 10:30 |
| noonedeadpunk | what I can suggest trying, is to run `openstack-ansible playbooks/rabbitmq-install.yml -e rabbitmq_upgrade=true` but again, this more fixes issue when you have a cluster rather than single instance | 10:30 |
| snadge | interesting.. so i see lots of errors like "ERROR oslo.messaging._drivers.impl_rabbit [req-1dc8f485-9008-4635-8cab-4560b2edb71d - - - - -] [4264d422-74f3-4764-8ad1-5cda53f65d2e] AMQP server on 172.29.237.199:5672 is unreachable: timed out. Trying again in 6 seconds.: timeout: timed out" | 10:32 |
| snadge | and too many heartbeats missed.. then eventually it goes 200 OK | 10:32 |
| snadge | looks like it makes about 8 connections to the rabbit server | 10:33 |
| snadge | is that one for each compute node or something? | 10:33 |
| snadge | Agent healthcheck: found 11 dead agents out of 13 | 10:34 |
| jrosser | snadge: if you've got somehow massive numbers of messages backed up in rabbitmq the whole thing can bog down | 10:45 |
| snadge | how do i check the queue depth? | 10:46 |
| jrosser | enabling the rabbitmq dashboard and looking at that is sometimes quite useful | 10:46 |
| jrosser | i think that dashboard is on port 15672 of the internal VIP and the password is in user secrets | 10:48 |
| jrosser | the 'monitoring' user has limited permissions which means it doesnt see everything | 10:49 |
| jrosser | you can 'rabbitmqctl set_user_tags monitoring administrator' | 10:49 |
| jrosser | to give it full permissions | 10:50 |
| snadge | i must have used an ssh port forward to do it before, thats the part i would have missed for usre | 10:50 |
| snadge | i can see i have accessed 15672 via localhost | 10:50 |
| jrosser | noonedeadpunk: yes centos-8 galera is looking really bad | 10:52 |
| jrosser | and thats blocking loads of patches | 10:52 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Remove opensuse-15 jobs https://review.opendev.org/737985 | 10:53 |
| noonedeadpunk | including move of ansible version, yeah | 10:54 |
| noonedeadpunk | and I'd rather not set it to non-voting.... | 10:55 |
| snadge | im logged in as "admin" user | 10:57 |
| noonedeadpunk | jrosser: btw, I've put us into PTG schedule - on wednesday on day slot, and on thursday on evening one | 10:58 |
| noonedeadpunk | Will that work for you? | 10:58 |
| noonedeadpunk | oh my | 10:59 |
| noonedeadpunk | that's bad | 11:00 |
| noonedeadpunk | I did huge mistake | 11:00 |
| snadge | it looks like neutron messages are queueing/unacked | 11:02 |
| snadge | specifically q-reports-plugin | 11:03 |
| noonedeadpunk | change of plans:) monday 13UTC-17UTC, Tuesday 21UTC-23UTC ? | 11:03 |
| noonedeadpunk | (it's October 26-27) | 11:07 |
| jrosser | should be ok | 11:18 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_magnum master: Only install devel packages during python_venv_build https://review.opendev.org/737844 | 11:25 |
| *** jawad_axd has quit IRC | 11:25 | |
| *** jawad_axd has joined #openstack-ansible | 11:26 | |
| noonedeadpunk | ok and train failed exact same way :( | 11:45 |
| noonedeadpunk | a bunch of PRs were merged for https://github.com/willshersystems/ansible-sshd/commits/master | 11:50 |
| jrosser | noonedeadpunk: we didnt remove something on master for centos-7 which has affected stable/train? | 11:50 |
| jrosser | zuul jobs stuff | 11:51 |
| noonedeadpunk | nope we didn;t I think it's ssh role that breaks us for some reason... | 11:52 |
| noonedeadpunk | Like https://github.com/willshersystems/ansible-sshd/commit/b9fb457d2b8a453e2e9e983216be11b857f91831 | 11:52 |
| noonedeadpunk | or not... | 11:53 |
| noonedeadpunk | I mean sshd doesn't really execute | 11:53 |
| noonedeadpunk | it just gather facts and fails... | 11:54 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Bump SHAs for stable/train https://review.opendev.org/752831 | 11:54 |
| *** rh-jelabarre has joined #openstack-ansible | 11:55 | |
| *** Brace has joined #openstack-ansible | 11:57 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/ussuri: Wait for aria2c to finish https://review.opendev.org/752905 | 12:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Add deployment of keystone_auth_default_policy https://review.opendev.org/751767 | 12:10 |
| jrosser | noonedeadpunk: what do you think of this https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/fc27e735a68b64cb3c67dd8abeaf324803a9845b/tasks/rabbitmq_post_install.yml#L56-L74 | 12:14 |
| jrosser | it tries to configure the plugins before restarting the service, that feels a bit odd | 12:14 |
| jrosser | with ansible 2.10 *all* the focal jobs fail like this https://zuul.opendev.org/t/openstack/build/26fd41c434e24643be1aef6fc12e84cc/log/job-output.txt#8724 | 12:16 |
| * noonedeadpunk wondering how ansible version changes that | 12:17 | |
| jrosser | yeah, odd | 12:18 |
| noonedeadpunk | maybe it started to correctly interper output of plugin... | 12:18 |
| jrosser | oh hrrm you know what else - focal we still take the distro package for rabbtmq | 12:19 |
| noonedeadpunk | oh, so it might get another version of rabbit | 12:19 |
| noonedeadpunk | it's 3.8.2 | 12:21 |
| jrosser | it's working elsewhere though isnt it | 12:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Simplify service creation https://review.opendev.org/751768 | 12:22 |
| jrosser | so maybe you're right and something changes with 2.10 and the rabbitmq_plugin module | 12:22 |
| jrosser | maybe not https://zuul.opendev.org/t/openstack/build/26fd41c434e24643be1aef6fc12e84cc/log/logs/host/rabbitmq/rabbit@aio1.log.txt#169 | 12:22 |
| noonedeadpunk | btw still no package on packagecloud | 12:23 |
| jrosser | no it seems just noting at all for focal | 12:23 |
| jrosser | *nothing | 12:23 |
| noonedeadpunk | hm, where do we store this plugin? | 12:25 |
| noonedeadpunk | oh, it's upstream... | 12:26 |
| noonedeadpunk | lol | 12:26 |
| jrosser | maybe i miss something but not even focal here https://bintray.com/rabbitmq/debian/rabbitmq-server | 12:26 |
| noonedeadpunk | I guess since focal uses 3.8 from their repos, they decided not to build 3.8 for it at all... | 12:27 |
| noonedeadpunk | https://launchpad.net/ubuntu/focal/+package/rabbitmq-server | 12:28 |
| jrosser | hmm ok - think i need a focal / ansible 2.10 AIO to see whats going on | 12:30 |
| noonedeadpunk | rabbitmq_plugin wasn't changed for years as well.... | 12:30 |
| noonedeadpunk | oh, wait, for 2.10 I think it comes from another repo | 12:31 |
| jrosser | right - so i'm wondering how this even works | 12:31 |
| jrosser | does ansible 2.10 ship the community collections? | 12:32 |
| *** jawad_ax_ has joined #openstack-ansible | 12:32 | |
| jrosser | https://github.com/ansible-collections/community.rabbitmq | 12:33 |
| jrosser | oh wow look who makes all the commits there :) odyssey4me | 12:33 |
| noonedeadpunk | yeah, I promised to help out with that but never did ;( | 12:34 |
| noonedeadpunk | huh, lol | 12:35 |
| *** dave-mccowan has joined #openstack-ansible | 12:35 | |
| noonedeadpunk | https://github.com/ansible-collections/community.rabbitmq/blob/main/plugins/modules/rabbitmq_plugin.py#L143 - that is new | 12:35 |
| *** jawad_axd has quit IRC | 12:36 | |
| noonedeadpunk | so maybe not valid defaults were used... | 12:37 |
| noonedeadpunk | but we have pretty simple fix though | 12:38 |
| noonedeadpunk | but we jsut need to set `broker_state: offline` there | 12:39 |
| jrosser | yeah, i think from our code it does this after dropping the config but before restarting the service, which could explain that | 12:40 |
| noonedeadpunk | and seems default was offline (but no default here https://www.rabbitmq.com/rabbitmq-plugins.8.html ) | 12:40 |
| jrosser | might be intersting chicken/egg to land this patch | 12:41 |
| noonedeadpunk | I mean it seems that --online is actually what changes behaviour | 12:41 |
| noonedeadpunk | as it means `Treat a failure to connect to the running broker as fatal.` | 12:41 |
| noonedeadpunk | maybe we will just change default in collection?:) | 12:41 |
| *** jawad_axd has joined #openstack-ansible | 12:42 | |
| noonedeadpunk | it's tagged though.... | 12:42 |
| jrosser | does this mean we need to pull in the collection? | 12:43 |
| noonedeadpunk | I think ansible pulls it... | 12:43 |
| noonedeadpunk | btw I guess just landing patch to 2.9 won't make it broken because of unkown argument? | 12:44 |
| jrosser | thats an error isnt it? using non existant module arg? | 12:45 |
| jrosser | well - maybe bigger picture thing | 12:45 |
| *** jawad_ax_ has quit IRC | 12:46 | |
| jrosser | rabbitmq was always a pain point for the ansible module | 12:46 |
| jrosser | so perhaps we just switch to using the collection right now | 12:46 |
| jrosser | independant of 2.10 | 12:46 |
| noonedeadpunk | and drop afterwards? | 12:46 |
| odyssey4me | jrosser: yep, I got that setup as an individual collection after our chat about all the changes you and noonedeadpunk wanted to make | 12:46 |
| noonedeadpunk | as in 2.10 it will be collection anyway? | 12:47 |
| jrosser | well i'm kind of not sure | 12:47 |
| jrosser | i really don't understand how my wip patch on 2.10 is getting rabbitmq modules at all right now | 12:47 |
| jrosser | o/ hello odyssey4me | 12:47 |
| noonedeadpunk | from what I can recall they were going to install all collections with ansible and provide minimal package along with that | 12:48 |
| odyssey4me | yeah, 2.10 should include that collection | 12:48 |
| noonedeadpunk | no to bother users with installing all manually to get things working as was in 2.9 | 12:48 |
| odyssey4me | I think 'ansible' from pypi will be ansible-base+maintained collections | 12:49 |
| noonedeadpunk | +1 | 12:49 |
| jrosser | odyssey4me: so if i run a job today with ansible 2.10 can i find which version of rabbitmq_plugin is used? | 12:51 |
| odyssey4me | jrosser: it should be the 1.0.0 collection release that is used | 12:53 |
| odyssey4me | jrosser: although quite honestly I have no idea how one would confirm that | 12:54 |
| jrosser | :) | 12:54 |
| noonedeadpunk | jrosser: yeah, probably it would be great goal to use ansible-minimal | 12:54 |
| jrosser | noonedeadpunk: looks like it - you just can no longer go to github/ansible/ansible and dive straight to the module code | 12:54 |
| noonedeadpunk | I'm more bothered with docs tbh | 12:55 |
| odyssey4me | noonedeadpunk: the docs should also be getting published - if not, there's a bug | 12:55 |
| noonedeadpunk | as https://docs.ansible.com/ansible/latest/modules/rabbitmq_plugin_module.html is obviously not relevant | 12:55 |
| noonedeadpunk | broker_state was already in 1.0.0 | 12:56 |
| *** yasemind has quit IRC | 12:57 | |
| admin0 | hi all .. anyone using magnum/kubernetes on openstack | 13:07 |
| *** yasemind has joined #openstack-ansible | 13:10 | |
| *** jawad_axd has quit IRC | 13:21 | |
| *** jawad_axd has joined #openstack-ansible | 13:22 | |
| *** cshen has quit IRC | 13:28 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/stein: Bump SHAs for stable/stein https://review.opendev.org/752830 | 13:36 |
| mgariepy | can i have some votes for the OVN backport to U ? https://review.opendev.org/#/c/751269 | 13:37 |
| *** dwilde has joined #openstack-ansible | 13:44 | |
| *** d34dh0r53 has quit IRC | 13:44 | |
| *** MickyMan77 has quit IRC | 13:49 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: DNM set broker_state when enabling plugins https://review.opendev.org/752958 | 14:19 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - test ansible 2.10 https://review.opendev.org/749484 | 14:19 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Test ansible-base 2.10.1 https://review.opendev.org/752963 | 14:26 |
| noonedeadpunk | jrosser: so, train has failed because of the sshd role.... | 14:43 |
| noonedeadpunk | I guess we need to bump it back for ussuri as well then... | 14:43 |
| noonedeadpunk | and llok into it... | 14:43 |
| admin0 | checking if anyone is using kubernetes on openstack.. i am getting "Create FailedResource Create Failed: Authorizationfailure: Resources.Kube Masters.Resources[0].Resources.Kube-Master: Authorization Failed." -- i think i have to manually fix some permissions for this, but I cannot find a documentation or pinpoint it | 14:43 |
| *** spatel has joined #openstack-ansible | 14:46 | |
| jrosser | noonedeadpunk: https://github.com/willshersystems/ansible-sshd/compare/3fb34ad9de7757739071bc777f8cb89c28a8ea74..6e9c4c181163a3863fc4ad76ebb32600333e903f | 14:50 |
| jrosser | this looks suspicious https://github.com/willshersystems/ansible-sshd/compare/3fb34ad9de7757739071bc777f8cb89c28a8ea74..6e9c4c181163a3863fc4ad76ebb32600333e903f#diff-468007bd6f59ca63e12c1a3613323a17R3 | 14:50 |
| noonedeadpunk | but https://github.com/willshersystems/ansible-sshd/blob/master/vars/RedHat_7.yml#L28 | 14:52 |
| noonedeadpunk | however it's exactly where task fails | 14:52 |
| noonedeadpunk | *it ends | 14:53 |
| jrosser | that code feels kind of suspect | 14:53 |
| * noonedeadpunk wondering if meta can be used with when | 14:53 | |
| jrosser | for a non supported OS won't it fail with __sshd_os_supported variable is not defined | 14:53 |
| jrosser | oh hold on | 14:54 |
| jrosser | its int defaults/main.yml as 'no' | 14:54 |
| noonedeadpunk | it is | 14:55 |
| jrosser | so the other possiblity is that no OS specific file got included here https://github.com/willshersystems/ansible-sshd/blob/9e79cc3802343d789baabf5f8233c4fab779dd30/tasks/variables.yml#L2 | 14:57 |
| noonedeadpunk | hm, shouldn't it be ansible_distribution_major_version rather then ansible_distribution_version? | 14:59 |
| noonedeadpunk | https://github.com/ansible/ansible/issues/57463 | 15:00 |
| jrosser | yes i was just looking at this https://github.com/willshersystems/ansible-sshd/blob/9e79cc3802343d789baabf5f8233c4fab779dd30/tasks/variables.yml#L22 | 15:00 |
| noonedeadpunk | and ansible_distribution_lts_version is kinda the same | 15:00 |
| jrosser | our issue is centos-7 though? | 15:01 |
| noonedeadpunk | yeah | 15:01 |
| noonedeadpunk | 8 was passing | 15:01 |
| jrosser | for the 7 jobs we get | 15:02 |
| jrosser | ansible_os_family: RedHat | 15:02 |
| jrosser | ansible_distribution_version: '7.8' | 15:03 |
| noonedeadpunk | we might got another ansible_distribution_version | 15:03 |
| jrosser | on stable/train | 15:03 |
| noonedeadpunk | yeah | 15:03 |
| jrosser | so theres an ansible version factor here too | 15:03 |
| noonedeadpunk | is it out of logs? | 15:03 |
| jrosser | the 8 jobs are on a later ansible | 15:03 |
| jrosser | https://zuul.opendev.org/t/openstack/build/e5ae88e08ac546ccb0e7ab99f8f0a051/log/zuul-info/host-info.centos-7.yaml | 15:03 |
| jrosser | seems our with-first-found pattern is a bit more robust than this | 15:04 |
| noonedeadpunk | it is | 15:04 |
| noonedeadpunk | our is really perfect:) | 15:04 |
| jrosser | oh err right so it's this then? https://github.com/willshersystems/ansible-sshd/commit/9e79cc3802343d789baabf5f8233c4fab779dd30 | 15:13 |
| noonedeadpunk | it is... | 15:13 |
| noonedeadpunk | good news that we can bump 0.9.1? | 15:14 |
| noonedeadpunk | just in case I pushed https://github.com/willshersystems/ansible-sshd/pull/132 | 15:14 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Bump SHAs for stable/train https://review.opendev.org/752831 | 15:17 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder stable/ussuri: Use cinder_service_setup_host for qos and types creation https://review.opendev.org/753016 | 15:20 |
| *** jawad_axd has quit IRC | 15:20 | |
| *** cshen has joined #openstack-ansible | 15:22 | |
| *** redrobot has joined #openstack-ansible | 15:23 | |
| *** jawad_axd has joined #openstack-ansible | 15:40 | |
| *** jawad_axd has quit IRC | 15:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Only install devel packages during python_venv_build https://review.opendev.org/737844 | 16:07 |
| *** dwilde has quit IRC | 16:30 | |
| *** d34dh0r53 has joined #openstack-ansible | 16:30 | |
| *** cshen has quit IRC | 16:31 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/ussuri: Add integrated test for Ubuntu using ML2/OVN driver https://review.opendev.org/751269 | 16:36 |
| openstackgerrit | Merged openstack/openstack-ansible master: Remove references to eth10/eth11 container interfaces https://review.opendev.org/751183 | 16:36 |
| mgariepy | \o/ OVN | 16:37 |
| mgariepy | i've been waiting for this moment, for weeks, and a great number of recheck ! haha | 16:37 |
| mgariepy | and i found the root cause of the ceph-volume race condition. | 16:39 |
| *** gyee has joined #openstack-ansible | 16:40 | |
| *** cshen has joined #openstack-ansible | 16:58 | |
| *** andrewbonney has quit IRC | 16:59 | |
| arxcruz | noonedeadpunk: https://review.opendev.org/#/c/752571/ if you still have time, this is blocking our upgrade/update jobs as well :) | 17:00 |
| *** cshen has quit IRC | 17:02 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: DNM set broker_state when enabling plugins https://review.opendev.org/752958 | 17:16 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - test ansible 2.10 https://review.opendev.org/749484 | 17:40 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - test ansible 2.10 https://review.opendev.org/749484 | 17:43 |
| jrosser | noonedeadpunk: ^ ansible-base + collections gets at least to setup-openstack.yml with that | 17:44 |
| jrosser | oh argh wrong patch | 17:49 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Test ansible-base 2.10.1 https://review.opendev.org/752963 | 17:50 |
| *** cshen has joined #openstack-ansible | 17:51 | |
| *** SecOpsNinja has left #openstack-ansible | 18:00 | |
| *** ChiTo has joined #openstack-ansible | 18:22 | |
| ChiTo | Hi openstack-ansible team | 18:22 |
| *** ChiTo has quit IRC | 18:22 | |
| *** ChiTo has joined #openstack-ansible | 18:25 | |
| *** ChiTo has quit IRC | 18:26 | |
| *** ChiTo has joined #openstack-ansible | 18:27 | |
| admin0 | checking again if anyone is doing osa+magnum(kubernetes) | 18:28 |
| admin0 | my issue might be related to magnum_keystone policies and i need some pointers/help | 18:28 |
| ChiTo | Hi guys, I have a question, how can I protect my usernames variables/passwords within the user_variables with Vault? I have already encrypt them with Vault but how can I send the variable within user_variables.yml? | 18:29 |
| jrosser | admin0: i think guilhermesp has a lot of experience on osa+magnum | 18:29 |
| admin0 | thanks jrosser .. ping guilhermesp | 18:30 |
| jrosser | ChiTo: if you mean ansible vault then you could encrypt the contents of user_secrets.yml as described here https://docs.ansible.com/ansible/latest/user_guide/vault.html | 18:33 |
| ChiTo | jrosser: thx!, yes I encrypted them but now I want to send specicfic variables stored in the user_variables.yml without plain text, is tht possible? | 18:34 |
| ChiTo | because even if I have encrypted everything I still have to read the usernames/passwords fromm the user_variables.yml, is that correct? | 18:35 |
| *** cshen has quit IRC | 18:35 | |
| ChiTo | or there is no need to include them in my user_variables.yml after my secret password was vault-encrypted? | 18:35 |
| ChiTo | not sure why I am including them again in the user_variables, I think I got some errors since I was deploying keystone and it was looking for the variables in the user_variables | 18:35 |
| jrosser | when running the playbooks theres not really anything special about user_secrets.yml | 18:36 |
| jrosser | it's just another file names user_*.yml | 18:36 |
| jrosser | you can have as many files as you like user_foo.yml user_bar.yml and so on, they all get included | 18:37 |
| *** jawad_axd has joined #openstack-ansible | 18:41 | |
| *** jawad_axd has quit IRC | 18:46 | |
| ChiTo | jrosser: Oh I see sorry for the confusion, so the only thing I need to do is to encrypt the yml file and automatically everytime I run the openstack-ansible the password or key would be asked right? | 18:47 |
| jrosser | i believe so yes, i think you have to pass some extra parameters as described in the ansible vault docs and it should work | 18:48 |
| ChiTo | understood thx for the clarification! | 18:48 |
| jrosser | if you get it to work we could add this to our documentation, would be nice | 18:49 |
| *** jawad_axd has joined #openstack-ansible | 19:02 | |
| *** jawad_axd has quit IRC | 19:06 | |
| ChiTo | sure thing, I am doing some tests now | 19:08 |
| ChiTo | jrosser: btw I just submitted a comment this weekend just for your awareness guys, regarding the ironic-conductor service https://bugs.launchpad.net/openstack-ansible/+bug/1896355 nothing critical, it is just to refer them perhaps in the example files when users enable Ironic | 19:10 |
| openstack | Launchpad bug 1896355 in openstack-ansible " 'ironic_conductor-infra_hosts' is missing from the example user config " [Undecided,New] | 19:10 |
| *** pcaruana has quit IRC | 19:10 | |
| guilhermesp | oh sorry admin0 missed the ping. Actually, im not doing osa+magnum nowadays. I was doing until train. What exactly are you facing ? | 19:11 |
| jrosser | ChiTo: did you see that there is a fix on the related issue now? https://bugs.launchpad.net/openstack-ansible/+bug/1872427 | 19:11 |
| openstack | Launchpad bug 1872427 in openstack-ansible "'placement-infra_hosts' is missing from the example user config" [Undecided,In progress] - Assigned to Dmitriy Rabotyagov (noonedeadpunk) | 19:11 |
| jrosser | ChiTo: oh sorry - placement / ironic mix up there.... | 19:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_cinder master: Revert use of _cinder_first_play_host for post install tasks https://review.opendev.org/753131 | 19:22 |
| *** jawad_axd has joined #openstack-ansible | 19:23 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Revert use of _cinder_first_play_host for post install tasks https://review.opendev.org/753133 | 19:24 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Test ansible-base 2.10.1 https://review.opendev.org/752963 | 19:25 |
| ChiTo | jrosser: Yep that is correct, actually I based on that previous "bug" | 19:27 |
| *** jawad_axd has quit IRC | 19:27 | |
| *** spatel has quit IRC | 19:35 | |
| *** jawad_axd has joined #openstack-ansible | 19:44 | |
| *** jawad_axd has quit IRC | 19:48 | |
| *** jawad_axd has joined #openstack-ansible | 20:04 | |
| *** jawad_axd has quit IRC | 20:09 | |
| *** cshen has joined #openstack-ansible | 20:30 | |
| *** MickyMan77 has joined #openstack-ansible | 20:34 | |
| *** cshen has quit IRC | 20:35 | |
| admin0 | guilhermesp, when i try to bring up a cluster, i get "Create FailedResource Create Failed: Authorizationfailure: Resources.Kube Masters.Resources[0].Resources.Kube-Master: Authorization Failed. | 20:41 |
| admin0 | i think it could be due to some policy or stuff not being in place | 20:41 |
| *** MickyMan77 has quit IRC | 20:43 | |
| *** jawad_axd has joined #openstack-ansible | 20:46 | |
| *** jawad_axd has quit IRC | 20:51 | |
| ChiTo | I am trying to set a manila_manila_conf_overrides variables within my user_variables to configure an Isilon backend for Manila, but for some reason the parser of ansible puts those variables within the DEFAULT section in manila.conf instead of [isilon] | 21:07 |
| ChiTo | I am setting this in my user_variables http://paste.ubuntu.com/p/kVTCyrfSH6/ | 21:09 |
| ChiTo | but I get this: http://paste.ubuntu.com/p/BGbWGKfrkP/ | 21:11 |
| ChiTo | in just one line | 21:11 |
| ChiTo | but within the DEFAULT section instead of the isilon one | 21:11 |
| ChiTo | I created the isilon in my openstack_user_config.yml in the manila section: http://paste.ubuntu.com/p/PZ3FXbxz9r/ | 21:13 |
| *** jbadiapa has quit IRC | 21:29 | |
| *** martalais has joined #openstack-ansible | 21:34 | |
| *** kaiokmo has joined #openstack-ansible | 21:35 | |
| guilhermesp | admin0: sorry for the delay, do you have a stack created? what about heat logs? | 21:52 |
| jrosser | ChiTo: your yaml needs to use : instead of = | 21:54 |
| jrosser | key: value | 21:54 |
| *** dave-mccowan has quit IRC | 21:58 | |
| *** MickyMan77 has joined #openstack-ansible | 21:59 | |
| *** dave-mccowan has joined #openstack-ansible | 22:02 | |
| *** MickyMan77 has quit IRC | 22:07 | |
| *** rh-jelabarre has quit IRC | 22:12 | |
| *** rh-jelabarre has joined #openstack-ansible | 22:12 | |
| guilhermesp | admin0: also i have the feeling that we could be missing role assignments on heat domain... can you check `openstack role assignment list --domain heat` | 22:12 |
| guilhermesp | also which osa version | 22:13 |
| *** tacco has quit IRC | 22:17 | |
| *** jawad_axd has joined #openstack-ansible | 22:29 | |
| *** cshen has joined #openstack-ansible | 22:31 | |
| *** nsmeds has joined #openstack-ansible | 22:32 | |
| *** jawad_axd has quit IRC | 22:34 | |
| *** cshen has quit IRC | 22:35 | |
| *** tosky has quit IRC | 22:49 | |
| *** jawad_axd has joined #openstack-ansible | 22:50 | |
| *** jawad_axd has quit IRC | 22:55 | |
| *** kaiokmo has quit IRC | 22:59 | |
| *** MickyMan77 has joined #openstack-ansible | 23:04 | |
| *** jawad_axd has joined #openstack-ansible | 23:11 | |
| *** MickyMan77 has quit IRC | 23:13 | |
| *** jawad_axd has quit IRC | 23:15 | |
| *** djhankb has quit IRC | 23:16 | |
| *** djhankb has joined #openstack-ansible | 23:17 | |
| *** jawad_axd has joined #openstack-ansible | 23:32 | |
| *** martalais has quit IRC | 23:36 | |
| *** jawad_axd has quit IRC | 23:37 | |
| ChiTo | jrosser: You are totally right, thanks a lot! | 23:45 |
| *** MickyMan77 has joined #openstack-ansible | 23:48 | |
| *** spatel has joined #openstack-ansible | 23:49 | |
| *** jawad_axd has joined #openstack-ansible | 23:52 | |
| *** jawad_axd has quit IRC | 23:57 | |
| *** MickyMan77 has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!