| *** dviroel|afk is now known as dviroel|out | 00:37 | |
| opendevreview | Merged openstack/openstack-ansible-os_mistral stable/xena: Add mistral-extra in the mistral venv https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/849575 | 02:47 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible stable/xena: Do not try to run etcd in serial mode https://review.opendev.org/c/openstack/openstack-ansible/+/848003 | 03:24 |
| *** ysandeep|out is now known as ysandeep | 05:00 | |
| *** ysandeep is now known as ysandeep|lunch | 07:35 | |
| jrosser | morning | 08:18 |
| noonedeadpunk | o/ | 09:30 |
| jrosser | do we need this? https://review.opendev.org/c/openstack/openstack-ansible/+/850478 | 09:31 |
| jrosser | a bit confused by the needed-by being abandoned | 09:31 |
| noonedeadpunk | wrong needed-by, should have been https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/847993 | 09:34 |
| noonedeadpunk | or no | 09:37 |
| noonedeadpunk | yeah, correct, wrong Needed-By | 09:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Increase manila flavor size https://review.opendev.org/c/openstack/openstack-ansible/+/846123 | 09:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Set minimum and maximum microversions for manila api https://review.opendev.org/c/openstack/openstack-ansible/+/850478 | 09:39 |
| *** ysandeep|lunch is now known as ysandeep | 09:59 | |
| *** ysandeep is now known as ysandeep|afk | 10:53 | |
| *** dviroel|out is now known as dviroel | 11:24 | |
| opendevreview | Danila Balagansky proposed openstack/openstack-ansible-os_neutron master: Dynamic ovn-encap-type configuration https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/850639 | 12:47 |
| anskiy | ^ no idea what `stt` is, I've took it from man: https://www.man7.org/linux/man-pages/man8/ovn-controller.8.html | 12:48 |
| noonedeadpunk | fair... | 12:48 |
| anskiy | I can modify it to get the list from some pre-defined variable, which would reside in openstack-ansible-os_neutron, but it would be kinda same. | 12:54 |
| mgariepy | https://tools.ietf.org/id/draft-davie-stt-06.html | 13:03 |
| anskiy | latest docs still mention it as supported: https://www.ovn.org/support/dist-docs/ovn-controller.8.html | 13:08 |
| mgariepy | yep. good enough i guess. it's an alternative tunel encap | 13:10 |
| mgariepy | what's up with [python_venv_build : gather build target facts] ? on placement | 13:15 |
| mgariepy | i've seen it on a few different distro / patches since yesterday. | 13:15 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/4211b9d532a247029df82a57cd7e2fa3/log/job-output.txt#13621 | 13:15 |
| mgariepy | seen debian/ focal/ c8s irrc | 13:15 |
| anskiy | that's EPERM, isn't it? | 13:18 |
| *** ysandeep|afk is now known as ysandeep | 13:20 | |
| mgariepy | no idea | 13:20 |
| mgariepy | anskiy, where do you spot that in the logs? | 13:22 |
| anskiy | "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": -13} rc :) | 13:23 |
| mgariepy | why would it fail on the same tasks for placement and not for keystone >? | 13:24 |
| anskiy | well, I don't know, but it's keystone is the first one, who goes into repo_container... | 13:37 |
| mgariepy | ho not placement.. | 13:42 |
| mgariepy | but uwsig | 13:42 |
| mgariepy | but uwsgi | 13:42 |
| mgariepy | utility - keystone - placement ok. but when it's called from uwsgi it fails on this check | 13:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Use release-ansible-collections from project-config https://review.opendev.org/c/openstack/ansible-config_template/+/850666 | 15:03 |
| mgariepy | should we disable NetworkManager on c8s ? https://zuul.opendev.org/t/openstack/build/55c677673cf24f1f894dac3f9915b177/log/job-output.txt#14686 | 15:05 |
| mgariepy | https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml#L37-L46 | 15:06 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/55c677673cf24f1f894dac3f9915b177/log/logs/etc/host/resolv.conf.txt#2 >> https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml#L166 | 15:09 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/55c677673cf24f1f894dac3f9915b177/log/logs/host/ironic-inspector-dnsmasq.service.journal-22-39-16.log.txt#44 | 15:10 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: Stop NetworkManager on RHEL https://review.opendev.org/c/openstack/openstack-ansible/+/850667 | 15:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Use release-ansible-collections from project-config https://review.opendev.org/c/openstack/ansible-config_template/+/850666 | 15:13 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_ironic stable/yoga: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/846777 | 15:15 |
| *** dviroel_ is now known as dviroel | 15:26 | |
| *** ysandeep is now known as ysandeep|out | 15:30 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Use release-ansible-collections from project-config https://review.opendev.org/c/openstack/ansible-config_template/+/850666 | 15:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Use release-ansible-collections from project-config https://review.opendev.org/c/openstack/ansible-config_template/+/850666 | 15:31 |
| mgariepy | >.< | 15:45 |
| mgariepy | i didn't saw it was yoga.. :/ | 15:45 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_ironic stable/yoga: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/846777 | 15:46 |
| noonedeadpunk | sorry mgariepy I'm quite distracted today and while I saw your questions really will have no chance answering them until late | 15:51 |
| mgariepy | no worries. | 15:56 |
| jrosser | mgariepy: similar here had a ridiculous day | 16:38 |
| jrosser | but i think i saw some other thing about networkmanager "interfering" in some other IRC so you might be onto something there | 16:38 |
| jrosser | and as far as MODULE FAILURE goes that seems to be one of the most frequent things that breaks our jobs | 16:39 |
| mgariepy | i saw in the inspector log that it networkmanager is in the way | 16:39 |
| mgariepy | only if the nodes renew it's lease in a timely fashion. | 16:39 |
| jrosser | for the MODULE FAILURE i am not sure what to do as it gives nothing useful in stderr/stdout | 16:40 |
| jrosser | i was considering making some sort of recursive playbook to just call that task infinitely and try to trigger it | 16:40 |
| mgariepy | it's kinda weird. might need to take some time to find more occurence but when i saw it it was always on placement | 16:41 |
| jrosser | i'm thinking something along the lines of bad / wrong permissions in /tmp or a race condition in the setup module that only breaks when the gather_subet is !all | 16:41 |
| mgariepy | not 100% sure if it was on uwsgi call from placement | 16:41 |
| jrosser | oh there are *loads* of these | 16:41 |
| mgariepy | can it be race on the socket? | 16:42 |
| jrosser | maybe a ~5 a day average when we are trying to merge stuff | 16:42 |
| jrosser | it's generally in the setup module i think? | 16:43 |
| mgariepy | ie. the ssh socket closing for some reason right on time.. and cause the module to fail ? | 16:43 |
| mgariepy | yeah | 16:43 |
| mgariepy | from what is saw it was setup. | 16:43 |
| jrosser | that itself is just wierd | 16:43 |
| mgariepy | yeah. | 16:43 |
| jrosser | it doesnt really seem to happen for any other module | 16:43 |
| mgariepy | do we have logs aggregate somewhere? | 16:44 |
| jrosser | perhaps something ends up in syslog | 16:44 |
| jrosser | though we throw away all of the logs that don't match a service we're interested in | 16:44 |
| jrosser | i've not had time but i think this is a good thing to try in an AIO | 16:45 |
| jrosser | if you have one a simple playbook runnning that task in the same way as when it fails would be fairly easy | 16:45 |
| mgariepy | jrosser, i seems to be able to reproduce is quite well now :D.. just need to figure out what param does make it fails. | 18:26 |
| jrosser | mgariepy: oh wow how did you do it? | 18:26 |
| mgariepy | with some magic ;) | 18:27 |
| mgariepy | export ANSIBLE_SSH_ARGS="${ANSIBLE_SSH_ARGS:-'-C -o ControlMaster=auto -o ControlPersist=2s'}" | 18:27 |
| jrosser | I asked about this in #ansible and they just told me that wasn’t how they would use setup module | 18:27 |
| jrosser | which was just wtf as we have little alternative | 18:28 |
| mgariepy | https://paste.ubuntu.com/p/kRK329btN9/ | 18:28 |
| mgariepy | don't evaluate the code tho lol | 18:28 |
| mgariepy | it does fail quite reliably | 18:28 |
| jrosser | is that url right? | 18:29 |
| mgariepy | hmm | 18:29 |
| mgariepy | https://paste.openstack.org/show/bDIHiGEyRlx5NbYsG84M/ | 18:30 |
| jrosser | oh need to be logged into something to see it i think | 18:30 |
| mgariepy | stupid need to be logged in.. | 18:30 |
| mgariepy | lol | 18:30 |
| jrosser | niiiice | 18:30 |
| mgariepy | 12.2.7 does have the issue. testing 12.2.6 now.. | 18:30 |
| jrosser | i never tried it but this had some interesting ideas https://stackoverflow.com/questions/54938299/how-to-call-the-ansible-play-recursively-based-on-until-condition | 18:31 |
| mgariepy | the default is 60s. for the controlePersist | 18:31 |
| jrosser | particularly the use of include_tasks: loop.yml from inside loop.yml | 18:31 |
| mgariepy | i did try a couple diffenrent sleep pattern ;) haha | 18:35 |
| jrosser | would be nice to be able to strip out wierdness and see if it still happens, like the delegate | 18:37 |
| mgariepy | nice loop :) seems to work :D | 18:40 |
| mgariepy | it's delegating | 18:42 |
| mgariepy | so far i've only seen it on ansible 2.12.7 | 18:45 |
| mgariepy | https://paste.openstack.org/show/bxyA4EhibRWyaow8Wo0b/ | 18:48 |
| mgariepy | https://paste.openstack.org/show/bzPXLwY2wLWdV7YrCB64/ | 18:57 |
| mgariepy | with the export of ANSIBLE_SSH_ARGS | 18:58 |
| mgariepy | hmm fun.. 2.13.2 is still affected. | 19:10 |
| jrosser | i wonder if it’s specific to our connection plugins or if it also does this with regular ansible | 19:11 |
| jrosser | the same play could target two real hosts | 19:11 |
| jrosser | and also in an AIO I wonder what it means when the AIO node is also the deploy node, if that becomes a corner case too | 19:12 |
| mgariepy | if i target aio. and not delegate it still have the same issue | 19:14 |
| jrosser | right - I guess the simplest possible case is needed for a bug report | 19:14 |
| mgariepy | i'm just swapping the ansible version in bootstrap-ansible.sh script and run it to install the correct version. | 19:22 |
| mgariepy | so our connection pluging should stay the same. | 19:23 |
| mgariepy | and 2.12.6 seems to be ok | 19:23 |
| *** dviroel is now known as dviroel|out | 21:07 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!