| *** spatel has joined #openstack-ansible | 00:09 | |
| *** spatel has quit IRC | 00:33 | |
| *** macz_ has joined #openstack-ansible | 00:51 | |
| *** macz_ has quit IRC | 00:55 | |
| *** dave-mccowan has quit IRC | 01:01 | |
| *** spatel has joined #openstack-ansible | 01:22 | |
| snadge | my neutron backend keeps falling over.. i think it has something to do with the rabbit queue filling up and messages timing out | 01:43 |
|---|---|---|
| snadge | this is on cent 7.7 and latest train | 01:43 |
| snadge | not sure why it has only just started doing this, very difficult to debug too, tried blowing everything away and starting again | 01:43 |
| snadge | things appear to be working ok apart from annoying messages on the console like "haproxy[11028]:backend neutron_server-back has no server available!" | 01:44 |
| snadge | if i could get some help debugging this when folks are awake later.. that would be great :D | 01:44 |
| *** weifan has joined #openstack-ansible | 02:00 | |
| *** weifan has quit IRC | 02:00 | |
| *** macz_ has joined #openstack-ansible | 02:39 | |
| *** macz_ has quit IRC | 02:43 | |
| snadge | i thought it might be related to this: https://review.opendev.org/#/c/697396/1 | 03:22 |
| snadge | i tried that patch and it doesn't appear to do anything | 03:22 |
| snadge | which was linked from this bug: https://bugs.launchpad.net/neutron/+bug/1853071 | 03:23 |
| openstack | Launchpad bug 1853071 in neutron "AMQP disconnects, q-reports-plugin queue grows, leading to DBDeadlocks while trying to update agent heartbeats" [High,In progress] - Assigned to Arjun Baindur (abaindur) | 03:23 |
| *** spatel has quit IRC | 03:26 | |
| *** mmercer has quit IRC | 03:53 | |
| *** mmercer has joined #openstack-ansible | 03:54 | |
| *** andrea15 has quit IRC | 04:03 | |
| *** andrea15 has joined #openstack-ansible | 04:04 | |
| *** macz_ has joined #openstack-ansible | 04:27 | |
| *** macz_ has quit IRC | 04:32 | |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-ansible | 04:36 | |
| *** cp- has quit IRC | 04:53 | |
| *** udesale has joined #openstack-ansible | 05:12 | |
| *** udesale has quit IRC | 05:14 | |
| *** udesale has joined #openstack-ansible | 05:14 | |
| *** cp- has joined #openstack-ansible | 05:57 | |
| *** miloa has joined #openstack-ansible | 06:42 | |
| *** irclogbot_1 has quit IRC | 06:49 | |
| *** waxfire has quit IRC | 06:50 | |
| *** waxfire has joined #openstack-ansible | 06:51 | |
| *** prometheanfire has quit IRC | 06:51 | |
| *** spotz has quit IRC | 06:51 | |
| *** prometheanfire has joined #openstack-ansible | 06:51 | |
| *** irclogbot_1 has joined #openstack-ansible | 06:51 | |
| *** jhesketh has quit IRC | 06:52 | |
| *** irclogbot_1 has quit IRC | 06:53 | |
| *** jhesketh has joined #openstack-ansible | 06:53 | |
| *** irclogbot_0 has joined #openstack-ansible | 06:53 | |
| *** jbadiapa has joined #openstack-ansible | 07:08 | |
| *** gokhani has joined #openstack-ansible | 07:24 | |
| *** tosky has joined #openstack-ansible | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:37 | |
| *** DanyC has joined #openstack-ansible | 07:43 | |
| *** DanyC has quit IRC | 07:48 | |
| jrosser | noonedeadpunk: this is an interesting error as we already split the add/remove tasks apart to prevent package db contention https://zuul.opendev.org/t/openstack/build/252cba0ccb9c4190867dc4c75591330d/log/job-output.txt#8083 | 07:54 |
| *** DanyC has joined #openstack-ansible | 07:59 | |
| *** Liwsaki has joined #openstack-ansible | 08:07 | |
| Liwsaki | Hello | 08:07 |
| Liwsaki | I'm from thailamd | 08:07 |
| Liwsaki | thailand | 08:07 |
| Liwsaki | I have a question? | 08:07 |
| *** kleini has joined #openstack-ansible | 08:10 | |
| noonedeadpunk | jrosser: btw we have bug in hardening.... | 08:28 |
| Liwsaki | Now python2.7 expire but I can't deploy openstack-ansible version ocata have a error pip can't load packge. | 08:29 |
| noonedeadpunk | https://github.com/openstack/ansible-hardening/commit/2093f503a64f213f58f9cab97a10a6f915a02ce9#commitcomment-37660759 | 08:30 |
| noonedeadpunk | Liwsaki: oh... ocata is not supported for years now... | 08:30 |
| noonedeadpunk | Why don't you want to consider deploying smth relevant? | 08:30 |
| jrosser | noonedeadpunk: for that ansible-hardening bug as it's a string we should be able to make that conditional on something | 08:48 |
| jrosser | i wonder why it didnt break our centos7 jobs though? | 08:49 |
| noonedeadpunk | I think we may just select not absent | 08:49 |
| noonedeadpunk | jrosser: because we use it in venv | 08:49 |
| jrosser | aaaahhhh | 08:50 |
| *** gokhani has quit IRC | 08:52 | |
| Liwsaki | @noonedeadpunk TT | 08:55 |
| *** DanyC has quit IRC | 09:01 | |
| *** gshippey has joined #openstack-ansible | 09:02 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: Fix role to work with Jinja 2.7 https://review.opendev.org/716528 | 09:02 |
| *** gokhani has joined #openstack-ansible | 09:08 | |
| *** DanyC has joined #openstack-ansible | 09:18 | |
| *** andrewbonney has joined #openstack-ansible | 09:18 | |
| *** jbadiapa is now known as jbadiapa|away | 09:27 | |
| *** owalsh has quit IRC | 09:50 | |
| *** owalsh has joined #openstack-ansible | 09:50 | |
| *** rpittau is now known as rpittau|bbl | 10:16 | |
| *** Liwsaki has quit IRC | 10:41 | |
| *** DanyC has quit IRC | 11:24 | |
| jrosser | oh dear opensuse job fails again in the gate | 11:29 |
| jrosser | evrardjp: has there been some kind of regression on package installing on suse? seeing a bunch of this http://paste.openstack.org/show/791447/ in the last 24 hours | 11:30 |
| *** DanyC has joined #openstack-ansible | 11:30 | |
| noonedeadpunk | and I've realized we have broken gates for hardening.... | 11:34 |
| noonedeadpunk | and not because of suse | 11:35 |
| jrosser | i saw fedora broken there | 11:38 |
| noonedeadpunk | yeah.... | 11:38 |
| * jrosser was just uploading fedora images and got distracted | 11:38 | |
| noonedeadpunk | Already have it and testing... | 11:39 |
| *** shyamb has joined #openstack-ansible | 11:58 | |
| *** jbadiapa|away is now known as jbadiapa | 11:59 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop https://review.opendev.org/710598 | 12:07 |
| *** rpittau|bbl is now known as rpittau | 12:07 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop https://review.opendev.org/710598 | 12:08 |
| evrardjp | mmm | 12:10 |
| noonedeadpunk | jrosser: btw https://review.opendev.org/#/c/715215/9 suse passed | 12:10 |
| evrardjp | is that transient? | 12:11 |
| evrardjp | Error: Subprocess failed. Error: RPM failed: warning: waiting for exclusive lock on /var/lib/rpm/Packages | 12:11 |
| evrardjp | might be worth checking if operations are done atomicly | 12:12 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-repo_build master: Retire openstack-ansible-repo_build role https://review.opendev.org/716389 | 12:13 |
| noonedeadpunk | I think they supposed to since package removal and installation jobs were separated.... | 12:14 |
| jrosser | we did this because of add/remove at the same time colliding https://review.opendev.org/#/c/693614/1/tasks/rhel7stig/packages.yml | 12:15 |
| *** rh-jelabarre has joined #openstack-ansible | 12:18 | |
| *** shyamb has quit IRC | 12:24 | |
| *** ericzolf has joined #openstack-ansible | 12:36 | |
| *** rh-jlabarre has joined #openstack-ansible | 12:36 | |
| *** rh-jelabarre has quit IRC | 12:37 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-repo_build master: Retire openstack-ansible-repo_build role https://review.opendev.org/716389 | 12:37 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop https://review.opendev.org/710598 | 12:42 |
| *** dave-mccowan has joined #openstack-ansible | 12:50 | |
| *** dave-mccowan has quit IRC | 12:54 | |
| *** jbadiapa has quit IRC | 13:02 | |
| *** jbadiapa has joined #openstack-ansible | 13:04 | |
| *** udesale_ has joined #openstack-ansible | 13:09 | |
| *** udesale has quit IRC | 13:11 | |
| *** mgariepy has quit IRC | 13:24 | |
| *** mgariepy has joined #openstack-ansible | 13:25 | |
| *** jamesden_ has joined #openstack-ansible | 13:37 | |
| kleini | jrosser: today I tried to deploy 19.0.11 and your suggested setting http://paste.openstack.org/show/791390/ fails with missing openstacksdk on the utility container | 13:45 |
| kleini | it did not fail during the initial bootstrapping of Keystone but when deploying the Users LDAP domain. | 13:45 |
| jrosser | it's really helpful if you paste the actual error output you got | 13:46 |
| jrosser | and the previous couple of tasks output to give the context | 13:47 |
| jrosser | kleini: i am thinking we have a bug here https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_ldap_setup.yml#L33 | 13:55 |
| kleini | http://paste.openstack.org/show/791468/ | 13:58 |
| jrosser | can you adjust this line https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_ldap_setup.yml#L33 | 14:01 |
| jrosser | and make it be like this one https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_federation_sp_idp_setup.yml#L24 | 14:01 |
| kleini | that change solves my deployment problem | 14:32 |
| jrosser | kleini: would you like to submit a patch for that? | 14:36 |
| jrosser | i can give you some pointers for that if you need, else i can patch it for you | 14:37 |
| *** macz_ has joined #openstack-ansible | 14:40 | |
| noonedeadpunk | kleini: to motivate you a bit, in case you path it time for patch to land will decrease dramatically because me with jrosser will be able to vote on it and land once CI will be finished | 14:41 |
| noonedeadpunk | otherwise we will need to wait for another core to review it :p | 14:41 |
| kleini | will try to provide the patch | 14:42 |
| jrosser | kleini: we already patched for ceph keys for you https://review.opendev.org/#/c/716274/1 | 14:43 |
| jrosser | thats on stable/train now | 14:43 |
| noonedeadpunk | Oh, maybe need to add it to the bump? | 14:44 |
| jrosser | the train bump patch has failed gate so it won't hurt | 14:45 |
| noonedeadpunk | why suse fails only in gates?... | 14:45 |
| jrosser | becasue it *knows* :) | 14:45 |
| noonedeadpunk | yeah, let me jsut finish another thing... | 14:45 |
| mnaser | i'll gladly remove suse from the gate and move it to non voting | 14:50 |
| mnaser | :) | 14:50 |
| *** gokhani has quit IRC | 15:12 | |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-os_horizon master: Config options around federation URLs https://review.opendev.org/716026 | 15:23 |
| *** pcaruana has quit IRC | 15:32 | |
| *** velmeran has quit IRC | 15:33 | |
| *** gyee has joined #openstack-ansible | 15:36 | |
| *** miloa has quit IRC | 15:45 | |
| *** pcaruana has joined #openstack-ansible | 15:48 | |
| *** DanyC has quit IRC | 15:49 | |
| *** DanyC has joined #openstack-ansible | 15:49 | |
| kleini | thank you very much for the ceph keys from files! Is it possible to get this backported to stable/stein? if I get Keystone bootstrapped with latest stable/stein (currently testing 19.0.11) | 15:57 |
| kleini | I will try to register and provide a patch tomorrow for the Keystone LDAP setup. | 15:58 |
| jrosser | kleini: if you look top right of the patch you see "cherry-picks" | 15:59 |
| jrosser | it is merged into train and the CI for stein is running right now on this patch https://review.opendev.org/#/c/716278 | 15:59 |
| kleini | oh, very nice. I would to totally lost without you guys! | 16:01 |
| *** jamesden_ has quit IRC | 16:14 | |
| *** rpittau is now known as rpittau|afk | 16:15 | |
| *** jamesden_ has joined #openstack-ansible | 16:21 | |
| *** jamesden_ has quit IRC | 16:29 | |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #openstack-ansible | 16:36 | |
| *** jamesden_ has joined #openstack-ansible | 16:57 | |
| *** d34dh0r53 has joined #openstack-ansible | 17:23 | |
| *** udesale_ has quit IRC | 17:29 | |
| *** ericzolf has quit IRC | 17:36 | |
| *** jamesden_ has quit IRC | 17:39 | |
| *** jbadiapa has quit IRC | 17:49 | |
| *** andrewbonney has quit IRC | 17:49 | |
| *** jbadiapa has joined #openstack-ansible | 17:50 | |
| *** jamesden_ has joined #openstack-ansible | 17:52 | |
| *** jamesden_ has quit IRC | 17:58 | |
| *** jamesden_ has joined #openstack-ansible | 17:59 | |
| *** mloza has joined #openstack-ansible | 18:11 | |
| *** DanyC has quit IRC | 18:22 | |
| *** jamesden_ has quit IRC | 18:26 | |
| *** jamesden_ has joined #openstack-ansible | 18:31 | |
| *** jamesden_ has quit IRC | 18:37 | |
| *** jamesden_ has joined #openstack-ansible | 19:00 | |
| *** DanyC has joined #openstack-ansible | 19:01 | |
| *** DanyC has quit IRC | 19:06 | |
| *** d34dh0r53 has quit IRC | 19:09 | |
| noonedeadpunk | Folks, am I doing smth wrong here? http://paste.openstack.org/show/791483/ | 19:17 |
| noonedeadpunk | as once I set ansible_python_interpreter=python3 I get this result: "msg": "[Errno 2] No such file or directory: '/root/.ansible/tmp/ansible-tmp-1585768520.19-272200180423918/find.py': '/root/.ansible/tmp/ansible-tmp-1585768520.19-272200180423918/find.py'" | 19:17 |
| noonedeadpunk | when it's python2 it's ok | 19:18 |
| noonedeadpunk | when I do without async it also works | 19:20 |
| *** d34dh0r53 has joined #openstack-ansible | 19:20 | |
| noonedeadpunk | huh, when I set ansible_python_interpreter=/usr/bin/python3 things are ok as well..... | 19:28 |
| noonedeadpunk | feels like some ansible bug.... | 19:30 |
| *** DanyC has joined #openstack-ansible | 19:33 | |
| *** DanyC has quit IRC | 19:39 | |
| noonedeadpunk | wtf, why suse always pass check but always fail in gate???? | 19:44 |
| noonedeadpunk | placed https://github.com/ansible/ansible/issues/68624.... | 19:52 |
| *** weshay is now known as weshay|ruck | 20:00 | |
| *** jbadiapa has quit IRC | 20:07 | |
| *** jbadiapa has joined #openstack-ansible | 20:08 | |
| *** gshippey has quit IRC | 20:21 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop https://review.opendev.org/710598 | 20:23 |
| *** thuydang has joined #openstack-ansible | 20:23 | |
| noonedeadpunk | ok, now we now, that we do what we shouldn't do... | 20:24 |
| noonedeadpunk | and I think we should fix it then? :( | 20:24 |
| *** kleini has quit IRC | 20:59 | |
| *** jbadiapa has quit IRC | 21:14 | |
| jrosser | noonedeadpunk: https://zuul.opendev.org/t/openstack/build/dc90c65b0e194aaaba56cfc8c2ed5ac5/log/logs/host/keystone-wsgi-public.service.journal-19-03-39.log.txt#6607-6624 | 21:21 |
| jrosser | thats the suse gate failure on https://review.opendev.org/#/c/715215/ | 21:22 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/stein: Importing keyrings from files rather than from mons https://review.opendev.org/716278 | 21:39 |
| mnaser | i swear i've seen that failure somewhere before jrosser | 21:39 |
| jrosser | it's cropped up a couple of times on train jobs | 21:40 |
| mnaser | but i'm in favour of nv and remove suse from gate, and if it goes non voting, i dont think anyone will actually bother looking at it | 21:40 |
| jrosser | just i couldnt find the previous cases, but thats one from just now | 21:40 |
| mnaser | and we earn some 3 hours of CI run-time per job | 21:40 |
| *** DanyC has joined #openstack-ansible | 22:08 | |
| *** melwitt has quit IRC | 22:08 | |
| *** aspiers has quit IRC | 22:08 | |
| *** Carcer has quit IRC | 22:08 | |
| *** spatel has joined #openstack-ansible | 22:19 | |
| *** rh-jlabarre has quit IRC | 22:26 | |
| *** aspiers has joined #openstack-ansible | 22:28 | |
| *** macz_ has quit IRC | 22:44 | |
| *** spatel has quit IRC | 22:44 | |
| *** melwitt has joined #openstack-ansible | 22:51 | |
| *** jamesden_ has quit IRC | 23:00 | |
| *** tosky has quit IRC | 23:03 | |
| *** DanyC has quit IRC | 23:45 | |
| *** sshnaidm is now known as sshnaidm|afk | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!