| *** spatel has joined #openstack-ansible | 00:18 | |
| *** spatel has quit IRC | 00:23 | |
| *** ChiTo has joined #openstack-ansible | 00:33 | |
| *** macz_ has quit IRC | 00:41 | |
| ChiTo | Hi everyone, happy new year, I want to ask if someone has experienced this issue: I have deployed an OpenStack with 6 compute nodes with OVS and DVR on the 6 nodes, so deploying neutron components as part of the compute nodes as well, I have enabled l3_ha as well as dvr and dvr_snat for the agent for those VMs that do not need floating IPs. My issue is that when I spin up x qty of instances and I allocate floating IPs on them everything | 00:45 |
|---|---|---|
| ChiTo | works without any setbacks, but if on the same network I spin up another "10-20-x" qty of instances then I noticed that the connectivity with the floating IPs enter in an outage state and I noticed that after exactly 300 sec they come back, I did some tcpdumps on the fip net namespaces as well as the qrouter and it looks like the MAC addresses are not immediately updated to the virtual routers, not sure if it has to do with OVS or linux | 00:45 |
| ChiTo | kernel in terms of the MAC learning functionality, by chance can you guide me on how to troubleshoot this issue and fix it? I would not expect that the communication with the external provider network is lost and restored after 300 sec, not sure if it has to do with some aging on the mac values from OVS. thanks! | 00:45 |
| *** NewJorg has quit IRC | 02:21 | |
| *** NewJorg has joined #openstack-ansible | 02:23 | |
| *** macz_ has joined #openstack-ansible | 02:42 | |
| *** macz_ has quit IRC | 02:47 | |
| *** spatel has joined #openstack-ansible | 03:22 | |
| *** macz_ has joined #openstack-ansible | 04:43 | |
| *** macz_ has quit IRC | 04:47 | |
| *** pto has joined #openstack-ansible | 05:30 | |
| *** pto has joined #openstack-ansible | 05:31 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** gyee has quit IRC | 06:11 | |
| *** ChiTo has quit IRC | 06:18 | |
| *** spatel has quit IRC | 06:33 | |
| *** pto has quit IRC | 06:53 | |
| *** spatel has joined #openstack-ansible | 07:05 | |
| *** spatel has quit IRC | 07:05 | |
| *** dirk has quit IRC | 07:09 | |
| *** admin0 has quit IRC | 07:09 | |
| *** dirk has joined #openstack-ansible | 07:10 | |
| *** kukacz has quit IRC | 07:30 | |
| *** pto has joined #openstack-ansible | 07:39 | |
| *** kukacz has joined #openstack-ansible | 08:05 | |
| *** rh-jelabarre has quit IRC | 08:19 | |
| *** miloa has joined #openstack-ansible | 08:22 | |
| *** andrewbonney has joined #openstack-ansible | 08:24 | |
| *** pto has quit IRC | 08:31 | |
| *** pto has joined #openstack-ansible | 08:32 | |
| *** rpittau|afk is now known as rpittau | 08:32 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/ussuri: Add trusted.gpg to the copy_from_host list for 20.04 https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/769743 | 08:49 |
| *** maharg101 has joined #openstack-ansible | 08:56 | |
| *** tosky has joined #openstack-ansible | 08:56 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Add timed cleanup script to handle old Docker data https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/769688 | 09:14 |
| *** jbadiapa has joined #openstack-ansible | 09:23 | |
| *** fanfi has quit IRC | 09:25 | |
| *** gokhani has quit IRC | 09:26 | |
| frickler | is the rationale for using direct root ssh login documented somewhere? would adding an option to use regular user account(s) and "become" have a chance of being accepted? | 09:42 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/victoria: Add trusted.gpg to the copy_from_host list for 20.04 https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/769742 | 09:48 |
| jrosser | frickler: i guess it's just historical, there is some work done to allow regular users here https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/615403 | 09:48 |
| frickler | jrosser: ah, great, thx for the pointer. likely needs some docs updates and such, but at least I can cross off having to discuss this topic with my compliance team \o/ | 09:53 |
| jrosser | if you fancy testing out regular users i don't think there is anything particularly fundamental other than maybe some extra setting to globally drive become: in all the playbooks | 09:53 |
| frickler | I'm currently still busy setting up a PoC using root access, but that will be the next step, then, yes | 09:54 |
| jrosser | i expect there to be some bugs with that as i don't think anyone does it right now, but they should all minor things | 09:54 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Remove support for lxc2 config keys https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/756247 | 09:59 |
| noonedeadpunk | considering | 10:16 |
| noonedeadpunk | mornings | 10:16 |
| noonedeadpunk | considering, that V due date is 14 Januray I'm thinking if we should release rc2 or jump right away to stable? | 10:17 |
| noonedeadpunk | I think better to have it then not... | 10:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun stable/victoria: defaults: start kuryr service before docker https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/769593 | 10:19 |
| jrosser | we got dozens of patches like these https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/768689 | 10:20 |
| jrosser | not sure if they would normally be automated as part of making the release, or they are legitimate things we missed in the rc | 10:20 |
| noonedeadpunk | a lot of them are abandoned btw | 10:21 |
| *** rh-jelabarre has joined #openstack-ansible | 10:22 | |
| jrosser | check the review dashboard, i see loads | 10:23 |
| noonedeadpunk | Ie https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/768866 | 10:23 |
| jrosser | well, either way, it does ask the question if we need to update those files everywhere for the V release | 10:25 |
| noonedeadpunk | well... I don;t have good answer now. From things like they are currently - we need them | 10:26 |
| jrosser | looks like these were maybe part of the release process in the past https://github.com/openstack/openstack-ansible-os_glance/blob/master/releasenotes/source/train.rst | 10:27 |
| noonedeadpunk | But from ther side, I think that we probably should drop publishing release notes for roles somehow, since we copy them to integrated repo | 10:27 |
| noonedeadpunk | or we should stop copying them | 10:27 |
| noonedeadpunk | and btw they're totally broken | 10:28 |
| noonedeadpunk | since we do not tag them | 10:28 |
| noonedeadpunk | So I think we don't need them, and instead I should ask about how to stop publushig release notes for roles on https://docs.openstack.org/releasenotes/ | 10:29 |
| jrosser | anyway, in terms of rc2 or go for stable there is probably not much value in rc2 | 10:34 |
| jrosser | spatel is already running the rc and major issues would have come up there | 10:34 |
| jrosser | and anything broken needs to be fixed on master and backported anyway so it really makes little difference tbh | 10:35 |
| *** mjwales has joined #openstack-ansible | 11:37 | |
| noonedeadpunk | well, in rc1 ansible bootstrap is broken for sure | 11:38 |
| noonedeadpunk | but probably agree with that is does not make a lot of sense | 11:38 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: zun: set configuration where compute host shared with nova https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/769116 | 11:39 |
| mjwales | mgariepy: we managed to get the Ceph cluster back online by manually mounting the OSDs then restarting the ceph-osd@ service. However, they still fail to mount if we reboot the node. The only thing that looks strange in our setup is the addition of four new compute nodes. These nodes are part of the ceph cluster and use LVM whereas the rest of the ceph cluster is using bluestore on disks. Not sure why there | 11:41 |
| mjwales | is a difference as they were provisioned using the same playbooks when we upgraded the cluster from Stein -> Train. | 11:41 |
| noonedeadpunk | wehat is ceph version you're running? | 11:41 |
| mjwales | noonedeadpunk: Nautilus | 11:41 |
| noonedeadpunk | was you upgrading to it from Luminous or smth? | 11:42 |
| noonedeadpunk | as eventually ceph-volume should manage osds since Nautilus | 11:42 |
| mjwales | We upgraded from Xenial to Bionic and at the same time rocky -> stein -> train. As part of that we upgrade Ceph to Luminous then to Nautilus | 11:43 |
| noonedeadpunk | try running `ceph-volume simple scan` and `ceph-volume simple activate --all` after that | 11:44 |
| noonedeadpunk | I think this should be done on each node with osds | 11:44 |
| noonedeadpunk | ie step 7 from upgrade guide https://docs.ceph.com/en/latest/releases/nautilus/#instructions | 11:45 |
| mjwales | Ahh looks like that step was missed. We upgrade by running the ceph-nautilus-upgrades.yml that someone linked to off this IRC a while back | 11:48 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Gather minimal facts to use ansible_distribution https://review.opendev.org/c/openstack/openstack-ansible/+/769913 | 12:06 |
| *** mjwales has quit IRC | 12:31 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/victoria: Use systemd sockets for libvirt https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/769746 | 13:08 |
| *** rfolco has joined #openstack-ansible | 13:09 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Add support for kata container runtime https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/769555 | 13:30 |
| *** miloa has quit IRC | 13:34 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/ussuri: Use systemd sockets for libvirt https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/769747 | 13:35 |
| *** miloa has joined #openstack-ansible | 13:37 | |
| *** admin0 has joined #openstack-ansible | 13:38 | |
| *** jawad_axd has joined #openstack-ansible | 14:03 | |
| *** spatel has joined #openstack-ansible | 14:07 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/769925 | 14:20 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/769925 | 14:22 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/769925 | 14:23 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/769925 | 14:25 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/769925 | 14:26 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/769929 | 14:32 |
| *** rpittau is now known as rpittau|afk | 14:46 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/769932 | 14:47 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/769933 | 14:51 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/769935 | 14:55 |
| *** omid-babaei has joined #openstack-ansible | 15:01 | |
| *** omid-babaei has quit IRC | 15:02 | |
| *** klamath_atx has joined #openstack-ansible | 15:07 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/769938 | 15:08 |
| *** pcaruana has quit IRC | 15:13 | |
| *** pcaruana has joined #openstack-ansible | 15:13 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/769940 | 15:14 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/769941 | 15:18 |
| *** ivve has joined #openstack-ansible | 15:29 | |
| *** miloa has quit IRC | 15:32 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/769944 | 15:36 |
| *** pto has quit IRC | 15:36 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/769945 | 15:39 |
| *** ivve has quit IRC | 15:39 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/769946 | 15:41 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/769947 | 15:44 |
| *** macz_ has joined #openstack-ansible | 15:46 | |
| *** macz_ has joined #openstack-ansible | 15:47 | |
| *** spatel has quit IRC | 15:48 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/769951 | 16:00 |
| *** spatel has joined #openstack-ansible | 16:04 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/769952 | 16:07 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_placement master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/769954 | 16:13 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_panko master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/769955 | 16:15 |
| spatel | jrosser: hey! | 16:16 |
| spatel | jrosser: i remember you gave me nice inventory group by racks, do you have that anywhere now? | 16:17 |
| spatel | I am trying to look into my notes but not able to find. | 16:17 |
| spatel | jrosser: nevermind i found it :) | 16:22 |
| jrosser | spatel: ah good - not back at work till next week | 16:23 |
| spatel | http://paste.openstack.org/show/770649/ | 16:23 |
| jrosser | would be nice to make an example for this though with yaml anchors | 16:23 |
| spatel | how long paste.openstack.org keep data ? this is very old post :) | 16:23 |
| spatel | that would be great to put it in example so other people can enjoy! | 16:24 |
| jrosser | i think it keeps it forever | 16:24 |
| spatel | oh! | 16:24 |
| spatel | anyway! happy new year and enjoy your holiday! | 16:25 |
| jrosser | there are example files in etc/openstack_deploy dir, so we could easily take one of those and make an anchors example | 16:25 |
| spatel | totally! i will try to put it there :) | 16:27 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/769956 | 16:36 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/769958 | 16:47 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/769959 | 16:55 |
| *** gyee has joined #openstack-ansible | 16:58 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tacker master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/769960 | 16:58 |
| mgariepy | https://yaml.org/type/merge.html are quite useful also. | 17:01 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/769964 | 17:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova stable/victoria: Use systemd sockets for libvirt https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/769746 | 17:03 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build stable/victoria: Fix tests for Centos8 EPEL repo paths https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/769585 | 17:08 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/769965 | 17:09 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Use internal endpoint for tempestconf https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/769966 | 17:17 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/769935 | 17:18 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/769951 | 17:31 |
| ThiagoCMC | The TASK '[lxc_hosts : Disable and stop lxc-net]' is failing, error: 'FAILED! => {"changed": false, "msg": "Service is in unknown state", "status": {}}' - Any idea why? | 17:42 |
| ThiagoCMC | I'm just tyring to re-run the `openstack-ansible setup-everything.yml` - OSA Ussuri / Ubuntu 20.04 (with HWE from Canonical Linux 5.8) | 17:42 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/769929 | 17:44 |
| ThiagoCMC | The command `systemctl status lxc-net` returns = 'Loaded: masked' / 'Active: inactive (dead)' | 17:44 |
| *** andrewbonney has quit IRC | 18:01 | |
| *** jawad_axd has quit IRC | 18:05 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Cleanup after service variables merged https://review.opendev.org/c/openstack/openstack-ansible/+/769974 | 18:07 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/769935 | 18:09 |
| *** itsjg has quit IRC | 18:32 | |
| *** yann-kaelig has joined #openstack-ansible | 18:45 | |
| *** maharg101 has quit IRC | 19:05 | |
| *** jawad_axd has joined #openstack-ansible | 19:38 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova stable/ussuri: Use systemd sockets for libvirt https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/769747 | 19:45 |
| *** itsjg has joined #openstack-ansible | 20:17 | |
| *** yann-kaelig has quit IRC | 20:40 | |
| *** maharg101 has joined #openstack-ansible | 21:02 | |
| *** maharg101 has quit IRC | 21:06 | |
| *** rfolco has quit IRC | 21:11 | |
| *** cloudnull has quit IRC | 21:18 | |
| *** jawad_axd has quit IRC | 21:20 | |
| *** cloudnull has joined #openstack-ansible | 21:28 | |
| *** spatel has quit IRC | 22:37 | |
| ThiagoCMC | Damn... The task I mentioned in my previous message, "TASK [lxc_hosts : Disable and stop lxc-net]", is broken with Linux Kernel 5.8 (Ubuntu 20.04 package `linux-generic-hwe-20.04`). I just downgraded the kernel and the task passed! | 23:02 |
| *** ThiagoCMC has quit IRC | 23:12 | |
| *** ThiagoCMC has joined #openstack-ansible | 23:24 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Gather minimal facts to use ansible_distribution https://review.opendev.org/c/openstack/openstack-ansible/+/769913 | 23:45 |
| *** spatel has joined #openstack-ansible | 23:52 | |
| *** spatel has quit IRC | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!