| *** maharg101 has quit IRC | 00:03 | |
| *** yolanda has quit IRC | 00:38 | |
| *** yolanda has joined #openstack-ansible | 00:39 | |
| *** cshen has joined #openstack-ansible | 00:48 | |
| *** cshen has quit IRC | 00:52 | |
| *** gyee has quit IRC | 01:04 | |
| *** cshen has joined #openstack-ansible | 01:55 | |
| *** maharg101 has joined #openstack-ansible | 01:59 | |
| *** cshen has quit IRC | 02:00 | |
| *** d34dh0r53 has quit IRC | 02:03 | |
| *** maharg101 has quit IRC | 02:06 | |
| *** rh-jelabarre has quit IRC | 02:09 | |
| *** nurdie has joined #openstack-ansible | 02:26 | |
| *** nurdie has quit IRC | 02:30 | |
| *** d34dh0r53 has joined #openstack-ansible | 02:32 | |
| *** MickyMan77 has joined #openstack-ansible | 03:32 | |
| *** MickyMan77 has quit IRC | 03:40 | |
| *** cshen has joined #openstack-ansible | 03:55 | |
| *** cshen has quit IRC | 04:00 | |
| *** shyamb has joined #openstack-ansible | 04:03 | |
| *** maharg101 has joined #openstack-ansible | 04:03 | |
| *** maharg101 has quit IRC | 04:08 | |
| *** MickyMan77 has joined #openstack-ansible | 04:10 | |
| *** idlemind has quit IRC | 04:16 | |
| *** idlemind_ has joined #openstack-ansible | 04:16 | |
| *** shyamb has quit IRC | 04:18 | |
| *** MickyMan77 has quit IRC | 04:18 | |
| *** nurdie has joined #openstack-ansible | 04:27 | |
| *** nurdie has quit IRC | 04:31 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-ansible | 04:33 | |
| *** zigo has quit IRC | 04:38 | |
| *** shyamb has joined #openstack-ansible | 04:46 | |
| *** cshen has joined #openstack-ansible | 04:48 | |
| *** MickyMan77 has joined #openstack-ansible | 04:51 | |
| *** cshen has quit IRC | 04:53 | |
| *** MickyMan77 has quit IRC | 04:59 | |
| *** shyam89 has joined #openstack-ansible | 05:05 | |
| *** shyamb has quit IRC | 05:08 | |
| *** cshen has joined #openstack-ansible | 05:15 | |
| *** cshen has quit IRC | 05:19 | |
| *** suryasingh has joined #openstack-ansible | 05:30 | |
| *** MickyMan77 has joined #openstack-ansible | 05:38 | |
| *** MickyMan77 has quit IRC | 05:41 | |
| *** itandops has joined #openstack-ansible | 05:44 | |
| *** miloa has joined #openstack-ansible | 05:57 | |
| *** itandops has quit IRC | 05:57 | |
| *** maharg101 has joined #openstack-ansible | 06:04 | |
| *** pcaruana has joined #openstack-ansible | 06:05 | |
| *** maharg101 has quit IRC | 06:09 | |
| BlackFX | is it normal for rabbit to sit constantly at 80%+ CPU? | 06:24 |
|---|---|---|
| *** shyam89 has quit IRC | 06:26 | |
| janno_ | BlackFX: yes | 06:26 |
| *** nurdie has joined #openstack-ansible | 06:28 | |
| BlackFX | Okay | 06:28 |
| BlackFX | I seem to have a really slow horizon | 06:28 |
| janno_ | BlackFX: This is due to BEAM | 06:31 |
| janno_ | BlackFX: https://stressgrid.com/blog/beam_cpu_usage/ | 06:31 |
| *** nurdie has quit IRC | 06:32 | |
| BlackFX | memcached had too many open files | 06:36 |
| *** noonedeadpunk has quit IRC | 06:37 | |
| *** shyamb has joined #openstack-ansible | 06:42 | |
| *** noonedeadpunk has joined #openstack-ansible | 06:46 | |
| *** djhankb has quit IRC | 07:09 | |
| *** djhankb has joined #openstack-ansible | 07:10 | |
| *** maharg101 has joined #openstack-ansible | 07:10 | |
| *** dirk has quit IRC | 07:13 | |
| *** shyamb has quit IRC | 07:14 | |
| *** dirk has joined #openstack-ansible | 07:15 | |
| *** cshen has joined #openstack-ansible | 07:16 | |
| jrosser | morning | 07:19 |
| jrosser | noonedeadpunk: if you have an idea on this - i don't really see why it still breaks https://review.opendev.org/#/c/754722 | 07:19 |
| noonedeadpunk | morning jrosser:) | 07:19 |
| *** shyamb has joined #openstack-ansible | 07:23 | |
| noonedeadpunk | ok, so aodh and panko fails due to gnocchi patch | 07:23 |
| noonedeadpunk | so in order to revert we need 754722 merged | 07:23 |
| noonedeadpunk | and 754722 doesn't seem to have DB creation delegated | 07:23 |
| *** shyam89 has joined #openstack-ansible | 07:26 | |
| jrosser | i was expecting to see 754722 pass to give confidence it was OK to merge the aodh and panko patches without CI | 07:27 |
| *** shyamb has quit IRC | 07:29 | |
| *** shyam89 has quit IRC | 07:31 | |
| noonedeadpunk | let's probably try re-checking it, but dunno if that gonna work | 07:32 |
| noonedeadpunk | as seems like it didn't pull changes for me | 07:32 |
| jrosser | yeah, or there is some other underlying problem in aodh role that i don't spot | 07:34 |
| *** andrewbonney has joined #openstack-ansible | 07:44 | |
| *** tosky has joined #openstack-ansible | 07:45 | |
| *** cshen has quit IRC | 07:45 | |
| *** jbadiapa has joined #openstack-ansible | 07:49 | |
| *** cshen has joined #openstack-ansible | 08:00 | |
| *** shyamb has joined #openstack-ansible | 08:02 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use nodepool epel mirror in CI for systemd-networkd package https://review.opendev.org/754706 | 08:03 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Require the use of community.rabbitmq ansible collection https://review.opendev.org/754657 | 08:06 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Add user defined collections https://review.opendev.org/753411 | 08:06 |
| *** cshen has quit IRC | 08:26 | |
| *** mensis has joined #openstack-ansible | 08:27 | |
| *** djhankb has quit IRC | 08:28 | |
| *** djhankb has joined #openstack-ansible | 08:28 | |
| *** nurdie has joined #openstack-ansible | 08:29 | |
| *** cshen has joined #openstack-ansible | 08:33 | |
| *** sshnaidm|off is now known as sshnaidm | 08:33 | |
| *** nurdie has quit IRC | 08:34 | |
| jrosser | i updated the linter version, and it doesnt like this https://github.com/openstack/openstack-ansible-tests/blob/master/test-prepare-host.yml#L224 | 08:45 |
| jrosser | and i agree :) | 08:46 |
| noonedeadpunk | in terms of replace?:) | 08:48 |
| jrosser | 'Don't compare to literal True/False' | 08:48 |
| noonedeadpunk | ah | 08:48 |
| jrosser | it's taken me a while just to work out what an earth it is doing | 08:48 |
| noonedeadpunk | lol, yes | 08:48 |
| noonedeadpunk | wait, really... | 08:51 |
| noonedeadpunk | what are we appending here xD | 08:51 |
| jrosser | yes exactly | 08:51 |
| jrosser | it's quite special | 08:51 |
| jrosser | i think it makes a list of 'true true true true' | 08:51 |
| jrosser | my head hurts now! | 08:52 |
| noonedeadpunk | and we're asserting list of trues?:) | 08:52 |
| jrosser | yup | 08:52 |
| noonedeadpunk | I want to unsee it | 08:54 |
| jrosser | i need to go sit in a quiet dark place for a while now | 08:56 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-galera_server master: DNM Try to understand what's wrong in CI https://review.opendev.org/754610 | 08:57 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use nodepool epel mirror in CI for systemd-networkd package https://review.opendev.org/754706 | 09:01 |
| openstackgerrit | James Gibson proposed openstack/openstack-ansible-ops master: Change ansible tests to prefer Python3 over Python2 in vitualenv https://review.opendev.org/751773 | 09:03 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts stable/ussuri: Use xt_MASQUERADE instead of ipt_MASQUERADE for kernels > 5.2 https://review.opendev.org/754833 | 09:09 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-galera_server master: DNM Try to understand what's wrong in CI https://review.opendev.org/754610 | 09:27 |
| *** arxcruz has quit IRC | 09:35 | |
| *** shyamb has quit IRC | 09:45 | |
| *** arxcruz has joined #openstack-ansible | 09:46 | |
| *** mensis has quit IRC | 10:05 | |
| *** nurdie has joined #openstack-ansible | 10:13 | |
| *** nurdie has quit IRC | 10:18 | |
| masterpe | Cinder-volume had too many open files, I think it was default (1024 4096), we change it by creating a files /etc/systemd/system/cinder-volume.service.d/limits.conf with content LimitNOFILE=16384. | 10:23 |
| masterpe | is LimitNOFILE managed by openstack-ansible? | 10:24 |
| *** shyamb has joined #openstack-ansible | 10:43 | |
| *** nurdie has joined #openstack-ansible | 10:44 | |
| *** nurdie has quit IRC | 10:48 | |
| *** shyam89 has joined #openstack-ansible | 10:51 | |
| *** miloa has quit IRC | 10:52 | |
| *** shyamb has quit IRC | 10:53 | |
| noonedeadpunk | jrosser: galera is super weird... like it fails always 2nd container, and service restart just got stuck locally. but in case of container restart it just spawns and joins cluster without issues... | 10:55 |
| noonedeadpunk | I really not sure what's wrong with it... | 10:55 |
| *** cshen has quit IRC | 10:59 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-galera_server master: DNM Try to understand what's wrong in CI https://review.opendev.org/754610 | 10:59 |
| *** cshen has joined #openstack-ansible | 11:03 | |
| *** djhankb has quit IRC | 11:07 | |
| *** djhankb has joined #openstack-ansible | 11:08 | |
| *** cshen has quit IRC | 11:10 | |
| *** shyam89 has quit IRC | 11:12 | |
| *** cshen has joined #openstack-ansible | 11:13 | |
| *** shyamb has joined #openstack-ansible | 11:22 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_mount master: Install required packages for NFS/CephFS mounts https://review.opendev.org/754978 | 11:27 |
| openstackgerrit | James Gibson proposed openstack/openstack-ansible-ops master: Change ansible tests to prefer Python3 over Python2 in vitualenv https://review.opendev.org/751773 | 11:29 |
| jrosser | masterpe: we already make limits.conf for galera and memcached, plus also for the swift service like this https://opendev.org/openstack/openstack-ansible-os_swift/src/branch/master/defaults/main.yml#L346 | 11:30 |
| jrosser | if you think that we need to increase the default for cinder then we can do something similar | 11:30 |
| jrosser | for the time being you can make a config override of this variable https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/defaults/main.yml#L330 | 11:31 |
| jrosser | and add whatever you need to the cinder volume systemd unit | 11:31 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_mount master: Install required packages for NFS/CephFS mounts https://review.opendev.org/754978 | 11:41 |
| *** shyamb has quit IRC | 11:49 | |
| *** rh-jelabarre has joined #openstack-ansible | 11:53 | |
| *** rh-jelabarre has quit IRC | 11:53 | |
| *** rh-jelabarre has joined #openstack-ansible | 11:54 | |
| *** shyamb has joined #openstack-ansible | 11:58 | |
| jrosser | noonedeadpunk: in the galera functional test, do we run it in serial container1/2/3 or all at the same time? | 12:02 |
| *** shyam89 has joined #openstack-ansible | 12:03 | |
| *** cshen has quit IRC | 12:04 | |
| *** shyamb has quit IRC | 12:06 | |
| noonedeadpunk | all at the same time | 12:07 |
| noonedeadpunk | I was thinking about serial tbh | 12:08 |
| noonedeadpunk | as we run serial in prod | 12:08 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Update ansible-linti==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/754982 | 12:11 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/754982 | 12:11 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use nodepool epel mirror in CI for systemd-networkd package https://review.opendev.org/754706 | 12:17 |
| *** rfolco|ruck has joined #openstack-ansible | 12:22 | |
| *** shyam89 has quit IRC | 12:29 | |
| *** shyam89 has joined #openstack-ansible | 12:29 | |
| *** shyam89 has quit IRC | 12:30 | |
| *** shyam89 has joined #openstack-ansible | 12:31 | |
| snadge | noonedeadpunk: how did you know about jumbo frames and no route to host issue i was having? | 12:31 |
| noonedeadpunk | so it was them?:) | 12:31 |
| snadge | i've done some testing since.. and local to container itself, i dont seem to have any connectivity issues | 12:31 |
| snadge | but from the controller to the container.. i get no route to host, which doesn't make sense to me | 12:31 |
| noonedeadpunk | from controller to container within same controler or container on other controller? | 12:32 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Remove python3 packages from bindep.txt https://review.opendev.org/754987 | 12:32 |
| snadge | i get no route to host from other hosts on the same br-mgmt network | 12:32 |
| snadge | which could be a firewalling issue or anything.. i honestly hate networks and its not my thing | 12:33 |
| snadge | what i do know is.. i shouldn't get an intermittent fault no route to network which then disappears from the controller to the neutron container | 12:33 |
| snadge | i'd love to know why that happens, and i suspect it has something to do with using centos 7 | 12:34 |
| noonedeadpunk | eventually you shouldn't have it at all:) | 12:34 |
| *** shyam89 has quit IRC | 12:34 | |
| noonedeadpunk | but in case of jubmo frames I had to set lc container interfaces MTU specificly to 1450 | 12:34 |
| *** shyam89 has joined #openstack-ansible | 12:34 | |
| snadge | interesting | 12:35 |
| noonedeadpunk | I guess I did that even in lxc config or smth like that | 12:35 |
| mgariepy | jumbo frame is a mess;) hahaah | 12:35 |
| snadge | if that fixes it i owe you a carton of beers | 12:35 |
| openstackgerrit | James Gibson proposed openstack/openstack-ansible-ops master: Change ansible tests to prefer Python3 over Python2 in vitualenv https://review.opendev.org/751773 | 12:38 |
| *** yolanda has quit IRC | 12:38 | |
| noonedeadpunk | snadge: I probably set `lxc_container_default_mtu` for that | 12:39 |
| *** yolanda has joined #openstack-ansible | 12:39 | |
| noonedeadpunk | or maybe set it in container_networks.... | 12:40 |
| noonedeadpunk | can't really recall nowadays... | 12:40 |
| jrosser | mtu and no route to host is a really odd combination | 12:41 |
| jrosser | i wonder what 'ip r' has to say | 12:41 |
| snadge | it shows the routes obviously.. and the issue is with the br-mgmt network (apparently) | 12:42 |
| snadge | at the time that it has the problem maybe that route is missing.. is what i should check | 12:43 |
| *** nurdie has joined #openstack-ansible | 12:45 | |
| *** shyam89 has quit IRC | 12:46 | |
| snadge | ok i have confirmed the route is still there according to ip r | 12:47 |
| snadge | but i got "no route to host" from telnet to the neutron backend server port (9696) immediately prior to and after that | 12:47 |
| snadge | and then however many seconds later.. it connects and starts working again. frustratign | 12:47 |
| noonedeadpunk | I think what may happen here is that vlan interface with mtu 1450 is part of the br-mgmt, but lxc tries to use mtu 1500 by default | 12:49 |
| *** nurdie has quit IRC | 12:49 | |
| snadge | im curious to try changing that to 1450.. where can i do that? | 12:50 |
| snadge | i mean.. thats a simple thing to try right | 12:50 |
| noonedeadpunk | yep, you can either set in lxc directly, or if doing it normally, you should put into /var/lib/lxc/container_name/eth1.ini or smth like this | 12:51 |
| noonedeadpunk | and restart container | 12:51 |
| jrosser | theres no encapsulation going on with the lxc bridges so really the whole thing should be 1500-mtu transparent all they way | 12:52 |
| noonedeadpunk | or, set `lxc_container_default_mtu` in user_variables, and run some playbook... like containers-lxc-create.yml | 12:52 |
| jrosser | you can do 'ping -M do -s <number-28> <destination-ip>' and fiddle around with the value of 'number' to find the actual mtu that will pass | 12:54 |
| jrosser | the 28 accounts for ICMP and ethernet header | 12:55 |
| snadge | i just changed neutron to 1450 from 1500 | 12:55 |
| snadge | and rebooted the container | 12:55 |
| *** shyamb has joined #openstack-ansible | 12:56 | |
| jrosser | so 1450 would generally be the setting in neutron when your project network type is vxlan | 12:56 |
| jrosser | that means that the packets created by your VM will be small enough to fit inside a vxlan packet and still be smaller than 1500 | 12:57 |
| *** shyamb has quit IRC | 12:57 | |
| *** shyamb has joined #openstack-ansible | 12:57 | |
| snadge | the mtu didn't seem to apply so im just rebooting the entire controller ;) | 12:59 |
| *** shyamb has quit IRC | 12:59 | |
| *** shyamb has joined #openstack-ansible | 12:59 | |
| jrosser | ip -d link show <- that'll show you what you've got | 12:59 |
| snadge | thats what i should've done yeah.. this will take about 5-10 minutes to come back up (blade server) | 12:59 |
| jrosser | so "right answer" here depends what you want to happen | 13:01 |
| jrosser | if you need to pass vxlan traffic over a 1500mtu underlying network then the neutron networks need to be 1450 | 13:01 |
| snadge | but that doesn't really make sense why it would work intermittently | 13:02 |
| jrosser | and that can propagate across interfaces and affect other stuff if you share the bridges with your containers and so on | 13:02 |
| *** shyamb has quit IRC | 13:04 | |
| *** shyamb has joined #openstack-ansible | 13:05 | |
| *** cshen has joined #openstack-ansible | 13:06 | |
| *** shyamb has quit IRC | 13:07 | |
| *** shyamb has joined #openstack-ansible | 13:08 | |
| snadge | ok well the neutron container has a 1450 mtu now and its still doing the layer 4 no route thing | 13:15 |
| *** shyamb has quit IRC | 13:15 | |
| snadge | im starting to wonder if i should just give someone else a login to this server somehow via a port forward or whatever | 13:16 |
| *** nurdie has joined #openstack-ansible | 13:31 | |
| snadge | i need to find a resolution on this problem ideally within the next week or so.. i've at least narrowed it down to some kind of lxc networking issue | 13:33 |
| snadge | since the problem is easily reproducible between the lxc host, and the container which is running on the same blade | 13:34 |
| snadge | if i telnet to localhost within the container.. it always connects and i never get the no route to host issue | 13:34 |
| jrosser | noonedeadpunk: are we missing a release for the most recent set of SHA bumps? | 13:42 |
| noonedeadpunk | we do as we had a bug there, which we closed right afterwards, but I couldn't recall what exactly it is | 13:43 |
| *** sshnaidm has quit IRC | 13:48 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Updated from OpenStack Ansible Tests https://review.opendev.org/754159 | 13:55 |
| *** sshnaidm has joined #openstack-ansible | 13:58 | |
| *** nurdie_ has joined #openstack-ansible | 13:59 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/754982 | 14:02 |
| *** nurdie has quit IRC | 14:02 | |
| *** pcaruana has quit IRC | 14:15 | |
| *** pcaruana has joined #openstack-ansible | 14:24 | |
| *** spatel has joined #openstack-ansible | 14:24 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/754982 | 14:24 |
| spatel | jrosser: spotz does this make sense to you guys? I have created my blog for octavia networking - https://satishdotpatel.github.io//openstack-ansible-octavia/ | 14:25 |
| jrosser | spatel: that is cool - nice diagram | 14:31 |
| jrosser | i am guessing you don't use neutron l3 agent? | 14:31 |
| spatel | jrosser: no i have my Cisco ASA is my gateway so pure vlan base provider | 14:32 |
| jrosser | and the DHCP, does neutron do that for you? | 14:33 |
| *** theintern has joined #openstack-ansible | 14:33 | |
| jrosser | like amphora IP | 14:33 |
| spatel | Yes everything neutron does that for me. | 14:33 |
| spatel | i get IP address on amphora (dual IP, 1. mgmt and 2. vm traffic) | 14:34 |
| jrosser | right - so there will be something like qdhcp namespace on the controller also talking to eth14 | 14:34 |
| jrosser | well actually i'm not sure how it'll be wired actually, but it would be great to add that too | 14:35 |
| openstackgerrit | James Gibson proposed openstack/openstack-ansible-ops master: Change ansible tests to prefer Python3 over Python2 in vitualenv https://review.opendev.org/751773 | 14:38 |
| spatel | jrosser: let me find that out how does my neutron DHCP agent get wire-up with br-lbaas (currently my lab is broken and trying to bring it up) | 14:39 |
| jrosser | spatel: cool - it's good to add becasue theres three things in play, the octavia container, neutron dhcp and the wiring to the amphora | 14:40 |
| spatel | jrosser: i do have namespace for VLAN 27 subnet on controller node running qdhcp | 14:41 |
| spotz | spatel: I can't help myself... no did here and configured - how did i configure:) | 14:41 |
| spotz | spatel: same changes here - how did i wire - how I wired | 14:42 |
| spatel | let me show you how | 14:42 |
| spotz | spatel: i didn’t created - I didn't create | 14:43 |
| spotz | Help I'm doing reviews not in Gerrit:) | 14:43 |
| spatel | here you go - http://paste.openstack.org/show/798522/ | 14:44 |
| spatel | tap interface tapbbe749e9-8f is connected to vlan.27 and namespace for qdhcp | 14:45 |
| spatel | spotz: I am also kinda new for octavia so lets clear all doubt here and then we will write up nice official doc with good example for new folks.. | 14:47 |
| spatel | jrosser: does that make sense to you - http://paste.openstack.org/show/798522/ | 14:47 |
| CeeMac | snadge: its definitely worth performing the ping exercise jrosser mentioned to try and work out the maximum MTU | 14:48 |
| spotz | spatel: I'll put what you have in a local doc and clean it up when I get home in a bit, if you want PM my your email to send it back to you | 14:48 |
| jrosser | spatel: i think so - neutron has created vlan.27, and you can see that the tap name matches up with the ns name at the top of your paste | 14:48 |
| CeeMac | I've had all kinds of crazy issues in various network environments where the MTU has been out of alignment. | 14:49 |
| spatel | jrosser: spotz i will add DHCP namespace in diagram also so it will be little clear to understand how dhcp handing over lbaas-mgmt ip | 14:49 |
| jrosser | CeeMac: i was wondering if the issue there was using br-vlan for the mgmt traffic as well as the neutron vlans | 14:50 |
| jrosser | neutron will fiddle with the MTU on the interfaces and that could easily mess up other things that you use the bridge for if they don't account for the changed MTU | 14:51 |
| CeeMac | jrosser: yeah. makes sense. at first I thought it might have been similar to the issues i'd seen trying to run controller on vmware, but then its intermittent whereas I had constant no route to host | 14:52 |
| CeeMac | mtu issues are haunting me at the moment it seems | 14:52 |
| jrosser | well, if it's not dns it'll be mtu :/ | 14:52 |
| CeeMac | +1 | 14:55 |
| *** theintern has quit IRC | 14:58 | |
| jrosser | spatel: also in /etc/openstack_deploy/openstack_user_config.yml do you have a used_ips section keeping the containers out of 172.27.40.200-172.27.40.250 ? | 14:58 |
| spatel | yes i do but i missed that in my doc | 14:59 |
| spatel | i will add that | 14:59 |
| jrosser | quite a small range btw - only 50 amphora ip there out of a whole /24 | 14:59 |
| spatel | its my lab :) | 15:00 |
| spatel | in production i have /21 range | 15:00 |
| jrosser | ah cool | 15:01 |
| spatel | now i am build new datacenter using VxLAN+EVPN (spine-leaf) and going to run octavia and senlin in production there so doing all preliminary exercise in lab. | 15:02 |
| spatel | 200 node private cloud. | 15:03 |
| CeeMac | spatel: nice :D | 15:03 |
| CeeMac | what hardware are you running that on switch/router wise? | 15:03 |
| spatel | I am planning to to make 6 node controller ( 3 node for all API and other 3 nodes for shared services like mysql, rabbitmq etc..) | 15:04 |
| spatel | We are using Cisco nexus 9336-FX2 for spine and Cisco nexus 9396PX for leaf | 15:04 |
| jrosser | ^ snap | 15:04 |
| jrosser | i have evpn on 9336-FX2 here | 15:04 |
| jrosser | also same split of 3 x infra / 3x shared nodes | 15:05 |
| CeeMac | haven't looked at Nexus switches for a while | 15:05 |
| spatel | that switches are beast, it can support 10G to 100G :) | 15:05 |
| CeeMac | guess they're pretty pricey :) | 15:05 |
| jrosser | do you have the evpn running? | 15:06 |
| spatel | jrosser: are running 3x shared nodes in LXC or metal way? | 15:06 |
| jrosser | lxc | 15:06 |
| spatel | cool that is what i am thinking | 15:06 |
| spatel | I am going to run OSFP+BGP style evpn for datacenter | 15:06 |
| spatel | currently practicing them on Cisco VIRL simulator :) + in my network lab | 15:07 |
| jrosser | i have octavia lbaas network on an evpn, works nicely | 15:07 |
| spatel | nice! | 15:08 |
| spatel | jrosser: do you guys run multicast for BUM traffic? | 15:08 |
| jrosser | yes | 15:08 |
| spatel | same here :) | 15:08 |
| jrosser | oh - you mean neutron VXLAN or the nxos stuff, becasue both | 15:09 |
| spatel | nxos VXLAN | 15:09 |
| jrosser | right yes using multicast for that | 15:09 |
| spatel | EVPN multicast :) | 15:09 |
| jrosser | then we also made TRM work inside the evpn tunnels for muticast applications | 15:09 |
| spatel | Are you guys using anycast gatewya on leaf? | 15:10 |
| jrosser | yes | 15:10 |
| jrosser | also leaf are VPC pair | 15:10 |
| spatel | same here | 15:10 |
| spatel | vPC for all TOR | 15:10 |
| spatel | jrosser: it would be great if you share your config if that is not very confidential :) i would like to see if i am following all best practices. | 15:11 |
| jrosser | we did eBGP for the underlay | 15:11 |
| spatel | i can share mine next week when i will start rolling out all config | 15:11 |
| spatel | I was thinking to use eBGP but its little complicated so i decided to use OSFP for underlay | 15:12 |
| jrosser | the config will be much smaller or OSPF | 15:12 |
| jrosser | *for | 15:12 |
| spatel | eBGP required lots of typing and peers while OSPF is very simple and copy paste conifig:) | 15:12 |
| spatel | eBGP is good for massive datacenter design but we have only 10 racks :) | 15:13 |
| noonedeadpunk | not _so_ small, considering potential growth | 15:13 |
| spatel | now we are planning to build multiple datacenter instead putting all eggs on single bucket. | 15:14 |
| spatel | soon planning to open another datacenter in EU and then Singapore | 15:15 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/754982 | 15:17 |
| spatel | jrosser: this is what i am trying to build in new DC - https://ibb.co/5vc4bn2 | 15:17 |
| jrosser | really 40g? :) | 15:18 |
| spatel | yes | 15:18 |
| spatel | why? | 15:18 |
| jrosser | 100G optics are cheap | 15:19 |
| jrosser | if you don't buy cisco..... | 15:20 |
| mgariepy | lol. | 15:20 |
| spatel | We already have lots of optics in stock so thought lets use them.. i don't think we will ever max out any link | 15:20 |
| CeeMac | laser2000 FTW | 15:20 |
| jrosser | oh well thats ok if you have them :) | 15:20 |
| spatel | We used fs.com mostly | 15:20 |
| CeeMac | i'd love to get 40GB DCIs | 15:21 |
| CeeMac | stuck at 10GB without large bag of cash | 15:21 |
| spatel | We also have 10G DCI | 15:22 |
| spatel | we don't have L2 stretch between DC | 15:22 |
| CeeMac | thats what EVPN is there for :p | 15:24 |
| jrosser | spatel: these have been good in the 9336 https://www.fs.com/uk/products/65210.html | 15:24 |
| CeeMac | we use MPLS-EVPN for DCI | 15:25 |
| spatel | jrosser: someday we will upgrade from 40G to 100G :) | 15:26 |
| openstackgerrit | James Gibson proposed openstack/openstack-ansible-ops master: Change ansible tests to prefer Python3 over Python2 in vitualenv https://review.opendev.org/751773 | 15:31 |
| spatel | jrosser: what is the configuration of you 3x shared infra nodes? cpu + memory etc.. | 15:31 |
| *** gyee has joined #openstack-ansible | 15:31 | |
| jrosser | they are fairly small, xeon-d 8C/16T with 64G | 15:32 |
| fridtjof[m] | I just upgraded from Stein to Train, and somehow the placement service broke | 15:36 |
| fridtjof[m] | When creating an instance, nova-scheduler complains: "Failed to retrieve allocation candidates from placement API for filters [...]" and gives me a 503 | 15:36 |
| fridtjof[m] | (with an HTML body) | 15:37 |
| fridtjof[m] | I checked both my (new, apparently) placement containers, and they're running fine | 15:37 |
| fridtjof[m] | both are UP in haproxy | 15:37 |
| *** mensis has joined #openstack-ansible | 15:38 | |
| fridtjof[m] | oh, seems like nova-scheduler is still trying to access nova_api_placement_front? | 15:38 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/755065 | 15:39 |
| jrosser | fridtjof[m]: there are i think some specific options on the upgrade scripts for placement S->T, did you see those? | 15:40 |
| fridtjof[m] | I just ran the run-upgrade script, assuming it would do all that's described in the major upgrade documentation | 15:40 |
| fridtjof[m] | I see it sets up the new placement containers, which exist now and seem to be fine | 15:41 |
| fridtjof[m] | what it definitely missed was removing the legacy backends from haproxy | 15:41 |
| fridtjof[m] | also, nova config seems to be untouched in that matter | 15:41 |
| jrosser | see https://github.com/openstack/openstack-ansible/blob/stable/train/scripts/run-upgrade.sh#L179 | 15:42 |
| jrosser | placement_migrate_flag=true is intended to make the changes you need | 15:43 |
| fridtjof[m] | yeah, I think i'll just rerun them and take a close look | 15:44 |
| jrosser | theres step-by-step at the bottom of here https://docs.openstack.org/openstack-ansible/train/admin/upgrades/major-upgrades.html | 15:48 |
| fridtjof[m] | ah, looking at the haproxy config it seems like there's both the old and new placement frontends defined with the same ports | 15:49 |
| fridtjof[m] | and because of order, the old one takes precedence | 15:49 |
| fridtjof[m] | just going to redeploy haproxy then | 15:49 |
| fridtjof[m] | yup, got that page open :) | 15:50 |
| spatel | jrosser: i do have 64GB + 2.5GHz cpu with 48 cores, i have 200 compute nodes so hope it should be enough | 15:50 |
| jrosser | fridtjof[m]: the haproxy role works by dropping lots of config fragments then using the ansible 'assemble' module to glue them together into one config file | 15:52 |
| jrosser | i'm not quite seeing at the moment where in the upgrade process the old placement frontend is removed | 15:52 |
| fridtjof[m] | yeah, it's just ignoring /etc/haproxy/conf.d/nova_api_placement | 15:53 |
| fridtjof[m] | I see the two steps generating config files and dropping files for non present services, but the list for that does not seem to contain nova_api_placement | 15:54 |
| jrosser | to remove it, the entry should be in the list of endpoints but state: absent | 15:57 |
| jrosser | then it gets deleted | 15:57 |
| dmsimard | pleasantly surprised to see upgrading to ansible 2.10 didn't seem to break much ? | 15:58 |
| jrosser | dmsimard: seems to be working out ok | 15:58 |
| dmsimard | that's neat considering the amount of changes under the hood | 15:58 |
| jrosser | would be interested to see if the whitespace/padding can be tightened up with 1000's of tasks | 15:59 |
| dmsimard | in the ara reports you mean ? | 15:59 |
| jrosser | yeah | 15:59 |
| dmsimard | yeah, definitely | 15:59 |
| jrosser | i futzed around a bit in the browser developer tools - just confirmed that i actually don't know what i'm doing :) | 15:59 |
| dmsimard | it's probably some <tr> css somewhere ¯\_(ツ)_/¯ | 16:00 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 16:01 |
| openstack | Meeting started Tue Sep 29 16:01:03 2020 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:01 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:01 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:01 |
| fridtjof[m] | jrosser: found the issue i think | 16:01 |
| noonedeadpunk | #topic office hours | 16:01 |
| *** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)" | 16:01 | |
| noonedeadpunk | \o/ | 16:01 |
| jrosser | o/ hello | 16:01 |
| noonedeadpunk | Ok, so telemetry failure? | 16:02 |
| noonedeadpunk | I'd say let's maybe trying to merge aodh at least? | 16:04 |
| jrosser | worst case we have to revert it | 16:05 |
| jrosser | i've not had opportunity to test that locally yet | 16:05 |
| noonedeadpunk | I think worst case we will have just another patch to fix it | 16:05 |
| jrosser | right, thats fine | 16:05 |
| jrosser | so thats this https://review.opendev.org/#/c/754791 | 16:06 |
| jrosser | followed by https://review.opendev.org/#/c/754720/ | 16:06 |
| noonedeadpunk | yep | 16:06 |
| noonedeadpunk | ok, then next thingis galera.... | 16:07 |
| noonedeadpunk | I tried to look into it and it fails in so many different ways.... | 16:07 |
| noonedeadpunk | When I deployed it locally it was passing 3 or 4 times in a row when I decided that it's ok | 16:08 |
| noonedeadpunk | in some cases there was smth weird with container, as service start was just hanging... | 16:09 |
| noonedeadpunk | so at the moment, we have 2 scenarios | 16:09 |
| *** nurdie has joined #openstack-ansible | 16:10 | |
| noonedeadpunk | 1st is old one, when one of the containers don't see address of another partner. and it's the issue of this specific member, and it goes back to ok state in case of restart | 16:10 |
| noonedeadpunk | while cluster is synced in this state | 16:11 |
| noonedeadpunk | 2nd case when one of the containers is really down and didn't get up. IN this case we should restart not containers which don't see neighboor but down member... | 16:12 |
| noonedeadpunk | And I dunno how to make ogic to make it work | 16:12 |
| noonedeadpunk | *logic | 16:12 |
| noonedeadpunk | From other side, we can add serial and probably forget about the issue at once | 16:12 |
| *** nurdie_ has quit IRC | 16:13 | |
| jrosser | for the first case do you think that the container networking is completely broken | 16:13 |
| noonedeadpunk | no, for the first 3 members are up and synced, but one of them show only 2 addresses in wsrep_incoming_addresses | 16:15 |
| noonedeadpunk | which doesn't affect anything functionally, except it's weird and our tests fail | 16:15 |
| jrosser | the functional test is kind of tech debt somehow | 16:17 |
| jrosser | we could have an integrated test with affinity=3 on the container | 16:17 |
| jrosser | then expand the galera role to have cluster status checks | 16:17 |
| noonedeadpunk | have no idea how to do the last part | 16:17 |
| noonedeadpunk | or just do cluster checks by default? | 16:19 |
| noonedeadpunk | or with some var passed? | 16:19 |
| noonedeadpunk | hm, yeah, might be | 16:19 |
| jrosser | see affinity on here https://github.com/openstack/openstack-ansible/blob/master/doc/source/admin/maintenance-tasks/containers.rst | 16:22 |
| jrosser | i never used this though.... maybe works!? | 16:22 |
| noonedeadpunk | yeah. I was not about affinity, but about how to extend role with tests) | 16:22 |
| noonedeadpunk | me too lol | 16:22 |
| jrosser | yes so it would be optional sanity checks i guess, you don't want that interfering when trying to rescue a broken galera cluster | 16:23 |
| jrosser | and some flag to make everything stop after setup-openstack | 16:23 |
| jrosser | *setup-infrastructure | 16:23 |
| openstackgerrit | James Gibson proposed openstack/openstack-ansible-ops master: Change ansible tests to prefer Python3 over Python2 in vitualenv https://review.opendev.org/751773 | 16:24 |
| noonedeadpunk | hm, yeah, makes sense | 16:24 |
| jrosser | noonedeadpunk: i have to head out for a bit but there is still a lot to go over for V release | 16:25 |
| jrosser | i sort of took over the PTG etherpad to track all these patches | 16:25 |
| jrosser | we needs the linters fixed for at least openstack-ansible-tests to land 2.10.1 patch there | 16:25 |
| noonedeadpunk | And I think it's about time to freeze master bumps? | 16:26 |
| noonedeadpunk | or at least switch master to victoria... | 16:26 |
| noonedeadpunk | so we don't start figting with W issues | 16:27 |
| *** djhankb has quit IRC | 16:33 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh master: Remove CI jobs to allow db setup patch to merge https://review.opendev.org/754791 | 16:33 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh master: Use the utility host for db setup tasks https://review.opendev.org/754720 | 16:33 |
| *** djhankb has joined #openstack-ansible | 16:33 | |
| jrosser | yes though we also start fighting requirements changes too as they’re based off the branch name | 16:36 |
| *** d34dh0r53 has quit IRC | 16:36 | |
| jrosser | noonedeadpunk: maybe an extra keyword on the scenario “infra” we could just run the first part of the deploy | 16:37 |
| *** d34dh0r53 has joined #openstack-ansible | 16:38 | |
| noonedeadpunk | jrosser: we can actually just to break here in case of some scenarios https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/gate-check-commit.sh#L188 | 16:41 |
| jrosser | right - perhaps a small step to getting rid of the functional tests | 16:42 |
| noonedeadpunk | yeah, I think I will try doing that tomorrow instead of trying to revive functional tests as is | 16:44 |
| jrosser | better value time I think | 16:44 |
| noonedeadpunk | yeah | 16:45 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/754982 | 16:47 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_mount master: Install required packages for NFS/CephFS mounts https://review.opendev.org/754978 | 16:48 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_mount master: Install required packages for NFS/CephFS mounts https://review.opendev.org/754978 | 16:49 |
| *** olivierbourdon38 has quit IRC | 16:51 | |
| *** olivierbourdon38 has joined #openstack-ansible | 16:52 | |
| * jrosser back | 16:57 | |
| noonedeadpunk | btw, mensis have completed fixing monasca role at least for train | 16:58 |
| jrosser | thats good - so long as we can keep on top of it | 16:59 |
| jrosser | like senlin CI already seems completely broken :( | 16:59 |
| noonedeadpunk | oh damn | 17:00 |
| * jrosser wish we had a better dashboard for periodic jobs | 17:00 | |
| jrosser | it's kind of easy if you're a one-repo project to look in zuul state | 17:00 |
| jrosser | but with so many it's just really hard | 17:01 |
| noonedeadpunk | yeah it is... | 17:01 |
| noonedeadpunk | but what I was going to say about monasca - we have retired roles | 17:01 |
| noonedeadpunk | and I was thinking about reviving it | 17:01 |
| noonedeadpunk | the thing was, that monasca had 2 repos - for service and agent | 17:01 |
| noonedeadpunk | and I was thinking if it's worth mmerging them now | 17:02 |
| noonedeadpunk | like we did for galera | 17:02 |
| jrosser | that makes sense, it's not unlike neutron or nova really | 17:02 |
| noonedeadpunk | point in separation might be, that agent installation can be provided to customers who know nothing about osa | 17:02 |
| jrosser | what does it create? | 17:03 |
| noonedeadpunk | I think it grabs data from vms? | 17:03 |
| noonedeadpunk | like prometheus expoter or smth... | 17:03 |
| mensis | its for grabbing metrics, and it has several plugins which including gathering metrics from vms | 17:04 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/755065 | 17:05 |
| noonedeadpunk | but the thing is, that monasca can be left without PTL and not sure about project future because of that... | 17:06 |
| noonedeadpunk | #endmeeting | 17:12 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://bit.ly/2SAcGAn" | 17:12 | |
| openstack | Meeting ended Tue Sep 29 17:12:48 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:12 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-09-29-16.01.html | 17:12 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-09-29-16.01.txt | 17:12 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-09-29-16.01.log.html | 17:12 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_keystone master: Fix keystone nginx behaviour https://review.opendev.org/754382 | 17:16 |
| *** ianychoi_ has joined #openstack-ansible | 17:17 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_keystone master: Fix keystone nginx behaviour https://review.opendev.org/754382 | 17:18 |
| *** cshen has quit IRC | 17:19 | |
| *** ianychoi has quit IRC | 17:20 | |
| *** andrewbonney has quit IRC | 17:32 | |
| *** maharg101 has quit IRC | 17:34 | |
| *** cyberpear has quit IRC | 17:36 | |
| *** PrinzElvis has quit IRC | 17:37 | |
| *** sri_ has quit IRC | 17:37 | |
| *** suryasingh has quit IRC | 17:37 | |
| *** mnaser has quit IRC | 17:37 | |
| *** fyx has quit IRC | 17:37 | |
| *** viks____ has quit IRC | 17:37 | |
| *** gixx has quit IRC | 17:37 | |
| *** gundalow has quit IRC | 17:37 | |
| *** jrosser has quit IRC | 17:38 | |
| *** mwhahaha has quit IRC | 17:38 | |
| *** mubix has quit IRC | 17:38 | |
| *** nicolasbock has quit IRC | 17:38 | |
| *** johnsom has quit IRC | 17:38 | |
| *** alanmeadows has quit IRC | 17:38 | |
| *** jungleboyj has quit IRC | 17:38 | |
| *** guilhermesp has quit IRC | 17:38 | |
| *** CeeMac has quit IRC | 17:38 | |
| *** Open10K8S has quit IRC | 17:38 | |
| *** gouthamr has quit IRC | 17:39 | |
| *** johnsom has joined #openstack-ansible | 17:40 | |
| *** mubix has joined #openstack-ansible | 17:47 | |
| *** jungleboyj has joined #openstack-ansible | 17:48 | |
| *** guilhermesp has joined #openstack-ansible | 17:48 | |
| *** fyx has joined #openstack-ansible | 17:48 | |
| *** sri_ has joined #openstack-ansible | 17:49 | |
| *** mwhahaha has joined #openstack-ansible | 17:49 | |
| *** nicolasbock has joined #openstack-ansible | 17:49 | |
| *** cyberpear has joined #openstack-ansible | 17:49 | |
| *** CeeMac has joined #openstack-ansible | 17:50 | |
| *** Open10K8S has joined #openstack-ansible | 17:50 | |
| *** mnaser has joined #openstack-ansible | 17:55 | |
| *** gundalow has joined #openstack-ansible | 17:58 | |
| *** mensis has quit IRC | 17:58 | |
| *** suryasingh has joined #openstack-ansible | 17:58 | |
| *** alanmeadows has joined #openstack-ansible | 17:59 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use nodepool epel mirror in CI for systemd-networkd package https://review.opendev.org/754706 | 18:00 |
| *** gixx has joined #openstack-ansible | 18:00 | |
| *** PrinzElvis has joined #openstack-ansible | 18:00 | |
| *** jrosser has joined #openstack-ansible | 18:05 | |
| *** nurdie has quit IRC | 18:05 | |
| fridtjof[m] | jrosser: i found the cause, but didnt want to interrupt the meeting | 18:08 |
| fridtjof[m] | Now i'm no longer on my desktop so i dont have the draft message i typed out | 18:08 |
| fridtjof[m] | But it's a commit between 20.1.5 and 20.1.6 affecting inventory/group_vars/haproxy/<something>.yml | 18:09 |
| jrosser | no worries - theres always tomorrow :) | 18:09 |
| *** spatel has quit IRC | 18:14 | |
| *** maharg101 has joined #openstack-ansible | 18:14 | |
| fridtjof[m] | Found it: https://opendev.org/openstack/openstack-ansible/commit/095bc436b7237ff3aa03d38d552a1e8a6e4859a7 | 18:15 |
| *** nurdie has joined #openstack-ansible | 18:19 | |
| *** maharg101 has quit IRC | 18:24 | |
| *** gouthamr__ has joined #openstack-ansible | 18:26 | |
| *** olivierbourdon38 has quit IRC | 18:36 | |
| *** olivierbourdon38 has joined #openstack-ansible | 18:38 | |
| masterpe | jrosser: about the systemd init_config_overrides and LimitNOFILE we currently have about 80 compute nodes and somehow we are hitting the limits. Cinder-volume is giving the "too many open files" error. We use Ceph as backend. I'm not sure why 4096 what is default is not enough. | 19:00 |
| *** spatel has joined #openstack-ansible | 19:03 | |
| jrosser | masterpe: ‘lsof’ might help see what it is | 19:14 |
| jrosser | we can certainly increase the default though if it’s too small | 19:14 |
| *** cshen has joined #openstack-ansible | 19:15 | |
| *** cshen has quit IRC | 19:20 | |
| *** gouthamr__ is now known as gouthamr | 19:39 | |
| *** tosky has quit IRC | 19:51 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible stable/train: Revert "Remove nova_api_placement from inventory" https://review.opendev.org/755117 | 20:17 |
| *** maharg101 has joined #openstack-ansible | 20:21 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update ansible-lint==4.3.5, flake8==3.8.3, bashate==2.0.0 https://review.opendev.org/755065 | 20:22 |
| *** maharg101 has quit IRC | 20:26 | |
| *** BlackFX has quit IRC | 20:42 | |
| *** spatel has quit IRC | 20:52 | |
| spotz | jrosser: I've got booth duty tonight poke with any reviews you need to get through | 20:53 |
| *** theintern has joined #openstack-ansible | 21:14 | |
| *** theintern has quit IRC | 21:14 | |
| *** cshen has joined #openstack-ansible | 21:15 | |
| *** cshen has quit IRC | 21:20 | |
| *** jbadiapa has quit IRC | 21:31 | |
| *** gundalow has quit IRC | 21:52 | |
| *** johnsom has quit IRC | 21:52 | |
| *** jungleboyj has quit IRC | 21:52 | |
| *** alanmeadows has quit IRC | 21:52 | |
| *** Open10K8S has quit IRC | 21:52 | |
| *** CeeMac has quit IRC | 21:52 | |
| *** fyx has quit IRC | 21:52 | |
| *** sri_ has quit IRC | 21:52 | |
| *** guilhermesp has quit IRC | 21:53 | |
| *** rpittau|afk has quit IRC | 21:53 | |
| *** johnsom has joined #openstack-ansible | 21:54 | |
| *** cyberpear has quit IRC | 21:54 | |
| *** PrinzElvis has quit IRC | 21:54 | |
| *** suryasingh has quit IRC | 21:54 | |
| *** alanmeadows has joined #openstack-ansible | 21:54 | |
| *** gixx has quit IRC | 21:54 | |
| *** gundalow has joined #openstack-ansible | 21:55 | |
| *** suryasingh has joined #openstack-ansible | 21:56 | |
| *** sri_ has joined #openstack-ansible | 21:56 | |
| *** jungleboyj has joined #openstack-ansible | 21:56 | |
| *** Open10K8S has joined #openstack-ansible | 21:56 | |
| *** fyx has joined #openstack-ansible | 21:56 | |
| *** rpittau|afk has joined #openstack-ansible | 21:57 | |
| *** PrinzElvis has joined #openstack-ansible | 21:57 | |
| *** cyberpear has joined #openstack-ansible | 21:57 | |
| *** guilhermesp has joined #openstack-ansible | 21:57 | |
| *** gixx has joined #openstack-ansible | 21:57 | |
| *** CeeMac has joined #openstack-ansible | 21:57 | |
| fridtjof[m] | jrosser: thanks for the proposal! | 22:16 |
| fridtjof[m] | I found another issue! | 22:16 |
| fridtjof[m] | when creating an instance, the linuxbridge agent on the corrresponding compute host is stuck with this error: | 22:16 |
| fridtjof[m] | 2020-09-29 22:13:44.231 1781 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-... - - - - -] Error in agent loop. Devices info: {'current': {'tap8e53ab18-3d', 'tape3d6613a-44', 'tapb1d81907-c2', 'tapd63475d5-04'}, 'timestamps': {'tap8e53ab18-3d': 43, 'tape3d6613a-44': 42, 'tapb1d81907-c2': 46, 'tapd63475d5-04': 44}, 'added': {'tap8e53ab18-3d', 'tape3d6613a-44', 'tapb1d81907-c2', 'tapd63475d5-04'}, | 22:16 |
| fridtjof[m] | 'removed': set(), 'updated': set()}: pyroute2.netlink.exceptions.NetlinkError: (13, 'Permission denied') | 22:16 |
| fridtjof[m] | from what i can see, the agent is running as user 'neutron', but it does all that through rootwrap? | 22:20 |
| *** maharg101 has joined #openstack-ansible | 22:22 | |
| *** MickyMan77 has joined #openstack-ansible | 22:23 | |
| *** maharg101 has quit IRC | 22:29 | |
| *** MickyMan77 has quit IRC | 22:31 | |
| *** nurdie has quit IRC | 22:34 | |
| fridtjof[m] | seems like my placement service's database is kind of broken :/ | 22:42 |
| fridtjof[m] | http://paste.openstack.org/show/798548/ | 22:45 |
| fridtjof[m] | shouldn't these two tables match? | 22:45 |
| fridtjof[m] | because nova-compute (on compute2) is now regularily giving me this: http://paste.openstack.org/show/798549/ | 22:47 |
| *** nurdie has joined #openstack-ansible | 22:50 | |
| *** nurdie has quit IRC | 22:55 | |
| *** ianychoi_ is now known as ianychoi | 23:00 | |
| *** klamath_atx has joined #openstack-ansible | 23:14 | |
| *** cshen has joined #openstack-ansible | 23:16 | |
| *** cshen has quit IRC | 23:20 | |
| *** nurdie has joined #openstack-ansible | 23:26 | |
| *** nurdie has quit IRC | 23:31 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!