| *** vesper has joined #openstack-ansible | 01:38 | |
| *** vesper11 has quit IRC | 01:39 | |
| *** gyee has quit IRC | 02:00 | |
| *** djhankb has joined #openstack-ansible | 02:05 | |
| *** rh-jelabarre has quit IRC | 03:13 | |
| *** raukadah is now known as chkumar|rover | 04:22 | |
| *** udesale has joined #openstack-ansible | 04:53 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:34 | |
| *** errr has quit IRC | 05:55 | |
| *** errr has joined #openstack-ansible | 05:56 | |
| *** errr has quit IRC | 06:05 | |
| *** errr has joined #openstack-ansible | 06:07 | |
| *** jbadiapa has joined #openstack-ansible | 06:51 | |
| *** southquist has joined #openstack-ansible | 06:55 | |
| *** bhyrted has quit IRC | 06:59 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_nova master: pre-install: check if path is defined https://review.opendev.org/705892 | 07:21 |
|---|---|---|
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_nova master: pre-install: check if path is defined https://review.opendev.org/705892 | 07:41 |
| *** elenalindq has joined #openstack-ansible | 08:16 | |
| *** rpittau|afk is now known as rpittau | 08:34 | |
| *** gshippey has joined #openstack-ansible | 08:45 | |
| *** gillesMo has joined #openstack-ansible | 08:49 | |
| *** DanyC has joined #openstack-ansible | 08:57 | |
| jrosser | i think this is the next piece needed to unbreak stable/train https://review.opendev.org/#/c/703572/ | 08:57 |
| *** feichh has joined #openstack-ansible | 09:23 | |
| *** shyamb has joined #openstack-ansible | 10:06 | |
| *** mjwales has joined #openstack-ansible | 10:06 | |
| mjwales | Enabling page_size in an ldap keystone domain causes a critical error in Keystone: ERROR keystone IOError: write error. Has anyone come across this before? | 10:07 |
| *** shyamb has quit IRC | 10:38 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Bump SHAs for master https://review.opendev.org/704253 | 10:46 |
| *** lkoranda has joined #openstack-ansible | 11:05 | |
| *** rpittau is now known as rpittau|bbl | 11:10 | |
| *** udesale_ has joined #openstack-ansible | 11:13 | |
| *** shyamb has joined #openstack-ansible | 11:15 | |
| *** udesale has quit IRC | 11:16 | |
| *** pcaruana has quit IRC | 11:37 | |
| *** ahosam has joined #openstack-ansible | 11:39 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Do not append container_name for metal containers https://review.opendev.org/705778 | 11:40 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Remove log compression before upload https://review.opendev.org/703374 | 11:43 |
| *** ahosam has quit IRC | 11:46 | |
| *** pcaruana has joined #openstack-ansible | 11:50 | |
| *** cshen has joined #openstack-ansible | 12:07 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Do not append container_name for metal containers https://review.opendev.org/705778 | 12:09 |
| *** tosky has joined #openstack-ansible | 12:12 | |
| *** cshen_ has joined #openstack-ansible | 12:12 | |
| *** shyamb has quit IRC | 12:13 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Use python3 to drop hosts file script where possible https://review.opendev.org/705849 | 12:15 |
| *** cshen has quit IRC | 12:32 | |
| *** cshen has joined #openstack-ansible | 12:32 | |
| mathlin | jrosser: thanks, worked like a charm | 12:42 |
| *** rh-jelabarre has joined #openstack-ansible | 12:45 | |
| *** shyamb has joined #openstack-ansible | 12:54 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Use python3 to drop hosts file script where possible https://review.opendev.org/705849 | 13:01 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Install gpg binary for Debian based systems https://review.opendev.org/705955 | 13:07 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Use python3 to drop hosts file script where possible https://review.opendev.org/705849 | 13:14 |
| *** rpittau|bbl is now known as rpittau | 13:15 | |
| *** bhyrted has joined #openstack-ansible | 13:16 | |
| *** shyamb has quit IRC | 13:18 | |
| *** electrofelix has joined #openstack-ansible | 13:29 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ceph_client master: ansible_distribution_release insted of ansible_lsb https://review.opendev.org/705962 | 13:36 |
| *** cshen_ has quit IRC | 13:38 | |
| openstackgerrit | Sam Choraria proposed openstack/openstack-ansible-ops master: Allow beat processors to be defined through configuration data https://review.opendev.org/705965 | 13:45 |
| *** lkoranda has quit IRC | 13:58 | |
| *** rpittau is now known as rpittau|mtg | 14:03 | |
| *** DanyC_ has joined #openstack-ansible | 14:06 | |
| *** DanyC has quit IRC | 14:10 | |
| *** lkoranda has joined #openstack-ansible | 14:12 | |
| *** pcaruana has quit IRC | 14:17 | |
| *** rpittau|mtg is now known as rpittau | 14:25 | |
| *** jamesdenton has joined #openstack-ansible | 14:34 | |
| *** cshen has quit IRC | 14:40 | |
| *** DanyC_ has quit IRC | 14:43 | |
| *** DanyC has joined #openstack-ansible | 14:44 | |
| *** DanyC has quit IRC | 14:45 | |
| *** DanyC has joined #openstack-ansible | 14:46 | |
| *** cshen has joined #openstack-ansible | 14:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 14:51 | |
| *** cshen has quit IRC | 14:56 | |
| *** DanyC has quit IRC | 14:58 | |
| *** lkoranda has quit IRC | 15:01 | |
| *** DanyC has joined #openstack-ansible | 15:07 | |
| *** DanyC has quit IRC | 15:13 | |
| *** DanyC has joined #openstack-ansible | 15:13 | |
| *** DanyC has quit IRC | 15:18 | |
| *** CeeMac has joined #openstack-ansible | 15:25 | |
| *** chkumar|rover is now known as raukadah | 15:27 | |
| bjoernt | Any ETA on 19.0.10 ? | 15:31 |
| *** mjwales has quit IRC | 15:33 | |
| *** ianychoi_ is now known as ianychoi | 15:40 | |
| *** pcaruana has joined #openstack-ansible | 15:49 | |
| *** DanyC has joined #openstack-ansible | 15:57 | |
| *** openstackstatus has joined #openstack-ansible | 16:04 | |
| *** ChanServ sets mode: +v openstackstatus | 16:04 | |
| openstackgerrit | Sam Choraria proposed openstack/openstack-ansible-ops master: Allow beat processors to be defined through configuration data https://review.opendev.org/705965 | 16:04 |
| *** southquist has quit IRC | 16:06 | |
| *** udesale_ has quit IRC | 16:24 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Change nova_lock_path mode https://review.opendev.org/706006 | 16:31 |
| noonedeadpunk | bjoernt: patch is placed https://review.opendev.org/#/c/705447/1 | 16:33 |
| noonedeadpunk | waiting for the release team approval | 16:33 |
| bjoernt | yes I was wondering why the patch didnt get merged | 16:34 |
| *** DanyC_ has joined #openstack-ansible | 16:35 | |
| *** spatel has joined #openstack-ansible | 16:37 | |
| *** DanyC has quit IRC | 16:38 | |
| spatel | now my single openstack cluster has 350 compute nodes... do i need to be worry? or should i keep adding more? | 16:38 |
| spatel | Everything looks good on infra nodes.. rabbit, mysql etc... | 16:39 |
| bjoernt | We have clusters running with well over 400 nodes but you need to watchout for api threads on neutron nova | 16:46 |
| *** DanyC_ has quit IRC | 16:46 | |
| bjoernt | rabbitmq ha policy I would watch out for | 16:46 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch data path configuration https://review.opendev.org/704770 | 16:46 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Exposed config for logstash elasticsearch endpoints https://review.opendev.org/705483 | 16:46 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Explicitly use Elasticsearch data node for ILM index update https://review.opendev.org/706014 | 16:46 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Update ILM policy if non-existent https://review.opendev.org/706015 | 16:46 |
| *** DanyC has joined #openstack-ansible | 16:46 | |
| spatel | bjoernt: what to watch in ha policy? | 16:48 |
| spatel | do you know metrics ? | 16:48 |
| bjoernt | to limit to 2 copies | 16:48 |
| bjoernt | and that you run hipe which should be default | 16:49 |
| spatel | do you know any command or tool i should be checking that? | 16:49 |
| spatel | I am not very expert in rabbitMQ | 16:49 |
| spatel | just monitoring basic mesg rate at this point | 16:50 |
| bjoernt | look inside the rabbitmq console to see message growth and latency | 16:50 |
| bjoernt | also run the controller nodes on performance governor | 16:50 |
| *** DanyC has quit IRC | 16:51 | |
| spatel | bjoernt: you meant -> cpupower frequency-set -g governor | 16:51 |
| bjoernt | one way out of millions, lol | 16:51 |
| bjoernt | yes | 16:51 |
| spatel | Do you monitoring rabbitMQ with builtin monitoring GUI or you have other metrics? | 16:52 |
| spatel | I have telegraf script pumping data in influx | 16:52 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Explicitly use Elasticsearch data node for ILM index update https://review.opendev.org/706014 | 17:05 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Update ILM policy if non-existent https://review.opendev.org/706015 | 17:05 |
| openstackgerrit | Sam Choraria proposed openstack/openstack-ansible-ops master: Allow beat processors to be defined through configuration data https://review.opendev.org/705965 | 17:10 |
| bjoernt | we have custom monitoring which use the management api | 17:15 |
| *** DanyC has joined #openstack-ansible | 17:19 | |
| noonedeadpunk | gshippey: regarding https://opendev.org/openstack/openstack-ansible-os_nova/commit/149d555d6b19ccfd903fff2797d901bfa3a0c69f are you sure that qemu-system and qemu-system-misc are required, since they just bring more archs to be supported? | 17:21 |
| *** DanyC has quit IRC | 17:22 | |
| *** DanyC has joined #openstack-ansible | 17:22 | |
| jrosser | noonedeadpunk: i think the issue was at upgrade time | 17:24 |
| jrosser | when you had older versions of those things installed, but becasue the extra packages had been dropped and are not coming in through a depends-on you end up with broken installation | 17:25 |
| noonedeadpunk | jrosser: yeah, I can recall it, and I agree that qemu-utils _must_ be in the list | 17:25 |
| noonedeadpunk | so we kinda need to drop them in an appropriate way? | 17:26 |
| noonedeadpunk | so that be no conflict during packages update? | 17:26 |
| jrosser | if they're no longer needed then they should be uninstalled, rather than just dropped from the install list | 17:27 |
| jrosser | then i think it would be ok | 17:27 |
| *** gyee has joined #openstack-ansible | 17:29 | |
| *** shananigans has joined #openstack-ansible | 17:29 | |
| noonedeadpunk | I just do one pretty hacky thing, and like in older nova if archs not in the list are installed https://opendev.org/openstack/nova/src/branch/master/nova/objects/fields.py#L208-L216 nova-compute just fails with "InvalidArchitectureName: Architecture name 'armv6l' is not recognised" | 17:30 |
| noonedeadpunk | while I have x86_64 host | 17:30 |
| noonedeadpunk | I think it's fixed now though in nova, so not a big deal | 17:32 |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-ansible | 17:34 | |
| jrosser | do you think we shave something broken that needs fixing? | 17:35 |
| *** DanyC has quit IRC | 17:35 | |
| *** DanyC has joined #openstack-ansible | 17:36 | |
| jrosser | i did some more centos-8 today, seems unclear if repo priorities are a thing any more though | 17:38 |
| jrosser | mnaser: ^ have you done any centos-8? | 17:39 |
| *** DanyC_ has joined #openstack-ansible | 17:39 | |
| *** DanyC has quit IRC | 17:40 | |
| *** NobodyCam has quit IRC | 17:50 | |
| *** NobodyCam has joined #openstack-ansible | 17:51 | |
| *** errr has quit IRC | 17:56 | |
| *** errr has joined #openstack-ansible | 17:56 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Propagated ILM changes to auditbeat install https://review.opendev.org/706069 | 18:00 |
| *** DanyC has joined #openstack-ansible | 18:01 | |
| *** DanyC has quit IRC | 18:01 | |
| *** rpittau is now known as rpittau|afk | 18:01 | |
| *** johnsom has quit IRC | 18:03 | |
| *** johnsom has joined #openstack-ansible | 18:03 | |
| *** DanyC_ has quit IRC | 18:05 | |
| spatel | jrosser: if you want i can give it a try with centos-8 (i was thinking of deploying osa on centos-8 in my lab) | 18:28 |
| jrosser | well this is extremely early attempt to build an AIO without containers | 18:29 |
| jrosser | and I seem to fail at simple things like making the rpm repo priorities work currently | 18:30 |
| jrosser | and also having to compile non existent packages from C code :( | 18:30 |
| jrosser | if you want to hack on getting centos 8 working that would be great | 18:31 |
| spatel | jrosser: i will give it a shot | 18:42 |
| *** theintern_ has joined #openstack-ansible | 18:51 | |
| *** theintern_ is now known as the_intern | 18:52 | |
| *** electrofelix has quit IRC | 18:59 | |
| *** gshippey has quit IRC | 19:01 | |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-os_designate master: Fix paging link protocol when behind haproxy https://review.opendev.org/706090 | 19:14 |
| *** spatel has quit IRC | 19:22 | |
| *** NobodyCam has quit IRC | 19:39 | |
| *** johnsom has quit IRC | 19:40 | |
| *** bhyrted has quit IRC | 20:24 | |
| guilhermesp | jamesdenton: hey man, I've faced some strange behavious with iptables_hybrid driver for ovs... like after creating batches of vms at once, there is a point that neutron agent breaks like http://paste.openstack.org/show/789175/ and then vms fails to allocate a ports. I talked with some guys on neutron and they said like ovs firewall is mostly used in train now | 20:47 |
| guilhermesp | after changing my driver from iptables_hybrid to openvswitch, it seems the iptable errors on agents are not happenening aymore | 20:48 |
| guilhermesp | are you aware of something regarding support or testing of iptables_hybrid driver so far? | 20:49 |
| jamesdenton | one sec | 20:49 |
| jamesdenton | on a call | 20:49 |
| guilhermesp | sure! no worries :) | 20:49 |
| *** mgariepy has quit IRC | 20:54 | |
| *** the_intern has quit IRC | 20:58 | |
| jamesdenton | can't say i have seen that, as we are dealing primarily with Stein right now (w/ iptables_hybrid IIRC) | 21:07 |
| jamesdenton | but it does look like some invalid lines may be at fault | 21:07 |
| jamesdenton | how many are you spinning up simultaneously | 21:08 |
| guilhermesp | i did 4 batches of 15 instances, the 4th batch usually breaks agent | 21:09 |
| guilhermesp | so when I changed to openvswitch driver, like much more batches ( >100vm) in an interval of 5 to 10 minutes didn't break for now | 21:09 |
| guilhermesp | we are with a deployment which is under a stress test since yesterday | 21:09 |
| jamesdenton | it would not surprise me if there was some regression in that driver | 21:10 |
| guilhermesp | so at some point, with lots of vms being created in a small interval breaks ovs agents | 21:10 |
| jamesdenton | how are you recovering from the issue? | 21:11 |
| guilhermesp | haleyb on #openstack-neutron commented that ovs firewall is mostly used in train and is more tested in gates nowadays i guess | 21:11 |
| guilhermesp | restarting the agent jamesdenton | 21:11 |
| jamesdenton | yeah, i know that ovs firewall has become used more and more. there was a bug i didn't like and had refrained from using it. it just got patched but i have not tested it | 21:12 |
| jamesdenton | https://bugs.launchpad.net/neutron/+bug/1732067 | 21:12 |
| openstack | Launchpad bug 1732067 in OpenStack Security Notes "openvswitch firewall flows cause flooding on integration bridge" [Undecided,New] | 21:12 |
| guilhermesp | hum i see, yeah im not sure about ovs firewall tbh... but we might have an issue with iptables_hybrid as we use as the default driver for ovs | 21:13 |
| jamesdenton | yeah, you're kinda screwed either way! | 21:13 |
| jamesdenton | port_security=false. fixed. | 21:13 |
| guilhermesp | jamesdenton: lol yeah, i think stress tests screws us all at some point | 21:14 |
| guilhermesp | hum so you leave the port unsecure to fix the bug above o.O ? | 21:16 |
| jamesdenton | naw, just saying don't run port security and you won't have any ovs firewall or iptables issues :D | 21:17 |
| guilhermesp | ah yeah | 21:18 |
| *** tosky has quit IRC | 21:18 | |
| guilhermesp | well, in that case I will try out openvswitch and see how it behaves in subsequent stress testings | 21:20 |
| jamesdenton | worth a shot. we had seen performance degradation (actual lower throughput) with that bug i referenced | 21:21 |
| jamesdenton | but it was a whiel ago | 21:21 |
| jamesdenton | if i have some time tomorrow, i will try to replicate your test in my env | 21:21 |
| jamesdenton | will have to see where i am at neutron-wise | 21:21 |
| guilhermesp | nice jamesdenton thanks! I will keep watching metrics of our rally tests to see if the error occours again. Currently I have some computes that are going to be tested running openvswitch driver | 21:23 |
| guilhermesp | i will keep you informed as well | 21:23 |
| jamesdenton | please do! | 21:23 |
| jamesdenton | were those the only relevant logs you had? at that url? | 21:24 |
| guilhermesp | yeah... i've spent the whoole day looking for other relevant logs | 21:24 |
| guilhermesp | but all rally tests were failing with timeouts to build ( allocate the port). Then I started to do some manual testss and creating lots of vms on only one compue, at some point, broke the agent and I was not able to create more instnace in this compute | 21:25 |
| guilhermesp | only after a agent restart ( restart and wait for some time to flush error logs ) | 21:25 |
| jamesdenton | can you share your server create command? | 21:27 |
| guilhermesp | and tbh, never seen such errors.... we monitor the agents, and even with those iptables rules breaking the agent, we could see the agent up through the openstack perspective. | 21:27 |
| guilhermesp | usually openstack server create --flavor test --image ubuntu-bionic --key-name deploy --availability-zone nova:kvm3.specterops.iad1.vexxhost.net --network public --max 15 test-kvm3 | 21:27 |
| jamesdenton | yeah, it's "healthy" | 21:28 |
| jamesdenton | ok pretty standard. cool, thanks | 21:28 |
| jamesdenton | what um, neutron version are you at? /openstack/venvs/neutron-* | 21:28 |
| guilhermesp | 20.0.1 | 21:29 |
| jamesdenton | ok, i have some upgrading to do | 21:29 |
| jamesdenton | i will keep ya posted | 21:30 |
| guilhermesp | the only difference for now is that we were using centos compute nodes, but now we are running debian ones | 21:30 |
| jrosser | looks like theres a fix to the ovs flooding in stable/train now | 21:30 |
| guilhermesp | i did a conversion recentrly, but not sure if I can say that is related to distro as the cloud started to be heavily used 1 day ago | 21:30 |
| *** elenalindq has quit IRC | 21:30 | |
| jamesdenton | hmmm, maybe it's a debian thing :D | 21:31 |
| guilhermesp | i guess that may the reason it is more tested on gates now ;P | 21:31 |
| guilhermesp | it's a debian buster compute running 20.0.1 | 21:31 |
| guilhermesp | ok so will have a quick rehearsal in a while but I will try to keep an eye here in case you need some more details | 21:32 |
| jamesdenton | i won't have time to look today but it's on my list | 21:32 |
| guilhermesp | cool cool thanks anyways \m/ | 21:33 |
| *** hwoarang has quit IRC | 21:35 | |
| *** hwoarang has joined #openstack-ansible | 21:37 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/train: Always use virtualenv and pip to install tempest on Ubuntu https://review.opendev.org/705336 | 21:55 |
| *** nicolasbock has joined #openstack-ansible | 21:57 | |
| *** nicolasbock has quit IRC | 22:23 | |
| *** yolanda has joined #openstack-ansible | 22:35 | |
| *** shananigans has quit IRC | 22:40 | |
| *** rh-jelabarre has quit IRC | 22:46 | |
| *** NobodyCam has joined #openstack-ansible | 23:00 | |
| *** johnsom has joined #openstack-ansible | 23:02 | |
| *** hwoarang has quit IRC | 23:51 | |
| *** hwoarang has joined #openstack-ansible | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!