| *** odyssey4me is now known as Guest6506 | 10:03 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Drop Nginx webserver support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 10:45 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Increase Apache thread limit for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/818733 | 10:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Backported fixes for healthcheck playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/803926 | 11:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano stable/wallaby: Fix murano role https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/818734 | 11:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Add variables for rabbitmq ssl configuration https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/791726 | 11:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/772039 | 11:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Use ansible_facts[] instead of fact variables https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/780653 | 11:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano stable/wallaby: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/798617 | 11:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/786845 | 12:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Ensure prettytable is present in the ansible-runtime virtualenv https://review.opendev.org/c/openstack/openstack-ansible/+/777061 | 12:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove provider_networks from neutron playbook https://review.opendev.org/c/openstack/openstack-ansible/+/817914 | 12:16 |
| maaz012345 | Hi Captains .. | 13:05 |
| maaz012345 | Can I get some help here related to openstack-ansible deployment | 13:06 |
| maaz012345 | https://bugs.launchpad.net/openstack-ansible/+bug/1951679 | 13:06 |
| noonedeadpunk | hey | 14:05 |
| noonedeadpunk | jrosser: well we replued same stuff there :D | 14:23 |
| noonedeadpunk | *replied | 14:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Include openstack_services for murano role https://review.opendev.org/c/openstack/openstack-ansible/+/818736 | 14:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Include openstack_services for murano role https://review.opendev.org/c/openstack/openstack-ansible/+/818736 | 14:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano stable/wallaby: Fix murano role https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/818734 | 14:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Increase Apache thread limit for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/818733 | 14:28 |
| spatel | jrosser noonedeadpunk jamesdenton_alt here is the POC for OVN SSL - https://satishdotpatel.github.io/ovn-ssl-setup-with-openstack/ | 14:29 |
| spatel | now i am going to see how we can wire up it with OSA/PKI | 14:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/ussuri: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817250 | 14:29 |
| opendevreview | Merged openstack/openstack-ansible stable/train: Fix mistral DB row size https://review.opendev.org/c/openstack/openstack-ansible/+/817631 | 14:31 |
| opendevreview | Merged openstack/openstack-ansible stable/train: add missing step to manual upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/817572 | 14:31 |
| spatel | noonedeadpunk does nova create crt for each node and sign or we are using single cert for every compute nodes? | 14:38 |
| noonedeadpunk | tbh I wasn;'t digging much into patch details that we merged | 14:38 |
| noonedeadpunk | I would need to check code to answer | 14:39 |
| noonedeadpunk | from what I see it's unique per compute | 14:39 |
| jrosser | spatel: imho you should have the os_neutron role create it's own certificates | 14:42 |
| jrosser | if you re-use the nova ones you will then be in trouble when the host is not also a compute host | 14:42 |
| spatel | jrosser sure, let me understand how does nova pki works and then i will create dedicated one for neutron role | 14:43 |
| spatel | noonedeadpunk around? | 15:26 |
| spatel | i need little advice related rabbitMQ | 15:26 |
| noonedeadpunk | I can try at least :D | 15:27 |
| spatel | we are getting lots of DDoS and that DDoS creating network blips that causing my RabbitMQ cluster partition and after that rabbitMQ not getting recover itself | 15:27 |
| spatel | OSA default use pause_minority i believe | 15:27 |
| spatel | my rabbitMQ nodes across multiple racks.. | 15:28 |
| spatel | do you think changing from pasue_minority to autoheal should help? | 15:28 |
| noonedeadpunk | well yes, by default it's `pause_minority` | 15:29 |
| noonedeadpunk | but it's pretty easily configurable with rabbitmq_cluster_partition_handling | 15:29 |
| spatel | But look like its not helping me | 15:29 |
| spatel | i would like to try out autoheal if that fix my issue.. | 15:29 |
| spatel | setting rabbitmq_cluster_partition_handling: autoheal should work right in user_variables.yml | 15:31 |
| noonedeadpunk | hm, I'm wondering if we're missing quorum queues... | 15:31 |
| noonedeadpunk | yes it should | 15:31 |
| spatel | what do you mean quorum queue | 15:31 |
| noonedeadpunk | https://www.rabbitmq.com/quorum-queues.html\ | 15:32 |
| spatel | Page not found :D | 15:32 |
| noonedeadpunk | drop last `\` | 15:32 |
| noonedeadpunk | added it accidentally | 15:32 |
| noonedeadpunk | so evenually it's new feature of 3.8 | 15:32 |
| spatel | :) | 15:32 |
| noonedeadpunk | and I haven't spend time reflecting defaults for these features | 15:33 |
| spatel | I am running 3.7.x release | 15:33 |
| spatel | rabbitmq-server-3.7.28-1.el7.noarch | 15:34 |
| noonedeadpunk | oh, there're even streams in 3.9 hehe | 15:34 |
| spatel | i am running stein release.. | 15:34 |
| noonedeadpunk | these looks like a good read... | 15:34 |
| spatel | can i bump my rabbitMQ version without upgrading cluster ? | 15:34 |
| noonedeadpunk | yes, you can use newer rabbitmq version with overrides | 15:34 |
| noonedeadpunk | and just upgrade it | 15:35 |
| spatel | oh even i am running stein release right? hope no dependency there | 15:35 |
| noonedeadpunk | but don't forget to also upgrade erlang! | 15:35 |
| spatel | i have noticed when you upgrade rabbitMQ it automatically upgrade erlang also | 15:35 |
| noonedeadpunk | um, not really | 15:36 |
| spatel | does Quorum Queues feature default enabled or i need to do it manually? | 15:36 |
| noonedeadpunk | ie we have it fixed here https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/stable/stein/vars/debian.yml#L39 | 15:36 |
| spatel | nice! i will take that code and upgrade my rabbitMQ to latest | 15:37 |
| noonedeadpunk | I don't think it's default. And I'm not sure if osa even supports them atm | 15:37 |
| noonedeadpunk | Need to have a read about that | 15:37 |
| spatel | i will mess with it.. | 15:37 |
| noonedeadpunk | `Quorum queues should be the default choice for a replicated queue type. Classic queue mirroring will be removed in a future version of RabbitMQ` | 15:38 |
| spatel | oh! must better then | 15:38 |
| spatel | i have one more question related removing HA for all queue | 15:39 |
| noonedeadpunk | oh, yes, it's the way better... | 15:39 |
| spatel | if i remove HA then how does all nodes will talk to rabbitMQ | 15:39 |
| spatel | example: compute config saying x.x.x.x,y.y.y.y,z,z,z,z | 15:40 |
| spatel | does they randomly pick rabbitMQ node? | 15:40 |
| noonedeadpunk | without replication - kind of | 15:40 |
| noonedeadpunk | and messages might be lost in case of one member outage | 15:40 |
| spatel | i don't care about lost mesg at this point :) i need stability | 15:41 |
| noonedeadpunk | but it's still cluster, so if yo uconnect to wrong member you still can be "routed" and read message you asked | 15:41 |
| mgariepy | if one server is out the whole queue will be unavailable. not only 1 message. | 15:42 |
| mgariepy | if it';s the hosting node. | 15:42 |
| spatel | if compute1 connect to rabbit1 first time then it will always stay there or do round-robin for each request ? | 15:42 |
| spatel | i am thinking i can add haproxy in front of rabbit (non-HA cluster) | 15:43 |
| spatel | mgariepy - if one server is out the whole queue will be unavailable. not only 1 message. ----- all queues on that dead node right? but other two should be working, correct? | 15:44 |
| noonedeadpunk | it is bad idea | 15:44 |
| spatel | adding haproxy? | 15:44 |
| noonedeadpunk | yep | 15:44 |
| spatel | ah! | 15:44 |
| noonedeadpunk | rabbit is designed the way that you can connect to any cluster member | 15:45 |
| noonedeadpunk | but all depends on mirroring settings then | 15:45 |
| spatel | if 3 node running in non-HA mirror and single node die.. (what is the impact here?) | 15:46 |
| jrosser | do you have some shared network path between the controllers and compute which suffers with this DDOS? | 15:49 |
| spatel | We have HP C7000 chassis and controller and computes are all mix in those chassis, we have 3 chassis in each racks. yes we have some short of shared network. when DDoS hit my network i noticed switch port buffer running out of capacity and start dropping packets.. for small time before DDoS mitigation kick in. | 15:52 |
| spatel | During that period i have noticed error in rabbitMQ saying cluster partitions. (now interesting thing.. after that rabbit never get recover from that incident) | 15:53 |
| spatel | i have 300 compute nodes in this cluster and 1000 vms (not sure if this is higher number for rabbit) | 15:54 |
| noonedeadpunk | hm, feels like quorum is not supported by oslo yet? | 15:54 |
| spatel | some article saying - We also know that rabbit before 3.8 may have some issues on clustering side, so you might consider running at least rabbitmq 3.8.x. | 15:54 |
| spatel | I am running 3.7.x (may be i need to upgrade to 3.8) | 15:55 |
| spatel | or get rid of HA | 15:55 |
| noonedeadpunk | but fwiw - 3.8 is more stable according to my experience | 15:55 |
| noonedeadpunk | so with train a lot of rabbit related stuff has been solved for me | 15:55 |
| spatel | 100% agreed, i have 3.8 running on other location which has zero issue.. (we are getting less DDoD there but) | 15:56 |
| * noonedeadpunk still a bit upset that x-queue-type can't be set with oslo yet | 15:56 | |
| spatel | reason i don't want to upgrade stein because its running centos7 and god knows upgrade will work or not :( | 15:57 |
| noonedeadpunk | from other side it means there's nothing we can/should do now :D | 15:57 |
| noonedeadpunk | upgrade rabbit? | 15:57 |
| noonedeadpunk | it should since it's using independent repos provided by erlang | 15:57 |
| spatel | That is what i am going to do. | 15:57 |
| noonedeadpunk | *external repos | 15:58 |
| spatel | vm_memory_high_watermark: 0.2 | 16:00 |
| spatel | do you think i should add more memory ? | 16:00 |
| mnaser | rabbitmq cluster recovering?! good one.... :p | 16:01 |
| noonedeadpunk | hehe | 16:01 |
| spatel | :) wow! good to see you after long time | 16:01 |
| mnaser | i'm always lurking :) | 16:02 |
| noonedeadpunk | I'm actually eager to test out stream queues | 16:02 |
| noonedeadpunk | they look like a suitable solution at glance | 16:02 |
| spatel | rabbit = no fun (100s of time i nuke my cluster since i build this cloud) | 16:03 |
| jrosser | mnaser: did you get cloud-init + ipv6 in the end? | 16:03 |
| mnaser | i really wish we start to get a fix for rabbitmq trobules | 16:08 |
| mnaser | since we use k8s world now, | 16:09 |
| mnaser | we have a script called restart cloud | 16:09 |
| mnaser | wipes rabbitmq and reinit, and then restarts * | 16:09 |
| mnaser | jrosser: i didn't' hack on it yet | 16:09 |
| spatel | mnaser that is what i am going, ./rabbit-nuke.sh crafted script to rebuild everything for me in 10 minute.. | 16:10 |
| spatel | doing* | 16:10 |
| jrosser | mnaser: for something like the ubuntu cloud image it's disabled by default, as you get a huuuuuuge timeout when theres no ipv6 metadata service available before it tries the ipv4 one | 16:11 |
| spatel | I think OVN can solve 50% issue of rabbitMQ | 16:11 |
| jrosser | if you want to try v6 only with ubuntu you'd need to modify metadata_urls in the image here https://cloudinit.readthedocs.io/en/latest/topics/datasources/openstack.html | 16:11 |
| jrosser | or else config drive | 16:12 |
| mnaser | jrosser: ipv6 only images then? | 16:12 |
| mnaser | i feel like going to OVN is going to bubble up a whole boatload of other fun things lol | 16:12 |
| jrosser | if you really want ipv6 metadata then you need to at least make it dual stack | 16:12 |
| jrosser | and accept the boot delay for v4 | 16:12 |
| jrosser | or make v6 images | 16:12 |
| mnaser | jrosser: at that point, what's the point of dual stack heh | 16:12 |
| jrosser | it's kind of unpleasnat | 16:12 |
| mnaser | sorry i mean | 16:13 |
| mnaser | whats the point of v6 only if you're going to make it dual stack tog et metadata | 16:13 |
| mnaser | i've been burned a lot with force_config_drive=true | 16:13 |
| jrosser | it's just a bummer that the stock cloud images disable v6 metadata, but i kind of see why they do it | 16:13 |
| mnaser | jrosser: solution for all network issues is disable ipv6, amirite? :) | 16:14 |
| mgariepy | something similar to the selinux stuff ;p | 16:15 |
| spatel | I believe only DoD using selinux.. lol | 16:15 |
| spatel | but trust me ipv6 getting real, now mobile carriers only providing ipv6 and apple and google won't let you add your apps if its not ipv6 certified | 16:17 |
| spatel | just because of certification we are running dual stack ipv6 cloud | 16:17 |
| mnaser | nice | 16:18 |
| mnaser | thats good to hear | 16:18 |
| mgariepy | i sure hope that there will be some real push in canada for the ISP to deploy it. | 16:20 |
| jrosser | we also run dual stack here though i'm not sure the users are doing much with it | 16:20 |
| jrosser | except for unknowingly accessing horizon over v6 if their ISP does it | 16:20 |
| spatel | https://www.internetsociety.org/blog/2016/05/starting-june-1-apple-requires-all-ios-apps-to-work-in-ipv6-only-networks/ | 16:21 |
| jrosser | neutron bgp agent scheduling is a whole extra class of special behaviour though | 16:21 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Implement ironic_neutron_agent and baremetal driver https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/813011 | 16:38 |
| *** sshnaidm is now known as sshnaidm|afk | 16:45 | |
| noonedeadpunk | let's merge https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818577 to unblock W | 16:49 |
| jrosser | done | 16:53 |
| opendevreview | Merged openstack/openstack-ansible-os_ceilometer master: Remove references to deprecated python-ceilometerclient https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/815317 | 16:54 |
| opendevreview | Merged openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/786845 | 16:54 |
| spatel | noonedeadpunk shouldn't we do vars/centos-8.yml instead of 8.5.yml ? (sorry if i am missing something) | 16:55 |
| noonedeadpunk | we're trying to distinguish centos-8 vs centos-stream | 16:55 |
| spatel | hmm | 16:56 |
| noonedeadpunk | while for ansible it's centos 8 stream is same centos-8 | 16:56 |
| noonedeadpunk | only without minor version | 16:56 |
| spatel | CentOS Linux 8 will reach End Of Life (EOL) on December 31st, 2021 | 16:57 |
| noonedeadpunk | yep, it will... | 16:57 |
| noonedeadpunk | so nasty hack is everything we need here hehe | 16:57 |
| jrosser | EOL doesnt stop people having deployments and the code needing to work on older branches :/ | 16:58 |
| spatel | hope stream will be smooth and not full of hacks | 16:58 |
| spatel | EOL may decommission some upstream repo | 16:58 |
| noonedeadpunk | well, will see | 16:59 |
| jrosser | spatel: in ansible it is very difficult to distinguish between centos-8 and centos-8 stream, theres no way to do it except checking for version=8 -> stream, version=8.x -> not stream | 16:59 |
| jrosser | so when we have to do different things for stream and not-stream it gets hacky and unreadable code very fast | 16:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano stable/wallaby: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/798617 | 17:00 |
| noonedeadpunk | btw, apache onyl keystone looks fair enough now I believe https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 17:01 |
| jrosser | thats why we need that vars/centos-8.<blah> to account for all the point releases of centos, and vars/centos-8.yml covers stream | 17:01 |
| noonedeadpunk | it needs https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 though for upgrade jobs to pass | 17:01 |
| jrosser | and we need this https://review.opendev.org/c/openstack/openstack-ansible/+/818674 | 17:02 |
| jrosser | and https://review.opendev.org/c/openstack/openstack-ansible/+/818733 | 17:03 |
| noonedeadpunk | yeah - same but for master... | 17:03 |
| noonedeadpunk | oh, damn, I pasted wrong link | 17:03 |
| jrosser | time to merge all the things to release | 17:04 |
| noonedeadpunk | yep | 17:04 |
| noonedeadpunk | I was looking through changes now and I guess we can for sure make a beta now so that ppl could test for week or so | 17:04 |
| noonedeadpunk | I just started working on proxysql role.. | 17:05 |
| noonedeadpunk | Hopefully it won't take more then couple of days | 17:05 |
| noonedeadpunk | but we'll release without it if not | 17:06 |
| noonedeadpunk | upsetting situation is with galera... | 17:06 |
| noonedeadpunk | I probably also will push bunch of patches to all roles to make config_template be usable as collection | 17:08 |
| mgariepy | noonedeadpunk, if you see galera hangs can you ask for a hold on the vm ? | 17:08 |
| mgariepy | or ping me to ask for it. | 17:08 |
| noonedeadpunk | mgariepy: the problem is that hold should be asked before it launched | 17:08 |
| noonedeadpunk | or maybe when it hang might be also fine? | 17:09 |
| mgariepy | if the check is not over. | 17:10 |
| mgariepy | you can ask for a hold | 17:10 |
| noonedeadpunk | ok, cool, I forgot it's an option. probably because when I see it failed it's too late ususally) | 17:11 |
| mgariepy | maybe we could add some stuff when the build fail | 17:11 |
| mgariepy | like mysql show processlist | 17:11 |
| jrosser | if your horizon is on the internet this may be of interest https://review.opendev.org/c/openstack/openstack-ansible/+/818533 | 17:12 |
| noonedeadpunk | oh, I was looking at it | 17:12 |
| noonedeadpunk | I guess it might be applicable also for keystone? | 17:14 |
| noonedeadpunk | and actually all apis? | 17:14 |
| jrosser | these are browser directives really | 17:14 |
| jrosser | but if the browser in the future (skyline?) called the other API directly, that might be a factor | 17:15 |
| *** sshnaidm|afk is now known as sshnaidm | 17:16 | |
| noonedeadpunk | commented | 17:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 17:29 |
| jrosser | looks like there are most of the ansible collections we need now available as required-projects in zuul jobs these days | 17:48 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Fix inconsistency in haproxy_frontend_raw naming https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/817727 | 18:30 |
| *** sshnaidm is now known as sshnaidm|afk | 19:26 | |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Update metering agent to use interface_driver alias https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/799911 | 20:07 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!