| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Disable CentOS check https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820447 | 07:53 |
|---|---|---|
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add documentation of security improvements made to Openstack Ansible https://review.opendev.org/c/openstack/openstack-ansible/+/820370 | 08:34 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add documentation of security improvements made to Openstack Ansible https://review.opendev.org/c/openstack/openstack-ansible/+/820370 | 08:38 |
| * noonedeadpunk commenting ^ atm | 08:44 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Disable CentOS check https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820447 | 08:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/819861 | 08:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/810237 | 08:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/819802 | 08:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820235 | 09:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820433 | 09:00 |
| mgoddard | hi evrardjp, sorry I missed your recent message. What's up? | 10:20 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Drop designate notifications topic https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/819314 | 10:54 |
| opendevreview | Merged openstack/openstack-ansible master: Reduce ceph memory overhead for AIO by setting is_hci to true https://review.opendev.org/c/openstack/openstack-ansible/+/820417 | 10:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/ussuri: Add CentOS 8.4 support https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/818488 | 11:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/victoria: Add CentOS 8.4 support https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/818487 | 11:18 |
| opendevreview | Merged openstack/openstack-ansible master: Do not fail when nova console is disabled https://review.opendev.org/c/openstack/openstack-ansible/+/820192 | 11:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Add integrated linters test https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/799037 | 11:35 |
| noonedeadpunk | let's merge manila patches and I will proceed with branching asap https://review.opendev.org/q/project:openstack/openstack-ansible-os_manila+status:open+label:Verified | 12:18 |
| noonedeadpunk | jrosser: would you mind if around?:) | 12:18 |
| jrosser | sure | 12:22 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/820439 | 12:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820504 | 12:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820504 | 12:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820505 | 12:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820505 | 12:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820506 | 12:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820506 | 12:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Bump OpenStack-Ansible Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/820529 | 14:46 |
| spatel | noonedeadpunk how do i find how which mysql cluster nodes in cluster - https://paste.opendev.org/show/811482/ | 14:49 |
| spatel | look like my galera not in good state | 14:49 |
| noonedeadpunk | `wsrep_incoming_addresses` shoudl contain list of IPs of the clusterm members | 14:51 |
| spatel | out of 3 nodes i have one node showing all 3 in this example - https://paste.opendev.org/show/811483/ | 14:52 |
| spatel | how do i know which nodes is out of cluster because my cluster size is 2 currently | 14:52 |
| noonedeadpunk | well, that node says `wsrep_cluster_size: 0` | 14:53 |
| spatel | 0 means its outside cluster pool correct? | 14:53 |
| noonedeadpunk | yeah | 14:54 |
| spatel | let me restart that node.. | 14:54 |
| noonedeadpunk | and wsrep_ready is OFF for it | 14:54 |
| spatel | we had large DDoS caused partition in cluster | 14:54 |
| spatel | I am seeing wsrep_ready is OFF on all 3 nodes | 14:54 |
| opendevreview | James Gibson proposed openstack/ansible-role-uwsgi master: Add support for TLS to UWSGI https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/820532 | 14:55 |
| noonedeadpunk | tbh I'd check which node has latest state which I think is done with `wsrep_last_committed` but not 100% sure | 14:55 |
| spatel | wsrep_last_committed is same on all 3 nodes | 14:55 |
| spatel | let me restart mariadb on node which has cluster size zero ( 0 ) | 14:56 |
| noonedeadpunk | and /var/lib/mysql/grastate.dat has -1 in seqno? | 14:57 |
| noonedeadpunk | as it feels like you might have split brain right now | 14:57 |
| spatel | yes DDoS split my network | 14:58 |
| noonedeadpunk | so I'd tried to find out what member has latest data and tried to re-bootstrap from it most likely | 14:58 |
| spatel | all of my nodes has seqno: -1 in grastate.dat | 14:59 |
| noonedeadpunk | it probably would be the one that has haproxy being routed to as "master" | 14:59 |
| spatel | its node 2 in my case | 14:59 |
| spatel | noonedeadpunk when i restart 3rd node i got this error - https://paste.opendev.org/show/811484/ | 15:02 |
| spatel | should i remove /var/lib/mysql/grastate.dat and then restart? | 15:04 |
| noonedeadpunk | sorry in a meeting | 15:15 |
| noonedeadpunk | is your networking actually fine now? | 15:34 |
| noonedeadpunk | or cluster members are still not reachable between each other | 15:35 |
| noonedeadpunk | um, no, see no reason to remove grastate | 15:35 |
| spatel | This is what i did, rename /var/lib/mysql/grastate.dat and run galera_new_cluster | 15:41 |
| spatel | all good now | 15:41 |
| spatel | cluster is back and running | 15:42 |
| noonedeadpunk | oh, well, you could place safe_to_bootstrap: 1 but you must do this on the node with latest state only | 15:46 |
| spatel | all nodes has seqno: -1 so i assuming all nodes are latest | 15:53 |
| spatel | + i have LB pointing to nodes-2 for write so in my case node-2 is latest and greatest | 15:53 |
| spatel | that is what i did on node-2 | 15:54 |
| spatel | I didn't set safe_to_bootstrap: 1 in file but just delete that file. (i think that is ok correct?) | 15:54 |
| spatel | now my rabbitMQ is dead :) | 15:55 |
| spatel | what a mess | 15:55 |
| opendevreview | Merged openstack/openstack-ansible master: Reduce manila CI check memory consumption https://review.opendev.org/c/openstack/openstack-ansible/+/820010 | 16:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Bump OpenStack-Ansible Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/820547 | 16:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/ussuri: Always upgrade ca-certificates https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/819689 | 16:06 |
| spatel | noonedeadpunk i am trying to upgrade rabbitMQ on stein release so i bump version to 3.8.14 and i git this error when re-building cluster - https://paste.opendev.org/show/811487/ | 16:15 |
| spatel | rabbitmqctl -q cluster_status | paste -sd '' - | sed 's/ //g' | grep -oP '(?<={cluster_name,<<\").*(?=\">>})' | 16:16 |
| spatel | no output | 16:16 |
| spatel | may be i need to change something in playbook which not able to extracting new output.. damn it | 16:18 |
| noonedeadpunk | oh, well, yes, 3.8 changed output a lot | 16:18 |
| spatel | yes | 16:18 |
| spatel | i noticed that | 16:18 |
| spatel | trying to see if i can change that in playbook | 16:18 |
| spatel | or better i should stick to 3.7 :( | 16:18 |
| spatel | or edit playbook to do rabbitmqctl -q cluster_status | sed 's/ //g' | grep -oP '(?<=Clustername:).*' | 16:20 |
| noonedeadpunk | I won't suggest later role version as might be we already using collection there... | 16:20 |
| spatel | i can just edit this file to fix this issue correct - shell: rabbitmqctl -q cluster_status | sed 's/ //g' | grep -oP '(?<=Clustername:).*' | 16:21 |
| spatel | otherwise it should work | 16:21 |
| noonedeadpunk | yeah, you can | 16:22 |
| spatel | done.. re-running play | 16:22 |
| spatel | that works! | 16:27 |
| spatel | I am also going to remove HA by hand to see how does it behave (because ansible does support non-HA cluster) | 16:28 |
| spatel | what do you suggest of vm_memory_high_watermark, 0.2 setting | 16:30 |
| spatel | should i change this to 0.4 ? | 16:30 |
| noonedeadpunk | um, but in cotnainer by default available 100% of controller node memory? | 16:38 |
| noonedeadpunk | unless you've set extra limits with cgroups | 16:38 |
| noonedeadpunk | its good to adjust if you have standalone rabbit nodes, that's true | 16:53 |
| *** sshnaidm is now known as sshnaidm|afk | 17:25 | |
| opendevreview | James Gibson proposed openstack/openstack-ansible-haproxy_server master: Add option to force encryption of all health checks over SSL https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/820572 | 17:25 |
| noonedeadpunk | we should defenitely finish uwsgi for neutron.... | 17:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/486156 | 17:29 |
| noonedeadpunk | I really wonder what's wrong there .... | 17:29 |
| noonedeadpunk | as neutron runs uwsgi in devstack... | 17:30 |
| mgariepy | noonedeadpunk, is that reprocucible on a specific test/distro ? | 17:34 |
| mgariepy | hoo. august.. | 17:35 |
| mgariepy | let's wait on the new result ahah | 17:35 |
| jrosser | the patch is from 2017 :/ | 17:38 |
| jrosser | somehow all this time it still doesn't work in an obvious way | 17:38 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Disable CentOS check https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820447 | 17:47 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/819861 | 17:51 |
| noonedeadpunk | while I can imagine in 2017 uwsgi was not that common, in 2021 it must work for sure.... | 18:20 |
| opendevreview | Merged openstack/openstack-ansible stable/ussuri: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820506 | 18:26 |
| opendevreview | Merged openstack/openstack-ansible stable/victoria: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820505 | 18:29 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820235 | 18:41 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/819802 | 18:41 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820433 | 18:41 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/820504 | 18:41 |
| jrosser | ah something is wrong with https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/810237 | 18:49 |
| noonedeadpunk | yeah just spotted ( | 18:49 |
| noonedeadpunk | already rebasing ( | 18:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/810237 | 18:52 |
| noonedeadpunk | damn 2 more hours( | 19:03 |
| noonedeadpunk | um, neutron failed now with `qemu-kvm: allocate 1018329600 bytes for jit buffer: Cannot allocate memory` doh | 19:04 |
| noonedeadpunk | (nova-compute failed on neutron uwsgi patch to be specific) | 19:05 |
| noonedeadpunk | well, fair, considering I haven't limited that... | 19:06 |
| jrosser | that's like 1G? | 19:06 |
| noonedeadpunk | yeah, it is.... | 19:07 |
| noonedeadpunk | we don't have such flavors though... | 19:10 |
| noonedeadpunk | for neutron I believe default is applied which is https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/defaults/main.yml#L286 | 19:10 |
| noonedeadpunk | still 512 is too much though | 19:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Reduce neutron WSGI workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/820586 | 19:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/486156 | 19:12 |
| noonedeadpunk | if all this tie it was just oom.... | 19:13 |
| noonedeadpunk | *time | 19:13 |
| opendevreview | Merged openstack/ansible-role-systemd_service master: Add integrated linters test https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/799037 | 19:14 |
| noonedeadpunk | should we also merge https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/819689 for U before EMing? | 19:14 |
| mgariepy | yes please :D | 19:15 |
| noonedeadpunk | great! | 19:16 |
| noonedeadpunk | oh, damn, it's actually 4 hours for manila to merge :( well, then going to sleep in the meanwhile... | 19:19 |
| spatel | welcome - https://blog.centos.org/2021/12/introducing-centos-stream-9/ | 19:19 |
| noonedeadpunk | meh | 19:19 |
| mgariepy | er. | 19:19 |
| noonedeadpunk | damn it, nodepool already have image.... | 19:20 |
| noonedeadpunk | I wonder how badly will it fail.... | 19:20 |
| mgariepy | lol. | 19:20 |
| mgariepy | if you know you will have to fix it.. ;p | 19:20 |
| noonedeadpunk | well, at least we have redhat-8 somewhere which doesn't cout really... | 19:21 |
| noonedeadpunk | *count | 19:21 |
| noonedeadpunk | so currently it's rhel beta, or?:)) | 19:22 |
| noonedeadpunk | isn't it's 2 more days before release? https://centos.org/stream9/ | 19:23 |
| noonedeadpunk | ah, lol, disregard | 19:23 |
| spatel | lol | 19:25 |
| spatel | i am still confused in centos how does versioning works? | 19:26 |
| mgariepy | https://xkcd.com/2224/ | 19:27 |
| spatel | heh | 19:28 |
| mgariepy | isn't stream only for beta testing for rhel and at some point stuff get merged in rhel from stream(?not quite sure when/how or why) ? | 19:29 |
| spatel | yes stream is QA/Staging environment for RHEL release | 19:30 |
| spatel | fedora ---> stream ----> RHEL | 19:31 |
| spatel | i won't recommend to not deploy it in production, we never know when redhat change policy | 19:32 |
| spatel | Sorry (IBM) | 19:33 |
| spatel | What is the status of Rocky Linux? | 19:33 |
| noonedeadpunk | nobody cares I think) | 19:34 |
| noonedeadpunk | but _if_ it is compatible - it should just work | 19:34 |
| noonedeadpunk | as it's reported as ansible_os_family as redhat | 19:35 |
| noonedeadpunk | but we can't know since there're no CI images and probably won't ever be | 19:35 |
| spatel | after CentOS EOL people we start moving to Rocky Linux | 19:36 |
| noonedeadpunk | I honestly put more trust in almalinux tbh, since CloudLinux haven't failed me previously | 19:37 |
| noonedeadpunk | and they have good experience and team tbh... | 19:37 |
| noonedeadpunk | and it's whole their business that standed on CentOS | 19:37 |
| noonedeadpunk | and they have ELevate :D | 19:38 |
| spatel | never heard this name before "almalinux" | 19:38 |
| noonedeadpunk | https://almalinux.org/elevate | 19:38 |
| noonedeadpunk | which allow to upgrade between centos major versions | 19:39 |
| mgariepy | first time i heard you can upgrade between major release ! :P | 19:39 |
| spatel | lol | 19:40 |
| noonedeadpunk | well, I heard about them as I used several of products from their maintainers pretty successfully. https://www.kernelcare.com/ was made by them but sold afterwards :( | 19:40 |
| noonedeadpunk | and they sold it for $1 per server :P | 19:41 |
| noonedeadpunk | (per month ofc) | 19:41 |
| noonedeadpunk | oh, wait, no, it hasn't been sold lol | 19:42 |
| noonedeadpunk | tixcare appears to be owned by cloudlinux | 19:42 |
| noonedeadpunk | *tuxcare | 19:42 |
| mgariepy | on paper it seems nice. but not rebooting a server for 4 year then a power outage do happen. do not seem fun to me :) | 19:42 |
| noonedeadpunk | we had uptime for 1000+days lol)) | 19:43 |
| mgariepy | yep then it too 3 days to reboot the damn server ! | 19:43 |
| noonedeadpunk | But it's more about - urgent security flaw is found and you need to reboot your all your computes in a week.... | 19:43 |
| noonedeadpunk | It's soo cheaper to jsut do planned upgrades say once a year... | 19:44 |
| mgariepy | depending on the setup it might not be too long to remove load from servers . | 19:45 |
| noonedeadpunk | depending on the amount of servers) | 19:45 |
| mgariepy | but it depends on the usecase. | 19:45 |
| noonedeadpunk | as abvoisuly you should have tons of internal stuff that must be upgraded as well | 19:45 |
| noonedeadpunk | *obviously | 19:45 |
| noonedeadpunk | but whatever) | 19:45 |
| mgariepy | yeah | 19:45 |
| noonedeadpunk | They also have extended support for dead OS, like they still provide patches for centos 6 :D | 19:46 |
| noonedeadpunk | or ubuntu xenial | 19:47 |
| noonedeadpunk | And it's sooo much cheaper they buy from canonical their advantage... | 19:47 |
| noonedeadpunk | *then | 19:48 |
| mgariepy | hehe. | 19:48 |
| spatel | can we run infra nodes on ubuntu and compute on CentOS? | 20:28 |
| mgariepy | i guess you could. | 20:29 |
| spatel | hmm | 20:29 |
| spatel | i have 300 nodes cluster running centOS (thinking to migrate to Ubunut) | 20:29 |
| mgariepy | i guess that it's not muck worst that when you do the upgrade | 20:29 |
| mgariepy | like major ubuntu upgrade. | 20:30 |
| spatel | but may be mixing OS will bring some odd | 20:30 |
| mgariepy | hmm | 20:32 |
| mgariepy | you would do controller 1 by 1 ? | 20:33 |
| mgariepy | probably best to test it in dev. | 20:33 |
| spatel | I am thinking buy hardware and build 3 node controller using ubunut and then slowly migrate compute one by one | 20:34 |
| spatel | that is easy and simple | 20:34 |
| mgariepy | migrate vms ? | 20:35 |
| mgariepy | like live migration or something like that ? | 20:35 |
| spatel | yes migrate them, we don't have Ceph storage. | 20:35 |
| spatel | We are running voice application and we don't store data | 20:36 |
| mgariepy | if your flavor have ephemeral storage you might need to adjust some stuff | 20:36 |
| spatel | i can delete vm and create on new cluster | 20:36 |
| mgariepy | ho. | 20:36 |
| mgariepy | easy then i guess ! :D | 20:36 |
| spatel | all my vms running on local-storage | 20:36 |
| spatel | we are just use cpu/network to run voice | 20:37 |
| mgariepy | yes local storage might be ok since it does start off from a qemu file backed by the glance image (in case of ephemeral nova will create the backing file on the compute) and between 16>18 upgrade the options for creating this file changed | 20:39 |
| mgariepy | so both files are not compatible. | 20:39 |
| spatel | oh! wait i am not following you.. what is backing file? | 20:40 |
| mgariepy | when you start a vm nova will donwload the qcow2 file from glance and create a new file for the vm. | 20:41 |
| spatel | yes, that is true | 20:42 |
| mgariepy | but by default the vm file is dependent on the cached image from glance on the node. | 20:42 |
| spatel | yes.. that is correct | 20:42 |
| spatel | oh wait.. no i don't think that is true | 20:43 |
| spatel | i am running queens and up version so i don't think that is the case with me | 20:44 |
| mgariepy | https://paste.opendev.org/show/811494/ | 20:45 |
| mgariepy | but it depend on the configuration also. | 20:45 |
| spatel | its been 3 years now and i am running all local-storage and had not a single issue | 20:46 |
| mgariepy | it's not what i'm saying. | 20:46 |
| mgariepy | :) | 20:46 |
| spatel | :) i thought you are saying it might be a problem when they backing up | 20:47 |
| mgariepy | https://paste.openstack.org/show/811495/ | 20:47 |
| mgariepy | if you have epehmeral storage like in this flavor | 20:47 |
| mgariepy | live migration can cause issue. | 20:48 |
| mgariepy | because of the ephemeral disk | 20:48 |
| spatel | This is what my flavor saying - https://paste.openstack.org/show/811496/ | 20:49 |
| mgariepy | ok | 20:49 |
| spatel | what does Ephemeral do? | 20:50 |
| mgariepy | the ephemeral disk is attached as vdb in the vm and is a separate file in the hypervisor | 20:50 |
| mgariepy | like : disk.eph0 | 20:50 |
| spatel | hmm just like we attached cinder disk in vm? | 20:50 |
| mgariepy | with a backing file (which is only a pre-formated disk image | 20:51 |
| spatel | what is the use of it? | 20:51 |
| mgariepy | if the options for formating the disk changes the inodes don't align when migrating. | 20:51 |
| mgariepy | store ephemeral data | 20:51 |
| spatel | who use that? | 20:52 |
| mgariepy | some researcher do. to storge temporary data. | 20:53 |
| spatel | oh so very special case | 20:53 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/ussuri: Always upgrade ca-certificates https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/819689 | 21:03 |
| noonedeadpunk | you can have such mix, BUT there some tricks that needs to be done with repo server | 21:10 |
| noonedeadpunk | at least you need to have 1 repo server on the OS you will run on computes | 21:10 |
| noonedeadpunk | also there's broken stuff with haproxy since it will point to the "master" repo server while you will have wheels for centos on the 1 specific one | 21:11 |
| noonedeadpunk | Probably there should be some acl rule written but I wasn't digging too deep | 21:11 |
| mgariepy | ho that's true. | 21:13 |
| mgariepy | indeed. | 21:13 |
| noonedeadpunk | hm, where ubuntu would log oom? | 21:19 |
| noonedeadpunk | I just don't see any real reason for neutron-server to fail with uwsgi, except being killed. | 21:19 |
| jrosser | general syslog i think for oom | 21:20 |
| noonedeadpunk | oh, well, found some ovn error which says noting to me: HashRing is empty, error: Hash Ring returned empty when hashing "b'456d1ba5-4477-447f-89c3-ef1e944cef91'" | 21:20 |
| noonedeadpunk | but the thing is that in haproxy I see 200, and then it's just 504. And the last log msg in neutron-server journal is before first 504 in haproxy... | 21:21 |
| noonedeadpunk | nothing leads to api failure... | 21:21 |
| noonedeadpunk | yeah. syslog is clean.... | 21:21 |
| noonedeadpunk | hm, so what fails now with uwsgi is OVN only... (well, and calico, but that could be random) | 21:30 |
| noonedeadpunk | I wonder if we just can set `neutron_use_uwsgi` based on the provider picked.... | 21:30 |
| jrosser | is neutron-server in some sort of crash loop? | 21:33 |
| jrosser | hrrm, well it restarts over and over again many times until here https://zuul.opendev.org/t/openstack/build/7989cb6094fc4415b321603b46b2fcfa/log/logs/openstack/aio1_neutron_server_container-3f03ecf0/neutron-server.service.journal-21-14-44.log.txt#39328 | 21:35 |
| noonedeadpunk | well its not oom as at the end in ps I see processes | 21:38 |
| noonedeadpunk | well, it might do that until we make db migrations? | 21:39 |
| noonedeadpunk | I wonder if now these 2 workers we have are just occupied with waiting for reply from ovn or smth like that... | 21:40 |
| noonedeadpunk | but smth is specifically wrong with ovn.... | 21:41 |
| noonedeadpunk | mgariepy: logs have been posted for https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/486156 :p | 21:42 |
| noonedeadpunk | and you had some hands on with OVN? | 21:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/486156 | 21:46 |
| mgariepy | i can take a look tomorrow | 21:46 |
| jrosser | interestingly neutron server is the only thing with data stillin recv-q here https://zuul.opendev.org/t/openstack/build/9ac242757c00471aafa97ffcfd43968a/log/logs/openstack/instance-info/ss_20-43-19.log.txt | 21:46 |
| noonedeadpunk | hm, yes | 21:49 |
| jrosser | oh well thats 5 requests after the haproxy backend went down maybe? | 21:49 |
| noonedeadpunk | I'd say there should be 6 requests but yeah, you're probably right it's haproxy | 21:50 |
| noonedeadpunk | ah, yes, it's 5, sixth is 504 ie incoming request for neutron I guess | 21:51 |
| noonedeadpunk | so it's not killed, just doesn't respond.... | 21:52 |
| noonedeadpunk | and neutron doesn't care to gather enough logs for uwsgi test.... | 21:54 |
| noonedeadpunk | but they seem to use apache for some reason.... | 21:55 |
| noonedeadpunk | worth probably talking to them in the morning.... | 21:56 |
| noonedeadpunk | I bet they also test like ovs.... | 21:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Bump OpenStack-Ansible Ussuri https://review.opendev.org/c/openstack/openstack-ansible/+/820604 | 22:16 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/810237 | 23:29 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!