| noonedeadpunk | I will be semi-around today just in case | 09:54 |
|---|---|---|
| jrosser | good morning | 10:12 |
| noonedeadpunk | o/ | 10:30 |
| noonedeadpunk | let's try to land backports as would be great to make another releases for stable branches https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%255Estable/.*+status:open+ | 10:34 |
| jrosser | thats the ones for Z on the way | 10:42 |
| jrosser | though it looks like one of the infra mirrors might be broken so we could be in for some rechecks | 10:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Upgrade MariaDB to 10.11 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/875381 | 10:46 |
| noonedeadpunk | I'm gonna spawn some sandbox to check out haproxy changes.... | 10:47 |
| noonedeadpunk | As I don't quite understand some things still | 10:47 |
| jrosser | there is something about handlers that i want to understand | 10:47 |
| noonedeadpunk | yeah | 10:48 |
| jrosser | and i was wondering if actually the problem there was we need to make the hander names more specific in the PKI role | 10:48 |
| jrosser | i think you can `notify: "{{ handler_name }}"` | 10:48 |
| jrosser | so we could have `glance cert installed` rather than just `cert installed` but tbh i was not even understading the underlying issue at all right now | 10:49 |
| noonedeadpunk | yeah, I wanted to try out couple of things as it's tough to understand just looking at code | 10:51 |
| jrosser | right - i am really struggling to find time to make AIO/understand which is why my review of this has been so tough to do | 10:52 |
| jrosser | there was also a bug for LE which needed a throttle: 1 that should be called out as it's own patch | 10:52 |
| noonedeadpunk | yeah, kind of same here... but will invest time today at least to spawn env and can give you access it to play around if you want to | 10:54 |
| noonedeadpunk | But yes, I was going to check these things as well | 10:55 |
| jrosser | i was sure that we had some other role that allowed a list of handler names to be passed in | 10:59 |
| jrosser | but not found that obviously | 10:59 |
| Elnaz | ^ Hi | 11:00 |
| jrosser | noonedeadpunk: i was also thinking that `add_host` could be used to make temporary group for running a play against all the haproxy hosts whilst having them access the group vars from some other service group | 11:04 |
| jrosser | that would be really neat and take away some of the un-ansible-ness of whats there now | 11:05 |
| noonedeadpunk | Oh, yes, that's really good idea. We indeed have temp groups in couple of places as of today | 11:06 |
| jrosser | yes we do, like lxc stuff | 11:06 |
| jrosser | then it could be just an extra play in the service playbooks and super obvious how it works | 11:07 |
| noonedeadpunk | ++ | 11:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Allow to manage more the one vhost with mq_setup https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/875399 | 11:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace HA policies for RabbitMQ with quorum https://review.opendev.org/c/openstack/openstack-ansible/+/873618 | 11:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Add quorum support for service https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/875408 | 11:26 |
| *** dviroel_ is now known as dviroel | 11:28 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Add quorum support for glance https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/873632 | 11:30 |
| noonedeadpunk | jrosser: btw, have you checked for fixed haproxy packages? As I still don't see them, but maybe I'm blind? | 12:26 |
| jrosser | i think they were available for focal pretty quick when i looked | 12:26 |
| jrosser | i did need to check the ubuntu changelog for the package though | 12:27 |
| noonedeadpunk | hm.... | 12:27 |
| noonedeadpunk | It was just mentioning 2.0.31 but might be indeed that just backported patch to 2.0.29 for some reason.... | 12:29 |
| noonedeadpunk | Though they used to bump minor version quite freely... | 12:29 |
| noonedeadpunk | Ah, yes... It's indeed 2.0.29-0ubuntu1.3 | 12:31 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/xena: Add authentication for [cinder] section of nova.conf https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/873571 | 12:41 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Add authentication for [cinder] section of nova.conf https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/873469 | 12:41 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Use SSL database connections with nova-manage https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/873466 | 12:41 |
| *** lowercase_ is now known as lowercase | 15:29 | |
| lowercase | Hey guys, im having an issue where the playbook keeps returning this error: "sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1091, \"Can't DROP COLUMN `vm_state`; check that it exists\")", "[SQL: ALTER TABLE build_requests DROP COLUMN vm_state]". The table, build_requests does not exist. and continues to cause api sync to fail. I confirmed this by logging into the db and confirming it indeed doesn't exist. | 15:31 |
| jrosser | lowercase: which playbook is this? | 15:32 |
| lowercase | nova | 15:32 |
| lowercase | "/openstack/venvs/nova-25.2.1.dev3/bin/nova-manage", "api_db", "sync"] is the ocmmand being run. | 15:32 |
| lowercase | im on 25.3.0 | 15:33 |
| jrosser | lowercase: this is the patch in qestion - part of nova https://review.opendev.org/c/openstack/nova/+/812145 | 15:42 |
| lowercase | that's it. | 15:46 |
| jrosser | lowercase: this really is a nova issue tbh - i wonder if there is some error with the version of alembic/sqlalchemy causing issues here | 15:47 |
| lowercase | im looking at seeing if there is a better way to write this so it tests if the table and column exists, before attempting to remove it | 15:47 |
| lowercase | alright, ill ping them | 15:47 |
| jrosser | note that this is the first migration that is not a "legacy" migration too | 15:48 |
| jrosser | so there could easily be a gotcha there | 15:48 |
| lowercase | what? | 15:48 |
| lowercase | what do you mean by legacy, sorry? | 15:48 |
| jrosser | here https://github.com/openstack/nova/tree/master/nova/db/api/migrations/versions | 15:49 |
| jrosser | vs here https://github.com/openstack/nova/tree/master/nova/db/api/legacy_migrations/versions | 15:49 |
| jrosser | but tbh this is the extent of my understanding of the db migrations, knowing that they exist :) | 15:49 |
| lowercase | same.. same.. | 15:50 |
| jrosser | i think nova-manage puts stuff in syslog | 15:50 |
| jrosser | possibly there is more there, and i believe that there are debug options too | 15:51 |
| lowercase | Running upgrade d67eeaabee36 -> b30f573d3377, Remove unused build_requests columns | 15:51 |
| lowercase | that's all it says sadly | 15:51 |
| lowercase | and then it spits out the python error i linked earlier | 15:52 |
| jrosser | well at least that matches up with the version numbers in the migrations/ folder | 15:52 |
| jrosser | is this a previous install, then upgrade to 25.x, now minor upgrade to 25.y ? | 15:52 |
| jrosser | it would be interesting to know if in a fresh AIO running `nova-manage api_db sync` more than once runs into this issue | 15:54 |
| lowercase | it does | 15:58 |
| lowercase | i dropped the whole nova database, and reran the api_db sync | 15:59 |
| jrosser | so delete db -> nova-manage api_db sync (works) -> nova-manage api_db sync (fails) | 16:04 |
| jrosser | i wonder what `nova-manage api_db version` says during those steps | 16:06 |
| lowercase | api_db sync fails with the same error | 16:20 |
| lowercase | Running upgrade d67eeaabee36 -> b30f573d3377, Remove unused build_requests columns | 16:20 |
| jrosser | lowercase: did you do W->Y upgrade, missing out X? i expect nova to say thats not supported | 16:36 |
| lowercase | no, that was me misremebering version names | 16:37 |
| lowercase | we are going x -> y | 16:37 |
| lowercase | https://bugs.launchpad.net/nova/+bug/2008716 | 16:38 |
| lowercase | I just created that to see where it goes, in the meantime im going to start writing a patch | 16:38 |
| jrosser | i would edit that to avoid a distraction on skipping versions | 16:39 |
| lowercase | but first gym and lunch | 16:39 |
| lowercase | i just edited it | 16:39 |
| jrosser | cool | 16:39 |
| lowercase | refresh just to confirm i did it right? | 16:39 |
| jrosser | yes i see it correct now | 16:40 |
| spatel | is this SSD good for ceph ? - https://www.dell.com/en-us/shop/dell-384tb-ssd-sas-mixed-use-12gbps-fips-140-512e-25in-pm5-v-3-dwpd-21024-tbw/apd/400-beoy/storage-drives-media | 17:04 |
| jrosser | spatel: seems pretty expensive :) and thats an older toshiba from before they rebrand to kioxia? | 17:09 |
| spatel | Does it good for ceph.. ? | 17:10 |
| jrosser | it's impossible to say | 17:10 |
| spatel | hmm | 17:10 |
| jrosser | first it doesnt say if it has PLP | 17:10 |
| spatel | what is PLP? | 17:10 |
| jrosser | and it looks like "enterprise" type drive rather than scale out type | 17:10 |
| jrosser | Power Loss Protection | 17:11 |
| spatel | If its enterprise then PLP must be there.. | 17:11 |
| spatel | based on cost | 17:11 |
| jrosser | the only SSD drives that are good for ceph are those with sufficient power storage on the SSD to flush the RAM cache to the flash when the power fails | 17:12 |
| jrosser | and then depending on use case you might not need anywhere near 3 DWPD | 17:12 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/yoga: Use SSL database connections with nova-manage https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/873467 | 17:13 |
| jrosser | like a 12Tb kioxia CD6 is ~2000eur | 17:15 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/zed: Fix typo in ansible_facts['pkg_mgr'] https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/874742 | 17:58 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/zed: Drop empty elements from constraint/requirement files https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/874734 | 17:58 |
| spatel | jrosser is Kioxia is good brand | 17:58 |
| spatel | never heard of it | 17:58 |
| lowercase | our considerations for drives are: does dell certify the drive? And price. 11k is way to high. We use microns mostly. we are targeting nvme's for our next cluster. | 19:02 |
| lowercase | i don't have any experience with fips drives tho so that's a completly different animal | 19:03 |
| spatel | We have bunch of drives in stock spare.. thinking to use for ceph but not sure if they are good or not. I have to run some stuff to find good result | 19:07 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/xena: Use SSL database connections with nova-manage https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/873468 | 19:35 |
| Elnaz | Hi | 19:36 |
| Elnaz | Do you log this channel somewhere? | 19:37 |
| Elnaz | By 'Access` channel on the provider network, it didn't work in VLAN mode. Why it needs to have trunk channel between the nodes and TOR switch? | 19:37 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org experienced severe performance degradation between 17:50 and 19:45 due to excessive API query activity; the addresses involved are now blocked but any changes missing job results from that timeframe should be rechecked | 19:54 | |
| supamatt | spatel: koxia is a good brand, cm6's are beasts | 20:06 |
| supamatt | the new cm8's are just gonzo fast | 20:06 |
| spatel | supamatt thanks for your input | 20:07 |
| supamatt | the cm8 is hands down the fastest NVMe on the market today | 20:07 |
| supamatt | it's u.3, gets 2Miops+ on reads, and writes are around 1.2Miops... Ceph though will never see that performnace. | 20:08 |
| supamatt | too much overhead | 20:08 |
| supamatt | any enterprise NVMe that can do 200Kiops on reads, or 80Kiops on writes is going to be fast enough for Ceph today. until crismon goes ga | 20:09 |
| BobZannapolis | Anyone know how to make Rally's "NovaServers.boot_and_delete_server" execute on EVERY hypervisor we have so we can verify that all compute nodes are being accessed ? tia | 20:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!