*** DanyC has quit IRC | 00:14 | |
*** spotz has joined #openstack-ansible | 00:28 | |
*** gyee has quit IRC | 00:38 | |
*** jbadiapa has quit IRC | 00:53 | |
*** dave-mccowan has quit IRC | 00:55 | |
*** nicolasbock has quit IRC | 01:01 | |
*** boldo has quit IRC | 01:08 | |
*** ivve has quit IRC | 01:27 | |
*** bold has joined #openstack-ansible | 02:07 | |
*** rh-jelabarre has quit IRC | 03:10 | |
*** raukadah is now known as chandankumar | 04:37 | |
*** udesale has joined #openstack-ansible | 04:54 | |
*** evrardjp has quit IRC | 05:35 | |
*** evrardjp has joined #openstack-ansible | 05:36 | |
*** mkuf has joined #openstack-ansible | 06:44 | |
*** miloa has joined #openstack-ansible | 07:06 | |
ebbex | Anyone seen this before? Unhandled error: CappedVersionUnk | 07:07 |
---|---|---|
ebbex | nown: Unrecoverable Error: Versioned Objects in DB are capped to unknown version 1.38. | 07:07 |
ebbex | I'm upgrading from rocky to stein, and cinder-api won't start because of this. | 07:08 |
ebbex | (Stein 19.0.10) | 07:08 |
*** mkuf_ has joined #openstack-ansible | 07:15 | |
*** mkuf has quit IRC | 07:19 | |
*** miloa has quit IRC | 07:26 | |
*** cshen has joined #openstack-ansible | 07:33 | |
*** djhankb has quit IRC | 07:37 | |
*** miloa has joined #openstack-ansible | 07:39 | |
*** bold has quit IRC | 07:46 | |
*** dmsimard|off7 has joined #openstack-ansible | 07:48 | |
*** dmsimard|off has quit IRC | 07:48 | |
*** dmsimard|off7 is now known as dmsimard|off | 07:48 | |
*** miloa has quit IRC | 07:56 | |
*** tosky has joined #openstack-ansible | 07:57 | |
*** miloa has joined #openstack-ansible | 08:01 | |
*** djhankb has joined #openstack-ansible | 08:04 | |
*** andrewbonney has joined #openstack-ansible | 08:10 | |
*** yoctozepto9 has joined #openstack-ansible | 08:14 | |
*** yoctozepto has quit IRC | 08:15 | |
*** yoctozepto9 is now known as yoctozepto | 08:15 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/713590 | 08:24 |
*** sshnaidm has quit IRC | 08:29 | |
*** cshen has quit IRC | 08:38 | |
*** rpittau|afk is now known as rpittau | 08:45 | |
*** yoctozepto has quit IRC | 08:49 | |
*** cshen has joined #openstack-ansible | 08:57 | |
*** yoctozepto has joined #openstack-ansible | 08:59 | |
*** ygk_12345 has joined #openstack-ansible | 08:59 | |
ebbex | I bumped osa to stein/master as i suspected something was wrong with the cinder at the git-hash in 19.0.10, and now it's working. Which is kinda odd, as I've done this upgrade on 19.0.10 previously without a hitch. | 09:00 |
jrosser | ebbex double check that cinder-volume and cinder-api are actually the version you expect | 09:02 |
jrosser | also that the service is restarted properly so that the installed version is actually the running one | 09:03 |
jrosser | i think a mismatch there can cause that sort of error message | 09:03 |
*** DanyC has joined #openstack-ansible | 09:11 | |
*** jbadiapa has joined #openstack-ansible | 09:36 | |
openstackgerrit | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/713590 | 09:36 |
*** ygk_12345 has quit IRC | 09:37 | |
*** admin0 has quit IRC | 09:39 | |
ebbex | this was during the 'scripts/run-upgrade.sh', the service with the error was /openstack/venvs/cinder-19.0.10/bin/cinder-scheduler | 09:40 |
ebbex | on the production cluster which i upgraded earlier in february, the object version is 1.37 | 09:42 |
jrosser | if you get those errors with git hashes clear out the specific (or all) ansible roles in /etc/ansible/roles | 09:43 |
ebbex | from the same 19.0.10 point release... | 09:44 |
jrosser | i found this which looks similar https://ask.openstack.org/en/question/100160/cinder-cappedversionunknown-unrecoverable-error-versioned-objects-in-db-are-capped-to-unknown-version-111/ | 09:46 |
ebbex | jrosser: already do, i run a playbook that checks out the 19.0.10 point release, clears out /etc/ansible/roles, runs boostrap-ansible, comment it out in the run-upgrade.sh, and apply additional patches/cherry-picks on the /etc/ansible/roles. | 09:46 |
openstackgerrit | Merged openstack/openstack-ansible-repo_build stable/train: repo_build: update tests to opendev.org https://review.opendev.org/713313 | 09:48 |
ebbex | none of the cherry-picks mess with cinder. The 19.0.10 release i upgraded to in production is as mentioned on object version 1.37. | 09:48 |
ebbex | Hmm, unless they bumped something in cinder during the rocky 18.1.19, that makes it incompatible with 19.0.10? (Production was upgraded from 18.1.17) | 09:50 |
openstackgerrit | Merged openstack/openstack-ansible-repo_server stable/rocky: lsyncd: use relative paths in exclude statement https://review.opendev.org/712482 | 09:52 |
openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty master: Refactor memcached_servers https://review.opendev.org/713232 | 09:56 |
*** bjoernt_ has joined #openstack-ansible | 09:58 | |
*** yoctozepto has quit IRC | 09:59 | |
openstackgerrit | Erik Berg proposed openstack/openstack-ansible-repo_build stable/stein: repo_build: update tests to opendev.org https://review.opendev.org/713606 | 10:00 |
*** keeper_ has joined #openstack-ansible | 10:01 | |
*** yoctozepto has joined #openstack-ansible | 10:03 | |
*** keeper has quit IRC | 10:04 | |
*** bjoernt has quit IRC | 10:04 | |
*** yoctozepto2 has joined #openstack-ansible | 10:13 | |
*** yoctozepto has quit IRC | 10:13 | |
*** yoctozepto2 is now known as yoctozepto | 10:13 | |
*** noonedeadpunk has joined #openstack-ansible | 10:14 | |
*** sshnaidm has joined #openstack-ansible | 10:20 | |
*** jawad_axd has joined #openstack-ansible | 10:50 | |
CeeMac | morning | 10:51 |
jrosser | morning | 10:52 |
*** sshnaidm has quit IRC | 10:54 | |
*** sshnaidm has joined #openstack-ansible | 10:55 | |
CeeMac | channel seems quiet these days? :( | 10:55 |
CeeMac | i've been trying to lurk when/where I can in between project-of-doom-that-never-ends | 10:56 |
CeeMac | hows everyone holding up? | 10:56 |
jrosser | no so bad | 10:56 |
jrosser | just getting used to full time working from home | 10:56 |
CeeMac | yeah | 10:56 |
CeeMac | i miss my monitors :( | 10:57 |
jrosser | i have a fairly ancient imac that turns out it can also be a display | 10:57 |
CeeMac | one of the screens with built in mac ones? like on zoolander? :) | 10:58 |
openstackgerrit | Merged openstack/openstack-ansible-os_panko master: Refactor memcached_servers https://review.opendev.org/713249 | 10:59 |
openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Refactor memcached_servers https://review.opendev.org/713262 | 10:59 |
openstackgerrit | Merged openstack/openstack-ansible-os_trove master: Refactor memcached_servers https://review.opendev.org/713263 | 11:01 |
jawad_axd | Hi ! A question: I created couple of instances with 1TB/12Vcpus/24GB_ram. I noticed hypervisor summary in horizon did not update itself...while instances are running successfully.What could be the reason? | 11:02 |
*** nicolasbock has joined #openstack-ansible | 11:03 | |
openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Refactor memcached_servers https://review.opendev.org/713244 | 11:05 |
openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: rabbitmq_server: there's no system change on a package query https://review.opendev.org/713124 | 11:14 |
jawad_axd | Answer of my above question: Restarting nova-compute on host fixed the issue and hypervisor summary got updated. | 11:19 |
*** ivve has joined #openstack-ansible | 11:47 | |
*** cshen has quit IRC | 11:52 | |
*** rh-jelabarre has joined #openstack-ansible | 11:53 | |
openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group https://review.opendev.org/707849 | 11:54 |
openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch discovery nodes https://review.opendev.org/708826 | 11:54 |
*** rpittau is now known as rpittau|bbl | 12:01 | |
*** cshen has joined #openstack-ansible | 12:03 | |
CeeMac | has anyone tried running block-storage ha across the cinder-api containers? From what I can see, this isnt baked in to the role currently | 12:07 |
CeeMac | looking at https://docs.openstack.org/ha-guide/storage-ha-block.html | 12:07 |
jrosser | CeeMac: you mean something beyond HAProxy load balancing the API requests? | 12:10 |
*** jbadiapa has quit IRC | 12:19 | |
CeeMac | jrosser: yeah, the doc has a bunch of services running as cluster resources in pacemaker. | 12:23 |
CeeMac | Guess I'll need to read some more on what difference that makes rather than using haproxy to balance api calls | 12:24 |
jrosser | im not really sure it's functionally any different to multiple haproxy+keepalived | 12:25 |
jrosser | isnt pacemaker an alternative take on that? | 12:25 |
CeeMac | Still a trying to work out where the pain points are. I think it's more probably around the cinder-volume side. Trying to wrap a process around performing maintenance on a storage node and how that affects mounted volumes etc | 12:25 |
jrosser | that page really does only seem to talk about the API service | 12:26 |
jrosser | thats not the same as HA for storage nodes | 12:26 |
*** jamesden_ has joined #openstack-ansible | 12:28 | |
jrosser | futher on in the same documentation they go on to point at things like ceph rbd for a solution to HA in the actual backend https://docs.openstack.org/ha-guide/storage-ha-backend.html | 12:28 |
noonedeadpunk | YEar, working from home is pretty nasty thing... Just getting too much distracted by different stuff... | 12:32 |
*** gshippey has joined #openstack-ansible | 12:40 | |
*** udesale_ has joined #openstack-ansible | 12:44 | |
*** udesale has quit IRC | 12:47 | |
*** rpittau|bbl is now known as rpittau | 13:01 | |
*** ygk_12345 has joined #openstack-ansible | 13:04 | |
*** fghaas has joined #openstack-ansible | 13:09 | |
*** sep has quit IRC | 13:09 | |
CeeMac | jrosser: yeah think I need to do more reading. | 13:10 |
CeeMac | On an unrelated note. How easy/hard is it to add a custom container to the inventory and deployment? | 13:14 |
jrosser | thats reasonably straightforward | 13:17 |
jrosser | you make new stuff in /etc/openstack_deploy/env.d/.... | 13:17 |
jrosser | i have a little example of that here https://github.com/jrosser/openstack-ansible-ops/blob/designate-bind/designate-bind/env.d/dnsaas-bind.yml | 13:19 |
CeeMac | Ah, great thanks | 13:19 |
jrosser | but really it's no more than taking one of the existing container defintoins from the openstack-ansible env.d directory and adjusting the names to suit | 13:20 |
jrosser | then assign hosts for those continaers in openstack_user_config | 13:20 |
mnaser | jrosser: also i think there is a doc about it | 13:21 |
mnaser | i cant seem to find it tho | 13:22 |
jrosser | no me neither | 13:23 |
CeeMac | I'll take a punt and see how I get on | 13:24 |
*** jbadiapa has joined #openstack-ansible | 13:24 | |
*** cshen has quit IRC | 13:25 | |
jrosser | once you've fiddled env.d and o_u_c.yml use inventory_manage.py to see if the new groups/hosts have been created | 13:25 |
*** cshen has joined #openstack-ansible | 13:25 | |
mnaser | ^ and run dynamic inventory at least once to update it | 13:26 |
CeeMac | Right | 13:26 |
CeeMac | I was going to ask about that, I think I've got a record of that command | 13:27 |
CeeMac | Thanks guys :) | 13:27 |
ygk_12345 | hi all | 13:27 |
CeeMac | Hi | 13:28 |
ygk_12345 | how to install placement plugin for openstack client in openstack ansible ? | 13:28 |
ygk_12345 | i am receiving these errors when in rocky when launching a vm on a specific compute node | 13:29 |
ygk_12345 | Failed to retrieve resource provider tree from placement API for UUID ea3af0ff-fb69-449a-b484-5ac722010a1f | 13:29 |
openstackgerrit | Merged openstack/openstack-ansible-repo_build stable/stein: repo_build: update tests to opendev.org https://review.opendev.org/713606 | 13:30 |
ygk_12345 | can anyone help me with that ? | 13:34 |
*** spatel has joined #openstack-ansible | 13:35 | |
jrosser | ygk_12345: on rocky isnt placement part of nova? | 13:36 |
ygk_12345 | @jross | 13:37 |
ygk_12345 | jrosser it is I think so | 13:37 |
jrosser | right - so i don't think there is anything specific you need to do - the OSA deployment will/should take care of that | 13:38 |
ygk_12345 | @jrso | 13:38 |
jrosser | it's not really a plugin either | 13:38 |
ygk_12345 | jrosser but I dont fins placement option in openstack client to list them | 13:38 |
*** DanyC has quit IRC | 13:39 | |
ygk_12345 | I am facing this error when lauching a vm on a specific compute | 13:39 |
ygk_12345 | Failed to retrieve resource provider tree from placement API for UUID ea3af0ff-fb69-449a-b484-5ac722010a1f | 13:39 |
*** ansmith_ has joined #openstack-ansible | 13:42 | |
*** keeper_ has quit IRC | 13:44 | |
*** ansmith has quit IRC | 13:44 | |
*** keeper has joined #openstack-ansible | 13:46 | |
*** DanyC has joined #openstack-ansible | 13:47 | |
ygk_12345 | i also see this line in the openvswitch status in compute node | 13:49 |
ygk_12345 | ovs|00003|db_ctl_base|ERR|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identical values | 13:49 |
*** fghaas has left #openstack-ansible | 13:51 | |
*** dave-mccowan has joined #openstack-ansible | 13:57 | |
*** yoctozepto1 has joined #openstack-ansible | 14:12 | |
*** yoctozepto has quit IRC | 14:13 | |
*** yoctozepto1 is now known as yoctozepto | 14:13 | |
*** sluna has quit IRC | 14:29 | |
*** sluna has joined #openstack-ansible | 14:29 | |
*** ygk_12345 has quit IRC | 14:30 | |
openstackgerrit | Merged openstack/openstack-ansible-repo_server stable/stein: lsyncd: use relative paths in exclude statement https://review.opendev.org/712456 | 14:39 |
*** fridtjof[m] has quit IRC | 14:50 | |
*** fridtjof[m] has joined #openstack-ansible | 14:54 | |
*** macz_ has joined #openstack-ansible | 14:55 | |
*** udesale_ has quit IRC | 15:32 | |
*** prometheanfire has joined #openstack-ansible | 15:37 | |
*** mkuf_ has quit IRC | 15:38 | |
*** joshualyle has joined #openstack-ansible | 15:44 | |
*** jawad_axd has quit IRC | 15:45 | |
idlemind | is centos now largely deprecated for osa? i'm assuming a lack of maintainership? | 15:51 |
jrosser | it's not deprecated, but it could sure use some extra effort | 16:08 |
idlemind | jrosser thanks, is centos 8 in the works? do we just need testing? | 16:10 |
spotz | back on IRC if needed:) | 16:11 |
*** joshualyle has quit IRC | 16:13 | |
jrosser | idlemind: centos8 needs more effort | 16:13 |
*** cshen has quit IRC | 16:14 | |
*** fresta has joined #openstack-ansible | 16:22 | |
*** cshen has joined #openstack-ansible | 16:26 | |
*** sshnaidm has quit IRC | 16:28 | |
*** sshnaidm has joined #openstack-ansible | 16:28 | |
*** cshen has quit IRC | 16:30 | |
CeeMac | jrosser: in your sample env.d if understand correctly, you would have dnsaas-bond_hosts populated out in o_u_c.yml right? I'm a little rusty with this again | 16:46 |
openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group https://review.opendev.org/707849 | 16:46 |
openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch discovery nodes https://review.opendev.org/708826 | 16:46 |
jrosser | CeeMac: i think so yes | 16:47 |
jrosser | just like one ofthe other sections in there | 16:47 |
CeeMac | just found the readme, so i'm reading that :) | 16:48 |
CeeMac | do I also remember seeing someone posted up a playbook for adding extra services into haproxy too? | 16:50 |
CeeMac | a while back | 16:51 |
*** johanssone has quit IRC | 16:51 | |
jrosser | if you look in the openstack-ansible-ops repo for the ELK stack there is an example of that | 16:52 |
CeeMac | ah, sweet | 16:52 |
CeeMac | 6 or 7 or either | 16:52 |
jrosser | you don't need a playbook, there is a var ready to take extra haproxy endpoints | 16:52 |
jrosser | which you can put in user_variabls | 16:52 |
CeeMac | great | 16:52 |
CeeMac | found it, awesome. I think i can work with this | 16:55 |
*** johanssone has joined #openstack-ansible | 16:56 | |
*** johanssone has quit IRC | 16:57 | |
*** johanssone has joined #openstack-ansible | 17:03 | |
*** johanssone has quit IRC | 17:03 | |
*** johanssone has joined #openstack-ansible | 17:05 | |
*** johanssone has quit IRC | 17:05 | |
*** johanssone has joined #openstack-ansible | 17:06 | |
*** pcaruana has quit IRC | 17:08 | |
*** johanssone has quit IRC | 17:08 | |
*** DanyC has quit IRC | 17:09 | |
*** johanssone has joined #openstack-ansible | 17:14 | |
*** johanssone has quit IRC | 17:15 | |
*** johanssone has joined #openstack-ansible | 17:20 | |
*** johanssone has quit IRC | 17:20 | |
*** johanssone has joined #openstack-ansible | 17:23 | |
*** rmcall has joined #openstack-ansible | 17:26 | |
*** gyee has joined #openstack-ansible | 17:27 | |
*** johanssone has quit IRC | 17:27 | |
*** pcaruana has joined #openstack-ansible | 17:28 | |
*** cshen has joined #openstack-ansible | 17:31 | |
*** cshen has quit IRC | 17:35 | |
*** evrardjp has quit IRC | 17:36 | |
*** evrardjp has joined #openstack-ansible | 17:36 | |
openstackgerrit | Merged openstack/openstack-ansible master: ceph: run handlers after common https://review.opendev.org/711845 | 17:37 |
*** DanyC has joined #openstack-ansible | 17:45 | |
*** DanyC has quit IRC | 17:50 | |
*** idlemind has quit IRC | 18:00 | |
*** miloa has quit IRC | 18:05 | |
*** idlemind has joined #openstack-ansible | 18:08 | |
*** jawad_axd has joined #openstack-ansible | 18:13 | |
*** DanyC has joined #openstack-ansible | 18:40 | |
*** sshnaidm is now known as sshnaidm|afk | 18:41 | |
*** nicolasbock has quit IRC | 18:47 | |
*** nicolasbock has joined #openstack-ansible | 18:54 | |
*** jawad_axd has quit IRC | 18:54 | |
*** DanyC has quit IRC | 18:58 | |
*** DanyC has joined #openstack-ansible | 19:27 | |
*** rpittau is now known as rpittau|afk | 19:40 | |
Nick_A | Anyone using ceph for volumes? Looking for tips on increasing iops | 19:47 |
*** DanyC has quit IRC | 19:48 | |
mgariepy | Nick_A, i do but basically more nvme ? | 20:01 |
mgariepy | Nick_A, which version of ceph are you using and what are your osds? | 20:03 |
mgariepy | but it depends a lot on your workload | 20:03 |
Nick_A | We're using ssds already, but HW RAID (which I understand is not a typical ceph setup). version 13.2.7 | 20:06 |
openstackgerrit | Douglas Mendizábal proposed openstack/ansible-role-atos-hsm stable/train: Add HA (load balancing) support https://review.opendev.org/713732 | 20:06 |
mgariepy | hw raid ? why ? | 20:07 |
Nick_A | It's a dev setup - we're trying to repurpose some existing gear | 20:07 |
mgariepy | can't you get the drive to get passed through? | 20:07 |
mgariepy | directly | 20:07 |
Nick_A | I have not seen a way to do that with these particular cards | 20:11 |
Nick_A | Do you think that has a bearing on the iops? | 20:11 |
mgariepy | not sure. | 20:11 |
mgariepy | you created one raid0/drive i guess ? | 20:12 |
Nick_A | I have not tried that but I see that as a suggested workaround | 20:12 |
mgariepy | what is your current raid config ? | 20:13 |
mgariepy | you are on bluestore ? | 20:14 |
*** Openk10s has joined #openstack-ansible | 20:14 | |
Nick_A | RAID10 sorry | 20:15 |
mgariepy | triple replication over double replication | 20:15 |
mgariepy | try to have one device / exposed virutal one and it should be way better | 20:16 |
mgariepy | how many osd do you have ? | 20:16 |
Nick_A | 3 | 20:17 |
Nick_A | 2 atm actually | 20:17 |
mgariepy | hoe many ssds ? | 20:17 |
mgariepy | s/hoe/how/ | 20:18 |
Nick_A | 8 total, 4 per RAID10 / osd | 20:19 |
*** DanyC has joined #openstack-ansible | 20:25 | |
*** DanyC has quit IRC | 20:31 | |
mgariepy | i guess you will have a better throughput with single drive per raid-ish devices, there wont be any replication on the raid so performance should be better. | 20:33 |
openstackgerrit | Merged openstack/ansible-role-atos-hsm stable/train: Add HA (load balancing) support https://review.opendev.org/713732 | 20:33 |
Nick_A | thanks - looking into raid0/jbod again | 20:34 |
*** mkuf_ has joined #openstack-ansible | 20:34 | |
*** gshippey has quit IRC | 20:36 | |
*** rmcall has quit IRC | 20:38 | |
mgariepy | Nick_A, let me know if it helps. | 20:39 |
*** mkuf_ has quit IRC | 20:39 | |
Nick_A | will do thank you | 20:40 |
*** nicolasbock has quit IRC | 20:40 | |
*** jamesden_ has quit IRC | 21:02 | |
*** jbadiapa has quit IRC | 21:13 | |
*** openstackstatus has joined #openstack-ansible | 21:15 | |
*** ChanServ sets mode: +v openstackstatus | 21:15 | |
openstackgerrit | Merged openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group https://review.opendev.org/707849 | 22:07 |
*** spatel has quit IRC | 22:14 | |
*** nicolasbock has joined #openstack-ansible | 22:23 | |
*** nicolasbock has quit IRC | 22:27 | |
*** nicolasbock has joined #openstack-ansible | 22:29 | |
openstackgerrit | Henry Bonath proposed openstack/openstack-ansible-ops master: Add "rabbit-rescue" script https://review.opendev.org/713751 | 22:36 |
*** DanyC has joined #openstack-ansible | 22:36 | |
*** DanyC has quit IRC | 22:42 | |
*** ivve has quit IRC | 22:50 | |
*** macz_ has quit IRC | 22:56 | |
*** threestrands has joined #openstack-ansible | 22:58 | |
*** spatel has joined #openstack-ansible | 23:27 | |
*** spatel has quit IRC | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!