Wednesday, 2020-03-18

*** DanyC has quit IRC00:14
*** spotz has joined #openstack-ansible00:28
*** gyee has quit IRC00:38
*** jbadiapa has quit IRC00:53
*** dave-mccowan has quit IRC00:55
*** nicolasbock has quit IRC01:01
*** boldo has quit IRC01:08
*** ivve has quit IRC01:27
*** bold has joined #openstack-ansible02:07
*** rh-jelabarre has quit IRC03:10
*** raukadah is now known as chandankumar04:37
*** udesale has joined #openstack-ansible04:54
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-ansible05:36
*** mkuf has joined #openstack-ansible06:44
*** miloa has joined #openstack-ansible07:06
ebbexAnyone seen this before? Unhandled error: CappedVersionUnk07:07
ebbexnown: Unrecoverable Error: Versioned Objects in DB are capped to unknown version 1.38.07:07
ebbexI'm upgrading from rocky to stein, and cinder-api won't start because of this.07:08
ebbex(Stein 19.0.10)07:08
*** mkuf_ has joined #openstack-ansible07:15
*** mkuf has quit IRC07:19
*** miloa has quit IRC07:26
*** cshen has joined #openstack-ansible07:33
*** djhankb has quit IRC07:37
*** miloa has joined #openstack-ansible07:39
*** bold has quit IRC07:46
*** dmsimard|off7 has joined #openstack-ansible07:48
*** dmsimard|off has quit IRC07:48
*** dmsimard|off7 is now known as dmsimard|off07:48
*** miloa has quit IRC07:56
*** tosky has joined #openstack-ansible07:57
*** miloa has joined #openstack-ansible08:01
*** djhankb has joined #openstack-ansible08:04
*** andrewbonney has joined #openstack-ansible08:10
*** yoctozepto9 has joined #openstack-ansible08:14
*** yoctozepto has quit IRC08:15
*** yoctozepto9 is now known as yoctozepto08:15
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/71359008:24
*** sshnaidm has quit IRC08:29
*** cshen has quit IRC08:38
*** rpittau|afk is now known as rpittau08:45
*** yoctozepto has quit IRC08:49
*** cshen has joined #openstack-ansible08:57
*** yoctozepto has joined #openstack-ansible08:59
*** ygk_12345 has joined #openstack-ansible08:59
ebbexI bumped osa to stein/master as i suspected something was wrong with the cinder at the git-hash in 19.0.10, and now it's working. Which is kinda odd, as I've done this upgrade on 19.0.10 previously without a hitch.09:00
jrosserebbex double check that cinder-volume and cinder-api are actually the version you expect09:02
jrosseralso that the service is restarted properly so that the installed version is actually the running one09:03
jrosseri think a mismatch there can cause that sort of error message09:03
*** DanyC has joined #openstack-ansible09:11
*** jbadiapa has joined #openstack-ansible09:36
openstackgerritMerged openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/71359009:36
*** ygk_12345 has quit IRC09:37
*** admin0 has quit IRC09:39
ebbexthis was during the 'scripts/run-upgrade.sh', the service with the error was /openstack/venvs/cinder-19.0.10/bin/cinder-scheduler09:40
ebbexon the production cluster which i upgraded earlier in february, the object version is 1.3709:42
jrosserif you get those errors with git hashes clear out the specific (or all) ansible roles in /etc/ansible/roles09:43
ebbexfrom the same 19.0.10 point release...09:44
jrosseri found this which looks similar https://ask.openstack.org/en/question/100160/cinder-cappedversionunknown-unrecoverable-error-versioned-objects-in-db-are-capped-to-unknown-version-111/09:46
ebbexjrosser: already do, i run a playbook that checks out the 19.0.10 point release, clears out /etc/ansible/roles, runs boostrap-ansible, comment it out in the run-upgrade.sh, and apply additional patches/cherry-picks on the /etc/ansible/roles.09:46
openstackgerritMerged openstack/openstack-ansible-repo_build stable/train: repo_build: update tests to opendev.org  https://review.opendev.org/71331309:48
ebbexnone of the cherry-picks mess with cinder. The 19.0.10 release i upgraded to in production is as mentioned on object version 1.37.09:48
ebbexHmm, unless they bumped something in cinder during the rocky 18.1.19, that makes it incompatible with 19.0.10? (Production was upgraded from 18.1.17)09:50
openstackgerritMerged openstack/openstack-ansible-repo_server stable/rocky: lsyncd: use relative paths in exclude statement  https://review.opendev.org/71248209:52
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Refactor memcached_servers  https://review.opendev.org/71323209:56
*** bjoernt_ has joined #openstack-ansible09:58
*** yoctozepto has quit IRC09:59
openstackgerritErik Berg proposed openstack/openstack-ansible-repo_build stable/stein: repo_build: update tests to opendev.org  https://review.opendev.org/71360610:00
*** keeper_ has joined #openstack-ansible10:01
*** yoctozepto has joined #openstack-ansible10:03
*** keeper has quit IRC10:04
*** bjoernt has quit IRC10:04
*** yoctozepto2 has joined #openstack-ansible10:13
*** yoctozepto has quit IRC10:13
*** yoctozepto2 is now known as yoctozepto10:13
*** noonedeadpunk has joined #openstack-ansible10:14
*** sshnaidm has joined #openstack-ansible10:20
*** jawad_axd has joined #openstack-ansible10:50
CeeMacmorning10:51
jrossermorning10:52
*** sshnaidm has quit IRC10:54
*** sshnaidm has joined #openstack-ansible10:55
CeeMacchannel seems quiet these days? :(10:55
CeeMaci've been trying to lurk when/where I can in between project-of-doom-that-never-ends10:56
CeeMachows everyone holding up?10:56
jrosserno so bad10:56
jrosserjust getting used to full time working from home10:56
CeeMacyeah10:56
CeeMaci miss my monitors :(10:57
jrosseri have a fairly ancient imac that turns out it can also be a display10:57
CeeMacone of the screens with built in mac ones? like on zoolander? :)10:58
openstackgerritMerged openstack/openstack-ansible-os_panko master: Refactor memcached_servers  https://review.opendev.org/71324910:59
openstackgerritMerged openstack/openstack-ansible-os_swift master: Refactor memcached_servers  https://review.opendev.org/71326210:59
openstackgerritMerged openstack/openstack-ansible-os_trove master: Refactor memcached_servers  https://review.opendev.org/71326311:01
jawad_axdHi ! A question: I created couple of instances with 1TB/12Vcpus/24GB_ram. I noticed hypervisor summary in horizon did not update itself...while instances are running successfully.What could be the reason?11:02
*** nicolasbock has joined #openstack-ansible11:03
openstackgerritMerged openstack/openstack-ansible-os_masakari master: Refactor memcached_servers  https://review.opendev.org/71324411:05
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: rabbitmq_server: there's no system change on a package query  https://review.opendev.org/71312411:14
jawad_axdAnswer of my above question: Restarting nova-compute on host fixed the issue and hypervisor summary got updated.11:19
*** ivve has joined #openstack-ansible11:47
*** cshen has quit IRC11:52
*** rh-jelabarre has joined #openstack-ansible11:53
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group  https://review.opendev.org/70784911:54
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch discovery nodes  https://review.opendev.org/70882611:54
*** rpittau is now known as rpittau|bbl12:01
*** cshen has joined #openstack-ansible12:03
CeeMachas anyone tried running block-storage ha across the cinder-api containers? From what I can see, this isnt baked in to the role currently12:07
CeeMaclooking at https://docs.openstack.org/ha-guide/storage-ha-block.html12:07
jrosserCeeMac: you mean something beyond HAProxy load balancing the API requests?12:10
*** jbadiapa has quit IRC12:19
CeeMacjrosser: yeah, the doc has a bunch of services running as cluster resources in pacemaker.12:23
CeeMacGuess I'll need to read some more on what difference that makes rather than using haproxy to balance api calls12:24
jrosserim not really sure it's functionally any different to multiple haproxy+keepalived12:25
jrosserisnt pacemaker an alternative take on that?12:25
CeeMacStill a trying to work out where the pain points are. I think it's more probably around the cinder-volume side. Trying to wrap a process around performing maintenance on a storage node and how that affects mounted volumes etc12:25
jrosserthat page really does only seem to talk about the API service12:26
jrosserthats not the same as HA for storage nodes12:26
*** jamesden_ has joined #openstack-ansible12:28
jrosserfuther on in the same documentation they go on to point at things like ceph rbd for a solution to HA in the actual backend https://docs.openstack.org/ha-guide/storage-ha-backend.html12:28
noonedeadpunkYEar, working from home is pretty nasty thing... Just getting too much distracted by different stuff...12:32
*** gshippey has joined #openstack-ansible12:40
*** udesale_ has joined #openstack-ansible12:44
*** udesale has quit IRC12:47
*** rpittau|bbl is now known as rpittau13:01
*** ygk_12345 has joined #openstack-ansible13:04
*** fghaas has joined #openstack-ansible13:09
*** sep has quit IRC13:09
CeeMacjrosser: yeah think I need to do more reading.13:10
CeeMacOn an unrelated note. How easy/hard is it to add a custom container to the inventory and deployment?13:14
jrosserthats reasonably straightforward13:17
jrosseryou make new stuff in /etc/openstack_deploy/env.d/....13:17
jrosseri have a little example of that here https://github.com/jrosser/openstack-ansible-ops/blob/designate-bind/designate-bind/env.d/dnsaas-bind.yml13:19
CeeMacAh, great thanks13:19
jrosserbut really it's no more than taking one of the existing container defintoins from the openstack-ansible env.d directory and adjusting the names to suit13:20
jrosserthen assign hosts for those continaers in openstack_user_config13:20
mnaserjrosser: also i think there is a doc about it13:21
mnaseri cant seem to find it tho13:22
jrosserno me neither13:23
CeeMacI'll take a punt and see how I get on13:24
*** jbadiapa has joined #openstack-ansible13:24
*** cshen has quit IRC13:25
jrosseronce you've fiddled env.d and o_u_c.yml use inventory_manage.py to see if the new groups/hosts have been created13:25
*** cshen has joined #openstack-ansible13:25
mnaser^ and run dynamic inventory at least once to update it13:26
CeeMacRight13:26
CeeMacI was going to ask about that, I think I've got a record of that command13:27
CeeMacThanks guys :)13:27
ygk_12345hi all13:27
CeeMacHi13:28
ygk_12345how to install placement plugin for openstack client in openstack ansible ?13:28
ygk_12345i am receiving these errors when in  rocky when launching a vm on a specific compute node13:29
ygk_12345Failed to retrieve resource provider tree from placement API for UUID ea3af0ff-fb69-449a-b484-5ac722010a1f13:29
openstackgerritMerged openstack/openstack-ansible-repo_build stable/stein: repo_build: update tests to opendev.org  https://review.opendev.org/71360613:30
ygk_12345can anyone help me with that ?13:34
*** spatel has joined #openstack-ansible13:35
jrosserygk_12345: on rocky isnt placement part of nova?13:36
ygk_12345@jross13:37
ygk_12345jrosser it is I think so13:37
jrosserright - so i don't think there is anything specific you need to do - the OSA deployment will/should take care of that13:38
ygk_12345@jrso13:38
jrosserit's not really a plugin either13:38
ygk_12345jrosser but I dont fins placement option in openstack client to list them13:38
*** DanyC has quit IRC13:39
ygk_12345I am facing this error when lauching a vm on a specific compute13:39
ygk_12345Failed to retrieve resource provider tree from placement API for UUID ea3af0ff-fb69-449a-b484-5ac722010a1f13:39
*** ansmith_ has joined #openstack-ansible13:42
*** keeper_ has quit IRC13:44
*** ansmith has quit IRC13:44
*** keeper has joined #openstack-ansible13:46
*** DanyC has joined #openstack-ansible13:47
ygk_12345i also see this line in the openvswitch status in compute node13:49
ygk_12345ovs|00003|db_ctl_base|ERR|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identical values13:49
*** fghaas has left #openstack-ansible13:51
*** dave-mccowan has joined #openstack-ansible13:57
*** yoctozepto1 has joined #openstack-ansible14:12
*** yoctozepto has quit IRC14:13
*** yoctozepto1 is now known as yoctozepto14:13
*** sluna has quit IRC14:29
*** sluna has joined #openstack-ansible14:29
*** ygk_12345 has quit IRC14:30
openstackgerritMerged openstack/openstack-ansible-repo_server stable/stein: lsyncd: use relative paths in exclude statement  https://review.opendev.org/71245614:39
*** fridtjof[m] has quit IRC14:50
*** fridtjof[m] has joined #openstack-ansible14:54
*** macz_ has joined #openstack-ansible14:55
*** udesale_ has quit IRC15:32
*** prometheanfire has joined #openstack-ansible15:37
*** mkuf_ has quit IRC15:38
*** joshualyle has joined #openstack-ansible15:44
*** jawad_axd has quit IRC15:45
idlemindis centos now largely deprecated for osa? i'm assuming a lack of maintainership?15:51
jrosserit's not deprecated, but it could sure use some extra effort16:08
idlemindjrosser thanks, is centos 8 in the works? do we just need testing?16:10
spotzback on IRC if needed:)16:11
*** joshualyle has quit IRC16:13
jrosseridlemind: centos8 needs more effort16:13
*** cshen has quit IRC16:14
*** fresta has joined #openstack-ansible16:22
*** cshen has joined #openstack-ansible16:26
*** sshnaidm has quit IRC16:28
*** sshnaidm has joined #openstack-ansible16:28
*** cshen has quit IRC16:30
CeeMacjrosser: in your sample env.d if understand correctly, you would have dnsaas-bond_hosts populated out in o_u_c.yml right? I'm a little rusty with this again16:46
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group  https://review.opendev.org/70784916:46
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch discovery nodes  https://review.opendev.org/70882616:46
jrosserCeeMac: i think so yes16:47
jrosserjust like one ofthe other sections in there16:47
CeeMacjust found the readme, so i'm reading that :)16:48
CeeMacdo I also remember seeing someone posted up a playbook for adding extra services into haproxy too?16:50
CeeMaca while back16:51
*** johanssone has quit IRC16:51
jrosserif you look in the openstack-ansible-ops repo for the ELK stack there is an example of that16:52
CeeMacah, sweet16:52
CeeMac6 or 7 or either16:52
jrosseryou don't need a playbook, there is a var ready to take extra haproxy endpoints16:52
jrosserwhich you can put in user_variabls16:52
CeeMacgreat16:52
CeeMacfound it, awesome.  I think i can work with this16:55
*** johanssone has joined #openstack-ansible16:56
*** johanssone has quit IRC16:57
*** johanssone has joined #openstack-ansible17:03
*** johanssone has quit IRC17:03
*** johanssone has joined #openstack-ansible17:05
*** johanssone has quit IRC17:05
*** johanssone has joined #openstack-ansible17:06
*** pcaruana has quit IRC17:08
*** johanssone has quit IRC17:08
*** DanyC has quit IRC17:09
*** johanssone has joined #openstack-ansible17:14
*** johanssone has quit IRC17:15
*** johanssone has joined #openstack-ansible17:20
*** johanssone has quit IRC17:20
*** johanssone has joined #openstack-ansible17:23
*** rmcall has joined #openstack-ansible17:26
*** gyee has joined #openstack-ansible17:27
*** johanssone has quit IRC17:27
*** pcaruana has joined #openstack-ansible17:28
*** cshen has joined #openstack-ansible17:31
*** cshen has quit IRC17:35
*** evrardjp has quit IRC17:36
*** evrardjp has joined #openstack-ansible17:36
openstackgerritMerged openstack/openstack-ansible master: ceph: run handlers after common  https://review.opendev.org/71184517:37
*** DanyC has joined #openstack-ansible17:45
*** DanyC has quit IRC17:50
*** idlemind has quit IRC18:00
*** miloa has quit IRC18:05
*** idlemind has joined #openstack-ansible18:08
*** jawad_axd has joined #openstack-ansible18:13
*** DanyC has joined #openstack-ansible18:40
*** sshnaidm is now known as sshnaidm|afk18:41
*** nicolasbock has quit IRC18:47
*** nicolasbock has joined #openstack-ansible18:54
*** jawad_axd has quit IRC18:54
*** DanyC has quit IRC18:58
*** DanyC has joined #openstack-ansible19:27
*** rpittau is now known as rpittau|afk19:40
Nick_AAnyone using ceph for volumes? Looking for tips on increasing iops19:47
*** DanyC has quit IRC19:48
mgariepyNick_A, i do but basically more nvme ?20:01
mgariepyNick_A, which version of ceph are you using and what are your osds?20:03
mgariepybut it depends a lot on your workload20:03
Nick_AWe're using ssds already, but HW RAID (which I understand is not a typical ceph setup). version 13.2.720:06
openstackgerritDouglas Mendizábal proposed openstack/ansible-role-atos-hsm stable/train: Add HA (load balancing) support  https://review.opendev.org/71373220:06
mgariepyhw raid ? why ?20:07
Nick_AIt's a dev setup - we're trying to repurpose some existing gear20:07
mgariepycan't you get the drive to get passed through?20:07
mgariepydirectly20:07
Nick_AI have not seen a way to do that with these particular cards20:11
Nick_ADo you think that has a bearing on the iops?20:11
mgariepynot sure.20:11
mgariepyyou created one raid0/drive i guess ?20:12
Nick_AI have not tried that but I see that as a suggested workaround20:12
mgariepywhat is your current raid config ?20:13
mgariepyyou are on bluestore ?20:14
*** Openk10s has joined #openstack-ansible20:14
Nick_ARAID10 sorry20:15
mgariepytriple replication over double replication20:15
mgariepytry to have one device / exposed virutal one and it should be way better20:16
mgariepyhow many osd do you have ?20:16
Nick_A320:17
Nick_A2 atm actually20:17
mgariepyhoe many ssds ?20:17
mgariepys/hoe/how/20:18
Nick_A8 total, 4 per RAID10 / osd20:19
*** DanyC has joined #openstack-ansible20:25
*** DanyC has quit IRC20:31
mgariepyi guess you will have a better throughput with single drive per raid-ish devices, there wont be any replication on the raid so performance should be better.20:33
openstackgerritMerged openstack/ansible-role-atos-hsm stable/train: Add HA (load balancing) support  https://review.opendev.org/71373220:33
Nick_Athanks - looking into raid0/jbod again20:34
*** mkuf_ has joined #openstack-ansible20:34
*** gshippey has quit IRC20:36
*** rmcall has quit IRC20:38
mgariepyNick_A, let me know if it helps.20:39
*** mkuf_ has quit IRC20:39
Nick_Awill do thank you20:40
*** nicolasbock has quit IRC20:40
*** jamesden_ has quit IRC21:02
*** jbadiapa has quit IRC21:13
*** openstackstatus has joined #openstack-ansible21:15
*** ChanServ sets mode: +v openstackstatus21:15
openstackgerritMerged openstack/openstack-ansible-ops master: Subdivision of elastic-logstash group  https://review.opendev.org/70784922:07
*** spatel has quit IRC22:14
*** nicolasbock has joined #openstack-ansible22:23
*** nicolasbock has quit IRC22:27
*** nicolasbock has joined #openstack-ansible22:29
openstackgerritHenry Bonath proposed openstack/openstack-ansible-ops master: Add "rabbit-rescue" script  https://review.opendev.org/71375122:36
*** DanyC has joined #openstack-ansible22:36
*** DanyC has quit IRC22:42
*** ivve has quit IRC22:50
*** macz_ has quit IRC22:56
*** threestrands has joined #openstack-ansible22:58
*** spatel has joined #openstack-ansible23:27
*** spatel has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!