| *** gmann is now known as gmann_afk | 00:07 | |
| *** gmann_afk is now known as gmann | 01:11 | |
| tabacha | Hi, I think i found a bug in the flavour cmd, I reported it here: https://storyboard.openstack.org/#!/story/2009260 and fixed it here: https://review.opendev.org/c/openstack/ansible-collections-openstack/+/811739 what are the next steps to get it back to the git repo? | 06:14 |
|---|---|---|
| *** frickler_ is now known as frickler | 07:25 | |
| noonedeadpunk | tabacha: I guess you need to ask this in #opanstack-ansible-sig if that's about merging that patch to a collection | 07:49 |
| tabacha | noonedeadpunk: okay I asked there. | 07:56 |
| *** odyssey4me is now known as Guest1483 | 08:12 | |
| baptistemm | hello | 14:53 |
| noonedeadpunk | o/ | 14:54 |
| baptistemm | I'm trying to make the ansible inventory plugin for openstack returning something, but it gives me a 403 http error, where using openstack-inventory script works | 14:55 |
| baptistemm | ansible-inventory does not give me useful input for troubleshooting | 14:56 |
| baptistemm | can a good soul give me some hints | 14:56 |
| baptistemm | I've modified ansible.cfg to have a section [inventory] which has 'enable_plugins = openstack, auto' | 14:57 |
| noonedeadpunk | um.... | 14:57 |
| baptistemm | I've put a clouds.yaml file in ~/.config/openstack/ | 14:58 |
| noonedeadpunk | I'm not quite sure I understand what exactly you're trying to do and why | 14:58 |
| baptistemm | https://pastebin.com/EsDKB9Jk | 14:59 |
| noonedeadpunk | are you sure you're trying to use OSA? | 14:59 |
| noonedeadpunk | Or it's question regarding openstack collections? | 14:59 |
| baptistemm | I'm an openstack user so the latter | 15:00 |
| noonedeadpunk | yeah, I got this :) this channel is related to specific deployment tooling https://opendev.org/openstack/openstack-ansible | 15:00 |
| noonedeadpunk | * https://docs.openstack.org/openstack-ansible | 15:00 |
| noonedeadpunk | which I'm not sure if used in your usecase | 15:01 |
| baptistemm | ah that's a bit misleading | 15:01 |
| baptistemm | :) | 15:01 |
| noonedeadpunk | eventually we have #opanstack-ansible-sig that is related to ansible collections for openstack | 15:01 |
| spatel | Folks, i have very strange issue... with old openstack vs new Wallaby | 15:02 |
| spatel | we have same hardware between Stein and Wallaby and looking at disk IO i am seeing bad performance in Wallaby | 15:03 |
| spatel | is anything changed in Wallaby ? | 15:03 |
| noonedeadpunk | um, you mean storage performance? | 15:04 |
| spatel | https://paste.opendev.org/show/809727/ | 15:05 |
| spatel | I am using local Disk | 15:05 |
| spatel | no ceph storage | 15:05 |
| spatel | noonedeadpunk what do you use for disk_cachemodes = ? | 15:09 |
| noonedeadpunk | and nvmes? | 15:13 |
| noonedeadpunk | But for ceph we do use writeback | 15:13 |
| noonedeadpunk | I'm not sure dd is valid thing to measure | 15:14 |
| noonedeadpunk | What does fio says with randrw? | 15:15 |
| spatel | if i run dd test on host i get very good numbers | 15:15 |
| spatel | but in VM its 3x lower | 15:15 |
| spatel | let me try fio.. | 15:15 |
| spatel | i think there must be some setting in nova which is bottleneck | 15:15 |
| noonedeadpunk | well, if you don't have writeback - that most likely the thing | 15:18 |
| spatel | noonedeadpunk what do you think of these numbers - https://paste.opendev.org/show/809730/ | 15:22 |
| spatel | writeback ? | 15:23 |
| spatel | disk_cachemodes = block=writeback ? | 15:23 |
| noonedeadpunk | `disk_cachemodes = network=writeback` but it's ceph | 15:23 |
| spatel | i have local storage so it should be disk_cachemodes = block=writeback | 15:24 |
| spatel | correct? | 15:24 |
| noonedeadpunk | yeah | 15:24 |
| noonedeadpunk | but your numbers are kind of close | 15:24 |
| spatel | why my dd is very different ? | 15:24 |
| spatel | also my client complaining performance isn't good | 15:25 |
| spatel | they are not complaining about stein release even they have same hardware.. | 15:25 |
| spatel | do i need to rebuild VM after changing disk_cachemodes = block=writeback | 15:25 |
| noonedeadpunk | with same firware versions as well? | 15:25 |
| spatel | i think yes.. | 15:26 |
| spatel | i didn't look at firmware yet.. | 15:26 |
| noonedeadpunk | I mean - cpu vulnarabilities being patched can lead to 10-20% of performance degradation | 15:26 |
| spatel | i am checking now all hardware level details and going from bottom to top | 15:26 |
| spatel | i know what you saying | 15:28 |
| spatel | but in compute node directly i am getting 225 MB/s write speed | 15:29 |
| spatel | and in VM i am getting 80 MB/s write speed | 15:29 |
| noonedeadpunk | and dd from /dev/zero can be hitting some entropy issue | 15:29 |
| noonedeadpunk | fio shows 153MB/s for centos and 136MB/s for ubuntu | 15:29 |
| noonedeadpunk | and 37.3kiops vs 33.2k iops | 15:30 |
| spatel | i know.. but very odd same time other environment dd working great so kind of not feeling good | 15:31 |
| spatel | also customer complaining and i trust them.. | 15:31 |
| spatel | they had no issue in old environment | 15:31 |
| spatel | something is changed, either ubuntu vs centos compute or wallaby has some extra overhead | 15:32 |
| noonedeadpunk | Well, it might be because of missing Haveged or smth like that... | 15:35 |
| spatel | hmm | 15:35 |
| spatel | let me try apt install haveged and see | 15:36 |
| spatel | noonedeadpunk should i use disk_cachemodes=file=writeback ? | 15:52 |
| spatel | kvm using file not block right? | 15:52 |
| spatel | disk_cachemodes = file=writethrough,block=none | 15:52 |
| spatel | noonedeadpunk good news :) | 16:44 |
| spatel | This solved my issue, now performance amazing, 3x more - disk_cachemodes = file=writeback,block=none | 16:45 |
| spatel | one interesting thing if i build ubuntu vm then i don't see any issue but if i build centos 7 vm then i see issue.. | 16:55 |
| spatel | look like its centos and ubuntu combination issue | 16:55 |
| spatel | noonedeadpunk there? | 17:56 |
| mgariepy | spatel, same FS in the vm ? | 18:13 |
| spatel | FS? | 18:13 |
| mgariepy | filesystem | 18:13 |
| spatel | ext3 ? | 18:13 |
| spatel | i have to check that part | 18:14 |
| mgariepy | the hosts is ubuntu ? | 18:14 |
| spatel | compute on ubuntu and vm using centos7 | 18:15 |
| -opendevstatus- NOTICE: The review.opendev.org Gerrit server has become unreachable as of approximately 19:10 UTC due to a networking issue in the provider, but should be reachable again shortly | 19:43 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!