Monday, 2021-05-03

*** spatel_ has joined #openstack-ansible00:57
*** spatel_ is now known as spatel00:57
*** spatel has quit IRC02:02
*** evrardjp has quit IRC02:33
*** evrardjp has joined #openstack-ansible02:33
*** ChipOManiac has joined #openstack-ansible03:56
ChipOManiac@noon04:11
ChipOManiacOh for crissakes...04:11
ChipOManiacnoonedeadpunk So I using the specific Git branch when deploying Masakari with OpenStack-Ansible. Good news is that it finally does get the nodes from "compute service list", bad news is that it now attempts to used the resolved IP address of the hostname *as a hostname* and errors out. -_-04:13
ChipOManiacAt least that's what it looks like.04:13
*** shyamb has joined #openstack-ansible06:16
noonedeadpunkum....06:43
noonedeadpunkso in compute service list you have ip addresses now???06:44
*** shyamb has quit IRC06:56
*** shyamb has joined #openstack-ansible07:08
*** rpittau|afk is now known as rpittau07:19
*** tosky has joined #openstack-ansible07:50
*** shyamb has quit IRC07:56
*** ChipOManiac has quit IRC07:58
*** sshnaidm|off is now known as sshnaidm08:12
*** ChipOManiac has joined #openstack-ansible08:27
ChipOManiacnoonedeadpunk no, but somehow the error from masakari tells me that its trying to resolve an IP address instead of a hostname.08:28
noonedeadpunkcan you paste it with tstack trace?08:29
*** shyamb has joined #openstack-ansible08:31
ChipOManiacnoonedeadpunk Yes. The scenario was I made the segments and added the hosts, the did a pkill -KILL of the qemu-system process to simulate a down. The other one was doing a link down of the network interfaces for management and storage (on a VPN so can't disconnect physically).08:46
noonedeadpunkWell eventually masakari monitors have 3 parts - instance monitor, process monitor and host monitor. Killing process might trigger only process monitor in case it's configured to check for existance of the specific process08:49
noonedeadpunkwhich is not smth set by default by osa08:49
*** shyamb has quit IRC08:49
*** shyamb has joined #openstack-ansible08:49
ChipOManiacOf course, in this case the instancemonitor understood that the process was down. Putting the stack trace and stuff into a paste bin now.08:50
noonedeadpunkbringing link down should actually cause node failover, since then hostmonitor is triggered. as that's when pacemaker marks node as down and masakari hostmnitor listen on the interface for actions that happen08:50
noonedeadpunkSo eventually you need to check if your pacemaker cluster is healthy (ie with `crm_mon -1`) and what's going on the masakari monitors journald08:51
noonedeadpunkAs I don't really understand what happens there workflow wise and where it fails with what issue08:52
ChipOManiacNothing wrong with the Pacemaker/Corosync cluster, I had to set on up anyway to get LINSTOR to HA. :)  The journald output for instancemonitor is the one with said error.08:53
noonedeadpunkso, instancemonitor check up for instance ha specifically. so when it detects instance is down, it tries to evacuate it or reboot. Eventually, I haven't tried this one too much, as was more interested in whole nodes failures rather than in single instance08:56
noonedeadpunkBut would be aesome to see stack trace with error08:56
ChipOManiacStill trying to get you that stack trace. :)09:00
*** gokhani has joined #openstack-ansible09:03
gokhaniHi folks, when I install cinder service on OSA 22.1.0 I am getting 'AnsibleUnicode' object has no attribute 'items' error. May be this bug resolved in new tags. http://paste.openstack.org/show/804916/09:05
gokhaniI defined 2 cinder backends > http://paste.openstack.org/show/804917/09:07
ChipOManiacnoonedeadpunk http://paste.openstack.org/show/804918/ there's the paste.09:09
noonedeadpunkum, and what's crm_mon -1 ?09:11
noonedeadpunkgokhani: and how you've used cinder_backends ?09:14
gokhaninoonedeadpunk, http://paste.openstack.org/show/804917/09:15
noonedeadpunkgokhani: well, I mean cinder_backends are supposed to be as a part of the `container_vars` https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/openstack_user_config.yml.pod.example#L467-L50609:16
*** luksky has joined #openstack-ansible09:16
gokhaninoonedeadpunk, ok got it :) I have defined cinder_backends in user_variables.yml file.09:18
noonedeadpunkwhat I did to repeat code less: http://paste.openstack.org/show/804919/09:19
noonedeadpunkChipOManiac: I kind of suspect, that in pacemaker/corosync nodes are added by IP?09:20
ChipOManiacYes they are. Is that the problem? <facepalm>09:20
noonedeadpunkas node names must match in corosync and compute service lsit09:21
gokhanithanks a lot noonedeadpunk :)09:21
noonedeadpunkas hostmonitor listens to corosync, finds that node is down, and the only identifier it has is node name from pacemaker custer09:21
noonedeadpunkand it assumes that it's the name that should be in nova as well09:21
ChipOManiacnoonedeadpunk Aigoo. Well, I'll change that and get back to you ASAP. Thanks for the help so far, you've been very helpful. :)09:22
noonedeadpunkso as ring0_addr in corosync.conf you should have an IP of interface (ie mgmt) and name match hostname and what's in compute servce list09:23
admin0morning  .. checking if anyone has done stein in focal .. so that xenial -> focal upgrade bypassing bionic09:23
noonedeadpunk(that's the default role behaviour for me for some reason)09:23
noonedeadpunkstein on focal is not working09:24
noonedeadpunkwe added focal in U iirc09:24
admin0ok .. at what point/tag is bionic -> focal upgrade recommended09:24
noonedeadpunkbionic still supported. We drop it in Xena only09:24
admin0aah .. long life :)09:24
noonedeadpunkyeah :) don't want to upgrade haha09:25
*** jbadiapa has joined #openstack-ansible09:25
ChipOManiacnoonedeadpunk Okay, that is the way it is actually, I thought you needed to use the hostname instead of the ip address.09:25
noonedeadpunkBut probably V is most suitable for upgrade...09:25
ChipOManiacThe names are the same as what shows up on compute service list09:26
noonedeadpunkChipOManiac: oh, no....09:26
ChipOManiacnoonedeadpunk Its basically09:28
ChipOManiacnode {09:28
ChipOManiac  ring0_addr: 192.168.51.1009:28
ChipOManiac  name: kalmyk109:28
ChipOManiac  nodeid: 1009:28
ChipOManiac}09:28
ChipOManiacfor each node.09:28
noonedeadpunkyeah, looks correct09:28
noonedeadpunkI wonder if it's jsut instancemonitor that's broken somehow....09:29
ChipOManiacI thought you meant to make it to ring0_addr: hn-1.openstack.local.09:29
noonedeadpunkno-no, ring0_addr is correct for you09:29
noonedeadpunkhave you checked hostmonitor as well?09:29
ChipOManiacYes. Says everything is normal, shows the other two nodes as well as up.09:30
noonedeadpunkand shows them by name?09:30
ChipOManiacInitially wasn't working because I used the wrong network interface.09:31
ChipOManiacYes shows by name.09:31
ChipOManiackalmyk1 is 'online' (current: 'online') etc.09:31
noonedeadpunkhm... this looks like instancemonitor specific issue to me...09:32
ChipOManiacPython issue?09:32
noonedeadpunkoh https://opendev.org/openstack/masakari-monitors/commit/85dda1b0ebab75a8f9997a616cf1a759cad7955809:32
ChipOManiacHost is set in the config as the IP address.09:34
ChipOManiacAs OSA did.09:34
ChipOManiacShould that be changed to the hostname?09:34
noonedeadpunkoh, yeah, that might be the issue!09:34
ChipOManiacLet's see.09:34
noonedeadpunkcan you try replacing `ansible_host` with smth like `inventory_hostname`?09:34
noonedeadpunkor better `ansible_facts['hostname']`?09:35
noonedeadpunkhere https://opendev.org/openstack/openstack-ansible-os_masakari/src/branch/master/templates/masakarimonitors.conf.j2#L209:35
*** shyam89 has joined #openstack-ansible09:35
noonedeadpunkor probably better to add another option there...09:36
noonedeadpunkwhich will be `hostname = {{ ansible_facts['hostname'] }}`09:36
ChipOManiacSure. I just edited it manually now to see if it works.09:37
ChipOManiacAnd restarted the service.09:37
ChipOManiacBut I'll go ahead and do it ansible-wise.09:37
*** shyamb has quit IRC09:38
*** luksky has quit IRC09:47
openstackgerritMerged openstack/openstack-ansible-os_horizon stable/victoria: Install adjutant-ui when the adjutant service is deployed  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/78861109:52
noonedeadpunkChipOManiac: yeah would be awesome to patch that09:58
*** shyam89 has quit IRC10:01
*** shyam89 has joined #openstack-ansible10:02
noonedeadpunkanyway dunno why it picks up IP as `hostname = CONF.hostname or socket.gethostname()` - so should default to hostname anyway...10:04
ChipOManiacnoonedeadpunk I guess I'll post an issue on Github. I've also noticed that the libvirt development packages, and the python development packages aren't installed by openstack-ansible-masakari. Should I add those too?10:05
ChipOManiacI think that's because the entry was host: instead of host: and hostname:10:05
ChipOManiacI saw an OSLO warning complaining about host being deprecated.10:05
noonedeadpunkI think they are? https://opendev.org/openstack/openstack-ansible-os_masakari/src/branch/master/vars/debian.yml#L2310:06
ChipOManiacEither way I added both host: and hostname: to the ansible masakari.conf template.10:06
ChipOManiacOdd, I had to manually install those.10:07
noonedeadpunkoh, so you had to install them on computes?10:07
ChipOManiacYes.10:07
noonedeadpunkhm, I actually thought it's required only to build package, but not to operate10:09
noonedeadpunkit's playbook failed with some error or there was issue in log of the monitors?10:09
ChipOManiacWell the issue happens at 'building wheels" or something.10:10
noonedeadpunkhuh, that's weird indeed. as it's not supposed to run against compute hosts at all10:10
noonedeadpunkit's supposed to be delegated to repo container10:11
ChipOManiacYes.10:11
ChipOManiacAny idea of where I can get a list of properties that can be applied to instances (eg. making them HA).10:11
ChipOManiac?10:11
gokhaninoonedeadpunk, I am getting rbd incomplete errors with glance + ceph like this bug definition. https://bugs.launchpad.net/glance/+bug/1916482 . Do we need to do anything on haproxy side ?10:14
openstackLaunchpad bug 1916482 in Glance "rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) since Victoria Upgrade, ceph v nautilus" [Undecided,New]10:14
noonedeadpunkChipOManiac: I think it's not documented10:15
ChipOManiacAhwell. Too bad.10:15
ChipOManiacThe entry to make an instance HA for Masakari is HA_Enabled is correct however?10:16
noonedeadpunkiirc if you want to explicitly enable/didable that there was some tag required in the metadata of the instance ie ha_enabled or smth like that...10:16
ChipOManiacDoes case matter here?10:17
noonedeadpunkI don't think it's exact tag...10:17
ChipOManiacBecause I just did an interface down, and it didn't move the instance.10:17
ChipOManiacIt does however get the node right. So thanks for that. :)10:17
noonedeadpunkso you set it to the hostname?10:18
ChipOManiacI set both. The host and the hostname.10:18
ChipOManiacSo:10:18
ChipOManiachost: 192.168.50.X10:18
ChipOManiachostname: kalmykX10:19
ChipOManiacThe OSLO complaint has disappeared.10:19
noonedeadpunkyeah, I think host is not needed nowadays indeed10:20
ChipOManiacI just left it as is. Anyway, thank you for helping me fix that. :)10:21
*** shyam89 has quit IRC10:40
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Allow to configure corosync ports  https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/78924110:42
noonedeadpunkgokhani: hm, that's interesting read...10:43
noonedeadpunknot sure atm10:43
noonedeadpunkI'm wondering if just disabling uwsgi for glance and using native service will work better for you10:45
noonedeadpunkie setting `glance_use_uwsgi: false`10:45
gokhaninoonedeadpunk, you are awesome, thanks:)  It could be good for disabling uwsgi when use gilance with ceph10:47
noonedeadpunksetting mode to tcp for http backed client, dunno10:47
noonedeadpunksounds like pretty nasty workaround as well10:48
noonedeadpunkI'm not sure if disabling uwsgi will actually help - worth checking though10:48
gokhaninoonedeadpunk, yes I changed mode to tcp10:48
gokhaniI think we are getting these errors when upload large images10:48
*** dave-mccowan has joined #openstack-ansible10:55
*** jbadiapa has quit IRC10:57
gokhaninoonedeadpunk, after changing mode from http to tcp, it worked. Now ı can upload images to ceph.10:57
gokhaniIn glance docs, it says uWSGI Server HTTP Mode has never worked properly > https://docs.openstack.org/glance/victoria/admin/apache-httpd.html11:05
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Replace deprecated host param for monitors  https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/78924311:07
*** shyam89 has joined #openstack-ansible11:10
*** shyam89 has quit IRC11:10
*** shyamb has joined #openstack-ansible11:10
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Add masakari-introspectiveinstancemonitor support  https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/78924411:12
*** mgariepy has quit IRC11:42
jonherAIO question with cloudkitty dashboard, OPENSTACK_ENDPOINT_TYPE in horizon config is set to "internalURL" by default. Does that not mean it should try the internalURL for cloudkitty(rating service) ? It's trying to use the public one but gets a ssl error since the self-signed cert on HAproxy is not trusted11:48
jonher"OPENSTACK_ENDPOINT_TYPE specifies the endpoint type to use for the endpoints in the Keystone service catalog."11:49
noonedeadpunkI think it actually should11:59
noonedeadpunkBut no idea if cloudkitty respects that actuall11:59
jonheryeah that's what i'm suspecting, just wanted to make sure11:59
*** ThiagoCMC has quit IRC12:04
*** ChipOManiac has quit IRC12:08
*** shyamb has quit IRC12:13
*** shyamb has joined #openstack-ansible12:14
*** shyamb has quit IRC12:17
*** shyamb has joined #openstack-ansible12:17
noonedeadpunkI actually catched several services on that, but never looked into cloudkitty though12:17
jonheryeah i found octavia setting the interface based on OPENSTACK_ENDPOINT_TYPE designate not for example (i think)12:18
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Enable senlin tempest plugin when senlin is deployed  https://review.opendev.org/c/openstack/openstack-ansible/+/75410512:20
*** dave-mccowan has quit IRC12:24
*** shyamb has quit IRC12:26
*** dave-mccowan has joined #openstack-ansible12:26
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add centos-8 stream jobs  https://review.opendev.org/c/openstack/openstack-ansible/+/77622612:26
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add centos-8 stream jobs  https://review.opendev.org/c/openstack/openstack-ansible/+/77622612:26
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add centos-8 stream jobs  https://review.opendev.org/c/openstack/openstack-ansible/+/77622612:28
*** mgariepy has joined #openstack-ansible12:28
*** spatel_ has joined #openstack-ansible12:52
*** spatel_ is now known as spatel12:52
openstackgerritJonathan Herlin proposed openstack/openstack-ansible-os_cloudkitty stable/victoria: Fix wsgi_venv path for cloudkitty-api  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/78926713:03
jonherrebase 780874 ? that would change to using integrated build instead of ansible-functional13:08
*** ThiagoCMC has joined #openstack-ansible13:23
ThiagoCMCMorning guys! Long time no see...  =P13:24
ThiagoCMCMy OpenStack+Ceph Cloud that I deployment in my basement is running for about 6 months already! However, there is one, very annoying, problem. There are a LOT of random "Error 500", which forces me to try again and again until it works. I already replaced the hardware and even a re-deploy but, same problem. Is it just me?13:26
*** d34dh0r53 has joined #openstack-ansible13:30
*** gokhani has quit IRC14:43
*** zbr has quit IRC14:51
*** zbr has joined #openstack-ansible14:52
*** macz_ has joined #openstack-ansible15:26
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix  https://review.opendev.org/c/openstack/openstack-ansible/+/78937615:36
noonedeadpunkjonher: oh, yes:)15:36
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: [goal] Deprecate the JSON formatted policy file  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/78087415:36
noonedeadpunkThiagoCMC:  Error 500 where? That shouldn't happen actually15:37
noonedeadpunk500 produce stack traces somewhere in logs. Depending on what produce the error15:37
*** macz_ has quit IRC15:38
noonedeadpunkcores, any reviews for https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/777607 ? holds adjutant a bit15:40
ThiagoCMCnoonedeadpunk, they're really random, sometimes Horizon can't display the Glance images, sometimes the Ceph volumes, sometimes the Neutron networks... Via CLI, `openstack stack create ...` is particularly bad, I have to run it many times, until the stack is actually created successfully... Sometimes the stack is created but it fails on Nova, while trying to create the instances. :-(15:42
noonedeadpunksounds smth like rabbit or keystone issue...15:42
ThiagoCMCHmmm15:43
noonedeadpunkor, well DB...15:43
ThiagoCMCDamn lol15:43
ThiagoCMCWell, I'm glad to hear that you don't see them! I'll make a new deployment then, based on Wallaby, and compare both...15:43
*** mgariepy has quit IRC15:43
ThiagoCMCI'll also keep an eye on those 3 pieces, Rabbit, Keystone and SQL15:44
noonedeadpunkif they're often - you should be able to easily reproduce them then?15:44
noonedeadpunkand check logs for stack trace once you see it?15:44
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Enable senlin tempest plugin when senlin is deployed  https://review.opendev.org/c/openstack/openstack-ansible/+/75410515:45
ThiagoCMCIt's hard to replicate them consistently (it's kinda random, here and there), but good call. I might take most problematic one, the stack create, and monitor for a while.15:46
noonedeadpunkfor stack create - heat usualy capable of showing error for resource creation15:46
ThiagoCMCIt shows as 50015:47
noonedeadpunkor at least you can see what resource failed and go there for logs15:47
ThiagoCMCgot it15:47
noonedeadpunkif it's failing at create request - the go for heat logs15:47
noonedeadpunk500 is kind of super general thing - means some service fails with python stack trace in logs15:48
*** macz_ has joined #openstack-ansible15:48
ThiagoCMCI see, makes sense... I started to think that this was kinda normal lol - But good to know that it isn't suppose to happen, at all!   =P15:49
noonedeadpunkyeah, it not, otherwise nobody would use that in production :)15:49
ThiagoCMCVery true!  LOL15:49
*** shyamb has joined #openstack-ansible15:54
*** shyamb has quit IRC16:08
*** mgariepy has joined #openstack-ansible16:10
*** rpittau is now known as rpittau|afk16:32
*** gyee has joined #openstack-ansible16:46
*** cp- has joined #openstack-ansible17:40
mgariepynoonedeadpunk, jrosser have you ever seen galera mysql config lost on an upgrade?17:51
mgariepysomehow galera pkg update overwrote the my.cnf file in my container..17:52
*** spatel has quit IRC18:34
*** spatel_ has joined #openstack-ansible18:35
*** spatel_ is now known as spatel18:35
*** ThiagoCMC has quit IRC19:32
*** oleksandry has joined #openstack-ansible19:35
*** ThiagoCMC has joined #openstack-ansible19:47
*** oleksandry has quit IRC19:53
*** cp- has quit IRC19:53
*** gyee has quit IRC19:53
*** poopcat has quit IRC19:53
*** owalsh has quit IRC19:53
*** priteau has quit IRC19:54
*** akahat|ruck has quit IRC19:54
*** bradm has quit IRC19:54
*** lemko has quit IRC19:54
*** oleksandry has joined #openstack-ansible19:55
*** cp- has joined #openstack-ansible19:55
*** gyee has joined #openstack-ansible19:55
*** poopcat has joined #openstack-ansible19:55
*** owalsh has joined #openstack-ansible19:55
*** priteau has joined #openstack-ansible19:55
*** akahat|ruck has joined #openstack-ansible19:55
*** bradm has joined #openstack-ansible19:55
*** lemko has joined #openstack-ansible19:55
*** cp- has quit IRC19:55
*** cp- has joined #openstack-ansible19:56
*** jbadiapa has joined #openstack-ansible20:31
spateljamesdenton around ?20:38
*** jbadiapa has quit IRC20:41
*** spatel has quit IRC20:58
*** snapdeal has joined #openstack-ansible22:14
*** snapdeal has quit IRC22:27
*** tosky has quit IRC22:50
*** ThiagoCMC has quit IRC22:50
*** ThiagoCMC has joined #openstack-ansible23:05
*** macz_ has quit IRC23:24

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!