| mjwales | Following on from Friday we have re-run the rabbitmq and neutron playbooks but the HA routers are not scheduling. All the router HA ports are showing as down. Everything was working on Train before we did the Ussuri upgrade. | 08:18 |
|---|---|---|
| jrosser | morning | 08:40 |
| mjwales | Are we better off trying to upgrade to Victoria so we are on the latest release to aid in debugging? | 08:47 |
| jrosser | have you turned debug on for your neutron l3 agent? | 08:50 |
| mjwales | Yes nothing out of the ordinary was logged i.e. no errors | 08:54 |
| mjwales | The l3 agent logs "Function 'neutron.service.Service.periodic_tasks' run outlasted interval by 73.70 sec" and seems to stop | 08:57 |
| mjwales | l3 agent with debug enabled: http://paste.openstack.org/show/806983/ | 08:59 |
| rohit02 | hi team,we have deployed OSA ussuri/victoria both with all endpoints on TLS but facing one issue while creating the bootable volume | 11:18 |
| rohit02 | https://bugs.launchpad.net/openstack-ansible/+bug/1929913 | 11:19 |
| jrosser | rohit02: without further debug info from your side there is not much we can do | 11:22 |
| jrosser | you will need to look in the cinder api service logs next | 11:23 |
| rohit02 | jrosser: thanx....what debug info you required can please let me know so i can provide you | 11:24 |
| jrosser | there are a couple of things you can do | 11:26 |
| jrosser | use --debug on the cli tool to get better info on what it is doing | 11:26 |
| jrosser | and look in the cinder API log to try to find the root cause | 11:26 |
| rohit02 | jrosser: http://paste.openstack.org/show/806987/ please have a look | 11:40 |
| jrosser | rohit02: HTTP exception thrown: Invalid image identifier or unable to access requested image. | 11:48 |
| jrosser | btw all I can do is look at exactly the same log as you - i don't have any special insight into what is wrong here | 11:49 |
| jrosser | so either the image ID is wrong, or the image is not accessible | 11:49 |
| rohit02 | jrosser: image id is right...we are able to deploy image booted instance with the same image | 12:14 |
| jrosser | rohit02: have you checked that the exact same thing works without ssl-everywhere? | 12:16 |
| jrosser | your bug is "when all endpoints on SSL" so this suggests that you have good evidence that it is the SSL causing the trouble? | 12:16 |
| rohit02 | jrooser: yes...when we keep only public endpoint on ssl and rest on non ssl it works well | 12:16 |
| rohit02 | jrosser: yes ssl causing the issue as per our testing | 12:18 |
| rohit02 | when we keep non ssl it all openstack operations works well | 12:19 |
| jrosser | have you done any debugging in this function https://github.com/openstack/cinder/blob/master/cinder/api/v2/volumes.py#L137-L178 | 12:20 |
| jrosser | see that it is querying the glance service `images = list(image_service.detail(context, **params))` | 12:22 |
| jrosser | have you next checked the glance logs? | 12:22 |
| rohit02 | jrooser: we actually using the volume v3 api right? | 13:00 |
| rohit02 | jrooser: in debug mode we are not seeing any volume v2 call | 13:01 |
| jrosser | rohit02: look in your paste (http://paste.openstack.org/show/806987/), the version numbers are shown there in some of the URL | 13:07 |
| rohit02 | jrosser: we are unable to set pdb on mentioned file to debug it....can you please guide us on the same | 13:36 |
| jrosser | rohit02: you could add some logging statements like these https://github.com/openstack/cinder/blob/master/cinder/api/v2/volumes.py#L185 | 14:06 |
| jrosser | rohit02: did you look in the glance logs? | 14:06 |
| spatel | folks, does someone has haproxy grafana template ? | 14:18 |
| spatel | in market all i am seeing prometheus template but not influx :) | 14:18 |
| jrosser | we did prometheus exporter for haproxy | 14:21 |
| spatel | https://grafana.com/grafana/dashboards/1234 | 14:26 |
| spatel | i found this and look like works.. | 14:26 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Drop CentOS overrides for systemd version https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797610 | 16:55 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!