| *** mhen_ is now known as mhen | 01:46 | |
| *** __ministry is now known as Guest1775 | 02:08 | |
| bhola | frickler, The strange thing is logs showing error. | 13:14 |
|---|---|---|
| frickler | bhola: please use paste.opendev.org and show the complete error | 13:14 |
| bhola | I have installed every service on a separate vm on virtualbox. | 13:14 |
| bhola | frickler, yup I am going to do it. But just to let you know that there are 9 VMs running on my Virtualbox. | 13:16 |
| bhola | Controller has chrony server, etcd mariadb server, memcached and rabbitmq | 13:17 |
| bhola | keystone vm is running keystone service and horizon. | 13:17 |
| bhola | then every single VM is running a single service from openstack services | 13:18 |
| frickler | that's a strange selection. the usual first step would be to run everything in a single vm | 13:18 |
| bhola | frickler, that also didn't work. So I distributed them on different VMs. The good thing is they are communicating with each other without any error. | 13:19 |
| bhola | frickler, Looking at the log, at the moment only Glance vm is showing the error. I am going to put it in the pastebin. | 13:21 |
| bhola | frickler, log from Glance VM https://pastebin.com/eCX9JSLg | 13:24 |
| frickler | bhola: this looks ok, the errors at startup should not be relevant if you use the file backend as I think is done in the install guide. so let's look at nova logs, what do you see there when you start an instance? | 13:28 |
| frickler | also are you using horizon for that or CLI? | 13:28 |
| bhola | frickler, I am using both of them but error is appearing on both. By the way I can't see log files for keystone in /var/log/keystone directory. this is empty. keystone log files are in /var/log/apache2 directory. Is this normal? | 13:31 |
| frickler | bhola: yes, this is normal with the way ubuntu deploys the service | 13:34 |
| bhola | Is there any site where I can upload images as well just like pastebin? | 13:35 |
| frickler | bhola: can you run the CLI command as "openstack --debug server create" and paste the complete output? feel free to redact the URLs that are mentioned | 13:35 |
| bhola | frickler, sure | 13:36 |
| frickler | bhola: there is imgur.com, but let's try to stick to text based logs as far as possible | 13:36 |
| bhola | frickler, Here is the output. I sources demo-openrc https://pastebin.com/2zamxUfy | 13:39 |
| frickler | bhola: thx, so there must be some traceback matching that 500 error in the nova-api log. btw. do you run all nova services in the same vm, then? | 13:42 |
| bhola_ | hi. | 13:54 |
| bhola_ | This is bhola. I am out to pick up my child from school. I will be back in 20 mins. | 13:56 |
| bhola | frickler, I am back. | 14:27 |
| bhola | No nova-api is running on separate vm and nova-compute on separate vm | 14:27 |
| frickler | ok, so check logs for both and possibly also nova-conductor | 14:28 |
| bhola | frickler, OK | 14:29 |
| bhola | frickler, This is from nova-api.log https://pastebin.com/iCBdXvYA | 14:31 |
| bhola | nova-conductor has nothing about it after I issued the command. | 14:33 |
| bhola | Nova-Schedular log also got nothing in it. | 14:33 |
| bhola | nova-compute long on other vm also got nothing in it. | 14:34 |
| bhola | This is how openstack services are deployed on different VMs https://imgur.com/7LVz3Mt.png https://imgur.com/JcpRn32.png https://imgur.com/1YZasZo.png https://imgur.com/GmSRKNl.png | 14:38 |
| frickler | bhola: o.k., so nova cannot talk to keystone properly, getting some SSL error | 14:39 |
| frickler | in your OSC log you have http://keystone:5000/identity/v3, not https | 14:40 |
| bhola | It cannot talk to keystone only when creating an instance? | 14:40 |
| frickler | can you try to do "curl https://keystone:5000/identity/" from the nova-api vm? | 14:41 |
| bhola | https is used in separate section [service_user]. As I said I just followed the install guide. All I modified was to add port no 5000 to it. The original setting under section [service_user] was auth_url = https://keystone/identity and I changed it to https://keystone:5000/identity | 14:44 |
| bhola | frickler, it has gone into a prompt waiting for something to enter. | 14:46 |
| bhola | frickler, | 14:47 |
| bhola | frickler, ah. got it. | 14:47 |
| bhola | without port number I got error "curl: (7) Failed to connect to keystone port 443 after 0 ms: Connection refused" and with port number I got error "curl: (35) error:0A00010B:SSL routines::wrong version number" | 14:49 |
| frickler | and what about "curl http://keystone:5000/identity/" ? | 14:49 |
| bhola | looks like I got the response. "versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://keystone:5000/identity/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}} | 14:51 |
| bhola | frickler, you want me to test it with unsecure http? | 14:52 |
| frickler | bhola: yes, seems that is what you deployed, so need to use that in nova.conf, too | 14:54 |
| bhola | unsecure request without port number I get a response but like this <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> | 14:55 |
| bhola | <html><head> | 14:55 |
| bhola | <title>403 Forbidden</title> | 14:55 |
| bhola | </head><body> | 14:55 |
| bhola | <h1>Forbidden</h1> | 14:55 |
| bhola | <p>You don't have permission to access this resource.</p> | 14:55 |
| bhola | <hr> | 14:55 |
| bhola | <address>Apache/2.4.52 (Ubuntu) Server at keystone Port 80</address> | 14:55 |
| bhola | </body></html> | 14:55 |
| frickler | yes, just use http://keystone:5000/... in your nova.conf everywhere | 14:57 |
| *** __ministry is now known as Guest1815 | 15:00 | |
| bhola | frickler, I added a setting on compute node under [DEFAULT] "instances_path = /var/lib/nova/instances" after reading from a forum. It fixed a number of errors. This setting is not in install guide. Do I need it? | 15:00 |
| frickler | well you didn't show errors related to that yet, but that setting doesn't sound wrong, either | 15:05 |
| bhola | frickler, Now It created the instance but with error "Error: Failed to perform requested operation on instance "provider-instance1", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance afa38e79-56a3-4765-80e1-26fd7c98540b.] | 15:05 |
| *** dasTor_ is now known as dasTor | 15:08 | |
| bhola | frickler, it fails at building the image while spawining. it even gets ip address and then removes it and then fails at spawning. | 15:16 |
| bhola | keystone log showing this "WARNING keystone.server.flask.application [req-9096a675-dfa1-4bd2-a885-d5d25538d696 bca8d7a0353246f884bd73abfcab323f - - default -] Authorization failed. The request you have made requires authentication. from 192.168.200.12: keystone.exception.Unauthorized: The request you have made requires authentication.\x1b[00m" | 15:19 |
| frickler | which VM is 192.168.200.12? | 15:34 |
| Fionan | that's keystone-vm @fricker | 15:38 |
| frickler | ok, so since keystone usually doesn't talk to itself, that will likely have been from horizon | 15:43 |
| frickler | so for the instance build failure, there should be a lot of logs on nova-compute | 15:43 |
| bhola | frickler, this is keystone and horizon | 15:53 |
| bhola | frickler, Fionan I think I neeed to setup rootwrap setting on nova and/or compute node | 15:54 |
| bhola | the error I am getting is WARNING oslo.privsep.daemon [-] privsep log: sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper | 15:57 |
| bhola | 2024-08-29 15:25:40.311 1042 WARNING oslo.privsep.daemon [-] privsep log: sudo: a password is required | 15:57 |
| bhola | 2024-08-29 15:25:40.333 1042 CRITICAL oslo.privsep.daemon [req-342ea2ef-b433-42c6-bcbc-bc1db62c7082 - - - - -] privsep helper command exited non-zero (1) | 15:57 |
| bhola | 2024-08-29 15:25:40.334 1042 ERROR oslo_service.periodic_task [req-342ea2ef-b433-42c6-bcbc-bc1db62c7082 - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) | 15:57 |
| bhola | This is on neutron node | 15:58 |
| frickler | so that looks like you need to set up passwordless sudo. but I would be surprised if that wasn't mentioned in the install guide | 16:03 |
| *** rebtoor_ is now known as rebtoor | 16:08 | |
| bhola | I followed (Bobcat) guide and It is not there. I googled this error and found this page https://stackoverflow.com/questions/61139735/neutron-error-oslo-privsep-daemon-failedtodropprivileges-privsep-helper-comman | 16:09 |
| frickler | bhola: see https://docs.openstack.org/nova/2024.1/admin/root-wrap-reference.html | 16:09 |
| frickler | it is meant to be linked to from https://docs.openstack.org/install-guide/environment-security.html , but that only has pretty old pages | 16:10 |
| bhola | nova ALL = (root) NOPASSWD: /usr/bin/nova-rootwrap /etc/nova/rootwrap.conf * This entry needs to be put in sudoer file? | 16:19 |
| frickler | I'm not sure about the exact syntax, but something similar like that for sure | 16:23 |
| cru | I see in the latest documentation that the charms install/deployment path does not exist in the latest version, is this just a documentation thing or will the charms path not be supported in the future? | 17:02 |
| frickler | cru: this may just be a documentation issue, let me check some things | 17:06 |
| *** __ministry is now known as Guest1822 | 17:12 | |
| cru | thanks @frickler | 17:18 |
| bhola | frickler, moved a bit closer. Permissions fixed. Now Neutron keep throwing an error in log file. ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Bridge enp0s9 for physical network provider does not exist. Agent terminated! | 23:17 |
| bhola | enp0s9 is the provider interface | 23:18 |
| bhola | mentioned in the config file as outlined in the install guide. | 23:18 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
