| *** VW has quit IRC | 00:03 | |
| *** knoxy has quit IRC | 00:05 | |
| *** blacklotus89 has joined #openstack | 00:08 | |
| *** AlexeyAbashkin has joined #openstack | 00:10 | |
| *** catintheroof has joined #openstack | 00:11 | |
| *** eckesicle has joined #openstack | 00:13 | |
| *** burgerk has joined #openstack | 00:13 | |
| *** AlexeyAbashkin has quit IRC | 00:14 | |
| *** jamesbenson has joined #openstack | 00:15 | |
| *** blacklotus89 has quit IRC | 00:17 | |
| *** Cybodog has joined #openstack | 00:17 | |
| *** jamesbenson has quit IRC | 00:19 | |
| *** VW has joined #openstack | 00:20 | |
| *** lucendio1 has quit IRC | 00:20 | |
| *** VW has quit IRC | 00:20 | |
| *** Son_Goku has quit IRC | 00:20 | |
| *** VW has joined #openstack | 00:20 | |
| *** rcernin has joined #openstack | 00:21 | |
| *** dtrainor has quit IRC | 00:22 | |
| *** bobh has joined #openstack | 00:22 | |
| *** gkadam has joined #openstack | 00:23 | |
| *** eroux has quit IRC | 00:25 | |
| *** gkadam_ has quit IRC | 00:25 | |
| *** catintheroof has quit IRC | 00:32 | |
| *** blacklotus89 has joined #openstack | 00:33 | |
| masber | good morning, I have an instance with a volume attached to it, unfortunately the volume is detach every time the instance is rebooted, what would be the way of mount the volume automatically? thank you | 00:33 |
|---|---|---|
| *** burgerk has quit IRC | 00:36 | |
| *** catintheroof has joined #openstack | 00:38 | |
| *** auraka has joined #openstack | 00:39 | |
| *** gyee has quit IRC | 00:40 | |
| *** sree has joined #openstack | 00:40 | |
| *** catintheroof has quit IRC | 00:42 | |
| *** catintheroof has joined #openstack | 00:44 | |
| *** sree has quit IRC | 00:45 | |
| *** AlexeyAbashkin has joined #openstack | 00:46 | |
| *** davidbowlby is now known as davidbowlby-afk | 00:49 | |
| *** AlexeyAbashkin has quit IRC | 00:51 | |
| *** Son_Goku has joined #openstack | 00:52 | |
| *** achadha has joined #openstack | 00:55 | |
| *** achadha has quit IRC | 00:55 | |
| *** catintheroof has quit IRC | 00:55 | |
| *** achadha has joined #openstack | 00:55 | |
| *** catintheroof has joined #openstack | 00:56 | |
| *** charcol has quit IRC | 01:00 | |
| *** knoxy has joined #openstack | 01:00 | |
| *** woodard_ has quit IRC | 01:00 | |
| *** woodard has joined #openstack | 01:01 | |
| *** otherBeeTeeBeats has quit IRC | 01:02 | |
| *** charcol has joined #openstack | 01:04 | |
| *** nshetty has joined #openstack | 01:04 | |
| *** gkadam has quit IRC | 01:07 | |
| *** Son_Goku has quit IRC | 01:12 | |
| *** snowman4839 has quit IRC | 01:13 | |
| *** snowman4839 has joined #openstack | 01:15 | |
| *** redcavalier has joined #openstack | 01:16 | |
| *** thorst has joined #openstack | 01:16 | |
| *** thorst has quit IRC | 01:17 | |
| redcavalier | Hi, I've been running into a neutron bug since upgrading to the latest Ocata in September and I've been trying to figure out the cause for a while now. | 01:18 |
| redcavalier | First I must say we use L3-high availability between 2 network-only nodes | 01:19 |
| *** amodi has joined #openstack | 01:20 | |
| *** catintheroof has quit IRC | 01:20 | |
| *** namnh has joined #openstack | 01:21 | |
| *** fzdarsky_ has joined #openstack | 01:21 | |
| redcavalier | Randomly, for no apparent reasons, when we create a floating ip, the floating ip and iptables rules are created in the namespace, but there's no port created in the namespace to link it to the instance | 01:21 |
| *** catintheroof has joined #openstack | 01:21 | |
| *** VW has quit IRC | 01:22 | |
| *** Son_Goku has joined #openstack | 01:23 | |
| *** catintheroof has quit IRC | 01:24 | |
| *** fzdarsky|afk has quit IRC | 01:24 | |
| *** Gwayne has joined #openstack | 01:31 | |
| *** Gwayne has joined #openstack | 01:31 | |
| *** uZiel has joined #openstack | 01:37 | |
| *** uZiel has quit IRC | 01:42 | |
| *** rcernin has quit IRC | 01:43 | |
| *** yamamoto has joined #openstack | 01:43 | |
| *** blacklotus89 has quit IRC | 01:44 | |
| *** rcernin has joined #openstack | 01:44 | |
| *** bvdk has quit IRC | 01:46 | |
| *** snowman4839 has quit IRC | 01:46 | |
| *** snowman4839 has joined #openstack | 01:46 | |
| *** yamamoto has quit IRC | 01:49 | |
| *** m00dawg has quit IRC | 01:50 | |
| *** Son_Goku has quit IRC | 01:52 | |
| *** blacklotus89 has joined #openstack | 01:53 | |
| *** snowman4839 has quit IRC | 01:53 | |
| *** snowman4839 has joined #openstack | 01:54 | |
| *** catintheroof has joined #openstack | 01:55 | |
| *** salv-orlando has joined #openstack | 01:55 | |
| *** gin0 has joined #openstack | 01:57 | |
| *** VW has joined #openstack | 01:59 | |
| *** salv-orlando has quit IRC | 02:00 | |
| *** catintheroof has quit IRC | 02:00 | |
| *** achadha has quit IRC | 02:00 | |
| *** Son_Goku has joined #openstack | 02:01 | |
| *** achadha has joined #openstack | 02:01 | |
| *** Cybodog has quit IRC | 02:03 | |
| *** achadha has quit IRC | 02:05 | |
| *** dalgaaf has quit IRC | 02:07 | |
| *** dalgaaf has joined #openstack | 02:08 | |
| *** bobh has quit IRC | 02:11 | |
| *** mn3m has quit IRC | 02:14 | |
| *** pryorda has quit IRC | 02:16 | |
| *** gli has joined #openstack | 02:17 | |
| *** cyborg-one has quit IRC | 02:17 | |
| *** thorst has joined #openstack | 02:18 | |
| *** achadha has joined #openstack | 02:18 | |
| *** achadha has quit IRC | 02:18 | |
| *** achadha has joined #openstack | 02:19 | |
| *** thorst has quit IRC | 02:23 | |
| *** achadha_ has joined #openstack | 02:23 | |
| *** mn3m has joined #openstack | 02:23 | |
| *** achadha has quit IRC | 02:24 | |
| *** dparkes has quit IRC | 02:24 | |
| *** Son_Goku has quit IRC | 02:24 | |
| *** catintheroof has joined #openstack | 02:28 | |
| *** yamahata has quit IRC | 02:29 | |
| *** iyamahat has quit IRC | 02:30 | |
| *** jeroen92 has joined #openstack | 02:31 | |
| *** catintheroof has quit IRC | 02:33 | |
| *** zhangfei has joined #openstack | 02:33 | |
| *** jeroen92 has quit IRC | 02:36 | |
| *** dparkes has joined #openstack | 02:36 | |
| *** mn3m has quit IRC | 02:39 | |
| *** fragatina has quit IRC | 02:40 | |
| *** cdub has joined #openstack | 02:43 | |
| *** dparkes has quit IRC | 02:44 | |
| *** cloudrancher has quit IRC | 02:45 | |
| *** dparkes has joined #openstack | 02:45 | |
| *** yamamoto has joined #openstack | 02:45 | |
| *** fragatina has joined #openstack | 02:45 | |
| *** cloudrancher has joined #openstack | 02:46 | |
| *** AlexeyAbashkin has joined #openstack | 02:47 | |
| *** trozet has joined #openstack | 02:47 | |
| *** Son_Goku has joined #openstack | 02:48 | |
| *** fragatina has quit IRC | 02:50 | |
| *** spzala has quit IRC | 02:50 | |
| *** yamamoto has quit IRC | 02:51 | |
| *** AlexeyAbashkin has quit IRC | 02:52 | |
| *** amodi has quit IRC | 02:53 | |
| *** cloudrancher has quit IRC | 02:55 | |
| *** woodard has quit IRC | 02:55 | |
| *** cloudrancher has joined #openstack | 02:56 | |
| *** Son_Goku has quit IRC | 02:59 | |
| *** armaan has joined #openstack | 02:59 | |
| *** snowman48391 has joined #openstack | 03:00 | |
| *** snowman4839 has quit IRC | 03:00 | |
| *** achadha has joined #openstack | 03:01 | |
| *** achadha has quit IRC | 03:01 | |
| *** achadha has joined #openstack | 03:01 | |
| *** achadha has quit IRC | 03:01 | |
| *** achadha_ has quit IRC | 03:01 | |
| *** spzala has joined #openstack | 03:02 | |
| *** spzala has quit IRC | 03:06 | |
| *** dparkes has quit IRC | 03:06 | |
| *** AlexeyAbashkin has joined #openstack | 03:08 | |
| *** gvrangan has joined #openstack | 03:08 | |
| *** trozet has quit IRC | 03:11 | |
| *** AlexeyAbashkin has quit IRC | 03:12 | |
| *** yikun has quit IRC | 03:12 | |
| *** VW has quit IRC | 03:14 | |
| *** yamamoto has joined #openstack | 03:16 | |
| *** knoxy has quit IRC | 03:18 | |
| *** dparkes has joined #openstack | 03:18 | |
| *** Son_Goku has joined #openstack | 03:20 | |
| *** VW has joined #openstack | 03:24 | |
| redcavalier | Out of curiosity, is it the l3 agent that creates a port for a vm in the namespace, or the openvswitch/linuxbridge agent? | 03:24 |
| *** dparkes has quit IRC | 03:24 | |
| *** nicolasbock has quit IRC | 03:30 | |
| *** aselius has quit IRC | 03:30 | |
| *** Calvin` has quit IRC | 03:32 | |
| *** Calvin` has joined #openstack | 03:34 | |
| *** achadha has joined #openstack | 03:35 | |
| *** uZiel has joined #openstack | 03:36 | |
| SamYaple | redcavalier: there are no namespaces on the compute node (without DVR) | 03:36 |
| *** yamamoto has quit IRC | 03:36 | |
| *** imfearless has joined #openstack | 03:37 | |
| SamYaple | redcavalier: the port in the namespace with the l3-controller in the router namespace is for the router and thats iptables so, there is a little dance to get iptables rules applied | 03:38 |
| *** yamahata has joined #openstack | 03:40 | |
| *** charcol-v2 has joined #openstack | 03:41 | |
| *** snowman48391 has quit IRC | 03:41 | |
| *** charcol has quit IRC | 03:41 | |
| *** snowman4839 has joined #openstack | 03:42 | |
| *** bvdk has joined #openstack | 03:42 | |
| *** markvoelker has quit IRC | 03:43 | |
| *** achadha has quit IRC | 03:43 | |
| *** achadha has joined #openstack | 03:43 | |
| *** Son_Goku has quit IRC | 03:43 | |
| *** zhangfei has quit IRC | 03:44 | |
| *** mihdih has quit IRC | 03:47 | |
| *** dparkes has joined #openstack | 03:47 | |
| *** achadha has quit IRC | 03:48 | |
| *** tyrefors has quit IRC | 03:50 | |
| *** VW has quit IRC | 03:52 | |
| *** yamamoto has joined #openstack | 03:54 | |
| *** yamamoto has quit IRC | 03:54 | |
| *** Yarboa has joined #openstack | 03:55 | |
| *** yamamoto has joined #openstack | 03:55 | |
| *** flwang1 has quit IRC | 03:55 | |
| *** woodard has joined #openstack | 03:56 | |
| *** salv-orlando has joined #openstack | 03:56 | |
| redcavalier | SamYaple, sorry, I might not have been clear. I'm talking on a network node in it's namespace. I'm running in an issue where and instance's interface inside the namespace is not created. | 03:57 |
| *** AnilV4 has joined #openstack | 03:57 | |
| *** fragatina has joined #openstack | 03:57 | |
| *** fragatina has quit IRC | 03:58 | |
| *** fragatina has joined #openstack | 03:58 | |
| SamYaple | redcavalier: there is no instance interface on the network node | 03:59 |
| redcavalier | SamYaple, you're right. I'm explaining this poorly | 04:00 |
| *** shintaro has joined #openstack | 04:00 | |
| *** salv-orlando has quit IRC | 04:00 | |
| redcavalier | SamYaple, so, when I add a floating ip to an instance, it doesn't show up in the list when I do ip netns *namespace* ip a. Its iptables rules are there though. | 04:01 |
| *** catintheroof has joined #openstack | 04:01 | |
| SamYaple | redcavalier: so the floating ip wont be a *new* interface, it will be a /32 on the routers interface | 04:02 |
| SamYaple | and you are saying thats not showing up? | 04:02 |
| redcavalier | exactly | 04:02 |
| *** germs1 has quit IRC | 04:02 | |
| SamYaple | legacy router? (no dvr, no ha)? | 04:02 |
| redcavalier | this happens randomly though. Like, it can work for a week, and then suddenly fail | 04:02 |
| redcavalier | HA | 04:02 |
| SamYaple | if its HA, are you sure its not in one of the other network namespaces? (it should onyl live in one) | 04:03 |
| *** spzala has joined #openstack | 04:03 | |
| redcavalier | SamYaple, all floating IPs should be in the same one. If it pops up in the inactive one, there'S still a problem. | 04:03 |
| SamYaple | oh you are saying some are still there? | 04:04 |
| redcavalier | yes, what happens is that we are able to set up floating ips for like 2 or 3 weeks and then, suddenly, new floating ips stop showing up in the namespace | 04:04 |
| SamYaple | so a few questions, when this happens you have other working floating ip address in that router-namespace? does detaching and reattaching the ip help? | 04:05 |
| SamYaple | first question answered | 04:05 |
| redcavalier | Despite iptables being set properly and openstack believing everything is alright | 04:05 |
| redcavalier | detaching and reattaching does not help | 04:05 |
| *** catintheroof has quit IRC | 04:05 | |
| redcavalier | to have it work, we've had to reboot both network nodes | 04:06 |
| SamYaple | certainly sounds like a bug. i would dig into the keepalived logs (assuming no errors in neutron-l3-agent logs) | 04:06 |
| SamYaple | sounds like a keepalived issue | 04:06 |
| redcavalier | alright. To be honest, it'S been happening since we updated Ocata at the end of september | 04:06 |
| SamYaple | in l3-ha i belive keepalived handles all of that | 04:06 |
| SamYaple | so when this happens next time, try killing all the keepalived processes and restarting the l3 agents | 04:07 |
| redcavalier | in what file are the keepalived logs located by default? | 04:07 |
| redcavalier | ok | 04:07 |
| *** spzala has quit IRC | 04:07 | |
| SamYaple | hmmm good question | 04:07 |
| *** woodard has quit IRC | 04:07 | |
| SamYaple | i belive /var/log/neutron/* | 04:07 |
| SamYaple | but im not sure they are logged by default | 04:07 |
| SamYaple | i dont use l3-ha and havent in a while | 04:08 |
| redcavalier | I don't remember seeing a keepalived log file | 04:08 |
| SamYaple | ps aux | grep keepalived | 04:08 |
| SamYaple | that might tell you where its logging | 04:08 |
| SamYaple | i cant remember if keepalived can log to a file directly, or requires syslog | 04:08 |
| SamYaple | if its syslog, then you should have that somewhere | 04:08 |
| SamYaple | redcavalier: when moving to ocata, did keepalived version change? | 04:09 |
| redcavalier | SamYaple, good question. let me check the yum logs | 04:09 |
| *** jdwidari has quit IRC | 04:10 | |
| redcavalier | also, the process doesn'T indicate any logs | 04:10 |
| SamYaple | in anycase, if killing keepalived and restarting l3-agents (to regen keeplived process) fixes it, i think its safe to say keepalived is having issues | 04:10 |
| SamYaple | at that point you would just need to track down *why* | 04:10 |
| *** rideh has quit IRC | 04:10 | |
| SamYaple | but all openstack can do is inform keepalived of the update, it assumes keepalived is oging to handle it form there | 04:11 |
| redcavalier | I see | 04:11 |
| redcavalier | doesn'T openstack also generate the keepalived configuration? | 04:12 |
| *** rideh has joined #openstack | 04:12 | |
| SamYaple | yes, but i doubt the keepalived config is broken | 04:13 |
| SamYaple | its probably a nohup thats not working or something | 04:13 |
| SamYaple | you can verify that now that you have a direction to look | 04:13 |
| redcavalier | ok. and yes, there was indeed an update of keepalived | 04:13 |
| SamYaple | so its eitehr a new bug in keepalived, or a bug in neutron configuring the new keepalived | 04:13 |
| *** davidbowlby-afk has quit IRC | 04:13 | |
| SamYaple | possibly it needs a new option or something | 04:14 |
| SamYaple | more debuging and info is required | 04:14 |
| *** vks1 has joined #openstack | 04:14 | |
| redcavalier | yea, I'll see what I can find. Is it documented how to have keepalived log somewhere when its config is managed by openstack? | 04:15 |
| *** ldl has joined #openstack | 04:15 | |
| ldl | Hi， friends, who can help me with this? | 04:15 |
| ldl | https://ask.openstack.org/en/question/110857/after-use-admin-change-the-normal-users-password-i-can-not-use-the-new-password-to-login/ | 04:15 |
| *** fragatina has quit IRC | 04:16 | |
| SamYaple | redcavalier: its keepalived, so the config and cli is the only thing that its going to read. neutron might havea way to append args or something | 04:16 |
| SamYaple | redcavalier: but probably not documented well if it exists | 04:16 |
| SamYaple | the core configuration would not be configurable given neutrons history | 04:16 |
| SamYaple | redcavalier: source code doesnt lie, if its an option, itll be in github.com/openstack/neutron/neutron/conf | 04:17 |
| redcavalier | SamYaple, so basically, if it'S not there, there's not much I can do beside coding it myself | 04:18 |
| *** dparkes has quit IRC | 04:18 | |
| *** imfearless has quit IRC | 04:18 | |
| SamYaple | redcavalier: lol lets identify the problem before we start talking about the limitations of neutron :P | 04:18 |
| *** thorst has joined #openstack | 04:19 | |
| redcavalier | hey, I'm just trying to give myself log to work from :p | 04:19 |
| SamYaple | you probably dont need to change anything, its possibly a bug in neutron, its probably an issue with keepalived | 04:19 |
| SamYaple | redcavalier: you could always *downgrade* keepalived, run with that for a bit | 04:19 |
| *** gkadam has joined #openstack | 04:19 | |
| SamYaple | but to get any type of resolution on this you need to be able to reproduce it and identify the cause | 04:20 |
| *** dbecker has quit IRC | 04:20 | |
| SamYaple | reproducing it should help with that | 04:20 |
| redcavalier | SamYaple, yea, I'll try to convince my boss with downgrading. | 04:20 |
| SamYaple | (wait for it to happen again and then try killing keepalived first) | 04:20 |
| SamYaple | for all we know thats not going to fix it | 04:21 |
| *** pleia2 has quit IRC | 04:21 | |
| *** zymurgy has quit IRC | 04:21 | |
| redcavalier | Alright | 04:21 |
| *** pleia2 has joined #openstack | 04:22 | |
| *** thorst has quit IRC | 04:24 | |
| *** gildub has quit IRC | 04:24 | |
| *** Guest has joined #openstack | 04:25 | |
| *** gin0 has quit IRC | 04:26 | |
| Guest | I have installed openstack using devstack , but when I try to use the services it gives me "Unable to determine the Keystone version to authenticate with using the given auth_url." | 04:26 |
| *** masco has joined #openstack | 04:27 | |
| *** gin0 has joined #openstack | 04:27 | |
| *** rmcallis has quit IRC | 04:27 | |
| Guest | and when I determine the version on OS_AUTH_URL it now shows me "Unable to establish connection to https://104.239.150.29:5000/v3/auth/tokens: HTTPSConnectionPool(host='104.239.150.29', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53dc9ed250>: Failed to establish a new connection: " | 04:28 |
| *** dparkes has joined #openstack | 04:30 | |
| *** jeroen92 has joined #openstack | 04:32 | |
| *** ldl has quit IRC | 04:32 | |
| *** janki has joined #openstack | 04:36 | |
| *** dbecker has joined #openstack | 04:36 | |
| *** jeroen92 has quit IRC | 04:37 | |
| *** sree has joined #openstack | 04:37 | |
| *** logan- has quit IRC | 04:38 | |
| *** rj291196 has joined #openstack | 04:39 | |
| SamYaple | Guest: looks like you are having trouble talkign to keystone | 04:40 |
| SamYaple | Guest: keystone is either not running, or there is a firewall issue of some kind | 04:40 |
| *** gildub has joined #openstack | 04:40 | |
| *** ivveh- is now known as ivveh | 04:41 | |
| *** logan- has joined #openstack | 04:42 | |
| Guest | SamYaple: or maybe having problems finding my auth_url as I manually set it and also I am unable to find keystonerc_admin file but my ./stack.sh successfully completed, don't know whats wrong | 04:42 |
| SamYaple | Guest: no, the message is pretty clear and not openstack specfic at all. you cannot connect to port 5000 at 104.239.150.29 | 04:44 |
| SamYaple | its because nothing is running there, or there is a firewall issue | 04:44 |
| Guest | SamYample: alright do you know how do we find our keystone auth url if we wanted to ? | 04:46 |
| SamYaple | Guest: figure out where keystone is running, at what ip and what port, then use that | 04:46 |
| Guest | SamYample: alright, tahnsk | 04:47 |
| *** rmk has quit IRC | 04:50 | |
| *** rideh has quit IRC | 04:51 | |
| *** _Cyclone_ has quit IRC | 04:51 | |
| *** xar- has quit IRC | 04:51 | |
| *** pleia2 has quit IRC | 04:51 | |
| *** Graypup_ has quit IRC | 04:51 | |
| *** kuromagi has quit IRC | 04:51 | |
| *** cburgess has quit IRC | 04:51 | |
| *** suramya_ has joined #openstack | 04:51 | |
| *** achadha has joined #openstack | 04:51 | |
| *** achadha has quit IRC | 04:51 | |
| *** gvrangan has quit IRC | 04:52 | |
| *** achadha has joined #openstack | 04:52 | |
| *** gvrangan has joined #openstack | 04:52 | |
| *** rmcallis has joined #openstack | 04:53 | |
| *** Suramya has joined #openstack | 04:53 | |
| *** dparkes has quit IRC | 04:54 | |
| *** rmcallis has quit IRC | 04:58 | |
| *** rmcallis has joined #openstack | 04:59 | |
| *** mihdih has joined #openstack | 05:00 | |
| *** dparkes has joined #openstack | 05:00 | |
| *** spzala has joined #openstack | 05:02 | |
| *** smccarth_ has joined #openstack | 05:02 | |
| *** Graypup_ has joined #openstack | 05:03 | |
| *** _Cyclone_ has joined #openstack | 05:03 | |
| *** cburgess has joined #openstack | 05:03 | |
| *** vvargaszte has joined #openstack | 05:03 | |
| *** rideh has joined #openstack | 05:04 | |
| *** woodard has joined #openstack | 05:04 | |
| *** dparkes has quit IRC | 05:05 | |
| *** rmk has joined #openstack | 05:05 | |
| *** rmk has quit IRC | 05:05 | |
| *** rmk has joined #openstack | 05:05 | |
| *** pleia2 has joined #openstack | 05:05 | |
| *** kuromagi has joined #openstack | 05:05 | |
| *** zymurgy has joined #openstack | 05:06 | |
| *** dparkes has joined #openstack | 05:06 | |
| *** smccarthy has quit IRC | 05:06 | |
| *** spzala has quit IRC | 05:06 | |
| *** xar- has joined #openstack | 05:10 | |
| *** ChubYann has quit IRC | 05:10 | |
| *** slaweq has joined #openstack | 05:11 | |
| *** macermak has joined #openstack | 05:12 | |
| *** vvargaszte has quit IRC | 05:14 | |
| *** masuberu has joined #openstack | 05:15 | |
| *** woodard has quit IRC | 05:16 | |
| *** slaweq has quit IRC | 05:16 | |
| *** dparkes has quit IRC | 05:17 | |
| *** masber has quit IRC | 05:18 | |
| *** masber has joined #openstack | 05:20 | |
| *** LocaMocha has joined #openstack | 05:21 | |
| *** yamamoto has quit IRC | 05:21 | |
| *** masuberu has quit IRC | 05:23 | |
| *** flwang1 has joined #openstack | 05:24 | |
| *** salv-orlando has joined #openstack | 05:25 | |
| *** martinus__ has joined #openstack | 05:25 | |
| *** omie888777 has quit IRC | 05:26 | |
| *** achadha has quit IRC | 05:29 | |
| *** achadha has joined #openstack | 05:29 | |
| *** flwang1 has quit IRC | 05:30 | |
| *** newmember has joined #openstack | 05:31 | |
| *** atmark has joined #openstack | 05:31 | |
| *** LocaMocha has quit IRC | 05:33 | |
| *** achadha has quit IRC | 05:33 | |
| *** flwang1 has joined #openstack | 05:33 | |
| *** LocaMocha has joined #openstack | 05:34 | |
| *** lunarlamp has quit IRC | 05:35 | |
| *** suuuper has joined #openstack | 05:36 | |
| *** salv-orlando has quit IRC | 05:38 | |
| *** salv-orlando has joined #openstack | 05:38 | |
| *** iyamahat has joined #openstack | 05:39 | |
| *** ogny has joined #openstack | 05:40 | |
| *** markvoelker has joined #openstack | 05:43 | |
| *** gin0 has quit IRC | 05:45 | |
| *** BenderRodriguez has quit IRC | 05:47 | |
| *** Guest has quit IRC | 05:47 | |
| *** cshastri has joined #openstack | 05:51 | |
| *** jparrill has joined #openstack | 05:51 | |
| *** iyamahat has quit IRC | 05:52 | |
| *** jprovazn has joined #openstack | 05:53 | |
| *** ldl has joined #openstack | 05:55 | |
| *** iranzo has joined #openstack | 05:56 | |
| *** flwang1 has quit IRC | 05:57 | |
| *** agurenko has quit IRC | 05:58 | |
| *** mosulica has joined #openstack | 05:59 | |
| *** Son_Goku has joined #openstack | 06:00 | |
| *** spzala has joined #openstack | 06:03 | |
| *** omie888777 has joined #openstack | 06:04 | |
| *** ldl has quit IRC | 06:04 | |
| *** cah_link has joined #openstack | 06:04 | |
| *** smccarth_ has quit IRC | 06:06 | |
| *** spzala has quit IRC | 06:07 | |
| *** jbadiapa has joined #openstack | 06:10 | |
| *** egonzalez has joined #openstack | 06:12 | |
| *** woodard has joined #openstack | 06:13 | |
| *** cah_link has quit IRC | 06:13 | |
| *** Son_Goku has quit IRC | 06:14 | |
| *** ldl has joined #openstack | 06:14 | |
| *** grafuls has quit IRC | 06:17 | |
| *** markvoelker has quit IRC | 06:18 | |
| *** gildub has quit IRC | 06:19 | |
| *** thorst has joined #openstack | 06:20 | |
| *** spectr has joined #openstack | 06:22 | |
| *** zhangfei has joined #openstack | 06:23 | |
| *** bvdk has quit IRC | 06:23 | |
| *** woodard has quit IRC | 06:24 | |
| *** thorst has quit IRC | 06:25 | |
| *** dparkes has joined #openstack | 06:26 | |
| *** gvrangan has quit IRC | 06:26 | |
| *** Gwayne has quit IRC | 06:31 | |
| *** kei-ichi has quit IRC | 06:31 | |
| *** kei-ichi has joined #openstack | 06:32 | |
| *** lunarlamp has joined #openstack | 06:32 | |
| *** aojea has joined #openstack | 06:32 | |
| *** gvrangan has joined #openstack | 06:32 | |
| *** jeroen92 has joined #openstack | 06:34 | |
| *** bvdk has joined #openstack | 06:37 | |
| *** jeroen92 has quit IRC | 06:38 | |
| *** robbiew has quit IRC | 06:39 | |
| *** kei-ichi has quit IRC | 06:40 | |
| *** kei-ichi has joined #openstack | 06:41 | |
| *** hseipp has joined #openstack | 06:42 | |
| *** gvrangan has quit IRC | 06:44 | |
| *** rj291196 has quit IRC | 06:48 | |
| *** Son_Goku has joined #openstack | 06:48 | |
| *** kickinz1 has joined #openstack | 06:56 | |
| *** aojea has quit IRC | 06:58 | |
| *** pblaho has joined #openstack | 07:00 | |
| *** suramya_ has quit IRC | 07:01 | |
| *** kei-ichi has quit IRC | 07:03 | |
| *** aojea has joined #openstack | 07:03 | |
| *** takamatsu has joined #openstack | 07:03 | |
| *** kei-ichi has joined #openstack | 07:03 | |
| *** armaan has quit IRC | 07:04 | |
| *** grafuls has joined #openstack | 07:04 | |
| *** armaan has joined #openstack | 07:05 | |
| *** rbowen has joined #openstack | 07:05 | |
| *** kickinz1 has quit IRC | 07:05 | |
| *** kickinz1 has joined #openstack | 07:06 | |
| *** rossella_s has joined #openstack | 07:06 | |
| *** mrch has joined #openstack | 07:06 | |
| *** robbiew has joined #openstack | 07:07 | |
| *** Zophar has quit IRC | 07:07 | |
| *** aojea has quit IRC | 07:08 | |
| *** KickInZOne has joined #openstack | 07:11 | |
| *** rikmeg has joined #openstack | 07:12 | |
| *** slaweq has joined #openstack | 07:12 | |
| *** kickinz1 has quit IRC | 07:13 | |
| *** vvargaszte has joined #openstack | 07:13 | |
| *** vvargaszte has quit IRC | 07:14 | |
| *** markvoelker has joined #openstack | 07:14 | |
| *** armaan has quit IRC | 07:15 | |
| *** armaan has joined #openstack | 07:15 | |
| *** Son_Goku has quit IRC | 07:16 | |
| *** slaweq has quit IRC | 07:16 | |
| *** Tecumseh has joined #openstack | 07:19 | |
| *** woodard has joined #openstack | 07:20 | |
| *** jackNemrod has joined #openstack | 07:20 | |
| *** tesseract has joined #openstack | 07:22 | |
| *** slaweq has joined #openstack | 07:26 | |
| *** lucendio has joined #openstack | 07:26 | |
| *** anshul has joined #openstack | 07:29 | |
| *** woodard has quit IRC | 07:30 | |
| *** salv-orlando has quit IRC | 07:30 | |
| *** salv-orlando has joined #openstack | 07:31 | |
| *** ffiore has joined #openstack | 07:31 | |
| *** tssurya has quit IRC | 07:31 | |
| *** tssurya has joined #openstack | 07:32 | |
| *** lucendio has quit IRC | 07:33 | |
| *** thorst has joined #openstack | 07:33 | |
| *** charcol-v2 has quit IRC | 07:34 | |
| *** namnh has quit IRC | 07:34 | |
| *** salv-orlando has quit IRC | 07:35 | |
| *** rbowen has quit IRC | 07:36 | |
| *** pblaho has quit IRC | 07:37 | |
| *** flwang1 has joined #openstack | 07:37 | |
| *** rmart04 has joined #openstack | 07:38 | |
| *** fsimonce has joined #openstack | 07:38 | |
| *** rbowen has joined #openstack | 07:38 | |
| *** thorst has quit IRC | 07:38 | |
| *** josecastroleon has joined #openstack | 07:41 | |
| *** jpsandiego42 has quit IRC | 07:41 | |
| *** jpsandiego42 has joined #openstack | 07:41 | |
| *** ttrifonov has quit IRC | 07:42 | |
| *** pblaho has joined #openstack | 07:42 | |
| *** geaaru has joined #openstack | 07:43 | |
| *** e0ne has joined #openstack | 07:45 | |
| *** ttrifonov has joined #openstack | 07:46 | |
| *** markvoelker has quit IRC | 07:48 | |
| *** flwang1 has quit IRC | 07:49 | |
| *** toanju has joined #openstack | 07:49 | |
| *** ralonsoh has joined #openstack | 07:50 | |
| *** salv-orlando has joined #openstack | 07:52 | |
| *** rbowen has quit IRC | 07:53 | |
| *** AlexeyAbashkin has joined #openstack | 07:56 | |
| *** makowals has joined #openstack | 07:58 | |
| *** rcernin has quit IRC | 07:58 | |
| *** rbowen has joined #openstack | 07:58 | |
| *** MrAbaddon has quit IRC | 08:01 | |
| *** jackNemrod has quit IRC | 08:01 | |
| *** jackNemrod has joined #openstack | 08:03 | |
| *** itxaka has joined #openstack | 08:03 | |
| *** itxaka has quit IRC | 08:03 | |
| *** ivveh has quit IRC | 08:04 | |
| *** slaweq has quit IRC | 08:04 | |
| *** slaweq has joined #openstack | 08:05 | |
| *** ivveh has joined #openstack | 08:05 | |
| *** jackNemrod_ has joined #openstack | 08:07 | |
| *** jackNemrod has quit IRC | 08:07 | |
| *** jackNemrod_ is now known as jackNemrod | 08:07 | |
| *** cnf has quit IRC | 08:07 | |
| *** cnf has joined #openstack | 08:11 | |
| *** OutBackDingo has quit IRC | 08:13 | |
| *** yamahata has quit IRC | 08:14 | |
| *** pbandark has joined #openstack | 08:15 | |
| *** zhangfei has quit IRC | 08:15 | |
| *** armaan has quit IRC | 08:18 | |
| *** armaan has joined #openstack | 08:19 | |
| *** OutBackDingo has joined #openstack | 08:23 | |
| *** woodard has joined #openstack | 08:27 | |
| *** rsimai_away is now known as rsimai | 08:29 | |
| *** mhickey has joined #openstack | 08:30 | |
| *** iranzo has quit IRC | 08:36 | |
| *** drbroiler has joined #openstack | 08:37 | |
| *** iranzo has joined #openstack | 08:37 | |
| *** iranzo has quit IRC | 08:37 | |
| *** iranzo has joined #openstack | 08:37 | |
| *** dasTor has joined #openstack | 08:37 | |
| *** gszasz has joined #openstack | 08:38 | |
| *** woodard has quit IRC | 08:38 | |
| *** nshetty is now known as nshetty|lunch | 08:40 | |
| *** nshetty|lunch is now known as nshetty | 08:40 | |
| *** pravins has joined #openstack | 08:41 | |
| *** fzdarsky_ is now known as fzdarsky | 08:43 | |
| *** funabashi has quit IRC | 08:44 | |
| *** rmcallis has quit IRC | 08:44 | |
| *** markvoelker has joined #openstack | 08:45 | |
| *** rmcallis has joined #openstack | 08:46 | |
| *** electrofelix has joined #openstack | 08:48 | |
| *** bbcmicrocomputer has joined #openstack | 08:49 | |
| *** bbcmicrocomputer has joined #openstack | 08:49 | |
| *** rvd has joined #openstack | 08:49 | |
| *** rvd is now known as dave|gone | 08:49 | |
| *** dave|gone has quit IRC | 08:50 | |
| *** namnh has joined #openstack | 08:51 | |
| *** pblaho has quit IRC | 08:51 | |
| *** MarkBaker has quit IRC | 08:53 | |
| *** rmcallis has quit IRC | 08:53 | |
| *** dizquierdo has joined #openstack | 08:54 | |
| *** racedo has joined #openstack | 08:54 | |
| *** MrAbaddon has joined #openstack | 08:56 | |
| *** darkalia has quit IRC | 08:58 | |
| *** bvdk has quit IRC | 08:59 | |
| *** forgotmynick has joined #openstack | 09:00 | |
| *** salmankhan has joined #openstack | 09:02 | |
| *** darkalia has joined #openstack | 09:03 | |
| *** darkalia has quit IRC | 09:03 | |
| *** darkalia has joined #openstack | 09:04 | |
| *** spzala has joined #openstack | 09:04 | |
| *** ivveh has quit IRC | 09:05 | |
| *** jeroen92 has joined #openstack | 09:07 | |
| *** ginsul has joined #openstack | 09:08 | |
| *** spzala has quit IRC | 09:09 | |
| *** funabashi has joined #openstack | 09:10 | |
| *** jparrill has quit IRC | 09:10 | |
| *** mguiney has quit IRC | 09:11 | |
| *** jeroen92 has quit IRC | 09:12 | |
| *** ffiore has quit IRC | 09:12 | |
| *** ffiore_ has joined #openstack | 09:12 | |
| *** masco has quit IRC | 09:15 | |
| *** slaweq has quit IRC | 09:16 | |
| *** slaweq has joined #openstack | 09:17 | |
| *** markvoelker has quit IRC | 09:18 | |
| *** ffiore has joined #openstack | 09:20 | |
| *** ffiore_ has quit IRC | 09:20 | |
| *** slaweq has quit IRC | 09:22 | |
| *** slaweq has joined #openstack | 09:22 | |
| *** mguiney has joined #openstack | 09:24 | |
| *** bvdk has joined #openstack | 09:27 | |
| *** schlitzer has joined #openstack | 09:29 | |
| *** bvdk has quit IRC | 09:30 | |
| *** MarkBaker has joined #openstack | 09:31 | |
| *** toscalix has joined #openstack | 09:31 | |
| *** armaan has quit IRC | 09:32 | |
| *** armaan has joined #openstack | 09:32 | |
| *** v3nd3tta is now known as v3nd3tta`` | 09:33 | |
| *** thorst has joined #openstack | 09:34 | |
| *** grafuls has quit IRC | 09:35 | |
| *** woodard has joined #openstack | 09:36 | |
| *** ivveh has joined #openstack | 09:36 | |
| *** cnf has quit IRC | 09:37 | |
| *** cah_link has joined #openstack | 09:37 | |
| *** rmart04 has quit IRC | 09:37 | |
| *** v3nd3tta`` is now known as v3nd3tta | 09:39 | |
| *** thorst has quit IRC | 09:39 | |
| *** links has quit IRC | 09:42 | |
| *** woodard has quit IRC | 09:46 | |
| *** flwang1 has joined #openstack | 09:46 | |
| *** fragatina has joined #openstack | 09:48 | |
| *** v3nd3tta is now known as v3nd3tta`` | 09:48 | |
| *** MarkBaker_ has joined #openstack | 09:49 | |
| *** Suramya has quit IRC | 09:50 | |
| *** MarkBaker has quit IRC | 09:52 | |
| *** redcavalier has quit IRC | 09:52 | |
| *** maciejjozefczyk has joined #openstack | 09:52 | |
| *** slaweq_ has joined #openstack | 09:52 | |
| *** SmearedBeard has joined #openstack | 09:53 | |
| *** dizquierdo is now known as dizquierdo_afk | 09:54 | |
| *** slaweq has quit IRC | 09:54 | |
| *** maciejjo1 has quit IRC | 09:55 | |
| *** dizquierdo_afk is now known as dizquierdo | 09:55 | |
| *** flwang1 has quit IRC | 09:55 | |
| *** v3nd3tta`` is now known as v3nd3tta | 09:58 | |
| *** armaan has quit IRC | 09:58 | |
| *** v3nd3tta is now known as v3nd3tta`` | 10:00 | |
| *** nshetty has quit IRC | 10:00 | |
| *** v3nd3tta`` is now known as v3nd3tta | 10:06 | |
| *** v3nd3tta is now known as v3nd3tta`` | 10:07 | |
| *** Son_Goku has joined #openstack | 10:09 | |
| *** jeroen92 has joined #openstack | 10:09 | |
| hughsaunders | Hey, I have a deploy with ironic and kvm hypervisors, tempest test minimum basic is failing when its instance is scheduled to an ironic node rather than a kvm hypervisor. How can I ensure these instances are not scheduled to ironic nodes? | 10:09 |
| *** mosulica has quit IRC | 10:10 | |
| *** eckesicle has quit IRC | 10:12 | |
| *** mn3m has joined #openstack | 10:14 | |
| *** markvoelker has joined #openstack | 10:15 | |
| *** mhickey has quit IRC | 10:17 | |
| *** MarkBaker_ has quit IRC | 10:19 | |
| *** jaicaa has quit IRC | 10:23 | |
| *** mosulica has joined #openstack | 10:24 | |
| *** mn3m has quit IRC | 10:24 | |
| *** yamamoto has joined #openstack | 10:24 | |
| *** mn3m has joined #openstack | 10:24 | |
| *** blacklotus89 has quit IRC | 10:25 | |
| *** beerdrop has quit IRC | 10:27 | |
| *** Son_Goku has quit IRC | 10:28 | |
| *** mn3m_ has joined #openstack | 10:29 | |
| *** mn3m_ has quit IRC | 10:31 | |
| *** armaan has joined #openstack | 10:31 | |
| *** mn3m has quit IRC | 10:31 | |
| *** gli has quit IRC | 10:31 | |
| *** cnf has joined #openstack | 10:32 | |
| *** gkadam has quit IRC | 10:34 | |
| *** blacklotus89 has joined #openstack | 10:35 | |
| *** namnh has quit IRC | 10:36 | |
| *** mn3m_ has joined #openstack | 10:38 | |
| *** mn3m_ is now known as mn3m | 10:39 | |
| *** dizquierdo is now known as dizquierdo_afk | 10:41 | |
| *** beerdrop has joined #openstack | 10:41 | |
| *** armaan has quit IRC | 10:43 | |
| *** ArminderSingh has quit IRC | 10:43 | |
| *** woodard has joined #openstack | 10:43 | |
| *** pblaho has joined #openstack | 10:44 | |
| *** yamamoto has quit IRC | 10:45 | |
| *** gkadam has joined #openstack | 10:47 | |
| *** markvoelker has quit IRC | 10:49 | |
| *** rmcallis has joined #openstack | 10:49 | |
| *** rossella_s has quit IRC | 10:51 | |
| *** jprovazn has quit IRC | 10:53 | |
| *** metachr0n has joined #openstack | 10:53 | |
| *** woodard has quit IRC | 10:53 | |
| *** rmcallis has quit IRC | 10:54 | |
| *** lucendio has joined #openstack | 10:59 | |
| lvdombrkr | folks trying to find any notes/documentation how to upgrade from Kilo to Liberty, tried look in offcial documentation, and google but without success | 10:59 |
| *** jeroen92 has quit IRC | 10:59 | |
| *** jeroen92 has joined #openstack | 10:59 | |
| lvdombrkr | maybe someone have ideas? | 10:59 |
| *** _sid has joined #openstack | 11:01 | |
| *** ArminderSingh has joined #openstack | 11:02 | |
| *** robcresswell has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack | 11:05 | |
| *** _sid has quit IRC | 11:06 | |
| *** _sid has joined #openstack | 11:06 | |
| *** spzala has joined #openstack | 11:06 | |
| *** Acinonyx has quit IRC | 11:11 | |
| *** jparrill has joined #openstack | 11:12 | |
| *** spzala has quit IRC | 11:12 | |
| *** Acinonyx has joined #openstack | 11:13 | |
| *** shintaro has quit IRC | 11:14 | |
| *** dparkes has quit IRC | 11:16 | |
| *** Son_Goku has joined #openstack | 11:16 | |
| *** MarkBaker_ has joined #openstack | 11:16 | |
| *** rbowen has quit IRC | 11:17 | |
| *** dyasny has quit IRC | 11:18 | |
| *** yamamoto has quit IRC | 11:21 | |
| *** nwonknu has quit IRC | 11:22 | |
| *** r0kc4t has quit IRC | 11:23 | |
| *** dizquierdo_afk is now known as dizquierdo | 11:23 | |
| *** r0kc4t has joined #openstack | 11:24 | |
| *** salmankhan has quit IRC | 11:25 | |
| *** sep_ is now known as sep | 11:26 | |
| hughsaunders | lvdombrkr: Largely depends how you deployed, openstack-ansible have some notes on upgrading from kilo to liberty at https://github.com/openstack/openstack-ansible/blob/liberty-eol/doc/source/upgrade-guide/overview.rst | 11:26 |
| *** bvdk has joined #openstack | 11:26 | |
| *** jeroen92 has quit IRC | 11:28 | |
| *** ldl has quit IRC | 11:29 | |
| *** donghao has joined #openstack | 11:30 | |
| *** forgotmynick has quit IRC | 11:30 | |
| *** Son_Goku has quit IRC | 11:30 | |
| *** donghao has quit IRC | 11:31 | |
| *** donghao has joined #openstack | 11:32 | |
| *** _sid has quit IRC | 11:32 | |
| *** baoli has joined #openstack | 11:34 | |
| *** ffiore has quit IRC | 11:34 | |
| *** donghao has quit IRC | 11:35 | |
| *** thorst has joined #openstack | 11:35 | |
| *** ldl has joined #openstack | 11:36 | |
| *** nwonknu has joined #openstack | 11:36 | |
| *** donghao has joined #openstack | 11:37 | |
| *** GenteelBen has joined #openstack | 11:37 | |
| *** baoli has quit IRC | 11:37 | |
| *** tlaxkit has joined #openstack | 11:38 | |
| *** baoli has joined #openstack | 11:38 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:38 | |
| *** grafuls has joined #openstack | 11:38 | |
| *** omie888777 has quit IRC | 11:38 | |
| *** thorst has quit IRC | 11:40 | |
| *** omie888777 has joined #openstack | 11:40 | |
| *** omie88877777 has joined #openstack | 11:43 | |
| *** electrofelix has quit IRC | 11:43 | |
| *** electrofelix has joined #openstack | 11:43 | |
| *** nicolasbock has joined #openstack | 11:43 | |
| *** gringao has quit IRC | 11:44 | |
| *** grafuls has quit IRC | 11:45 | |
| *** belmoreira has joined #openstack | 11:45 | |
| *** thorst has joined #openstack | 11:46 | |
| *** markvoelker has joined #openstack | 11:46 | |
| *** cnf has quit IRC | 11:47 | |
| *** Son_Goku has joined #openstack | 11:47 | |
| *** cnf has joined #openstack | 11:48 | |
| *** v3nd3tta`` is now known as v3nd3tta | 11:49 | |
| *** yamamoto has joined #openstack | 11:50 | |
| *** woodard has joined #openstack | 11:50 | |
| *** atmark has quit IRC | 11:50 | |
| *** ikhan_ has joined #openstack | 11:51 | |
| *** mosulica has quit IRC | 11:51 | |
| *** rbowen has joined #openstack | 11:53 | |
| *** smccarthy has joined #openstack | 11:54 | |
| *** dparkes has joined #openstack | 11:55 | |
| *** dizquierdo_afk is now known as dizquierdo | 11:57 | |
| *** gringao has joined #openstack | 11:57 | |
| *** smccarthy has quit IRC | 11:59 | |
| *** jeroen92 has joined #openstack | 11:59 | |
| *** yamamoto has quit IRC | 11:59 | |
| *** Son_Goku has quit IRC | 11:59 | |
| *** toscalix has quit IRC | 12:00 | |
| *** _sid has joined #openstack | 12:00 | |
| *** grafuls has joined #openstack | 12:00 | |
| *** woodard has quit IRC | 12:00 | |
| *** gringao has quit IRC | 12:02 | |
| *** jeroen92 has quit IRC | 12:03 | |
| *** ivveh has quit IRC | 12:04 | |
| *** tnogisto has joined #openstack | 12:04 | |
| *** liverpooler has joined #openstack | 12:05 | |
| *** lahari has joined #openstack | 12:08 | |
| tnogisto | hi, if I want to put compute nodes in multiple different physical locations, would it be better to set the region_name variable or availability-zone variable? | 12:08 |
| *** lahari has quit IRC | 12:08 | |
| *** spzala has joined #openstack | 12:08 | |
| *** lahari has joined #openstack | 12:08 | |
| *** vks1 has quit IRC | 12:11 | |
| *** _sid has quit IRC | 12:12 | |
| *** drbroiler has quit IRC | 12:12 | |
| *** spzala has quit IRC | 12:13 | |
| *** salmankhan has joined #openstack | 12:13 | |
| *** ffiore has joined #openstack | 12:13 | |
| *** yamamoto has joined #openstack | 12:14 | |
| *** blacklotus89 has quit IRC | 12:15 | |
| *** gringao has joined #openstack | 12:16 | |
| *** gaowen has quit IRC | 12:16 | |
| *** gaowen has joined #openstack | 12:17 | |
| *** salmankhan has quit IRC | 12:17 | |
| *** imfearless has joined #openstack | 12:18 | |
| *** markvoelker has quit IRC | 12:20 | |
| *** jeroen92 has joined #openstack | 12:20 | |
| *** morazi has joined #openstack | 12:24 | |
| *** blacklotus89 has joined #openstack | 12:25 | |
| *** jeroen92 has quit IRC | 12:25 | |
| *** lahari has quit IRC | 12:26 | |
| *** lahari has joined #openstack | 12:27 | |
| *** markvoelker has joined #openstack | 12:27 | |
| *** lahari has quit IRC | 12:27 | |
| *** davidbowlby has joined #openstack | 12:29 | |
| *** StevenK has quit IRC | 12:29 | |
| *** bl4ckb1rd has quit IRC | 12:29 | |
| *** classicsnail has quit IRC | 12:29 | |
| *** freyes has quit IRC | 12:29 | |
| *** v3nd3tta is now known as v3nd3tta`` | 12:29 | |
| *** classicsnail has joined #openstack | 12:29 | |
| *** bl4ckb1rd has joined #openstack | 12:30 | |
| *** freyes__ has joined #openstack | 12:30 | |
| *** shoogz has quit IRC | 12:32 | |
| *** baoli has quit IRC | 12:32 | |
| *** Son_Goku has joined #openstack | 12:32 | |
| *** shoogz has joined #openstack | 12:33 | |
| *** shoogz has joined #openstack | 12:33 | |
| *** uZiel has quit IRC | 12:35 | |
| *** VW has joined #openstack | 12:36 | |
| *** VW has quit IRC | 12:36 | |
| *** VW has joined #openstack | 12:37 | |
| *** rikmeg has quit IRC | 12:37 | |
| *** gkadam has quit IRC | 12:37 | |
| *** baoli has joined #openstack | 12:37 | |
| *** armaan has joined #openstack | 12:37 | |
| *** jeroen92 has joined #openstack | 12:38 | |
| *** snuffkin has quit IRC | 12:38 | |
| *** freyes__ is now known as freyes | 12:40 | |
| *** freyes has quit IRC | 12:40 | |
| *** freyes has joined #openstack | 12:40 | |
| *** StevenK has joined #openstack | 12:41 | |
| *** Son_Goku has quit IRC | 12:42 | |
| *** jeroen92 has quit IRC | 12:42 | |
| *** jcarter2 has joined #openstack | 12:42 | |
| *** AnilV4 has quit IRC | 12:43 | |
| *** snuffkin has joined #openstack | 12:45 | |
| *** jsheeren has joined #openstack | 12:45 | |
| *** knoxy has joined #openstack | 12:47 | |
| *** knoxy has joined #openstack | 12:47 | |
| *** jeroen92 has joined #openstack | 12:48 | |
| *** pchavva has joined #openstack | 12:50 | |
| *** spzala has joined #openstack | 12:50 | |
| *** jeroen92_ has joined #openstack | 12:53 | |
| *** rand0m has joined #openstack | 12:53 | |
| *** jeroen92 has quit IRC | 12:54 | |
| rand0m | Hello y'all | 12:54 |
| rand0m | I have a question regarding keystone: Is keystone intended to not automatically add systemd unit files during its installation? | 12:55 |
| rand0m | I have installed keystone on ubuntu server 16.04 which does not let me manage keystone using the systemctl or service command | 12:56 |
| *** grafuls has quit IRC | 12:56 | |
| *** VW has quit IRC | 12:57 | |
| *** woodard has joined #openstack | 12:58 | |
| *** marst has joined #openstack | 13:01 | |
| *** pblaho has quit IRC | 13:01 | |
| *** egonzalez has quit IRC | 13:01 | |
| *** edleafe is now known as figleaf | 13:02 | |
| *** pblaho has joined #openstack | 13:02 | |
| *** vvargaszte has joined #openstack | 13:02 | |
| *** armaan has quit IRC | 13:02 | |
| *** pravins has quit IRC | 13:04 | |
| *** spzala has quit IRC | 13:04 | |
| *** catintheroof has joined #openstack | 13:04 | |
| *** bobmel has joined #openstack | 13:06 | |
| *** bobmel_ has quit IRC | 13:06 | |
| *** shardy has quit IRC | 13:07 | |
| *** naichuans has quit IRC | 13:07 | |
| *** jackNemrod has quit IRC | 13:08 | |
| *** baoli has quit IRC | 13:09 | |
| *** woodard has quit IRC | 13:09 | |
| *** ldl has quit IRC | 13:10 | |
| *** KickInZOne has quit IRC | 13:12 | |
| *** Son_Goku has joined #openstack | 13:14 | |
| *** toscalix has joined #openstack | 13:18 | |
| *** jeroen92 has joined #openstack | 13:18 | |
| *** jeroen92_ has quit IRC | 13:21 | |
| *** vks1 has joined #openstack | 13:22 | |
| *** dxiri_ has quit IRC | 13:22 | |
| frickler | rand0m: since (I think) Ocata, keystone is running as a wsgi service behind apache, so no dedicated keystone.service anymore | 13:23 |
| *** rand0m has quit IRC | 13:24 | |
| *** jackNemrod has joined #openstack | 13:24 | |
| *** rbowen has quit IRC | 13:30 | |
| *** dansmith is now known as superdan | 13:30 | |
| *** dizquierdo has quit IRC | 13:31 | |
| *** galstrom_zzz is now known as galstrom | 13:32 | |
| *** dizquierdo has joined #openstack | 13:32 | |
| *** kindjal has joined #openstack | 13:33 | |
| *** pravins has joined #openstack | 13:33 | |
| *** salmankhan has joined #openstack | 13:33 | |
| *** mhickey has joined #openstack | 13:33 | |
| *** catintheroof has quit IRC | 13:34 | |
| *** janki has quit IRC | 13:37 | |
| *** indistylo has joined #openstack | 13:40 | |
| *** vvargaszte has quit IRC | 13:40 | |
| *** designbybeck has quit IRC | 13:41 | |
| *** smccarthy has joined #openstack | 13:41 | |
| *** dyasny has joined #openstack | 13:42 | |
| *** catintheroof has joined #openstack | 13:42 | |
| *** Son_Goku has quit IRC | 13:44 | |
| *** jsheeren has quit IRC | 13:47 | |
| *** josok has quit IRC | 13:48 | |
| *** spectr has quit IRC | 13:49 | |
| DHE | newton runs keystone as wsgi | 13:50 |
| *** uZiel has joined #openstack | 13:50 | |
| *** janzian has joined #openstack | 13:52 | |
| *** morazi has quit IRC | 13:52 | |
| *** sree has quit IRC | 13:54 | |
| *** sree has joined #openstack | 13:55 | |
| *** ben_roose has joined #openstack | 13:55 | |
| *** arxcruz is now known as arxcruz|pto | 13:56 | |
| *** sree has quit IRC | 14:00 | |
| *** baoli has joined #openstack | 14:01 | |
| *** VW has joined #openstack | 14:03 | |
| *** lucendio has quit IRC | 14:03 | |
| *** Dinesh_Bhor has quit IRC | 14:04 | |
| *** krisd_ has quit IRC | 14:05 | |
| *** krisd has joined #openstack | 14:06 | |
| *** woodard has joined #openstack | 14:06 | |
| *** mchlumsky has joined #openstack | 14:08 | |
| *** SimAloo has joined #openstack | 14:08 | |
| *** spectr has joined #openstack | 14:09 | |
| *** ikhan_ has quit IRC | 14:11 | |
| *** amodi has joined #openstack | 14:12 | |
| *** ffiore has quit IRC | 14:15 | |
| *** lucendio has joined #openstack | 14:17 | |
| *** pravins has quit IRC | 14:17 | |
| *** woodard has quit IRC | 14:17 | |
| *** hippiepete has joined #openstack | 14:17 | |
| *** mchlumsky has quit IRC | 14:17 | |
| *** mchlumsky has joined #openstack | 14:18 | |
| *** exodusftw has quit IRC | 14:19 | |
| *** uZiel has quit IRC | 14:21 | |
| *** rmart04 has joined #openstack | 14:22 | |
| *** suuuper has quit IRC | 14:24 | |
| *** mchlumsky has quit IRC | 14:26 | |
| *** mchlumsky has joined #openstack | 14:28 | |
| *** baoli has quit IRC | 14:29 | |
| *** xemdetia has joined #openstack | 14:29 | |
| *** jidar has quit IRC | 14:29 | |
| *** ffiore has joined #openstack | 14:30 | |
| *** exodusftw has joined #openstack | 14:30 | |
| *** jidar has joined #openstack | 14:32 | |
| *** jamesbenson has joined #openstack | 14:34 | |
| *** baoli has joined #openstack | 14:35 | |
| *** dbecker has quit IRC | 14:35 | |
| *** dbecker has joined #openstack | 14:35 | |
| *** tlaxkit has quit IRC | 14:38 | |
| *** smccarthy has quit IRC | 14:38 | |
| *** spectr has quit IRC | 14:39 | |
| *** dbecker has quit IRC | 14:40 | |
| *** dbecker has joined #openstack | 14:40 | |
| *** bvdk has quit IRC | 14:41 | |
| *** baoli has quit IRC | 14:42 | |
| *** e0ne has quit IRC | 14:45 | |
| *** baoli has joined #openstack | 14:45 | |
| *** toanju has quit IRC | 14:46 | |
| *** Cybodog has joined #openstack | 14:46 | |
| *** josecastroleon has quit IRC | 14:46 | |
| *** baoli has quit IRC | 14:48 | |
| *** armaan has joined #openstack | 14:48 | |
| *** uZiel has joined #openstack | 14:48 | |
| *** raynold has quit IRC | 14:51 | |
| *** e0ne has joined #openstack | 14:53 | |
| *** e0ne has quit IRC | 14:53 | |
| *** jwitko has joined #openstack | 14:53 | |
| *** e0ne has joined #openstack | 14:54 | |
| *** e0ne has quit IRC | 14:54 | |
| *** bobh has joined #openstack | 14:55 | |
| *** e0ne has joined #openstack | 14:55 | |
| *** milk__ has joined #openstack | 14:55 | |
| *** e0ne has quit IRC | 14:56 | |
| *** e0ne has joined #openstack | 14:56 | |
| *** e0ne has quit IRC | 14:57 | |
| *** pblaho has quit IRC | 14:58 | |
| *** milk__ has left #openstack | 14:58 | |
| *** bobh has quit IRC | 14:58 | |
| *** jpsandiego42 has quit IRC | 15:01 | |
| *** jpsandiego42 has joined #openstack | 15:02 | |
| *** jpsandiego42 has quit IRC | 15:02 | |
| *** omie88877777 has quit IRC | 15:04 | |
| *** omie888777 has quit IRC | 15:04 | |
| *** iranzo has quit IRC | 15:06 | |
| *** chlong has quit IRC | 15:06 | |
| *** marst has quit IRC | 15:06 | |
| *** marst has joined #openstack | 15:06 | |
| *** rmart04 has quit IRC | 15:06 | |
| *** baoli has joined #openstack | 15:07 | |
| *** belmoreira has quit IRC | 15:09 | |
| *** vhosakot has joined #openstack | 15:11 | |
| *** ogny has quit IRC | 15:13 | |
| *** jackNemrod has quit IRC | 15:13 | |
| *** AlexeyAbashkin has quit IRC | 15:13 | |
| *** rbowen has joined #openstack | 15:14 | |
| *** MarkBaker_ has quit IRC | 15:14 | |
| *** yamahata has joined #openstack | 15:14 | |
| *** crose has joined #openstack | 15:14 | |
| *** woodard has joined #openstack | 15:14 | |
| *** jmlowe_ has joined #openstack | 15:14 | |
| *** newmember has quit IRC | 15:15 | |
| *** sree has joined #openstack | 15:16 | |
| *** jmlowe has quit IRC | 15:16 | |
| *** iyamahat has joined #openstack | 15:16 | |
| *** links has joined #openstack | 15:19 | |
| *** janzian has quit IRC | 15:20 | |
| *** rand0m has joined #openstack | 15:21 | |
| *** grafuls has joined #openstack | 15:21 | |
| *** rand0m has quit IRC | 15:21 | |
| *** xehbit has quit IRC | 15:21 | |
| *** salv-orlando has quit IRC | 15:22 | |
| *** salv-orlando has joined #openstack | 15:23 | |
| *** saybeano has joined #openstack | 15:23 | |
| *** woodard has quit IRC | 15:24 | |
| *** yamamoto has quit IRC | 15:24 | |
| *** cshastri has quit IRC | 15:25 | |
| *** slaweq_ has quit IRC | 15:25 | |
| *** indistylo has quit IRC | 15:26 | |
| *** salv-orlando has quit IRC | 15:27 | |
| *** grafuls has quit IRC | 15:29 | |
| *** toscalix has quit IRC | 15:32 | |
| *** mwynne has quit IRC | 15:35 | |
| *** kjackal_ has quit IRC | 15:35 | |
| *** janzian has joined #openstack | 15:36 | |
| *** ricardoas has joined #openstack | 15:36 | |
| *** rsimai has quit IRC | 15:38 | |
| *** jamesbenson has quit IRC | 15:40 | |
| *** Tecumseh has quit IRC | 15:40 | |
| *** anshul has quit IRC | 15:41 | |
| *** shoogz has quit IRC | 15:42 | |
| *** jamesbenson has joined #openstack | 15:44 | |
| *** shoogz has joined #openstack | 15:44 | |
| *** shoogz has joined #openstack | 15:44 | |
| *** VW has quit IRC | 15:47 | |
| *** VW has joined #openstack | 15:48 | |
| *** tssurya has quit IRC | 15:49 | |
| *** BenderRodriguez has joined #openstack | 15:49 | |
| *** BenderRodriguez has quit IRC | 15:49 | |
| *** BenderRodriguez has joined #openstack | 15:49 | |
| *** dasTor has quit IRC | 15:49 | |
| *** rbowen has quit IRC | 15:52 | |
| *** OutBackDingo has quit IRC | 15:52 | |
| *** OutBackDingo has joined #openstack | 15:53 | |
| *** anshul has joined #openstack | 15:54 | |
| *** sree has quit IRC | 15:55 | |
| *** sree has joined #openstack | 15:55 | |
| *** iyamahat has quit IRC | 15:56 | |
| *** vishvendra has joined #openstack | 15:56 | |
| *** ChiTo has joined #openstack | 15:58 | |
| *** anshul has quit IRC | 15:58 | |
| *** iyamahat has joined #openstack | 15:59 | |
| *** smccarthy has joined #openstack | 15:59 | |
| *** sree has quit IRC | 16:00 | |
| *** smccarthy has quit IRC | 16:00 | |
| *** smccarthy has joined #openstack | 16:01 | |
| *** yamamoto has joined #openstack | 16:01 | |
| *** vishvendra has quit IRC | 16:04 | |
| *** vishvendra has joined #openstack | 16:05 | |
| *** ffiore has quit IRC | 16:05 | |
| *** hippiepete has quit IRC | 16:11 | |
| *** nick123 has quit IRC | 16:12 | |
| *** anshul has joined #openstack | 16:15 | |
| *** hippiepete has joined #openstack | 16:16 | |
| *** hippiepete has quit IRC | 16:17 | |
| *** hippiepete has joined #openstack | 16:17 | |
| *** Xiti has quit IRC | 16:18 | |
| *** mihdih has quit IRC | 16:18 | |
| *** Xiti has joined #openstack | 16:18 | |
| *** janzian has quit IRC | 16:19 | |
| *** robcresswell has joined #openstack | 16:19 | |
| *** crose has quit IRC | 16:19 | |
| *** iyamahat has quit IRC | 16:19 | |
| *** hseipp has quit IRC | 16:21 | |
| *** suuuper has joined #openstack | 16:21 | |
| *** e0ne has joined #openstack | 16:21 | |
| *** woodard has joined #openstack | 16:22 | |
| *** catintheroof has quit IRC | 16:23 | |
| *** salv-orlando has joined #openstack | 16:23 | |
| *** mwynne has joined #openstack | 16:23 | |
| *** jmlowe_ has quit IRC | 16:23 | |
| mwynne | Hi guys. The apache2 process used by keystone is always pinned at 100% CPU on my stack. Is there anything I can do to improve performance there? | 16:24 |
| *** catintheroof has joined #openstack | 16:27 | |
| *** tvinson has quit IRC | 16:27 | |
| *** tvinson has joined #openstack | 16:27 | |
| *** salv-orlando has quit IRC | 16:27 | |
| *** nick123 has joined #openstack | 16:28 | |
| *** dizquierdo has quit IRC | 16:29 | |
| *** jamesden_ has joined #openstack | 16:31 | |
| *** jeroen92 has quit IRC | 16:32 | |
| *** jamesdenton has quit IRC | 16:32 | |
| *** tvinson has quit IRC | 16:32 | |
| *** tvinson has joined #openstack | 16:32 | |
| *** woodard has quit IRC | 16:32 | |
| *** gvrangan has joined #openstack | 16:32 | |
| *** gvrangan has quit IRC | 16:33 | |
| *** dizquierdo has joined #openstack | 16:33 | |
| *** smccarthy has quit IRC | 16:33 | |
| *** gvrangan has joined #openstack | 16:33 | |
| *** kjackal_ has joined #openstack | 16:35 | |
| *** catintheroof has quit IRC | 16:35 | |
| *** salmankhan has quit IRC | 16:35 | |
| *** vishvendra has quit IRC | 16:36 | |
| *** catintheroof has joined #openstack | 16:36 | |
| *** smccarthy has joined #openstack | 16:38 | |
| *** links has quit IRC | 16:38 | |
| *** catintheroof has quit IRC | 16:39 | |
| *** catintheroof has joined #openstack | 16:39 | |
| *** gvrangan has quit IRC | 16:39 | |
| *** chito_ has joined #openstack | 16:40 | |
| *** jeroen92 has joined #openstack | 16:40 | |
| *** blacklotus89 has quit IRC | 16:40 | |
| *** gvrangan has joined #openstack | 16:42 | |
| *** cloudrancher has quit IRC | 16:43 | |
| *** ChiTo has quit IRC | 16:43 | |
| *** ttsiouts has quit IRC | 16:43 | |
| *** gvrangan has quit IRC | 16:43 | |
| *** cloudrancher has joined #openstack | 16:44 | |
| *** jeroen92 has quit IRC | 16:44 | |
| *** raynold has joined #openstack | 16:44 | |
| *** gvrangan has joined #openstack | 16:46 | |
| *** gvrangan has quit IRC | 16:51 | |
| *** achadha has joined #openstack | 16:51 | |
| *** blacklotus89 has joined #openstack | 16:52 | |
| *** AlexeyAbashkin has joined #openstack | 16:53 | |
| *** sshnaidm|off has quit IRC | 16:54 | |
| *** achadha has quit IRC | 16:55 | |
| *** mrch has quit IRC | 16:56 | |
| *** electrofelix has quit IRC | 16:56 | |
| *** AlexeyAbashkin has quit IRC | 16:57 | |
| *** knoxyy has joined #openstack | 16:57 | |
| *** v3nd3tta`` is now known as v3nd3tta | 16:58 | |
| *** mhickey has quit IRC | 16:59 | |
| *** ralonsoh has quit IRC | 17:00 | |
| *** knoxy has quit IRC | 17:01 | |
| *** fzdarsky is now known as fzdarsky|afk | 17:03 | |
| *** MrAbaddon has quit IRC | 17:03 | |
| *** fragatina has quit IRC | 17:03 | |
| *** v3nd3tta is now known as v3nd3tta`` | 17:07 | |
| *** rmcallis has joined #openstack | 17:07 | |
| *** jeroen92 has joined #openstack | 17:08 | |
| *** kjackal_ has quit IRC | 17:09 | |
| *** dxiri has joined #openstack | 17:09 | |
| *** fsimonce has quit IRC | 17:11 | |
| *** yamamoto has quit IRC | 17:13 | |
| *** dyasny has quit IRC | 17:14 | |
| *** vishvendra has joined #openstack | 17:15 | |
| *** racedo_ has joined #openstack | 17:15 | |
| *** racedo has quit IRC | 17:17 | |
| *** ianychoi_ has quit IRC | 17:17 | |
| *** saviosaltamontes has joined #openstack | 17:19 | |
| *** blacklotus89 has quit IRC | 17:19 | |
| *** yamamoto has joined #openstack | 17:21 | |
| *** fragatina has joined #openstack | 17:21 | |
| *** idlemind has quit IRC | 17:22 | |
| *** sshnaidm has joined #openstack | 17:24 | |
| *** kjackal_ has joined #openstack | 17:24 | |
| *** MarkBaker has joined #openstack | 17:24 | |
| *** anshul has quit IRC | 17:26 | |
| mwynne | Anyone able to help me out there? | 17:27 |
| davidbowlby | mwynne, what do the logs say? | 17:27 |
| *** gszasz has quit IRC | 17:27 | |
| * davidbowlby 's catch phrase... with good reason | 17:27 | |
| *** v3nd3tta`` is now known as v3nd3tta | 17:28 | |
| *** v3nd3tta is now known as v3nd3tta`` | 17:30 | |
| *** jmlowe has joined #openstack | 17:30 | |
| *** sshnaidm has quit IRC | 17:30 | |
| *** woodard has joined #openstack | 17:30 | |
| *** kjackal_ has quit IRC | 17:30 | |
| *** robbiew has quit IRC | 17:30 | |
| *** idlemind has joined #openstack | 17:31 | |
| mwynne | davidbowlby: 2017-10-27 17:31:38.355 31477 INFO keystone.common.wsgi [req-6c9c7360-5dd7-4109-8b85-a09130d802c9 edae1a5301044b0c9924eb09df0f080d 4f779ae189834a9a9e193b1b9d7546a6 - default default] GET http://10.10.10.5:35357/v3/auth/tokens | 17:32 |
| mwynne | getting lots of those in the admin log | 17:32 |
| mwynne | Constantly getting those | 17:32 |
| mwynne | the other logs are pretty silent | 17:32 |
| davidbowlby | get for tokens wouldn't be a big deal, I would think | 17:33 |
| davidbowlby | unless you're public facing and getting slammed from a DOS | 17:33 |
| mwynne | Not public facing | 17:33 |
| *** kjackal_ has joined #openstack | 17:33 | |
| davidbowlby | ok so prob not an issue | 17:33 |
| davidbowlby | you don't see anything in the apache2 error log? | 17:33 |
| mwynne | davidbowlby: the error log's empty | 17:35 |
| davidbowlby | how about keystone logs? | 17:35 |
| *** germs has joined #openstack | 17:35 | |
| mwynne | The keystone admin log is just full of those messages I sent | 17:36 |
| mwynne | The others are pretty much idle | 17:37 |
| *** forgotmynick has joined #openstack | 17:37 | |
| *** MarkBaker_ has joined #openstack | 17:37 | |
| davidbowlby | what do you see in the apache2 access log | 17:38 |
| davidbowlby | just those token calls? | 17:38 |
| davidbowlby | you should see your horizon logins/etc | 17:38 |
| mwynne | yeah there are others in that log | 17:39 |
| mwynne | but I rarely use horizon so there aren't many of those.. | 17:39 |
| *** fatdragon has joined #openstack | 17:39 | |
| mwynne | I should mention that this is a stack with 92 tenants/networks and 180 vms | 17:40 |
| *** woodard has quit IRC | 17:40 | |
| *** MarkBaker has quit IRC | 17:40 | |
| *** macermak has quit IRC | 17:41 | |
| mwynne | davidbowlby: So that controller's pretty busy, but when I'm not using heat to spin up stacks it's using a lot of CPU | 17:41 |
| mwynne | Always apache/keystone | 17:41 |
| *** bbcmicrocomputer has quit IRC | 17:41 | |
| davidbowlby | mwynne, what kind of box are you running it on | 17:43 |
| mwynne | davidbowlby: I'm running this in a nested openstack cloud | 17:44 |
| mwynne | 8 CPUs + 16 GB of ram | 17:44 |
| *** ffiore has joined #openstack | 17:44 | |
| *** chlong has joined #openstack | 17:44 | |
| mwynne | the controller and all compute nodes are VMs inside an undercloud | 17:44 |
| davidbowlby | so is the instance performance bad? | 17:45 |
| davidbowlby | what do your undercloud reports say for performance? | 17:45 |
| *** takimbal has quit IRC | 17:45 | |
| *** iyamahat has joined #openstack | 17:45 | |
| davidbowlby | when you do top and see apache2 up top, is it the horizon one or the www-data one | 17:45 |
| *** takimbal has joined #openstack | 17:46 | |
| mwynne | The other VMs on the undercloud are OK, and the hardware is in decent shape. It's just this one VM that's getting punished. | 17:46 |
| davidbowlby | or the root one | 17:46 |
| *** jpsandiego42 has joined #openstack | 17:46 | |
| mwynne | 31476 keystone 20 0 647660 144180 12996 S 115.6 0.9 727:18.22 apache2 | 17:46 |
| mwynne | 31477 keystone 20 0 648172 139064 12908 S 112.3 0.8 727:13.98 apache2 | 17:46 |
| davidbowlby | ok | 17:46 |
| davidbowlby | what version of openstack are you using? | 17:46 |
| mwynne | Ocata | 17:46 |
| *** jdwidari has joined #openstack | 17:46 | |
| *** MrAbaddon has joined #openstack | 17:48 | |
| *** yamamoto has quit IRC | 17:49 | |
| *** kjackal_ has quit IRC | 17:49 | |
| *** woodard has joined #openstack | 17:50 | |
| *** penick has joined #openstack | 17:50 | |
| *** v3nd3tta`` is now known as v3nd3tta | 17:51 | |
| *** yamahata has quit IRC | 17:52 | |
| mwynne | davidbowlby: Any thoughts? | 17:52 |
| davidbowlby | I ran an ancient version of openstack so I don't have a great reference point to help you | 17:52 |
| *** iyamahat has quit IRC | 17:52 | |
| *** kristianjf has joined #openstack | 17:52 | |
| davidbowlby | but I'd be hitting the logs hard to figure out if there is a process out there with errors in its log | 17:52 |
| davidbowlby | I don't know what would make it constantly hit your tokens like that | 17:53 |
| davidbowlby | unless you have some backend process breaking and it keeps trying | 17:53 |
| *** yamamoto has joined #openstack | 17:53 | |
| *** kristianjf has quit IRC | 17:53 | |
| *** metachr0n has quit IRC | 17:53 | |
| davidbowlby | I'm running 2014.1.1 lol | 17:55 |
| davidbowlby | and I'm terrified to update at this point lol | 17:55 |
| *** drbroiler has joined #openstack | 17:55 | |
| davidbowlby | I'll most likely pull drives and rebuild from scratch haha | 17:56 |
| davidbowlby | mwynne, look for the same identifier to come up, like 4f779ae189834a9a9e193b1b9d7546a6 | 17:56 |
| davidbowlby | typically is a reference to the id of the process trying to perform the action | 17:57 |
| mwynne | davidbowlby: [req-c42231a1-5f70-4e42-a22c-3e9e979318a6 edae1a5301044b0c9924eb09df0f080d 4f779ae189834a9a9e193b1b9d7546a6 | 17:57 |
| mwynne | that third column is always the same | 17:57 |
| mwynne | or nearly always | 17:58 |
| *** bitchecker has quit IRC | 17:58 | |
| mwynne | no idea what that represents though | 17:58 |
| *** newmember has joined #openstack | 17:58 | |
| *** yamamoto has quit IRC | 17:59 | |
| mwynne | davidbowlby: Lots of "4f779ae189834a9a9e193b1b9d7546a6" in neutron server logs... | 17:59 |
| davidbowlby | there is somewhere, and it's been forever since I've been in there, where you can query that ID and see what process it is | 18:00 |
| *** snowman4839 has quit IRC | 18:00 | |
| *** v3nd3tta is now known as v3nd3tta`` | 18:00 | |
| *** snowman4839 has joined #openstack | 18:00 | |
| davidbowlby | mwynne, I'm no expert by any means, just set it up years ago and it just works for me | 18:01 |
| mwynne | davidbowlby: Thanks anyway :) | 18:01 |
| davidbowlby | mwynne, but I do recall the ID thing from a console interface, actually, now that I think about it it's in horizon | 18:01 |
| davidbowlby | too | 18:01 |
| *** jbadiapa has quit IRC | 18:02 | |
| *** rmart04 has joined #openstack | 18:02 | |
| *** bitchecker has joined #openstack | 18:03 | |
| davidbowlby | you can pull up the hypervisors in the system panel under admin in horizon | 18:04 |
| davidbowlby | it lists the instances and their IDs | 18:04 |
| davidbowlby | you could see if it is an instace ID | 18:04 |
| davidbowlby | *instance ID | 18:04 |
| davidbowlby | I don't use neutron in my deployment | 18:05 |
| *** harlowja has quit IRC | 18:05 | |
| *** harlowja has joined #openstack | 18:05 | |
| *** ffiore has quit IRC | 18:08 | |
| *** cloudrancher has quit IRC | 18:11 | |
| *** cloudrancher has joined #openstack | 18:12 | |
| *** vks1 has quit IRC | 18:12 | |
| *** saviosaltamontes has quit IRC | 18:12 | |
| *** mab__ has joined #openstack | 18:13 | |
| *** jeroen92 has quit IRC | 18:13 | |
| *** nick123 has quit IRC | 18:14 | |
| *** e0ne has quit IRC | 18:15 | |
| mwynne | davidbowlby: https://docs.openstack.org/keystone/latest/admin/identity-caching-layer.html | 18:15 |
| *** MarkBaker_ has quit IRC | 18:15 | |
| mwynne | Enabling caching made a huge difference | 18:15 |
| *** v3nd3tta`` is now known as v3nd3tta | 18:21 | |
| *** ginsul has quit IRC | 18:21 | |
| *** ginsul has joined #openstack | 18:22 | |
| *** dyasny has joined #openstack | 18:22 | |
| *** ginsul has joined #openstack | 18:22 | |
| *** jeroen92 has joined #openstack | 18:23 | |
| *** salv-orlando has joined #openstack | 18:24 | |
| davidbowlby | mwynne, cool! | 18:25 |
| mwynne | APIs are WAY faster now too :) | 18:26 |
| *** ginsul has quit IRC | 18:26 | |
| *** LocaMocha has quit IRC | 18:27 | |
| *** salv-orlando has quit IRC | 18:29 | |
| *** georgem1 has joined #openstack | 18:30 | |
| *** nick123 has joined #openstack | 18:31 | |
| *** crose has joined #openstack | 18:31 | |
| *** geaaru has quit IRC | 18:32 | |
| *** jackNemrod has joined #openstack | 18:33 | |
| *** uZiel has quit IRC | 18:33 | |
| *** armaan has quit IRC | 18:34 | |
| *** rtjure has quit IRC | 18:34 | |
| *** rtjure has joined #openstack | 18:35 | |
| *** vishvendra1 has joined #openstack | 18:36 | |
| *** cah_link has quit IRC | 18:37 | |
| *** jackNemrod has quit IRC | 18:38 | |
| *** rtjure has quit IRC | 18:39 | |
| *** vishvendra has quit IRC | 18:40 | |
| *** vishvendra1 is now known as vishvendra | 18:40 | |
| *** PityDaFool has quit IRC | 18:41 | |
| *** mab__ has quit IRC | 18:42 | |
| *** jeroen92 has quit IRC | 18:42 | |
| *** cloudrancher has quit IRC | 18:42 | |
| *** vishvendra1 has joined #openstack | 18:42 | |
| *** jpsandiego42 has quit IRC | 18:42 | |
| *** cloudrancher has joined #openstack | 18:43 | |
| *** vishvendra has quit IRC | 18:44 | |
| *** vishvendra1 is now known as vishvendra | 18:44 | |
| *** PityDaFool has joined #openstack | 18:45 | |
| *** rtjure has joined #openstack | 18:45 | |
| *** saviosaltamontes has joined #openstack | 18:45 | |
| *** fragatina has quit IRC | 18:48 | |
| *** pbandark has quit IRC | 18:49 | |
| *** upworklink has joined #openstack | 18:49 | |
| *** rtjure has quit IRC | 18:50 | |
| *** janzian has joined #openstack | 18:58 | |
| *** yamamoto has joined #openstack | 18:59 | |
| *** jeroen92 has joined #openstack | 19:04 | |
| *** takamatsu has quit IRC | 19:05 | |
| *** yamamoto has quit IRC | 19:06 | |
| *** cah_link has joined #openstack | 19:07 | |
| *** crose has quit IRC | 19:07 | |
| *** mn3m has quit IRC | 19:09 | |
| *** crose has joined #openstack | 19:10 | |
| *** sree has joined #openstack | 19:11 | |
| *** saviosaltamontes has quit IRC | 19:12 | |
| *** dneary has quit IRC | 19:12 | |
| *** cah_link has quit IRC | 19:13 | |
| *** jamesben_ has joined #openstack | 19:13 | |
| *** jamesbe__ has joined #openstack | 19:14 | |
| *** jamesbenson has quit IRC | 19:15 | |
| *** saviosaltamontes has joined #openstack | 19:16 | |
| *** sree has quit IRC | 19:16 | |
| *** vishvendra1 has joined #openstack | 19:17 | |
| *** cah_link has joined #openstack | 19:17 | |
| *** jamesben_ has quit IRC | 19:18 | |
| mwynne | Has anyone seen this before while trying to create networks? | 19:19 |
| mwynne | ERROR neutron.plugins.ml2.managers [req-302e58ec-cbf4-417f-8ad5-aabf8405aa62 - - - - -] Failed to bind port f0f57e65-2de5-4563-98a7-2682d7703950 on host controller-5 for vnic_type normal using segments [{'segmentation_id': 1082, 'physical_network': u'physnet1', 'id': u'ab0904fc-a6f1-4c3c-8267-52fa80c2d1d7', 'network_type': u'vlan'}] | 19:19 |
| *** crose has quit IRC | 19:19 | |
| *** pbandark has joined #openstack | 19:20 | |
| *** vishvendra has quit IRC | 19:20 | |
| *** vishvendra1 is now known as vishvendra | 19:20 | |
| *** uZiel has joined #openstack | 19:21 | |
| *** saviosaltamontes has quit IRC | 19:21 | |
| *** v3nd3tta is now known as v3nd3tta`` | 19:22 | |
| *** jamesbenson has joined #openstack | 19:23 | |
| *** ChubYann has joined #openstack | 19:23 | |
| *** jpsandiego42 has joined #openstack | 19:24 | |
| *** pbandark has quit IRC | 19:24 | |
| *** Anario has joined #openstack | 19:25 | |
| *** pbandark has joined #openstack | 19:26 | |
| SamYaple | mwynne: yea. pretty common. your logs should have more info | 19:26 |
| Anario | Hi All, Dumb question sorry, how to port dashboard to public ip address in liberty version? Can't find OPENSTACK_HOST = parameter | 19:26 |
| SamYaple | specific the host where it failed | 19:26 |
| *** jamesbe__ has quit IRC | 19:27 | |
| *** rmart04 has quit IRC | 19:27 | |
| *** uZiel has quit IRC | 19:28 | |
| mwynne | SamYaple: any idea which logs? | 19:28 |
| mwynne | that error's in neutron-server.log | 19:28 |
| SamYaple | mwynne: neutron ovs/linuxbridge logs on host where binding failed | 19:28 |
| SamYaple | Anario: horizon is typically run behind apache, so apache would be where you would change teh binding. But the sepcific option you are looking for is in /etc/openstack-dashboard/local_settings (or similiar, it can be in different spots depening on how you installed it) | 19:29 |
| *** catintheroof has quit IRC | 19:29 | |
| *** e0ne has joined #openstack | 19:29 | |
| *** e0ne has quit IRC | 19:30 | |
| *** OutBackDingo has quit IRC | 19:32 | |
| mwynne | SamYaple: Is there a common cause for that error? | 19:32 |
| *** tesseract has quit IRC | 19:33 | |
| *** irated has joined #openstack | 19:33 | |
| *** irated has joined #openstack | 19:33 | |
| *** takimbal has quit IRC | 19:33 | |
| SamYaple | mwynne: could be a number of common things. logs help | 19:34 |
| *** takimbal has joined #openstack | 19:36 | |
| *** cah_link has quit IRC | 19:36 | |
| *** cah_link has joined #openstack | 19:38 | |
| *** iyamahat has joined #openstack | 19:40 | |
| *** OutBackDingo has joined #openstack | 19:40 | |
| *** catintheroof has joined #openstack | 19:41 | |
| *** iyamahat_ has joined #openstack | 19:42 | |
| *** cah_link1 has joined #openstack | 19:43 | |
| *** cah_link has quit IRC | 19:44 | |
| *** cah_link1 is now known as cah_link | 19:44 | |
| *** ginsul has joined #openstack | 19:45 | |
| *** catintheroof has quit IRC | 19:45 | |
| *** iyamahat has quit IRC | 19:45 | |
| *** sree has joined #openstack | 19:46 | |
| Anario | SamYaple Thanks, but i can't find this parameter in openstack-dashboard/local_settings | 19:49 |
| Anario | SamYaple it should be somewhere :) | 19:49 |
| *** jamesbenson has quit IRC | 19:49 | |
| mwynne | Anario: ubuntu or rhel? | 19:51 |
| *** sree has quit IRC | 19:51 | |
| Anario | mwynne rhel | 19:51 |
| *** pchavva has quit IRC | 19:51 | |
| *** jamesbenson has joined #openstack | 19:52 | |
| SamYaple | Anario: it isn't consumed directly anywhere. its just used as a temp variable for OPENSTACK_KEYSTONE_URL | 19:53 |
| SamYaple | OPENSTACK_KEYSTONE_URL is the value you care about | 19:53 |
| mwynne | amodi: Can he just set a ServerAlias in httpd.conf for his public IP? | 19:53 |
| mwynne | whoops, SamYaple ^^ | 19:54 |
| *** suuuper has quit IRC | 19:55 | |
| *** suuuper has joined #openstack | 19:55 | |
| SamYaple | mwynne: depends on what he is trying to do. OPENSTACK_KEYSTONE_URL is what horizon uses to talk to keystone, not the ip/port horizon is listening on | 19:56 |
| mwynne | Oh, I see. | 19:56 |
| mwynne | I might have misinterpreted what he wanted. | 19:57 |
| *** rcernin has joined #openstack | 19:57 | |
| SamYaple | yea im just answering the questions asked, i dont know the end goal here | 19:57 |
| *** SmearedBeard has quit IRC | 19:57 | |
| *** sree has joined #openstack | 19:58 | |
| *** suuuper has quit IRC | 19:59 | |
| Anario | SamYaple I have two interfaces eth0 with local ip 1.1.1.1 and wth1 with public ip == I can Access dashboard through local ... but i want to access dashboard from public | 20:00 |
| *** yamamoto has joined #openstack | 20:02 | |
| *** SmearedBeard has joined #openstack | 20:02 | |
| *** bobh has joined #openstack | 20:02 | |
| *** sree has quit IRC | 20:03 | |
| *** rmart04 has joined #openstack | 20:03 | |
| *** rmart04 has quit IRC | 20:03 | |
| *** bobh has quit IRC | 20:04 | |
| *** bobh has joined #openstack | 20:05 | |
| SamYaple | Anario: can you pastebin `ss -plnt`? | 20:05 |
| *** yamamoto has quit IRC | 20:08 | |
| *** aselius has joined #openstack | 20:09 | |
| *** bobh has quit IRC | 20:09 | |
| *** georgem1 has quit IRC | 20:10 | |
| Anario | SamYaple fix it! Thanks ... they move it to 15-horizon_vhost.conf | 20:10 |
| *** penick has quit IRC | 20:11 | |
| Anario | mwynne Thank you! | 20:12 |
| *** jmlowe has quit IRC | 20:14 | |
| *** forgotmynick has quit IRC | 20:17 | |
| mwynne | Anario: No problem :) | 20:19 |
| *** vishvendra1 has joined #openstack | 20:21 | |
| *** mn3m has joined #openstack | 20:22 | |
| *** Raymii has quit IRC | 20:24 | |
| *** vishvendra has quit IRC | 20:24 | |
| *** salv-orlando has joined #openstack | 20:25 | |
| *** yamahata has joined #openstack | 20:25 | |
| *** ginsul has quit IRC | 20:26 | |
| *** ginsul has joined #openstack | 20:26 | |
| *** vishvendra1 has quit IRC | 20:28 | |
| *** salv-orlando has quit IRC | 20:29 | |
| *** ginsul has quit IRC | 20:31 | |
| *** ginsul has joined #openstack | 20:32 | |
| *** kjackal_ has joined #openstack | 20:34 | |
| *** salv-orlando has joined #openstack | 20:35 | |
| *** thorst has quit IRC | 20:38 | |
| *** imfearless has quit IRC | 20:42 | |
| *** dxiri has quit IRC | 20:43 | |
| *** dxiri has joined #openstack | 20:43 | |
| *** cah_link has quit IRC | 20:44 | |
| *** yamahata has quit IRC | 20:44 | |
| *** iyamahat_ has quit IRC | 20:45 | |
| *** pchavva has joined #openstack | 20:45 | |
| *** mn3m has quit IRC | 20:45 | |
| *** mloza has quit IRC | 20:45 | |
| *** dxiri has quit IRC | 20:48 | |
| *** mloza has joined #openstack | 20:48 | |
| *** mn3m has joined #openstack | 20:52 | |
| *** pchavva has quit IRC | 20:52 | |
| *** metachr0n has joined #openstack | 20:54 | |
| *** jamesbenson has quit IRC | 20:54 | |
| *** mn3m has quit IRC | 21:00 | |
| *** mab__ has joined #openstack | 21:01 | |
| *** Anario has quit IRC | 21:01 | |
| *** galstrom is now known as galstrom_zzz | 21:03 | |
| *** pchavva has joined #openstack | 21:04 | |
| *** fragatina has joined #openstack | 21:04 | |
| *** yamamoto has joined #openstack | 21:05 | |
| *** achadha has joined #openstack | 21:05 | |
| *** iyamahat has joined #openstack | 21:06 | |
| *** pchavva has quit IRC | 21:09 | |
| *** yamamoto has quit IRC | 21:10 | |
| *** mab__ has quit IRC | 21:10 | |
| *** jwitko has quit IRC | 21:12 | |
| *** marst has quit IRC | 21:15 | |
| *** jeroen92 has quit IRC | 21:16 | |
| *** OutBackDingo has quit IRC | 21:22 | |
| *** OutBackDingo has joined #openstack | 21:22 | |
| *** dizquierdo has quit IRC | 21:23 | |
| *** dxiri has joined #openstack | 21:23 | |
| *** ben_roose has quit IRC | 21:24 | |
| *** martinus__ has quit IRC | 21:24 | |
| *** yamahata has joined #openstack | 21:25 | |
| *** penick has joined #openstack | 21:25 | |
| *** xemdetia has quit IRC | 21:26 | |
| *** rcernin has quit IRC | 21:28 | |
| *** SimAloo has quit IRC | 21:30 | |
| mwynne | SamYaple: Any idea which pool size needs to be increased for this error? ERROR oslo_messaging.rpc.server TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 | 21:32 |
| mwynne | The error's in neutron-server.log | 21:32 |
| *** sree has joined #openstack | 21:32 | |
| *** newmember has quit IRC | 21:33 | |
| *** upworklink has quit IRC | 21:33 | |
| *** sree has quit IRC | 21:37 | |
| *** amodi has quit IRC | 21:38 | |
| *** jamesden_ is now known as jamesdenton | 21:41 | |
| *** baoli has quit IRC | 21:42 | |
| *** geaaru has joined #openstack | 21:42 | |
| *** janzian has quit IRC | 21:45 | |
| *** salv-orlando has quit IRC | 21:50 | |
| *** salv-orlando has joined #openstack | 21:50 | |
| *** takimbal has quit IRC | 21:51 | |
| *** mchlumsky has quit IRC | 21:53 | |
| *** salv-orlando has quit IRC | 21:55 | |
| *** aconole has quit IRC | 21:55 | |
| SamYaple | mwynne: da. thats the connection to the database | 21:57 |
| SamYaple | need to bump up max_connections on the DB side | 21:57 |
| SamYaple | probably | 21:57 |
| *** galstrom_zzz is now known as galstrom | 21:57 | |
| *** dxiri has quit IRC | 21:57 | |
| *** figleaf is now known as edleafe | 22:01 | |
| *** galstrom is now known as galstrom_zzz | 22:02 | |
| *** salv-orlando has joined #openstack | 22:03 | |
| *** abu0 has joined #openstack | 22:04 | |
| *** kevin_b has joined #openstack | 22:06 | |
| *** takimbal has joined #openstack | 22:06 | |
| *** yamamoto has joined #openstack | 22:06 | |
| mwynne | SamYaple: max connections for mysql? | 22:09 |
| mwynne | in the sql config? | 22:09 |
| *** yamamoto has quit IRC | 22:11 | |
| *** SmearedBeard has quit IRC | 22:12 | |
| *** iyamahat_ has joined #openstack | 22:13 | |
| *** jtomasek has quit IRC | 22:14 | |
| *** shubjero has quit IRC | 22:14 | |
| SamYaple | yup | 22:14 |
| *** iyamahat has quit IRC | 22:15 | |
| *** sree has joined #openstack | 22:19 | |
| *** kindjal has quit IRC | 22:20 | |
| *** achadha has quit IRC | 22:20 | |
| *** achadha has joined #openstack | 22:20 | |
| mwynne | I'll give that a try | 22:20 |
| *** iyamahat__ has joined #openstack | 22:21 | |
| *** iyamahat__ has quit IRC | 22:21 | |
| *** iyamahat has joined #openstack | 22:22 | |
| *** sree has quit IRC | 22:23 | |
| *** thorst has joined #openstack | 22:24 | |
| *** iyamahat_ has quit IRC | 22:24 | |
| *** achadha_ has joined #openstack | 22:26 | |
| *** achadha has quit IRC | 22:26 | |
| *** salv-orlando has quit IRC | 22:27 | |
| *** jeroen92 has joined #openstack | 22:28 | |
| *** salv-orlando has joined #openstack | 22:28 | |
| *** thorst has quit IRC | 22:29 | |
| *** abu0_ has joined #openstack | 22:29 | |
| *** achadha_ has quit IRC | 22:30 | |
| *** iyamahat_ has joined #openstack | 22:30 | |
| *** abu0 has quit IRC | 22:30 | |
| *** jeroen92 has quit IRC | 22:33 | |
| *** iyamahat__ has joined #openstack | 22:33 | |
| *** iyamahat has quit IRC | 22:33 | |
| *** achadha has joined #openstack | 22:33 | |
| kevin_b | anyone know of a broadcast issue with ovs when using it with ml2 vlan. basically we get a network storm if ovs gets restarted(manual or at boot) | 22:34 |
| *** salv-orlando has quit IRC | 22:35 | |
| *** salv-orlando has joined #openstack | 22:36 | |
| *** iyamahat_ has quit IRC | 22:37 | |
| *** sshnaidm has joined #openstack | 22:38 | |
| *** iyamahat__ has quit IRC | 22:38 | |
| *** achadha has quit IRC | 22:38 | |
| *** v3nd3tta`` is now known as v3nd3tta | 22:39 | |
| *** abu0 has joined #openstack | 22:39 | |
| *** metachr0n has quit IRC | 22:39 | |
| *** galstrom_zzz is now known as galstrom | 22:40 | |
| *** salv-orlando has quit IRC | 22:40 | |
| *** galstrom is now known as galstrom_zzz | 22:40 | |
| *** abu0_ has quit IRC | 22:41 | |
| *** Cybodog has quit IRC | 22:52 | |
| SamYaple | kevin_b: what version of ovs? | 22:52 |
| kevin_b | one sec | 22:53 |
| kevin_b | $ ovs-dpctl --version | 22:55 |
| kevin_b | ovs-dpctl (Open vSwitch) 2.3.1 | 22:55 |
| kevin_b | Compiled Apr 13 2015 15:07:34 | 22:55 |
| kevin_b | ubuntu 14.04 kernel 3.13 | 22:56 |
| SamYaple | kevin_b: are you using arp_responder? | 22:58 |
| SamYaple | kevin_b: specifically, can you pastebin your ml2_conf.ini | 22:59 |
| *** dxiri has joined #openstack | 23:00 | |
| *** tep has joined #openstack | 23:01 | |
| kevin_b | [agent] | 23:01 |
| kevin_b | l2_population = False | 23:01 |
| kevin_b | polling_interval = 2 | 23:01 |
| kevin_b | arp_responder = False | 23:01 |
| kevin_b | dont_fragment = False | 23:01 |
| kevin_b | the rest i really cant share | 23:01 |
| SamYaple | meh its all i need | 23:01 |
| SamYaple | ok so, last question before advice | 23:01 |
| SamYaple | what version of openstack? | 23:01 |
| kevin_b | Sadly Juno.. working on upgrades | 23:02 |
| kevin_b | basically the same problem if someone restarts the agent, know that's fixed in newer revs, but this looks to be an ovs level issue | 23:02 |
| SamYaple | cool. so are you *sure* its a storm and not just alot of arp requests? | 23:02 |
| kevin_b | takes 5+ minutes to recover | 23:03 |
| SamYaple | yea. thats not a storm. thats just alot of requests | 23:03 |
| SamYaple | you want to turn on l2_pop and arp_responder | 23:03 |
| SamYaple | should resolve your issues | 23:03 |
| kevin_b | for vlan based vm's ? | 23:04 |
| kevin_b | GRE based ones have always been fine in this config | 23:04 |
| SamYaple | oh i see. i read vxlan | 23:04 |
| SamYaple | hmmm. let me think | 23:04 |
| kevin_b | vm->ovs->vlan->bond0->world | 23:05 |
| kevin_b | if it helps | 23:05 |
| kevin_b | bond0 is lacp | 23:05 |
| SamYaple | you definetely should have l2_pop and arp_responder onfor any tunnel networks. they were specifically written for ovs. but back to the issue | 23:05 |
| kevin_b | thanks for thinking on it. I ran out of ideas | 23:07 |
| *** yamamoto has joined #openstack | 23:07 | |
| SamYaple | ok more questions | 23:07 |
| SamYaple | does it act like this when you restart *any* node, or just the comptues? or just the controllers? | 23:08 |
| *** smccarthy has quit IRC | 23:10 | |
| *** yamahata has quit IRC | 23:10 | |
| kevin_b | if ovs is restarted on any host with a direct to vlan vm it impacts the whole network | 23:10 |
| *** drbroiler has quit IRC | 23:10 | |
| SamYaple | so the compute nodes | 23:11 |
| kevin_b | yep | 23:11 |
| SamYaple | ok... bababaaa | 23:11 |
| SamYaple | does restarting neutron-openswitch-agent on the compute also cause the issue? | 23:11 |
| kevin_b | yes, mostly because on juno though. supposedly fixed in newer versions | 23:12 |
| SamYaple | right because back then it regened all the rules | 23:12 |
| SamYaple | i rememebr that | 23:12 |
| *** yamamoto has quit IRC | 23:12 | |
| SamYaple | i suspect that might be the entirety of the issue here | 23:12 |
| SamYaple | i havent had ovs issues like that in a couple of years | 23:13 |
| *** ginsul has quit IRC | 23:13 | |
| SamYaple | my best advice would be to migrate vms before you do maintenance on the computes | 23:13 |
| *** ginsul has joined #openstack | 23:13 | |
| SamYaple | but thats my general advice anyway | 23:13 |
| *** gnufied has quit IRC | 23:13 | |
| SamYaple | kevin_b: you have the option of pulling in the 4.4 kernel in 14.04 which has newer better ovs if you think its an ovs issue | 23:14 |
| kevin_b | That's another one i've been beating on... newer kernels. they are staying on 3.13 because it's "tested" :( | 23:14 |
| SamYaple | well 4.4 is the lts kernel | 23:15 |
| SamYaple | its "tested" in the same way | 23:15 |
| SamYaple | and it has support until 14.04 EOLs | 23:15 |
| kevin_b | by tested i mean tested in our setup. | 23:15 |
| kevin_b | 3.13 is also supported until 14.04 EOLS, so it's hard to convince them to change | 23:15 |
| SamYaple | i get your point | 23:15 |
| SamYaple | if you can do this on a test node, showing either diffeernt behaviour (it doesnt break everythign when a node reboots) or the same then youll have more info | 23:16 |
| kevin_b | so is it basically because the flow table is gone that the vlan's flood? | 23:16 |
| kevin_b | yep i'll see if i can get that tested. | 23:16 |
| SamYaple | kevin_b: thats what my suspicion is, yes | 23:17 |
| SamYaple | i mean that is behaviour ive seen before | 23:17 |
| *** ginsul has quit IRC | 23:17 | |
| kevin_b | i see there's an option to the init script to keep flows may have to test that and see if it stops the mess at least on planned restarts | 23:17 |
| kevin_b | probably not much we can do on reboots | 23:18 |
| SamYaple | kevin_b: you might try disabling neutron-openvswitch-agent and rebooting | 23:18 |
| SamYaple | if on reboot the "storm" doesnt happen, then you have your answer | 23:18 |
| kevin_b | i didn't think flow tables survived a reboot | 23:19 |
| SamYaple | tis my point | 23:19 |
| SamYaple | if you suspect its an issue with OVS, this will prove/disprove that | 23:20 |
| kevin_b | ahh | 23:20 |
| kevin_b | if we reboot and leave physical interfaces offline, let openvswitch-agent and ovs start, wait 15-20 minutes and bring the external interface up the problem doesn't happen | 23:21 |
| kevin_b | if that makes any sense | 23:21 |
| SamYaple | youre really testing my memory here, but if i *recall* correctly, the issue was with the way it laid down the flow rules, it doesnt do it all atomically, so there are times when the flows are partitially there (even mometarily) which caused issues | 23:23 |
| SamYaple | fixed with drop all rules later | 23:23 |
| kevin_b | so without the drop all rules it's like having a dumb hub in there | 23:23 |
| SamYaple | again, if my memory isnt letting me down yes | 23:24 |
| kevin_b | hmm maybe we can script something to force the drop all rules in before agent startup | 23:24 |
| kevin_b | have to think on that one | 23:24 |
| SamYaple | nah its just going to clear the rule before it lays down its own | 23:25 |
| *** penick has quit IRC | 23:25 | |
| SamYaple | so youll end up in the same situation | 23:25 |
| kevin_b | and when ovs is restarted but not the agent, the agent notices and wipes and restarts the same way | 23:26 |
| kevin_b | so restarting either makes for a mess | 23:26 |
| kevin_b | what i think we need to try is an ovs restart WITHOUT the agent running. | 23:27 |
| SamYaple | upgradingis the solution! :/ | 23:27 |
| *** thorst has joined #openstack | 23:27 | |
| kevin_b | yep painfully aware | 23:27 |
| SamYaple | yea possibly chanigng the orderingwill help you here | 23:27 |
| kevin_b | ok thanks for the help. gave me some ideas to try next round of debugging | 23:27 |
| SamYaple | good luck! | 23:28 |
| kevin_b | in the end it's because the agent isn't atomic. The next test on my list is disabling the agent, and just restarting ovs | 23:29 |
| SamYaple | i said dont quote me! | 23:29 |
| SamYaple | :) | 23:29 |
| kevin_b | it'll probably be fine without the agent making a mess | 23:29 |
| SamYaple | you should be happy to know though that i havent had less issues in a long time. at least since liberty | 23:29 |
| SamYaple | s/less/these | 23:29 |
| kevin_b | dont worry i wont.. they'd probably have a cow if they knew i was asking on IRC ;) | 23:30 |
| SamYaple | "hey i talked the guys that wrote this software and they said...." | 23:30 |
| SamYaple | whats wrong with that | 23:30 |
| kevin_b | heck i've been doing this since the days of diablo so.... | 23:30 |
| SamYaple | nice! diablo was my first cluster | 23:30 |
| SamYaple | i still worked on an essex cluster until last year | 23:31 |
| SamYaple | i mean "worked" we would just tell them to upgrade | 23:31 |
| kevin_b | that site until it was decommed was the most stable. It was just linux bridges to vlan :P | 23:31 |
| SamYaple | i happen to love OVS | 23:33 |
| SamYaple | even with its issues | 23:33 |
| SamYaple | do not like linux bridge. even with its (mostly) lack of issues | 23:33 |
| kevin_b | yeah, i'm still using linux bridges on my home stuff, but wonder if it is worth going to ovs | 23:33 |
| SamYaple | ovs can do alot of realyl cool stuff that LB never ever will be able too | 23:33 |
| SamYaple | forbetter or worse ;) | 23:33 |
| SamYaple | is it "worth" it? probably not | 23:34 |
| SamYaple | is it "cool" and are there alot of cool things to play with? absolutely | 23:34 |
| SamYaple | for your home cluster that is | 23:34 |
| kevin_b | my first ovs experience was with a commercial app on top of ovs 1.x | 23:34 |
| SamYaple | im sorry for your loss | 23:34 |
| kevin_b | nicira | 23:34 |
| kevin_b | or something like that | 23:34 |
| kevin_b | been so long ago | 23:34 |
| SamYaple | oh im well aware. i was at rackspace when that happened | 23:34 |
| kevin_b | yep Nicira | 23:35 |
| SamYaple | indeed. | 23:35 |
| kevin_b | thing could never scale so we moved on quick | 23:35 |
| SamYaple | went to private cloud and had to use 1.10.2 (which is a dev release BTW) for a long time after | 23:35 |
| SamYaple | then it had such a bad name that no one wanted to use it | 23:36 |
| kevin_b | megaflows made a big difference way back then | 23:36 |
| SamYaple | but i still love my ovs | 23:36 |
| kevin_b | Well thanks again. Gave me some ideas to take back to debugging. | 23:37 |
| SamYaple | +1 | 23:37 |
| kevin_b | i really want to find a solution that doesn't mean turning off interfaces to reboot :( and unplanned reboots are whole nother thing | 23:37 |
| SamYaple | kevin_b: might i suggest disabling neutron-*-agent again? | 23:38 |
| SamYaple | manualyl starting it when youre ready | 23:38 |
| SamYaple | would prevent thisissue unexpectedly happening | 23:38 |
| *** thorst has quit IRC | 23:38 | |
| kevin_b | yeah i think that was the current plan.. a sleep in the agent startup or something goofy. | 23:39 |
| SamYaple | now lets talk about rabbitmq clustering and galera issues! | 23:39 |
| kevin_b | but what doesn't make sense is no flood if we start ovs+agent with interfaces down, but then bring them up. I'd think if it was the agent it would happen when the agent could finally connect out and get the network config.. | 23:40 |
| kevin_b | anyway.. sorry forthe babbling | 23:40 |
| SamYaple | well that does make sense actually | 23:40 |
| SamYaple | if you are willing to lay the blame on neutron | 23:40 |
| SamYaple | because with the interfaces down you can safely lay downthe rules | 23:40 |
| kevin_b | We have those too. Pretty much had to toss of ceilometer cause it was killing rabbit. It'll need its own rabbit cluster. and nasty deadlock in galera in active/active/active configs :( | 23:41 |
| kevin_b | the base rules, but non of the info in neutron db? | 23:41 |
| kevin_b | none | 23:42 |
| SamYaple | kevin_b: im 95% sureall the deadlockissues aresolved my mitaka | 23:42 |
| SamYaple | there was 1 in nova and 1 in neutron in liberty | 23:42 |
| SamYaple | theremight stillbe one in neutron | 23:42 |
| SamYaple | but for all other services active/active til your heart is content | 23:42 |
| kevin_b | we have a race condition we are fighting in cinder too.. weeeee | 23:42 |
| SamYaple | and switch to rabbitmq autoclusterer | 23:42 |
| SamYaple | kevin_b: cinder-volume with shared backend? | 23:42 |
| kevin_b | 3 api hosts for cinder, with haproxy round robin.. makes a mess | 23:43 |
| kevin_b | if someone does something stupid like submit a bunch of attaches in a row | 23:43 |
| SamYaple | whats your cinder backend? | 23:44 |
| kevin_b | for the same volume. That ends up being a manual database cleanup :( | 23:44 |
| kevin_b | solidfire | 23:44 |
| SamYaple | ah yea. lots of those issues are resolved by mitaka timeframe too | 23:44 |
| kevin_b | yeah they chose to go to one api server managed in ha software, i said WTH didn't you just do active/passive/passive in haproxy | 23:45 |
| *** cyborg-one has joined #openstack | 23:45 | |
| SamYaple | yup thats the way you fix that :/ | 23:46 |
| kevin_b | anyway. So things make a lot more sense. Time to write and email while it's in my head. Need to get agent logs from a host started with interfaces down and see what gets layed down | 23:46 |
| SamYaple | good luck again:) | 23:46 |
| kevin_b | yep it's late here time to call it a night. | 23:47 |
| kevin_b | Thanks a lot | 23:47 |
| *** thorst has joined #openstack | 23:49 | |
| *** thorst has quit IRC | 23:49 | |
| *** ianychoi has joined #openstack | 23:56 | |
| *** vishvendra has joined #openstack | 23:58 | |
| *** achadha has joined #openstack | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!