| *** celebdor has quit IRC | 00:16 | |
| *** longkb has joined #openstack-lbaas | 00:43 | |
| openstackgerrit | Jacky Hu proposed openstack/octavia-dashboard master: Remove obsolete gate hooks https://review.openstack.org/593466 | 01:14 |
|---|---|---|
| *** hongbin has joined #openstack-lbaas | 02:04 | |
| sapd1 | johnsom: Are you there? | 02:58 |
| *** gans has joined #openstack-lbaas | 03:16 | |
| *** gans has quit IRC | 03:16 | |
| *** ramishra has joined #openstack-lbaas | 03:39 | |
| *** longkb has quit IRC | 03:55 | |
| *** longkb has joined #openstack-lbaas | 03:56 | |
| *** hongbin has quit IRC | 05:06 | |
| *** longkb has quit IRC | 05:12 | |
| *** longkb has joined #openstack-lbaas | 05:12 | |
| *** yboaron_ has joined #openstack-lbaas | 05:26 | |
| *** oanson has joined #openstack-lbaas | 05:43 | |
| *** ltomasbo has joined #openstack-lbaas | 06:29 | |
| *** hvhaugwitz has quit IRC | 06:40 | |
| *** hvhaugwitz has joined #openstack-lbaas | 06:42 | |
| *** luksky has joined #openstack-lbaas | 06:42 | |
| *** pcaruana has joined #openstack-lbaas | 06:46 | |
| *** ispp has joined #openstack-lbaas | 06:58 | |
| *** rcernin has quit IRC | 07:05 | |
| *** ispp has quit IRC | 07:21 | |
| *** velizarx has joined #openstack-lbaas | 07:25 | |
| *** yboaron_ has quit IRC | 07:27 | |
| *** rpittau has joined #openstack-lbaas | 07:41 | |
| *** celebdor has joined #openstack-lbaas | 07:45 | |
| *** celebdor has quit IRC | 08:06 | |
| *** velizarx has quit IRC | 08:22 | |
| *** ktibi has joined #openstack-lbaas | 08:29 | |
| *** yboaron_ has joined #openstack-lbaas | 08:33 | |
| *** celebdor has joined #openstack-lbaas | 08:34 | |
| *** celebdor has quit IRC | 08:42 | |
| *** luksky has quit IRC | 08:44 | |
| *** velizarx has joined #openstack-lbaas | 08:57 | |
| *** luksky has joined #openstack-lbaas | 09:22 | |
| *** longkb has quit IRC | 10:06 | |
| *** phuoc has joined #openstack-lbaas | 10:08 | |
| *** phuoc_ has quit IRC | 10:10 | |
| *** yboaron_ has quit IRC | 11:07 | |
| *** yboaron_ has joined #openstack-lbaas | 11:07 | |
| *** yboaron_ has quit IRC | 12:03 | |
| *** yboaron_ has joined #openstack-lbaas | 12:23 | |
| openstackgerrit | Kobi Samoray proposed openstack/neutron-lbaas master: nlbaas2octavia: improve member error log message https://review.openstack.org/593610 | 12:40 |
| *** yboaron_ has quit IRC | 13:05 | |
| *** yboaron_ has joined #openstack-lbaas | 13:06 | |
| *** fnaval has joined #openstack-lbaas | 13:21 | |
| openstackgerrit | Kobi Samoray proposed openstack/neutron-lbaas master: nlbaas2octavia: do not change SG owned by user https://review.openstack.org/592471 | 13:32 |
| openstackgerrit | Kobi Samoray proposed openstack/neutron-lbaas master: nlbaas2octavia: improve member error log message https://review.openstack.org/593610 | 13:32 |
| *** ktibi has quit IRC | 13:40 | |
| *** yboaron_ has quit IRC | 14:09 | |
| *** yboaron_ has joined #openstack-lbaas | 14:25 | |
| *** luksky has quit IRC | 15:07 | |
| *** pcaruana has quit IRC | 15:11 | |
| *** velizarx has quit IRC | 15:21 | |
| *** yboaron_ has quit IRC | 15:41 | |
| *** ramishra has quit IRC | 15:44 | |
| *** sapd1 has quit IRC | 16:14 | |
| *** strigazi has quit IRC | 16:19 | |
| *** strigazi has joined #openstack-lbaas | 16:24 | |
| *** dolly has joined #openstack-lbaas | 16:27 | |
| dolly | Hi guys! Great work with octavia! | 16:28 |
| johnsom | Thanks | 16:28 |
| *** KeithMnemonic has joined #openstack-lbaas | 16:28 | |
| cgoncalves | I sense a "but" coming?! :D | 16:29 |
| johnsom | lol, me too | 16:30 |
| *** dolly has quit IRC | 16:32 | |
| *** dolly_ has joined #openstack-lbaas | 16:32 | |
| *** dolly_ is now known as dolly | 16:33 | |
| *** dolly_ has joined #openstack-lbaas | 16:35 | |
| *** velizarx has joined #openstack-lbaas | 16:36 | |
| dolly_ | hm, this is weird. as soon as I type my message, I cant seem to send any new messages. Using the webchat client though. | 16:36 |
| dolly_ | test. | 16:36 |
| dolly_ | hm ok, well did my message about the bug I'm hitting go through ? | 16:37 |
| *** dolly has quit IRC | 16:37 | |
| johnsom | No | 16:39 |
| dolly_ | Hm ok, we run OSP 13 Queens and I think we hit the following bug(s) and I was just curious if anyone had any information on them. I have logs if needed | 16:40 |
| cgoncalves | johnsom, see?! there's always a "but" | 16:41 |
| cgoncalves | dolly_, which bugs? | 16:41 |
| *** dolly has joined #openstack-lbaas | 16:42 | |
| dolly | hm whats going on | 16:42 |
| dolly | as soon as I paste an url | 16:42 |
| dolly | I get disconnected | 16:42 |
| cgoncalves | dolly, are you registered at freenode? could be an anti-spam mech disconnecting you | 16:44 |
| dolly | yeah, well I just registered my nick | 16:44 |
| johnsom | Maybe try http://paste.openstack.org/ | 16:44 |
| cgoncalves | dolly, or just add them to https://etherpad.openstack.org/p/2espbE8KZg | 16:44 |
| *** dolly_ has quit IRC | 16:45 | |
| dolly | ok | 16:45 |
| dolly | pasted them in that etherpad | 16:45 |
| dolly | let me see if I can go through another connection | 16:45 |
| dolly | I didn't see any activity on those bugs and I was just curios | 16:46 |
| cgoncalves | well, and I missed johnsom's comment on storyboard (probably didn't get a notification from storyboard...) | 16:47 |
| *** dolly_ has joined #openstack-lbaas | 16:47 | |
| dolly_ | Alright, lets see now, https://bugzilla.redhat.com/show_bug.cgi?id=1577976 | 16:48 |
| openstack | bugzilla.redhat.com bug 1577976 in openstack-octavia "Disabling heart bit interface for failover- LB goes into error state, New backup amphora is not created" [High,Assigned] - Assigned to nmagnezi | 16:48 |
| dolly_ | Alrighty, this seems to work better. | 16:49 |
| dolly_ | As I was saying, I think we hit that bug. Running OSP 13 / Queens. | 16:49 |
| dolly_ | Great work with Octavia guys! | 16:50 |
| johnsom | Carlos, 1577976 is someone thinking "Role" is a status field and not it's settings. | 16:52 |
| johnsom | The failed failover however is a problem. We would need more information. | 16:52 |
| cgoncalves | johnsom, I know... /o\ | 16:52 |
| johnsom | dolly_ Can you paste us a "openstack subnet show" for your lb-mgmt-net? | 16:52 |
| cgoncalves | I don't have the env I used to report the story | 16:53 |
| dolly_ | sure, http://paste.openstack.org/show/728444/ | 16:56 |
| dolly_ | If you need more logs, I have from octavia with debug=true | 16:56 |
| johnsom | Ok, so DHCP looks good. | 16:57 |
| dolly_ | I should mentioned that we manually merged this -> https://review.openstack.org/#/c/558962/ <- | 16:57 |
| dolly_ | However, I'm not convinced that the patchset is the problem since other seem to have hit the same issue. | 16:58 |
| dolly_ | Can't rule it out though of course. | 16:58 |
| johnsom | I need to go to a meeting, but will look more later. Before it goes into reverting, did the nova instance get an IP on the lb-mgmt-net? Like in "openstack server list" | 16:59 |
| dolly_ | Well from what I can see the amp never got created. | 17:01 |
| dolly_ | I'll go home from work, but will be back in 45-60 minutes. | 17:06 |
| *** dolly has quit IRC | 17:07 | |
| *** dolly_ has quit IRC | 17:12 | |
| *** dolly_ has joined #openstack-lbaas | 17:57 | |
| *** dolly_ is now known as dolly | 18:17 | |
| *** velizarx has quit IRC | 18:23 | |
| *** KeithMnemonic has quit IRC | 18:33 | |
| *** abaindur has joined #openstack-lbaas | 18:50 | |
| dolly | johnsom, did you get any ideas about what could be the issue ? | 19:00 |
| *** luksky has joined #openstack-lbaas | 19:04 | |
| johnsom | dolly Caught me at lunch. | 19:37 |
| dolly | johnsom, interesting note while digging through the logs | 19:37 |
| dolly | johnsom, oh, sorry! | 19:38 |
| johnsom | Oh good | 19:38 |
| dolly | dont want to bother in lunch :p | 19:38 |
| johnsom | Just back from lunch now | 19:38 |
| dolly | well, I cant really see the creation of the amphore | 19:38 |
| dolly | http://paste.openstack.org/show/728457/ | 19:40 |
| dolly | When i grep for the id of the supposedly created compute (amphora) i can only see some GET on the nova-api, and the DELETE. | 19:40 |
| dolly | I would assume that I would see some creation if it was ever created. | 19:41 |
| dolly | I cant see it in the "openstack server list --deleted --all" either (those amphoras listed there are older with other id's). | 19:41 |
| johnsom | Well, when debug logging is on you should see: | 19:43 |
| johnsom | Aug 20 08:44:43.384913 ubuntu-xenial-rax-iad-0001406866 octavia-worker[8112]: DEBUG octavia.controller.worker.tasks.compute_tasks [-] Server created with id: 1cc97ca7-0bf5-4a0d-a879-765a8b41346b for amphora id: e2f2f0ec-1305-403f-8e82-ca2965ffbc6f {{(pid=8604) execute /opt/stack/new/octavia/octavia/controller/worker/tasks/compute_tasks.py:100}} | 19:43 |
| johnsom | What compute driver are you using? controller_worker.compute_driver = compute_nova_driver | 19:44 |
| dolly | That is not set in the config | 19:45 |
| johnsom | dolly That means you are running with the default driver, which is the no-op driver. It doesn't actually boot instances in nova. It's used for testing. You want the amphora_driver, compute_driver, and network_driver defined on worker, health, and housekeeping processes | 19:47 |
| dolly | johnsom, oh! | 19:48 |
| dolly | That is very interesting. | 19:48 |
| johnsom | Here is an example file from our testing gates: http://logs.openstack.org/29/592829/1/check/octavia-v2-dsvm-scenario/d27da70/controller/logs/etc/octavia/octavia_conf.txt.gz | 19:49 |
| dolly | no wait | 19:49 |
| dolly | I lied | 19:49 |
| dolly | I was actually looking for the controller_worker.compute_driver string, rather than under [controller_worker] | 19:50 |
| johnsom | Ah, yeah, I pasted the line from the debug log output | 19:50 |
| cgoncalves | right, it is being set by puppet-octavia | 19:50 |
| dolly | mm I see | 19:51 |
| johnsom | Oh, right, this is OSP 13, so it should be setup for you. | 19:51 |
| dolly | yea, one would think so at least =) | 19:51 |
| cgoncalves | I'd kick myself and our quality engineer if we had not caught that xD | 19:52 |
| dolly | :D | 19:52 |
| jiteka | Hey guys, need a second opinion on this and maybe some advice. When deleting a LB, I get that error | 19:55 |
| jiteka | ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] All attempts to remove security group f680ac35-5990-484d-887b-84f045a1ddf1 have failed.: Conflict: Security Group f680ac35-5990-484d-887b-84f045a1ddf1 in use. | 19:55 |
| jiteka | Which make me think that octavia-worker is actually waiting for the amphora where that secgroup is attached to be deleted before secgroup could be deleted | 19:55 |
| jiteka | but to avoid that error, is there is any config parameter that I could play with to make it more smooth ? | 19:55 |
| jiteka | full logs : http://paste.openstack.org/show/728459/ | 19:55 |
| *** korean101 has quit IRC | 19:55 | |
| dolly | jiteka, are you running a OSP 13 installation ? | 19:56 |
| jiteka | no I'm not running Red hat distro of Openstack | 19:57 |
| johnsom | jiteka Which version of Octavia are you running? | 19:57 |
| cgoncalves | there was a patch some time ago that was supposed to help in that | 19:57 |
| jiteka | Queens | 19:57 |
| jiteka | From source | 19:57 |
| johnsom | Hmmm, yeah, I'm going to do a release today of a new stable/queens. | 19:57 |
| dolly | jiteka, ok. The reason I'm wondering is because I stumbled on that as well. I deleted the ports and the sec-group manually. | 19:57 |
| jiteka | ok | 19:58 |
| johnsom | There were some strange edge cases where nova wasn't letting go of ports that caused some of that | 19:58 |
| jiteka | right now I'm heavily testing failover | 19:58 |
| jiteka | or killing randomely amphora's | 19:58 |
| jiteka | to see how it behave | 19:58 |
| cgoncalves | https://review.openstack.org/#/c/574215/ | 19:59 |
| johnsom | jiteka Yeah, you will want stable/queens master for the most fixes in that area. Or wait for the release later today | 19:59 |
| jiteka | will be an important ops feature to validate before we start getting users on-board | 19:59 |
| jiteka | I'm fine waiting for your release and deploying it tomorrow morning (I'm in EU timezone) | 19:59 |
| johnsom | jiteka Ok, it should be ready in your morning. | 20:00 |
| jiteka | johnsom: what's the easiest way to check when happened the last release ? github last commit ? | 20:01 |
| johnsom | jiteka This page shows the releases and git commit IDs: https://releases.openstack.org/queens/index.html#queens-octavia | 20:01 |
| jiteka | would like to anticipate better to make our CI/CD run more aligned with new push on the branch I'm tracking | 20:01 |
| jiteka | ho ! that's great :-o | 20:02 |
| jiteka | that's what I was looking for :) | 20:02 |
| jiteka | thanks johnsom | 20:02 |
| dolly | johnsom, there is no trace of "Server created with id: 7b4b557a-f677-4c84-ae8c-d9bb2d77b346" in the logs. | 20:12 |
| johnsom | dolly And you see "debug" log messages from Octavia? | 20:12 |
| dolly | So I'm wondering, where did it get the id of the vm from ? If its never created I mean. | 20:12 |
| nmagnezi | johnsom, I see you had some gates + KVM fun yesterday :-) | 20:14 |
| dolly | http://paste.openstack.org/show/728461/ <- johnsom well here's the log | 20:15 |
| johnsom | Yeah, the kernel KVM code regressed again, so had to pull OVH out again | 20:15 |
| johnsom | dolly Ah, so debug is not enabled on that instance, so you won't see the message I pasted. | 20:15 |
| dolly | ah ok, I thought I enabled debug in the octavia.conf | 20:16 |
| dolly | is it multiple debug settings ? | 20:16 |
| johnsom | Ah, there is this line however: 2018-08-17 13:54:21.294 24 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: 06219f4f-6d5b-45c5-beaf-3f55fc4dba84 with compute id 7b4b557a-f677-4c84-ae8c-d9bb2d77b346 for load balancer: 20ff2b24-5ccb-4a89-8c4f-5794f86912b6 | 20:16 |
| johnsom | dolly No, just the one debug = True setting | 20:16 |
| dolly | Well that one is enabled. Hm, maybe I did't restart the containers | 20:17 |
| dolly | Pretty sure I restarted all the containers. | 20:18 |
| dolly | =) | 20:18 |
| johnsom | dolly if debug is enabled you will see a dump of the current configuration settings on startup in the log | 20:19 |
| cgoncalves | dolly, where did you change the configuration file? | 20:19 |
| *** celebdor has joined #openstack-lbaas | 20:20 | |
| cgoncalves | if you modify configs inside containers and restart them, pufff! | 20:20 |
| dolly | yeah no under /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf | 20:20 |
| cgoncalves | perfect | 20:21 |
| *** KeithMnemonic has joined #openstack-lbaas | 20:23 | |
| *** dolly has quit IRC | 20:41 | |
| johnsom | rm_work nmagnezi dayou If you are around there is a translation that could use another +2/+W: https://review.openstack.org/593528 | 20:59 |
| johnsom | stable/queens release patch: https://review.openstack.org/593954 | 21:07 |
| abaindur | johnsom: is there a way we can test tls termination load balancers without barbican? noticed the local_cert_manager is an option in the conf file | 21:10 |
| johnsom | Yeah, I think I posted a comment for you about that last week. Let me find it | 21:10 |
| abaindur | sorry, might have lost my irc connection | 21:10 |
| abaindur | i remember asking | 21:10 |
| johnsom | abaindur https://github.com/openstack/octavia/blob/master/octavia/certificates/manager/local.py#L97-L102 | 21:11 |
| johnsom | Using the "local" manager, you can put your cert info on the local filesystem. | 21:11 |
| johnsom | It's one cert for all load balancers, but should allow testing. | 21:12 |
| johnsom | That said, I'm not sure how well that driver is tested, so you might run into issues. I can't remember if we have a functional gate using the local driver or not | 21:13 |
| abaindur | so this is what the storage_path in certificates section is used for | 21:13 |
| abaindur | otherwise its not used (if we use barbican)? | 21:14 |
| johnsom | I think that is correct | 21:14 |
| abaindur | i see it takes in a cert UUID... but how will our cert have one without barbican? also dont see "store_cert" being called or referenced anyhere else in Octavia | 21:21 |
| abaindur | whats the tls arg i would specify when creating a listener? | 21:23 |
| johnsom | Make up an uuid, name the files with the uuid. Or whatever you put into the listener as the reference | 21:23 |
| abaindur | --default-tls-container=$(openstack secret list | awk '/ tls_secret1 / {print $2}' | 21:23 |
| abaindur | oh so i just need to se the same ID/unique name i want | 21:23 |
| johnsom | =uuid or whatever the filename is | 21:23 |
| abaindur | that will be the arg for --default-tls-container= ? | 21:24 |
| johnsom | Yes | 21:24 |
| johnsom | Store cert never got used | 21:24 |
| abaindur | ah gotcha | 21:24 |
| openstackgerrit | Merged openstack/octavia-dashboard master: Remove obsolete gate hooks https://review.openstack.org/593466 | 21:27 |
| *** rcernin has joined #openstack-lbaas | 21:50 | |
| *** fnaval has quit IRC | 21:58 | |
| *** celebdor has quit IRC | 22:03 | |
| *** luksky has quit IRC | 22:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!