| johnsom | harlowja I'm using the patchset one behind the one you just posted. I'm having problems where the octavia-working isn't stopping after a devstack clean.sh | 00:11 |
|---|---|---|
| harlowja | johnsom hi ho | 00:11 |
| harlowja | hi ho silver | 00:11 |
| harlowja | lol | 00:11 |
| johnsom | Away! | 00:11 |
| harlowja | sooo whats hanging | 00:12 |
| johnsom | oh | 00:12 |
| johnsom | https://www.irccloud.com/pastebin/DTPIVXdS/ | 00:12 |
| johnsom | I've also given it the 60 seconds just to let the graceful timeout fire, still lives on | 00:12 |
| harlowja | ya, that makes me think cotyledon then | 00:13 |
| johnsom | I'm all for "takes a lickin' but keeps on ticking" but.... | 00:13 |
| johnsom | Yeah, this is new after that change | 00:13 |
| johnsom | Strange thing is if I kill -15 the master process it does eventually shutdown | 00:14 |
| harlowja | any idea if u can get where it is locked | 00:14 |
| harlowja | ? | 00:14 |
| johnsom | I would assume that devstack is kill -15 the processes... | 00:15 |
| johnsom | hmmm | 00:15 |
| johnsom | All that does is sit and wait for oslo messages | 00:15 |
| johnsom | The worst part is devstack nukes the log directory so not a lot of output to look at | 00:16 |
| harlowja | durn | 00:17 |
| harlowja | https://github.com/sileht/cotyledon/blob/master/cotyledon/_service_manager.py#L283 would be nice to know if it comes out | 00:17 |
| harlowja | can u try to get the logs :-P | 00:18 |
| johnsom | I am not oslo logging ninja enough to know if I can temp move that log to /tmp or something | 00:18 |
| harlowja | go for it | 00:19 |
| harlowja | i believe in u | 00:19 |
| harlowja | u are the ninja u want to be | 00:19 |
| johnsom | I guess I can hack clean.sh to just not nuke the directory | 00:19 |
| harlowja | ya, need more info about what is waiting to stop | 00:22 |
| johnsom | clean.sh hack in place, waiting for stack to finish to repo | 00:23 |
| johnsom | repo->repro | 00:23 |
| *** armax has quit IRC | 00:30 | |
| openstackgerrit | Jude Cross proposed openstack/python-octaviaclient master: Add Listener commands to client https://review.openstack.org/461231 | 00:32 |
| rm_work | I mean, I think the signal it's listening for might be specifically -9? | 00:32 |
| *** sanfern has quit IRC | 00:32 | |
| johnsom | No, -9 doesn't get to the process, the kernel traps that one and kills without prejudice | 00:33 |
| harlowja | only thing i can think is the fast_exit of cotleddon | 00:34 |
| rm_work | hmm | 00:34 |
| rm_work | then yeah dunno | 00:34 |
| harlowja | https://github.com/sileht/cotyledon/blob/master/cotyledon/_service_manager.py#L313-L324 | 00:34 |
| harlowja | LOG.info(reason) | 00:34 |
| harlowja | u should see that though | 00:34 |
| johnsom | Hmm, I don't see any logs from cotyledon | 00:34 |
| johnsom | except the config and the run service line | 00:34 |
| harlowja | logging configured for that? | 00:34 |
| harlowja | though u should see something :-/ | 00:35 |
| * harlowja doesn't see any special cotledyon stuff in https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L19-L30 | 00:36 | |
| harlowja | so something should pop out i guess | 00:36 |
| johnsom | http://paste.openstack.org/show/608559/ | 00:38 |
| johnsom | That is all I get, dump the config, say I'm running and nothing | 00:38 |
| johnsom | Even when I send it kill -15 | 00:39 |
| harlowja | all i see is config_glue, lol | 00:39 |
| johnsom | Though that does seem to start the 60 second graceful shutdown clock | 00:39 |
| harlowja | does http://paste.openstack.org/raw/608559/ havfe more :-P | 00:41 |
| harlowja | all i see is a bunch of config flue, lol | 00:41 |
| harlowja | glue | 00:41 |
| johnsom | Oh, crumb, that is odd. | 00:41 |
| johnsom | Paste fail. | 00:41 |
| johnsom | Just a second | 00:41 |
| johnsom | Though, 90% of it is config glue | 00:41 |
| johnsom | great, paste is truncating... gist on the way... | 00:43 |
| harlowja | :-/ | 00:43 |
| johnsom | https://gist.github.com/johnsom/c8046b0c3d7e659559383c034dfde67d | 00:43 |
| harlowja | looks ok to me :_P | 00:44 |
| harlowja | lol | 00:44 |
| harlowja | :-P | 00:44 |
| harlowja | except '172.21.21.135:5672 is unreachable' stuff | 00:44 |
| harlowja | how are u stopping that, ctrl-c? | 00:45 |
| johnsom | That's from the ./clean.sh killing rabbit | 00:45 |
| openstackgerrit | Jude Cross proposed openstack/python-octaviaclient master: Add loadbalancer commands to client https://review.openstack.org/454516 | 00:46 |
| JudeC | sorry about the spam just doing some cleanup :) | 00:46 |
| JudeC | load balancer commands look pretty good right now. | 00:47 |
| johnsom | JudeC I have zero problems with progress spam... Grin | 00:48 |
| harlowja | johnsom do u have the kill logs? or was that them ? | 00:50 |
| harlowja | wondering if u are even seeing https://github.com/openstack/octavia/blob/master/octavia/controller/queue/consumer.py#L49 | 00:50 |
| johnsom | That was it... | 00:51 |
| harlowja | oh | 00:51 |
| harlowja | lol | 00:51 |
| johnsom | I do have that in the code I'm running, but yeah, doesn't seem to get there or the logs go into the ehter | 00:52 |
| *** bread has joined #openstack-lbaas | 00:52 | |
| *** bread is now known as aamerine | 00:52 | |
| johnsom | Yeah, just the start, not the stop | 00:53 |
| harlowja | anyway u can add some kind print/output or something to https://github.com/sileht/cotyledon/blob/master/cotyledon/_service_manager.py#L313 | 00:57 |
| harlowja | that is supposed to log a reason, though perhaps its not? | 00:57 |
| johnsom | ^C2017-05-01 17:30:08.024 37227 INFO cotyledon._service_manager [-] Caught SIGIN | 00:59 |
| johnsom | T signal, instantaneous exiting | 00:59 |
| johnsom | I get that log message when I control-c it from the console | 00:59 |
| johnsom | Just not kill -15 | 00:59 |
| johnsom | nor clean.sh | 00:59 |
| harlowja | right SIGTERM is trying to graceful shutdown | 01:00 |
| harlowja | not sure what signal clean.sh is using | 01:01 |
| johnsom | It should use sigterm | 01:02 |
| harlowja | ya, the thing that seems odd is that u aren't seeing https://github.com/sileht/cotyledon/blob/master/cotyledon/_service_manager.py#L255 | 01:02 |
| *** blogan_ has joined #openstack-lbaas | 01:07 | |
| *** blogan has quit IRC | 01:09 | |
| harlowja | i mean lookin at clean | 01:11 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix an failure if the vip network has no gateway https://review.openstack.org/461582 | 01:11 |
| harlowja | it seems to just call into unstack | 01:11 |
| harlowja | like if https://github.com/openstack-dev/devstack/blob/master/unstack.sh#L172-L176 is happening | 01:12 |
| harlowja | then i wonder if that's just shutting things down incorrectly | 01:12 |
| harlowja | but ya, not sure, my devstack foo is weak | 01:12 |
| johnsom | Hmm, wonder if oslo service sees the terminal go away and dies but cotyl doesn' | 01:14 |
| harlowja | possibly | 01:15 |
| * johnsom grasps at straws | 01:16 | |
| johnsom | I think I need a break and will apply head to wall again tomorrow | 01:16 |
| rm_work | yeah would be nice to get the gate working and some of this pending stuff merged in the meantime too | 01:27 |
| openstackgerrit | Jude Cross proposed openstack/python-octaviaclient master: Add Listener commands to client https://review.openstack.org/461231 | 01:27 |
| openstackgerrit | Jude Cross proposed openstack/python-octaviaclient master: Add loadbalancer commands to client https://review.openstack.org/454516 | 01:27 |
| *** JudeC has quit IRC | 01:32 | |
| *** chlong has quit IRC | 01:50 | |
| *** fnaval has joined #openstack-lbaas | 02:33 | |
| *** fnaval has quit IRC | 03:31 | |
| *** gcheresh has joined #openstack-lbaas | 04:00 | |
| *** links has joined #openstack-lbaas | 04:05 | |
| *** fnaval has joined #openstack-lbaas | 04:06 | |
| *** csomerville has joined #openstack-lbaas | 04:11 | |
| *** cody-somerville has quit IRC | 04:14 | |
| *** aojea has joined #openstack-lbaas | 04:19 | |
| *** aojea has quit IRC | 04:23 | |
| *** gcheresh has quit IRC | 04:50 | |
| *** fnaval has quit IRC | 05:20 | |
| *** blogan_ has quit IRC | 05:25 | |
| *** cpuga has quit IRC | 05:33 | |
| *** cpuga has joined #openstack-lbaas | 05:33 | |
| *** cpuga has quit IRC | 05:33 | |
| *** cpuga has joined #openstack-lbaas | 05:34 | |
| *** JudeC has joined #openstack-lbaas | 05:37 | |
| *** cpuga has quit IRC | 05:39 | |
| openstackgerrit | cheng proposed openstack/octavia master: Add monitor address and port to pool member https://review.openstack.org/461649 | 06:19 |
| *** aojea has joined #openstack-lbaas | 06:50 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Extend api to accept qos_policy_id https://review.openstack.org/458308 | 06:51 |
| *** voelzmo has joined #openstack-lbaas | 06:52 | |
| *** pcaruana has joined #openstack-lbaas | 07:06 | |
| *** gcheresh has joined #openstack-lbaas | 07:23 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Extend api to accept qos_policy_id https://review.openstack.org/458308 | 07:30 |
| *** tesseract-RH has joined #openstack-lbaas | 07:30 | |
| *** JudeC has quit IRC | 07:36 | |
| *** gcheresh has quit IRC | 07:59 | |
| openstackgerrit | Merged openstack/octavia master: Fix a bug where ports may not be deleted https://review.openstack.org/431324 | 08:40 |
| *** aojea has quit IRC | 08:43 | |
| *** links has quit IRC | 09:07 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add support for single-create for APIv2 https://review.openstack.org/459032 | 09:08 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Implement sorting and pagination for octavia https://review.openstack.org/382147 | 09:10 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Update devstack to properly configure nova-lxd img https://review.openstack.org/453237 | 09:13 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 09:16 |
| rm_work | ^^ doing some rebasing of older patches that I think are still possibly relevant | 09:16 |
| rm_work | partly just so they'll float to the top of the change list and not get lost, because we need to take them over | 09:17 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: LB/Listener Stats https://review.openstack.org/449243 | 09:18 |
| *** links has joined #openstack-lbaas | 09:25 | |
| *** mdavidson has joined #openstack-lbaas | 09:25 | |
| *** aojea has joined #openstack-lbaas | 09:31 | |
| *** amit213 has quit IRC | 09:43 | |
| *** amitry has quit IRC | 09:43 | |
| *** sindhu has quit IRC | 09:43 | |
| *** dasanind has quit IRC | 09:43 | |
| *** amitry has joined #openstack-lbaas | 09:44 | |
| *** yamamoto has quit IRC | 09:52 | |
| rm_work | hmm | 09:56 |
| rm_work | where are you openstackgerrit | 09:56 |
| rm_work | just submitted 4 patchsets | 09:56 |
| rm_work | ah it must not announce for stable branches | 09:56 |
| openstackgerrit | cheng proposed openstack/octavia master: Add monitor address and port to pool member https://review.openstack.org/461649 | 09:59 |
| *** links has quit IRC | 10:01 | |
| *** amit213 has joined #openstack-lbaas | 10:16 | |
| *** dasanind has joined #openstack-lbaas | 10:17 | |
| *** sindhu has joined #openstack-lbaas | 10:17 | |
| *** links has joined #openstack-lbaas | 10:19 | |
| *** kbyrne has quit IRC | 11:08 | |
| *** kbyrne has joined #openstack-lbaas | 11:12 | |
| *** kbyrne has quit IRC | 11:16 | |
| *** kbyrne has joined #openstack-lbaas | 11:21 | |
| *** b3nt_pin is now known as beagles | 11:24 | |
| *** beagles is now known as Guest93618 | 11:24 | |
| *** Guest93618 is now known as b3nt_pin | 11:25 | |
| *** yamamoto has joined #openstack-lbaas | 11:32 | |
| *** yamamoto_ has joined #openstack-lbaas | 11:33 | |
| *** yamamoto has quit IRC | 11:36 | |
| *** yamamoto_ has quit IRC | 11:58 | |
| *** yamamoto has joined #openstack-lbaas | 11:59 | |
| *** yamamoto has quit IRC | 12:01 | |
| *** gcheresh has joined #openstack-lbaas | 12:03 | |
| *** lunarlamp has joined #openstack-lbaas | 12:06 | |
| *** krypto has joined #openstack-lbaas | 12:42 | |
| *** krypto has joined #openstack-lbaas | 12:42 | |
| xgerman | o/ | 12:43 |
| *** cpuga has joined #openstack-lbaas | 12:43 | |
| *** b3nt_pin is now known as beagles | 12:46 | |
| *** cpuga has quit IRC | 12:48 | |
| *** catintheroof has joined #openstack-lbaas | 12:55 | |
| *** cpuga has joined #openstack-lbaas | 12:56 | |
| *** yamamoto has joined #openstack-lbaas | 13:02 | |
| *** chlong has joined #openstack-lbaas | 13:13 | |
| *** links has quit IRC | 13:20 | |
| *** yamamoto has quit IRC | 13:25 | |
| *** yamamoto has joined #openstack-lbaas | 13:25 | |
| *** openstackgerrit has quit IRC | 13:48 | |
| *** openstackgerrit has joined #openstack-lbaas | 13:54 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Adds status tree to V2 API https://review.openstack.org/459880 | 13:54 |
| *** fnaval has joined #openstack-lbaas | 14:06 | |
| *** krypto has quit IRC | 14:08 | |
| *** voelzmo has quit IRC | 14:16 | |
| *** armax has joined #openstack-lbaas | 14:18 | |
| *** voelzmo has joined #openstack-lbaas | 14:24 | |
| *** voelzmo has quit IRC | 14:29 | |
| *** voelzmo has joined #openstack-lbaas | 14:29 | |
| *** krypto has joined #openstack-lbaas | 14:32 | |
| *** fnaval has quit IRC | 14:32 | |
| *** krypto has quit IRC | 14:37 | |
| *** basilAB has quit IRC | 14:42 | |
| *** vaishali has quit IRC | 14:43 | |
| *** robcresswell has quit IRC | 14:43 | |
| *** dougwig has quit IRC | 14:44 | |
| *** ankur-gupta-f4 has quit IRC | 14:44 | |
| *** redrobot has quit IRC | 14:44 | |
| *** m-greene has quit IRC | 14:45 | |
| *** ctracey has quit IRC | 14:45 | |
| *** jmccrory has quit IRC | 14:45 | |
| *** pcaruana has quit IRC | 14:45 | |
| *** fnaval has joined #openstack-lbaas | 14:46 | |
| *** ptoohill has quit IRC | 14:46 | |
| *** sshank has quit IRC | 14:47 | |
| *** m-greene has joined #openstack-lbaas | 14:47 | |
| *** greghaynes has quit IRC | 14:47 | |
| *** ptoohill has joined #openstack-lbaas | 14:48 | |
| *** ianychoi has quit IRC | 14:49 | |
| *** ianychoi has joined #openstack-lbaas | 14:50 | |
| *** jmccrory has joined #openstack-lbaas | 14:51 | |
| *** pcaruana has joined #openstack-lbaas | 14:51 | |
| *** basilAB has joined #openstack-lbaas | 14:51 | |
| *** sshank has joined #openstack-lbaas | 14:52 | |
| *** redrobot has joined #openstack-lbaas | 14:52 | |
| *** redrobot is now known as Guest92102 | 14:52 | |
| *** krypto has joined #openstack-lbaas | 14:53 | |
| *** krypto has quit IRC | 14:53 | |
| *** Guest12904 has joined #openstack-lbaas | 14:54 | |
| *** dougwig has joined #openstack-lbaas | 14:54 | |
| *** basilAB has quit IRC | 14:56 | |
| *** vaishali has joined #openstack-lbaas | 14:56 | |
| *** basilAB has joined #openstack-lbaas | 14:57 | |
| *** blogan has joined #openstack-lbaas | 14:57 | |
| *** greghaynes has joined #openstack-lbaas | 14:57 | |
| *** gcheresh has quit IRC | 15:01 | |
| johnsom | o/ | 15:01 |
| *** Guest12904 has quit IRC | 15:02 | |
| *** ctracey has joined #openstack-lbaas | 15:04 | |
| johnsom | rm_work Are you around? | 15:05 |
| *** chlong has quit IRC | 15:13 | |
| rm_work | johnsom: yeah | 15:14 |
| rm_work | so | 15:14 |
| rm_work | look at the results of the rechecks on the other stable branch patches | 15:14 |
| rm_work | the tempest ones | 15:14 |
| johnsom | looking | 15:14 |
| rm_work | something that isn't using stable branches (gate maybe?) is failing on LIBS_FROM_GIT for the client | 15:14 |
| rm_work | maybe the correct fix is to hack a bit on the gate side | 15:15 |
| rm_work | but i figured i'd try this and see if it was easier just to add the client bits | 15:15 |
| rm_work | otherwise our stable gates are blocked ATM | 15:15 |
| johnsom | I see | 15:15 |
| johnsom | https://www.irccloud.com/pastebin/R46HddFC/ | 15:15 |
| rm_work | interesting, this is a new one | 15:16 |
| rm_work | http://logs.openstack.org/46/442246/5/check/gate-neutron-lbaasv2-dsvm-api-ubuntu-xenial/d94c6ef/logs/devstacklog.txt.gz#_2017-05-02_09_46_52_153 | 15:16 |
| rm_work | that | 15:16 |
| rm_work | at least for the ocata gate | 15:16 |
| johnsom | Yeah, we need to fix the project-config | 15:17 |
| rm_work | same for newton | 15:17 |
| rm_work | http://logs.openstack.org/47/442247/5/check/gate-neutron-lbaasv2-dsvm-minimal-ubuntu-xenial/b511217/logs/devstacklog.txt.gz#_2017-05-02_09_33_11_131 | 15:17 |
| rm_work | ok | 15:17 |
| johnsom | Installing python-octaviaclient isn't consistent with the stable policy... | 15:17 |
| rm_work | just prevent from adding the devstack project line for the client, on stables? | 15:18 |
| rm_work | ... some stables | 15:18 |
| rm_work | until we get to stable/pike | 15:18 |
| rm_work | then we will need it | 15:18 |
| rm_work | so can't block it on ALL stable branches | 15:18 |
| johnsom | I still don't understand why we can't just do that in the gate hook | 15:20 |
| rm_work | do you have any ideas? I can ask infra, they might have some guidance | 15:20 |
| rm_work | because the gate hook is too late to affect PROJECTS | 15:20 |
| rm_work | because by then it's already checked out everything | 15:20 |
| johnsom | Ok, so remind me why we need this at all in the gate? None of the tests will use the client | 15:22 |
| rm_work | because in order to have OUR devstack plugin install the client | 15:23 |
| rm_work | it has to be installed via gate | 15:23 |
| rm_work | this is a VERY common pattern | 15:23 |
| rm_work | every other project does it | 15:23 |
| rm_work | I will admit it seems like a really weird confluence of crazy | 15:24 |
| rm_work | with regard to the little quirks in how all of this stuff works lining up in a very peculiar way | 15:24 |
| johnsom | Yeah, especially since I haven't had my coffee yet. | 15:24 |
| rm_work | but go look at any other project | 15:24 |
| rm_work | seriously, pick one | 15:24 |
| rm_work | pretty sure all other projects do it | 15:25 |
| rm_work | unless their plugin is in OSC by default maybe | 15:25 |
| rm_work | but even glance does it IIRC | 15:25 |
| rm_work | and magnum and barbican | 15:25 |
| rm_work | let me look at nova really quick | 15:25 |
| rm_work | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L729 | 15:27 |
| rm_work | aah they have a bash-completion thing too | 15:27 |
| rm_work | i think that's what we need to set u | 15:27 |
| rm_work | *set up | 15:27 |
| johnsom | So meet me halfway. Let's do patches that just fix the lines in gate_hook and not install the client. | 15:28 |
| rm_work | right, just not sure how | 15:28 |
| rm_work | just check for the two stable branches we know about | 15:28 |
| rm_work | and override? | 15:28 |
| rm_work | sec | 15:28 |
| *** chlong has joined #openstack-lbaas | 15:30 | |
| johnsom | Nevermind, I know that wont work | 15:30 |
| rm_work | it won't? | 15:30 |
| rm_work | it might | 15:30 |
| rm_work | looking for how to see if the job that's running is for stable | 15:30 |
| *** sanfern has joined #openstack-lbaas | 15:31 | |
| johnsom | You mean conditional in the jenkins job? That would work. I was thinking about the stable branch patches | 15:31 |
| rm_work | ah | 15:31 |
| rm_work | yeah i meant in project-config | 15:31 |
| rm_work | i'm there now | 15:31 |
| rm_work | just looking for the right var to check | 15:31 |
| johnsom | You can just check for the BRANCH_OVERRIDE | 15:31 |
| rm_work | i thought that was for ZUUL | 15:32 |
| rm_work | not for us | 15:32 |
| rm_work | so what will that var look like | 15:32 |
| johnsom | Could be. | 15:32 |
| rm_work | BRANCH_OVERRIDE=stable/ocata | 15:32 |
| rm_work | ? | 15:32 |
| rm_work | I can ask infra | 15:32 |
| johnsom | I'm going to go make coffee real quick | 15:32 |
| rm_work | kk | 15:32 |
| *** amotoki has joined #openstack-lbaas | 15:40 | |
| rm_work | johnsom: https://review.openstack.org/#/c/461830/ | 15:49 |
| johnsom | +1'd | 15:56 |
| rm_work | lol already +A'd | 15:57 |
| johnsom | Sigh, see above, coffee | 15:58 |
| rm_work | :P | 15:58 |
| rm_work | yeah so speaking of | 15:58 |
| rm_work | it's been ... a while since i slept | 15:58 |
| rm_work | so gonna take a 4h or so nap <_< | 15:58 |
| rm_work | feel free to poke any patches I have up | 15:59 |
| rm_work | make changes, do whatever, go nuts ^_^ | 15:59 |
| xgerman | go nuts? Is this an inviation… | 15:59 |
| rm_work | lol | 15:59 |
| *** krypto has joined #openstack-lbaas | 16:01 | |
| *** tesseract-RH has quit IRC | 16:01 | |
| *** pcaruana has quit IRC | 16:02 | |
| *** chlong has quit IRC | 16:04 | |
| *** voelzmo has quit IRC | 16:05 | |
| johnsom | Alright! REvert cotyledon.... | 16:05 |
| johnsom | Grin | 16:05 |
| xgerman | I was thinking rewrite everyhting in Clojure | 16:07 |
| johnsom | Yeah, that will help.... | 16:08 |
| *** csomerville has quit IRC | 16:11 | |
| *** cody-somerville has joined #openstack-lbaas | 16:11 | |
| *** amotoki has quit IRC | 16:21 | |
| *** csomerville has joined #openstack-lbaas | 16:23 | |
| *** cody-somerville has quit IRC | 16:26 | |
| *** blogan has quit IRC | 16:30 | |
| *** aojea has quit IRC | 16:31 | |
| *** robcresswell has joined #openstack-lbaas | 16:43 | |
| *** ankur-gupta-f4 has joined #openstack-lbaas | 16:45 | |
| *** blogan has joined #openstack-lbaas | 16:45 | |
| *** harlowja has quit IRC | 17:04 | |
| *** JudeC has joined #openstack-lbaas | 17:11 | |
| *** harlowja has joined #openstack-lbaas | 18:00 | |
| harlowja | did u guys figure out the cotyledon or whatever stuff? | 18:01 |
| harlowja | johnsom ? | 18:01 |
| johnsom | Hi, no still broken | 18:01 |
| harlowja | i'm sure medhi would like to know whatever u find out, especially if its a behavior difference | 18:01 |
| harlowja | just need to get some more logs, lol | 18:02 |
| harlowja | somebody took yer logs | 18:02 |
| harlowja | lol | 18:02 |
| *** chlong has joined #openstack-lbaas | 18:03 | |
| *** gcheresh has joined #openstack-lbaas | 18:05 | |
| johnsom | Yeah, I need to play with it more. | 18:06 |
| johnsom | Working on summit slides at the moment | 18:06 |
| johnsom | And working with Adam to un-break the stable branches.... grin | 18:07 |
| *** voelzmo has joined #openstack-lbaas | 18:09 | |
| *** aojea has joined #openstack-lbaas | 18:11 | |
| *** csomerville has quit IRC | 18:13 | |
| *** aojea has quit IRC | 18:15 | |
| *** voelzmo has quit IRC | 18:15 | |
| *** voelzmo has joined #openstack-lbaas | 18:16 | |
| harlowja | k | 18:23 |
| *** krypto has quit IRC | 18:28 | |
| *** csomerville has joined #openstack-lbaas | 18:43 | |
| *** csomerville has quit IRC | 18:43 | |
| *** csomerville has joined #openstack-lbaas | 18:44 | |
| *** csomerville has quit IRC | 18:54 | |
| *** cody-somerville has joined #openstack-lbaas | 18:58 | |
| *** voelzmo has quit IRC | 19:01 | |
| *** ianychoi_ has joined #openstack-lbaas | 19:10 | |
| *** ianychoi has quit IRC | 19:13 | |
| *** ianychoi_ is now known as ianychoi | 19:19 | |
| *** armax has quit IRC | 19:30 | |
| *** voelzmo has joined #openstack-lbaas | 19:36 | |
| *** aojea has joined #openstack-lbaas | 19:50 | |
| *** cody-somerville has quit IRC | 19:52 | |
| *** voelzmo_ has joined #openstack-lbaas | 19:56 | |
| *** voelzmo has quit IRC | 19:58 | |
| *** voelzmo_ has quit IRC | 20:01 | |
| rm_work | lolwut: http://logs.openstack.org/03/445003/1/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/aaeb962/console.html#_2017-05-02_09_49_06_515882 | 20:10 |
| rm_work | whatever | 20:10 |
| rm_work | johnsom: want me to look at the cotyledon thing? | 20:10 |
| *** voelzmo has joined #openstack-lbaas | 20:12 | |
| *** voelzmo has quit IRC | 20:13 | |
| *** cody-somerville has joined #openstack-lbaas | 20:16 | |
| *** cpuga has quit IRC | 20:27 | |
| *** blogan_ has joined #openstack-lbaas | 20:27 | |
| *** cody-somerville has quit IRC | 20:28 | |
| *** JudeC has quit IRC | 20:29 | |
| *** blogan has quit IRC | 20:30 | |
| *** Guest75 has joined #openstack-lbaas | 20:30 | |
| *** cody-somerville has joined #openstack-lbaas | 20:37 | |
| *** JudeC has joined #openstack-lbaas | 20:47 | |
| *** armax has joined #openstack-lbaas | 20:49 | |
| *** cody-somerville has quit IRC | 20:55 | |
| *** cody-somerville has joined #openstack-lbaas | 20:57 | |
| *** cody-somerville has quit IRC | 20:57 | |
| *** cody-somerville has joined #openstack-lbaas | 20:57 | |
| *** cody-somerville has quit IRC | 20:58 | |
| *** cody-somerville has joined #openstack-lbaas | 20:58 | |
| *** chlong has quit IRC | 21:01 | |
| *** catintheroof has quit IRC | 21:01 | |
| rm_work | johnsom: ok this one is passing: https://review.openstack.org/#/c/442246/ | 21:02 |
| *** blogan_ has quit IRC | 21:02 | |
| xgerman | let’s see if it progresses further | 21:08 |
| *** chlong has joined #openstack-lbaas | 21:16 | |
| *** gcheresh has quit IRC | 21:18 | |
| rm_work | fixed the newton one i think | 21:21 |
| rm_work | https://review.openstack.org/#/c/442247/ | 21:21 |
| rm_work | hopefully should pass now | 21:21 |
| johnsom | sudo not found, that is nice | 21:23 |
| *** dougwig has quit IRC | 21:25 | |
| *** cody-somerville has quit IRC | 21:25 | |
| rm_work | yeah IDKWTF | 21:25 |
| rm_work | might just recheck | 21:25 |
| johnsom | Our luck, someone posted a new cirros image | 21:26 |
| johnsom | Or devstack has played with the glance images more and it's picking yet a different image now | 21:27 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Switch to use stable remote_client https://review.openstack.org/445003 | 21:28 |
| johnsom | Oh, right, we hard coded the image to work around that devstack issue | 21:28 |
| rm_work | gonna rebase it anyway cause whatever | 21:28 |
| rm_work | johnsom: how do i repro this cotyledon thing | 21:31 |
| rm_work | got a devstack up | 21:31 |
| rm_work | ctrl-c seems to shit it down perfectly | 21:31 |
| johnsom | I do ./clean.sh | 21:31 |
| rm_work | yes, I see the typo, and I refuse to correct it | 21:31 |
| johnsom | haha | 21:31 |
| rm_work | lol well, what does clean.sh do? | 21:31 |
| rm_work | i don't want to clean.sh my devstack | 21:31 |
| johnsom | /opt/stack/devstack/clean.sh burns down the devstack | 21:32 |
| rm_work | that's what "revert to snapshot" or "openstack server delete" is for | 21:32 |
| johnsom | after it is done you should be able to ps -ef | grep stack and see no processes still running under the stack user | 21:32 |
| rm_work | ok so if this is ONLY a problem with clean.sh | 21:32 |
| rm_work | why are we even caring | 21:32 |
| rm_work | i thought this was something that could actually happen in a deployment | 21:33 |
| johnsom | Well, I use it, users use it. If you do this and then startup stack again, your lbaas requests go into the ether. | 21:33 |
| johnsom | Which becomes a support headache for said PTL | 21:33 |
| rm_work | ugh | 21:34 |
| rm_work | if you use clean.sh and startup stack again, you're 99% sure to get some other bug too | 21:34 |
| johnsom | Well, I think this will crop up with various init script setups too | 21:34 |
| johnsom | No, I use it all the time | 21:34 |
| rm_work | the whole idea of re-doing a stack on the same VM is bad | 21:34 |
| *** aojea has quit IRC | 21:35 | |
| johnsom | So, a test case that makes you slightly less crazy/rage-y at me. | 21:35 |
| rm_work | hmm, kill -15 works | 21:36 |
| johnsom | If you kill -15 the controller worker master process, why does it not stop it until after the 60 second "graceful shutdown timeout" expires? | 21:36 |
| johnsom | Did kill -15 work right way or did it have to wait 60 seconds? | 21:36 |
| rm_work | I think that's how cotyledon works | 21:36 |
| rm_work | it has a graceful thing | 21:37 |
| johnsom | Yeah, but it's not gracefully stopping, it's timing out and killing it | 21:37 |
| rm_work | 2017-05-03 06:35:40.195 25974 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process | 21:37 |
| rm_work | 2017-05-03 06:35:40.195 25984 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [25984] | 21:37 |
| rm_work | 2017-05-03 06:35:40.195 25986 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(1) [25986] | 21:37 |
| rm_work | 2017-05-03 06:35:40.196 25984 INFO octavia.controller.queue.consumer [-] Stopping consumer... | 21:37 |
| rm_work | 2017-05-03 06:35:40.196 25986 INFO octavia.controller.queue.consumer [-] Stopping consumer... | 21:37 |
| johnsom | Both of these behaviors started with this swtich. They both worked with oslo server | 21:37 |
| johnsom | service | 21:37 |
| rm_work | that's with kill -15 | 21:37 |
| rm_work | looks like it all shut down nicely | 21:37 |
| rm_work | didn't take long either | 21:38 |
| johnsom | Mine does not do that | 21:38 |
| rm_work | how long are you waiting after starting it to kill it | 21:38 |
| rm_work | i wonder if there's an initial period or something | 21:38 |
| johnsom | How are you starting o-cw? | 21:39 |
| johnsom | screen? | 21:39 |
| rm_work | devstack default | 21:39 |
| johnsom | Yeah, it's up a bit | 21:39 |
| *** cpuga has joined #openstack-lbaas | 21:39 | |
| rm_work | i just went over to the screen it was on | 21:39 |
| johnsom | Yeah, well, devstack default just changed.... | 21:39 |
| johnsom | screen | 21:39 |
| rm_work | i just redid my stack | 21:39 |
| rm_work | and i seem to still be on screen? | 21:40 |
| johnsom | Yeah, I could control-c no problem, but if I kill -15 it doesn't do the graceful | 21:40 |
| rm_work | if I ctrl-c it does the SIGINT instant exit | 21:40 |
| rm_work | which actually does look like it has an error? | 21:40 |
| rm_work | 2017-05-03 06:40:20.855 26953 INFO cotyledon._service_manager [-] Caught SIGINT signal, instantaneous exiting | 21:40 |
| rm_work | o-cw failed to start. Exit code: 1 | 21:40 |
| rm_work | oh is that second thing a different line | 21:40 |
| rm_work | output by the other bit on the console | 21:40 |
| rm_work | derp | 21:40 |
| rm_work | yep | 21:41 |
| rm_work | k nm works fine | 21:41 |
| rm_work | ah hold on that time it took like 10 seconds | 21:41 |
| rm_work | after the "Stopping consumer..." lines | 21:41 |
| johnsom | I don't get "Caught SIGTERM signal, graceful exiting of service ConsumerService" nor "Stopping consumer..." | 21:42 |
| rm_work | but at that point we're inside OUR code | 21:42 |
| rm_work | huh | 21:42 |
| rm_work | weird | 21:42 |
| rm_work | i wonder if clean.sh is busted | 21:43 |
| johnsom | restacking to get it back | 21:43 |
| rm_work | what signal is it sending | 21:43 |
| xgerman | don’t go that far | 21:43 |
| johnsom | I don't get it by just kill -15, no clean.sh | 21:43 |
| rm_work | ugh not like that'll tell you anything, restacks are inherently broken | 21:43 |
| rm_work | so tired of debugging shit because a restack did something dumb | 21:43 |
| johnsom | rm_work no, I restack (via clean.sh/stack.sh) all of the time | 21:43 |
| rm_work | to find out it's not a real problem | 21:44 |
| rm_work | johnsom: maybe that explains why you are having this weird problem? :P | 21:44 |
| rm_work | I mean | 21:44 |
| rm_work | just sayin' | 21:44 |
| johnsom | Grrrr, it worked fine with oslo service.... | 21:44 |
| rm_work | i stopped having dumb problems once I stopped using restack and just use a fresh VM every time | 21:45 |
| rm_work | i don't even want to think about all the time I wasted before I switched to that workflo | 21:45 |
| johnsom | I don't use restack.sh or rejoin.sh (both of which I don't think are a thing anymore | 21:45 |
| xgerman | yeah, rejoin is gone | 21:45 |
| xgerman | clean.sh/stack.sh | 21:46 |
| rm_work | hmm | 21:46 |
| rm_work | I always used unstack.sh and stack.sh | 21:46 |
| johnsom | Yeah, unstack.sh doesn't work reliabily | 21:46 |
| rm_work | maybe clean.sh does a better job? | 21:46 |
| rm_work | but i still don't really trust it | 21:46 |
| rm_work | yeah i dunno, kill -15 seems to be working super reliably here | 21:50 |
| johnsom | Yeah, I will test again in about 10 minutes | 21:50 |
| rm_work | if yours is systemd, maybe it's a bug with the systemd stuff? | 21:50 |
| johnsom | No, I'm on the old screen | 21:50 |
| johnsom | I am 99.9% sure it's that oslo.service dies when the tty goes away and cylon doesn't. Frankly, we probably need to fix this: | 21:53 |
| johnsom | https://www.irccloud.com/pastebin/mzRYSfqY/ | 21:53 |
| johnsom | Which makes me wonder, did we disable the service in apache with the wsgi patch...... | 21:53 |
| johnsom | Sigh | 21:53 |
| rm_work | hmm maybe not? | 21:54 |
| johnsom | Oh, yes we did.. good | 21:54 |
| rm_work | k | 21:54 |
| rm_work | excellent ;P | 21:54 |
| johnsom | Ok, stacked. Let me try something... | 21:55 |
| * rm_work lets johnsom try something | 21:56 | |
| rm_work | but just this once! | 21:56 |
| johnsom | Yeah, ok | 22:01 |
| johnsom | So, if you kill -15 while it is happy in screen it stops ok | 22:01 |
| johnsom | However, while in screen it's parent is bash | 22:01 |
| johnsom | When clean runs, it does a "screen -X -S $SESSION quit" | 22:01 |
| johnsom | Which all of the other octavia process then exit | 22:02 |
| johnsom | BUT! | 22:02 |
| johnsom | octavia-worker does not and it's parent process is now 1 | 22:02 |
| johnsom | At this point kill -15 doesn't work right | 22:02 |
| johnsom | I think we just need to fix the stop function | 22:03 |
| *** cpuga has quit IRC | 22:06 | |
| openstackgerrit | Jude Cross proposed openstack/python-octaviaclient master: Add Listener commands to client https://review.openstack.org/461231 | 22:09 |
| *** cpuga has joined #openstack-lbaas | 22:13 | |
| openstackgerrit | Merged openstack/octavia master: Fix an failure if the vip network has no gateway https://review.openstack.org/461582 | 22:18 |
| *** cody-somerville has joined #openstack-lbaas | 22:20 | |
| *** cody-somerville has quit IRC | 22:20 | |
| *** cody-somerville has joined #openstack-lbaas | 22:20 | |
| *** chlong has quit IRC | 22:22 | |
| rm_work | here's the next one! http://logs.openstack.org/47/442247/9/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/e1c9a46/console.html#_2017-05-02_22_18_31_826766 | 22:37 |
| rm_work | lol | 22:37 |
| johnsom | This one we have merged a fix for, it must just not be backported | 22:38 |
| *** fnaval has quit IRC | 22:38 | |
| johnsom | Ok, we are idiots sometimes.... Wait until you see the fix for this cylon issue | 22:47 |
| johnsom | Yes, I'm going to call it cylon | 22:47 |
| *** fnaval has joined #openstack-lbaas | 22:48 | |
| rm_work | lol | 22:48 |
| *** fnaval has quit IRC | 22:49 | |
| *** fnaval has joined #openstack-lbaas | 22:50 | |
| rm_work | looking for the list_networks fix | 22:57 |
| *** cpuga has quit IRC | 22:57 | |
| rm_work | was it in... the other stable branch? | 23:01 |
| rm_work | https://review.openstack.org/#/c/438112/ | 23:06 |
| rm_work | found it | 23:06 |
| rm_work | cherry-picking | 23:06 |
| johnsom | Yep, that one | 23:06 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix devstack plugin to stop octavia processes https://review.openstack.org/461927 | 23:10 |
| johnsom | Read and weep ^^^^ | 23:10 |
| rm_work | OK | 23:13 |
| rm_work | got it i think | 23:13 |
| rm_work | cherry-pick: https://review.openstack.org/#/c/461926/ | 23:13 |
| rm_work | and rebased the other on top | 23:13 |
| rm_work | umm | 23:13 |
| rm_work | I wonder if they need to be merged together to actually pass >_< | 23:13 |
| rm_work | :/ | 23:13 |
| rm_work | we'll see | 23:13 |
| rm_work | johnsom: wait what about the stop_process for all the other stuff | 23:14 |
| johnsom | What other stuff that doesn't have it's own devstack plugin? | 23:14 |
| rm_work | oh | 23:14 |
| rm_work | sorry i read that backwards | 23:15 |
| rm_work | you added it, not removed it | 23:15 |
| rm_work | k | 23:15 |
| rm_work | yep | 23:15 |
| rm_work | looks right | 23:15 |
| rm_work | did we NEVER have those? | 23:15 |
| johnsom | Yeah, it actually works too | 23:15 |
| johnsom | Nope | 23:16 |
| rm_work | lol | 23:16 |
| rm_work | because no one cares about cleanup | 23:16 |
| johnsom | You really don't want to look at the history for that function either.... At one point it was killing haproxy processes.... | 23:16 |
| johnsom | sigh | 23:16 |
| rm_work | <_< | 23:16 |
| * rm_work doesn't care about cleanup | 23:17 | |
| rm_work | i care so little, I'm +2ing that without testing it, because it looks right... | 23:17 |
| rm_work | and I trust you, the one person ever who tried to use the stop function :P | 23:17 |
| johnsom | https://github.com/openstack/octavia/blame/ecfca348904560c341e84644137b9c0aa09b3949/devstack/plugin.sh#L213 | 23:18 |
| *** catintheroof has joined #openstack-lbaas | 23:23 | |
| *** catintheroof has quit IRC | 23:23 | |
| *** catintheroof has joined #openstack-lbaas | 23:24 | |
| *** Guest75 has quit IRC | 23:26 | |
| *** catintheroof has quit IRC | 23:29 | |
| *** catintheroof has joined #openstack-lbaas | 23:31 | |
| johnsom | rm_work FYI, I am going fix the stats patch | 23:34 |
| rm_work | kk | 23:34 |
| rm_work | flying tomorrow, so i'll be out the whole day | 23:41 |
| rm_work | unless i get bored briefly in LAX | 23:41 |
| johnsom | Ok | 23:48 |
| johnsom | Hey, on the project ID thing | 23:48 |
| johnsom | I swear I commented on it, but we can discuss now if you want | 23:48 |
| rm_work | sure | 23:51 |
| rm_work | it didn't occur to me before that it's still super useful on responses | 23:52 |
| rm_work | until i thought about getting a response without it when querying like | 23:52 |
| rm_work | pools | 23:52 |
| rm_work | like sure, you should know who you are | 23:53 |
| rm_work | but | 23:53 |
| rm_work | it's pretty standard to RETURN on every object | 23:53 |
| rm_work | across all of openstack | 23:53 |
| johnsom | Yeah, I have no problem with the return. However, I would prefer we leave the project_id in the tables | 23:54 |
| rm_work | wut | 23:55 |
| rm_work | why | 23:55 |
| johnsom | It will make queries much easier/faster | 23:55 |
| rm_work | hmm | 23:55 |
| rm_work | i guess if you want | 23:55 |
| johnsom | That or we do foreign key columns | 23:55 |
| rm_work | i don't honestly care | 23:55 |
| rm_work | i just thought removing it was the plan | 23:55 |
| *** yamamoto has quit IRC | 23:56 | |
| johnsom | I'm just thinking of the "get_all" case | 23:56 |
| johnsom | Actually any of the gets we are going to filter by project id | 23:57 |
| rm_work | hmm k | 23:58 |
| *** catintheroof has quit IRC | 23:59 | |
| rm_work | i can undo the part that pulls the project_id from the highest level object | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!