| rm_work | yeah i saw | 00:12 |
|---|---|---|
| johnsom | Well, DNS puked so rechecking that too | 00:12 |
| rm_work | T_T | 00:13 |
| rm_work | https://review.openstack.org/#/c/474711/ has the 404 error, lol | 00:16 |
| rm_work | so no, not fixed | 00:17 |
| rm_work | i wonder if moving to webob from the shitty flask stuff actually somehow changed the way gunicorn loads the app to no longer 404? lol | 00:17 |
| rm_work | maybe it was a bug in flask's reponse objects >_> | 00:17 |
| eandersson | :'( | 00:17 |
| johnsom | Umm, there is no depends there | 00:17 |
| johnsom | So, that doesn't prove the fix doesn't work | 00:18 |
| openstackgerrit | Merged openstack/octavia master: Enable DIB trace logging https://review.openstack.org/474800 | 00:18 |
| *** wasmum has quit IRC | 00:19 | |
| *** KeithMnemonic has quit IRC | 00:20 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add RBAC enforcement to Octavia v2 API https://review.openstack.org/472872 | 00:22 |
| rm_work | johnsom: no, i mean it proves it wasn't magically fixed yesterday or something | 00:22 |
| rm_work | i still think MY fix works somehow (whether it's the switch to webob, or the json stuff) | 00:22 |
| johnsom | Ah, yeah | 00:22 |
| rm_work | http://logs.openstack.org/90/474790/6/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/993232f/console.html#_2017-06-16_22_39_24_608859 | 00:22 |
| johnsom | Yeah, I'm starting to feel that way too. Just about to point of just trying to merge it | 00:23 |
| rm_work | what is going on THERE | 00:23 |
| rm_work | just a normal "nova didn't boot a VM right" issue? | 00:23 |
| rm_work | well that won't happen until the mkfs thing merges, rofl | 00:23 |
| rm_work | which won't happen until DNS stops breaking | 00:23 |
| rm_work | fffffffff | 00:23 |
| johnsom | Well, when the test fails with that I check the o-cw log to see why it wasn't up | 00:24 |
| johnsom | <Response [404]> | 00:24 |
| johnsom | Wait, that was patch 6??? | 00:25 |
| johnsom | http://logs.openstack.org/90/474790/6/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/993232f/logs/screen-o-cw.txt.gz#_Jun_16_22_38_29_774571 | 00:25 |
| rm_work | wait where's a 404 | 00:26 |
| rm_work | I don't see that | 00:26 |
| johnsom | http://logs.openstack.org/90/474790/6/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/993232f/logs/screen-o-cw.txt.gz#_Jun_16_22_38_31_252277 | 00:27 |
| rm_work | oh | 00:27 |
| rm_work | uhh | 00:27 |
| johnsom | I have no idea what sweet heck happened there.... | 00:27 |
| rm_work | ummm | 00:28 |
| johnsom | Though raising that ConnectionError exception will pop to the outer loop, which is sill just two retries right???? | 00:28 |
| johnsom | So it was like two 111 connection refused and then a 404 | 00:29 |
| johnsom | I think | 00:29 |
| johnsom | Though I'm really tired at the moment | 00:29 |
| rm_work | ummm | 00:29 |
| rm_work | soooo | 00:29 |
| rm_work | i think what happened | 00:29 |
| rm_work | is it does that debug message on 404 | 00:29 |
| rm_work | on line 279/280 | 00:29 |
| rm_work | then it would have hit my code on 283 | 00:30 |
| rm_work | and it DIDN'T enter the loop!? | 00:30 |
| rm_work | so it *returned* | 00:30 |
| rm_work | it should have entered the loop I think | 00:30 |
| rm_work | ok i'm gonna do a thing | 00:30 |
| rm_work | more debug time | 00:31 |
| rm_work | I'm gonna have so many messages | 00:31 |
| johnsom | Make octavia great! | 00:32 |
| rm_work | ok only one extra | 00:38 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Agent: swap flask responses to webob, handle 404 retries better https://review.openstack.org/474790 | 00:38 |
| rm_work | at least we'll see more data about the connection on a 404 | 00:38 |
| *** yamamoto has joined #openstack-lbaas | 00:38 | |
| rm_work | rofl it failed again johnsom (mkfs patch) | 00:39 |
| *** links has joined #openstack-lbaas | 00:39 | |
| johnsom | As soon as I finish this RBAC testing I am applying beer to the day | 00:40 |
| rm_work | lol | 00:40 |
| rm_work | i need a bot that does a recheck every time this patch completes | 00:40 |
| *** cpuga has quit IRC | 00:48 | |
| *** cpuga has joined #openstack-lbaas | 00:49 | |
| *** cpuga has quit IRC | 00:53 | |
| *** yamamoto has quit IRC | 00:56 | |
| *** blogan has quit IRC | 00:57 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add RBAC enforcement to Octavia v2 API https://review.openstack.org/472872 | 01:06 |
| *** JudeC has quit IRC | 01:23 | |
| *** yamamoto has joined #openstack-lbaas | 01:56 | |
| *** yamamoto has quit IRC | 02:01 | |
| *** eanderson_ has joined #openstack-lbaas | 02:22 | |
| eanderson_ | rm_work, I am as close as you will get to a recheck bot :D | 02:23 |
| rm_work | lol | 02:23 |
| rm_work | thx | 02:23 |
| eanderson_ | diskbuilder failed on gate again though :( | 02:23 |
| eanderson_ | I wish you could recheck before it even finished | 02:23 |
| *** eanderson_ is now known as eandersson_ | 02:25 | |
| rm_work | T_T | 02:25 |
| rm_work | i need MY patch to have a 404 happen | 02:26 |
| rm_work | i've been going through each logs/screen-o-cw.txt.gz by hand and looking for "Got a 404" | 02:27 |
| rm_work | for the scenarios | 02:27 |
| rm_work | 7 scenarios for each run | 02:27 |
| rm_work | need it to catch a 404 to show me what's happening | 02:27 |
| *** ianychoi has quit IRC | 03:18 | |
| *** ianychoi has joined #openstack-lbaas | 03:19 | |
| *** yamamoto has joined #openstack-lbaas | 03:27 | |
| openstackgerrit | Merged openstack/octavia master: Small refactor for load_balancer v2 vip validation https://review.openstack.org/474711 | 03:28 |
| *** yamamoto has quit IRC | 03:31 | |
| *** SumitNaiksatam has quit IRC | 03:37 | |
| *** SumitNaiksatam has joined #openstack-lbaas | 03:38 | |
| *** krypto has joined #openstack-lbaas | 03:39 | |
| *** krypto has quit IRC | 03:43 | |
| *** krypto has joined #openstack-lbaas | 03:43 | |
| *** yamamoto has joined #openstack-lbaas | 04:14 | |
| *** yamamoto has quit IRC | 04:14 | |
| *** yamamoto has joined #openstack-lbaas | 04:58 | |
| *** yamamoto_ has joined #openstack-lbaas | 05:05 | |
| *** yamamoto has quit IRC | 05:08 | |
| *** armax has quit IRC | 05:25 | |
| *** yamamoto_ has quit IRC | 05:26 | |
| *** SumitNaiksatam_ has joined #openstack-lbaas | 05:43 | |
| *** SumitNaiksatam has quit IRC | 05:44 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 05:44 | |
| *** eandersson_ has quit IRC | 06:26 | |
| *** yamamoto has joined #openstack-lbaas | 06:27 | |
| *** yamamoto has quit IRC | 06:44 | |
| *** tesseract has joined #openstack-lbaas | 07:01 | |
| rm_work | LOL the current run hit the mkfs bug THREE TIMES | 07:50 |
| rm_work | somehow | 07:50 |
| rm_work | it started ... 50 minutes ago | 07:50 |
| rm_work | the merge was 67 minutes ago? | 07:50 |
| rm_work | not sure why it's still hitting it | 07:50 |
| rm_work | but the debug logs don't show it running the new command, so... ??? | 07:51 |
| rm_work | hopefully the next run won't break | 07:51 |
| *** kobis has joined #openstack-lbaas | 08:02 | |
| *** gtrxcb has joined #openstack-lbaas | 08:12 | |
| *** rcernin has joined #openstack-lbaas | 08:25 | |
| *** kobis has quit IRC | 08:37 | |
| *** gtrxcb has quit IRC | 08:47 | |
| *** kobis has joined #openstack-lbaas | 08:48 | |
| *** links has quit IRC | 09:38 | |
| *** yamamoto has joined #openstack-lbaas | 10:24 | |
| *** krypto has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** yamamoto has joined #openstack-lbaas | 11:25 | |
| *** yamamoto has quit IRC | 11:34 | |
| *** sanfern has joined #openstack-lbaas | 12:13 | |
| *** yamamoto has joined #openstack-lbaas | 12:16 | |
| *** yamamoto has quit IRC | 13:01 | |
| *** jerrygb has joined #openstack-lbaas | 13:22 | |
| *** sanfern has quit IRC | 13:56 | |
| *** sanfern has joined #openstack-lbaas | 13:58 | |
| *** jerrygb has quit IRC | 13:59 | |
| *** yamamoto has joined #openstack-lbaas | 14:01 | |
| *** yamamoto has quit IRC | 14:12 | |
| *** yamamoto has joined #openstack-lbaas | 14:49 | |
| *** yamamoto has quit IRC | 15:16 | |
| *** raginbajin has quit IRC | 15:33 | |
| *** raginbajin has joined #openstack-lbaas | 15:43 | |
| *** rcernin has quit IRC | 15:44 | |
| *** cpuga has joined #openstack-lbaas | 15:56 | |
| *** cpuga has quit IRC | 16:00 | |
| *** yamamoto has joined #openstack-lbaas | 16:17 | |
| *** catintheroof has joined #openstack-lbaas | 16:21 | |
| *** rcernin has joined #openstack-lbaas | 16:23 | |
| *** yamamoto has quit IRC | 16:24 | |
| *** ipsecguy_ has joined #openstack-lbaas | 16:32 | |
| *** ipsecguy has quit IRC | 16:33 | |
| *** jerrygb has joined #openstack-lbaas | 16:37 | |
| *** jerrygb has quit IRC | 16:37 | |
| *** catintheroof has quit IRC | 16:37 | |
| *** krypto has joined #openstack-lbaas | 16:39 | |
| *** krypto has quit IRC | 16:39 | |
| *** krypto has joined #openstack-lbaas | 16:39 | |
| *** tesseract has quit IRC | 16:56 | |
| *** openstackgerrit has quit IRC | 17:18 | |
| *** eandersson_ has joined #openstack-lbaas | 17:34 | |
| johnsom | Well, the mkfs fix is in the runs now, so one victory | 17:48 |
| johnsom | Ok, that is strange, it's in some of the runs... ???? | 17:49 |
| johnsom | I bet the lbaas jobs are still pulling the package... That is why | 17:50 |
| *** cpuga has joined #openstack-lbaas | 17:57 | |
| *** cpuga has quit IRC | 18:02 | |
| *** kobis has quit IRC | 18:13 | |
| *** gcheresh has joined #openstack-lbaas | 18:20 | |
| *** pcaruana has joined #openstack-lbaas | 18:24 | |
| *** pcaruana has quit IRC | 18:25 | |
| *** eandersson_ has quit IRC | 18:34 | |
| *** ctracey has quit IRC | 18:48 | |
| *** sbalukoff_ has quit IRC | 18:48 | |
| *** amitry has quit IRC | 18:48 | |
| *** sbalukoff_ has joined #openstack-lbaas | 18:49 | |
| *** amitry has joined #openstack-lbaas | 18:49 | |
| *** ctracey has joined #openstack-lbaas | 18:50 | |
| *** krypto has quit IRC | 19:05 | |
| *** armax has joined #openstack-lbaas | 19:31 | |
| *** armax has quit IRC | 19:31 | |
| *** armax has joined #openstack-lbaas | 19:40 | |
| *** openstackgerrit has joined #openstack-lbaas | 20:04 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add RBAC enforcement to Octavia v2 API https://review.openstack.org/472872 | 20:04 |
| johnsom | http://logs.openstack.org/90/474790/7/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/521d1b8/logs/screen-o-cw.txt.gz#_Jun_17_19_54_24_257344 | 20:06 |
| johnsom | SoB: https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/plug.py#L224 | 20:10 |
| johnsom | So... It's probably the same issue as the mkfs problem | 20:10 |
| johnsom | The OS has not yet finished plugging the port | 20:11 |
| johnsom | So I think we should change that to either return 503 Service Unavailable (The server is currently unable to handle the request due to a temporary overloading or maintenance of the server.) or retry in the amp agent for some time. We also might be able to do the "udevadm settle" in the amp agent, but that can block for like up to two minutes. | 20:16 |
| johnsom | I think 503 might be the best answer myself | 20:16 |
| *** ipsecguy has joined #openstack-lbaas | 20:17 | |
| johnsom | Sorry, house guests, have to disappear again for a bit | 20:17 |
| *** ipsecguy_ has quit IRC | 20:21 | |
| *** gcheresh has quit IRC | 20:32 | |
| *** cpuga has joined #openstack-lbaas | 22:00 | |
| *** cpuga has quit IRC | 22:05 | |
| *** rcernin has quit IRC | 22:18 | |
| *** catintheroof has joined #openstack-lbaas | 22:28 | |
| *** catintheroof has quit IRC | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!