| *** sshank has quit IRC | 00:00 | |
| *** pothole has quit IRC | 00:01 | |
| *** pothole has joined #openstack-lbaas | 00:01 | |
| *** csomerville has quit IRC | 00:03 | |
| *** armax has quit IRC | 00:05 | |
| *** amotoki has quit IRC | 00:07 | |
| *** pothole has quit IRC | 00:09 | |
| *** xgerman has joined #openstack-lbaas | 00:20 | |
| *** fnaval has joined #openstack-lbaas | 00:37 | |
| *** fnaval_ has quit IRC | 00:40 | |
| openstackgerrit | Merged openstack/octavia master: Fix devstack plugin to stop octavia processes https://review.openstack.org/461927 | 00:45 |
|---|---|---|
| openstackgerrit | German Eichberger proposed openstack/octavia master: Adds the ability to failover a load balancer https://review.openstack.org/444347 | 01:03 |
| *** sshank has joined #openstack-lbaas | 01:04 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Remove log translations from octavia https://review.openstack.org/447872 | 01:05 |
| johnsom | Oye, that was fun... | 01:05 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Remove log translations from octavia https://review.openstack.org/447872 | 01:07 |
| *** cpuga has quit IRC | 01:12 | |
| *** cpuga has joined #openstack-lbaas | 01:23 | |
| *** yamamoto has joined #openstack-lbaas | 01:31 | |
| *** yamamoto has quit IRC | 01:35 | |
| *** blogan has joined #openstack-lbaas | 01:36 | |
| *** cpuga has quit IRC | 01:38 | |
| *** blogan__ has quit IRC | 01:39 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add some hacking checks borrowed from neutron https://review.openstack.org/462334 | 01:41 |
| johnsom | rm_work https://review.openstack.org/462336 | 01:48 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Adds the ability to failover a load balancer https://review.openstack.org/444347 | 01:49 |
| xgerman | ^^ rolled back POST -> PUT | 01:50 |
| *** fnaval_ has joined #openstack-lbaas | 02:05 | |
| *** fnaval has quit IRC | 02:08 | |
| *** yamamoto has joined #openstack-lbaas | 02:36 | |
| *** yamamoto has quit IRC | 02:46 | |
| rm_work | kk | 02:49 |
| rm_work | johnsom: uhh, soooo | 02:49 |
| rm_work | johnsom: if you remove that from neutron-lbaas, the ones that include octavia will explode | 02:50 |
| rm_work | because our devstack client pulls in octaviaclient | 02:50 |
| johnsom | Well, it is blowing up the namespace driver gates | 02:50 |
| rm_work | :/ | 02:51 |
| rm_work | why | 02:51 |
| rm_work | it shouldn't matter | 02:51 |
| rm_work | so it pulls in some random client lib | 02:52 |
| rm_work | can you show me a log? | 02:52 |
| johnsom | Client does not get installed because octavia devstack plugin does not run | 02:52 |
| rm_work | Just adding it to PROJECTS installs it | 02:52 |
| rm_work | that's the point | 02:52 |
| johnsom | You aren't just adding it to projects! | 02:53 |
| rm_work | we need PROJECTS to install it, so that when the octavia plugin does a git_clone, it is already there, so it noops, and doesn't trigger the "cloning explosion" | 02:53 |
| rm_work | if we don't run the octavia plugin, PROJECTS should just install it and that's it | 02:53 |
| rm_work | ah, because it's also in LIBS_FROM_GIT? | 02:53 |
| johnsom | Look at the log translation patch in neutron lbaas | 02:53 |
| rm_work | devstack should handle that too | 02:53 |
| johnsom | Yeah, projects would be fine | 02:53 |
| johnsom | It is the libs from git | 02:54 |
| rm_work | hmmmm | 02:54 |
| johnsom | Namespace driver does not run octavia devstack plugin | 02:54 |
| rm_work | yeah but it shouldn't matter | 02:54 |
| rm_work | i should be able to add like | 02:54 |
| rm_work | python-kuryrclient | 02:55 |
| rm_work | and it should be fine | 02:55 |
| rm_work | even with LIBS_FROM_GIT | 02:55 |
| johnsom | We should not be putting libs from git for the osc client in nlbaas gates | 02:55 |
| rm_work | ugh OK so let me think about how we can do this better | 02:55 |
| rm_work | is there a way we can tell whether we're inside a gate job or not, from within the devstack plugin? | 02:55 |
| johnsom | Libs from git makes devstack check if it installed | 02:56 |
| rm_work | right, and it should be installed | 02:56 |
| johnsom | No | 02:56 |
| rm_work | ah but not from git | 02:56 |
| johnsom | I don't think so | 02:56 |
| rm_work | blegh part of me expected that arg to make devstack just *do that* | 02:56 |
| johnsom | It is useless with nlbaas namespace driver | 02:56 |
| rm_work | actually, I really still think it does | 02:56 |
| johnsom | Well, nlbaas overall | 02:56 |
| rm_work | yes it's useless but it also doesn't matter | 02:56 |
| rm_work | it's a noop | 02:56 |
| rm_work | it's useless in the octavia gate as well right now, but it's *fine* | 02:57 |
| rm_work | one sec | 02:57 |
| johnsom | Can't you just add it to projects and not do the libs fron get in nlbaas jobs? | 02:57 |
| rm_work | if it's not in LIBS_FROM_GIT I think our git_clone will still cause it to explode | 02:58 |
| rm_work | ok hold on. | 03:00 |
| rm_work | this is really dumb, but maybe I can figure it out | 03:00 |
| rm_work | we just need to do our clone conditionally | 03:00 |
| *** bread_ has joined #openstack-lbaas | 03:02 | |
| rm_work | yeah k sec | 03:03 |
| johnsom | Hmm, I am tired and grumpy (too much translation log patch review and work) so I am going to sign off for the night and see what the end result is in the morning. I just don't see why nlbaas should have anything to do with that code.... | 03:04 |
| rm_work | ugh so we already install conditionally on the LIBS_FROM_GIT thing | 03:07 |
| rm_work | maybe we can just do PROJECTS | 03:07 |
| rm_work | johnsom: one could ask why the namespace jobs include *octavia* | 03:09 |
| rm_work | I only include the client on jobs that include octavia.. | 03:10 |
| *** kbyrne_ has joined #openstack-lbaas | 03:10 | |
| rm_work | oh actually ... hmmm | 03:10 |
| rm_work | maybe it will be ok | 03:10 |
| rm_work | maybe I'm thinking of a different permutation that had an issue with not including it in neutron-lbaas | 03:11 |
| *** ankur-gupta-f4 has quit IRC | 03:11 | |
| *** kbyrne has quit IRC | 03:11 | |
| *** mdavidson has quit IRC | 03:11 | |
| *** aamerine has quit IRC | 03:11 | |
| *** kbyrne_ is now known as kbyrne | 03:11 | |
| rm_work | sure, let's try this way | 03:12 |
| rm_work | johnsom: should I continue to hack on the log translation patches? | 03:13 |
| rm_work | i see this one is still failing | 03:13 |
| johnsom | No, please fix it in a separate patch | 03:17 |
| *** mdavidson has joined #openstack-lbaas | 03:18 | |
| johnsom | Hmm, looks like i screwed up the octavia patch. Ugh | 03:20 |
| *** blogan has quit IRC | 03:30 | |
| *** blogan has joined #openstack-lbaas | 03:30 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Remove log translations from octavia https://review.openstack.org/447872 | 03:30 |
| *** armax has joined #openstack-lbaas | 03:31 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add some hacking checks borrowed from neutron https://review.openstack.org/462334 | 03:32 |
| *** gongysh has joined #openstack-lbaas | 03:32 | |
| rm_work | fix which in a separate patch? | 03:45 |
| rm_work | decided to leave your project-config patch as-is | 03:45 |
| rm_work | it might actually work with our current method | 03:45 |
| rm_work | last time i tried it was set up a little differently | 03:46 |
| *** fnaval has joined #openstack-lbaas | 03:52 | |
| *** fnaval_ has quit IRC | 03:55 | |
| *** armax has quit IRC | 03:59 | |
| *** sshank has quit IRC | 04:52 | |
| *** sshank has joined #openstack-lbaas | 05:07 | |
| *** sshank has quit IRC | 05:10 | |
| *** ankur-gupta-f4 has joined #openstack-lbaas | 05:13 | |
| *** yamamoto has joined #openstack-lbaas | 05:18 | |
| *** links has joined #openstack-lbaas | 05:18 | |
| *** gcheresh has joined #openstack-lbaas | 05:20 | |
| *** yamamoto has quit IRC | 05:21 | |
| rm_work | johnsom: i am not seeing pep8 failures due to this anywhere? | 05:29 |
| rm_work | was gonna just see pep8 succeeding instead of failing to know your translation patch was good, but | 05:29 |
| rm_work | i can't get a negative | 05:29 |
| johnsom | Because I am awesome | 05:30 |
| rm_work | lol | 05:31 |
| rm_work | yeah but like | 05:31 |
| rm_work | should i see pep8 fails on master? | 05:31 |
| johnsom | On lbaas, but octavia didn't have a hacking rule until now | 05:32 |
| rm_work | ahh i see | 05:32 |
| rm_work | so if i cherry-pick the hacking rule commit | 05:32 |
| rm_work | i should see a fail | 05:32 |
| johnsom | Nlbaas inherited the hacking from neutron-lib | 05:33 |
| rm_work | yeah and you copied it to octavia | 05:34 |
| rm_work | but on top of the log patch (obviously) | 05:34 |
| johnsom | Well, the translation thing is in. The log patch | 05:35 |
| rm_work | hmm k | 05:35 |
| rm_work | well erm | 05:35 |
| rm_work | k | 05:35 |
| rm_work | ah yeah i see | 05:35 |
| *** fnaval has quit IRC | 05:42 | |
| *** amotoki has joined #openstack-lbaas | 06:05 | |
| *** yamamoto has joined #openstack-lbaas | 06:21 | |
| *** voelzmo has joined #openstack-lbaas | 06:23 | |
| *** yamamoto has quit IRC | 06:26 | |
| *** yamamoto has joined #openstack-lbaas | 06:26 | |
| *** yamamoto has quit IRC | 06:28 | |
| *** mdurrant has quit IRC | 06:37 | |
| *** pcaruana has joined #openstack-lbaas | 06:45 | |
| *** yamamoto has joined #openstack-lbaas | 06:59 | |
| *** yamamoto has quit IRC | 07:09 | |
| *** tesseract has joined #openstack-lbaas | 07:29 | |
| *** aojea has joined #openstack-lbaas | 07:48 | |
| *** blogan_ has joined #openstack-lbaas | 07:50 | |
| *** blogan has quit IRC | 07:51 | |
| openstackgerrit | Jude Cross proposed openstack/python-octaviaclient master: Add l7policy commands to client https://review.openstack.org/462431 | 08:06 |
| *** mdurrant has joined #openstack-lbaas | 08:29 | |
| *** JudeC has quit IRC | 08:33 | |
| *** yamamoto has joined #openstack-lbaas | 09:15 | |
| *** yamamoto has quit IRC | 09:19 | |
| *** links has quit IRC | 09:45 | |
| *** blogan__ has joined #openstack-lbaas | 10:01 | |
| *** blogan_ has quit IRC | 10:03 | |
| *** ankur-gupta-f4 has quit IRC | 10:25 | |
| *** ankur-gupta-f4 has joined #openstack-lbaas | 10:26 | |
| *** gongysh has quit IRC | 10:30 | |
| *** catintheroof has joined #openstack-lbaas | 11:47 | |
| *** catintheroof has quit IRC | 11:51 | |
| *** krypto has joined #openstack-lbaas | 12:31 | |
| *** cpuga has joined #openstack-lbaas | 12:50 | |
| *** yamamoto has joined #openstack-lbaas | 12:54 | |
| *** yamamoto has quit IRC | 13:22 | |
| *** yamamoto has joined #openstack-lbaas | 13:24 | |
| *** chlong has joined #openstack-lbaas | 13:32 | |
| xgerman | o/ | 13:43 |
| *** fnaval has joined #openstack-lbaas | 13:58 | |
| *** chlong has quit IRC | 14:01 | |
| *** armax has joined #openstack-lbaas | 14:12 | |
| *** chlong has joined #openstack-lbaas | 14:14 | |
| *** catintheroof has joined #openstack-lbaas | 14:20 | |
| *** catinthe_ has joined #openstack-lbaas | 14:23 | |
| *** catintheroof has quit IRC | 14:24 | |
| *** armax has quit IRC | 14:38 | |
| *** voelzmo has quit IRC | 14:42 | |
| *** gongysh has joined #openstack-lbaas | 14:44 | |
| *** yamamoto has quit IRC | 14:44 | |
| *** voelzmo_ has joined #openstack-lbaas | 14:44 | |
| xgerman | Rosella stepping down — Neutron is imploding…. | 14:49 |
| *** gongysh has quit IRC | 14:49 | |
| *** armax has joined #openstack-lbaas | 14:56 | |
| *** jerrygb has joined #openstack-lbaas | 15:07 | |
| *** gcheresh has quit IRC | 15:08 | |
| *** sshank has joined #openstack-lbaas | 15:08 | |
| johnsom | o/ | 15:13 |
| *** armax has quit IRC | 15:25 | |
| xgerman | johnsom I will look at single create again… would be good if we can approve status before so I have less rework | 15:34 |
| xgerman | :-) | 15:34 |
| xgerman | merge conflicts | 15:34 |
| johnsom | Ok, yeah, I should be back to a place I can attempt to test that again | 15:34 |
| xgerman | thx | 15:34 |
| johnsom | xgerman rm_work This should probably go in sooner than later: https://review.openstack.org/#/c/447872/ | 15:35 |
| johnsom | Otherwise we are going to have patches that still add translations... | 15:35 |
| *** chlong has quit IRC | 15:36 | |
| *** blogan__ is now known as blogan | 15:38 | |
| *** yamamoto has joined #openstack-lbaas | 15:45 | |
| *** voelzmo_ has quit IRC | 15:46 | |
| *** voelzmo has joined #openstack-lbaas | 15:46 | |
| openstackgerrit | Merged openstack/octavia master: project_id should remain in the response for v2 objects https://review.openstack.org/461344 | 15:54 |
| *** yamamoto has quit IRC | 15:57 | |
| *** voelzmo has quit IRC | 15:59 | |
| *** JudeC has joined #openstack-lbaas | 16:03 | |
| xgerman | “ "fault": {"message": "Host 'octavia' is not mapped to any cell", "code": 400, "created": "2017-05-04T16:05:58Z"}, "metadata": {}}}" | 16:06 |
| johnsom | Hmm, that is neat | 16:07 |
| xgerman | mmh, how did the single create call pass the ate? | 16:07 |
| xgerman | might be my insane devstack… | 16:08 |
| *** fnaval has quit IRC | 16:08 | |
| *** fnaval has joined #openstack-lbaas | 16:08 | |
| johnsom | Could be something merged in nova between | 16:08 |
| xgerman | mmh, wonder if starting with a fresh vm helps but I am afraid of system-d | 16:09 |
| *** fnaval has quit IRC | 16:11 | |
| johnsom | Yeah, systemd threw a wrench in my demo yesterday. I am just not super proficient with it yet | 16:19 |
| johnsom | https://docs.openstack.org/developer/devstack/systemd.html | 16:21 |
| *** yamamoto has joined #openstack-lbaas | 16:26 | |
| *** ankur-gupta-f4 has quit IRC | 16:35 | |
| *** yamamoto has quit IRC | 16:37 | |
| *** pcaruana has quit IRC | 16:40 | |
| *** ankur-gupta-f4 has joined #openstack-lbaas | 16:44 | |
| *** armax has joined #openstack-lbaas | 16:47 | |
| *** krypto has quit IRC | 16:50 | |
| *** krypto has joined #openstack-lbaas | 16:50 | |
| *** krypto has quit IRC | 16:50 | |
| *** krypto has joined #openstack-lbaas | 16:50 | |
| *** fnaval has joined #openstack-lbaas | 16:54 | |
| *** krypto has quit IRC | 16:54 | |
| xgerman | live and learn: sudo journalctl -f --unit devstack@o-cw.service | 17:07 |
| xgerman | this looks like it will time out | 17:09 |
| xgerman | https://www.irccloud.com/pastebin/VODo7t4k/ | 17:10 |
| *** aojea has quit IRC | 17:10 | |
| *** aojea has joined #openstack-lbaas | 17:11 | |
| *** aojea has quit IRC | 17:15 | |
| *** tesseract has quit IRC | 17:30 | |
| *** SumitNaiksatam has joined #openstack-lbaas | 17:43 | |
| *** basilAB has quit IRC | 17:45 | |
| *** vaishali has quit IRC | 17:46 | |
| xgerman | johnsom any idea? traceroute goes to our interface but I am not sure which bridge we are using | 18:00 |
| johnsom | I haven't looked at it too much. Was fixing Ihar's pedantic comments on the log patch for nlbaas. I didn't go scrub all of the log lines in nlbaas because, yeah, short life over there. I did do that for octavia however. Ugh. | 18:02 |
| johnsom | So basically your issue is you can't reach the management interface on the amp? | 18:02 |
| xgerman | yep | 18:11 |
| openstackgerrit | Michael Johnson proposed openstack/neutron-lbaas master: Remove log translations from neutron-lbaas https://review.openstack.org/447921 | 18:11 |
| *** bread_ is now known as aamerine | 18:13 | |
| johnsom | Did you check the /opt/stack/data/nova/instances/<uuid>/console.log? | 18:14 |
| xgerman | empty | 18:15 |
| xgerman | might be systemd | 18:15 |
| johnsom | ummm, wow, that would be unfortunate | 18:15 |
| xgerman | yeah, I am now really worried about my lab :-( | 18:16 |
| johnsom | So, I have a console log | 18:16 |
| xgerman | cool | 18:16 |
| johnsom | what does nova say when you do a nova show on the amp instance? | 18:16 |
| xgerman | nothing out of the ordinary: | 18:17 |
| xgerman | https://www.irccloud.com/pastebin/GJRX4KRj/ | 18:17 |
| *** harlowja has quit IRC | 18:19 | |
| *** sanfern has joined #openstack-lbaas | 18:19 | |
| johnsom | Yeah, that looks right | 18:20 |
| johnsom | Can you do a "ip a" and "netstat -rn"? | 18:20 |
| xgerman | https://www.irccloud.com/pastebin/0YNt5NpZ/ | 18:21 |
| * xgerman gotta get with those new commands still using route and ifconfig | 18:21 | |
| xgerman | br-int is ipv6 only - hmm | 18:22 |
| johnsom | Hmmm, that all looks ok for mgmt net | 18:23 |
| johnsom | We really need the console log | 18:23 |
| openstackgerrit | Michael Johnson proposed openstack/neutron-lbaas master: Remove log translations from neutron-lbaas https://review.openstack.org/447921 | 18:23 |
| *** sanfern has quit IRC | 18:24 | |
| *** vaishali has joined #openstack-lbaas | 18:32 | |
| *** basilAB has joined #openstack-lbaas | 18:33 | |
| xgerman | well, I guess my fresh install is hosed | 18:41 |
| *** Guest92102 is now known as redrobot | 18:42 | |
| johnsom | Hmm | 18:43 |
| johnsom | I'm running the systemd "stuff" and I have console logs, so I would be worried that nova is broken | 18:43 |
| xgerman | yeah, tomorrow is. anew day to try again | 18:50 |
| johnsom | Yeah, I am going to eat lunch and give it a go | 18:51 |
| *** amotoki has quit IRC | 18:55 | |
| *** blogan has quit IRC | 19:22 | |
| *** aojea has joined #openstack-lbaas | 19:27 | |
| *** cpuga has quit IRC | 19:40 | |
| *** harlowja has joined #openstack-lbaas | 19:48 | |
| *** aojea has quit IRC | 19:50 | |
| *** aojea has joined #openstack-lbaas | 19:50 | |
| *** aojea_ has joined #openstack-lbaas | 19:52 | |
| *** aojea has quit IRC | 19:53 | |
| *** blogan has joined #openstack-lbaas | 19:59 | |
| xgerman | re-installed - console log still empty; and lb not working | 20:02 |
| *** ipsecguy_ has joined #openstack-lbaas | 20:05 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add support for single-create for APIv2 https://review.openstack.org/459032 | 20:06 |
| johnsom | ^^^ Rebase | 20:07 |
| *** ipsecguy has quit IRC | 20:07 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Implement sorting and pagination for octavia https://review.openstack.org/382147 | 20:07 |
| johnsom | ^^^ same | 20:07 |
| johnsom | xgerman does the console log file exist? | 20:08 |
| xgerman | yes | 20:08 |
| xgerman | but empty | 20:08 |
| *** cpuga has joined #openstack-lbaas | 20:09 | |
| johnsom | Hmm, I don't see anything obvious in the nova merge list since I stacked yesterday | 20:10 |
| johnsom | I will restack now too | 20:10 |
| xgerman | ok, let me know | 20:11 |
| johnsom | oh man... | 20:12 |
| johnsom | I just did a clean.sh with this systemd stuff. | 20:12 |
| johnsom | everything +dog is still running | 20:12 |
| johnsom | Wow, like nothing shut down | 20:13 |
| *** aojea_ has quit IRC | 20:19 | |
| johnsom | Why the @#*$%*@#$ can't journalctl learn to line wrap | 20:26 |
| johnsom | by default | 20:26 |
| openstackgerrit | Merged openstack/neutron-lbaas master: Remove log translations from neutron-lbaas https://review.openstack.org/447921 | 20:47 |
| openstackgerrit | Michael Johnson proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/462111 | 20:48 |
| openstackgerrit | Michael Johnson proposed openstack/neutron-lbaas master: Remove get_namespace from API extensions https://review.openstack.org/460831 | 20:54 |
| johnsom | xgerman Yeah, devstack is broken | 20:58 |
| xgerman | :-( | 20:58 |
| johnsom | My cirros web servers aren't coming up or logging to console.log either | 20:58 |
| johnsom | Oye, and since devstack doesn't have stable who knows if we can even use a stable branch for the hands on lab | 20:59 |
| xgerman | :-( | 21:00 |
| johnsom | xgerman I think it is the new qemu packages from ubuntu | 21:33 |
| johnsom | My qemu processes are 100% cpu | 21:33 |
| xgerman | k | 21:33 |
| johnsom | https://www.irccloud.com/pastebin/DYi4ZXLs/ | 21:33 |
| johnsom | I think that is the list of stuff that updated before this happened | 21:34 |
| johnsom | Another side effect with the systemd stuff, the syslog is full of openstack logs | 21:34 |
| *** jerrygb has quit IRC | 21:35 | |
| *** cpuga has quit IRC | 21:36 | |
| *** cpuga has joined #openstack-lbaas | 21:50 | |
| johnsom | Ok, it's the new ubuntu packages | 21:57 |
| johnsom | apt-get install seabios=1.8.2-1ubuntu1 qemu-system-sparc=1:2.5+dfsg-5ubuntu10.11 qemu-system-ppc=1:2.5+dfsg-5ubuntu10.11 qemu-system-arm=1:2.5+dfsg-5ubuntu10.11 qemu-kvm=1:2.5+dfsg-5ubuntu10.11 qemu-system-x86=1:2.5+dfsg-5ubuntu10.11 qemu-system-mips=1:2.5+dfsg-5ubuntu10.11 qemu-system-misc=1:2.5+dfsg-5ubuntu10.11 | 21:57 |
| johnsom | That rolls back the bad patches and you can boot instances again | 21:57 |
| johnsom | xgerman ^^^ | 21:59 |
| xgerman | Thanks | 22:00 |
| *** SumitNaiksatam has quit IRC | 22:01 | |
| rm_work | we can just pick a commit from before the systemd change | 22:02 |
| rm_work | and use that | 22:02 |
| rm_work | from devstack | 22:02 |
| openstackgerrit | Joshua Harlow proposed openstack/octavia master: Handle stopping the health manager and ejecting out of current health failovers https://review.openstack.org/462748 | 22:03 |
| harlowja | rm_work JudeC ^ | 22:04 |
| harlowja | another take/variation of https://review.openstack.org/#/c/456420/ | 22:04 |
| rm_work | will look eventually | 22:04 |
| harlowja | in general, i was also looking over how we could get over to tooz for the udp status checker | 22:04 |
| harlowja | but that stuff is pretty engrained :-/ | 22:05 |
| johnsom | I think there is a flag that revert to screen | 22:05 |
| rm_work | so much shit to do today and tomorrow though for the lab, we prolly don't have time to work on stuff that's already "mostly working" | 22:05 |
| rm_work | johnsom: yeah I am going to try using it | 22:05 |
| rm_work | omg what happened to single-create lol | 22:07 |
| rm_work | mind if i undo your rebase | 22:07 |
| johnsom | Did I screw it up????? | 22:08 |
| rm_work | not sure | 22:08 |
| rm_work | something is woooonky | 22:08 |
| rm_work | rather just start from before it | 22:08 |
| rm_work | looks like yet | 22:12 |
| rm_work | *yes | 22:12 |
| rm_work | you took the old minimal-pool stuff | 22:12 |
| rm_work | instead of the id-only stuff | 22:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add support for single-create for APIv2 https://review.openstack.org/459032 | 22:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Implement sorting and pagination for octavia https://review.openstack.org/382147 | 22:13 |
| johnsom | Oh, I see, darn it | 22:13 |
| johnsom | rm_work if you didn't see earlier, the new qemu packages from ubuntu break nova in a bad way. I have a command to revert to the older packages I pasted to the channel | 22:15 |
| rm_work | ok | 22:15 |
| rm_work | so just | 22:16 |
| rm_work | do that AFTER devstack but before doing anything with nova | 22:16 |
| johnsom | Actually do it before devstack. | 22:16 |
| johnsom | Otherwise you have to restart nova to get it to pick it up | 22:16 |
| rm_work | ah | 22:17 |
| johnsom | Basically if you do a dist-upgrade you need to downgrade those after | 22:17 |
| rm_work | doesn't devstack *install* the newer version then though? | 22:17 |
| johnsom | I don't think so | 22:17 |
| rm_work | hmm k | 22:17 |
| johnsom | Darn, yes it does | 22:18 |
| johnsom | So, yeah, after devstack then restart nova | 22:19 |
| *** fnaval has quit IRC | 22:19 | |
| johnsom | hmmm, that is going to suck | 22:19 |
| johnsom | You might be able to do it before devstack, then "apt-mark hold <package-name>" to stop devstack from upgrading it | 22:22 |
| rm_work | yes | 22:23 |
| rm_work | i think that's prolly best | 22:23 |
| rm_work | xgerman: i am working on making sure this script will actually *work* now | 22:24 |
| rm_work | xgerman: then we can look at testing it on the student machines :/ | 22:24 |
| *** blogan has quit IRC | 22:25 | |
| *** cpuga has quit IRC | 22:30 | |
| johnsom | harlowja What does tooz bring to the party for the udp heartbeats? | 22:34 |
| rm_work | oops missed some tenant_id references | 22:34 |
| harlowja | johnsom it replaces it with the technology/driver specific heartbeats | 22:34 |
| harlowja | which then depends on what driver u want to default to | 22:34 |
| johnsom | Ok, but why? | 22:35 |
| rm_work | guh the rebase i'm about to get to do | 22:35 |
| rm_work | T_T | 22:35 |
| harlowja | johnsom less code, more people that know how the heartbeating works, more people that know how the heartbeating works, thus more people that can fix it, or view what the current heartbeats are or ... | 22:35 |
| harlowja | those kinds of things | 22:35 |
| *** jerrygb has joined #openstack-lbaas | 22:37 | |
| johnsom | Hmm, I think I would like to get a architecture pitch. Right now this thing scales beautifully, so alternatives need to meet that bar | 22:37 |
| *** JudeC has quit IRC | 22:40 | |
| harlowja | ya, though i have a feeling etcd is going to outpace the ability of octavia | 22:40 |
| harlowja | with things like 'https://github.com/coreos/etcd/issues/7624' | 22:40 |
| harlowja | 'efficiently handle millions of unique watches ' | 22:40 |
| harlowja | key watching == heartbeat (and associated death detection) | 22:41 |
| *** jerrygb has quit IRC | 22:41 | |
| johnsom | Maybe etcd can borrow our code | 22:42 |
| johnsom | grin | 22:42 |
| johnsom | Yeah, but our heartbeats carry the stats and status tree. It's not just an "are you there" | 22:42 |
| harlowja | keys map to values :-P | 22:42 |
| harlowja | like a typically key -> value store | 22:43 |
| harlowja | in zookeeper or etcd | 22:43 |
| johnsom | Again, I want to understand the architecture, if this thing is going to pool for each key, game over man | 22:43 |
| johnsom | pool-> poll | 22:43 |
| harlowja | nah, clients that are interested in others keys (to watch them) subscribe to them | 22:44 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Purge more tenant_id references that I missed https://review.openstack.org/462750 | 22:45 |
| harlowja | using http long polls, which afaik are not really polls (at least this is what etcd is using) | 22:45 |
| johnsom | I mean right now we have basically one packet, from amp to a randomly selected endpoint from a list of endpoints. no handshakes, no persistent connections, no "master" health manager | 22:46 |
| johnsom | I'm just wearing my skeptics hat here... | 22:46 |
| *** JudeC has joined #openstack-lbaas | 22:47 | |
| JudeC | harlowja: Just saw your message looking now :) | 22:47 |
| harlowja | johnsom another angle is to look at this from operational aspect of octavia | 22:48 |
| johnsom | Ok, I'm listening... | 22:49 |
| harlowja | one sec | 22:50 |
| rm_work | UUUUUGH | 22:50 |
| rm_work | DIB WHY | 22:50 |
| rm_work | http://logs.openstack.org/72/447872/8/gate/gate-octavia-v1-dsvm-py3x-scenario-ubuntu-xenial/0865502/logs/devstacklog.txt.gz#_2017-05-04_22_25_17_669 | 22:50 |
| johnsom | Part of this comes from an implementation I saw using gearman for this sort of thing, oye, train wreck | 22:50 |
| rm_work | DIB py3 breakage | 22:50 |
| * rm_work sighs | 22:50 | |
| * rm_work goes and fixes it | 22:50 | |
| johnsom | That merged four days ago... | 22:53 |
| *** fnaval has joined #openstack-lbaas | 22:53 | |
| *** fnaval has quit IRC | 22:53 | |
| *** fnaval has joined #openstack-lbaas | 22:54 | |
| johnsom | rm_work The hold trick appears to work | 23:00 |
| rm_work | ok | 23:00 |
| rm_work | can you pastebin me the lines for the hold | 23:00 |
| rm_work | i assume we could do JUST that | 23:00 |
| rm_work | and then devstack will install the right one | 23:00 |
| rm_work | once it's pinned | 23:00 |
| johnsom | http://paste.openstack.org/show/608915/ | 23:02 |
| harlowja | johnsom so operationally, one of the benefits of a single system that everyone uses for liveness, is well a single system to understand, typically then a single system to turn off a 'worker' (whether that be a worker in neutron, or a worker in nova or ...) | 23:04 |
| harlowja | now maybe that goal is to lofty, lol | 23:04 |
| *** catinthe_ has quit IRC | 23:08 | |
| rm_work | ok time to see if depends-on works with dib now :P | 23:26 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add support for single-create for APIv2 https://review.openstack.org/459032 | 23:27 |
| rm_work | it's summit/demo time | 23:30 |
| xgerman | harlowja we will let nova go first — and then see | 23:30 |
| rm_work | of course EVERYTHING BREAKS AT ONCE | 23:30 |
| rm_work | this is to be expected | 23:30 |
| xgerman | yep, it’s demo time | 23:30 |
| harlowja | xgerman :-/ | 23:31 |
| rm_work | i need to put a caching proxy on my router and have it cache the disk images that devstack uses | 23:31 |
| harlowja | k8s is already going xgerman so that might be enough, lol | 23:31 |
| xgerman | I don’t doubt it works for k8 — but does it work for OpenStack… | 23:32 |
| xgerman | also the redundancy euquation of etcd is a bit different than ours. We get benefit from two health managers but etcd needs at least 3 | 23:33 |
| johnsom | Wait, all those k8s people keep using octavia. Maybe they have already made their decision and octavia's approach won.... | 23:33 |
| xgerman | this irked me since not everybody has three az;s | 23:33 |
| xgerman | johnsom +1 | 23:33 |
| xgerman | they mentioned to me that k8 sucks at load balancing | 23:34 |
| openstackgerrit | Joshua Harlow proposed openstack/octavia master: Handle stopping the health manager and ejecting out of current health failovers https://review.openstack.org/462748 | 23:38 |
| rm_work | ah harlowja: add this: | 23:39 |
| rm_work | Depends-On: I3d86482a2999197a60a81d42afc5ef7a6e71e313 | 23:39 |
| harlowja | well it may not really depend on :-P | 23:39 |
| rm_work | it does | 23:39 |
| harlowja | ok | 23:39 |
| rm_work | *everything* now does | 23:39 |
| rm_work | lol | 23:39 |
| harlowja | ah | 23:39 |
| harlowja | k | 23:39 |
| harlowja | one of those | 23:39 |
| rm_work | until they merge it | 23:39 |
| rm_work | but it'll speed up having your checks already done | 23:39 |
| openstackgerrit | Joshua Harlow proposed openstack/octavia master: Handle stopping the health manager and ejecting out of current health failovers https://review.openstack.org/462748 | 23:40 |
| harlowja | so a related question, in line with ^ | 23:40 |
| harlowja | how long does 'self.cw.failover_amphora' take | 23:40 |
| rm_work | it depends | 23:40 |
| harlowja | cause the time that takes, will be the time such a thing will take to turn off :-P | 23:40 |
| johnsom | Depends on your configuration | 23:40 |
| harlowja | (the app to turn off) | 23:40 |
| rm_work | could be 6 seconds | 23:41 |
| rm_work | could be 6 minutes | 23:41 |
| rm_work | depends entirely on your config and your cloud | 23:41 |
| johnsom | If you are in active/standby model you can failover in around a second | 23:41 |
| harlowja | no way to interrupt it? | 23:41 |
| rm_work | johnsom: not failover_amphora | 23:41 |
| harlowja | ie whatever self.cw.failover_amphora is doin | 23:41 |
| harlowja | can that be stoppe | 23:41 |
| rm_work | johnsom: that's the code to do the delete/create | 23:41 |
| rm_work | harlowja: not safely | 23:41 |
| harlowja | *stopped | 23:41 |
| harlowja | rm_work hmmm | 23:41 |
| rm_work | you'd end up with like | 23:41 |
| rm_work | some random VM created | 23:42 |
| johnsom | Ah, so he was asking specifically | 23:42 |
| rm_work | without the right sec groups and no management interface | 23:42 |
| harlowja | rm_work that's cool, u have a cleanup process right :-P | 23:42 |
| rm_work | uhhh | 23:42 |
| rm_work | yeah | 23:42 |
| rm_work | it's called "josh goes into nova and deletes that" | 23:42 |
| harlowja | hmmm | 23:42 |
| johnsom | With spares pool it's around 30 seconds if I remember right. The long tail is plugging port via neutron | 23:42 |
| harlowja | cleanup process doesn't wipe VMs that never finished? | 23:42 |
| harlowja | wipe/delete | 23:42 |
| rm_work | johnsom: it takes about 15s I think in our cloud :) | 23:43 |
| rm_work | i don't believe our taskflow revert workflows have VM delete calls :/ | 23:43 |
| rm_work | MAYBE? | 23:43 |
| johnsom | lower neutron penalty.. | 23:43 |
| rm_work | but i doubt it | 23:43 |
| johnsom | Yeah, it does delete | 23:43 |
| johnsom | on revert | 23:43 |
| rm_work | ah ok | 23:43 |
| rm_work | then nm maybe it'll be ok | 23:43 |
| rm_work | though if the FLOW dies | 23:43 |
| rm_work | there's nothing | 23:43 |
| rm_work | so if you just *kill it* | 23:43 |
| harlowja | rm_work well that's one way, though u'd need to have a way to signal to 'self.cw.failover_amphora' that that is happening | 23:44 |
| johnsom | That is exactly why I asked for the "always_resolve" setting.... | 23:44 |
| rm_work | there's no cleanup following | 23:44 |
| harlowja | so it can abort the taskflow engine run | 23:44 |
| harlowja | right now its just call into blocking function 'self.cw.failover_amphora' and <magic> | 23:44 |
| rm_work | i'd just let a failover finish | 23:44 |
| rm_work | HM can take a little longer to shut down >_> | 23:44 |
| johnsom | Sigh, more uwsgi fall out: 0curl: (7) Failed to connect to localhost port 5000: Connection refused | 23:44 |
| harlowja | rm_work up to 6 minutes sucks lol | 23:45 |
| rm_work | and failovers are *important*] | 23:45 |
| rm_work | uhh | 23:45 |
| rm_work | i thought we fixed those keystone changes | 23:45 |
| rm_work | harlowja: eh | 23:45 |
| rm_work | they should suck it up | 23:45 |
| rm_work | that's the LONG end though | 23:45 |
| rm_work | like, cloud with only software virt :P | 23:45 |
| johnsom | We did in the gates, not in my test scripts.... | 23:45 |
| rm_work | which doesn't really exist | 23:45 |
| rm_work | ahh lol | 23:45 |
| rm_work | harlowja: 6 minutes would be ... in virtualbox on my mac | 23:45 |
| harlowja | rm_work okies, other way is u start integrated more tightly with the taskflow engine | 23:46 |
| rm_work | or a triple-o install with no nested-virt | 23:46 |
| rm_work | harlowja: hmm, how do we integrate MORE tightly with taskflow? | 23:46 |
| rm_work | of course you'd say that :P | 23:46 |
| harlowja | well `self.cw.failover_amphora` | 23:46 |
| rm_work | that kicks a task | 23:47 |
| rm_work | isn't that pretty integrated? | 23:47 |
| harlowja | to make that interuptible what would u need to do | 23:47 |
| harlowja | one way, is to be able to preempt all running things in your app, telling them to shutdown (now) | 23:48 |
| johnsom | harlowja You are screwed on virtualbox.... no nested virtualization. Booting a VM takes like 10 minutes under virtualbox | 23:49 |
| harlowja | other way is to have those things in your app check the system is being turned off at various locations in code and handle that appropriatly | 23:49 |
| harlowja | a taskflow engine/workflow has those locations automatically (at each transition in the workflow) | 23:50 |
| xgerman | so what you want is a way to “drain” controller nodes? | 23:50 |
| johnsom | Well, not a good start to single call testing: 500 Internal Server Error | 23:50 |
| harlowja | xgerman there are a bunch of conceptual ways to do it :-P | 23:50 |
| johnsom | harlowja I thought that is what jobboard was for.... | 23:50 |
| harlowja | ^ is just how to turn off an application quickly, lol | 23:51 |
| harlowja | pretty sure every application out there needs to do something like this to turn off in reasonable time :-P | 23:51 |
| xgerman | well, to drain you can just disconnect from the queue | 23:51 |
| harlowja | so i'm just theorizing, how do u detect that your app is turning off and how does the rest of the components in your app react to that | 23:52 |
| harlowja | be those components threads, child processes, other | 23:52 |
| harlowja | *or coroutines | 23:52 |
| xgerman | is this per node? or for the whole set up? | 23:52 |
| harlowja | everywhere an app runs :-P | 23:52 |
| johnsom | UnknownConnectionError: Unexpected exception for http://172.21.21.135/identity/v3/auth/tokens: Invalid return character or leading space in header: X-Subject-Token | 23:53 |
| rm_work | johnsom: what is wrong with your stuff, lol | 23:53 |
| harlowja | so that would i assume be everywhere, lol | 23:53 |
| rm_work | T_T | 23:53 |
| xgerman | ok, an rm -rf for Octavia | 23:53 |
| xgerman | kill all, nova delete all ? | 23:53 |
| johnsom | rm_work It's too many people messing with too much stuff in other projects with little regard to the impact. | 23:54 |
| harlowja | maybe its better to just start with how does an standalone app shutdown correctly under crash (without leaving poop in places), or under non-crash (where it should also not leave poop places) | 23:55 |
| harlowja | lol | 23:55 |
| harlowja | i think a common solution is u end up with something like taskflow (but i might be wrong), lol | 23:55 |
| rm_work | harlowja: if i asked you how to cook a steak, would you start with "well, first import taskflow"? :P | 23:55 |
| harlowja | or u end up with callback spaghetti (and the runtime can just stop running callbacks for u, likely want node.js does) | 23:56 |
| harlowja | *want = what | 23:56 |
| rm_work | yeah I definitely wouldn't want to end up with spaghetti when cooking a steak! :P | 23:56 |
| johnsom | rm_work lol! | 23:56 |
| johnsom | "well, first import taskflow" OMG I needed that laugh | 23:57 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Implement sorting and pagination for octavia https://review.openstack.org/382147 | 23:57 |
| rm_work | ^^ just rebase | 23:57 |
| rm_work | johnsom: glad I can be of assistance :P | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!