| imacdonn | guess we'll have to "agree to disagree" on that part .. I'll see what the RDO guys have to say about it | 00:01 |
|---|---|---|
| rm_work | yep | 00:01 |
| rm_work | to me this is extremely unambiguous | 00:02 |
| rm_work | i have a feeling the RDO packagers are aware of this as well | 00:02 |
| rm_work | so again, i have a feeling something else is going on here | 00:02 |
| imacdonn | OK.. will see | 00:02 |
| rm_work | this isn't their first rodeo | 00:02 |
| imacdonn | since you're here ... I have another issue that you may comment on ... I think you've see it before | 00:02 |
| imacdonn | from amphora-agent.... [2018-03-05 21:02:45 +0000] [1105] [DEBUG] Invalid request from ip=::ffff:10.250.32.84: [SSL: SSL_SESSION_ID_CALLBACK_FAILED] ssl session id callback failed (_ssl.c:1783) | 00:03 |
| rm_work | hmmm | 00:03 |
| imacdonn | seems to be intermittent .. some amphorae do it, others work OK ... checked clock already .... NTP is good | 00:03 |
| rm_work | which log is that | 00:03 |
| rm_work | yeah clock would have been my first recommendation | 00:03 |
| imacdonn | /var/log/amphora-agent.log ... something like that | 00:03 |
| rm_work | yeah k | 00:04 |
| johnsom | You don’t want to run something different than CI, that means it is untested. Just FYI | 00:04 |
| johnsom | :ffff: seems suspect | 00:05 |
| rm_work | ^^ yes, i mentioned that | 00:05 |
| imacdonn | I think that's just because IPv6 is enabled | 00:06 |
| rm_work | testins should === production, or you're asking for trouble | 00:06 |
| rm_work | this is just a general rule | 00:06 |
| johnsom | Like a mixed ipv5 | 00:06 |
| rm_work | *testing | 00:06 |
| rm_work | yeah that looks like ipv4 in 6 | 00:06 |
| rm_work | but that should be fine | 00:06 |
| rm_work | i think? it's valid notation | 00:06 |
| rm_work | johnsom: why are you here? :P | 00:07 |
| imacdonn | this is troubling too ... I can't delete the listener that failed to create due to the jinja version issue... | 00:08 |
| imacdonn | [imacdonn@home ~]$ openstack loadbalancer listener delete listener1 | 00:08 |
| imacdonn | Load Balancer d3fe6073-4cce-476c-b968-9ed162d91e15 is immutable and cannot be updated. (HTTP 409) (Request-ID: req-be2a84e8-6d81-4097-9021-6691300e0579) | 00:08 |
| rm_work | what status is the LB in? | 00:08 |
| rm_work | did it get stuck in PENDING_UPDATE? | 00:08 |
| rm_work | or did it go to ERROR? | 00:08 |
| imacdonn | provisioning_status=ERROR | 00:08 |
| rm_work | our flows are generally supposed to keep the LB itself in ACTIVE and just send the Listener to ERROR so it can be cleaned up | 00:09 |
| rm_work | but something must have happened that was catastrophic | 00:09 |
| rm_work | once the LB itself goes into ERROR, there's not much you can do | 00:09 |
| imacdonn | mysql -e 'drop database octavia;' | 00:09 |
| imacdonn | :P | 00:09 |
| rm_work | I've been investigating ways we could try to do some correction, but... :* | 00:09 |
| rm_work | you can delete the whole LB | 00:09 |
| cgoncalves | o/ | 00:09 |
| rm_work | hey cgoncalves! I did get home finally :P | 00:10 |
| rm_work | it just took 24h+ | 00:10 |
| cgoncalves | imacdonn: catching up with the backlog. want to summarize your pip jinja2 issue? | 00:10 |
| cgoncalves | rm_work: hey! cool, glad to hear you made it safe :) | 00:10 |
| johnsom | Yeah, check your states. You will only get 409 if it is still trying. You should get lb and listener ERROR when it gives up trying, at which point you can delete | 00:10 |
| imacdonn | cgoncalves: I don't want to continue the debate ... but basically RDO Queens comes with python2-jinja2 2.8.1 ... which is not sufficient | 00:11 |
| cgoncalves | rm_work: think positive. there are still people stuck in dublin ;) | 00:11 |
| rm_work | yeah, RDO Queens should be packaging 2.10 which is what u-c prescribes | 00:11 |
| cgoncalves | imacdonn: ok. if octavia/requirements.txt prescribes 2.10 it should be what the Require: should set | 00:12 |
| imacdonn | cgoncalves: it doesn't ... requiremenrts.txt says >= 2.8 | 00:12 |
| cgoncalves | I'll have a look tomorrow (01:12 am here) | 00:12 |
| imacdonn | we've agreed that that's wrong and needs to be fixed | 00:12 |
| rm_work | yes | 00:12 |
| rm_work | but also if RDO doesn't package according to u-c then things will get really wonky when you use multiple projects | 00:12 |
| cgoncalves | imacdonn: alright. I'll check and submit a patch to RDO | 00:12 |
| rm_work | the whole point of g-r and u-c is to make it so multiple projects can be installed in the same system | 00:13 |
| johnsom | rm_work sipping sherry overlooking the lake from the castle. No worries on the PTO. Just trying to help imacdonn while I can / up. | 00:13 |
| rm_work | even without virtualenvs | 00:13 |
| imacdonn | cgoncalves: OK. I can file an RHBZ on it | 00:13 |
| rm_work | cgoncalves: we should update it upstream | 00:13 |
| rm_work | and backport | 00:13 |
| cgoncalves | imacdonn: much appreciated! assigned it to me (cgoncalves@redhat.com) | 00:13 |
| rm_work | and then rdo should pull that in | 00:13 |
| imacdonn | cgoncalves: will do - thanks! | 00:13 |
| rm_work | i would have done it already but i'm in the middle of a patch | 00:13 |
| johnsom | rm_work: sushi burrito this time? | 00:14 |
| cgoncalves | rm_work: not following. why is it needed to fix upstream? requirements.txt is not following u-c? | 00:14 |
| rm_work | cgoncalves: requirements.txt is supposed to be representative of the minimum version our project requires to function | 00:15 |
| rm_work | u-c is supposed to be used in conjunction when installing things | 00:15 |
| rm_work | what Openstack C-I tests (and thus absolutely what should be packaged/installed) is a combination of the two handled by pip | 00:15 |
| rm_work | so normally, pip would parse >=2.8 as "newest" | 00:16 |
| rm_work | and u-c would cap that at 2.10 | 00:16 |
| rm_work | meaning 2.10 is the version we test, even if 2.11 is out | 00:16 |
| cgoncalves | ah, i see the problem | 00:17 |
| cgoncalves | limitations of .spec | 00:17 |
| cgoncalves | Jinja2!=2.9.0,!=2.9.1,!=2.9.2,!=2.9.3,!=2.9.4,>=2.8 # BSD License (3 clause) | 00:17 |
| rm_work | and regardless of what our requirements says, we expect to be installed with 2.10 | 00:17 |
| rm_work | yes, so it has >=2.8 | 00:17 |
| rm_work | which as of queens release in pypi would have been 2.10 | 00:17 |
| cgoncalves | yeah we cannot add ANDs in Requires: | 00:17 |
| rm_work | so there's a thing that you can run i think that will generate an exact package list | 00:18 |
| johnsom | Yes, requirement repo has a tool | 00:18 |
| cgoncalves | https://github.com/rdo-packages/octavia-distgit/blob/rpm-master/openstack-octavia.spec#L109 | 00:18 |
| cgoncalves | johnsom: looking for trouble with your wife? if not, disconnect immediately :P | 00:20 |
| rm_work | i think this would do it: | 00:20 |
| rm_work | generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 -b blacklist.txt -r requirements.txt > real-constraints.txt | 00:20 |
| rm_work | need to test | 00:20 |
| rm_work | i just kinda mangled their example | 00:20 |
| rm_work | where requirements.txt is our reqs file and the rest is from the requirements repo | 00:20 |
| rm_work | let me see what that does | 00:20 |
| johnsom | cgoncalves: yeah, technically it is her birthday here, but don’t expect me around tomorrow. She is reading now, so safe for a quick irc checkin | 00:21 |
| cgoncalves | lol | 00:22 |
| imacdonn | cgoncalves: what would be the right "Component" for this? There is no "openstack-octavia" .. is there something for general packaging issues ? | 00:23 |
| imacdonn | "distribution" ? | 00:23 |
| rm_work | yeah, as this is not an octavia specific issue | 00:24 |
| rm_work | if RDO didn't package jinja 2.10, then it is wrong for all of openstack | 00:24 |
| rm_work | not just us | 00:24 |
| cgoncalves | imacdonn: product: red hat openstack; component: openstack-octavia | 00:24 |
| rm_work | yeah, it looks like that command works | 00:25 |
| imacdonn | cgoncalves: Oh, I was going for RDO .... not sure if I can file bugs againts RHOS | 00:25 |
| rm_work | generate-constraints -p /usr/bin/python2.7 -b blacklist.txt -r ../octavia/requirements.txt > real-requirements.txt | 00:25 |
| rm_work | ^^ this generated me a list of the actual things you should install for octavia to work | 00:25 |
| cgoncalves | imacdonn: anyone can | 00:25 |
| rm_work | https://gist.github.com/rm-you/8178ae242ed7606d58ec23efd648b415 | 00:26 |
| rm_work | cgoncalves: so RDO packaging should run that first | 00:26 |
| rm_work | then use the resulting requirements file | 00:26 |
| rm_work | (if your packaging stuff can't handle u-c natively) | 00:27 |
| johnsom | Be sure to checkout queens requirements repo and not master | 00:27 |
| cgoncalves | rm_work: ^ that's how octavia/requimrents.txt should look like? | 00:27 |
| rm_work | ah right whoops | 00:27 |
| rm_work | cgoncalves: err | 00:27 |
| rm_work | yes and no | 00:27 |
| rm_work | no in repo | 00:27 |
| rm_work | yes for anything packaging | 00:27 |
| cgoncalves | rm_work: k. I'll handle it tomorrow with fresh eyes | 00:28 |
| *** fnaval has joined #openstack-lbaas | 00:30 | |
| rm_work | kk | 00:30 |
| johnsom | Night from Galway | 00:32 |
| imacdonn | 'nite johnsom | 00:32 |
| imacdonn | thanks for the debate ;) | 00:33 |
| rm_work | night! | 00:33 |
| imacdonn | cgoncalves: Could you grab https://bugzilla.redhat.com/show_bug.cgi?id=1551821 ? I don't think I can assign it | 00:34 |
| cgoncalves | imacdonn: done | 00:37 |
| imacdonn | cgoncalves: thanks! | 00:38 |
| *** yamamoto has quit IRC | 00:40 | |
| rm_work | cgoncalves: but we SHOULD update in repo to require >= 2.10 I think? if possible | 00:41 |
| rm_work | the issue being I don't know if we can get out-of-sync with global-requirements | 00:41 |
| rm_work | we might have to ask g-r to update to simply require >=2.10 | 00:41 |
| rm_work | as i think it is required for us to match it exactly | 00:41 |
| *** yamamoto has joined #openstack-lbaas | 00:42 | |
| cgoncalves | rm_work: yeah I think so too. as it is now our requirements.txt looks bogus | 00:44 |
| cgoncalves | rm_work: I mean, it should have been automatically updated from g-r (i.e. patch by openstack proposal bot) | 00:46 |
| rm_work | err | 00:46 |
| rm_work | i'm not sure if g-r has different than us | 00:46 |
| rm_work | yeah https://github.com/openstack/requirements/blob/master/global-requirements.txt#L84 | 00:46 |
| rm_work | which is why ours is what it is | 00:47 |
| rm_work | because it is derived from that | 00:47 |
| rm_work | so we may need to update g-r | 00:47 |
| cgoncalves | "Packagers are advised to use the versions (upper-constraints)" | 00:48 |
| cgoncalves | rm_work: we could propose bumping it, yeah | 00:51 |
| *** fnaval has quit IRC | 00:51 | |
| rm_work | yes | 00:51 |
| rm_work | so we do need to update g-r | 00:51 |
| rm_work | as our line has to match theirs | 00:51 |
| *** fnaval has joined #openstack-lbaas | 00:51 | |
| rm_work | but they don't allow backports | 00:51 |
| rm_work | so we can fix it | 00:51 |
| rm_work | but | 00:51 |
| rm_work | you'll have to backport internally or else fix RDO's build process to use u-c | 00:52 |
| rm_work | the latter is the correct | 00:52 |
| rm_work | *more correct approach IMO | 00:52 |
| imacdonn | ;) | 00:52 |
| imacdonn | rm_work: I reproduced the SSL_SESSION_ID_CALLBACK_FAILED issue, if you have any other thoughts / things to check .... | 00:52 |
| rm_work | what were the repro steps | 00:53 |
| imacdonn | create a LB (Active-Passive, in case that matters) | 00:53 |
| imacdonn | observe: | 00:53 |
| imacdonn | 2018-03-06 00:48:29.106 24144 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url https://10.250.34.202:9443/0.5/plug/vip/10.250.34.205 request /usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:256 | 00:53 |
| imacdonn | 2018-03-06 00:48:29.112 24144 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) | 00:53 |
| imacdonn | and amphora-agent.log says: [2018-03-06 00:51:28 +0000] [1122] [DEBUG] Invalid request from ip=::ffff:10.159.210.42: [SSL: SSL_SESSION_ID_CALLBACK_FAILED] ssl session id callback failed (_ssl.c:1783) | 00:53 |
| imacdonn | can also make the error by connecting to port 9443 with openssl s_client | 00:54 |
| *** irenab has quit IRC | 00:55 | |
| cgoncalves | rm_work: right. let's bump to 2.10 first upstream | 00:56 |
| rm_work | yeah | 00:56 |
| rm_work | i can make a g-r patch | 00:56 |
| *** oanson has quit IRC | 00:56 | |
| imacdonn | I should confess that the amphora OS is Oracle Linux .. basically the same as CentOS / RHEL ... it seemed to work OK with Pike once I fixed NTP, but that doesn't seem to be the issue here (I did rebuild it with the new amphora-agent code, but maybe there's something else I need to update...) | 00:56 |
| cgoncalves | rm_work: thanks! add me so that I can follow please | 00:57 |
| rm_work | imacdonn: no worries, i had assumed | 00:57 |
| rm_work | i did review your patch BTW and agree with nir | 00:57 |
| imacdonn | the problem with supporting OL in DIB is that I don't know where to get a suitable base image .... but I haven't really looked | 00:58 |
| rm_work | though i just realized it might be difficult if your image is bot publically available | 00:58 |
| rm_work | yeah lol | 00:58 |
| rm_work | just realized that | 00:58 |
| rm_work | *is not | 00:59 |
| rm_work | which i'm guessing it isn't | 00:59 |
| imacdonn | there are some supposedly OpenStack-ready images out there | 00:59 |
| rm_work | does oracle not produce a cloud-ready image? | 01:01 |
| rm_work | i guess perhaps not | 01:01 |
| imacdonn | that's what I mean | 01:01 |
| rm_work | it doesn't necessarily have to be openstack-ready, it can be just "cloud-ready" | 01:01 |
| rm_work | our image builder can install stuff on it I believe to add cloud-init and such | 01:02 |
| rm_work | oh, though you'd need a base element | 01:02 |
| rm_work | which ... yeah | 01:02 |
| rm_work | THAT can be a complex step | 01:02 |
| rm_work | is it even allowed to be public due to licensing? | 01:02 |
| imacdonn | I think so | 01:03 |
| imacdonn | you just don't get any support | 01:03 |
| rm_work | I always assume I would have to sign away my firstborn and a pint of blood to be able to access anything oracle related | 01:03 |
| *** oanson has joined #openstack-lbaas | 01:03 | |
| imacdonn | https://community.oracle.com/thread/3984084 | 01:04 |
| rm_work | and also that oracle somehow becomes the owner of any hardware i run the software on, and becomes eligable to garnish my wages for license fees | 01:04 |
| imacdonn | Go to | 01:04 |
| imacdonn | https://edelivery.oracle.com | 01:04 |
| imacdonn | search by "Release" | 01:04 |
| imacdonn | start typing "OpenStack" | 01:04 |
| imacdonn | you will see the option to choose "Oracle Linux 7 Virtual Machine Image for Openstack 7.0.0.0.0" | 01:04 |
| rm_work | ah yeah great | 01:04 |
| rm_work | so they ship a qcow2 | 01:04 |
| *** irenab has joined #openstack-lbaas | 01:04 | |
| rm_work | that's A++ | 01:04 |
| imacdonn | yeah, I'll take the wages .. you can keep the firstborn, though | 01:04 |
| rm_work | though i can't get to that page without a userID | 01:05 |
| rm_work | which yeah, oracle, makes sense | 01:05 |
| imacdonn | I tihnk you can get one without paying anything .... in any case, I think it's OK if you have to download the image and make it available to DIB as a local file ... I think I read that the RHEL element works that way | 01:06 |
| rm_work | yes | 01:06 |
| rm_work | you can give DIB a local path to the image via an ENV variable | 01:06 |
| rm_work | which is a valid approach for licensed images | 01:06 |
| imacdonn | I'll try to find some of that "copious free time" that everyone talks about | 01:07 |
| imacdonn | ... to try it | 01:07 |
| rm_work | yeah, i think you'd still need a base element though | 01:08 |
| rm_work | but probably if it's close enough to rhel you could just pretend you're doing RHEL, tell disk-image-builder that, and then pass it the file for the oracle image | 01:09 |
| imacdonn | it should be very similar to the RHEL one .. yeah | 01:09 |
| rm_work | the image-build testing process is really more about "waiting" than having a ton of time to actively do stuff | 01:09 |
| imacdonn | waiting for the next thing to break ;) | 01:10 |
| imacdonn | switching gears again ... I thought I'd turn on debug for the amphora-agent that had the SSL issue... so edited octavia.conf and restarted it ... and now it works! | 01:11 |
| imacdonn | some Googling on the error pointed to PRNG readiness | 01:12 |
| rm_work | hmm | 01:16 |
| rm_work | we really need to just ... make an octavia amp distro | 01:17 |
| rm_work | because this whole "every distro feels a need to have an amp image" thing seems silly to me | 01:17 |
| imacdonn | AmphorOS ? | 01:17 |
| rm_work | it's a minimally functioning little black-box | 01:17 |
| rm_work | it shouldn't matter what distro is on it | 01:17 |
| imacdonn | yeah... that'd make it smaller (quicker to deploy) too | 01:17 |
| rm_work | but i understand that when a distro wants to release a cloud, they have to put a distro they can support throughout | 01:18 |
| rm_work | but honestly | 01:18 |
| rm_work | you could use the ubuntu image with an oracle cloud and if it wasn't a licensing problem, no one would even notice | 01:18 |
| imacdonn | yeah.. I doubt anyone would ever notice, but ................ | 01:18 |
| rm_work | it's just dumb, the distro on the amp has nothing to do with anything | 01:19 |
| rm_work | all it does is complicate stuff because we have to support a ton | 01:19 |
| rm_work | and do different templates and such | 01:19 |
| rm_work | we literally just need a minimal linux kernel, haproxy and keepalived and python compiled for it | 01:19 |
| rm_work | and that's it | 01:19 |
| imacdonn | wonder what it'd take to make it work on CirrOS | 01:20 |
| rm_work | heh | 01:20 |
| rm_work | i have looked at Alpine | 01:20 |
| rm_work | should look at cirros probably... | 01:20 |
| imacdonn | it has cloud-init already | 01:20 |
| imacdonn | never used Alpine | 01:20 |
| rm_work | alpine is a minimal tiny security-focused distro | 01:21 |
| rm_work | but yes, existing cloud-init support and DIB elements would make things trivial | 01:21 |
| rm_work | not sure why i didn't poke at this before | 01:21 |
| rm_work | hmm we might want to look at including https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/cloud-init-datasources in our image builds and explicitly setting only ConfigDrive | 01:27 |
| rm_work | since that's what we rely on | 01:27 |
| rm_work | it might speed up our amp boot times | 01:27 |
| imacdonn | hmm | 01:27 |
| imacdonn | I hadn't noticed you're using ConfigDrive | 01:28 |
| rm_work | err, I believe we rely on that | 01:29 |
| imacdonn | I'm not a huge fan, based on past experience ... it was incompatible with live-migration at one point | 01:29 |
| imacdonn | that's probably mostly fixed now | 01:29 |
| rm_work | hmm, we take them in but it seems optionally... except i'm pretty sure we hardcode stuff in whatever calls that | 01:30 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L94 | 01:30 |
| rm_work | looking | 01:30 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/controller/worker/tasks/compute_tasks.py#L78 | 01:31 |
| rm_work | yep | 01:31 |
| imacdonn | good to know | 01:34 |
| *** b_bezak has joined #openstack-lbaas | 01:36 | |
| *** b_bezak has quit IRC | 01:40 | |
| rm_work | added you guys to the g-r patch i proposed | 01:42 |
| imacdonn | yep, got it .. tnx | 01:42 |
| rm_work | if we can land that, then we will be allowed to (nay, required to -- the patch will be automatic) update our project's requirements | 01:43 |
| rm_work | though i guarantee you'll see comments like "not using upper-constraints is a really bad idea", lol | 01:47 |
| imacdonn | I wonder if the systemd octavia-amphora-agent.service needs to start "After" systemd-random-seed.service | 01:49 |
| imacdonn | although it appears it did anyway, but still failed)... | 01:51 |
| imacdonn | Mar 06 00:48:20 localhost.localdomain systemd[1]: Starting Load/Save Random Seed... | 01:51 |
| imacdonn | Mar 06 00:48:20 localhost.localdomain systemd[1]: Started Load/Save Random Seed. | 01:51 |
| imacdonn | Mar 06 00:48:23 host-10-250-34-202 systemd[1]: Starting OpenStack Octavia Amphora Agent service... | 01:51 |
| imacdonn | Mar 06 00:48:23 host-10-250-34-202 systemd[1]: Started OpenStack Octavia Amphora Agent service. | 01:51 |
| rm_work | we could adjust the "requires" in the template | 01:51 |
| imacdonn | seems it should really wait until cloud-init has finished too .... (?) | 01:52 |
| imacdonn | I notice that the hostname changed after it was started... | 01:53 |
| imacdonn | Mar 06 00:48:24 amphora-48a5cc1b-8898-4d10-8ec6-5c548e235cb6.novalocal systemd-hostnamed[962]: Changed static host name to 'amphora-48a5cc1b-8898-4d10-8ec6-5c548e235cb6.novalocal' | 01:53 |
| imacdonn | Mar 06 00:48:21 localhost.localdomain systemd[1]: Started Initial cloud-init job (pre-networking). | 01:54 |
| imacdonn | Mar 06 00:48:23 host-10-250-34-202 systemd[1]: Starting OpenStack Octavia Amphora Agent service... | 01:54 |
| imacdonn | Mar 06 00:48:23 host-10-250-34-202 systemd[1]: Starting Initial cloud-init job (metadata service crawler)... | 01:54 |
| imacdonn | Mar 06 00:48:23 host-10-250-34-202 systemd[1]: Started OpenStack Octavia Amphora Agent service. | 01:54 |
| imacdonn | Mar 06 00:48:24 host-10-250-34-202 cloud-init[856]: Cloud-init v. 0.7.9 running 'init' at Tue, 06 Mar 2018 00:48:24 +0000. Up 5.21 seconds. | 01:54 |
| imacdonn | wouldn't be surprised if there's a race condition in there somewhere | 01:54 |
| rm_work | i would have guessed that something else we required (like networking) would in turn require the random stuff | 02:00 |
| *** imacdonn has quit IRC | 02:02 | |
| *** imacdonn has joined #openstack-lbaas | 02:02 | |
| imacdonn | based on this log, it doesn't appear to be an issue with the random seed ... at least it's not as simple as the service dependency,,., I'm now wondering if cloud-init is changing something that's causing interference | 02:03 |
| imacdonn | does octavia ship a systemd service description for the agent itself? | 02:05 |
| *** slaweq_ has joined #openstack-lbaas | 02:05 | |
| rm_work | yes | 02:05 |
| imacdonn | I found the ones for haproxy, etc, but not the agent one yet | 02:05 |
| rm_work | https://github.com/openstack/octavia/blob/0949ee1b3c3616dd342732b937370c5d72fe9b51/elements/amphora-agent/install.d/amphora-agent-source-install/amphora-agent.service | 02:06 |
| rm_work | err | 02:06 |
| rm_work | https://github.com/openstack/octavia/blob/master/elements/amphora-agent/install.d/amphora-agent-source-install/amphora-agent.service | 02:06 |
| rm_work | it's in the element | 02:06 |
| imacdonn | ah, right | 02:06 |
| imacdonn | except.,.. | 02:07 |
| imacdonn | [root@amphora-48a5cc1b-8898-4d10-8ec6-5c548e235cb6 ~]# rpm -qf /usr/lib/systemd/system/octavia-amphora-agent.service | 02:07 |
| imacdonn | openstack-octavia-amphora-agent-2.0.0-1.el7.noarch | 02:07 |
| imacdonn | so it's part of the RDO RPM | 02:07 |
| imacdonn | and the RDO one looks slightly different | 02:08 |
| rm_work | pastebin it? | 02:08 |
| imacdonn | https://github.com/rdo-packages/octavia-distgit/blob/rpm-master/octavia-amphora-agent.service | 02:09 |
| *** slaweq_ has quit IRC | 02:10 | |
| rm_work | ah | 02:11 |
| rm_work | not much different | 02:11 |
| imacdonn | yeah | 02:11 |
| imacdonn | nothing that looks relevant | 02:11 |
| imacdonn | FWIW, I'm fairly sure that the DIB stuff doesn't use the one that you posted the link to ... I dissected the CentOS elements (IIRC) and didn't pick up that file | 02:12 |
| rm_work | the dib one exactly uses that one | 02:15 |
| rm_work | cent/rhel/ubuntu all use that one when installing from source | 02:15 |
| imacdonn | k, maybe I missed it | 02:16 |
| rm_work | it comes from https://github.com/openstack/octavia/tree/master/elements/amphora-agent | 02:16 |
| rm_work | which is our only amp element | 02:16 |
| rm_work | which we add explicitly regardless of distro: https://github.com/openstack/octavia/blob/master/diskimage-create/diskimage-create.sh#L340 | 02:16 |
| imacdonn | ahh, but I was installing from the RDO RPM, not source | 02:17 |
| *** slaweq has joined #openstack-lbaas | 02:17 | |
| imacdonn | academic, I suppose | 02:17 |
| imacdonn | now looking at this: | 02:17 |
| imacdonn | # cloud-init normally emits a "cloud-config" upstart event to inform third | 02:18 |
| imacdonn | # parties that cloud-config is available, which does us no good when we're | 02:18 |
| imacdonn | # using systemd. cloud-config.target serves as this synchronization point | 02:18 |
| imacdonn | # instead. Services that would "start on cloud-config" with upstart can | 02:18 |
| imacdonn | # instead use "After=cloud-config.target" and "Wants=cloud-config.target" | 02:18 |
| imacdonn | # as appropriate. | 02:18 |
| *** slaweq has quit IRC | 02:22 | |
| rm_work | hmmm | 02:27 |
| rm_work | so that's in cloud-init? | 02:27 |
| imacdonn | yeah | 02:27 |
| rm_work | so we could change our After in our service to start after cloud-config | 02:27 |
| rm_work | i'd probably be ok with that change | 02:27 |
| imacdonn | yes, that's what I'm trying now | 02:27 |
| dayou | sudo ip netns exec amphora-haproxy ping 10.0.0.7 | 02:28 |
| imacdonn | [imacdonn@home ~]$ openstack loadbalancer show lb6 -f value -c provisioning_status | 02:28 |
| imacdonn | ACTIVE | 02:28 |
| imacdonn | so-far so-good ;) | 02:28 |
| dayou | Anyone knows hwy inside the amphora-haproxy namespace, the ip address of eth1 can't be accessed | 02:29 |
| *** slaweq has joined #openstack-lbaas | 02:30 | |
| rm_work | err | 02:32 |
| rm_work | can you do | 02:32 |
| rm_work | sudo ip netns exec amphora-haproxy ip a | 02:32 |
| rm_work | and pastebin that | 02:32 |
| dayou | http://paste.openstack.org/show/692748/ | 02:34 |
| dayou | This seems like a bug to me | 02:34 |
| *** slaweq has quit IRC | 02:34 | |
| dayou | I also tried in the lab, it's same as on devstack | 02:34 |
| imacdonn | ip netns exec amphora-haproxy ip l set lo up | 02:39 |
| imacdonn | do that, then try your ping again | 02:39 |
| imacdonn | dayou: ^ | 02:39 |
| dayou | worked, :-D | 02:40 |
| dayou | why don't we put lo to up by default? | 02:41 |
| imacdonn | yeah... so, because eth1 is local, it wants to use lo .. dunno why it's not up by default | 02:41 |
| *** slaweq has joined #openstack-lbaas | 02:42 | |
| imacdonn | rm_work: That systemd thing made the agent start after cloud-init as intended .... unfortunately it doesn't solve the SSL problem :( | 02:43 |
| rm_work | :/ | 02:44 |
| rm_work | erm so, dayou, i do not have this problem | 02:45 |
| rm_work | i mean, my `lo` shows DOWN also | 02:46 |
| rm_work | but i have no traffic problems | 02:46 |
| imacdonn | I don't have traffic problems either ... being able to ping the eth1 address from inside the netns would not be required for normal operation | 02:46 |
| dayou | rm_work, The problem I had was try to announce route via bgp inside namespace | 02:47 |
| rm_work | hmmm | 02:47 |
| *** slaweq has quit IRC | 02:47 | |
| dayou | Also I am confused about whether I should annouce route inside the namespace or outside | 02:49 |
| rm_work | ah | 02:49 |
| rm_work | i think it's because | 02:49 |
| *** yamamoto has quit IRC | 02:49 | |
| *** yamamoto has joined #openstack-lbaas | 02:49 | |
| rm_work | we aren't actually supposed to accept traffic to the eth1 address directly | 02:50 |
| rm_work | that's the vrrp_ip | 02:50 |
| rm_work | not the actual VIP | 02:50 |
| rm_work | we bind HAProxy to listen only on the VIP address | 02:50 |
| rm_work | which is directed to that port but not actually that address | 02:50 |
| imacdonn | you can't ping the VIP either .. it's also a local route that wants to go through lo | 02:50 |
| rm_work | i can curl the VIP | 02:51 |
| imacdonn | not sure why ping locally has to do with BGP tho | 02:51 |
| rm_work | but not the vrrp_ip | 02:51 |
| rm_work | didn't try ping | 02:51 |
| rm_work | but we shouldn't respond to ping anyway | 02:51 |
| imacdonn | oh wait, I'm not looking at the VIP .. nevermind | 02:52 |
| rm_work | hmmm pinging the vrrp address doesn't respond, as expected, but for the VIP address it does actually respond for me, just oddly | 02:52 |
| rm_work | probably because of the way our network is routed | 02:52 |
| dayou | it needs the loopback for other traffic such as bgp port 179 | 02:53 |
| imacdonn | rm_work: are you pinging it on the active amphora ? | 02:53 |
| imacdonn | because that, for me, still wants to go through lo | 02:54 |
| imacdonn | [root@amphora-9896a337-6b25-4163-8326-d129c4b9e21f ~]# ip netns exec amphora-haproxy ip r g 10.250.34.212 | 02:54 |
| imacdonn | local 10.250.34.212 dev lo src 10.250.34.216 | 02:54 |
| imacdonn | cache <local> | 02:54 |
| *** slaweq has joined #openstack-lbaas | 02:55 | |
| rm_work | yeah, from within the namespace | 02:55 |
| rm_work | mine shows through eth1 | 02:55 |
| imacdonn | oh, interesting | 02:55 |
| rm_work | but our networking is a little different I think | 02:56 |
| *** slaweq has quit IRC | 02:59 | |
| *** slaweq has joined #openstack-lbaas | 03:07 | |
| *** slaweq has quit IRC | 03:12 | |
| *** fnaval has quit IRC | 03:17 | |
| *** harlowja has quit IRC | 03:18 | |
| *** slaweq has joined #openstack-lbaas | 03:19 | |
| *** slaweq has quit IRC | 03:24 | |
| *** slaweq has joined #openstack-lbaas | 03:32 | |
| *** slaweq has quit IRC | 03:36 | |
| *** slaweq has joined #openstack-lbaas | 03:44 | |
| *** slaweq has quit IRC | 03:49 | |
| *** links has joined #openstack-lbaas | 03:51 | |
| *** slaweq has joined #openstack-lbaas | 03:57 | |
| *** slaweq has quit IRC | 04:01 | |
| *** slaweq has joined #openstack-lbaas | 04:09 | |
| *** slaweq has quit IRC | 04:14 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 04:20 |
| *** slaweq has joined #openstack-lbaas | 04:22 | |
| *** slaweq has quit IRC | 04:26 | |
| *** slaweq has joined #openstack-lbaas | 04:34 | |
| *** sanfern has joined #openstack-lbaas | 04:36 | |
| *** slaweq has quit IRC | 04:39 | |
| *** slaweq has joined #openstack-lbaas | 04:46 | |
| *** slaweq has quit IRC | 04:51 | |
| *** slaweq has joined #openstack-lbaas | 04:59 | |
| *** chkumar246 is now known as chandankumar | 04:59 | |
| *** slaweq has quit IRC | 05:03 | |
| dayou | FYI, needs to rebase https://review.openstack.org/#/c/528850/ and fix merge conflict | 05:09 |
| *** slaweq has joined #openstack-lbaas | 05:11 | |
| *** harlowja has joined #openstack-lbaas | 05:12 | |
| *** slaweq has quit IRC | 05:16 | |
| *** slaweq has joined #openstack-lbaas | 05:24 | |
| *** gcheresh has joined #openstack-lbaas | 05:27 | |
| *** slaweq has quit IRC | 05:29 | |
| *** gcheresh has quit IRC | 05:34 | |
| *** slaweq has joined #openstack-lbaas | 05:36 | |
| *** slaweq has quit IRC | 05:40 | |
| *** slaweq has joined #openstack-lbaas | 05:48 | |
| *** slaweq has quit IRC | 05:53 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia master: ACTIVE-ACTIVE: Initial distributor data model https://review.openstack.org/528850 | 05:57 |
| openstackgerrit | Hengqing Hu proposed openstack/octavia master: L3 ACTIVE-ACTIVE data model https://review.openstack.org/524722 | 05:57 |
| openstackgerrit | Hengqing Hu proposed openstack/octavia master: Make frontend interface attrs less vrrp specific https://review.openstack.org/521138 | 05:57 |
| openstackgerrit | Hengqing Hu proposed openstack/octavia master: Able to set frontend network for loadbalancer https://review.openstack.org/529936 | 05:57 |
| *** slaweq has joined #openstack-lbaas | 06:01 | |
| *** ivve has quit IRC | 06:05 | |
| *** slaweq has quit IRC | 06:05 | |
| openstackgerrit | Min Sun proposed openstack/neutron-lbaas-dashboard master: Cannot update ssl certificate when update listener https://review.openstack.org/549947 | 06:08 |
| *** slaweq has joined #openstack-lbaas | 06:13 | |
| *** threestrands has quit IRC | 06:14 | |
| *** slaweq has quit IRC | 06:18 | |
| *** slaweq has joined #openstack-lbaas | 06:26 | |
| *** harlowja has quit IRC | 06:29 | |
| *** slaweq has quit IRC | 06:30 | |
| *** chandankumar has quit IRC | 06:35 | |
| *** chandankumar has joined #openstack-lbaas | 06:36 | |
| *** slaweq has joined #openstack-lbaas | 06:38 | |
| *** slaweq has quit IRC | 06:43 | |
| *** kobis has joined #openstack-lbaas | 06:52 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 06:52 | |
| *** rcernin has quit IRC | 07:14 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP support with netcat - DIB https://review.openstack.org/538105 | 07:22 |
| *** AlexeyAbashkin has quit IRC | 07:50 | |
| *** slaweq_ has joined #openstack-lbaas | 08:31 | |
| *** tesseract has joined #openstack-lbaas | 08:32 | |
| *** slaweq_ has quit IRC | 08:36 | |
| *** b_bezak has joined #openstack-lbaas | 08:38 | |
| *** oanson has quit IRC | 08:41 | |
| *** irenab has quit IRC | 08:42 | |
| *** pcaruana has joined #openstack-lbaas | 08:44 | |
| *** irenab has joined #openstack-lbaas | 08:49 | |
| *** oanson has joined #openstack-lbaas | 08:50 | |
| *** pcaruana has quit IRC | 08:54 | |
| *** ivve has joined #openstack-lbaas | 09:08 | |
| *** ivve has quit IRC | 09:13 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 09:17 | |
| *** slaweq_ has joined #openstack-lbaas | 09:32 | |
| *** slaweq_ has quit IRC | 09:37 | |
| *** slaweq_ has joined #openstack-lbaas | 09:43 | |
| *** rcernin has joined #openstack-lbaas | 09:45 | |
| *** dims has quit IRC | 09:45 | |
| *** slaweq_ has quit IRC | 09:48 | |
| *** dims has joined #openstack-lbaas | 09:49 | |
| *** slaweq_ has joined #openstack-lbaas | 09:53 | |
| *** sticker has joined #openstack-lbaas | 09:55 | |
| *** slaweq_ has quit IRC | 09:57 | |
| *** slaweq_ has joined #openstack-lbaas | 10:03 | |
| *** slaweq_ has quit IRC | 10:08 | |
| *** salmankhan has joined #openstack-lbaas | 10:12 | |
| *** slaweq_ has joined #openstack-lbaas | 10:13 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia-dashboard master: Being able to change insert headers of listener https://review.openstack.org/549999 | 10:15 |
| *** slaweq_ has quit IRC | 10:18 | |
| *** slaweq_ has joined #openstack-lbaas | 10:23 | |
| rm_work | nmagnezi_: so it's been a few days, what do you think of the haproxy 1.8 on cent7 thing? | 10:28 |
| rm_work | cgoncalves: you too | 10:28 |
| *** slaweq_ has quit IRC | 10:28 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia-dashboard master: Being able to change insert headers of listener https://review.openstack.org/549999 | 10:29 |
| cgoncalves | rm_work: I think it is time for you to go to bed :) | 10:31 |
| rm_work | meh | 10:31 |
| rm_work | i woke up at 3pm | 10:31 |
| cgoncalves | rm_work: I've brought that up yesterday in a sync call | 10:31 |
| rm_work | i'm gonna take another crack at that api config dump thing | 10:32 |
| cgoncalves | rm_work: we will try during coming weeks to have it distributed in the openstack channels/RDO | 10:32 |
| rm_work | sweet | 10:32 |
| rm_work | i can sit on the patch until you can get an official home for it | 10:32 |
| rm_work | so long as it's really happening | 10:33 |
| *** salmankhan has quit IRC | 10:33 | |
| *** salmankhan has joined #openstack-lbaas | 10:33 | |
| *** slaweq_ has joined #openstack-lbaas | 10:33 | |
| cgoncalves | rm_work: it's not confirmed. it wouldn't be the first precedence so I'd say we have good chances | 10:34 |
| cgoncalves | rm_work: it would be okay for me to merge your patch that installs haproxy from paas repo as I have reviewed the .spec and can tell it comes from rhel and other RH folks | 10:35 |
| *** slaweq_ has quit IRC | 10:38 | |
| rm_work | hmmm k | 10:39 |
| rm_work | we can always update the location | 10:39 |
| rm_work | i would like to poke the gate for it a little more | 10:39 |
| rm_work | or let me run it internally for a week or something :P | 10:39 |
| cgoncalves | internally in a production env xD | 10:41 |
| *** slaweq_ has joined #openstack-lbaas | 10:44 | |
| rm_work | ... yes | 10:44 |
| rm_work | hey, I'm fairly confident it'll be fine | 10:44 |
| rm_work | but i'd just smatter in a few LBs | 10:44 |
| *** slaweq_ has quit IRC | 10:49 | |
| *** b_bezak has quit IRC | 10:49 | |
| *** sanfern has quit IRC | 10:53 | |
| *** AlexeyAbashkin has quit IRC | 11:05 | |
| *** salmankhan has quit IRC | 11:10 | |
| *** salmankhan has joined #openstack-lbaas | 11:14 | |
| *** slaweq_ has joined #openstack-lbaas | 11:14 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 11:16 | |
| *** slaweq_ has quit IRC | 11:19 | |
| *** slaweq_ has joined #openstack-lbaas | 11:24 | |
| *** slaweq_ has quit IRC | 11:29 | |
| *** slaweq_ has joined #openstack-lbaas | 11:35 | |
| *** yamamoto has quit IRC | 11:39 | |
| *** slaweq_ has quit IRC | 11:39 | |
| *** slaweq_ has joined #openstack-lbaas | 12:15 | |
| *** slaweq_ has quit IRC | 12:20 | |
| *** slaweq_ has joined #openstack-lbaas | 12:26 | |
| *** slaweq_ has quit IRC | 12:30 | |
| *** slaweq_ has joined #openstack-lbaas | 12:36 | |
| *** yamamoto has joined #openstack-lbaas | 12:40 | |
| *** slaweq_ has quit IRC | 12:41 | |
| *** yamamoto has quit IRC | 12:45 | |
| *** slaweq_ has joined #openstack-lbaas | 12:46 | |
| *** slaweq_ has quit IRC | 12:50 | |
| *** sanfern has joined #openstack-lbaas | 12:53 | |
| *** slaweq_ has joined #openstack-lbaas | 12:56 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements https://review.openstack.org/549551 | 12:57 |
| *** slaweq_ has quit IRC | 13:01 | |
| *** aojea_ has joined #openstack-lbaas | 13:01 | |
| *** atoth has joined #openstack-lbaas | 13:03 | |
| prometheanfire | rm_work: we are working on publishing signed musl images | 13:05 |
| *** slaweq_ has joined #openstack-lbaas | 13:06 | |
| *** slaweq_ has quit IRC | 13:11 | |
| *** slaweq_ has joined #openstack-lbaas | 13:17 | |
| *** slaweq_ has quit IRC | 13:21 | |
| *** slaweq_ has joined #openstack-lbaas | 13:27 | |
| *** slaweq_ has quit IRC | 13:31 | |
| *** slaweq_ has joined #openstack-lbaas | 13:37 | |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas master: [WIP]: Nuke lazy loaders and flush db sessions https://review.openstack.org/549613 | 13:39 |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas master: [WIP]: Nuke lazy loaders and flush db sessions https://review.openstack.org/549613 | 13:40 |
| *** slaweq_ has quit IRC | 13:41 | |
| *** yamamoto has joined #openstack-lbaas | 13:42 | |
| *** fnaval has joined #openstack-lbaas | 13:44 | |
| *** slaweq_ has joined #openstack-lbaas | 13:47 | |
| *** yamamoto has quit IRC | 13:47 | |
| *** fnaval has quit IRC | 13:49 | |
| *** slaweq_ has quit IRC | 13:52 | |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas master: [DNM]: Test CI https://review.openstack.org/550085 | 13:56 |
| *** slaweq_ has joined #openstack-lbaas | 13:57 | |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas master: [DNM]: Test CI https://review.openstack.org/550085 | 14:00 |
| *** sanfern has quit IRC | 14:00 | |
| *** slaweq_ has quit IRC | 14:02 | |
| *** slaweq_ has joined #openstack-lbaas | 14:07 | |
| *** slaweq_ has quit IRC | 14:12 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia master: Provide devstack samples for l3 active active https://review.openstack.org/520878 | 14:16 |
| *** slaweq_ has joined #openstack-lbaas | 14:18 | |
| *** salmankhan has quit IRC | 14:20 | |
| *** slaweq_ has quit IRC | 14:23 | |
| *** fnaval has joined #openstack-lbaas | 14:27 | |
| *** sapd__ has joined #openstack-lbaas | 14:27 | |
| *** salmankhan has joined #openstack-lbaas | 14:28 | |
| *** slaweq_ has joined #openstack-lbaas | 14:28 | |
| *** sapd_ has quit IRC | 14:31 | |
| *** slaweq_ has quit IRC | 14:32 | |
| *** slaweq_ has joined #openstack-lbaas | 14:38 | |
| *** aojea_ has quit IRC | 14:39 | |
| *** slaweq_ has quit IRC | 14:42 | |
| *** yamamoto has joined #openstack-lbaas | 14:43 | |
| *** slaweq_ has joined #openstack-lbaas | 14:48 | |
| *** yamamoto has quit IRC | 14:49 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [WIP] Add grenade support https://review.openstack.org/549654 | 14:50 |
| *** slaweq_ has quit IRC | 14:53 | |
| *** slaweq_ has joined #openstack-lbaas | 14:58 | |
| *** sticker has quit IRC | 15:03 | |
| *** slaweq_ has quit IRC | 15:03 | |
| *** slaweq_ has joined #openstack-lbaas | 15:09 | |
| xgerman_ | cgoncalves: any idea why the pip install doesn’t work for https://review.openstack.org/#/c/549259/ | 15:13 |
| xgerman_ | apt install | 15:13 |
| xgerman_ | maybe should ask the people in infra | 15:13 |
| *** slaweq_ has quit IRC | 15:13 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 15:17 |
| xgerman_ | n.m. was silly | 15:17 |
| *** slaweq_ has joined #openstack-lbaas | 15:19 | |
| nmagnezi_ | xgerman_, johnsom, rm_work, Ziuul shows that our CI for neutron-lbaas seems to be broken (tested with https://review.openstack.org/#/c/550085/ ) | 15:20 |
| xgerman_ | mmh | 15:20 |
| xgerman_ | johnsom is still in Ireland sipping whiskey (I hope) — and my Zuul isn’t strong | 15:21 |
| xgerman_ | but will check | 15:21 |
| dayou | zuul seems to be broken at the moment | 15:22 |
| xgerman_ | ok, then let’s wait - if there are logs I can sift through let me know | 15:23 |
| *** slaweq_ has quit IRC | 15:23 | |
| nmagnezi_ | dayou, is that so? you get bad results for the dashboard patches? | 15:24 |
| *** links has quit IRC | 15:24 | |
| *** slaweq has joined #openstack-lbaas | 15:29 | |
| cgoncalves | nmagnezi_, dayou, xgerman_, johnsom, rm_work: ostestr is no more. I'll put a patch up for review switching to stestr | 15:31 |
| cgoncalves | ref: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122135.html | 15:31 |
| xgerman_ | thx | 15:32 |
| *** slaweq has quit IRC | 15:34 | |
| *** slaweq has joined #openstack-lbaas | 15:39 | |
| *** rcernin has quit IRC | 15:43 | |
| *** slaweq has quit IRC | 15:44 | |
| *** yamamoto has joined #openstack-lbaas | 15:45 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Migrate to stestr https://review.openstack.org/550134 | 15:46 |
| cgoncalves | xgerman_, nmagnezi_: ^ | 15:46 |
| *** slaweq has joined #openstack-lbaas | 15:49 | |
| *** yamamoto has quit IRC | 15:51 | |
| *** slaweq has quit IRC | 15:53 | |
| *** slaweq has joined #openstack-lbaas | 15:54 | |
| *** aojea_ has joined #openstack-lbaas | 15:57 | |
| xgerman_ | let’s wait until it passes Zuul :-) | 16:02 |
| xgerman_ | + we will need a patch for our other 3 projects cgoncalves | 16:02 |
| *** kobis has quit IRC | 16:03 | |
| cgoncalves | xgerman_: on it! :) | 16:03 |
| xgerman_ | sweet | 16:04 |
| cgoncalves | xgerman_: actually we don't. they don't use ostestr | 16:05 |
| xgerman_ | nice. even lbaasv2 | 16:05 |
| xgerman_ | n-lbaas | 16:05 |
| cgoncalves | grep returns no results for 'ostestr' :) | 16:06 |
| cgoncalves | we probably want to switch all to stestr. currently set to run 'python setup.py test [...]' in tox.ini | 16:10 |
| xgerman_ | yeah | 16:15 |
| *** sapd__ has quit IRC | 16:17 | |
| *** sapd__ has joined #openstack-lbaas | 16:17 | |
| *** sapd__ has quit IRC | 16:19 | |
| *** sapd__ has joined #openstack-lbaas | 16:20 | |
| *** aojea_ has quit IRC | 16:23 | |
| *** kevinbenton has quit IRC | 16:41 | |
| *** kevinbenton has joined #openstack-lbaas | 16:43 | |
| *** yamamoto has joined #openstack-lbaas | 16:47 | |
| *** yamamoto has quit IRC | 16:53 | |
| cgoncalves | uh! neutron-lbaasv2-dsvm-api failing with couple of "One or more ports have an IP allocation from this subnet" errors | 16:54 |
| *** AlexeyAbashkin has quit IRC | 16:57 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 17:03 |
| xgerman_ | ouch | 17:03 |
| johnsom | cgoncalves: that is usually the side effect of another failure | 17:12 |
| cgoncalves | johnsom: is your wife distracted reading? xD | 17:13 |
| cgoncalves | ok, I'll take a deeper look. I see same failure on other patches | 17:13 |
| johnsom | Yes actually | 17:13 |
| johnsom | Yeah, those tests will give that on cleanup if something else bombed | 17:14 |
| johnsom | Look i o-cw o-api | 17:15 |
| cgoncalves | will do. thanks for the pointers! | 17:15 |
| *** yamamoto has joined #openstack-lbaas | 17:49 | |
| *** aojea_ has joined #openstack-lbaas | 17:51 | |
| *** yamamoto has quit IRC | 17:54 | |
| rm_work | prometheanfire: cool | 18:08 |
| *** salmankhan has quit IRC | 18:12 | |
| *** salmankhan has joined #openstack-lbaas | 18:15 | |
| xgerman_ | +1 | 18:16 |
| *** salmankhan has quit IRC | 18:20 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:26 | |
| *** tesseract has quit IRC | 18:27 | |
| *** AlexeyAbashkin has quit IRC | 18:30 | |
| *** harlowja has joined #openstack-lbaas | 18:39 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 18:43 |
| imacdonn | rm_work: I *think* I got my SSL problem solved. Seems it was clock after all I noticed that the amphora-agent was starting before ntpd. I made two changes - enabled the "ntpdate" service in addition to "ntpd", and made the agent start "After" time-sync.target ... so-far, it's consistently worked since that | 18:45 |
| *** yamamoto has joined #openstack-lbaas | 18:50 | |
| *** yamamoto has quit IRC | 18:54 | |
| *** aojea_ has quit IRC | 19:00 | |
| rm_work | we need to merge a similar patch in neutron-lbaas | 19:01 |
| rm_work | since we co-gate it in octavia | 19:01 |
| *** ivve has joined #openstack-lbaas | 19:01 | |
| rm_work | or figure out why it's failing | 19:03 |
| cgoncalves | rm_work: are you referring to neutron-lbaasv2-dsvm-api? | 19:04 |
| rm_work | yeah, looks like it's failing on something | 19:04 |
| cgoncalves | rm_work: have you found the issue? | 19:04 |
| rm_work | not yet | 19:04 |
| rm_work | just started looking | 19:04 |
| rm_work | once i realized it wasn't just ostestr | 19:04 |
| cgoncalves | rm_work: issue is _vip port_id still exists | 19:05 |
| cgoncalves | Found port (8e036024-20e5-4fd1-a3ee-66f0efa33c7d, 10.1.0.10) having IP allocation on subnet f7f4dde8-ff91-4c0c-89dd-81a4d114d48c, cannot delete | 19:06 |
| cgoncalves | Calling delete_port for 8e036024-20e5-4fd1-a3ee-66f0efa33c7d owned by neutron:LOADBALANCERV2 {{(pid=13582) delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1566}} | 19:06 |
| rm_work | is that on the neutron side? | 19:07 |
| cgoncalves | yes | 19:07 |
| cgoncalves | http://logs.openstack.org/54/549654/4/check/neutron-lbaasv2-dsvm-api/0eaaf0c/logs/screen-q-svc.txt.gz | 19:07 |
| cgoncalves | hmmm I think we're not waiting for the port deletion to complete | 19:09 |
| cgoncalves | neutron-server[13497]: DEBUG neutron.plugins.ml2.plugin [None req-3e6b133d-4502-46a7-a09d-84baa618a581 None None] Deleting port 8e036024-20e5-4fd1-a3ee-66f0efa33c7d {{(pid=13582) _pre_delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1491}} | 19:10 |
| cgoncalves | ^ here it starts | 19:10 |
| cgoncalves | neutron-server[13497]: DEBUG neutron.api.rpc.handlers.resources_rpc [None req-3e6b133d-4502-46a7-a09d-84baa618a581 None None] Pushing event deleted for resources: {'Port': ['ID=8e036024-20e5-4fd1-a3ee-66f0efa33c7d,revision_number=None']} {{(pid=13582) push /opt/stack/new/neutron/neutron/api/rpc/handlers/resources_rpc.py:241}} | 19:10 |
| cgoncalves | ^ here it notifies port delete completed | 19:10 |
| cgoncalves | but in the middle we try to delete subnet where port is attached | 19:11 |
| rm_work | ah you're looking at a different change than i am | 19:12 |
| rm_work | i was looking at the failure on https://review.openstack.org/#/c/550134/ | 19:12 |
| cgoncalves | rm_work: different patch, same failure :) | 19:13 |
| rm_work | yeah prolly | 19:14 |
| rm_work | hmmm so what changed | 19:17 |
| rm_work | since like, two days ago | 19:17 |
| rm_work | or whenever our last +1 from zuul was | 19:17 |
| cgoncalves | rm_work: shouldn't we be deleting vip_port_id in https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/tests/tempest/v2/api/base.py#L86 ? | 19:21 |
| rm_work | i think it's created as part of the LB? | 19:22 |
| rm_work | so it should be deleted before the LB is finished and DELETED status | 19:22 |
| rm_work | ah | 19:23 |
| rm_work | https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/tests/tempest/v2/api/base.py#L125 | 19:23 |
| rm_work | we're missing a cls._wait_for_load_balancer_status(lb_id) | 19:23 |
| rm_work | after the delete | 19:23 |
| rm_work | we need status to go to DELETED | 19:24 |
| rm_work | we can't just say "delete" and then move on | 19:24 |
| rm_work | i can fix this | 19:24 |
| cgoncalves | rm_work: cool! | 19:24 |
| cgoncalves | rm_work: I think there's some cleanup waiter we could use | 19:24 |
| rm_work | yes | 19:25 |
| rm_work | specifically that one | 19:25 |
| cgoncalves | _wait_for_neutron_port_delete | 19:25 |
| rm_work | no | 19:25 |
| rm_work | cls._wait_for_load_balancer_status(lb_id) | 19:25 |
| rm_work | we just tell it to wait for status DELETED | 19:25 |
| rm_work | (out of the main loop, one sec) | 19:25 |
| rm_work | almost have it already | 19:25 |
| cgoncalves | that would also work :D | 19:26 |
| openstackgerrit | Adam Harwell proposed openstack/neutron-lbaas master: Wait for LB deletes on cleanup https://review.openstack.org/550222 | 19:27 |
| rm_work | ^^ | 19:27 |
| rm_work | err | 19:28 |
| rm_work | damnit | 19:28 |
| rm_work | bad copy/paste | 19:28 |
| openstackgerrit | Adam Harwell proposed openstack/neutron-lbaas master: Wait for LB deletes on cleanup https://review.openstack.org/550222 | 19:28 |
| rm_work | there we go | 19:28 |
| rm_work | though honestly that'd work anyway i think because the deleted=True kinda ends up being the thing that hits | 19:29 |
| cgoncalves | yeah was about to comment on that ;) | 19:29 |
| rm_work | when it is finally deleted, it's not going to come back as status DELETED | 19:29 |
| rm_work | it's going to 404 | 19:29 |
| cgoncalves | rm_work: hmmm operating_status='OFFLINE' too no? | 19:30 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep https://review.openstack.org/549295 | 19:30 |
| rm_work | no | 19:30 |
| rm_work | operating status doesn't matter | 19:30 |
| rm_work | it's a user status | 19:30 |
| cgoncalves | https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/tests/tempest/v2/api/base.py#L198 | 19:30 |
| rm_work | hmmmmmmmmm | 19:31 |
| rm_work | that's a bad check I think | 19:31 |
| rm_work | I should patch that out a little differently | 19:31 |
| cgoncalves | rm_work: one may want to wait for operating_status to toggle to whatever is being desired so in that sense makes sense having it there too | 19:33 |
| cgoncalves | e.g. wait for load balancer to be fully operational | 19:33 |
| rm_work | fixed | 19:34 |
| openstackgerrit | Adam Harwell proposed openstack/neutron-lbaas master: Wait for LB deletes on cleanup https://review.openstack.org/550222 | 19:34 |
| rm_work | updated the review but the bot didn't mention it | 19:34 |
| rm_work | that is what it SHOULD do | 19:34 |
| rm_work | ah there goes the bot | 19:35 |
| cgoncalves | LGTM | 19:35 |
| rm_work | i should check whether the tests that do care about operating_status actually check it explicitly as they should though | 19:36 |
| cgoncalves | I wonder what changed to cause the test failures now | 19:36 |
| rm_work | I could do it kinda "opposite" to maintain backwards compat | 19:36 |
| rm_work | it's dumb, but... | 19:36 |
| rm_work | maybe would be a lighter touch approach | 19:36 |
| rm_work | yeah this should basically always have been an issue | 19:37 |
| openstackgerrit | Adam Harwell proposed openstack/neutron-lbaas master: Wait for LB deletes on cleanup https://review.openstack.org/550222 | 19:38 |
| rm_work | there, fixed it to be backwards compatible | 19:38 |
| cgoncalves | perhaps jobs are running in faster machines now so that the port doesn't get deleted first before LB delete xD | 19:38 |
| rm_work | even though it's a little dumb | 19:38 |
| cgoncalves | neutron-lbaas is in life-support... | 19:38 |
| rm_work | ok one more and i'm done | 19:39 |
| openstackgerrit | Adam Harwell proposed openstack/neutron-lbaas master: Wait for LB deletes on cleanup https://review.openstack.org/550222 | 19:39 |
| cgoncalves | hmmm yeah dumb but backward compatible now | 19:39 |
| xgerman_ | well, we need to make installing Octavia easier… | 19:39 |
| rm_work | because that if/else structure was dumb | 19:39 |
| rm_work | you can depends-on that one | 19:41 |
| rm_work | in the octavia test | 19:41 |
| rm_work | err, the octavia stestr patch | 19:41 |
| rm_work | should work | 19:41 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Migrate to stestr https://review.openstack.org/550134 | 19:42 |
| cgoncalves | ^^ | 19:42 |
| *** yamamoto has joined #openstack-lbaas | 19:50 | |
| xgerman_ | cgoncalves: its 9 pm in Germany. Wonder what your working hours are? | 19:52 |
| xgerman_ | or is that all happening in some Brauhaus ;-) | 19:52 |
| rm_work | well, we caught each other on each side of our working hours, i think :P | 19:53 |
| xgerman_ | likely, I normally aim for a core of 8-3 sometimes start at 6, sometimes end at 5 | 19:54 |
| *** yamamoto has quit IRC | 19:56 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 20:18 |
| rm_work | i aim for a core of 12pm-4pm PST and 12am-4am PST | 20:32 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 20:51 |
| *** yamamoto has joined #openstack-lbaas | 20:52 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep https://review.openstack.org/549295 | 20:52 |
| cgoncalves | xgerman_: heh I could tell you the same :) | 20:57 |
| cgoncalves | rm_work: your patch fixes 3 out of 4 tempest failures | 20:57 |
| *** yamamoto has quit IRC | 20:57 | |
| rm_work | hmm | 20:58 |
| xgerman_ | cgoncalves: I am normally on PST so my team wouldn’t appreciate me switching time zones on them | 20:58 |
| *** gcheresh has joined #openstack-lbaas | 20:59 | |
| rm_work | which one still happened | 20:59 |
| cgoncalves | xgerman_: I see. my excuse is girlfriend being out this week :) | 20:59 |
| cgoncalves | rm_work: lemme check. I was on mobile monitoring :) | 20:59 |
| xgerman_ | those are the most productive weeks :-) | 21:00 |
| rm_work | yeah i mean, my wife and i are in different countries until next week | 21:00 |
| *** aojea_ has joined #openstack-lbaas | 21:01 | |
| cgoncalves | rm_work: neutron_lbaas.tests.tempest.v2.api.test_health_monitor_admin.TestHealthMonitors | 21:01 |
| rm_work | same issue tho? | 21:02 |
| cgoncalves | http://paste.openstack.org/show/693460/ | 21:02 |
| cgoncalves | yeah, same issue it seems | 21:02 |
| rm_work | hmm | 21:02 |
| cgoncalves | py27 and py35 also kaput | 21:03 |
| rm_work | i wonder if there's multiple places that happens | 21:03 |
| rm_work | same failures? | 21:03 |
| cgoncalves | http://logs.openstack.org/22/550222/5/check/openstack-tox-py35/d70dfa1/job-output.txt.gz | 21:03 |
| cgoncalves | broken pipe | 21:03 |
| cgoncalves | tools/pretty_tox.sh: 12: tools/pretty_tox.sh: subunit-trace: not found | 21:04 |
| rm_work | errr | 21:04 |
| rm_work | ah right | 21:04 |
| rm_work | i think that's because that file doesn't get created unless the tests pass | 21:04 |
| rm_work | ... maybe | 21:04 |
| cgoncalves | ok that one is part of ostestr switch to stestr | 21:05 |
| *** aojea_ has quit IRC | 21:05 | |
| *** prometheanfire has quit IRC | 21:10 | |
| rm_work | hmmmm | 21:10 |
| rm_work | this really should take care of it | 21:10 |
| -openstackstatus- NOTICE: The infrastructure team is aware of replication issues between review.openstack.org and github.com repositories. We're planning a maintenance to try and address the issue. We recommend using our official supported mirrors instead located at https://git.openstack.org. | 21:18 | |
| *** gcheresh has quit IRC | 21:23 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 21:35 |
| imacdonn | yay... found another way to get a LB into limbo state (immutable) | 21:44 |
| imacdonn | Just create a LB, then try to create a listener for it, specifying a secret container that the octavia user doesn't have ACL access to | 21:44 |
| *** irenab has left #openstack-lbaas | 21:47 | |
| rm_work | that should reject on the initial create call | 21:47 |
| imacdonn | it didn't | 21:48 |
| rm_work | I thought i had a test specifically for that... | 21:48 |
| * rm_work looks | 21:48 | |
| imacdonn | http://paste.openstack.org/show/693467/ | 21:49 |
| rm_work | yeah we should expect that | 21:50 |
| rm_work | and reject | 21:50 |
| rm_work | how sure how that got through to rpc | 21:50 |
| rm_work | that should reject on the API side | 21:50 |
| imacdonn | shall I create a story for this ? | 21:53 |
| rm_work | hmmm | 21:53 |
| rm_work | ah maybe I am thinking of the n-lbaas side | 21:53 |
| rm_work | tracing through | 21:53 |
| rm_work | and writing a test for it now to see what happens | 21:54 |
| imacdonn | k | 21:54 |
| *** yamamoto has joined #openstack-lbaas | 21:54 | |
| *** threestrands has joined #openstack-lbaas | 21:54 | |
| *** yamamoto has quit IRC | 21:59 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 22:02 |
| *** aojea_ has joined #openstack-lbaas | 22:02 | |
| *** aojea_ has quit IRC | 22:07 | |
| *** aojea_ has joined #openstack-lbaas | 22:12 | |
| *** aojea_ has quit IRC | 22:16 | |
| *** rcernin has joined #openstack-lbaas | 22:20 | |
| *** jdavis has joined #openstack-lbaas | 22:27 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Periodic job to build + publish diskimage https://review.openstack.org/549259 | 22:31 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 22:48 | |
| openstackgerrit | Carlos Goncalves proposed openstack/neutron-lbaas master: Switch to stestr https://review.openstack.org/550282 | 22:49 |
| cgoncalves | rm_work: ^ will hopefully fix the py{27,35} test failures | 22:49 |
| *** AlexeyAbashkin has quit IRC | 22:52 | |
| *** yamamoto has joined #openstack-lbaas | 22:56 | |
| *** yamamoto has quit IRC | 23:02 | |
| *** atoth has quit IRC | 23:03 | |
| *** atoth has joined #openstack-lbaas | 23:04 | |
| *** yamamoto has joined #openstack-lbaas | 23:06 | |
| openstackgerrit | Carlos Goncalves proposed openstack/neutron-lbaas master: Switch to stestr https://review.openstack.org/550282 | 23:10 |
| rm_work | cgoncalves: i think they'll have to be in the same patch | 23:11 |
| rm_work | sinc we'll need one patch that will pass for everything | 23:12 |
| rm_work | otherwise no merging | 23:12 |
| *** slaweq has quit IRC | 23:13 | |
| cgoncalves | rm_work: infinite depends-on loop :D | 23:14 |
| cgoncalves | rm_work: I guess you're right. I can't think of other way to have them separated | 23:15 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Periodic job to build + publish diskimage https://review.openstack.org/549259 | 23:16 |
| cgoncalves | d'oh! stupid .gitignore :P | 23:20 |
| openstackgerrit | Carlos Goncalves proposed openstack/neutron-lbaas master: Switch to stestr https://review.openstack.org/550282 | 23:22 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Periodic job to build + publish diskimage https://review.openstack.org/549259 | 23:24 |
| rm_work | cgoncalves: either copy in my change to yours and add a Co-Authored-By line for me | 23:47 |
| rm_work | or copy your change into mine and add a Co-Authored-By line for you | 23:47 |
| rm_work | and abandon one of the two | 23:47 |
| *** jdavis has quit IRC | 23:48 | |
| rm_work | we're going to need them to be the same change | 23:48 |
| cgoncalves | rm_work: agreed, I can do that | 23:48 |
| cgoncalves | py27 and py35 are suceeding with my latest patch set btw | 23:48 |
| openstackgerrit | Carlos Goncalves proposed openstack/neutron-lbaas master: Switch to stestr and wait for LB delete on cleanup https://review.openstack.org/550282 | 23:52 |
| *** fnaval has quit IRC | 23:53 | |
| cgoncalves | rm_work: ^ | 23:53 |
| rm_work | k | 23:55 |
| rm_work | still need to figure out how that port is persisting tho | 23:55 |
| cgoncalves | I need to go now. I'll catch up in 7h-ish | 23:56 |
| imacdonn | pfft .. part-timers | 23:57 |
| cgoncalves | :D | 23:57 |
| imacdonn | :) g'nite | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!