| *** tongl has quit IRC | 00:45 | |
| *** http_GK1wmSU has joined #openstack-neutron-ovn | 02:53 | |
| *** http_GK1wmSU has left #openstack-neutron-ovn | 02:56 | |
| *** anilvenkata has joined #openstack-neutron-ovn | 03:39 | |
| *** janki has joined #openstack-neutron-ovn | 05:12 | |
| openstackgerrit | venkata anil proposed openstack/networking-ovn master: Fix gate by skipping neutron test https://review.openstack.org/489966 | 05:21 |
|---|---|---|
| *** jchhatbar has joined #openstack-neutron-ovn | 05:34 | |
| *** Guest14 has joined #openstack-neutron-ovn | 05:36 | |
| *** janki has quit IRC | 05:36 | |
| *** Guest14 has quit IRC | 05:39 | |
| *** mmirecki has joined #openstack-neutron-ovn | 06:16 | |
| openstackgerrit | venkata anil proposed openstack/networking-ovn master: Fix gate by skipping neutron test https://review.openstack.org/489966 | 06:50 |
| *** pcaruana has joined #openstack-neutron-ovn | 07:38 | |
| *** yamamoto has quit IRC | 08:37 | |
| *** yamamoto has joined #openstack-neutron-ovn | 08:50 | |
| *** lucas-afk is now known as lucasagomes | 08:55 | |
| lucasagomes | dalvarez, I will look into the functional tests failures. Or, are you into it ? | 08:56 |
| dalvarez | lucasagomes, anilvenkata is i think | 09:00 |
| dalvarez | but i can help | 09:00 |
| dalvarez | im busy with some stuff but i can help :) | 09:00 |
| anilvenkata | lucasagomes, I am looking into them | 09:00 |
| anilvenkata | dalvarez, ^ | 09:01 |
| lucasagomes | right on, anilvenkata if you need any help lemme know | 09:01 |
| anilvenkata | lucasagomes, sure | 09:01 |
| dalvarez | anilvenkata, i saw your last comment :( | 09:01 |
| anilvenkata | dalvarez, thanks :) | 09:01 |
| *** Guest14 has joined #openstack-neutron-ovn | 09:28 | |
| *** anilvenkata is now known as anilvenkata|LUNC | 09:36 | |
| openstackgerrit | Miguel Angel Ajo proposed openstack/networking-ovn master: refarch: Update documentation and diagrams https://review.openstack.org/490812 | 09:52 |
| Guest14 | numans ^ I saw that we also need to update the details about DHCP :) this is talking about dhcp agent in many places :D | 09:53 |
| *** Guest14 is now known as ajo | 09:53 | |
| ajo | yikes | 09:53 |
| numans | ajo, thanks for the patch - I agree | 10:00 |
| *** anilvenkata|LUNC is now known as anilvenkata | 10:02 | |
| *** janki has joined #openstack-neutron-ovn | 10:11 | |
| *** mmirecki is now known as mmirecki_lunch | 10:12 | |
| *** jchhatbar has quit IRC | 10:12 | |
| *** openstackgerrit has quit IRC | 10:18 | |
| *** anilvenkata has quit IRC | 10:20 | |
| *** openstackgerrit has joined #openstack-neutron-ovn | 10:26 | |
| openstackgerrit | Daniel Alvarez proposed openstack/networking-ovn master: Make Metadata agent independent from other config files https://review.openstack.org/490822 | 10:27 |
| *** jchhatbar has joined #openstack-neutron-ovn | 10:27 | |
| dalvarez | ajo, ^ | 10:28 |
| dalvarez | ajo ^ | 10:28 |
| *** janki has quit IRC | 10:29 | |
| *** yamamoto has quit IRC | 10:32 | |
| *** yamamoto has joined #openstack-neutron-ovn | 10:33 | |
| *** yamamoto has quit IRC | 10:38 | |
| openstackgerrit | Daniel Alvarez proposed openstack/networking-ovn master: Rename metadata proxy config dir https://review.openstack.org/490827 | 10:39 |
| *** yamamoto has joined #openstack-neutron-ovn | 10:48 | |
| ajo | dalvarez yummy | 10:56 |
| *** jchhatbar has quit IRC | 11:06 | |
| openstackgerrit | Numan Siddique proposed openstack/networking-ovn master: Fix gate issues https://review.openstack.org/489966 | 11:14 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Idea proposal: Neutron/OVN database consistency problem https://review.openstack.org/490834 | 11:23 |
| *** lucasagomes is now known as lucas-hungry | 11:31 | |
| *** yamamoto has quit IRC | 11:39 | |
| *** janki has joined #openstack-neutron-ovn | 11:44 | |
| *** mmirecki_lunch is now known as mmirecki | 11:44 | |
| openstackgerrit | Miguel Angel Ajo proposed openstack/networking-ovn master: Make the metadata support work on multinode https://review.openstack.org/490568 | 11:46 |
| dalvarez | ajo, sry i was having lunchhhh | 11:53 |
| numans | dalvarez, with my hijacked patch approach - functional tests for py27 are passing | 12:14 |
| dalvarez | numans, PS7' | 12:14 |
| dalvarez | ? | 12:14 |
| numans | dalvarez, but failing for py34 for some other reason - which i think can be fixed :) - http://logs.openstack.org/66/489966/7/check/gate-networking-ovn-dsvm-functional-py35/5f27484/ | 12:14 |
| numans | dalvarez, yes | 12:14 |
| dalvarez | numans, 2017-08-04 11:25:11.419792 | 2017-08-04 11:25:11.419 | b"TypeError: can't pickle dict_keys objects" | 12:15 |
| dalvarez | ack | 12:15 |
| numans | dalvarez, not a python expert - you think it could be handled :) | 12:16 |
| dalvarez | numans, but PS7 sticks to OVS 2.7 | 12:16 |
| numans | dalvarez, yes for functional tests | 12:17 |
| numans | dalvarez, right now we are in a tricky situation - if we take anil's p6 patch, it breaks the ovs-release job | 12:17 |
| dalvarez | numans, yeah not sure what we want to do... i like Anil's patch but we had a look at that together so it doesn't count much | 12:18 |
| numans | dalvarez, the right approach is handle the db or release versioning properly in the code , but that's not straight forward | 12:18 |
| dalvarez | yes | 12:18 |
| dalvarez | let me take a look at the python thing | 12:18 |
| numans | dalvarez, so i think we can switch to ovs 2.7 for functional tests | 12:19 |
| numans | dalvarez, ++ | 12:19 |
| dalvarez | numans, i dont see why py3 is failing now | 12:23 |
| dalvarez | this patch doesn't have to do with the deepcopy | 12:24 |
| numans | dalvarez, yeah | 12:24 |
| dalvarez | it's some other patch recently merged in neutron?! | 12:24 |
| numans | dalvarez, but how that is related to neutron ? | 12:24 |
| dalvarez | i mean.. that should be failing in neutron as well.. dont know | 12:24 |
| numans | dalvarez, ok | 12:25 |
| numans | dalvarez, yeah you are right | 12:26 |
| numans | dalvarez, strange | 12:26 |
| dalvarez | weird! | 12:26 |
| numans | bizzare | 12:26 |
| dalvarez | numans, im trying to reproduce it locally by calling modify_fields_to_db | 12:26 |
| numans | dalvarez, cool | 12:26 |
| numans | i am out of adjectives now :) | 12:26 |
| dalvarez | LOL | 12:27 |
| dalvarez | numans, hahaha | 12:27 |
| dalvarez | numans++ | 12:27 |
| dalvarez | numans, https://bugs.launchpad.net/networking-ovn/+bug/1666113 | 12:36 |
| openstack | Launchpad bug 1666113 in networking-ovn "In python3.5 environment, synchronization router exception" [Undecided,Fix released] - Assigned to Guoshuai Li (liguoshuai1990) | 12:36 |
| dalvarez | numans, i think that fixed it for floating ips but for some reason it breaks now for us... | 12:37 |
| dalvarez | but trying to understand what does our patch change... | 12:38 |
| *** yamamoto has joined #openstack-neutron-ovn | 12:40 | |
| *** yamamoto has quit IRC | 12:45 | |
| numans | dalvarez, ok | 12:45 |
| dalvarez | numans, https://github.com/openstack/neutron/commit/2ca6b5bce67bad0cd5d59aefda2c7e4b6aaaeda0 | 12:48 |
| dalvarez | this looks like it broke it | 12:48 |
| dalvarez | numans, can you run py3 tests locally? | 12:48 |
| dalvarez | so that we can remove the list() and try? | 12:48 |
| numans | dalvarez, worth a try | 12:48 |
| dalvarez | man we're depending so much on neutron tests :( | 12:48 |
| dalvarez | numans, if you have it I can try | 12:48 |
| *** anilvenkata has joined #openstack-neutron-ovn | 12:49 | |
| numans | dalvarez, yeah | 12:49 |
| numans | dalvarez, you want to try where ? | 12:49 |
| dalvarez | lucas-hungry, numans ajo: https://review.openstack.org/#/c/424154/ | 12:49 |
| dalvarez | this is what broke it :) | 12:50 |
| dalvarez | ^ | 12:50 |
| *** lucas-hungry is now known as lucasagomes | 12:50 | |
| dalvarez | numans, i mean if you have some environment ready to run the functional tests and try removing the 'list()' there | 12:50 |
| numans | dalvarez, ok. got it | 12:50 |
| numans | dalvarez, i guess need to install py35 | 12:50 |
| * dalvarez reporting the bug | 12:50 | |
| dalvarez | numans, yup | 12:50 |
| lucasagomes | dalvarez, oh, lemme take a look | 12:50 |
| dalvarez | numans, lucasagomes i think we want to try without the 'list' here: https://review.openstack.org/#/c/424154/19/neutron/db/l3_db.py@1606 | 12:51 |
| lucasagomes | dalvarez, :-( another failure then | 12:51 |
| dalvarez | like the old code | 12:51 |
| dalvarez | yeah man | 12:51 |
| dalvarez | it's crazy lately | 12:51 |
| lucasagomes | :-/ | 12:52 |
| dalvarez | well not sure if without the 'list' there | 12:52 |
| dalvarez | it changed completely | 12:52 |
| numans | dalvarez, testing with python 3.4 (couldn't get py35 for centos from epel) | 12:54 |
| numans | dalvarez, are you suggesting to remove list from ovn_db_sync.py ? | 12:54 |
| numans | dalvarez, i.e to rever the bug 1666113 fix ? | 12:55 |
| openstack | bug 1666113 in networking-ovn "In python3.5 environment, synchronization router exception" [Undecided,Fix released] https://launchpad.net/bugs/1666113 - Assigned to Guoshuai Li (liguoshuai1990) | 12:55 |
| dalvarez | numans, lucasagomes not sure, look: | 12:55 |
| dalvarez | if i understood the code correctly this is what it's getting done: | 12:55 |
| dalvarez | http://paste.openstack.org/show/617535/ | 12:56 |
| dalvarez | numans, lucasagomes ^ it works for me regardless of the contents of device owners apparently | 12:57 |
| dalvarez | im running that script with py3.5 | 12:57 |
| dalvarez | >>> modify_fields_to_db({'port_type': owners}) | 12:57 |
| dalvarez | {'port_type': ['network:router_interface', 'network:router_gateway']} | 12:57 |
| dalvarez | same without 'list' | 12:57 |
| dalvarez | numans, lucasagomes also with kwargs: http://paste.openstack.org/show/617536/ | 13:01 |
| openstackgerrit | Numan Siddique proposed openstack/networking-ovn master: Fix gate issues https://review.openstack.org/489966 | 13:01 |
| * lucasagomes looks | 13:04 | |
| lucasagomes | dalvarez, on a side note, thanks for the review on the spec! Very useful, I've answer that | 13:04 |
| lucasagomes | answered* | 13:04 |
| dalvarez | lucasagomes, thanks for that! :) i enjoyed reading it | 13:05 |
| dalvarez | you did a really awesome job with the journaling and now rethiking the whole thing again | 13:05 |
| lucasagomes | dalvarez, you, numans, anil and others helped a lot ! | 13:06 |
| * lucasagomes is trying to find the python3.5 error | 13:06 | |
| * numans still couldn't look into it :( | 13:06 | |
| dalvarez | lucasagomes, if you take a look at my last pastebin where i tried to isolate the call | 13:07 |
| dalvarez | it should work | 13:07 |
| dalvarez | numans, i think your new PS will do nothing :P | 13:07 |
| dalvarez | but it's ok, i'll take care of it hopefully | 13:07 |
| numans | dalvarez, :( | 13:07 |
| dalvarez | numans, <3 | 13:07 |
| numans | dalvarez, i couldn't run it locally | 13:07 |
| lucasagomes | dalvarez, numans ins't the problem here the db_router.keys() from https://github.com/openstack/networking-ovn/blob/master/networking_ovn/ovn_db_sync.py#L349 ? | 13:13 |
| dalvarez | yeah trying that now ;P | 13:14 |
| lucasagomes | because that would return a iterable in python3 ? | 13:14 |
| dalvarez | i was on it | 13:14 |
| dalvarez | i think so | 13:14 |
| lucasagomes | if we just add a list() around that | 13:14 |
| lucasagomes | it should work I believe | 13:14 |
| lucasagomes | dalvarez, cool, try updating that patch of yours: https://review.openstack.org/#/c/489966/ with it | 13:15 |
| dalvarez | yeah! | 13:15 |
| dalvarez | that'll make it | 13:15 |
| dalvarez | >>> routers={'a': 1, 'b': 2} | 13:16 |
| dalvarez | >>> fun(None, router_ids=routers.keys(), port_type=owners) | 13:16 |
| dalvarez | Traceback (most recent call last): | 13:16 |
| dalvarez | File "<stdin>", line 1, in <module> | 13:16 |
| dalvarez | File "<stdin>", line 2, in fun | 13:16 |
| dalvarez | File "<stdin>", line 2, in modify_fields_to_db | 13:16 |
| dalvarez | File "/usr/lib64/python3.5/copy.py", line 155, in deepcopy | 13:16 |
| dalvarez | y = copier(x, memo) | 13:16 |
| dalvarez | File "/usr/lib64/python3.5/copy.py", line 243, in _deepcopy_dict | 13:16 |
| dalvarez | y[deepcopy(key, memo)] = deepcopy(value, memo) | 13:16 |
| dalvarez | File "/usr/lib64/python3.5/copy.py", line 174, in deepcopy | 13:16 |
| dalvarez | rv = reductor(4) | 13:16 |
| dalvarez | TypeError: can't pickle dict_keys objects | 13:16 |
| dalvarez | opss sry | 13:16 |
| dalvarez | lucasagomes, ^ that's it :) | 13:16 |
| dalvarez | with a list around that it works | 13:16 |
| numans | dalvarez, http://logs.openstack.org/66/489966/8/check/gate-networking-ovn-dsvm-functional-py35/afeadb5/logs/testr_results.html.gz | 13:17 |
| numans | dalvarez, it passed now | 13:17 |
| lucasagomes | dalvarez, cool! | 13:17 |
| openstackgerrit | Numan Siddique proposed openstack/networking-ovn master: Fix gate issues https://review.openstack.org/489966 | 13:18 |
| dalvarez | yeah! | 13:18 |
| dalvarez | :) | 13:18 |
| lucasagomes | dalvarez, btw, there's a pep8 error in that patch... also, would be good to squash anilvenkata's fix onto it too ? | 13:18 |
| numans | dalvarez, but i forgot to run pep8 locally and it failed in CI :) | 13:18 |
| lucasagomes | or leave as a separated thing ? | 13:18 |
| anilvenkata | lucasagomes, I will squash that | 13:18 |
| lucasagomes | I mean instead of skipping that test, because now we know what's the real problem | 13:18 |
| lucasagomes | anilvenkata, o/ thanks | 13:18 |
| anilvenkata | wc :) | 13:18 |
| dalvarez | lucasagomes, yeah we dont need to skip it anymore | 13:20 |
| dalvarez | we can squash it and add the two bug numbers | 13:20 |
| dalvarez | plumbling friday anilvenkata lucasagomes numans :) | 13:20 |
| dalvarez | plumbing* | 13:20 |
| ajo | you're the best, guys | 13:21 |
| lucasagomes | dalvarez, awesome! Thanks a lot | 13:22 |
| dalvarez | lucasagomes, anilvenkata numans \o/ | 13:23 |
| dalvarez | anilvenkata, are you squashing it? if so can you also please add "Closes-Bug: #1708660" to the commit message? | 13:27 |
| openstack | bug 1708660 in networking-ovn "functional tests failing for py35 due to a recent neutron patch" [Undecided,New] https://launchpad.net/bugs/1708660 - Assigned to Daniel Alvarez (dalvarezs) | 13:27 |
| anilvenkata | dalvarez, ok | 13:27 |
| dalvarez | anilvenkata++ thanks a lot! | 13:27 |
| anilvenkata | dalvarez, lucasagomes I need a suggestion, As you know, any updates to DB from ovn_client.py will go through commands.py. So I am thinking of adding the db checks( i.e if these new columns r present or not ) | 13:30 |
| anilvenkata | ajo, numans | 13:30 |
| ajo | whot :D | 13:31 |
| anilvenkata | ajo, numans dalvarez lucasagomes where to add them? in commands.py or ovn_client.py | 13:31 |
| dalvarez | anilvenkata, so you want to fix the schema thing and not run against 2.7? | 13:31 |
| lucasagomes | anilvenkata, that's about the name/severity thingie ? | 13:31 |
| ajo | I think it is | 13:31 |
| dalvarez | anilvenkata, i think i'd do it in commands.py, ovn_client doesn't know much about the db schema so it's gonna be more hidden in commands.py | 13:31 |
| anilvenkata | ajo, issue is to new columns added in ovn master name/severity which are not present in 2.7 | 13:31 |
| dalvarez | commands.py knows about the database schema | 13:32 |
| dalvarez | anilvenkata, numan's suggestion was to skip it for now until we have green light in our CI | 13:32 |
| ajo | well we may want to go back to 2.8 asap, it's needed for l3ha ;D | 13:32 |
| anilvenkata | dalvarez, but I have to change functional and unit tests then | 13:32 |
| dalvarez | and then address it but if you feel comfortable doing that that'd be fine :) | 13:32 |
| dalvarez | anilvenkata, only switching to 2.7 in functional testing should suffice right? | 13:32 |
| anilvenkata | dalvarez, but we need permanent solution | 13:33 |
| dalvarez | but +1 for addressing it now for master branch! i'd have it in commands.py, sounds good to you? | 13:33 |
| anilvenkata | dalvarez, ajo numans lucasagomes I did same thing in l3ha patch | 13:33 |
| dalvarez | anilvenkata, yeah i fully agree, i was just rephrasing numans which was also coherent since we didn't have a solution in mind for the schema thing | 13:33 |
| dalvarez | i think it makes a lot of sense | 13:33 |
| anilvenkata | dalvarez, lucasagomes ajo numans https://review.openstack.org/#/c/486971/4/networking_ovn/ovsdb/commands.py | 13:33 |
| dalvarez | let's do it in commands.py then! | 13:33 |
| dalvarez | :D | 13:33 |
| anilvenkata | dalvarez, lucasagomes ajo numans I have to change many unit tests to accomodate that | 13:34 |
| ajo | yeah, those things may move later in time to ovsdbapp | 13:34 |
| *** fzdarsky has joined #openstack-neutron-ovn | 13:34 | |
| ajo | btw | 13:34 |
| dalvarez | anilvenkata, yeah commands.py is the place | 13:34 |
| dalvarez | and since you're already doing it, let's go for it | 13:34 |
| anilvenkata | dalvarez, ajo numans lucasagomes but I have to change some unit tests, like I did for l3ha | 13:35 |
| lucasagomes | ajo, ++ for having it in ovsdbapp | 13:35 |
| dalvarez | anilvenkata, i tend to agree with numans: let's first have CI pass | 13:35 |
| dalvarez | and then address it the way you're doing it now? | 13:35 |
| anilvenkata | dalvarez, sure | 13:35 |
| dalvarez | and by having CI green we just need to switch to 2.7 branch | 13:36 |
| dalvarez | as the famous hijacked patch does right now :D | 13:36 |
| anilvenkata | :) | 13:36 |
| anilvenkata | dalvarez, ajo lucasagomes numans thanks guys | 13:37 |
| ajo | thank you anilvenkata | 13:37 |
| lucasagomes | yeah thank YOU anilvenkata | 13:38 |
| anilvenkata | :) | 13:38 |
| dalvarez | team team team | 13:39 |
| dalvarez | :p | 13:39 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Idea proposal: Neutron/OVN database consistency problem https://review.openstack.org/490834 | 13:45 |
| *** janki has quit IRC | 13:46 | |
| numans | lucasagomes, http://status.openstack.org/zuul/ - so far going good. functional job passing for py27 and py35 | 13:47 |
| russellb | happy friday everyone | 13:48 |
| numans | russellb, hi | 13:48 |
| numans | russellb, we are having with the gate today | 13:48 |
| numans | hopefully it will be resolved soon. | 13:48 |
| dalvarez | hey hey :) | 13:49 |
| numans | russellb, https://review.openstack.org/#/c/489966/ | 13:49 |
| dalvarez | numans, http://logs.openstack.org/22/490822/1/check/gate-tempest-dsvm-networking-ovn-ovs-master-nv/470f0e1/logs/screen-networking-ovn-metadata-agent.txt.gz#_Aug_04_11_54_28_384736 | 13:49 |
| dalvarez | man everything's broken today :p | 13:49 |
| dalvarez | i remember having seen something like that in the past in neutron gate | 13:49 |
| dalvarez | something related to sudo has changed lately? | 13:50 |
| dalvarez | anilvenkata, lucasagomes ^ | 13:50 |
| dalvarez | ajo ^ | 13:50 |
| russellb | cool - ping if/when it needs a +2 | 13:50 |
| * ajo looks | 13:50 | |
| numans | dalvarez, that part of code runs "sudo ip " commands right to create namespace, create veth etc | 13:51 |
| dalvarez | numans, it does | 13:51 |
| ajo | hmmm yikes, dalvarez that looks like if sudo was epecting to ask for a password | 13:51 |
| dalvarez | ajo but it didn't fail for the patch you submitted today | 13:51 |
| numans | dalvarez, ajo yeah looks like | 13:51 |
| dalvarez | it's the tempest nv job | 13:52 |
| dalvarez | ajo, numans: https://review.openstack.org/#/c/490568/2 | 13:52 |
| dalvarez | this one passed this morning | 13:52 |
| ajo | dalvarez may be they updated the image or devstack ? | 13:52 |
| dalvarez | woa in between the two? | 13:52 |
| dalvarez | prolly we uploaded almost simultaneously | 13:52 |
| dalvarez | anyhow anything executing sudo should fail (not sure if we have something else using sudo) | 13:52 |
| dalvarez | numans, however i'm not calling sudo directly IIRC, just rootwrap | 13:53 |
| dalvarez | ohhhhhhhhhh | 13:53 |
| ajo | no new change in devstack since Aug 1st | 13:53 |
| ajo | so not that | 13:53 |
| dalvarez | ajo, now im not passing in neutron.conf | 13:53 |
| dalvarez | maybe the rootwrap configuration is skipped now | 13:53 |
| dalvarez | crap | 13:53 |
| ajo | dalvarez hmm, it could be that | 13:53 |
| ajo | yes | 13:53 |
| dalvarez | ajo any suggestions around that? | 13:53 |
| ajo | but here locally it's working without that somehow :? | 13:53 |
| dalvarez | shall i assume that we have neutron.conf? | 13:53 |
| dalvarez | ajo cause you're still passing in /etc/neutron.conf | 13:53 |
| ajo | dalvarez no, but my neutron.conf is fake | 13:54 |
| ajo | I only have what I put | 13:54 |
| dalvarez | 620 [agent] | 13:54 |
| dalvarez | 621 root_helper_daemon = sudo /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf | 13:54 |
| dalvarez | 622 root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf | 13:54 |
| dalvarez | 623 | 13:54 |
| dalvarez | ajo ml2 conf maybe? | 13:54 |
| ajo | nope | 13:54 |
| ajo | I don't have it | 13:54 |
| dalvarez | man :( | 13:54 |
| ajo | let me check | 13:54 |
| dalvarez | then i dont know | 13:54 |
| ajo | dalvarez : | 13:55 |
| ajo | [vagrant@compute1 neutron]$ grep rootwrap * -R | 13:55 |
| ajo | [vagrant@compute1 neutron]$ | 13:55 |
| ajo | but may be | 13:55 |
| ajo | I have sudo permission for my user "vagrant" | 13:55 |
| ajo | which I do | 13:56 |
| dalvarez | oh man :) | 13:56 |
| *** mlavalle has joined #openstack-neutron-ovn | 13:56 | |
| ajo | and I have the notty option in sudoers file | 13:56 |
| ajo | so that's it | 13:56 |
| dalvarez | ajo that's it | 13:56 |
| ajo | you need to configure rootwrap-daemon in your agent file too | 13:56 |
| ajo | :) | 13:56 |
| dalvarez | ajo so what do you suggest? adding that option in devstack or in the code? | 13:56 |
| ajo | or make sure neutron will do it for you on the compute nodes too | 13:56 |
| dalvarez | neutron == devstack plugin in neutron? | 13:56 |
| dalvarez | like copying neutron.conf? | 13:56 |
| ajo | dalvarez it may be easier to put in the networking-ovn devstack plugin | 13:56 |
| ajo | compared to doing it in neutron repo | 13:57 |
| ajo | ;) | 13:57 |
| dalvarez | yeah | 13:57 |
| ajo | I leave it to your choice ;D , personally, I'd do it in networking-ovn ;D | 13:57 |
| dalvarez | ajo but how? picking those options from somewhere and writing them into agent ini file? | 13:58 |
| lucasagomes | dalvarez, damn yet another failure :D glad you already found the problem | 13:58 |
| dalvarez | lucasagomes, yeah not sure how to solve it tho | 13:58 |
| * lucasagomes is reading the scrollback | 13:59 | |
| dalvarez | because i think we don't want to assume that compute nodes will have /etc/neutron/neutron.conf present | 13:59 |
| dalvarez | lucasagomes, ^ | 13:59 |
| dalvarez | if we assume that, then i'm fine :) | 13:59 |
| ajo | dalvarez yes, look at how devstack does it for neutron | 13:59 |
| dalvarez | ajo https://github.com/openstack-dev/devstack/blob/b24bfac43dbec9c40a7274a6c51b602fc61226cd/lib/neutron-legacy#L733 | 14:00 |
| dalvarez | that's in neutron-legacy | 14:00 |
| dalvarez | in neutron not sure | 14:00 |
| dalvarez | but i could take those | 14:00 |
| ajo | you have it in lib/neutron too I think | 14:00 |
| ajo | configure_neutron_rootwrap | 14:00 |
| ajo | may be you can just call that | 14:00 |
| ajo | the same way we call other stuff in lib/neutron | 14:01 |
| dalvarez | ajo that creates directories and so on | 14:02 |
| dalvarez | ajo that doesn't set the root_helper_daemon in ini file if i'm not mistaken | 14:02 |
| ajo | hmm | 14:04 |
| anilvenkata | ajo, dalvarez lucasagomes should the higher layers like ovn_client.py know that columns like 'name', 'severity' not supported for ACL table and shouldn't pass those columns to commands.py ? | 14:05 |
| ajo | right, but we may need that done too | 14:05 |
| dalvarez | ajo iniset $NEUTRON_DHCP_CONF agent root_helper_daemon "$NEUTRON_ROOTWRAP_DAEMON_CMD" | 14:05 |
| dalvarez | yeah probably we're already doing it | 14:05 |
| dalvarez | otherwise i'll do it | 14:05 |
| dalvarez | youre right | 14:05 |
| ajo | dalvarez not doing it, (at least in the metadata case) | 14:05 |
| dalvarez | anilvenkata, good question... i think that 'code base' should support it | 14:05 |
| ajo | dalvarez I don't see it in my compute1 :D | 14:06 |
| anilvenkata | ajo, dalvarez lucasagomes ovn_client.py should be aware what columns it should/shouldn't pass to commands.py | 14:06 |
| dalvarez | anilvenkata, and let commands.py deal with the different schema versions :? | 14:06 |
| ajo | [vagrant@compute1 ~]$ ls /etc/neutron/ | 14:06 |
| ajo | metadata_agent.ini neutron.conf | 14:06 |
| dalvarez | ajo, looking at where to call that fun | 14:06 |
| anilvenkata | dalvarez, commands.py should only take care of how to write to db? | 14:06 |
| lucasagomes | anilvenkata, hmm maybe, how would ovn_client be aware of that ? By looking at the a speicifc version or schema or ? | 14:06 |
| ajo | yeah, may be it can compare the schema version | 14:07 |
| anilvenkata | lucasagomes, it will ask commands.py if these columns are supported or not | 14:07 |
| ajo | to know when the change took place ? | 14:07 |
| anilvenkata | dalvarez, ^ | 14:07 |
| dalvarez | anilvenkata, lucasagomes, i think that the codebase-wise we can assume the columns be there since latest ovn version has it | 14:07 |
| ajo | or well, the same thing if the columns are there or not | 14:07 |
| dalvarez | anilvenkata, lucasagomes but then have a FIXME and check for those columns to exist in the commands.py "layer" | 14:07 |
| lucasagomes | dalvarez, we need to have some sort of backward compat for when these columns are not available | 14:08 |
| * ajo overflows, he's awful multitasker | 14:08 | |
| dalvarez | so that we don't have to deal with schema versions there | 14:08 |
| dalvarez | lol ajo | 14:08 |
| dalvarez | ajo | 14:08 |
| *** fzdarsky has quit IRC | 14:08 | |
| dalvarez | <3 | 14:08 |
| ajo | X'D | 14:08 |
| anilvenkata | ajo, dalvarez lucasagomes other wise ovn_client.py is asking something and commands.py is hiding that | 14:08 |
| lucasagomes | anilvenkata, yeah, it feels like OVO again :D | 14:09 |
| dalvarez | anilvenkata, yeah... | 14:09 |
| anilvenkata | :) | 14:09 |
| anilvenkata | no need for OVO | 14:09 |
| lucasagomes | yeah maybe for now we could have ovn_client being smart enough to deal with different versions | 14:09 |
| lucasagomes | anilvenkata, yeah just saying it's a similar problem | 14:09 |
| anilvenkata | as both are in same code base and aprt of same process :) | 14:09 |
| lucasagomes | anilvenkata, but yeah, ovn_client.py dealing with it seems fair | 14:10 |
| russellb | this is likely to be an ongoing issue in networking-ovn -- having to deal with different versions of OVN | 14:10 |
| ajo | yes | 14:10 |
| ajo | we need an strategy for that | 14:10 |
| lucasagomes | and less misleading than requests X and in the end commands.py is requests Y | 14:10 |
| russellb | use the new features if available, but make sure networking-ovn can still be used if not | 14:10 |
| ajo | may be a good topic for next meeting | 14:10 |
| lucasagomes | russellb, yeah we should account to that | 14:10 |
| numans | we need to probably find a better solution with a proper design | 14:10 |
| russellb | we avoided this in the past by only supporting latest release, but as of ... right now basically, we can't really do that anymore :) | 14:11 |
| russellb | should be able to rely on schema version number | 14:11 |
| lucasagomes | ++ | 14:12 |
| * anilvenkata searching for some code (poor in multitasking) | 14:12 | |
| ajo | russellb I'd may be try to rely on looking at features of the DB | 14:12 |
| ajo | or not... I don't know | 14:13 |
| ajo | I was thinking about backport patches on core-ovn, that touch the DB | 14:13 |
| ajo | if we look at the schema, it'd be hard to know if the thing is present | 14:13 |
| dalvarez | yeah i agree with ajo.. | 14:14 |
| anilvenkata | dalvarez, ajo russellb lucasagomes numans actually I have seen some code in ovn_client which is checking if row is already present or not(asking commands.py), then doing some stuff. similarly it can check if the columns are supported or not (asking commands.py) | 14:14 |
| * anilvenkata searching again for that code | 14:15 | |
| dalvarez | ajo: sry for the task switching, are you ok with calling "configure_neutron_rootwrap" where we check if metadata service is enabled? | 14:15 |
| lucasagomes | ajo, as long as the changes to the db are backward compatible and we have a number bumped for every change in the schema we could rely on these numbers no ? | 14:15 |
| russellb | OK - i don't feel too strongly about schema version vs introspecting schema itself | 14:16 |
| ajo | dalvarez that'd be ok, yes | 14:16 |
| russellb | schema version sounds like slightly less code but | 14:16 |
| russellb | ¯\_(ツ)_/¯ | 14:16 |
| openstackgerrit | Daniel Alvarez proposed openstack/networking-ovn master: Make Metadata agent independent from other config files https://review.openstack.org/490822 | 14:16 |
| dalvarez | ajo thanks ^ | 14:16 |
| ajo | russellb yeah, it started thinking of that, but I started feeling the pain of bugs that required schema changes | 14:16 |
| ajo | still... well.. this is generally for new features | 14:16 |
| ajo | we don't tend to back port that, but who knows ':D | 14:17 |
| anilvenkata | ajo, dalvarez russellb dalvarez I was talking about this existing function check_for_row_by_value_and_retry , we can add a new similar function verify_attribute_supported http://paste.openstack.org/show/617544/ | 14:18 |
| anilvenkata | numans, ^ | 14:18 |
| dalvarez | anilvenkata, i like that one | 14:18 |
| dalvarez | i think we can stick to that until we come up with a more general approach | 14:18 |
| anilvenkata | dalvarez, thanks Daniel | 14:19 |
| anilvenkata | ajo, lucasagomes what do u say about that http://paste.openstack.org/show/617544/ similar to existing check_for_row_by_value_and_retry() | 14:19 |
| lucasagomes | anilvenkata, looks like a big workaround for me (-: | 14:20 |
| lucasagomes | insert a row and then delete it to know whether it's supported or not | 14:20 |
| anilvenkata | lucasagomes, ok | 14:21 |
| lucasagomes | if that's the only way, grand, but I would love to have a better way of doing that | 14:21 |
| anilvenkata | lucasagomes, I will modify that | 14:21 |
| lucasagomes | if possible | 14:21 |
| anilvenkata | lucasagomes, check if any rows already present, then check for column, if now rows then insert row | 14:21 |
| lucasagomes | I know that for now we only want to remove that workaround pinning a version | 14:21 |
| numans | i think at ml2 mech driver startup, we need to check the schema version and then load the features like metadata or dns, or ipv6 RA support etc. in a way the ml2 mech driver should be smart enough and make these features kind of pluggable. not sure if this makes sense | 14:21 |
| lucasagomes | but if we had a "less intrusive" way of detecting that I think we would be better off | 14:21 |
| lucasagomes | numans, ++ | 14:22 |
| lucasagomes | as long as we document that we need to restart the services after updating the schema it should be good | 14:23 |
| russellb | numans: ++ | 14:23 |
| anilvenkata | will that be applicable for new columns also ? | 14:23 |
| russellb | might be nice to check per request ... be a bit more dynamic | 14:24 |
| lucasagomes | anilvenkata, I would imagine so, any change to the schema would bump a certain version | 14:24 |
| russellb | requested an HA gateway -- do we support that? yes? create. no? reject. | 14:24 |
| lucasagomes | russellb, yeah that would be even better and would allow to update the db w/o restarting the neutron services | 14:25 |
| anilvenkata | but these requests are neutron commands right? | 14:26 |
| anilvenkata | if so, how can user check OVN specific things like these new columns('name' and 'severity')? | 14:27 |
| russellb | a user? | 14:31 |
| russellb | neutron user? | 14:31 |
| russellb | they get whatever the neutron API exposes | 14:31 |
| russellb | i guess | 14:31 |
| anilvenkata | russellb, "might be nice to check per request ... be a bit more dynamic. requested an HA gateway -- do we support that? yes? create. no? reject", - who issues this request? | 14:32 |
| russellb | i meant checking in the path of neutron API requests | 14:34 |
| russellb | check to see if we can fulfill the request, at request time | 14:34 |
| anilvenkata | ok, thanks | 14:36 |
| anilvenkata | lucasagomes, " I would imagine so, any change to the schema would bump a certain version" is it already supported in ovn ? | 14:48 |
| numans | anilvenkata, it is expected to, but may not be guaranteed to be the case | 14:49 |
| ajo | seen in our unit tests: | 14:49 |
| ajo | /opt/stack/networking-ovn/.tox/py27/lib/python2.7/site-packages/oslo_context/context.py:104: DeprecationWarning: Policy enforcement is depending on the value of tenant_id. This key is deprecated. Please update your policy file to use the standard policy values. | 14:49 |
| ajo | DeprecationWarning) | 14:49 |
| ajo | I guess we should move all tenant_id references to project_id, I guess? | 14:50 |
| ajo | I'm guessing twice | 14:50 |
| anilvenkata | numans, oh, then w ecant relay on that | 14:50 |
| numans | anilvenkata, too early to say now | 14:50 |
| openstackgerrit | Miguel Angel Ajo proposed openstack/networking-ovn master: Support for L3 gateway HA https://review.openstack.org/486971 | 14:51 |
| ajo | just a tiny change | 14:52 |
| ajo | I probably may update the unit test | 14:52 |
| anilvenkata | ajo++ | 14:53 |
| anilvenkata | ajo, thanks Miguel | 14:53 |
| numans | dalvarez, russellb ajo lucasagomes anilvenkata https://review.openstack.org/#/c/489966/ passed | 14:55 |
| dalvarez | numans, \o/ | 14:55 |
| dalvarez | oh yeah!! | 14:55 |
| dalvarez | let's merge it and unblock the gate :) | 14:56 |
| dalvarez | i have to recheck quite a good # of patches | 14:56 |
| numans | lucasagomes, and russellb can you please have a look and review it - https://review.openstack.org/#/c/489966/ | 14:56 |
| russellb | i'm fine approving this to fix the gate ... | 14:58 |
| russellb | but it seems really bad that new columns we don't use yet would break anything | 14:58 |
| russellb | it's one thing to deal with optionally using a new feature, but new features we're not using shouldn't break our tests ... | 14:59 |
| openstackgerrit | Terry Wilson proposed openstack/networking-ovn master: WIP Start converting to ovsdbapp impls https://review.openstack.org/489416 | 15:02 |
| lucasagomes | anilvenkata, numans sorry I was on my 1:1, reading now | 15:15 |
| lucasagomes | numans, can't we add some unittest to check for these versions bumps in OVN core then ? | 15:15 |
| lucasagomes | lucasagomes, like OVO, we could hash the schema and if there's any change to it we would detect in a unittest | 15:15 |
| russellb | we should make our tests not need to detect it | 15:16 |
| russellb | IMO | 15:16 |
| lucasagomes | russellb, but then what happens if you update the schema and forget to bump the version ? That patch could sneak in the repository no ? | 15:17 |
| russellb | i mean on networking-ovn side | 15:18 |
| lucasagomes | oh right | 15:18 |
| lucasagomes | oh I agree with that yeah | 15:18 |
| russellb | schema is hashed on core ovn side | 15:18 |
| lucasagomes | I was thinking about OVN core | 15:18 |
| russellb | you can't update the schema without updating the hash (which sits right next to the version number in the file) | 15:18 |
| russellb | so as a reviewer, you shouldn't see a hash change without the version changed too | 15:19 |
| lucasagomes | right on, so the tests are in place already | 15:19 |
| russellb | on core side yeah | 15:19 |
| russellb | roughly | 15:19 |
| lucasagomes | that's good then, so anilvenkata ^ sounds like we could rely on that version yes | 15:19 |
| russellb | could be more explicit ... maybe checkpatch.py to ensure a hash change comes with version change | 15:19 |
| lucasagomes | otherwiseguy, when you have some time, me and dalvarez have some doubts that you might help with here https://review.openstack.org/#/c/490834/1/doc/source/contributor/design/database_consistency.rst (L89) | 15:33 |
| * otherwiseguy looks | 15:33 | |
| dalvarez | :) | 15:33 |
| otherwiseguy | lucasagomes, dalvarez To be honest, I really know very little about the neutron db side of things. Are there cases where you would need to update an object and an object that is referenced as a foreign key at the same time and therefor run into lock-order/deadlock issues? | 15:43 |
| dalvarez | otherwiseguy, i think in neutron is already solved but what lucasagomes proposes is to use kind of a distributed lock in neutron | 15:44 |
| dalvarez | so we thought you would come up with some brilliant idea to not lock on neutron db and solve it at OVSDB level :) | 15:45 |
| otherwiseguy | I just know that most projects I've worked on that add locks after the fact have run into issues (not insurmountable ones, but still). | 15:45 |
| otherwiseguy | ok. | 15:45 |
| lucasagomes | otherwiseguy, dalvarez right yeah, the more specific question is whether we could rely on IDLs and not have to have a lock in neutron | 15:45 |
| lucasagomes | neutron db* | 15:45 |
| otherwiseguy | Well, my brilliant idea is to add the ability for ML2 to pass through db access to ml2 plugins and use the ovn db as the actual source for truth. I think both of you should implement that while I watch from afar. | 15:46 |
| dalvarez | otherwiseguy, lol | 15:46 |
| dalvarez | otherwiseguy, want some popcorn? | 15:46 |
| otherwiseguy | dalvarez, yes please. | 15:47 |
| dalvarez | haha | 15:47 |
| otherwiseguy | It would be possible in ovsdb to do locks based on the uuid as well. | 15:47 |
| dalvarez | anyways that sounds good, at least the part of having the ovn db as the source for truth | 15:47 |
| dalvarez | otherwiseguy, as you suggested, using named locks, right? | 15:47 |
| russellb | that's an enormous amount of work | 15:47 |
| russellb | (using only ovn db) | 15:47 |
| lucasagomes | otherwiseguy, yeah named locks right ? | 15:47 |
| otherwiseguy | they're just name-based locks, so ovsdb lock with name == uuid would be a row-level lock. | 15:47 |
| lucasagomes | otherwiseguy, yeah, I could add as an alternative on the spec | 15:48 |
| otherwiseguy | I have no idea what the performance impact of that would be. | 15:48 |
| lucasagomes | I was just unsure about the amount of locks that ovsdb can handle | 15:48 |
| otherwiseguy | all of them. | 15:48 |
| lucasagomes | because I know that for mysql it should be fine, but, idk ovsdb enough | 15:48 |
| otherwiseguy | it can handle all of the locks. | 15:48 |
| otherwiseguy | (i have no idea) | 15:48 |
| lucasagomes | lol | 15:48 |
| lucasagomes | ok yeah, it's definetely possible | 15:48 |
| lucasagomes | in the same spec I suggest using named locks in ovsdb for making sure that only one maintenance thread runs in the whole cluster | 15:49 |
| otherwiseguy | If ovsdb's distributed db was done that would be nice. | 15:49 |
| lucasagomes | like we have for ovsdb_monitor.py | 15:49 |
| otherwiseguy | If only there were algorithms out there for solving consensus issues. :) | 15:50 |
| lucasagomes | right (-: etcd! | 15:50 |
| lucasagomes | raft* | 15:50 |
| lucasagomes | but yeah... | 15:50 |
| otherwiseguy | raft, multi-paxos, whatever. i spent a bit of time researching all of that, but the time...who has the time... | 15:51 |
| lucasagomes | you need 3 dbs running for that tho (I think, it requires minimum 3 for consensus, I might be wrong tho...) | 15:51 |
| * lucasagomes needs to look into raft more | 15:51 | |
| otherwiseguy | lucasagomes, with ovsdb each worker is a db so... :D | 15:52 |
| lucasagomes | otherwiseguy, yup, that's the reason why I think we need a distributed lock actually, cause I'm afraid that relying on the in-memory replica model in the IDLs might lead to inconsistencies because they could be outdated | 15:53 |
| * lucasagomes already had too many problems with idls :-( | 15:53 | |
| otherwiseguy | even with etcd, i think by default you can get stale reads though. | 15:57 |
| otherwiseguy | or at least that was the case when I looked at it a while ago. | 15:57 |
| otherwiseguy | you can configure it so that isn't the case, just saying that the general issue is a pretty prevalent one. | 15:59 |
| *** pcaruana has quit IRC | 16:10 | |
| *** lucasagomes is now known as lucas-afk | 16:28 | |
| openstackgerrit | Merged openstack/networking-ovn master: Fix gate issues https://review.openstack.org/489966 | 16:29 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/networking-ovn master: Updated from global requirements https://review.openstack.org/488027 | 16:43 |
| *** anilvenkata has quit IRC | 17:14 | |
| *** openstackstatus has quit IRC | 17:27 | |
| *** openstack has joined #openstack-neutron-ovn | 17:29 | |
| *** openstackstatus has joined #openstack-neutron-ovn | 17:29 | |
| *** ChanServ sets mode: +v openstackstatus | 17:29 | |
| *** markmcclain has quit IRC | 18:24 | |
| *** markmcclain has joined #openstack-neutron-ovn | 18:24 | |
| *** openstackstatus has quit IRC | 18:26 | |
| *** openstack has joined #openstack-neutron-ovn | 18:28 | |
| *** openstackstatus has joined #openstack-neutron-ovn | 18:28 | |
| *** ChanServ sets mode: +v openstackstatus | 18:28 | |
| openstackgerrit | Akihiro Motoki proposed openstack/networking-ovn master: Improve oslo-config-generator setting https://review.openstack.org/486351 | 18:31 |
| openstackgerrit | Akihiro Motoki proposed openstack/networking-ovn master: Add auto-generated config reference https://review.openstack.org/486352 | 18:31 |
| -openstackstatus- NOTICE: Gerrit is being restarted to pick up CSS changes and should be back momentarily | 20:36 | |
| *** mmirecki has quit IRC | 21:08 | |
| *** yamamoto has joined #openstack-neutron-ovn | 22:00 | |
| *** yamamoto has quit IRC | 22:41 | |
| *** yamamoto has joined #openstack-neutron-ovn | 23:09 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!