| *** pmannidi has joined #openstack-kuryr | 00:01 | |
| *** hongbin has joined #openstack-kuryr | 01:00 | |
| *** hongbin has quit IRC | 03:19 | |
| *** rh-jelabarre has quit IRC | 03:28 | |
| *** janki has joined #openstack-kuryr | 04:25 | |
| openstackgerrit | Peng Liu proposed openstack/kuryr-tempest-plugin master: NPWG multi-vif driver tempest tests https://review.openstack.org/590643 | 04:37 |
|---|---|---|
| *** janki has quit IRC | 05:19 | |
| *** janki has joined #openstack-kuryr | 05:28 | |
| *** janki has quit IRC | 06:41 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add a new gate for network policies https://review.openstack.org/590304 | 06:41 |
| *** pcaruana has joined #openstack-kuryr | 06:43 | |
| openstackgerrit | Peng Liu proposed openstack/kuryr-kubernetes master: Move function get_subnet to kuryr_kubernetes.utils https://review.openstack.org/590662 | 06:45 |
| ltomasbo | celebdor[m], dmellado: do you now if there is any issue with containerized openshift on the gates? | 07:03 |
| dmellado | not that I'm aware ltomasbo | 07:03 |
| dmellado | what's going on? | 07:03 |
| ltomasbo | it's taking ages for me to make this patch pass them! https://review.openstack.org/#/c/580680 | 07:03 |
| ltomasbo | the 2 thatt timed out | 07:04 |
| ltomasbo | I'm checking the logs, and it seems the kuryr-cni was not ready | 07:04 |
| dmellado | I had some timouts on octavia yesterday | 07:04 |
| ltomasbo | and I see there is no crds/annotations being made | 07:04 |
| dmellado | in fact I'm still waiting for 1 patch to get merged | 07:04 |
| ltomasbo | Kuryrnet CRD could not be added. Rolling back network resources created for the namespace.: K8sClientException: <Response | 07:04 |
| dmellado | hmm werid | 07:05 |
| dmellado | let's check https://review.openstack.org/#/c/590304/ | 07:05 |
| dmellado | it's set to trigger experimentla gates | 07:05 |
| ltomasbo | dmellado, btw, I'll be posting one soon this morning regarding namespace and pool deletion race | 07:05 |
| ltomasbo | dmellado, great! let me know the outcome! | 07:05 |
| dmellado | ltomasbo: depending on how fast you are it might even cut in rc1 | 07:06 |
| ltomasbo | perhaps those gates are marked as non-voting for a reason... | 07:06 |
| ltomasbo | dmellado, I'm stacking to test if it fix the issue | 07:06 |
| ltomasbo | if it does, it will be in a couple of hours I guess | 07:06 |
| dmellado | heh, let me know if you can get it | 07:06 |
| dmellado | but be warned that I'll be going off early today | 07:06 |
| dmellado | like in 3 hours or so | 07:06 |
| ltomasbo | ohh, ok | 07:07 |
| ltomasbo | we'll see if I'm fast enough xD | 07:07 |
| ltomasbo | dmellado, celebdor[m]: look what I'm seeing: Aug 09 17:33:42.760263 ubuntu-xenial-limestone-regionone-0001248938 openshift[6215]: 2018-08-09 17:33:42.751 23 ERROR kuryr_kubernetes.cni.api [-] Looks like 127.0.0.1:50036 cannot be reached. Is kuryr-daemon running?: ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=50036): Max retries exceeded with url: /addNetwork (Caused by NewConnectionError('<urllib3.connection.HTTPConn | 07:07 |
| ltomasbo | ection object at 0x7f02389ea750>: Failed to establish a new connection: [Errno 111] Connection refused',)) | 07:07 |
| dmellado | hmm that's weird | 07:08 |
| ltomasbo | taht is on the openshift-node | 07:08 |
| dmellado | that's only on that experimental gate, isn't it? | 07:08 |
| ltomasbo | yep, and that should be independent on the namespace isolation (I suppose) | 07:08 |
| dmellado | ltomasbo: let's check if that duplicates on my patch | 07:08 |
| dmellado | if so, we'll open a bug for this | 07:08 |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: Change Pod annotations format to o.vo https://review.openstack.org/584377 | 07:20 |
| *** janki has joined #openstack-kuryr | 07:27 | |
| *** pmannidi has quit IRC | 07:42 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers https://review.openstack.org/580680 | 07:46 |
| *** garyloug has joined #openstack-kuryr | 08:06 | |
| *** celebdor has joined #openstack-kuryr | 08:27 | |
| pliu | Hi, I am working a patch for testing NWPG feature with tempest plugin, and hit a problem when using method exec_command_in_pod. | 08:31 |
| celebdor | pliu: which problem? | 08:31 |
| *** janki has quit IRC | 08:32 | |
| pliu | since kuryr-kubernetes cannot work with kube-dns, the kube-api cannot get ip of pod from dns. like you cannot run 'kubectl exec' with kuryr | 08:32 |
| pliu | I saw other tempest cases also use this method like test_namespace.py, I wonder how it works. | 08:34 |
| dmellado | damn, I'm having issues cutting rc1 | 08:36 |
| dmellado | damn 'paperworks' xD | 08:36 |
| celebdor | pliu: did you try with devstack openshift support? | 08:36 |
| celebdor | I added dns support there | 08:36 |
| pliu | so the kube-dns can run on openshift base deployment? | 08:38 |
| celebdor | pliu: kube-dns is run internally by openshift node when you choose openshift devstack | 08:38 |
| pliu | good to know that. I will try it with openshift. Thanks | 08:39 |
| celebdor | I didn't have time to integrate kube-dns for kubernetes deployment | 08:39 |
| celebdor | dmellado: we have flakiness again in getting answers from different backends | 08:39 |
| celebdor | why oh why can't we get reliable round robin from Octavia? | 08:40 |
| *** janki has joined #openstack-kuryr | 08:43 | |
| ltomasbo | celebdor, https://review.openstack.org/#/c/580680/ this one is finally working on containerized openshift octavia | 09:25 |
| ltomasbo | celebdor, I'm re-checking the experimental gate as the serial one failed due to the yum install issue that happens every now and then | 09:26 |
| celebdor | ltomasbo: what's the new apigroups for? | 09:26 |
| ltomasbo | otherwise openshift cannot push objects into the kuryrnet crds that are used for the namespaces | 09:27 |
| ltomasbo | celebdor, ^^ | 09:27 |
| ltomasbo | it was being rejected (403 exception) | 09:27 |
| celebdor | right! | 09:28 |
| celebdor | good ctach | 09:28 |
| ltomasbo | celebdor, well, it took me a few retries... | 09:28 |
| ltomasbo | celebdor, it was driving me crazy!! | 09:29 |
| celebdor | shouldn't the second apigroup be enough? | 09:29 |
| celebdor | I would think the first one is only to create new crd from specs | 09:29 |
| celebdor | not to create/update/delete crd instances | 09:29 |
| celebdor | ltomasbo: ^^ | 09:29 |
| ltomasbo | probably | 09:30 |
| ltomasbo | the crd was being created without that actuall (as it was created with kubectl apply) | 09:30 |
| ltomasbo | celebdor, do you want me to give it a try without that? | 09:30 |
| celebdor | ltomasbo: yes, please | 09:31 |
| ltomasbo | sure! | 09:31 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers https://review.openstack.org/580680 | 09:32 |
| ltomasbo | done | 09:32 |
| celebdor | ltomasbo: dmellado: do you know why the hell do we use curl in https://github.com/openstack/kuryr-tempest-plugin/blob/master/kuryr_tempest_plugin/tests/scenario/test_service.py#L45-L47 | 09:32 |
| ltomasbo | let's see if it likes it | 09:32 |
| celebdor | instead of just using requests? | 09:32 |
| celebdor | I don't quite get it | 09:32 |
| dmellado | celebdor: before going off | 09:32 |
| dmellado | I'd say because of gcheresh | 09:32 |
| dmellado | xD | 09:33 |
| dmellado | g'weekend y'all | 09:33 |
| celebdor | do you see anything wrong with changing it to requests? | 09:33 |
| dmellado | nope | 09:33 |
| celebdor | ok | 09:33 |
| celebdor | so then I'm changing it to parallel requests | 09:34 |
| celebdor | using a threadpool | 09:34 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure delete_network_pools include all the ports https://review.openstack.org/590739 | 10:25 |
| ltomasbo | celebdor, ^^ this fixes the race for deleting namespaces and ports | 10:32 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-tempest-plugin master: Use ThreadPool and requests instead of subprocess https://review.openstack.org/590740 | 10:32 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-tempest-plugin master: Use ThreadPool and requests instead of subprocess https://review.openstack.org/590740 | 10:37 |
| celebdor | ltomasbo: ^^ getting rid of curl | 10:37 |
| celebdor | ltomasbo: can you explain how that works? | 10:42 |
| ltomasbo | celebdor, sure! | 10:54 |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: drop deprecated oadm https://review.openstack.org/588303 | 10:55 |
| ltomasbo | celebdor, basically it ensures that _trigger_return_to_pool is not executed at the same time on the background (periodic task cleaning up ports to be reused) and by the deletE_neetwork_polls | 10:55 |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: Allow musl based distros to resolve using search https://review.openstack.org/580701 | 10:56 |
| celebdor | ltomasbo: at the same time as what? | 11:00 |
| ltomasbo | celebdor, so, _trigger_return_to_pool is the function that puts back the ports on the available_ports_pool dict | 11:00 |
| ltomasbo | celebdor, this is executed (in a different thread) every 15 seconds (by default) | 11:01 |
| ltomasbo | celebdor, and this function is also called from delete_network_pools | 11:01 |
| celebdor | oh | 11:02 |
| ltomasbo | celebdor, otherwise, delete_network_pools will delete the ports (belonging to the network) on available_ports_pool and then _trigger_return_to_pool will put some other ports of that network back to available_ports_pools | 11:02 |
| ltomasbo | that will lead to ports on that network not being deleted | 11:02 |
| ltomasbo | and therefore, the subnet/network/sg cannot be deleted if it is in used | 11:02 |
| ltomasbo | so, calling trigger_return_to_pool ensure taht all the ports belonging to delted pods are put back on the available_ports_pool | 11:03 |
| ltomasbo | and locking trigger_return_to_pool so taht it cannot be executed by different threads at the same time will ensure this is not f*ck up by executing the same fucntion by the periodic time | 11:04 |
| ltomasbo | task | 11:04 |
| celebdor | ltomasbo: thanks. Now go and make the commit message include this much more in depth explanation | 11:05 |
| celebdor | :-) | 11:05 |
| ltomasbo | man... | 11:06 |
| ltomasbo | I have it on the bug description | 11:06 |
| celebdor | ok, let me look at it again | 11:06 |
| ltomasbo | or you want more detailed info there too? | 11:06 |
| celebdor | cause I read both earlier and I missed part | 11:06 |
| celebdor | ok, ok | 11:07 |
| celebdor | I buy it as it is | 11:07 |
| ltomasbo | if not clear, I can try to extend it | 11:07 |
| ltomasbo | no problem, I will re-spin it anyway to fix the typo | 11:08 |
| celebdor | no, it's okay, just the typo | 11:08 |
| ltomasbo | ok! | 11:08 |
| celebdor | ltomasbo: I'm curious, why did you use separate lock names for baremetal and nested? | 11:08 |
| celebdor | is it for environments with both? | 11:09 |
| ltomasbo | yep | 11:09 |
| ltomasbo | I believe there is no need to block both at the same time | 11:09 |
| ltomasbo | as they will be touching different pool keys | 11:09 |
| ltomasbo | (perhaps I'm wrong) | 11:09 |
| celebdor | ok | 11:11 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure delete_network_pools include all the ports https://review.openstack.org/590739 | 11:15 |
| openstackgerrit | Alexey Perevalov proposed openstack/kuryr-kubernetes master: Separate helper function for pod https://review.openstack.org/590245 | 11:23 |
| *** rh-jelabarre has joined #openstack-kuryr | 11:26 | |
| ltomasbo | celebdor, I'm my explanation I forgot to mention the second part of the problem (deleting 2 namespaces with pods) | 11:31 |
| celebdor | ltomasbo: add it to the bug | 11:32 |
| ltomasbo | I'll do | 11:32 |
| ltomasbo | celebdor, seems removing the crd thing was not good: https://review.openstack.org/#/c/580680/21 | 12:10 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-tempest-plugin master: Use ThreadPool and requests instead of subprocess https://review.openstack.org/590740 | 12:11 |
| celebdor | ltomasbo: for which gate? | 12:11 |
| ltomasbo | containerized openshift with octavia | 12:11 |
| ltomasbo | the 2 experiemtnal ones that have failed | 12:11 |
| celebdor | ltomasbo: why didn't it fail with the other openshift one? | 12:12 |
| ltomasbo | no containerized | 12:12 |
| ltomasbo | then it does not need all the sa things | 12:12 |
| celebdor | the controller always uses a serviceaccount, doesn't it? | 12:12 |
| ltomasbo | only if is running on top of openshift/kubernetse, right? | 12:13 |
| ltomasbo | if it is a devstack@XXX service it does not need it, right> | 12:13 |
| celebdor | ltomasbo: how does it perform the operations on k8s otherwise? | 12:14 |
| celebdor | I don't think we use the admin kubeconfig ever | 12:14 |
| celebdor | let me check | 12:14 |
| ltomasbo | nop, I only have this problems for openshift gates | 12:14 |
| ltomasbo | and only for the ones containerized | 12:15 |
| celebdor | meh | 12:30 |
| celebdor | this is a bug | 12:30 |
| celebdor | we should be using service accounts and cluster role bindings in both cases :/ | 12:32 |
| celebdor | ltomasbo: did you find the permission denied message? | 12:35 |
| *** janki has quit IRC | 12:35 | |
| ltomasbo | nop | 12:35 |
| ltomasbo | I'm rechecking just in case | 12:35 |
| ltomasbo | I didn't find any error | 12:35 |
| ltomasbo | seems like a timeout issue | 12:36 |
| *** jistr is now known as jistr|call | 13:46 | |
| ltomasbo | celebdor, so it seems the failures are unrelated, recheck changed the type of gate failing... https://review.openstack.org/#/c/580680 | 13:54 |
| ltomasbo | so I guess you were right and the customresourcedefinition api was not needed | 13:55 |
| ltomasbo | going to recheck for the last time! | 13:56 |
| *** jistr|call is now known as jistr | 14:00 | |
| *** hongbin_ has joined #openstack-kuryr | 14:03 | |
| celebdor | :-) | 14:21 |
| *** AlexeyPerevalov has quit IRC | 14:37 | |
| openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-tempest-plugin master: Use ThreadPool and requests instead of subprocess https://review.openstack.org/590740 | 14:46 |
| *** celebdor has quit IRC | 16:44 | |
| *** celebdor has joined #openstack-kuryr | 16:45 | |
| *** aojea has joined #openstack-kuryr | 16:46 | |
| *** celebdor has quit IRC | 16:49 | |
| *** aojea has quit IRC | 16:52 | |
| *** pcaruana has quit IRC | 19:19 | |
| *** aojea has joined #openstack-kuryr | 21:56 | |
| *** aojea has quit IRC | 22:27 | |
| *** hongbin_ has quit IRC | 23:00 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!