| *** gianpietro has quit IRC | 00:09 | |
| *** gianpietro has joined #openstack-kuryr | 00:10 | |
| *** salv-orlando has joined #openstack-kuryr | 00:27 | |
| *** salv-orlando has quit IRC | 00:32 | |
| *** hongbin has joined #openstack-kuryr | 00:44 | |
| *** yamamoto has joined #openstack-kuryr | 00:52 | |
| *** yamamoto has quit IRC | 00:57 | |
| *** salv-orlando has joined #openstack-kuryr | 01:27 | |
| *** salv-orlando has quit IRC | 01:32 | |
| *** yamamoto has joined #openstack-kuryr | 01:53 | |
| *** yamamoto has quit IRC | 01:59 | |
| *** jlabarre has quit IRC | 02:15 | |
| *** salv-orlando has joined #openstack-kuryr | 02:28 | |
| *** salv-orlando has quit IRC | 02:34 | |
| *** yamamoto has joined #openstack-kuryr | 02:37 | |
| *** yamamoto has quit IRC | 02:46 | |
| *** caowei has joined #openstack-kuryr | 02:47 | |
| *** yamamoto has joined #openstack-kuryr | 02:56 | |
| *** premsankar has quit IRC | 03:24 | |
| *** hongbin has quit IRC | 03:30 | |
| *** salv-orlando has joined #openstack-kuryr | 03:30 | |
| *** salv-orlando has quit IRC | 03:35 | |
| *** janonymous has joined #openstack-kuryr | 04:05 | |
| *** salv-orlando has joined #openstack-kuryr | 04:31 | |
| *** salv-orlando has quit IRC | 04:36 | |
| *** isssp has joined #openstack-kuryr | 05:03 | |
| *** burned has quit IRC | 05:06 | |
| *** isssp has quit IRC | 05:09 | |
| *** isssp has joined #openstack-kuryr | 05:12 | |
| *** yboaron has joined #openstack-kuryr | 05:22 | |
| *** salv-orlando has joined #openstack-kuryr | 05:29 | |
| *** yboaron has quit IRC | 05:40 | |
| *** gcheresh_ has joined #openstack-kuryr | 06:15 | |
| *** yboaron has joined #openstack-kuryr | 06:15 | |
| *** phuoc_ has joined #openstack-kuryr | 06:19 | |
| *** yboaron has quit IRC | 06:20 | |
| *** phuoc has quit IRC | 06:22 | |
| *** natanbro has joined #openstack-kuryr | 06:23 | |
| *** threestrands has quit IRC | 06:25 | |
| *** aojea has joined #openstack-kuryr | 06:58 | |
| *** yboaron has joined #openstack-kuryr | 07:20 | |
| *** isssp has quit IRC | 07:25 | |
| *** isssp has joined #openstack-kuryr | 07:26 | |
| *** salv-orlando has quit IRC | 07:33 | |
| *** pcaruana has joined #openstack-kuryr | 07:42 | |
| *** pcaruana has quit IRC | 07:44 | |
| *** pcaruana has joined #openstack-kuryr | 07:44 | |
| *** pcaruana has quit IRC | 07:45 | |
| *** pcaruana has joined #openstack-kuryr | 07:45 | |
| *** pcaruana has quit IRC | 07:47 | |
| *** pcaruana has joined #openstack-kuryr | 07:47 | |
| *** pcaruana has quit IRC | 07:48 | |
| *** pcaruana has joined #openstack-kuryr | 07:48 | |
| *** pcaruana has quit IRC | 07:50 | |
| *** pcaruana has joined #openstack-kuryr | 07:50 | |
| *** pcaruana has quit IRC | 07:51 | |
| *** pcaruana has joined #openstack-kuryr | 07:51 | |
| *** pcaruana has quit IRC | 07:53 | |
| *** pcaruana has joined #openstack-kuryr | 07:53 | |
| *** pcaruana has quit IRC | 07:54 | |
| *** pcaruana has joined #openstack-kuryr | 07:55 | |
| *** pcaruana has quit IRC | 07:56 | |
| *** ispp has joined #openstack-kuryr | 07:58 | |
| *** isssp has quit IRC | 08:00 | |
| *** pcaruana has joined #openstack-kuryr | 08:06 | |
| *** pcaruana has quit IRC | 08:07 | |
| *** pcaruana has joined #openstack-kuryr | 08:08 | |
| *** pcaruana has quit IRC | 08:09 | |
| *** pcaruana has joined #openstack-kuryr | 08:09 | |
| *** pcaruana has quit IRC | 08:10 | |
| *** pcaruana has joined #openstack-kuryr | 08:11 | |
| *** pcaruana has quit IRC | 08:12 | |
| *** pcaruana has joined #openstack-kuryr | 08:13 | |
| *** pcaruana has quit IRC | 08:15 | |
| *** pcaruana has joined #openstack-kuryr | 08:15 | |
| *** pcaruana has quit IRC | 08:16 | |
| *** pcaruana has joined #openstack-kuryr | 08:18 | |
| *** yboaron has quit IRC | 08:19 | |
| *** pcaruana has quit IRC | 08:20 | |
| *** pcaruana has joined #openstack-kuryr | 08:20 | |
| *** pcaruana has quit IRC | 08:21 | |
| *** pcaruana has joined #openstack-kuryr | 08:21 | |
| *** pcaruana has quit IRC | 08:22 | |
| *** snapiri has joined #openstack-kuryr | 08:25 | |
| *** pcaruana has joined #openstack-kuryr | 08:29 | |
| *** pcaruana has quit IRC | 08:30 | |
| *** salv-orlando has joined #openstack-kuryr | 08:34 | |
| *** pcaruana has joined #openstack-kuryr | 08:36 | |
| *** pcaruana has quit IRC | 08:37 | |
| *** pcaruana has joined #openstack-kuryr | 08:38 | |
| *** celebdor has joined #openstack-kuryr | 08:38 | |
| *** pcaruana has quit IRC | 08:39 | |
| celebdor | dulek: I'll send a patch to add the kubelet interface to devstack openshift deployment | 08:39 |
|---|---|---|
| *** pcaruana has joined #openstack-kuryr | 08:40 | |
| celebdor | dmellado: ptal https://review.openstack.org/555023 | 08:40 |
| *** salv-orlando has quit IRC | 08:40 | |
| dmellado | ptal | 08:40 |
| dmellado | ? xD | 08:40 |
| celebdor | dmellado: Please Take A Look | 08:40 |
| *** pcaruana has quit IRC | 08:40 | |
| dulek | celebdor: I'm trying to run containerized jobs on 555040, but yeah - OpenShift ones will probably fail. | 08:41 |
| dmellado | celebdor: you? document? xD | 08:41 |
| *** pcaruana has joined #openstack-kuryr | 08:41 | |
| *** pcaruana has quit IRC | 08:41 | |
| celebdor | dmellado: https://www.youtube.com/watch?v=kwLbphePcLM | 08:44 |
| dmellado | xD | 08:50 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-kubernetes master: devstack: add kubelet iface to openshift nodes https://review.openstack.org/555206 | 09:00 |
| *** pcaruana has joined #openstack-kuryr | 09:02 | |
| celebdor | dulek: ^^ | 09:03 |
| celebdor | irenab: ^^ | 09:03 |
| *** pcaruana has quit IRC | 09:03 | |
| *** pcaruana has joined #openstack-kuryr | 09:03 | |
| *** pcaruana has quit IRC | 09:05 | |
| openstackgerrit | Merged openstack/kuryr-tempest-plugin master: demo container: Include ssl certificates https://review.openstack.org/554945 | 09:05 |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: devstack: Make service subnet routable https://review.openstack.org/555015 | 09:08 |
| celebdor | gcheresh_: dulek: We should probably have a gate that exercises the container probes | 09:11 |
| dulek | celebdor: Hm, it shouldn't be too hard for liveness probes. Just login into container, add a rule blocking traffic to K8s API and wait for container restart. | 09:14 |
| dulek | celebdor: I'm not sure about readiness though. | 09:14 |
| celebdor | dulek: It's not important | 09:15 |
| celebdor | I just want to make sure we don't fuck up the access to the pods from kubelet/origin-node | 09:15 |
| dulek | celebdor: Uhm… Well… | 09:20 |
| dulek | celebdor: I'm discussing an error with gcheresh_ at the moment. It might be related… | 09:21 |
| *** yboaron has joined #openstack-kuryr | 09:24 | |
| *** garyloug has joined #openstack-kuryr | 09:30 | |
| dulek | dmellado: Do you know if it'll be possible to add container logs into gate results? | 09:33 |
| dulek | dmellado: When running containerized it'll be useful to have them copied for debugging. | 09:33 |
| *** salv-orlando has joined #openstack-kuryr | 09:35 | |
| *** salv-orlando has quit IRC | 09:38 | |
| *** salv-orlando has joined #openstack-kuryr | 09:38 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-tempest-plugin master: Updated from global requirements https://review.openstack.org/555219 | 09:41 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Revert "Watcher restarts watching resources in failure" https://review.openstack.org/555221 | 09:44 |
| *** gcheresh_ has quit IRC | 09:45 | |
| *** yboaron has quit IRC | 09:46 | |
| celebdor | dulek: I'd like a more detailed explanation than "This commit most likely broke containerized deployments." | 09:47 |
| celebdor | :-) | 09:47 |
| celebdor | dulek: btw, very clever trick with the envs in https://review.openstack.org/#/c/555040/1/cni_ds_init | 09:49 |
| *** gcheresh_ has joined #openstack-kuryr | 09:52 | |
| dulek | celebdor: That's why the revert is on -W and I'm producing the explanation. I hope I'll find a fix instead of a revert. | 09:53 |
| dulek | celebdor: Just we don't have container logs, so I need to debug that locally. | 09:53 |
| dulek | celebdor: This trick is so magical that I always need to spend 20 minutes googling for it when needed. ;) | 09:54 |
| celebdor | dmellado: How can we add logs to the gate? | 09:55 |
| celebdor | dulek: it's common in bash tricks | 09:55 |
| celebdor | :P | 09:55 |
| *** gcheresh has joined #openstack-kuryr | 09:57 | |
| *** gcheresh_ has quit IRC | 09:57 | |
| *** yboaron has joined #openstack-kuryr | 10:03 | |
| *** janonymous has quit IRC | 10:17 | |
| dmellado | lol | 10:29 |
| dmellado | celebdor: what kind of log would you like to add | 10:30 |
| dulek | dmellado: `kubectl logs -lapp=kuryr` | 10:30 |
| dulek | dmellado: I'm currently looking at adding dummy run_process that will do `kubectl logs -f <container_name>` | 10:31 |
| dmellado | I'll check later, being in meetings re: Downstream CI all day long to get an overview later to you | 10:31 |
| dmellado | in any case we can always just add a playbook | 10:31 |
| dmellado | to our run | 10:31 |
| dmellado | so it'll trigger those and store it somehow | 10:31 |
| dmellado | dulek: let me know when you have a draft and I'll use it as a base for my patch | 10:32 |
| dulek | dmellado: I'm blocked at the moment, so I'll try to do that. | 10:32 |
| dulek | dmellado: In case I'll fail with dummy run_process - how can I add a playbook to existing job? | 10:32 |
| dmellado | dulek: in the job definition | 10:33 |
| dmellado | i.e. run: playbooks/sahara-tests-scenario.yaml | 10:33 |
| dmellado | dulek: https://github.com/openstack/sahara-tests/blob/master/.zuul.yaml#L71 | 10:33 |
| dulek | dmellado: Oh, awesome. | 10:34 |
| dmellado | dulek: let me know if you get blocked | 10:34 |
| dulek | dmellado: Sure, thanks! | 10:34 |
| dmellado | dulek: you might also want to check this | 10:36 |
| dmellado | https://docs.openstack.org/infra/manual/zuulv3.html | 10:36 |
| dmellado | https://github.com/openstack-infra/zuul-jobs/tree/master/roles/upload-logs | 10:42 |
| dmellado | https://github.com/openstack-dev/devstack/blob/master/playbooks/post.yaml | 10:43 |
| celebdor | dulek: would you mind if I try to get rid of the "consumes" thing? | 11:02 |
| dulek | celebdor: Like… Completely? In what context?\ | 11:02 |
| celebdor | dulek: well, I'm doing the new Handler interface | 11:03 |
| celebdor | and I always found the consumes thing to be clunky | 11:03 |
| dulek | celebdor: Hm, definitely it's not intuitive to use. You have an alternative? Just filter on the handler method level? | 11:04 |
| celebdor | dulek: I'm thinking about keeping a dict of resource_kind -> handler | 11:04 |
| dulek | celebdor: consumes was able to do more elaborate filtering than on Kind property. | 11:05 |
| celebdor | really? | 11:05 |
| celebdor | I must have missed it | 11:05 |
| * celebdor looking | 11:05 | |
| dulek | celebdor: I've used it to overcome lack of fieldSelector on CRDs and do filtering on nodeName. | 11:06 |
| celebdor | oh, I see | 11:06 |
| celebdor | meh | 11:06 |
| dulek | celebdor: https://review.openstack.org/#/c/527243/11/kuryr_kubernetes/cni/handlers.py@137 | 11:06 |
| dulek | celebdor: Yeah, I can do that in the handler method if needed. And it will probably come without performance penalty at all. | 11:07 |
| *** garyloug has quit IRC | 11:07 | |
| celebdor | dulek: the interface is odd as fuck | 11:08 |
| dulek | celebdor: I definitely agree here. It's impossible to do OR or AND without looking deeply into the code. | 11:08 |
| *** aojea has quit IRC | 11:09 | |
| dulek | dmellado: http://paste.openstack.org/show/708929/ - it was easier than I thought. xD | 11:09 |
| celebdor | dulek: ok, looking at the thing right now... | 11:10 |
| celebdor | wouldn't it make sense to just have a predicate | 11:10 |
| celebdor | why is having the 'key' useful? | 11:10 |
| dulek | celebdor: I don't really know. :P | 11:10 |
| celebdor | in your case https://review.openstack.org/#/c/527243/11/kuryr_kubernetes/cni/handlers.py@137 | 11:10 |
| celebdor | you'd just make a consumes like | 11:11 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Execute kuryr-cni as `docker exec` https://review.openstack.org/555040 | 11:11 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Add Kuryr pods logs to gate results https://review.openstack.org/555254 | 11:11 |
| *** garyloug has joined #openstack-kuryr | 11:11 | |
| celebdor | return k8s_base.object_kind(event) == self.OBJECT_KIND and event['object']['spec']['nodeName'] == self.node_Name | 11:12 |
| dulek | celebdor: Yep, being able to do that would be cool. | 11:14 |
| dulek | celebdor: Instead of weird dict manipulations. | 11:14 |
| celebdor | exactly | 11:14 |
| * dulek -> lunchbreak. | 11:15 | |
| *** gcheresh has quit IRC | 11:16 | |
| celebdor | smacznégo | 11:17 |
| *** gcheresh has joined #openstack-kuryr | 11:26 | |
| *** maysamacedos has joined #openstack-kuryr | 12:06 | |
| *** caowei has quit IRC | 12:06 | |
| *** caowei has joined #openstack-kuryr | 12:06 | |
| *** jlabarre has joined #openstack-kuryr | 12:14 | |
| *** aojea has joined #openstack-kuryr | 12:18 | |
| *** caowei has quit IRC | 12:33 | |
| *** garyloug has quit IRC | 12:36 | |
| *** yamamoto has quit IRC | 12:54 | |
| *** gianpietro has quit IRC | 13:03 | |
| *** gianpietro has joined #openstack-kuryr | 13:03 | |
| dulek | celebdor: http://logs.openstack.org/54/555254/1/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/9ecada1/controller/logs/screen-kuryr-kubernetes.txt.gz#_Mar_22_11_50_32_968611 | 13:03 |
| *** gianpietro has quit IRC | 13:03 | |
| dulek | celebdor: Any idea what was merged recently that broke this? | 13:03 |
| dulek | celebdor: It's healthcheck trying to connect to Keystone. | 13:04 |
| *** gianpietro has joined #openstack-kuryr | 13:04 | |
| dulek | Hm, to be honest I've only pinged you because you've did this CA patch for kuryr/demo. I don't think it's related, maybe it's something on Keystone side? It inserts certs onto the host, but not into the container? | 13:05 |
| *** gianpietro has quit IRC | 13:09 | |
| *** gianpietro has joined #openstack-kuryr | 13:09 | |
| *** garyloug has joined #openstack-kuryr | 13:15 | |
| celebdor | dulek: what inserts certs? | 13:18 |
| dulek | celebdor: DevStack - I guess. Something must have changed in this matter recently. | 13:20 |
| dulek | Though me and Keystone folks were unable to find the commit that caused this. | 13:20 |
| dulek | celebdor: Anyway this raises a fair point - how do we mount certificates into the container? | 13:21 |
| *** yamamoto has joined #openstack-kuryr | 13:21 | |
| celebdor | dulek: that's not the right question | 13:21 |
| dulek | Hm? | 13:21 |
| celebdor | we don't even have those certificates in nested deployments to be able to mount them | 13:22 |
| celebdor | dulek: Since when are we testing devstack with https keystone by default? | 13:23 |
| dulek | celebdor: Oh, right, there are no constraints on where kuryr-controller will land… VM will not have it. | 13:23 |
| celebdor | exactly | 13:23 |
| dulek | celebdor: I think that it's HTTPS from the start. | 13:23 |
| celebdor | did we switch to verifying? | 13:23 |
| dulek | Just a sec. | 13:24 |
| celebdor | ah, we only have the option for k8s https | 13:24 |
| celebdor | for keystone it is probably in oslo | 13:24 |
| dulek | celebdor: Yeah, but it should still set auth_url in kuryr.conf, shouldn't it? | 13:26 |
| dulek | celebdor: auth_url = https://10.209.132.158/identity | 13:27 |
| celebdor | ok | 13:27 |
| dulek | (that's on the gate) | 13:27 |
| celebdor | dulek: what do you have on the configmap | 13:27 |
| celebdor | (for neutron auth) | 13:27 |
| dulek | So we most likely depend on some DevStack variable, though I'm unable to exactly pinpoint it. | 13:27 |
| dulek | celebdor: On DevStack ConfigMap is generated exactly like in non-containerized case. I'm just copying generated kuryr.conf into it. | 13:28 |
| celebdor | dulek: right | 13:28 |
| celebdor | I'm just asking about its content | 13:28 |
| celebdor | since I don't have it fresh :P | 13:29 |
| dulek | celebdor: http://logs.openstack.org/54/555254/1/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/9ecada1/controller/logs/etc/kuryr/kuryr_conf.txt.gz | 13:30 |
| dulek | celebdor: Something like that, it's actually from containerized job. | 13:30 |
| celebdor | dulek: hmmmf | 13:31 |
| celebdor | dulek: can you try with insecure=true | 13:32 |
| celebdor | in the [neutron] section | 13:33 |
| celebdor | but yeah, the fix for devstack is easy | 13:33 |
| dulek | celebdor: Well, I definitely can, though I'd rather find something that's less of a workaround. | 13:33 |
| celebdor | we just need to mount /opt/stack/data/ca-bundle.pem | 13:33 |
| dulek | Yup, but then - I don't really want to have certs in /opt/stack/data *in the container*. | 13:33 |
| celebdor | dulek: please, explain | 13:34 |
| dulek | Those failing logs are from inside the container, right? | 13:34 |
| celebdor | so | 13:34 |
| celebdor | ? | 13:34 |
| dulek | So something sets keystoneauth (or something else) to look for certificates in /opt/stack/data. | 13:35 |
| dulek | I'd rather have this option unset and put the certificates in /etc/ssl, or other standard certs directory. | 13:36 |
| *** aojea has quit IRC | 13:36 | |
| dulek | So I'd mount /opt/stack/data/ca-bundle.cert -> /etc/ssl/ca-bundle.cert. | 13:36 |
| celebdor | dulek: ah, that | 13:36 |
| dulek | But to do that I need to figure out why it had broke in the first place. | 13:36 |
| celebdor | yeah, I don't mind at all | 13:36 |
| celebdor | xD | 13:36 |
| celebdor | whatever you like, in devstack I don't care if it is /opt/stack or /etc/ssl/ca-bundle.cert | 13:37 |
| dulek | celebdor: Still I'd like upstream container to look for certs in /etc/ssl, not /opt/stack/data. I'll figure it out. :) | 13:39 |
| celebdor | well, it's relatively easy to do | 13:39 |
| celebdor | dulek: please open a bug in launchpad and on bz | 13:40 |
| dulek | celebdor: https://bugs.launchpad.net/kuryr-kubernetes/+bug/1758061 | 13:42 |
| openstack | Launchpad bug 1758061 in kuryr-kubernetes "Containerized gate is broken due to OpenStack API certs missing" [Undecided,New] | 13:42 |
| dulek | celebdor: And with BZ you mean the bug we've talked about yesterday, right? | 13:42 |
| *** aojea has joined #openstack-kuryr | 13:46 | |
| celebdor | and this one as well | 13:47 |
| celebdor | dulek: queens downstream will need a way for the ks ca bundle to be accessible to the controller | 13:48 |
| celebdor | although IIRC juriarte made a bug for ssl overcloud support | 13:48 |
| celebdor | which this issue would be related to | 13:48 |
| *** aojea has quit IRC | 13:50 | |
| *** yamamoto has quit IRC | 13:53 | |
| *** celebdor has quit IRC | 13:54 | |
| *** celebdor has joined #openstack-kuryr | 13:57 | |
| *** aojea has joined #openstack-kuryr | 14:00 | |
| *** yamamoto has joined #openstack-kuryr | 14:08 | |
| *** yamamoto has quit IRC | 14:13 | |
| *** hongbin has joined #openstack-kuryr | 14:13 | |
| *** atoth has joined #openstack-kuryr | 14:14 | |
| *** kiennt26_ has joined #openstack-kuryr | 14:15 | |
| *** yamamoto has joined #openstack-kuryr | 14:24 | |
| *** yamamoto has quit IRC | 14:28 | |
| *** garyloug has quit IRC | 14:38 | |
| *** yamamoto has joined #openstack-kuryr | 14:39 | |
| *** yamamoto has quit IRC | 14:43 | |
| *** yamamoto has joined #openstack-kuryr | 14:46 | |
| *** yamamoto has quit IRC | 14:46 | |
| *** garyloug has joined #openstack-kuryr | 15:04 | |
| juriarte | celebdor: yep, the BZ I opened was related to cert support in Openshift-on-Openstack playboo | 15:08 |
| juriarte | when using SSL in the overcloud, you need to mount the CA cert in the container in order to reach the overcloud openstack | 15:08 |
| juriarte | celebdor: don't know if that has something to do with the issue you and dulek mentioned | 15:09 |
| *** aojea_ has joined #openstack-kuryr | 15:09 | |
| dulek | juriarte: Well, partially. For some reason containers started to need that in DevStack as well. | 15:10 |
| dulek | juriarte: And I'm not sure why. | 15:10 |
| juriarte | could it be because devstack started using SSL as well? | 15:10 |
| juriarte | for all services | 15:11 |
| dulek | juriarte: It started recently? I thought it is using it for a while now. | 15:11 |
| juriarte | dulek: don't know, I was just trying to find a reason, but if you think devstack was already using SSL it must be something different | 15:12 |
| dulek | juriarte: I'm quite sure that SSL was enabled for a long time, but I might still be wrong. | 15:13 |
| *** aojea_ has quit IRC | 15:14 | |
| juriarte | dulek, mounting the cert did work? | 15:15 |
| dulek | juriarte: It'll work, I just want to do it properly. ;) | 15:15 |
| dulek | juriarte: That's why I'm looking for root cause. | 15:15 |
| juriarte | dulek: hehe, sure! | 15:16 |
| *** maysamacedos has quit IRC | 15:19 | |
| dulek | Hm… global-requirements.txt update is failing on python-nss installation. That might be related… | 15:20 |
| *** kiennt26_ has quit IRC | 15:27 | |
| *** gcheresh has quit IRC | 15:28 | |
| *** pcaruana has joined #openstack-kuryr | 15:42 | |
| *** pcaruana has quit IRC | 15:44 | |
| *** yamamoto has joined #openstack-kuryr | 15:47 | |
| *** pcaruana has joined #openstack-kuryr | 15:48 | |
| *** yamamoto has quit IRC | 15:53 | |
| *** natanbro has quit IRC | 15:57 | |
| dulek | FYI - newest global-requirements are broken: http://lists.openstack.org/pipermail/openstack-dev/2018-March/128649.html | 16:01 |
| celebdor | lol | 16:04 |
| celebdor | thanks dulek | 16:04 |
| dulek | celebdor: Yeah, well… 2016 is higher than 0.12. ;) | 16:04 |
| dulek | celebdor: http://logs.openstack.org/40/555040/2/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/5e29ef7/controller/logs/screen-kuryr-daemon.txt.gz#_Mar_22_11_41_52_287460 | 16:05 |
| dulek | celebdor: It's with your patch adding the route applied. :( | 16:05 |
| celebdor | wtf | 16:05 |
| dulek | celebdor: I'm pretty sure it worked on my env. I've also checked if the route is added *before* running the container. It is. | 16:05 |
| celebdor | dulek: how did you test that? | 16:06 |
| dulek | celebdor: Well, I've just run the DevStack with your change. | 16:06 |
| celebdor | dulek: oh, I thought you meant in the gate :P | 16:06 |
| dulek | celebdor: Nah, the gate exploded as you can see. :) Only now we can see the logs because I made it depend on 555254. | 16:07 |
| dulek | celebdor: http://logs.openstack.org/40/555040/2/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/5e29ef7/job-output.txt.gz#_2018-03-22_11_35_12_668227 | 16:07 |
| dulek | celebdor: 128? It's 129 then. | 16:08 |
| celebdor | http://logs.openstack.org/40/555040/2/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/5e29ef7/controller/logs/devstacklog.txt.gz | 16:08 |
| dulek | celebdor: Eeeeh. | 16:08 |
| dulek | celebdor: I think we're hitting the same issue that the abandoned patch was hitting. | 16:08 |
| dulek | Damn! | 16:08 |
| celebdor | dulek: which was it? | 16:08 |
| celebdor | I forgot | 16:08 |
| celebdor | the subnets look ok, don't they? | 16:08 |
| dulek | celebdor: Just a moment. | 16:09 |
| dulek | celebdor: So that patch was randomly failing with "already allocated" error | 16:10 |
| dulek | And it had something to do with being unable to reliably pull the IP allocated to K8s service? | 16:10 |
| dulek | yboaron might remember more. | 16:10 |
| dulek | yboaron: https://review.openstack.org/#/c/533343/ | 16:10 |
| celebdor | dulek: but in this case, the route is correctly being put | 16:11 |
| celebdor | am I missing something? | 16:11 |
| yboaron | checking .. | 16:11 |
| dulek | celebdor: Nope. It's set for 10.0.0.128. And K8s service has 10.0.0.129 | 16:11 |
| celebdor | dulek: 10.1.0.128/26 encompasses 10.0.0.129 | 16:12 |
| celebdor | 10.0.0.129 is the first address of the subnet | 16:12 |
| * dulek feels very dumb. :P | 16:12 | |
| dulek | Haven't noticed the /26 | 16:12 |
| celebdor | http://logs.openstack.org/40/555040/2/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/5e29ef7/controller/logs/devstacklog.txt.gz#_2018-03-22_11_35_12_667 | 16:12 |
| celebdor | yeah | 16:12 |
| celebdor | no worries | 16:12 |
| celebdor | so either the interface was down or the router was kaputt | 16:13 |
| yboaron | the abandoned patch , create new if with IP from service cidr, bur forgot to eliminate this IP from Kubernetes , as K8S plays the IPAM for service subnet | 16:13 |
| yboaron | so, this is not the case here, I just run openshift devstack with your change , route and ovs if created | 16:14 |
| dulek | Okay! | 16:14 |
| yboaron | but I can't create pod/services , CNI fails | 16:14 |
| dulek | Uhhh… | 16:14 |
| dulek | We get too many issues. :P | 16:14 |
| yboaron | it's non-containerized | 16:15 |
| celebdor | what? | 16:15 |
| yboaron | the POD watcher at CNI side failed | 16:15 |
| dulek | celebdor: So the issue with this route is repeatable in the gate - just FYI, it's not temporary. | 16:15 |
| yboaron | looking at logs .. | 16:15 |
| yboaron | I don't think it's related to celebdor patch | 16:16 |
| celebdor | dulek: thanks. That comforts me | 16:16 |
| *** yamamoto has joined #openstack-kuryr | 16:17 | |
| celebdor | the router ip is correct as well | 16:17 |
| celebdor | dulek: are we sure the API cluster IP is ready at that point? | 16:19 |
| dulek | celebdor: Wouldn't we get anything else than "no route to host" then? | 16:19 |
| celebdor | dulek: I should know this, but I always end up forgetting | 16:20 |
| dulek | :D | 16:20 |
| dulek | You might be right, looks like there's no waiting for API server at that point. I'll check again. | 16:20 |
| dulek | No, no, wait. | 16:21 |
| dulek | API server is up and running. Otherwise I wouldn't be able to spawn pods in the first place. :P | 16:21 |
| celebdor | dulek: I don't see a wait_for in run_openshift_node | 16:21 |
| celebdor | but then the openshift node would fail anyway | 16:21 |
| dulek | celebdor: And this gate is K8s actually. | 16:22 |
| celebdor | oh | 16:22 |
| dulek | So… What is creating this Service for kubernetes? | 16:22 |
| *** yamamoto has quit IRC | 16:22 | |
| celebdor | just a sec | 16:22 |
| celebdor | dulek: wrong! | 16:22 |
| celebdor | the wait_for that we do is with the controller IP | 16:22 |
| celebdor | not with the cluster IP | 16:23 |
| celebdor | (we also use the host IP for the pod creation and kubelet registration) | 16:23 |
| dulek | celebdor: Yes, yes, that's why I'm asking who's doing `kubectl expose` for kubernetes service. | 16:23 |
| celebdor | dulek: lol | 16:23 |
| celebdor | man... It is automatically done in K8s | 16:24 |
| celebdor | kubernetes always takes the first IP of the service subnet | 16:24 |
| celebdor | and creates the service | 16:24 |
| *** maysamacedos has joined #openstack-kuryr | 16:25 | |
| celebdor | dulek: but here's the deal | 16:25 |
| celebdor | if you look at devstack/plugin.sh | 16:25 |
| celebdor | you'll see that create_k8s_api_service (which creates the LB) | 16:25 |
| celebdor | runs at the very end! | 16:25 |
| celebdor | in stack / test-config phase | 16:25 |
| dulek | Damn. It's not my best day. | 16:26 |
| celebdor | so... | 16:26 |
| celebdor | this won't be that easy of a fix | 16:26 |
| celebdor | we'll have to move the create_k8s_api_service after the API is up | 16:27 |
| celebdor | but before daemonsets are created | 16:27 |
| celebdor | possibly even before the kubelet/openshift-node are created | 16:27 |
| dulek | celebdor: Naaah, there's an easy way. xD | 16:27 |
| celebdor | if I would just remember why the hell I put it so late... | 16:27 |
| yboaron | dulek, celebdor : because ocavia is ready only at this phase! | 16:28 |
| dulek | celebdor: Just make cni_ds_init explode if curl fails. xD | 16:28 |
| dulek | Is this the moment we all get headache? | 16:28 |
| dulek | It's awesome circular dependency! | 16:29 |
| yboaron | I'm confused, what problem are trying to solve ? | 16:29 |
| dulek | yboaron: I'd like to have access to the K8s API when initializing kuryr-cni container. | 16:29 |
| yboaron | and is it related to the Toni's IP route patch ? | 16:29 |
| dulek | yboaron: It is - a bit, because lack of ip route was blocking me in the first place. | 16:30 |
| dulek | yboaron: But now it turns out that without LB for services we're still screwed. | 16:30 |
| yboaron | dulek, thanks! , now I understand the context and problem | 16:31 |
| *** yamamoto has joined #openstack-kuryr | 16:32 | |
| dulek | celebdor: Are we able to simply move CNI DaemonSet creation after LBaaS is created? | 16:32 |
| celebdor | dulek: we could do that | 16:34 |
| celebdor | but frankly, I'm thinking more of moving the lbaas creation | 16:34 |
| celebdor | it would be more proper to create it just after the API is up | 16:34 |
| dulek | celebdor: yboaron says that Octavia is not ready until that late phase. | 16:34 |
| celebdor | and the rest of the stuff should use the K8s clusterip instead of the host ip | 16:35 |
| yboaron | so, for Octavia , I struggled a lot with LB creation pre test-config phase - no success. | 16:35 |
| celebdor | dulek: ah, fuck | 16:35 |
| celebdor | so that's why | 16:35 |
| celebdor | xD | 16:35 |
| celebdor | thanks yboaron | 16:35 |
| celebdor | I totally forgot why we put it there | 16:35 |
| celebdor | damn it to hell | 16:35 |
| celebdor | dulek: well, in that case, we need to move the kuryr containers creation after the create_k8s_api_service and have it be after a wait_for | 16:36 |
| celebdor | dulek: wanna do it or should I? | 16:36 |
| *** yamamoto has quit IRC | 16:36 | |
| dulek | celebdor: I'll do it in my patch. | 16:37 |
| celebdor | ok | 16:37 |
| celebdor | dulek: maybe you can just move the run_containerized_resources call | 16:38 |
| yboaron | are we fine with creating CNI containers in 'test-config' phase ? | 16:38 |
| celebdor | yboaron: you mean tempest wise? | 16:38 |
| dulek | I don't see any obstacles immediately. | 16:38 |
| yboaron | I mean CNI containers are vital component of Kuryr , and as such component I would expect to run in phase other than 'test-config' | 16:39 |
| celebdor | yboaron: the problem is with having such deps | 16:41 |
| celebdor | dulek: can you remind me where you get the service IP from? | 16:41 |
| dulek | celebdor: env vars in container. | 16:42 |
| dulek | celebdor: Simple as that. | 16:42 |
| celebdor | ah, right | 16:42 |
| celebdor | :-) | 16:42 |
| celebdor | it's either envs or dns | 16:42 |
| dulek | celebdor: Yup. | 16:43 |
| dulek | Okay, you know how do you setup SSL in DevStack? :D | 16:43 |
| dulek | enable_service tls-proxy | 16:43 |
| dulek | We have it in the gate but not in sample configs. Now I should reproduce this locally. | 16:44 |
| dulek | Now I'll find the job config commit that broke us… | 16:44 |
| celebdor | :-) | 16:45 |
| *** yamamoto has joined #openstack-kuryr | 16:47 | |
| *** yamamoto has quit IRC | 16:52 | |
| *** gcheresh has joined #openstack-kuryr | 16:57 | |
| *** aojea_ has joined #openstack-kuryr | 16:58 | |
| *** aojea_ has quit IRC | 17:02 | |
| *** yamamoto has joined #openstack-kuryr | 17:02 | |
| dulek | celebdor: OpenShift-Ansible will have access to the required certs? I can make those a K8s Secret. | 17:06 |
| *** yboaron has quit IRC | 17:06 | |
| *** yamamoto has quit IRC | 17:06 | |
| celebdor | well, it's a CA cert only, so I'm not sure how private they need to be | 17:07 |
| dulek | celebdor: Well, whatever, Secret is just a fancy name for ConfigMap. :P | 17:07 |
| celebdor | but I guess it's nice to use a K8s secret if it's not too much trouble | 17:07 |
| celebdor | ConfigMap readability also depends on the user, doesn't it? | 17:07 |
| dulek | It's the best way IMO - like kuryr.conf those are generated on deployment, so should be added into K8s. | 17:08 |
| dulek | celebdor: Ha, I'm not sure about readability. | 17:08 |
| dulek | celebdor: "In the future there will likely be some differentiators for secrets like rotation or support for backing the secret API w/ HSMs, etc. In general we like intent-based APIs, and the intent is definitely different for secret data vs. plain old configs." | 17:09 |
| dulek | ;) | 17:09 |
| celebdor | dulek: right | 17:10 |
| *** yamamoto has joined #openstack-kuryr | 17:17 | |
| * celebdor -> with kids | 17:17 | |
| *** gcheresh has quit IRC | 17:18 | |
| *** yamamoto has quit IRC | 17:22 | |
| *** gianpietro has quit IRC | 17:27 | |
| *** gianpietro has joined #openstack-kuryr | 17:28 | |
| *** yamamoto has joined #openstack-kuryr | 17:32 | |
| *** gianpietro has quit IRC | 17:32 | |
| *** yamamoto has quit IRC | 17:37 | |
| dulek | celebdor: Can you get your -W from https://review.openstack.org/#/c/554826/ ? It happened not to be the cause of the issues. | 17:37 |
| *** yamamoto has joined #openstack-kuryr | 17:47 | |
| *** yamamoto has quit IRC | 17:52 | |
| *** yamamoto has joined #openstack-kuryr | 18:02 | |
| *** garyloug has quit IRC | 18:05 | |
| *** yamamoto has quit IRC | 18:06 | |
| *** yamamoto has joined #openstack-kuryr | 18:06 | |
| *** yamamoto has quit IRC | 18:06 | |
| *** aojea has quit IRC | 18:43 | |
| *** aojea has joined #openstack-kuryr | 18:46 | |
| *** celebdor has quit IRC | 18:50 | |
| *** aojea has quit IRC | 18:50 | |
| *** aojea has joined #openstack-kuryr | 19:01 | |
| *** celebdor has joined #openstack-kuryr | 19:05 | |
| *** yamamoto has joined #openstack-kuryr | 19:07 | |
| *** yamamoto has quit IRC | 19:13 | |
| *** gianpietro has joined #openstack-kuryr | 19:22 | |
| *** aojea has quit IRC | 19:23 | |
| *** gianpietro has quit IRC | 19:33 | |
| *** aojea has joined #openstack-kuryr | 19:43 | |
| *** aojea has quit IRC | 19:45 | |
| *** aojea has joined #openstack-kuryr | 19:45 | |
| *** aojea has quit IRC | 19:46 | |
| *** aojea has joined #openstack-kuryr | 19:49 | |
| *** vikasc has quit IRC | 19:54 | |
| *** yamamoto has joined #openstack-kuryr | 20:09 | |
| *** vikasc has joined #openstack-kuryr | 20:10 | |
| *** yamamoto has quit IRC | 20:14 | |
| *** aojea has quit IRC | 20:17 | |
| *** aojea has joined #openstack-kuryr | 20:18 | |
| openstackgerrit | Doug Hellmann proposed openstack/fuxi master: add lower-constraints job https://review.openstack.org/555448 | 20:25 |
| *** gcheresh has joined #openstack-kuryr | 20:25 | |
| *** aojea has quit IRC | 20:33 | |
| *** aojea has joined #openstack-kuryr | 20:34 | |
| *** aojea has quit IRC | 20:57 | |
| *** aojea has joined #openstack-kuryr | 21:07 | |
| *** yamamoto has joined #openstack-kuryr | 21:10 | |
| *** yamamoto has quit IRC | 21:16 | |
| *** salv-orlando has quit IRC | 21:19 | |
| *** salv-orlando has joined #openstack-kuryr | 21:19 | |
| *** salv-orlando has quit IRC | 21:24 | |
| *** celebdor has quit IRC | 21:25 | |
| *** salv-orlando has joined #openstack-kuryr | 21:26 | |
| *** maysamacedos has quit IRC | 21:28 | |
| *** gcheresh has quit IRC | 21:43 | |
| *** pcaruana has quit IRC | 21:53 | |
| *** yamamoto has joined #openstack-kuryr | 21:55 | |
| *** livelace has joined #openstack-kuryr | 22:10 | |
| livelace-link | Hello, everybody. Does Kuryr work with cri-o ? | 22:11 |
| *** livelace has quit IRC | 22:11 | |
| *** dougbtv_ has joined #openstack-kuryr | 22:25 | |
| *** dougbtv has quit IRC | 22:28 | |
| *** maysamacedos has joined #openstack-kuryr | 22:36 | |
| *** aojea has quit IRC | 22:49 | |
| *** jlabarre has quit IRC | 22:54 | |
| *** hongbin has quit IRC | 23:01 | |
| *** maysamacedos has quit IRC | 23:06 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Add CA certificates Secret and mount it https://review.openstack.org/555502 | 23:09 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!