| goldyfruit | So far I'm stuck to the function execution | 00:14 |
|---|---|---|
| goldyfruit | qinling-engine is able to connect to Kubernetes API and etcd | 00:15 |
| goldyfruit | but for any reason it gets the internal API of my master | 00:15 |
| goldyfruit | https://paste.api-zulu.com/raw/ijayenetun | 00:15 |
| goldyfruit | I' ve to leave | 00:20 |
| goldyfruit | If you want more detail, I sent you my email by DM | 00:20 |
| *** goldyfruit has quit IRC | 00:26 | |
| *** goldyfruit has joined #openstack-qinling | 00:38 | |
| goldyfruit | Back | 00:40 |
| goldyfruit | I'll try to stay connected | 00:40 |
| goldyfruit | (my laptop at least) | 00:40 |
| *** goldyfruit has quit IRC | 01:58 | |
| *** goldyfruit_ has joined #openstack-qinling | 01:59 | |
| *** goldyfruit_ has quit IRC | 02:20 | |
| *** zigo has quit IRC | 07:28 | |
| *** zigo has joined #openstack-qinling | 09:36 | |
| *** zigo has quit IRC | 10:30 | |
| *** zigo has joined #openstack-qinling | 10:40 | |
| *** goldyfruit_ has joined #openstack-qinling | 11:55 | |
| *** goldyfruit_ has quit IRC | 12:19 | |
| *** goldyfruit_ has joined #openstack-qinling | 12:36 | |
| *** goldyfruit_ has quit IRC | 15:24 | |
| *** goldyfruit has joined #openstack-qinling | 16:24 | |
| goldyfruit | Hey guys | 16:32 |
| goldyfruit | lxkong, I'm still facing my issue with nodeport, my security groups for masters/minions are wild open | 16:33 |
| goldyfruit | Do I need something else | 16:34 |
| goldyfruit | Even directly in the master node I'm not able to reach the cluster IP | 17:12 |
| *** goldyfruit has quit IRC | 17:59 | |
| *** goldyfruit has joined #openstack-qinling | 18:23 | |
| goldyfruit | If I spawn the PODs and the SVC from my own template (using the openstackqinling runtime image), I'm able to get the /pong | 18:31 |
| goldyfruit | meaning something goes wrong during the provisioning by Qinling | 18:32 |
| goldyfruit | The only difference I found from my template and the Qinling's one is "nodePort: 3XXXX" value missing from Qinling | 18:36 |
| goldyfruit | The main difference is: Endpoints: 192.168.241.148:9090 | 18:58 |
| goldyfruit | I don't have this when I'm able to reach my pod | 18:58 |
| goldyfruit | And I have annotation | 18:59 |
| goldyfruit | https://paste.api-zulu.com/raw/gogeqiqumo | 18:59 |
| lxkong | goldyfruit: just double check, your k8s cluster is created using Magnum, right? | 21:17 |
| lxkong | `url: http://10.0.0.58:30978/execute`, do you mean `10.0.0.58` is your internal IP address? | 21:23 |
| lxkong | goldyfruit: does each node in your cluster have type: ExternalIP address? | 21:23 |
| goldyfruit | My cluster is deployed with Magnum | 21:30 |
| goldyfruit | 10.0.0.58 was a private IP | 21:30 |
| goldyfruit | I fixed that, now it try to reach a FIP | 21:31 |
| goldyfruit | but without success | 21:31 |
| goldyfruit | lxkong, | 21:32 |
| lxkong | what's the error? | 21:32 |
| goldyfruit | https://paste.api-zulu.com/raw/dupikudoji | 21:33 |
| goldyfruit | I feel like I miss something | 21:33 |
| lxkong | ? | 21:34 |
| goldyfruit | I don't understand what I missed in my setup | 21:35 |
| lxkong | if you create a nodeport service in the cluster, can you access it from the qinling control plane? | 21:37 |
| goldyfruit | yep | 21:39 |
| lxkong | hmm... | 21:39 |
| lxkong | if the service and all the pods are still alive, you can check the connection using curl | 21:40 |
| goldyfruit | DEBUG qinling.engine.default_engine [req-ae22590f-37c6-407f-a3ec-020938c214c6 e9c13653afc24d0e85d97d42fc781810 5eeb5916ae4b43699f06ea422e581b83 - default default] Found service url for function: a8805e50-014f-4b3d-9ab4-3d200b43b2b7(version 0), execution: e387a2ed-79d7-4a77-88bf-de05da7956f6, url: http://192.207.60.181:30410/execute create_execution /var/lib/kolla/venv/local/lib/python2.7/site-packages/qinling/engine/default_engine.py:168 | 21:42 |
| goldyfruit | curl timeout | 21:43 |
| goldyfruit | on k8s I see the service created | 21:43 |
| goldyfruit | qinling service-a8805e50-014f-4b3d-9ab4-3d200b43b2b7-0 NodePort 10.254.21.2 <none> 9090:30410/TCP 82s function_id=a8805e50-014f-4b3d-9ab4-3d200b43b2b7,function_version=0,pod-template-hash=5f7bb6fb44,runtime_id=0cbd4f27-57bc-411a-ab9b-9348c9546179 | 21:43 |
| goldyfruit | 192.207.60.181 is the FIP of one of my master | 21:43 |
| goldyfruit | This is the pod and the svc from the qinling namespace: https://paste.api-zulu.com/raw/onuzidokel | 21:48 |
| goldyfruit | I'm using Kubernetes 1.13.4 | 21:50 |
| goldyfruit | Here the describe of the service created by Qinling: https://paste.api-zulu.com/raw/goyahafoga | 21:52 |
| goldyfruit | Even from inside the cluster I'm not able to get the cluster ip | 21:55 |
| lxkong | what's your magnum version? | 21:58 |
| goldyfruit | 8.0 | 21:58 |
| goldyfruit | (Stein) | 21:58 |
| lxkong | can you show me the security group rules associated with the worker node port? | 21:58 |
| lxkong | also, just for sure, please also check the network policy in the cluster? | 21:58 |
| goldyfruit | https://paste.api-zulu.com/zawasecija.rb | 21:59 |
| goldyfruit | security gorup rules for master and minions | 22:00 |
| goldyfruit | lxkong, what do you mean pas network policy in the cluster, how can I do that ? | 22:00 |
| goldyfruit | (not a k8s expert :() | 22:00 |
| lxkong | `kubectl get networkpolicy -n <namespace>` | 22:01 |
| lxkong | if there is one, describe it | 22:01 |
| goldyfruit | NAME POD-SELECTOR AGE | 22:02 |
| goldyfruit | allow-qinling-engine-only <none> 8h | 22:02 |
| goldyfruit | https://paste.api-zulu.com/fitaxuwemi.pas | 22:02 |
| lxkong | could you try to remove that network policy? | 22:04 |
| lxkong | and try again to curl | 22:05 |
| goldyfruit | :o ! | 22:06 |
| goldyfruit | OMG | 22:06 |
| lxkong | ? | 22:06 |
| goldyfruit | I'm able to reach the nodeport | 22:06 |
| lxkong | any luck? | 22:06 |
| lxkong | nice, then the network policy is the problem | 22:06 |
| goldyfruit | # curl http://192.207.60.181:30410/ping -I | 22:06 |
| goldyfruit | HTTP/1.1 200 OK | 22:06 |
| goldyfruit | Content-Type: text/html; charset=utf-8 | 22:06 |
| goldyfruit | Content-Length: 4 | 22:06 |
| lxkong | we should fix that | 22:06 |
| lxkong | goldyfruit: please continue the journey, i'm ready to answer the next question :-) | 22:07 |
| goldyfruit | the trusted_cidrs should not avoid the IPBlock ? | 22:07 |
| goldyfruit | trusted_cidrs = 10.128.150.4/32 | 22:07 |
| goldyfruit | Ok, now I got Failed to download function package from https://api.mycloud.com:7070/v1/functions/a8805e50-014f-4b3d-9ab4-3d200b43b2b7?download=true | 22:09 |
| goldyfruit | I guess I'll set the internal endpoint in qinling | 22:09 |
| lxkong | goldyfruit: well, if you are using calico as the network plugin, the IPBlock cidr is only valid for the internal cluster ips | 22:10 |
| goldyfruit | Hum | 22:10 |
| goldyfruit | I'm using calico | 22:10 |
| lxkong | for now, just remove the network policy. | 22:10 |
| lxkong | goldyfruit: you are working on a public cloud or private? | 22:11 |
| goldyfruit | public | 22:11 |
| goldyfruit | and private | 22:11 |
| lxkong | :-) | 22:11 |
| lxkong | ok, for now, just remove the network policy, i will create an issue for tracking | 22:12 |
| goldyfruit | I removed the network policy | 22:12 |
| lxkong | and the pod needs to talk to qinling-api for package downloading | 22:12 |
| goldyfruit | :o | 22:13 |
| goldyfruit | That is my issue there | 22:13 |
| goldyfruit | I need to open 7070 | 22:13 |
| lxkong | where it's blocked? | 22:13 |
| lxkong | 7070 is the qinling service port | 22:13 |
| goldyfruit | My k8s cluster is in a different place than the API | 22:14 |
| goldyfruit | Waiting from the network guy :p | 22:14 |
| lxkong | ok. In our cloud, we allow the cluster nodes to talk to the api services. | 22:14 |
| goldyfruit | Yeah, I just requested the same :) | 22:15 |
| goldyfruit | are you running go runtime ? | 22:15 |
| goldyfruit | In fact, what runtimes are you running ? | 22:15 |
| lxkong | no, qinling only supports python. We are not running qinling in production actually, it's in alpha, and we plan to integrate qinling with knative at some time in the future. | 22:16 |
| lxkong | because in openstack community, there are not many people asking for FaaS, even after almost 1.5 years since Qinling was created | 22:17 |
| goldyfruit | To be honest, the installation is quite complicated | 22:17 |
| goldyfruit | None of the installers are able to deploy Qinling | 22:18 |
| lxkong | goldyfruit: yeah, how did you install other openstack services? | 22:18 |
| goldyfruit | I integrated Qinling to Kolla in our case, I'll push it upstream (hoping they will merge it) | 22:18 |
| goldyfruit | I really think that it's a very cool project | 22:19 |
| lxkong | yeah, usually it's up to other community (e.g. ubuntu, kolla, openstack-ansible) to help to support qinling installation, but according to the current upstream situation in openstack...you know | 22:19 |
| goldyfruit | Just not enough "marketing" around it | 22:19 |
| lxkong | and not feedbacks | 22:20 |
| lxkong | as a result, it's in the maintenance for now. I'm answering question here but don't put much time working on that. | 22:20 |
| goldyfruit | We install OpenStack using Kolla from OpenStack tarballs, which avoid the packages situation | 22:20 |
| goldyfruit | I understand | 22:21 |
| lxkong | until we are at a point to say, ok, we need to provide FaaS to the public, and we spend time on evalutating knative and so integration and development | 22:21 |
| goldyfruit | You said Qinling support only Python | 22:21 |
| lxkong | yes | 22:21 |
| goldyfruit | If we build our runtimes it will support other langages, right ? | 22:21 |
| lxkong | yep, sure | 22:22 |
| goldyfruit | (you scared me :p) | 22:22 |
| lxkong | the runtime part is actually just an image | 22:22 |
| lxkong | as cloud provider, you can use any image you want | 22:22 |
| goldyfruit | Yeah, I checked a bit on github/docker hub | 22:22 |
| lxkong | only need to implement some functions | 22:22 |
| lxkong | the image in Qinling upstream is only for dev purpose, i believe most of the cloud provider will have their own image for security reasons | 22:23 |
| goldyfruit | Yeah for the CI as I read | 22:23 |
| lxkong | correct | 22:23 |
| goldyfruit | Resources on Qinling are missing which is why it's not popular I thing | 22:24 |
| goldyfruit | think* | 22:24 |
| lxkong | yeah | 22:24 |
| goldyfruit | When I was looking on Google to install it I only got few links and most of them were from your blog post | 22:25 |
| goldyfruit | which I had to translate :D | 22:25 |
| lxkong | hah | 22:25 |
| lxkong | i should write in English from the beginning | 22:26 |
| goldyfruit | I think one thing that could help a lot is the official documentation and the integration with Magnum | 22:26 |
| lxkong | i agree, we had one person who were mainly working on the doc a while ago, but he left openstack unfortunately | 22:28 |
| goldyfruit | As many people | 22:29 |
| goldyfruit | But OpenStack still there | 22:29 |
| goldyfruit | More stable as ever | 22:29 |
| goldyfruit | (I started in 2012) | 22:29 |
| lxkong | yeah, and especially after k8s is becoming a buzz, we see a log of advantages to run k8s on top of openstack | 22:30 |
| goldyfruit | Yep! | 22:30 |
| lxkong | it's much flexible to integrate the cloud services with k8s, do auto-scaling, auto-healing | 22:31 |
| lxkong | etc. | 22:31 |
| goldyfruit | Exactly | 22:31 |
| lxkong | so we have deployed magnum (now in beta), and myself has been working on cloud-provider-openstack since last year. | 22:32 |
| lxkong | that's another reason i didn't have much time on QInling :-) | 22:32 |
| lxkong | because costomers are still asking from container infra at the moment, FaaS may be the next thing | 22:33 |
| goldyfruit | Totally understand | 22:33 |
| goldyfruit | I found one of your PR | 22:33 |
| goldyfruit | about PVC | 22:33 |
| goldyfruit | https://github.com/kubernetes/cloud-provider-openstack/pull/405 | 22:34 |
| goldyfruit | Are you using the cloud provider 1.14 ? | 22:34 |
| goldyfruit | So I opened the port | 22:34 |
| lxkong | 1.14 you mean CPO(cloud-provider-openstack)? | 22:34 |
| goldyfruit | yes | 22:35 |
| lxkong | yes, we are using the latest version | 22:35 |
| goldyfruit | Good to know | 22:35 |
| goldyfruit | I got this error when I execute the function | 22:35 |
| goldyfruit | https://paste.api-zulu.com/raw/epaxipowap | 22:35 |
| goldyfruit | I followed this doc: https://docs.openstack.org/qinling/latest/quick_start.html | 22:36 |
| goldyfruit | I'm using Qinling master branch | 22:39 |
| lxkong | goldyfruit: can you check the pod log? | 22:40 |
| lxkong | you can use `kubectl get po --show-labels` to find out which pod is working | 22:40 |
| goldyfruit | https://paste.api-zulu.com/uximemuzec.sql | 22:41 |
| goldyfruit | I'll try the python3 runtime | 22:43 |
| goldyfruit | python3 is a success | 22:43 |
| goldyfruit | +------------------+--------------------------------------+ | 22:43 |
| goldyfruit | | Field | Value | | 22:43 |
| goldyfruit | +------------------+--------------------------------------+ | 22:43 |
| goldyfruit | | id | 221294ff-2922-4ee7-9c8f-03f8e796a689 | | 22:43 |
| goldyfruit | | function_id | c73357a6-8fb0-46f1-9c6b-1103ca1656ab | | 22:43 |
| goldyfruit | | function_version | 0 | | 22:44 |
| goldyfruit | | description | None | | 22:44 |
| goldyfruit | | input | None | | 22:44 |
| goldyfruit | | result | {"duration": 0.505, "output": 30} | | 22:44 |
| goldyfruit | | status | success | | 22:44 |
| goldyfruit | | sync | True | | 22:44 |
| goldyfruit | | project_id | 5eeb5916ae4b43699f06ea422e581b83 | | 22:44 |
| goldyfruit | | created_at | 2019-05-15 22:43:42 | | 22:44 |
| goldyfruit | | updated_at | 2019-05-15 22:43:45 | | 22:44 |
| goldyfruit | +------------------+--------------------------------------+ | 22:44 |
| goldyfruit | openstack function execution log show 221294ff-2922-4ee7-9c8f-03f8e796a689 | 22:45 |
| goldyfruit | WARN: Resource limiting failed, run in unlimit mode. | 22:45 |
| goldyfruit | Start execution: 221294ff-2922-4ee7-9c8f-03f8e796a689 | 22:45 |
| goldyfruit | Finished execution: 221294ff-2922-4ee7-9c8f-03f8e796a689 | 22:45 |
| goldyfruit | I'm going forward ! | 22:45 |
| lxkong | hooray! | 22:47 |
| goldyfruit | Just looking for Resource limiting failed, run in unlimit mode | 22:47 |
| lxkong | Resource limiting is a feature wasn't well implemented | 22:48 |
| lxkong | if yo don't mind, could you please create several stories in https://storyboard.openstack.org/#!/project/openstack/qinling for all the issues you've seen? | 22:49 |
| goldyfruit | I'll | 22:49 |
| lxkong | much appreciate | 22:49 |
| goldyfruit | openstack function create --cpu unlimited should work right ? | 22:49 |
| goldyfruit | No problem, this is the least i can do ! | 22:49 |
| goldyfruit | I guess it wants a numeric value | 22:50 |
| lxkong | it's based on cgroup on the worker node, if it's not supported, then all the resource limit parameter won't work, so you could ignore those params | 22:52 |
| lxkong | fyi, https://paste.api-zulu.com/zixolasiku.py | 22:53 |
| goldyfruit | Great | 22:53 |
| goldyfruit | So, I have FaaS working on my environment! | 22:54 |
| lxkong | congrat! (i know how frustration you were in the jouney) | 22:54 |
| lxkong | anyway, i will leave you here and have to spend some time on my daily job | 22:54 |
| goldyfruit | Not that much, don't worry. I started to work on it yesterday morning | 22:54 |
| goldyfruit | Sure! | 22:54 |
| lxkong | feel free to ping me or leave msg here if you have any other questiosn | 22:55 |
| goldyfruit | Thanks again for your help, I'll create the stories | 22:55 |
| goldyfruit | I'll, thanks | 22:55 |
| lxkong | np | 22:55 |
| *** goldyfruit has quit IRC | 23:22 | |
| *** goldyfruit has joined #openstack-qinling | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!