| *** goldyfruit_ has joined #openstack-containers | 00:07 | |
| *** flwang1 has joined #openstack-containers | 00:14 | |
| *** sapd1_x has joined #openstack-containers | 00:25 | |
| *** goldyfruit_ has quit IRC | 00:33 | |
| *** goldyfruit has joined #openstack-containers | 00:35 | |
| *** goldyfruit has quit IRC | 01:00 | |
| *** sapd1_x has quit IRC | 01:09 | |
| *** sapd1_x has joined #openstack-containers | 01:25 | |
| *** goldyfruit has joined #openstack-containers | 01:26 | |
| *** sapd1_x has quit IRC | 01:37 | |
| *** hongbin has joined #openstack-containers | 01:49 | |
| openstackgerrit | Merged openstack/magnum-ui master: Update the constraints url https://review.opendev.org/682856 | 02:22 |
|---|---|---|
| openstackgerrit | Merged openstack/magnum-ui master: Generate PDF documentation https://review.opendev.org/682897 | 02:22 |
| *** hongbin has quit IRC | 02:36 | |
| *** ykarel has joined #openstack-containers | 02:39 | |
| *** ricolin has joined #openstack-containers | 02:47 | |
| *** ykarel_ has joined #openstack-containers | 02:48 | |
| *** ykarel has quit IRC | 02:51 | |
| *** dave-mccowan has quit IRC | 02:53 | |
| *** flwang1 has quit IRC | 03:28 | |
| *** ramishra has joined #openstack-containers | 03:44 | |
| *** udesale has joined #openstack-containers | 04:07 | |
| *** goldyfruit has quit IRC | 04:24 | |
| *** ykarel_ has quit IRC | 04:56 | |
| *** iokiwi has quit IRC | 05:09 | |
| *** iokiwi has joined #openstack-containers | 05:09 | |
| *** ykarel_ has joined #openstack-containers | 05:16 | |
| *** pcaruana has joined #openstack-containers | 06:42 | |
| *** ykarel_ is now known as ykarel | 07:26 | |
| *** dims has quit IRC | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:56 | |
| *** ivve has joined #openstack-containers | 08:01 | |
| strigazi | brtknr: meeting today? Feilong is not online. | 08:23 |
| brtknr | yes, in 30 mins right? | 08:23 |
| strigazi | I don't know, whiithout feilong doesn't make much sense. | 08:28 |
| strigazi | brtknr: any issues with nodegroups? feilong said nothing works? | 08:42 |
| brtknr | strigazi: i use a dedicated baremetal devstack for testing magnum changes and i didnt see any of those problem... | 08:45 |
| brtknr | strigazi: I'd like to retest but could we rebase all the changes to the current master? theres quite a few merge conflicts | 08:46 |
| brtknr | is ttstious arround? | 08:48 |
| strigazi | he is | 08:50 |
| strigazi | but we can't rebase indefinetely, let's test the current state and then rebase? | 08:50 |
| *** ykarel|lunch is now known as ykarel | 08:53 | |
| brtknr | strigazi: i was worried it was missing the calico changes but its not | 08:55 |
| brtknr | i'll verify again | 08:55 |
| brtknr | strigazi: did flwang tell you he's not able to attend? | 09:04 |
| brtknr | lets start the meeting, its 9AM UTC | 09:06 |
| strigazi | #startmeeting containers | 09:08 |
| openstack | Meeting started Wed Sep 25 09:08:41 2019 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:08 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:08 |
| *** openstack changes topic to " (Meeting topic: containers)" | 09:08 | |
| openstack | The meeting name has been set to 'containers' | 09:08 |
| strigazi | #topic Roll Call | 09:08 |
| *** openstack changes topic to "Roll Call (Meeting topic: containers)" | 09:08 | |
| strigazi | o/ | 09:08 |
| jakeyip | o/ | 09:09 |
| strigazi | brtknr: | 09:10 |
| brtknr | o/ | 09:12 |
| *** ttsiouts has joined #openstack-containers | 09:12 | |
| strigazi | #topic Stories and Tasks | 09:13 |
| *** openstack changes topic to "Stories and Tasks (Meeting topic: containers)" | 09:13 | |
| ttsiouts | o/ | 09:13 |
| strigazi | let's discuss quickly fedora coreos status and reasoning | 09:13 |
| strigazi | then nodegroups | 09:13 |
| strigazi | brtknr: jakeyip anything else you want to discuss | 09:13 |
| strigazi | ? | 09:13 |
| brtknr | stein backports | 09:14 |
| jakeyip | nothing from me | 09:14 |
| strigazi | ok | 09:14 |
| brtknr | also when to cut the train release | 09:15 |
| strigazi | So for CoreOS | 09:15 |
| strigazi | 1. we need to change from Atomic, there is no discussion around it | 09:15 |
| strigazi | 2. Fedora CoreOS is the "replacement" supported by the same team | 09:16 |
| strigazi | I say replacement because it is not drop in replacement | 09:16 |
| strigazi | I mean replacement in quotes | 09:16 |
| strigazi | reasons to use it, at least from my POV | 09:17 |
| strigazi | we have good communication with that community | 09:17 |
| strigazi | the goal is to run the stock OS and run everything in containers | 09:18 |
| brtknr | also they told me yesterday that they would like to support our use case transition from `atomic install --system ...` | 09:18 |
| strigazi | the transition is probably podman run | 09:18 |
| strigazi | any counter argument? | 09:20 |
| jakeyip | sounds good | 09:20 |
| brtknr | at first, my worry was no more `atomic` but i am more reassured by the fact that the intended replacement is podman/docker | 09:20 |
| strigazi | the work required is around the heat agent and replacement for the atomic cli | 09:21 |
| brtknr | we should be able to run privileged container for kube-* services right? | 09:21 |
| strigazi | atomic is just a python cli that writes a systemd unit which does "runc run" | 09:21 |
| strigazi | we could | 09:22 |
| brtknr | and podman is containers running under systemd iiuc | 09:22 |
| strigazi | I hope at least, because k8s 1.16 is not playing nice in a container | 09:23 |
| strigazi | yes | 09:23 |
| strigazi | like my comment in https://review.opendev.org/#/c/678458/ | 09:23 |
| brtknr | >I hope at least, because k8s 1.16 is not playing nice in a container | 09:23 |
| brtknr | in what way? | 09:23 |
| strigazi | the kubelet container is not propagating the mounts to the host | 09:24 |
| strigazi | only kubelet, the others are fine | 09:24 |
| strigazi | let's move to nodegroups? we won't solve this here | 09:25 |
| strigazi | I mean the 1.16 issue | 09:25 |
| brtknr | sounds like problem with podman? | 09:25 |
| strigazi | that was with atomic, not podman | 09:26 |
| strigazi | podman, atomic, they all use runc | 09:26 |
| brtknr | okay.. anyway i think what we need is to convince ourselves fcos is the best alternative | 09:26 |
| brtknr | before we build more momentum | 09:27 |
| strigazi | I'm convinced, whenever I ask the ubuntu community for help, I found the door closed | 09:28 |
| brtknr | I suppose the community is an important aspect... | 09:28 |
| strigazi | brtknr: what are your concerns? | 09:29 |
| strigazi | anyway, we will go to their meeting and we see. | 09:29 |
| strigazi | #topic Nodegroups | 09:30 |
| brtknr | i am just concerned about the risks as it seems experimantal | 09:30 |
| *** openstack changes topic to "Nodegroups (Meeting topic: containers)" | 09:30 | |
| brtknr | as with all exciting things XD | 09:30 |
| strigazi | compared with centos, openstack and kubernetes is too experimental | 09:30 |
| strigazi | or compared with debian, apache server, I can bring more :) | 09:31 |
| brtknr | i think we need to find out from fcos community what are the things that are going to stay and things that may be uprooted | 09:32 |
| brtknr | but happy to move on to nodegroups | 09:32 |
| strigazi | we are fixing an issue with labels, brtknr did you find anything? | 09:35 |
| strigazi | did manage to add nodegroups? | 09:35 |
| strigazi | did you manage to add nodegroups? | 09:35 |
| brtknr | strigazi: i just created a cluster but i was using kube_tag=v1.16.0 so it failed | 09:36 |
| brtknr | retrying now with v1.15.3 | 09:36 |
| brtknr | but i have tested the full lifecycle in one of the earlier patchsets | 09:38 |
| brtknr | create update and delete, also scaling | 09:38 |
| brtknr | and everything seemed to work for me | 09:38 |
| brtknr | also nice work adding the tests ttsiouts | 09:39 |
| brtknr | it feels like a complete package now | 09:39 |
| ttsiouts | brtknr: i'll push again today adapting your comments | 09:39 |
| ttsiouts | brtknr: we also identified this issue with labels that strigazi mentioned | 09:40 |
| ttsiouts | brtknr: thanks for testing! | 09:40 |
| strigazi | brtknr: perfect | 09:43 |
| brtknr | ttsiouts: i will repost my output to ng-6 saying everything is working for me | 09:43 |
| strigazi | excellent | 09:44 |
| strigazi | oh, one more thing | 09:44 |
| strigazi | for nodegroups, we (CERN) need to spawn cluster across projects | 09:44 |
| strigazi | for nodegroups, we (CERN) need to spawn clusters across projects | 09:44 |
| strigazi | eg ng1 in project p1 and ng2 in project p2 | 09:45 |
| brtknr | so one nodegroup in 1 project another in a different project? | 09:45 |
| strigazi | yes | 09:45 |
| brtknr | that sounds messy... | 09:45 |
| strigazi | in the db og ngs, we have proejct_id already | 09:45 |
| brtknr | isnt there tenant isolation between networks? | 09:45 |
| strigazi | nova is messy | 09:45 |
| strigazi | so the mess comes from tehre | 09:46 |
| strigazi | so the mess comes from there | 09:46 |
| brtknr | or are you planning to use public interface? | 09:46 |
| brtknr | s/public/external | 09:46 |
| strigazi | public inteface | 09:46 |
| brtknr | hmm interesting | 09:46 |
| strigazi | yes, it depends on what you use the cluster for | 09:46 |
| *** flwang1 has joined #openstack-containers | 09:46 | |
| flwang1 | sorry for the late | 09:47 |
| brtknr | hi flwang1 :) | 09:47 |
| flwang1 | was taking care sick kids | 09:47 |
| flwang1 | brtknr: hello | 09:47 |
| flwang1 | is strigazi around? | 09:47 |
| strigazi | for our usage and it is not an issue, and it is opt-in anyway | 09:47 |
| brtknr | yep | 09:47 |
| strigazi | hi flwang1 | 09:47 |
| flwang1 | strigazi: hello | 09:47 |
| brtknr | flwang1: hope the kids get better! | 09:47 |
| flwang1 | brtknr: thanks | 09:48 |
| flwang1 | my daughter got fever since yesterday | 09:48 |
| brtknr | strigazi: is multi project supported in current ng implementation? | 09:48 |
| flwang1 | anything i can help provide my opinion? | 09:48 |
| flwang1 | oh, you're discussing the ng stuff | 09:49 |
| strigazi | brtknr: no, but it is a small change | 09:49 |
| brtknr | flwang1: i cannot reproduce the issue you commented in ng-6 patch | 09:49 |
| *** ArchiFleKs has quit IRC | 09:49 | |
| brtknr | strigazi: is this for admins only? | 09:49 |
| strigazi | brtknr: no | 09:49 |
| strigazi | brtknr: 100% for uers | 09:50 |
| strigazi | brtknr: 100% for users | 09:50 |
| flwang1 | brtknr: in my testing, after removing the 4 new fields from nodegroup table, the cluster is getting stable | 09:51 |
| strigazi | brtknr: nova doesn't have accounting, for gpus, FPGAs, Ironic cpus are == to vcpus | 09:51 |
| flwang1 | i haven't dig into the root cause | 09:51 |
| strigazi | flwang1: what fields? | 09:51 |
| strigazi | what are you talking about? | 09:51 |
| flwang1 | stack_id, status, status_reason, version | 09:51 |
| strigazi | you dropped things from the db? | 09:51 |
| strigazi | are all migrations done? | 09:51 |
| brtknr | flwang1: did you apply `magnum-db-manage upgrade` after checking out ng-9? | 09:52 |
| flwang1 | i did | 09:52 |
| flwang1 | for sure | 09:52 |
| brtknr | i didn't need to delete anything | 09:52 |
| flwang1 | i mean | 09:52 |
| strigazi | what is the error? | 09:52 |
| strigazi | not the VM restarts/rebuils that is irrelevant | 09:52 |
| flwang1 | i have mentioned the error i saw in the ng6 patch | 09:52 |
| brtknr | i also had to checkout the change in python-magnumclient then `pip install -e .` | 09:53 |
| flwang1 | the problem i got is the vm restart/rebuild | 09:53 |
| strigazi | bad nova | 09:54 |
| strigazi | no resources | 09:54 |
| strigazi | when heat sends a req to nova | 09:54 |
| strigazi | and nova fails | 09:54 |
| strigazi | heat retries | 09:54 |
| strigazi | deletes the old vm and tries again | 09:54 |
| strigazi | same everything but diferent uuid | 09:54 |
| flwang1 | strigazi: so you mean, it's because my env(devstack) is lacking resource? | 09:55 |
| strigazi | this happens when you don't have resources | 09:55 |
| strigazi | yes | 09:55 |
| strigazi | or it missbehaves in some other way | 09:55 |
| flwang1 | strigazi: ok, i will test again tomorrow then | 09:55 |
| strigazi | eg can't create ports | 09:55 |
| strigazi | try the minumum possible | 09:55 |
| flwang1 | ok, i don't really worry about the ng work, overall looks good for me | 09:56 |
| strigazi | ok, if I +2 and bharat verifies you are ok? | 09:56 |
| strigazi | we test at cern in three different dev envs plus bharat's tests | 09:57 |
| flwang1 | strigazi: i'm ok with that | 09:58 |
| strigazi | flwang1: for train? | 09:58 |
| brtknr | I'm mostly happy to get things merged after rebase and addressing all the minor comments, now that we also have solid unit tests... i am sure we will find minor issues with it later but its been hanging around for too long :) | 09:59 |
| flwang1 | strigazi: for train | 09:59 |
| flwang1 | just one silly question | 09:59 |
| flwang1 | what's the 'version' standing for in the ng table? | 09:59 |
| flwang1 | i can't see a description for that | 09:59 |
| strigazi | placeholder for uprades with nodereplacement | 10:00 |
| strigazi | now it will work as it is implemented | 10:00 |
| strigazi | or we can leverage it now | 10:00 |
| flwang1 | so it's a version as kube_tag? | 10:01 |
| strigazi | give me 5', sorru | 10:01 |
| strigazi | give me 5', sorry | 10:01 |
| brtknr | i have to leave in 30 minutes for our team standup | 10:02 |
| flwang1 | brtknr: no problem | 10:03 |
| flwang1 | i will be offline in 15 mins as well | 10:03 |
| flwang1 | i'm addressing the comments from heat team for the ignition patch | 10:03 |
| flwang1 | i'm very happy they're generally OK with that | 10:04 |
| brtknr | btw can we start using etherpad for agenda like other teams, e.g. keystone: https://etherpad.openstack.org/p/keystone-weekly-meeting | 10:04 |
| brtknr | and put a link to this in the channel's idle topic | 10:04 |
| flwang1 | brtknr: we were using wiki, but i'm ok with etherpad | 10:04 |
| brtknr | or a link to wiki... i prefer the etherpad UI... | 10:05 |
| brtknr | https://etherpad.openstack.org/p/magnum-weekly-meeting | 10:05 |
| brtknr | there | 10:05 |
| brtknr | :) | 10:05 |
| flwang1 | cool | 10:07 |
| flwang1 | i just proposed a new patchset for the ignition patch | 10:08 |
| brtknr | flwang1: looks pretty solid | 10:14 |
| flwang1 | after it's done, we still have quite a lot work on magnum side to get the fedora coreos driver ready | 10:16 |
| strigazi | where were we? | 10:16 |
| flwang1 | strigazi: the fedora coreos driver | 10:16 |
| strigazi | flwang1: for ngs | 10:16 |
| flwang1 | for ngs | 10:16 |
| flwang1 | (22:01:04) flwang1: so it's a version as kube_tag? | 10:17 |
| strigazi | flwang1: the ngs in different projects, is there an issue? | 10:17 |
| strigazi | flwang1: oh, this | 10:17 |
| strigazi | flwang1: we can use it now too | 10:17 |
| flwang1 | so the version is the coe version of current node group? | 10:17 |
| flwang1 | the current name 'version' is quite confused for me | 10:18 |
| strigazi | this is an incremental version for the ng | 10:19 |
| strigazi | so that we have some tracking | 10:19 |
| strigazi | when a user upgrades something | 10:19 |
| strigazi | but for now it is a placeholder, to implement it | 10:20 |
| strigazi | makes sense? | 10:20 |
| strigazi | brtknr: before you go, any reason to not have ngs in different projects as an opt-in option? | 10:21 |
| strigazi | brtknr: flwang1: still here? | 10:22 |
| brtknr | strigazi: i dont have major objections to it but perhaps this can be added on later? | 10:22 |
| flwang1 | i'm | 10:22 |
| flwang1 | i'm thinking and checking the gke api | 10:22 |
| brtknr | or is it required imminintly | 10:22 |
| flwang1 | https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.nodePools#NodePool | 10:22 |
| strigazi | brtknr: why? for us it is | 10:22 |
| flwang1 | what do you mean ngs in different project? | 10:23 |
| *** pcaruana has quit IRC | 10:23 | |
| *** rcernin has quit IRC | 10:23 | |
| strigazi | flwang1: because nova doesn't have accounting for GPUs, FPGAs, and ironic cpus are accounted as vcpus | 10:23 |
| brtknr | strigazi: i find it slightly unintuitive | 10:23 |
| strigazi | brtknr: we won't advertise this | 10:24 |
| brtknr | i was under the impression that projects imply complete separation | 10:24 |
| strigazi | this doesn't say much ^^ | 10:24 |
| brtknr | i prefer supporting ng per region under the same project | 10:24 |
| strigazi | we do multicloud applications | 10:24 |
| flwang1 | strigazi: can you explain 'ngs in different projects? | 10:24 |
| brtknr | flwang1: so ng1 lives in project A and ng2 lives in project B, both part of the same cluster | 10:25 |
| flwang1 | does that mean cluster 1 in project A can have a NG which belongs to project B? | 10:25 |
| strigazi | again, this is opt-in | 10:25 |
| flwang1 | brtknr: that doesn't sound good for me | 10:25 |
| brtknr | i was under the impresssion that a cluster belongs to a project | 10:26 |
| strigazi | if magnum doesn't have it, we will investigate something else | 10:26 |
| flwang1 | if we want to have it, it needs to be disabled by default | 10:26 |
| flwang1 | unless the cloud operators enable it | 10:26 |
| brtknr | it then seems like a jump in logic to have children nodegroups spanning different projects | 10:26 |
| strigazi | that makes 100% sense | 10:26 |
| strigazi | if we want to have it, it needs to be disabled by default that | 10:27 |
| strigazi | brtknr: how do you do accounting for ironic nodes mixed with vms? | 10:27 |
| strigazi | everything starts from there | 10:28 |
| strigazi | and nova cells | 10:28 |
| strigazi | in the ideal openstack cloud, I understand, it does not make sens. | 10:28 |
| brtknr | strigazi: okay i'm happy with disabled by default. | 10:29 |
| strigazi | flwang1: ? | 10:30 |
| strigazi | policy or config option? | 10:30 |
| flwang1 | i'm ok, if it's disabled by default | 10:31 |
| flwang1 | config | 10:31 |
| flwang1 | i just worry about the security hell | 10:31 |
| strigazi | flwang1: brtknr I'll send you a presentation why we do it | 10:31 |
| flwang1 | strigazi: pls do | 10:31 |
| strigazi | flwang1: don't, because for a cloud with proper network it won't work anywya | 10:32 |
| strigazi | well, it can work | 10:32 |
| strigazi | but you need an extra router | 10:32 |
| strigazi | vrouter | 10:32 |
| flwang1 | from a public cloud pov, it doesn't make sense | 10:33 |
| strigazi | flwang1: this might be also usefull for running the master nodes in the operators tenant :) | 10:33 |
| strigazi | well, see my comment above :) | 10:33 |
| flwang1 | strigazi: i can see the extra benefit ;) | 10:33 |
| flwang1 | i have to go, sorry | 10:33 |
| strigazi | sorry, I completely forgot about that | 10:33 |
| flwang1 | it's late here | 10:33 |
| strigazi | ok | 10:33 |
| strigazi | see you | 10:33 |
| strigazi | #endmeeting | 10:34 |
| *** openstack changes topic to "OpenStack Containers Team" | 10:34 | |
| openstack | Meeting ended Wed Sep 25 10:34:02 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 10:34 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/containers/2019/containers.2019-09-25-09.08.html | 10:34 |
| flwang1 | strigazi: last question | 10:34 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/containers/2019/containers.2019-09-25-09.08.txt | 10:34 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/containers/2019/containers.2019-09-25-09.08.log.html | 10:34 |
| strigazi | tell me | 10:34 |
| flwang1 | as for the fedora coreos driver, are we going to use docker to install the k8s components? | 10:34 |
| flwang1 | and keep anything else same like the fedora atomic driver? | 10:34 |
| strigazi | flwang1: we will discuss it with the fedora coreos devs to see what is better | 10:34 |
| flwang1 | ok | 10:35 |
| strigazi | we will try to run whatever is possible in containers | 10:35 |
| strigazi | all the CNI and plugins we have are not affected by this | 10:35 |
| strigazi | 90% of our work is reusable | 10:35 |
| flwang1 | yep, only the k8s components so far | 10:35 |
| flwang1 | pls keep me in the loop | 10:36 |
| flwang1 | i want to start the work asap | 10:36 |
| *** pcaruana has joined #openstack-containers | 10:36 | |
| strigazi | sure, the meeting has logs too, I'll send you the relavant links | 10:36 |
| flwang1 | ok, have to go really | 10:36 |
| strigazi | good night | 10:36 |
| flwang1 | have a good day | 10:36 |
| strigazi | brtknr: I'll be back later | 10:36 |
| brtknr | see you :) i need to be another meeting now | 10:37 |
| flwang1 | strigazi: brtknr: pls help review https://review.opendev.org/675511 thank you very much | 10:43 |
| *** ttsiouts has quit IRC | 11:01 | |
| *** udesale has quit IRC | 11:15 | |
| *** ttsiouts has joined #openstack-containers | 11:22 | |
| *** goldyfruit has joined #openstack-containers | 11:32 | |
| *** yoctozepto has quit IRC | 12:10 | |
| *** yoctozepto has joined #openstack-containers | 12:15 | |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-6: Add new fields to nodegroup objects https://review.opendev.org/667088 | 12:16 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-7: Adapt parameter and output mappings https://review.opendev.org/667089 | 12:16 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-8: APIs for nodegroup CRUD operations https://review.opendev.org/647792 | 12:16 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-9: Driver for nodegroup operations https://review.opendev.org/667090 | 12:16 |
| *** goldyfruit has quit IRC | 12:16 | |
| *** dave-mccowan has joined #openstack-containers | 12:22 | |
| *** dave-mccowan has quit IRC | 12:26 | |
| *** yoctozepto has quit IRC | 12:26 | |
| *** yoctozepto has joined #openstack-containers | 12:26 | |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-6: Add new fields to nodegroup objects https://review.opendev.org/667088 | 12:28 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-7: Adapt parameter and output mappings https://review.opendev.org/667089 | 12:28 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-8: APIs for nodegroup CRUD operations https://review.opendev.org/647792 | 12:28 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-9: Driver for nodegroup operations https://review.opendev.org/667090 | 12:28 |
| *** dave-mccowan has joined #openstack-containers | 12:28 | |
| *** dave-mccowan has quit IRC | 12:42 | |
| ttsiouts | brtknr: are you around? | 12:43 |
| brtknr | ttsiouts: yep hi | 12:46 |
| ttsiouts | brtknr: thanks again for reviewing | 12:46 |
| ttsiouts | brtknr: I ended up refactoring some things in k8s template definitions | 12:47 |
| ttsiouts | brtknr: the thing is that for output mappings is a bit more complex | 12:47 |
| brtknr | ttsiouts: no worries! is it still working? | 12:47 |
| ttsiouts | brtknr: looks like it's working fine | 12:48 |
| ttsiouts | brtknr: whenever you have the time, check what I did and if you want something else I'm happy to address it | 12:48 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-9: Driver for nodegroup operations https://review.opendev.org/667090 | 12:50 |
| ttsiouts | brtknr: ^^ adapted the comments in the last patch | 12:50 |
| brtknr | I will retest cluster creation, nodegroup creation/scaling/deletion and report back | 12:51 |
| ttsiouts | brtknr: thanks again! | 12:51 |
| brtknr | ttsiouts: did you respond to flwang1's comment about min_nodes/max_nodes not being labels? | 12:58 |
| ttsiouts | brtknr: doing that now | 12:58 |
| brtknr | ttsiouts: do you think it should be a label? | 13:00 |
| strigazi | no, why labels? | 13:02 |
| ttsiouts | brtknr: not really. | 13:03 |
| ttsiouts | brtknr: it would be much easier to set/update them as NG attributes | 13:03 |
| strigazi | they will be consumed by the autoscaler | 13:03 |
| brtknr | so we are going to leave it as it? fine by me | 13:04 |
| brtknr | btw is there a scope for creating multiple nodegroups at cluster creation time as a continuation of this work? | 13:09 |
| brtknr | ttsiouts: strigazi ^ | 13:10 |
| brtknr | i suppose openstack coe cluster create k8s; openstack coe nodegroup create k8s ng1; can both be fired off simulatenously? | 13:10 |
| brtknr | strigazi: how do i simulate autoscaling? | 13:18 |
| brtknr | i.e. generating fake workload | 13:18 |
| brtknr | ttsiouts: ng-7 +343, -337... nice :) | 13:19 |
| *** udesale has joined #openstack-containers | 13:22 | |
| brtknr | ubuntu@devstack-master:/opt/stack/magnum$ openstack coe cluster resize k8s-flannel --nodegroup bharat 1 | 13:33 |
| brtknr | Resizing %(nodegroup)s outside the allowed range: min_node_count = %(min_node_count)s, max_node_count = %(max_node_count)s (HTTP 400) (Request-ID: req-b7b798c8-d9ac-43c7-bbbd-d1d59f3efce5) | 13:33 |
| brtknr | ttsiouts: ^ | 13:33 |
| brtknr | doesnt look right | 13:34 |
| brtknr | if nodegroup.min_node_count > cluster_resize_req.node_count: | 13:36 |
| brtknr | raise exception.NGResizeOutBounds( | 13:36 |
| brtknr | nodegroup=nodegroup.name, min_nc=nodegroup.min_node_count, | 13:36 |
| brtknr | max_nc=nodegroup.max_node_count) | 13:36 |
| brtknr | if (nodegroup.max_node_count and | 13:36 |
| brtknr | nodegroup.max_node_count < cluster_resize_req.node_count): | 13:36 |
| brtknr | raise exception.NGResizeOutBounds( | 13:36 |
| brtknr | nodegroup=nodegroup.name, min_nc=nodegroup.min_node_count, | 13:36 |
| brtknr | max_nc=nodegroup.max_node_count) | 13:36 |
| ttsiouts | brtknr: checking | 13:37 |
| brtknr | looks like the change was made in ng-2 and went unnoticed | 13:37 |
| *** ykarel is now known as ykarel|afk | 13:51 | |
| ttsiouts | brtknr: yeap you are right.. | 13:53 |
| brtknr | ttsiouts: 1 more thing, if i do `openstack coe cluster resize k8s-flannel --nodegroup default-worker 2`, both default-master and default-worker enter UPDATE_IN_PROGRESS state | 13:53 |
| ttsiouts | brtknr: yes. default NGs are in the same stack | 13:54 |
| brtknr | but if i do this to individual nodegroups, only the nodegroup is affected | 13:54 |
| brtknr | ah i understand now :) | 13:54 |
| brtknr | sorry my bad | 13:54 |
| ttsiouts | brtknr: no worries | 13:54 |
| brtknr | ttsiouts: i cant find any more faults with it tbh | 13:55 |
| brtknr | the only thing i'd like to test is autoscaling | 13:55 |
| brtknr | but i am not totally sure how | 13:55 |
| brtknr | do you have any pointers? | 13:55 |
| ttsiouts | brtknr: I could add the fix for the exception in ng-8 | 13:55 |
| ttsiouts | brtknr: it's a one liner fix | 13:56 |
| brtknr | ttsiouts: sounds good to me | 13:56 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-8: APIs for nodegroup CRUD operations https://review.opendev.org/647792 | 13:56 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-9: Driver for nodegroup operations https://review.opendev.org/667090 | 13:56 |
| ttsiouts | brtknr: ^^ | 13:57 |
| brtknr | thanks | 13:57 |
| ttsiouts | brtknr: the autoscaler is missing functionality at the moment | 13:57 |
| *** spiette has quit IRC | 13:58 | |
| *** ykarel|afk has quit IRC | 13:58 | |
| dioguerra | brtknr: long time ago, when i tested autoscaling only worked for default NG | 13:58 |
| ttsiouts | dioguerra: brtknr: yeah this what I think too | 13:59 |
| *** spiette has joined #openstack-containers | 14:00 | |
| brtknr | okay we can leave that for NG v2 | 14:00 |
| brtknr | dioguerra: can you tell me how to simulate workload to activate autoscaling? | 14:01 |
| brtknr | ttsiouts: any thoughts on this: https://review.opendev.org/#/c/667089/8/magnum/drivers/heat/swarm_fedora_template_def.py@82 | 14:09 |
| brtknr | more line 81 than 82 actually | 14:10 |
| *** goldyfruit has joined #openstack-containers | 14:16 | |
| *** ykarel has joined #openstack-containers | 14:32 | |
| *** goldyfruit has quit IRC | 14:46 | |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-8: APIs for nodegroup CRUD operations https://review.opendev.org/647792 | 14:58 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: ng-9: Driver for nodegroup operations https://review.opendev.org/667090 | 14:58 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/python-magnumclient master: Add nodegroup CRUD commands https://review.opendev.org/647793 | 14:58 |
| brtknr | ttsiouts: why did you remvoe image? | 15:01 |
| *** ttsiouts has quit IRC | 15:13 | |
| *** ttsiouts has joined #openstack-containers | 15:15 | |
| *** ttsiouts has quit IRC | 15:15 | |
| strigazi | we discussed it and image should be taken from the cluster-template | 15:17 |
| strigazi | brtknr: we will try to improve it further for upgrades | 15:17 |
| strigazi | brtknr: thoughts? doesn't it belong better to CT? | 15:18 |
| brtknr | strigazi: i was imagining NGs would be the perfect way to deploy custom images per NG | 15:18 |
| *** goldyfruit has joined #openstack-containers | 15:18 | |
| brtknr | e.g. a GPU NG will have an image with drivers baked in for example | 15:19 |
| *** ttsiouts has joined #openstack-containers | 15:20 | |
| brtknr | this seemingly takes away that advantage | 15:20 |
| strigazi | brtknr: i'll be back online in a bit | 15:20 |
| brtknr | unless there is another way to install gpu drivers at run time | 15:20 |
| strigazi | brtknr: for the meeting | 15:21 |
| strigazi | will you be here? | 15:21 |
| brtknr | the fcos one? | 15:21 |
| *** openstackgerrit has quit IRC | 15:21 | |
| strigazi | yes | 15:21 |
| brtknr | yeah | 15:21 |
| brtknr | for 30 mins | 15:21 |
| strigazi | ok, see you later then | 15:21 |
| brtknr | okay | 15:21 |
| *** ttsiouts has quit IRC | 15:24 | |
| dioguerra | brtknr: just schedule pods so that the scheduller gets stuck on pending | 15:28 |
| dioguerra | pods* | 15:28 |
| *** pcaruana has quit IRC | 15:30 | |
| *** ivve has quit IRC | 15:45 | |
| brtknr | dioguerra: thanks | 15:55 |
| strigazi | brtknr: for installing driver it is possible from a container | 15:56 |
| strigazi | brtknr: for installing GPU drivers it is possible from a container | 15:56 |
| strigazi | brtknr: for the image, what do you think? | 15:57 |
| *** itlinux has joined #openstack-containers | 15:57 | |
| strigazi | brtknr: we can leave it and pass it on cluster upgrade too? | 15:57 |
| strigazi | brtknr: we can improve that after | 15:57 |
| brtknr | strigazi: hmm okay if thats the case then i'm happy with it... ideally we dont want to build custom images either | 15:58 |
| brtknr | custom OS images* | 15:58 |
| strigazi | we don't want to build qcows | 15:58 |
| strigazi | but a need might exist | 15:59 |
| strigazi | I don't know | 15:59 |
| brtknr | the only problem i foresee is if we are using non fedora atomic image | 15:59 |
| brtknr | which dont do everything in containers | 15:59 |
| strigazi | maybe for them is useful | 16:00 |
| strigazi | brtknr: let's keep it? | 16:01 |
| brtknr | keep --image? | 16:01 |
| strigazi | we can have mixed cluster like this, eg with atomic and coreos when we upgrade | 16:02 |
| strigazi | yes | 16:02 |
| brtknr | sounds good | 16:02 |
| strigazi | and then gradually drop the atomic nodes | 16:02 |
| strigazi | user will love this IMO | 16:02 |
| strigazi | users will love this IMO | 16:02 |
| strigazi | we tried this before: | 16:02 |
| brtknr | is it possible to replace image in default-master and worker? | 16:02 |
| strigazi | create a cluster in stein, upgrade magnum to train, add a nodegroup (make sure you don't mix very old k8s versions though) | 16:03 |
| strigazi | brtknr: replace no, but we can add more nodegroups and drop old ones | 16:04 |
| strigazi | brtknr: this won't work for master nodes now, but we can do it | 16:04 |
| strigazi | brtknr: for workers it works with the current patches | 16:04 |
| *** ykarel is now known as ykarel\afk | 16:04 | |
| brtknr | i thought image was immutable for default-worker | 16:05 |
| strigazi | the image is immutable for all nodes at the moment anyway, no? | 16:06 |
| brtknr | are you saying that even the default-worker instances could be rebuilt? | 16:06 |
| strigazi | or replaced with another NG | 16:06 |
| strigazi | which would be better | 16:06 |
| strigazi | so, for the --image +2 ? | 16:07 |
| brtknr | we cant currently delete default-worker NG, i guess that is just a technicilty | 16:07 |
| strigazi | yeap ^^ | 16:07 |
| brtknr | strigazi: yep i already did | 16:08 |
| strigazi | he can rebase and your +2 automatically will come back | 16:08 |
| brtknr | cool | 16:08 |
| strigazi | brtknr: break and we meet again in fcos? | 16:08 |
| brtknr | sure, my head hurts from all the meetings today | 16:09 |
| strigazi | me too, when we had the IRC meeting three people were inmy office discussing ingresses | 16:10 |
| strigazi | s/me/mine | 16:10 |
| *** goldyfruit_ has joined #openstack-containers | 16:12 | |
| *** goldyfruit has quit IRC | 16:15 | |
| *** ykarel\afk is now known as ykarel | 16:23 | |
| *** jmlowe has quit IRC | 16:24 | |
| *** ykarel is now known as ykarel|away | 16:37 | |
| brtknr | strigazi: are you coming to #fedora-meeting-1 | 16:37 |
| *** ramishra has quit IRC | 16:43 | |
| *** jmlowe has joined #openstack-containers | 16:50 | |
| *** ivve has joined #openstack-containers | 16:53 | |
| *** henriqueof has joined #openstack-containers | 16:54 | |
| *** henriqueof1 has quit IRC | 16:55 | |
| goldyfruit_ | I'm facing an issue related to Neutron/Magnum | 17:04 |
| goldyfruit_ | When Neutron attach a foating IP to the port, it goes to the SNAT and then it goes to the qrouter where the instance is running | 17:05 |
| goldyfruit_ | (we are using DVR) | 17:05 |
| goldyfruit_ | During the time the FIP is bind in the SNAT, it sends a request which makes the MAC appears on the switch and then when the FIP is detached from the SNAT to the qrouter an another request is sent to the router with a different MAC | 17:06 |
| goldyfruit_ | Which means the router has 2 MACs for the same FIP which prevent the instance the get Internet access and download the images, etc... | 17:06 |
| goldyfruit_ | There is an issue in neutron for sure about not cleaning his shit in time but I guess we could prevent this by doing the right orchestration in Heat for Master/Nodes | 17:08 |
| goldyfruit_ | For example, we could first create the Neutron port, then create the instance and then attach the FIP to the instance/port | 17:08 |
| goldyfruit_ | From here https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_atomic_v1/templates/kubeminion.yaml#L440-L491 is seems we are creating the instance then the port and then attach the FIP to the port | 17:09 |
| goldyfruit_ | If a port is created without been attached and if a FIP is assigned to it then the FIP goes to the SNAT | 17:10 |
| goldyfruit_ | If a port is created and attached to an instance and then a FIP is assigned to the port then the FIP goes directly to the qrouter | 17:11 |
| goldyfruit_ | Which avoid the duplicate MAC address because the FIP has been plumbed only once | 17:11 |
| *** mrodriguez has joined #openstack-containers | 17:15 | |
| goldyfruit_ | mrodriguez, o/ | 17:16 |
| mrodriguez | goldyfruit_: o/ | 17:17 |
| *** ykarel|away has quit IRC | 17:19 | |
| *** udesale has quit IRC | 17:19 | |
| *** jmlowe has quit IRC | 17:35 | |
| *** jmlowe has joined #openstack-containers | 17:48 | |
| *** henriqueof1 has joined #openstack-containers | 17:53 | |
| *** henriqueof has quit IRC | 17:53 | |
| goldyfruit_ | We opened a bug related to Neutron: https://bugs.launchpad.net/neutron/+bug/1845360 | 17:58 |
| openstack | Launchpad bug 1845360 in neutron "ARP advertisement issue with DVR" [Undecided,New] | 17:58 |
| *** ricolin has quit IRC | 18:55 | |
| *** openstackgerrit has joined #openstack-containers | 19:13 | |
| openstackgerrit | Matthew Fuller proposed openstack/magnum master: PDF documentation build https://review.opendev.org/684436 | 19:13 |
| goldyfruit_ | So basically Neutron guys said: "Can the workflow be changed to first attach the port then assign the floating IP?" | 19:33 |
| *** flwang1 has quit IRC | 20:53 | |
| openstackgerrit | Matthew Fuller proposed openstack/magnum master: PDF documentation build https://review.opendev.org/684436 | 21:07 |
| openstackgerrit | Matthew Fuller proposed openstack/magnum master: PDF documentation build https://review.opendev.org/684436 | 21:09 |
| *** flwang has joined #openstack-containers | 21:38 | |
| flwang | brtknr: around? | 21:38 |
| *** mrodriguez has quit IRC | 22:06 | |
| flwang | brtknr: as for https://review.opendev.org/678458 you need to checkout the branch and install it to 'register' the fedora coreos driver | 22:15 |
| *** rcernin has joined #openstack-containers | 22:15 | |
| brtknr | flwang: hi, again almost bedtime here | 22:34 |
| brtknr | So it’s not enough to restart magnum services? | 22:34 |
| brtknr | What is the command for installation? Is pip install -e . enough? | 22:35 |
| flwang | brtknr: no, it's not enough | 22:35 |
| flwang | just pip install -e . | 22:35 |
| flwang | you're good to go | 22:35 |
| brtknr | Cool okay I’ll try that tomorrow :) I tested your status changed on top the latest master but seemed to return None as status rather that unknow | 22:36 |
| brtknr | s/changed/rebased | 22:37 |
| brtknr | Could you verify it works for you too? | 22:37 |
| brtknr | Also got the same outcome without rebase | 22:38 |
| brtknr | flwang^ | 22:39 |
| flwang | brtknr: sure, will do, thanks for the review | 23:09 |
| *** goldyfruit_ has quit IRC | 23:12 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Improve log of k8s health status check https://review.opendev.org/675511 | 23:20 |
| *** dave-mccowan has joined #openstack-containers | 23:28 | |
| *** ivve has quit IRC | 23:28 | |
| *** dave-mccowan has quit IRC | 23:33 | |
| brtknr | flwang: are you happy to +2 these backports https://review.opendev.org/#/q/status:open+project:openstack/magnum+branch:stable/stein+topic:stein-8.1.0 | 23:33 |
| brtknr | I’d like to push for cutting a new stein release tomorrow | 23:33 |
| brtknr | Would help to make stein more stable with fa29 | 23:34 |
| flwang | https://review.opendev.org/#/c/648935/ doesn't look good for backport, why do you need it? | 23:35 |
| flwang | brtknr: ^ | 23:36 |
| brtknr | Without that patch, Traefik is broken because it automatically downloads version 2.0.0+ containers since we didn’t fix it | 23:36 |
| brtknr | Make sense? | 23:37 |
| brtknr | The api has changed in the new version | 23:37 |
| brtknr | flwang^ | 23:37 |
| brtknr | V2 was released 9 days ago | 23:40 |
| flwang | ok, fair enough | 23:40 |
| brtknr | flwang: thank you, going to sleep now! Have a good day:) | 23:43 |
| flwang | brtknr: thank you, have a good night | 23:46 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!