Tuesday, 2019-06-25

*** hongbin has quit IRC00:19
*** rcernin has quit IRC01:48
*** rcernin has joined #openstack-containers01:48
*** xiangdong-huang has joined #openstack-containers01:57
*** hongbin has joined #openstack-containers02:04
*** rcernin has quit IRC02:42
*** rcernin has joined #openstack-containers02:42
*** rcernin has quit IRC03:04
*** rcernin has joined #openstack-containers03:04
*** sapd1_x has joined #openstack-containers03:07
*** mnaser has quit IRC03:19
*** mnaser has joined #openstack-containers03:20
*** ykarel|afk has joined #openstack-containers03:30
*** rcernin has quit IRC03:30
*** rcernin has joined #openstack-containers03:31
*** udesale has joined #openstack-containers03:53
*** ykarel|afk is now known as ykarel04:23
*** hongbin has quit IRC04:25
*** ykarel has quit IRC04:32
*** hongbin has joined #openstack-containers04:42
*** ykarel has joined #openstack-containers04:48
*** hongbin has quit IRC04:49
*** ivve has joined #openstack-containers05:04
lxkongNobodyCam: it has nothing to do with Magnum, it's up to the storage driver of the cloud provider (in-tree or CSI)05:32
*** sapd1_x has quit IRC05:34
*** ysandeep has joined #openstack-containers05:43
*** pcaruana has joined #openstack-containers05:56
*** pcaruana has quit IRC05:57
*** pcaruana has joined #openstack-containers05:57
*** lpetrut has joined #openstack-containers06:01
*** rcernin has quit IRC06:41
*** danil has joined #openstack-containers06:42
*** altlogbot_2 has quit IRC06:46
*** altlogbot_3 has joined #openstack-containers06:49
*** altlogbot_3 has quit IRC06:50
*** altlogbot_0 has joined #openstack-containers06:55
*** stefanocuck has joined #openstack-containers07:06
*** ttsiouts has joined #openstack-containers07:28
*** ttsiouts has quit IRC07:33
*** ykarel is now known as ykarel|lunch07:41
*** ttsiouts has joined #openstack-containers08:00
*** udesale has quit IRC08:01
openstackgerritJonathan Rosser proposed openstack/magnum stable/stein: Revert "support http/https proxy for discovery url"  https://review.opendev.org/66728008:01
*** udesale has joined #openstack-containers08:03
*** udesale has quit IRC08:03
openstackgerritJonathan Rosser proposed openstack/magnum stable/rocky: Revert "support http/https proxy for discovery url"  https://review.opendev.org/66728108:03
*** udesale has joined #openstack-containers08:03
*** ricolin has joined #openstack-containers08:05
openstackgerritJonathan Rosser proposed openstack/magnum stable/pike: Revert "support http/https proxy for discovery url"  https://review.opendev.org/66728308:05
openstackgerritJonathan Rosser proposed openstack/magnum stable/queens: Revert "support http/https proxy for discovery url"  https://review.opendev.org/66728408:07
*** ricolin has quit IRC08:13
*** ricolin has joined #openstack-containers08:22
*** ykarel|lunch is now known as ykarel08:40
*** yolanda has joined #openstack-containers09:07
openstackgerritBharat Kunwar proposed openstack/magnum master: Convert fixed_subnet name to uuid so that OCCM does not error  https://review.opendev.org/62951409:09
brtknrttsiouts: hmm i dont seem to be able to list nodegroups for a cluster09:20
brtknrI get a 40409:20
ttsioutsbrtknr: did you run the migrations first? (magnum-db-manage upgrade)09:24
ttsioutsbrtknr: also could you please share api logs?09:31
brtknrttsiouts: after a db upgrade, I'm hitting this: Remote error: DBError (pymysql (HTTP 500) (Request-ID: req-bee66ea1-228a-4959-9dfe-954c5e2f548b)09:36
brtknrubuntu@devstack-master:/opt/stack/devstack$ openstack coe cluster list09:36
brtknrThis is my journalctl logs live: https://seashells.io/v/A4GWFTwt09:37
ttsioutsbrtknr: I guess the services (both api and conductor) were restarted right?09:37
brtknrthis may be easier: https://seashells.io/p/A4GWFTwt09:37
brtknrttsiouts: ah, restart was the key09:40
brtknrthanks09:40
ttsioutsbrtknr: no prob! Thanks for testing!!09:41
ttsioutsbrtknr: ping me if you need anything09:41
brtknrttsiouts: thanks for the quick response :)09:41
brtknrttsiouts: hmm09:43
brtknrttsiouts: why is the initial state when creating a new nodegroup "UPDATE_FAILED"09:43
ttsioutsbrtknr: can you list the stacks?09:45
ttsioutsthe new nodegroup is a new stack09:45
brtknralso this is the output of my nodegroup list: https://seashells.io/v/FcfkvzCE09:45
ttsioutsdid the creation fail?09:45
brtknri only have 3 workers and 1 master but it tells me I have 2 default worker nodegroups and 2 master09:46
brtknryeah it is a new stack09:47
brtknris it supposed to be like that?09:47
brtknrrather than updating existing stack?09:47
ttsioutsbrtknr: yes. initially it was supposed to be an update in the existing stack09:49
ttsioutsbrtknr: but the complexity we introduced in heat side was huge09:49
brtknrttsiouts: okay, I almost prefer it this way...09:50
brtknrHeat is quite fragile..09:50
brtknr:)09:50
ttsioutsbrtknr: yes09:50
ttsioutsbrtknr: this way we can support old clusters too09:50
brtknrNice, my new nodegroup has joined the existing cluster09:51
brtknrso thats all working09:51
ttsioutsbrtknr: now the cluster status is an aggregation of the stacks underneath09:51
brtknrdid you see this: https://seashells.io/v/FcfkvzCE09:51
brtknrwhy I am being told i have double the number of default nodegroups?09:51
ttsioutsbrtknr: yeah this is weird09:51
ttsioutscould it be leftovers in the DB?09:52
brtknrI wwill see if it goes away when I create a new cluster09:52
openstackgerritDiogo Guerra proposed openstack/magnum master: [k8s] Update prometheus monitoring helm based configuration  https://review.opendev.org/65740309:52
ttsioutsalso please don't test adding master NGs yet09:52
ttsioutsIt won't work since we are missing a step09:52
ttsioutsbrtknr: I'm working on this09:53
brtknrttsiouts: okay thanks, i wont :)09:54
ttsioutsbrtknr: we need to persist the service keys and etcd keys so that we can reuse them when adding a master NG09:54
brtknrubuntu@devstack-master:/opt/stack/devstack$ openstack coe cluster list09:55
brtknr+--------------------------------------+------+---------+------------+--------------+-----------------+---------------+09:55
brtknr| uuid                                 | name | keypair | node_count | master_count | status          | health_status |09:55
brtknr+--------------------------------------+------+---------+------------+--------------+-----------------+---------------+09:55
brtknr| 7070046d-3c0f-448a-af64-689081b6c67b | k8s  | default |          6 |            2 | CREATE_COMPLETE | HEALTHY       |09:55
brtknr+--------------------------------------+------+---------+------------+--------------+-----------------+---------------+09:55
brtknralso look, node_could=6 master_count=2 after migration when it was only half that09:55
ttsioutsbrtknr: I guess the old api (before restarting) created the extra NGs..09:56
brtknrnot sure if this is because of migration logic?09:56
brtknrno there was only 3 nodes 1 master in reality09:56
ttsioutshmmm09:56
ttsioutsbrtknr: I have to test the migration again10:00
ttsioutsbrtknr: indeed there is something fishy10:01
brtknrhmm i wish i had backed up my database10:01
brtknris there a way to downgrade after migration?10:01
ttsioutsbrtknr: I am doing it manually10:02
ttsioutsif you don't care for what you have in the db i would do this:10:03
ttsiouts- stop both api and conductor10:03
ttsiouts- drop magnum db10:04
ttsiouts- checkout master10:04
ttsiouts- run the migrations (from master)10:04
ttsiouts- start the services10:04
ttsiouts- create a cluster10:04
ttsiouts- checkout magnum_nodegroups10:05
ttsiouts- stop services10:05
ttsiouts- run migrations10:05
ttsiouts- start the services10:05
ttsioutsI know it's not great...10:05
openstackgerritDiogo Guerra proposed openstack/magnum master: [k8s] Update prometheus monitoring helm based configuration  https://review.opendev.org/65740310:08
*** xiangdong-huang has quit IRC10:08
*** ttsiouts has quit IRC10:21
*** ttsiouts has joined #openstack-containers10:21
*** ttsiouts has quit IRC10:26
*** ykarel is now known as ykarel|meeting10:35
*** stefanocuck has quit IRC10:37
brtknrttsiouts: what do you mean by run migrations? "magnum-db-manage upgrade"?10:48
*** ykarel_ has joined #openstack-containers10:59
*** ykarel_ has quit IRC11:00
*** ykarel has joined #openstack-containers11:01
*** ykarel|meeting has quit IRC11:01
*** udesale has quit IRC11:06
*** udesale has joined #openstack-containers11:08
*** ykarel_ has joined #openstack-containers11:09
*** ykarel has quit IRC11:12
*** ttsiouts has joined #openstack-containers11:16
ttsioutsbrtknr: I was on a lunch break11:16
ttsioutsbrtknr: please create an empty DB called "magnum"11:16
ttsioutsbrtknr: the tool assumes that the DB is already there...11:17
ttsioutsbrtknr: sorry I forgot that step11:17
*** 7JTAANFPK has joined #openstack-containers11:39
7JTAANFPKHello, could anybody have a look on these: https://storyboard.openstack.org/#!/story/200594911:42
7JTAANFPKand: https://storyboard.openstack.org/#!/story/200582911:43
*** 7JTAANFPK has quit IRC11:51
*** slivik_ has joined #openstack-containers11:52
*** goldyfruit has joined #openstack-containers12:00
*** goldyfruit has quit IRC12:05
ttsioutsbrtknr: did it work?12:10
brtknrttsiouts: i managed to create a cluster using master branch12:14
brtknrnow trying to migrate the db again but getting this: alembic.util.exc.CommandError: Can't locate revision identified by 'd247ff520513'12:14
ttsioutsbrtknr: you are trying to run the migrations from magnum_nodegroups right?12:18
ttsioutsbrtknr: I cannot find the revision 'd247ff520513'12:27
ttsioutsnormally from master to magnum_nodegroups only one migration runs 461d798132c7 -> c04e925e65c212:28
brtknrttsiouts: hmm maybe it as an artifact of a previous attempt12:29
brtknri redid the steps and its not happening this time12:29
*** goldyfruit has joined #openstack-containers12:33
ttsioutsbrtknr: cool12:34
ttsioutsbrtknr: did the migration work for your cluster?12:34
brtknrttsiouts: just waiting for the new cluster to finish creating on master12:34
ttsioutsbrtknr: great12:35
*** ykarel_ is now known as ykarel|away12:48
*** ttsiouts has quit IRC12:55
*** ttsiouts has joined #openstack-containers12:56
*** ykarel|away has quit IRC13:01
*** ttsiouts has quit IRC13:01
*** ysandeep has quit IRC13:04
*** ttsiouts has joined #openstack-containers13:06
*** ttsiouts has left #openstack-containers13:06
*** ttsiouts has joined #openstack-containers13:06
brtknrttsiouts: strangely, the migration was fine this time13:11
goldyfruitHi guys13:12
ttsioutsbrtknr: can you share the output of nodegroup list?13:12
goldyfruitAny of you could review these one: https://review.opendev.org/#/c/665697/ ?13:12
brtknrhttps://seashells.io/v/zCA9jge913:13
brtknrttsiouts: ^13:13
ttsioutsbrtknr: could you also show the NGs?13:15
ttsioutsbrtknr: just to be sure..13:16
brtknrgoldyfruit: I pursued this path 1 year ago, ended up being abandoned... i dont know why.. https://review.opendev.org/#/c/561605/13:17
goldyfruitbrtknr arf :/13:18
brtknrgoldyfruit: look at the comments here: https://bugs.launchpad.net/magnum/+bug/176340513:19
openstackLaunchpad bug 1763405 in Magnum "Kubernetes node events warns about FailedNodeAllocatableEnforcement" [Undecided,In progress] - Assigned to Bharat Kunwar (brtknr)13:19
brtknrsolution is to use cgroupfs instead of systemd driver13:19
goldyfruitbrtknr, I did use cgroupfs but I had to add these two options to make the limits worked13:20
goldyfruitcgroupfs is the default value in the Heat templates13:20
brtknrttsiouts: https://seashells.io/v/gtDvdkCf13:22
goldyfruitAnd this doesn't seem to work: https://github.com/openstack/magnum/blob/d3c56b6b9071859bd99e8fa17f65518ac66704fb/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh#L164-L16713:22
ttsioutsbrtknr: looks good! thanks!!13:23
goldyfruithttps://paste.api-zulu.com/icegafevav.pl13:23
ttsioutsbrtknr: I'll also check the migration as soon as I get some time13:24
goldyfruitbrtknr, The /etc/systemd/system/docker.service.d/ directory is empty13:24
brtknrttsiouts: thanks but I think the first time was probably my own fault...13:27
brtknrI have been using this dev machine for a while... looks like some of the changes from previous13:27
brtknrbeacuse we now use default-{master,worker} convention13:27
ttsioutsbrtknr: hmmm...13:30
ttsioutsbrtknr: https://github.com/openstack/magnum/blob/master/magnum/db/sqlalchemy/alembic/versions/461d798132c7_change_cluster_to_support_nodegroups.py#L9313:30
ttsioutsbrtknr: I think I missed this...13:31
*** dave-mccowan has joined #openstack-containers13:32
brtknrttsiouts: how do you mean? i can see its there:         'name': '%s-master' % rs[1],13:33
ttsioutsbrtknr: this has to be fixed...13:33
brtknrgoldyfruit: check /etc/systemd/system/docker.service13:34
brtknrttsiouts: whats wrong with #L93?13:34
ttsioutsbrtknr: should be 'name': 'default-master'13:34
brtknroh13:35
brtknrwhy did it work for me second time?13:35
ttsioutsbrtknr: now I know why it was not found in your test13:35
goldyfruitbrtknr, yeah it's there13:35
ttsioutsbrtknr: the initial cluster has to be created using stable/stein13:35
goldyfruit          --exec-opt native.cgroupdriver=cgroupfs \13:35
ttsioutsbrtknr: not on master...13:35
brtknrttsiouts: ohhh okay13:36
brtknrttsiouts: so if i create a cluster using stable/stein and try to migrate, it would have not worked?13:37
brtknrgoldyfruit: so are you satisfied?13:38
ttsioutsbrtknr: I think it will work, but this breaks the naming convention... default-<role>13:38
goldyfruitbrtknr, ahah nop, because even with cgroupfs, I need the two options :)13:38
brtknrttsiouts: makes sense13:39
brtknrttsiouts: well, glad to be of help :P13:39
brtknrI'm very impressed with nodegroups overall! great work!13:40
goldyfruitThe two options are the only way for me to get "/sys/fs/cgroup/cpu|memory|etc.../kubepods/"13:40
brtknrgoldyfruit: what version of k8s are you using?13:40
brtknrand what version of magnum?13:40
brtknrgoldyfruit: as  I cannot reproduce your problem13:41
brtknrI dont get those warnings13:41
*** munimeha1 has joined #openstack-containers13:41
goldyfruitbrtknr, I'm running 1.13.4, Magnum from Stein, Fedora Atomix 2913:42
goldyfruitAtomic*13:43
brtknrgoldyfruit: how did you deploy magnum?13:43
goldyfruitKolla13:43
brtknrkolla-ansible?13:43
goldyfruitAnd kolla-ansible yes13:43
goldyfruitbrtknr, so you brtknr, so you get "/sys/fs/cgroup/memory/kubepods/" without  --cgroups-per-qos=true --enforce-node-allocatable=pods ?13:44
goldyfruit(sorry for the double so...)13:44
brtknrgoldyfruit: okay I understand your problem13:45
brtknrits different to mine...13:45
brtknrI have +1ed your PS13:45
goldyfruitbrtknr, \o/ thanks !13:46
goldyfruitbrtknr, I put the information in the story if you need more details13:46
goldyfruithttps://storyboard.openstack.org/#!/story/200590313:46
brtknrMay be worth adding to your story why its different to the earlier bug report13:47
*** openstackgerrit has quit IRC13:48
goldyfruitDone, I added that it would be an extension of commit https://github.com/openstack/magnum/commit/ec58c233618ad82c1a6694c91491306623ba1f1513:51
goldyfruitThanks brtknr wait and see :)13:51
brtknrgoldyfruit: does qinling work on top of magnum?13:53
goldyfruitbrtknr, yes13:55
goldyfruithttps://docs.openstack.org/kolla-ansible/latest/reference/compute/qinling-guide.html#connect-to-an-existing-kubernetes-cluster13:56
*** spiette has quit IRC14:05
*** ykarel has joined #openstack-containers14:06
*** spiette has joined #openstack-containers14:08
*** openstackgerrit has joined #openstack-containers14:14
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: Hardcode the names of the default NGs  https://review.opendev.org/66737314:14
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: [WIP] ng-6: Add new fields to nodegroup objects  https://review.opendev.org/66708814:15
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: [WIP] ng-7: Adapt parameter and output mappings  https://review.opendev.org/66708914:15
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: [WIP] ng-8: APIs for nodegroup CRUD operations  https://review.opendev.org/64779214:15
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: [WIP] ng-9: Driver for nodegroup operations  https://review.opendev.org/66709014:15
ttsioutsbrtknr: the fix is here: https://review.opendev.org/66737314:15
ttsioutsbrtknr: I rebased all ng-* on of the fix14:16
ttsioutsbrtknr: I quickly tested the migration from stable/stein (with the fix) and it seems to be working as expected14:17
ttsioutsbrtknr: thanks again for your effort!!14:17
brtknrttsiouts: I just reproduced the bug by upgrading from stein :https://seashells.io/v/4hCbxR6Z14:18
brtknrhttps://seashells.io/v/4hCbxR6Z14:18
ttsioutsbrtknr: yeah if you test it again on 667090 it should be fine14:20
ttsioutsbecause now it's rebased on top of the fix14:20
ttsioutsbrtknr: I created a cluster in stein. and after upgrading list shows this: http://paste.openstack.org/show/753365/14:28
brtknrttsiouts: hmm the flavor_id doesnt look right14:28
ttsioutsbrtknr: yeah it comes from the cluster and the cluster template...14:29
ttsioutsbrtknr: if you don't specify it, it will get the default from the templates...14:29
brtknrttsiouts: yess! its working for me too!14:32
ttsioutsbrtknr: cool!14:33
ttsioutsbrtknr: thanks again for testing!14:33
ttsioutsbrtknr: ng-8 contains the update API for NGs14:38
brtknrttsiouts: thank you, all looking great!14:38
*** lpetrut has quit IRC14:38
brtknrttsiouts: i am looking at the code in detail now14:38
ttsioutsbrtknr: I don't know if we want to support it for scaling though...14:38
ttsioutsbrtknr: I think, since we already have the resize api, everything should go from there..14:39
ttsioutsbrtknr: we could the use the update API for NG internals e.g. min/max_node_count etc14:40
brtknrttsiouts: i havent looked at the resize api14:40
brtknris this for autoscaling?14:41
ttsioutsmin/max_node_count you mean?14:41
brtknryeag14:42
ttsioutsyeah, we should use it for limiting the autoscaler14:42
*** ykarel is now known as ykarel|afk14:45
brtknrttsiouts: btw have ng-1->ng-5 all merged?14:47
ttsioutsbrtknr: yes14:48
brtknrttsiouts: cool14:48
*** udesale has quit IRC14:55
*** munimeha1 has quit IRC14:59
*** slivik_ has quit IRC15:12
*** sapd1_x has joined #openstack-containers15:16
*** lpetrut has joined #openstack-containers15:17
*** lpetrut has quit IRC15:23
*** ttsiouts has quit IRC15:39
*** ttsiouts has joined #openstack-containers15:40
*** sapd1_x has quit IRC15:40
*** ttsiouts has quit IRC15:44
*** sapd1_x has joined #openstack-containers15:51
*** sapd1_x has quit IRC16:02
*** udesale has joined #openstack-containers16:03
*** ricolin has quit IRC16:33
*** ramishra has quit IRC16:46
*** kplant has joined #openstack-containers16:51
kplantwas there a change in magnum recently that has octavia as a requirement? i'm getting the following error when trying to create a customer _without_ loadbalancers:16:52
kplantERROR: The Parameter (octavia_ingress_controller_tag) was not defined in template.16:52
kplants/customer/cluster16:53
*** udesale has quit IRC17:07
*** ykarel|afk has quit IRC17:45
*** guimaluf has joined #openstack-containers18:10
*** ianychoi_ has quit IRC18:49
*** ianychoi_ has joined #openstack-containers18:50
*** hongbin has joined #openstack-containers19:02
*** realgweep has joined #openstack-containers19:02
*** lpetrut has joined #openstack-containers19:18
brtknrIs there a meeting happening tonight?19:22
*** lpetrut has quit IRC19:23
*** pcaruana has quit IRC19:38
*** kplant has quit IRC19:46
*** realgweep has quit IRC19:55
*** dioguerra51 has joined #openstack-containers20:14
*** dioguerra51 has quit IRC20:14
*** dioguerra15 has joined #openstack-containers20:15
*** dioguerra15 has quit IRC20:15
*** dioguerra32 has joined #openstack-containers20:18
dioguerra32#JOIN openstack-containers20:18
dioguerra32Hello, meeting?20:18
*** ivve has quit IRC20:24
*** dioguerra32 has quit IRC20:28
*** ttsiouts has joined #openstack-containers20:55
*** dioguerra15 has joined #openstack-containers20:56
dioguerra15#join openstack-containers20:56
*** ianychoi_ has quit IRC20:59
*** ianychoi_ has joined #openstack-containers21:00
ttsioutsbrtknr: dioguerra15 is the meeting on?21:02
*** dioguerra15 has quit IRC21:03
brtknrnot srue21:03
*** ianychoi_ has quit IRC21:05
*** dioguerra35 has joined #openstack-containers21:06
*** ianychoi_ has joined #openstack-containers21:09
brtknrhmm21:12
brtknrdont think there's been a meeting in a few weeks21:12
ttsioutsbrtknr: hmmm21:15
*** dioguerra35 has quit IRC21:15
goldyfruit;?21:16
goldyfruit:/21:16
ttsioutslet's ping flwang to get the meeting back on from next week21:17
brtknryeah, a question on the mailing list probably a good move forwardd21:18
*** openstackgerrit has quit IRC21:18
*** ianychoi_ is now known as ianychoi21:18
ttsioutscool. I have to go now21:19
ttsioutsbrtknr: let's talk tomorrow if you want something regarding NGs21:21
brtknrttsiouts: sounds good, I am composing an email now for the ML21:21
ttsioutsbrtknr: I hope to get some time to refresh these patches this week.21:21
ttsioutsbrtknr: cool21:21
*** ttsiouts has quit IRC21:22
*** ttsiouts has joined #openstack-containers21:22
brtknrttsiouts: hello again!21:22
*** ttsiouts has quit IRC21:27
*** dioguerra44 has joined #openstack-containers21:35
dioguerra44#join openstack-containers21:35
*** dioguerra44 has quit IRC21:43
*** tobberydberg has quit IRC21:49
*** tobberydberg has joined #openstack-containers21:51
*** sapd1_x has joined #openstack-containers22:41
*** rcernin has joined #openstack-containers23:05
*** hongbin has quit IRC23:07
*** flwang1 has joined #openstack-containers23:11
flwang1brtknr: I'm in Shanghai for kubecon23:11
flwang1i have emailed strigazi to help host the meeting when i'm away but seems he is busy23:12
flwang1brtknr: i'm sorry if there is any confusion23:12
*** flwang1 has quit IRC23:23

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!