Wednesday, 2019-10-23

*** rcernin has joined #openstack-containers00:05
*** ykarel|away has joined #openstack-containers02:18
*** goldyfruit has joined #openstack-containers02:25
*** jerryz has quit IRC02:34
*** spsurya has joined #openstack-containers02:52
*** ramishra has joined #openstack-containers03:11
*** ykarel|away has quit IRC03:13
*** ricolin has joined #openstack-containers03:16
*** ykarel|away has joined #openstack-containers03:19
*** ykarel|away has quit IRC03:24
*** ykarel|away has joined #openstack-containers04:27
*** ykarel|away is now known as ykarel04:27
*** ykarel is now known as ykarel|afk06:21
*** rcernin has quit IRC06:37
*** trident has quit IRC06:55
*** arbrandes has quit IRC07:00
*** trident has joined #openstack-containers07:00
*** arbrandes has joined #openstack-containers07:01
*** ttsiouts has joined #openstack-containers07:09
*** ttsiouts has quit IRC07:25
*** ykarel|afk is now known as ykarel07:25
*** ttsiouts has joined #openstack-containers07:26
*** ttsiouts has quit IRC07:30
*** ivve has joined #openstack-containers07:39
openstackgerritpengyuesheng proposed openstack/magnum master: Switch to Ussuri jobs  https://review.opendev.org/69048807:45
*** ttsiouts has joined #openstack-containers07:53
openstackgerritSpyros Trigazis proposed openstack/magnum master: heat-agent: Check if scripts exists  https://review.opendev.org/68970407:58
*** yolanda has joined #openstack-containers07:58
*** mgoddard has joined #openstack-containers08:05
*** flwang1 has joined #openstack-containers08:21
*** ttsiouts has quit IRC08:40
*** ttsiouts has joined #openstack-containers08:41
*** ttsiouts has quit IRC08:41
*** ttsiouts_ has joined #openstack-containers08:41
openstackgerritSpyros Trigazis proposed openstack/magnum master: k8s_fedora: Add use_podman label  https://review.opendev.org/69005308:50
*** ttsiouts_ has quit IRC08:52
*** ttsiouts has joined #openstack-containers08:53
*** ttsiouts has quit IRC08:53
*** ttsiouts has joined #openstack-containers08:53
*** ykarel is now known as ykarel|lunch08:55
flwang1strigazi: brtknr: around?08:59
flwang1meeting?08:59
strigaziI'm here08:59
brtknro/09:00
flwang1#startmeeting magnum09:01
openstackMeeting started Wed Oct 23 09:01:09 2019 UTC and is due to finish in 60 minutes.  The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot.09:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.09:01
*** openstack changes topic to " (Meeting topic: magnum)"09:01
openstackThe meeting name has been set to 'magnum'09:01
flwang1#topic roll call09:01
*** openstack changes topic to "roll call (Meeting topic: magnum)"09:01
flwang1o/09:01
strigazio/09:02
*** pcaruana has joined #openstack-containers09:02
brtknro/09:02
flwang1let's stop typing on etherpad for now09:03
flwang1and focus here?09:04
flwang1#topic updates09:04
*** openstack changes topic to "updates (Meeting topic: magnum)"09:04
flwang1as i mentioned in the ehterpad09:04
flwang1the ignition patch has been merged and cherrypicked to train09:04
flwang1we may need a tiny patch to drop the copy  local-data to cfn-init-data09:05
flwang1strigazi: brtknr: ^09:06
strigaziI don't know if we need it09:06
flwang1strigazi: brtknr: i think we should focus on irc09:06
flwang1strigazi: why?09:06
strigaziflwang1: why we need it09:06
strigazi?09:06
brtknrif its copying, it will behave the same as before?09:07
brtknrdoes the heat patch prioritise local-data?09:07
flwang1strigazi: see https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_coreos_v1/templates/user_data.json#L8009:07
flwang1you added   ExecStartPre=-mv /var/lib/os-collect-config/local-data /var/lib/cloud/data/cfn-init-data09:08
flwang1not copy, it's a mv09:08
brtknrdoh! its mv /var/lib/os-collect-config/local-data /var/lib/cloud/data/cfn-init-data, not cp!09:08
brtknreither we should change it to cp....09:08
flwang1we should drop it09:08
brtknrbecause heat will have already cut a release09:09
brtknrwhen magnum cuts a release, it will break compatibility09:09
flwang1ah, i see your point09:09
flwang1that's shit09:09
brtknryeah :(09:09
flwang1anyway, we can keep it maybe09:10
brtknrthats assuming heat has cut a release09:10
strigaziExecStartPre=- means fail safely09:10
brtknrwe should check with them09:10
flwang1strigazi: that's good09:10
strigaziwhy we need it?09:10
flwang1strigazi: you're confusing me again, why we need what?09:11
strigaziI did it this way because I was expecting this09:11
strigaziWe don't need to drop the mv09:11
flwang1because it's like a patching09:11
flwang1not a solution09:11
strigazi;et09:12
flwang1but as brtknr said, heat have already released a version, so we can keep it09:12
flwang1don't you think it's ugly to have a mv like this?09:13
strigaziwe can drop it, the agent will work unless someone has a broken heat. we can keep it, the agent will work with a fixed or broken heat09:13
flwang1yep, that's why i think now we can live with that09:13
flwang1after I realized that heat have already released09:14
flwang1clear now?09:14
strigaziyes. That's why I said, "why we need it?"09:15
strigazimove on, we keep it.09:15
flwang1ok, fail enough09:15
brtknrlol...09:15
strigaziflwang1: let's move on?09:16
flwang1as for fa29 support, personally i would say support it until we think fa30 is stable09:16
brtknr#topic FA29 support09:17
flwang1#topic fa29 support09:17
*** openstack changes topic to "fa29 support (Meeting topic: magnum)"09:17
brtknr:( I dont have the wizardry09:17
flwang1brtknr: i can sell it to you if you can buy me a beer09:17
brtknrseems like a good trade09:18
flwang1strigazi: how do you think?09:18
brtknrflwang1: I am happy with that proposal09:18
brtknror when FA31 enters development?09:19
flwang1brtknr: either works i think, and i think it won't take too long, because the coreos driver is sharing most of the atomic driver09:20
brtknrworks for me09:20
flwang1next one?09:21
brtknrI would prefer is use_podman=true09:21
strigazibrtknr: no, default behaviour is better to not change.09:22
brtknris that doable? or would it cause issues for upgrade?09:22
brtknrokay fine09:22
strigaziuser creates a cluster in stein with a CT. In train they should get the same. makes sense?09:22
flwang1strigazi: i would like to understand why didn't you introduce this label at the first time?09:22
strigaziflwang1: because I got complains later :)09:23
flwang1personally, like this09:23
*** rcernin has joined #openstack-containers09:23
flwang1as i mentioned in the comments, it gives us more time to get the confidence to migrate to fedora coreos09:23
brtknrsure, I'm happy with it09:24
flwang1the only thing i'd like to propose is, just skip this label for fedora coreos09:24
brtknrbut needed some convincing09:24
brtknrmaybe we should drop podman support for atomic altogether?09:25
brtknrso its a clean separation09:25
flwang1it's a bit late at this moment, i would say09:25
brtknrits a lot of extra code for a feature with a short shelf life09:26
flwang1we can have a separate, fully isolated code, as long as we deprecate the fedora atomic driver09:26
flwang1are you suggesting only for master?09:26
brtknrwe just need to drop the label... default to using podman for coreos and atomic for fa2909:26
brtknrwith the same driver09:27
strigazigive me a moment09:27
strigaziI didn't mention this in the commit message unfortunately, but i mentioned it in a meeting.09:28
brtknrso USE_PODMAN is an internal label rather than something that the user specifies09:28
strigazibrtknr:  please waot09:28
strigazibrtknr:  please wait09:28
strigazikubelet in an atomic container, doesn't work09:28
strigazithat is why we have podman in the atomic driver.09:29
flwang1iirc, that's for v1.16.x, right?09:29
strigaziIf you don't want to have fedora atomic and and >= 1.16 we can drop it09:29
*** ykarel|lunch is now known as ykarel09:30
strigazibrtknr: flwang1 what do you prefer?09:30
flwang1strigazi: i'm ok for have it for fedora atomic driver09:30
strigazibrtknr: ?09:30
strigaziyou have some concerns09:31
flwang1but i prefer to use true for fedora coreos09:31
flwang1instead of letting user to set it for fedora coreos09:31
strigaziflwang1: https://review.opendev.org/#/c/690053/2/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml@77509:31
strigaziflwang1: https://review.opendev.org/#/c/690053/2/doc/source/user/index.rst@138509:32
brtknrhmm let me thing09:32
brtknri think by the time we start deploying train release to customer sites, which is at least 3 months away, fcos30 should be stable09:34
flwang1strigazi: what happened if i set use_podman=false for driver=fedora_coreos?09:34
brtknrwe dont need the use_podman label, if flwang1 needs it, lets feep it09:34
strigaziwe also have users, so don't forget us too :)09:34
strigaziflwang1: brtknr creation will fail09:34
strigaziwith:09:34
brtknrstrigazi: yes you too! if your users need it!09:35
flwang1yep, but i don't want to see that kind of "surprise"09:35
flwang1wait09:35
strigaziflwang1: | status_reason        | ERROR: Parameter 'use_podman' is invalid: "false" is not an allowed value [true] |09:36
strigaziit is pretty clear and prertty fast09:36
strigazihttp://paste.openstack.org/show/785502/09:36
flwang1unless i set use_podman=false for a fedora_coreos template09:36
flwang1ok, i'm good then09:36
strigazibrtknr: thoughts?09:37
brtknris it better to call it podman_enabled to keep with the convention09:38
flwang1brtknr: good point?09:38
flwang1good point09:38
strigazibrtknr: anything else than naming?09:38
brtknrnope09:39
strigaziare you sure?09:39
flwang1hah09:39
flwang1strigazi: don't be pushy for our lovely brtknr09:39
brtknri am outnumberd09:39
strigaziI mean, I might miss something09:40
brtknrwe need to document it09:40
strigazihttps://review.opendev.org/#/c/690053/2/doc/source/user/index.rst09:40
brtknrexplain in the doc that we might need use_podman in atomic to support 1.16.x+09:40
brtknrexplain in the doc that we need use_podman in atomic to support 1.16.x+09:40
brtknri will add this to review09:41
*** yolanda__ has joined #openstack-containers09:41
brtknri need to look at the code again09:42
brtknri was off yesterday so havent had a chance to review09:42
flwang1let's move?09:43
flwang1we have 17 mins left09:43
*** yolanda has quit IRC09:43
flwang1heat-agent: Check if scripts exists https://review.opendev.org/#/c/689704/09:43
flwang1i have +2ed09:43
brtknrI will review today09:44
brtknrBut happy with it in principle09:45
brtknrShould address the weird behaviour during reboot in heat-contaner-agent log09:45
flwang1brtknr: yep09:46
flwang1i notice that recently as well09:46
strigaziweird behaviour during reboot in heat-contaner-agent log09:46
strigaziplease elaborate09:46
flwang1my case is not reboot, i saw heat-container-agent stuck several times09:47
brtknrstrigazi: once the heat-container-agent finishes its thing, it starts writing: Oct 22 20:48:26 k8s-dlgmc7wmwxlo-master-0.novalocal podman[1492]: /var/lib/os-collect-config/local-data not found. Skipping to journald09:47
flwang1not useful log09:47
flwang1no09:47
brtknrbut after reboot, it would complain about a missing file09:47
strigazibrtknr: I mean about reboot09:47
strigaziok09:47
brtknrjust rebooting the node in the dev environment now09:48
strigaziwhich file? I think my patch fixes it09:48
brtknrstrigazi: one sec, just waiting for the reboot to complete09:48
strigazihttps://github.com/openstack/magnum/commit/3674b3617a770bd71d09e23137ff96f90eb1241a My mistake, it wasn't verbose enough. Using the atomic cli to install kubelet breaks mount09:50
strigazipropagation of secrets, configmaps and so on. Using podman09:50
strigaziin a systemd unit works.09:50
brtknrstrigazi: why do you strongly dislike PODMAN_ENABLED btw?09:50
brtknri am not that fussed09:50
strigaziit doesn't enable anything. But maybe from the user/ops pov it is better09:50
strigazilogs?09:52
flwang1strigazi: that's a valid point09:52
flwang1generally, we use XXX_enabled only for addons09:52
brtknrlook at the end: https://seashells.io/p/WseYPMUE09:52
flwang1not for this kind of case09:52
brtknrlets leave it as USE_PODMAN then09:53
strigazibrtknr: after the stop?09:53
strigazibrtknr: which line?09:54
brtknrOct 23 09:50:2909:54
strigazibrtknr: my patch fixes this09:54
flwang1strigazi: brtknr: can we discuss details offline?09:54
brtknrstrigazi: yes, i already ack this09:54
strigaziflwang1:09:55
strigaziflwang1: ok09:55
flwang1sorry, guys09:55
strigazianything last comments?09:55
flwang1it's late here09:55
brtknrokay, there's the issue with octavia and master lb fip09:55
strigazioh, have a look at the three patches from Theodoros I added in the etherpad09:55
brtknrits working for flwang09:55
brtknrnot for me09:55
brtknrcould you please test this strigazi09:56
flwang1strigazi: i will leave that for brtknr09:56
strigazione final thing from me09:57
strigaziWe need the patches I proposed, for podman and heat-agent, plus the three fixes from Theodoros:09:58
strigazihttps://review.opendev.org/#/c/687879/09:58
strigazihttps://review.opendev.org/#/c/688400/09:58
strigazihttps://review.opendev.org/#/c/688346/09:58
strigaziin the train branch09:58
flwang1CERN is trying to upgrade to train ?09:58
strigaziand release 9.1.009:58
strigaziyes09:59
flwang1ok, i see then09:59
flwang1to use the v1.16.x and later, right?09:59
strigaziyes09:59
strigazi17 in two months09:59
flwang1same here09:59
flwang1catalyst is on stable/stein10:00
flwang1with several private patches10:00
strigaziwe as well10:00
flwang1cherrypicked from master10:00
strigaziflwang1: how many?10:00
flwang1more than 1010:00
strigazi48 here10:00
flwang1from master(train)10:00
strigaziwho wins?10:00
flwang1you win10:00
strigazixD10:00
flwang1my friend, so you're keen than me10:00
strigaziCherryPickaaS10:01
flwang1ok, let's wrap up this meeting10:01
brtknr:D10:01
flwang1anything else?10:01
brtknrwe have only started upgrading to stein10:01
strigaziI'm good, before closing10:02
strigazi+1 or -1 to: We need the patches I proposed, for podman and heat-agent, plus the three fixes from Theodoros:10:02
strigazibrtknr: flwang1 ^^10:02
strigaziand then 9.1.010:02
strigazibrtknr: flwang1 ^^10:02
flwang1strigazi: buy me a beer10:02
strigazisure thing? IPA?10:02
strigazisure thing, IPA?10:02
*** yolanda__ is now known as yolanda10:03
flwang1hopefully i can join the next openstack summit10:03
flwang1do you know where is it?10:03
flwang1after shanghai10:03
strigazithis is a good local one https://www.magictomato.ch/fr/geneve/item_details/531-biere-pale-ale-tangente-brasserie-du-virage10:03
strigaziflwang1: I think Vancouver, but not sure10:03
flwang1the logo looks cool10:04
flwang1if it's Vancouver, i want to go10:04
strigazibrtknr: flwang1 agree with the plan for 9.1.0?10:04
flwang1if it's EU, no go10:04
brtknrstrigazi: sounds good10:04
flwang1strigazi: works for me10:04
flwang1i have to go10:04
strigaziexcellent10:04
flwang1have fun, guys10:04
strigazigood night, get some rest10:04
flwang1o/10:05
brtknrgood nite!10:05
strigazi#endmeeting10:05
*** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting"10:05
openstackMeeting ended Wed Oct 23 10:05:53 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)10:05
openstackMinutes:        http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-10-23-09.01.html10:05
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-10-23-09.01.txt10:05
openstackLog:            http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-10-23-09.01.log.html10:05
strigazibrtknr: do you want to push the image for the nvidia container drivers?10:06
strigazibrtknr: maybe we need to ask someong from the foundation for the license?10:07
strigazibrtknr: does this fix the logs of the agent? https://review.opendev.org/#/c/689704/10:08
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: No new NGs for clusters without an api_address  https://review.opendev.org/68840010:09
brtknrstrigazi: testing now10:12
brtknrstrigazi: yes, I'd like to push image for nvidia container drivers... but dont have the means to test them until November10:13
*** ykarel has quit IRC10:24
*** ttsiouts has quit IRC10:25
*** ykarel has joined #openstack-containers10:25
*** ttsiouts has joined #openstack-containers10:26
*** ttsiouts has quit IRC10:30
strigazibrtknr: did https://review.opendev.org/#/c/689704/ work for you in the end10:30
strigazi?10:30
brtknrstrigazi: its failing Error: unable to pull docker.io/openstackmagnum/heat-container-agent:ussuri-dev10:31
brtknrcan we revert the ussusi-dev for now10:32
strigazibrtknr it will be built when this patch is merged10:32
brtknrit is confusing to bundle it into the same change10:32
strigazibrtknr: we don't have a way to test10:33
strigaziyou can use this: gitlab-registry.cern.ch/strigazi/containers/heat-container-agent:safe-restart10:33
strigazibrtknr:  to have access to the registry secret, the patch needs to be merged10:34
brtknri will just rerun with train-dev tag10:35
brtknrthat should work too right, strigazi ?10:35
strigazitrain-dev, doesn't have the fix10:35
strigazibrtknr: I don't think there is a way to have "scratch" builds10:36
strigazibrtknr: I can push it to the registry if that help10:36
strigazibrtknr: I can push it to the registry if that helps10:36
brtknrstrigazi: ah okay sorry didnt realise, yes please10:36
strigazidocker.io/openstackmagnum/heat-container-agent:68970410:38
strigazitry with this tag 68970410:38
strigazibrtknr: ^^10:38
*** pcaruana has quit IRC10:41
openstackgerritSpyros Trigazis proposed openstack/magnum master: k8s_fedora: Add use_podman label  https://review.opendev.org/69005310:43
strigazibrtknr: ^^10:44
strigazidocs and reno updated10:44
brtknrstrigazi: doing it now10:45
*** ttsiouts has joined #openstack-containers10:58
*** ykarel is now known as ykarel|afk10:58
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: Docker volume size from nodegroups  https://review.opendev.org/68787911:11
*** ykarel|afk has quit IRC11:22
*** ykarel|afk has joined #openstack-containers11:22
*** ykarel|afk is now known as ykarel11:23
*** ramishra has quit IRC11:32
*** ttsiouts has quit IRC12:01
*** ttsiouts has joined #openstack-containers12:02
*** ttsiouts has quit IRC12:06
*** goldyfruit has quit IRC12:14
*** ttsiouts has joined #openstack-containers12:45
strigazibrtknr: did it work?12:45
* brtknr checks12:46
ttsioutsstrigazi: brtknr: CI is having some issues. I can see failures when tox installs dependencies.12:50
brtknrall seemed to be related to keystoneauth1===3.18.012:52
brtknrfrom #openstack-infra: <smcginnis> FWIW, I've been seeing a lot of requirements update jobs fail because some mirrors don't have newer packages yet. Not consistent, so it seems like there are intermittent mirror sync issues.12:53
*** pcaruana has joined #openstack-containers12:53
*** dave-mccowan has joined #openstack-containers12:57
*** ykarel_ has joined #openstack-containers12:58
brtknrstrigazi: its working, thanks :)12:59
*** ykarel_ is now known as ykarel|afk13:01
*** ykarel has quit IRC13:01
*** dave-mccowan has quit IRC13:17
*** rcernin has quit IRC13:24
*** goldyfruit has joined #openstack-containers13:25
strigazibrtknr: +2?13:31
brtknrstrigazi: already done13:32
strigazibrtknr: thanks13:32
*** ykarel|afk is now known as ykarel13:42
*** ramishra has joined #openstack-containers13:48
openstackgerritMerged openstack/magnum master: heat-agent: Check if scripts exists  https://review.opendev.org/68970413:58
*** ivve has quit IRC14:05
*** yolanda has quit IRC14:07
*** yolanda has joined #openstack-containers14:14
*** dave-mccowan has joined #openstack-containers14:16
*** ykarel is now known as ykarel|afk14:21
*** ykarel|afk is now known as ykarel14:32
*** dave-mccowan has quit IRC14:37
*** pcaruana has quit IRC14:57
*** ttsiouts has quit IRC15:09
*** ttsiouts has joined #openstack-containers15:09
*** ttsiouts has quit IRC15:14
openstackgerritMerged openstack/magnum master: Failed state was ignored for default ngs  https://review.opendev.org/68834615:16
*** goldyfruit has quit IRC15:46
*** ykarel is now known as ykarel|afk16:07
*** ykarel|afk is now known as ykarel|away16:52
*** ykarel|away has quit IRC17:43
*** ricolin has quit IRC17:55
*** spsurya has quit IRC17:59
*** jerryz has joined #openstack-containers18:01
*** ricolin has joined #openstack-containers18:03
*** ricolin has quit IRC18:25
*** pcaruana has joined #openstack-containers18:38
*** pcaruana has quit IRC18:44
*** flwang1 has quit IRC19:18
*** gouthamr has quit IRC19:43
*** gouthamr_ has joined #openstack-containers19:54
*** henriqueof has quit IRC21:15
*** goldyfruit has joined #openstack-containers21:38
*** goldyfruit has quit IRC22:04
*** ramishra has quit IRC22:07
*** rcernin has joined #openstack-containers22:33
openstackgerritBharat Kunwar proposed openstack/magnum stable/train: heat-agent: Check if scripts exists  https://review.opendev.org/69076022:58
openstackgerritBharat Kunwar proposed openstack/magnum stable/train: Failed state was ignored for default ngs  https://review.opendev.org/69076423:02

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!