| *** mdnadeem has joined #openstack-containers | 00:20 | |
| *** yamamoto has joined #openstack-containers | 00:34 | |
| *** yamamoto has quit IRC | 00:39 | |
| *** hongbin has joined #openstack-containers | 00:43 | |
| *** fragatina has quit IRC | 00:44 | |
| *** gsimondon has joined #openstack-containers | 01:01 | |
| *** gsimondon has quit IRC | 01:05 | |
| *** dardelean_ has joined #openstack-containers | 01:05 | |
| *** dardelean_ has quit IRC | 01:10 | |
| *** Nel1x has joined #openstack-containers | 01:22 | |
| *** georgem11 has quit IRC | 01:39 | |
| *** zhubingb_ has joined #openstack-containers | 01:49 | |
| *** zhubingb_ has quit IRC | 01:54 | |
| *** gsimondon has joined #openstack-containers | 01:55 | |
| *** gsimondon has quit IRC | 01:59 | |
| *** pengdake has joined #openstack-containers | 02:19 | |
| *** hongbin has quit IRC | 02:26 | |
| *** gsimondon has joined #openstack-containers | 02:49 | |
| *** gsimondon has quit IRC | 02:54 | |
| *** Nel1x has quit IRC | 03:02 | |
| *** fragatina has joined #openstack-containers | 03:04 | |
| *** fragatina has quit IRC | 03:05 | |
| *** fragatin_ has joined #openstack-containers | 03:05 | |
| *** ramishra has joined #openstack-containers | 03:06 | |
| *** fragatin_ has quit IRC | 03:17 | |
| *** fragatina has joined #openstack-containers | 03:17 | |
| *** yamamoto has joined #openstack-containers | 03:26 | |
| *** yamamoto has quit IRC | 03:26 | |
| *** yamamoto has joined #openstack-containers | 03:31 | |
| *** Roybent has joined #openstack-containers | 03:36 | |
| *** gsimondon has joined #openstack-containers | 03:43 | |
| *** dave-mcc_ has quit IRC | 03:44 | |
| *** lpetrut has joined #openstack-containers | 03:45 | |
| *** gsimondon has quit IRC | 03:48 | |
| *** udesale has joined #openstack-containers | 03:48 | |
| *** ricolin has joined #openstack-containers | 03:52 | |
| *** pengdake has quit IRC | 04:22 | |
| *** lpetrut has quit IRC | 04:23 | |
| *** chhagarw has joined #openstack-containers | 04:25 | |
| *** chhagarw has quit IRC | 04:32 | |
| *** gsimondon has joined #openstack-containers | 04:37 | |
| *** gsimondon has quit IRC | 04:42 | |
| *** ianychoi has quit IRC | 04:44 | |
| *** zhubingb_ has joined #openstack-containers | 04:51 | |
| *** flwang1 has quit IRC | 04:54 | |
| *** zhubingb_ has quit IRC | 04:56 | |
| *** gsimondon has joined #openstack-containers | 05:31 | |
| *** gsimondon has quit IRC | 05:35 | |
| *** AlexeyAbashkin has joined #openstack-containers | 05:48 | |
| *** pengdake has joined #openstack-containers | 05:51 | |
| *** Alexey_Abashkin has joined #openstack-containers | 05:54 | |
| *** AlexeyAbashkin has quit IRC | 05:55 | |
| *** Alexey_Abashkin is now known as AlexeyAbashkin | 05:55 | |
| *** serlex has joined #openstack-containers | 05:56 | |
| *** iranzo has joined #openstack-containers | 05:57 | |
| *** pcaruana has joined #openstack-containers | 05:57 | |
| -openstackstatus- NOTICE: Zuul stopped receiving gerrit events around 04:00UTC; any changes submitted between then and now will probably require a "recheck" comment to be requeued. Thanks! | 06:00 | |
| *** armaan has joined #openstack-containers | 06:04 | |
| *** pcaruana has quit IRC | 06:14 | |
| *** pcaruana has joined #openstack-containers | 06:15 | |
| *** armaan has quit IRC | 06:20 | |
| *** armaan has joined #openstack-containers | 06:20 | |
| *** lpetrut has joined #openstack-containers | 06:22 | |
| *** gsimondon has joined #openstack-containers | 06:30 | |
| *** gsimondo1 has joined #openstack-containers | 06:36 | |
| *** gsimondon has quit IRC | 06:38 | |
| *** armaan has quit IRC | 06:44 | |
| *** armaan has joined #openstack-containers | 06:45 | |
| *** zhubingb_ has joined #openstack-containers | 06:53 | |
| *** zhubingb_ has quit IRC | 06:57 | |
| *** b_bezak has joined #openstack-containers | 07:02 | |
| *** rcernin has quit IRC | 07:13 | |
| *** ramishra has quit IRC | 07:14 | |
| *** dardelean_ has joined #openstack-containers | 07:15 | |
| *** Alexey_Abashkin has joined #openstack-containers | 07:23 | |
| *** AlexeyAbashkin has quit IRC | 07:26 | |
| *** Alexey_Abashkin is now known as AlexeyAbashkin | 07:26 | |
| *** dardelean_ has quit IRC | 07:32 | |
| *** hongbin has joined #openstack-containers | 07:32 | |
| *** dardelean_ has joined #openstack-containers | 07:32 | |
| *** hongbin has quit IRC | 07:35 | |
| *** dardelean_ has quit IRC | 07:36 | |
| *** ramishra has joined #openstack-containers | 07:44 | |
| *** rcernin has joined #openstack-containers | 07:48 | |
| *** ktibi has joined #openstack-containers | 07:57 | |
| *** AlexeyAbashkin has quit IRC | 08:06 | |
| *** yamamoto has quit IRC | 08:08 | |
| *** ramishra has quit IRC | 08:11 | |
| *** ramishra has joined #openstack-containers | 08:13 | |
| *** mgoddard has joined #openstack-containers | 08:15 | |
| *** ricolin has quit IRC | 08:19 | |
| *** yamamoto has joined #openstack-containers | 08:22 | |
| *** AlexeyAbashkin has joined #openstack-containers | 08:28 | |
| *** chhagarw has joined #openstack-containers | 08:30 | |
| *** pengdake has quit IRC | 08:46 | |
| *** dardelean_ has joined #openstack-containers | 08:48 | |
| *** pengdake has joined #openstack-containers | 08:54 | |
| *** Alexey_Abashkin has joined #openstack-containers | 09:00 | |
| *** AlexeyAbashkin has quit IRC | 09:01 | |
| *** Alexey_Abashkin is now known as AlexeyAbashkin | 09:01 | |
| *** flwang1 has joined #openstack-containers | 09:16 | |
| *** ramishra has quit IRC | 09:17 | |
| *** ramishra has joined #openstack-containers | 09:19 | |
| *** openstackgerrit has joined #openstack-containers | 09:21 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Fix race condition issue for k8s multi masters https://review.openstack.org/573639 | 09:21 |
|---|---|---|
| flwang1 | strigazi: ping | 09:22 |
| strigazi | the idea looks ok, we need to modify the scripts to not run on the secondary masters | 09:23 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Fix race condition issue for k8s multi masters https://review.openstack.org/573639 | 09:24 |
| flwang1 | which scripts? | 09:24 |
| flwang1 | strigazi: see https://github.com/kubernetes/kubernetes/issues/44165 | 09:25 |
| flwang1 | it's a known issue in kubectl | 09:25 |
| flwang1 | and it's still open unfortunately | 09:25 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Add option to specify Cgroup driver for Kubelet https://review.openstack.org/571583 | 09:25 |
| strigazi | all scripts that try to create something on the api server | 09:26 |
| strigazi | if is_not_primary_master : exit 0 | 09:26 |
| flwang1 | how the script know it's not on primary_master? | 09:28 |
| strigazi | https://github.com/openstack/magnum/blob/master/magnum/drivers/swarm_fedora_atomic_v2/templates/fragments/write-swarm-master-service.sh#L13 | 09:36 |
| *** Roybent has quit IRC | 09:40 | |
| flwang1 | strigazi: cool, thanks | 09:51 |
| *** mdnadeem has quit IRC | 09:55 | |
| *** zhubingb_ has joined #openstack-containers | 09:55 | |
| *** zhubingb_ has quit IRC | 10:00 | |
| *** pengdake has quit IRC | 10:05 | |
| *** Roybent has joined #openstack-containers | 10:06 | |
| *** Roybent has quit IRC | 10:14 | |
| *** ricolin has joined #openstack-containers | 10:16 | |
| *** udesale_ has joined #openstack-containers | 10:27 | |
| *** udesale has quit IRC | 10:29 | |
| *** udesale_ has quit IRC | 10:31 | |
| *** rcernin has quit IRC | 10:37 | |
| brtknr | flwang1: how does creating a secondary master resolve race condition issue? is this becuase you've defined dependence on primary master for kube minion? | 10:40 |
| strigazi | brtknr: yes, the secondary masters and the minions depend on the primary master | 11:15 |
| *** pengdake has joined #openstack-containers | 11:20 | |
| *** pengdake has quit IRC | 11:26 | |
| *** AlexeyAbashkin has quit IRC | 11:53 | |
| *** AlexeyAbashkin has joined #openstack-containers | 11:54 | |
| *** vabada has joined #openstack-containers | 12:05 | |
| *** yamamoto has quit IRC | 12:11 | |
| *** AlexeyAbashkin has quit IRC | 12:12 | |
| *** AlexeyAbashkin has joined #openstack-containers | 12:12 | |
| *** yamamoto has joined #openstack-containers | 12:15 | |
| *** yamamoto has quit IRC | 12:16 | |
| *** armaan has quit IRC | 12:25 | |
| *** yamamoto has joined #openstack-containers | 12:27 | |
| *** yamamoto has quit IRC | 12:28 | |
| *** AlexeyAbashkin has quit IRC | 12:30 | |
| *** AlexeyAbashkin has joined #openstack-containers | 12:33 | |
| *** AlexeyAbashkin has quit IRC | 12:41 | |
| *** armaan has joined #openstack-containers | 12:41 | |
| *** armaan has quit IRC | 12:41 | |
| *** AlexeyAbashkin has joined #openstack-containers | 12:41 | |
| *** armaan has joined #openstack-containers | 12:41 | |
| *** georgem1 has joined #openstack-containers | 12:45 | |
| *** armaan has quit IRC | 12:46 | |
| *** georgem1 has left #openstack-containers | 12:56 | |
| *** AlexeyAbashkin has quit IRC | 12:56 | |
| *** AlexeyAbashkin has joined #openstack-containers | 12:57 | |
| *** zhubingb_ has joined #openstack-containers | 12:58 | |
| *** zhubingb_ has quit IRC | 13:02 | |
| *** vijaykc4 has joined #openstack-containers | 13:04 | |
| *** Nel1x has joined #openstack-containers | 13:09 | |
| *** Nel1x has quit IRC | 13:16 | |
| *** vabada has quit IRC | 13:16 | |
| *** yamamoto has joined #openstack-containers | 13:29 | |
| *** yamamoto has quit IRC | 13:31 | |
| *** yamamoto has joined #openstack-containers | 13:32 | |
| *** jmlowe has quit IRC | 13:35 | |
| *** ramishra has quit IRC | 13:38 | |
| -openstackstatus- NOTICE: A misapplied distro security package update caused many jobs to fail with a MERGER_FAILURE error between ~06:30-12:30 UTC; these can be safely rechecked now that the problem has been addressed | 13:46 | |
| *** markguz_ has joined #openstack-containers | 13:57 | |
| *** lpetrut has quit IRC | 13:58 | |
| *** ianychoi has joined #openstack-containers | 14:04 | |
| *** hongbin has joined #openstack-containers | 14:05 | |
| *** jmlowe has joined #openstack-containers | 14:07 | |
| *** markguz_ has quit IRC | 14:11 | |
| *** gsimondo1 has quit IRC | 14:15 | |
| *** jmlowe has quit IRC | 14:15 | |
| *** ricolin has quit IRC | 14:18 | |
| *** gsimondon has joined #openstack-containers | 14:20 | |
| *** AlexeyAbashkin has quit IRC | 14:21 | |
| *** serlex has quit IRC | 14:22 | |
| *** lpetrut has joined #openstack-containers | 14:22 | |
| *** mgoddard has quit IRC | 14:31 | |
| *** georgem1 has joined #openstack-containers | 14:37 | |
| *** b_bezak has quit IRC | 14:39 | |
| *** jmlowe has joined #openstack-containers | 14:39 | |
| *** mgoddard has joined #openstack-containers | 14:47 | |
| *** lpetrut has quit IRC | 14:54 | |
| *** zhubingb_ has joined #openstack-containers | 14:59 | |
| georgem1 | Any idea what can be done about https://bugs.launchpad.net/magnum/+bug/1775460 ? | 15:00 |
| openstack | Launchpad bug 1775460 in Magnum "Cluster creation fails because of password complexity" [Undecided,New] | 15:00 |
| *** zhubingb_ has quit IRC | 15:04 | |
| *** armaan has joined #openstack-containers | 15:05 | |
| *** zhubingb_ has joined #openstack-containers | 15:10 | |
| *** gsimondon has quit IRC | 15:14 | |
| *** ricolin has joined #openstack-containers | 15:25 | |
| *** dave-mccowan has joined #openstack-containers | 15:25 | |
| imdigitaljim | george1: just curious roughly at what point of cluster creation does this failure get thrown for you? | 15:27 |
| georgem1 | imdigitaljim: stack resource: "kube_masters" and Stack Resource Type: OS::Heat::ResourceGroup | 15:28 |
| georgem1 | imdigitaljim: https://photos.app.goo.gl/6bONrIZsI7ZdfNzS2 | 15:30 |
| imdigitaljim | the vms got created? | 15:31 |
| imdigitaljim | get* | 15:32 |
| *** dave-mccowan has quit IRC | 15:32 | |
| *** iranzo has quit IRC | 15:32 | |
| georgem1 | nope | 15:32 |
| imdigitaljim | hmm so not that far | 15:33 |
| imdigitaljim | ok, ill see if i can spot something | 15:33 |
| imdigitaljim | thanks! | 15:33 |
| *** zhubingb_ has quit IRC | 15:40 | |
| georgem1 | thank you too | 15:45 |
| imdigitaljim | flwang1: https://review.openstack.org/#/c/573639/ I was thinking about how to resolve this issue as well and I believe this approach will cause other complexity-related issues such as if you ever want to scale masters. What do you all think about leaving it as one resource group but adding logic on the k8s/fragments side for bootstrapping that 'primary master'. Make a parameter out of the %index% value and write it to the heat-params | 15:53 |
| imdigitaljim | file and then during the software configuration/cloud-init have a set of common master scripts but also some that run for a bootstrapping master vs a regular master. The host can just look at the heat-param and if it is master 0 you could execute one thing and if not exit 0 and continue to the common scripts (if any). Alternatively, if the host is master 1, it can wait for some indication that master 0 has completed its work and then | 15:53 |
| imdigitaljim | continue. There could be retry logic, maybe wait a few seconds between each check for the okay to continue, and maximum retries before returning a failure to heat through the wait handler so it wont have to wait until the timeout rather than reasonable time passed. The only thing here is if new hosts come up as master 0 (e.g. master 0 failed and a new one brought up) later they would need to check if the cluster is already active, it | 15:53 |
| imdigitaljim | should just then take the master 1 route. | 15:53 |
| *** mago_ has quit IRC | 15:53 | |
| *** ktibi_ has joined #openstack-containers | 15:57 | |
| *** ktibi has quit IRC | 16:00 | |
| imdigitaljim | https://review.openstack.org/#/c/571583/ +1 for this feature but would someone enlighten me on what would be the rationale for whether I'd use one or the other (in terms of Magnum usage)? I'm doing Magnum/Queens Fedora-k8s on Fedora27 but I've only been getting it functional with cgroupfs (against the system default of both Fedora and magnum) | 16:06 |
| *** dardelean_ has quit IRC | 16:12 | |
| *** dardelean_ has joined #openstack-containers | 16:12 | |
| *** armaan has quit IRC | 16:16 | |
| *** armaan has joined #openstack-containers | 16:16 | |
| *** vijaykc4 has quit IRC | 16:23 | |
| *** lpetrut has joined #openstack-containers | 16:29 | |
| *** gsimondon has joined #openstack-containers | 16:29 | |
| *** mgoddard has quit IRC | 16:32 | |
| *** fragatina has quit IRC | 16:33 | |
| *** fragatina has joined #openstack-containers | 16:34 | |
| *** vijaykc4 has joined #openstack-containers | 16:34 | |
| *** zhubingb_ has joined #openstack-containers | 16:41 | |
| *** Guest14735 is now known as sdake | 16:41 | |
| *** zhubingb_ has quit IRC | 16:46 | |
| *** lpetrut has quit IRC | 16:59 | |
| *** pcaruana has quit IRC | 17:02 | |
| *** chhagarw has quit IRC | 17:10 | |
| *** markguz_ has joined #openstack-containers | 17:12 | |
| *** markguz_ has quit IRC | 17:16 | |
| *** dardelean_ has quit IRC | 17:20 | |
| *** dardelean_ has joined #openstack-containers | 17:20 | |
| *** iranzo has joined #openstack-containers | 17:21 | |
| *** iranzo has joined #openstack-containers | 17:21 | |
| -openstackstatus- NOTICE: The Zuul scheduler was offline briefly to clean up from debugging a nodepool issue, so changes uploaded or approved between 16:50 and 17:15 UTC may need to be rechecked or reapproved (all already queued changes are in the process of being reenqueued now) | 17:23 | |
| *** dardelean_ has quit IRC | 17:25 | |
| *** lpetrut has joined #openstack-containers | 17:41 | |
| *** dardelean_ has joined #openstack-containers | 17:54 | |
| *** dardelean_ has quit IRC | 17:58 | |
| *** georgem1 has left #openstack-containers | 18:00 | |
| *** markguz_ has joined #openstack-containers | 18:10 | |
| *** markguz_ has quit IRC | 18:25 | |
| *** yamamoto has quit IRC | 18:25 | |
| *** yamamoto has joined #openstack-containers | 18:26 | |
| *** yamamoto has quit IRC | 18:31 | |
| *** ricolin has quit IRC | 18:36 | |
| *** zhubingb_ has joined #openstack-containers | 18:43 | |
| *** zhubingb_ has quit IRC | 18:48 | |
| *** fragatina has quit IRC | 18:49 | |
| flwang1 | imdigitaljim: still around? | 18:58 |
| imdigitaljim | yeah | 19:00 |
| flwang1 | imdigitaljim: thanks for your comments, as I mentioned in the launchpad bug, I know https://github.com/kubernetes/kubernetes/issues/44165 is the root cause | 19:03 |
| flwang1 | however, it has been there more than 1 year, so we may have to fix it in magnum | 19:04 |
| flwang1 | imdigitaljim: and i like your idea in the comments #1 | 19:04 |
| flwang1 | not sure if we can pass the %index% as a heat param, if we can, that could work | 19:05 |
| imdigitaljim | yeah i can confirm we can (i did it on a basic centos7 driver i was doing) | 19:05 |
| imdigitaljim | node_id: '%index%' | 19:06 |
| imdigitaljim | something like that would be sufficient to pass the index on the OS::Heat::ResourceGroup | 19:07 |
| imdigitaljim | then if [ "$NODE_ID" == "0" ]; then (do singleton operations) fi | 19:08 |
| imdigitaljim | this method works as expected | 19:09 |
| imdigitaljim | i can setup a PR with it over next week if you | 19:10 |
| imdigitaljim | 'd like | 19:10 |
| flwang1 | thank you, i'm keen to test it asap because it will block our release. it's kind a good idea | 19:11 |
| flwang1 | thank you for the valuable comments | 19:11 |
| flwang1 | are you using magnum in your prod? | 19:12 |
| flwang1 | imdigitaljim: i will propose a patch set today | 19:16 |
| flwang1 | here is Sat now ;) | 19:16 |
| imdigitaljim | We want to get Magnum to production soon but we've got some patching we've had to do (upstreaming soon) and some tuning for environment but we hope to have it in prod ASAP | 19:28 |
| imdigitaljim | but only in a dev capacity right now | 19:28 |
| *** yamamoto has joined #openstack-containers | 19:29 | |
| flwang1 | imdigitaljim: cool, pls pls upstream and i'm happy to review them | 19:32 |
| *** jmlowe has quit IRC | 19:33 | |
| flwang1 | as for the multi masters bug, did you see it in your test env? | 19:33 |
| flwang1 | imdigitaljim: ^ | 19:34 |
| *** armaan has quit IRC | 19:34 | |
| *** armaan has joined #openstack-containers | 19:34 | |
| *** yamamoto has quit IRC | 19:37 | |
| imdigitaljim | flwang1: yes it has intermittently come up | 20:01 |
| flwang1 | imdigitaljim: cool | 20:01 |
| flwang1 | looking forward to reviewing | 20:01 |
| *** armaan has quit IRC | 20:12 | |
| *** armaan has joined #openstack-containers | 20:13 | |
| *** mgoddard has joined #openstack-containers | 20:22 | |
| *** armaan has quit IRC | 20:31 | |
| *** armaan has joined #openstack-containers | 20:31 | |
| *** iranzo has quit IRC | 20:35 | |
| *** vijaykc4 has quit IRC | 20:44 | |
| *** ktibi_ has quit IRC | 20:47 | |
| *** gsimondon has quit IRC | 21:01 | |
| *** fragatina has joined #openstack-containers | 21:02 | |
| *** fragatina has quit IRC | 21:03 | |
| *** fragatina has joined #openstack-containers | 21:04 | |
| *** fragatina has quit IRC | 21:05 | |
| *** fragatina has joined #openstack-containers | 21:05 | |
| *** vijaykc4 has joined #openstack-containers | 21:05 | |
| *** mgoddard has quit IRC | 21:07 | |
| *** lpetrut has quit IRC | 21:15 | |
| *** jmlowe has joined #openstack-containers | 21:19 | |
| *** markguz_ has joined #openstack-containers | 21:26 | |
| *** markguz_ has quit IRC | 21:31 | |
| *** armaan has quit IRC | 21:41 | |
| *** jmlowe has quit IRC | 21:43 | |
| *** jmlowe has joined #openstack-containers | 21:48 | |
| *** jmlowe has quit IRC | 22:00 | |
| *** jmlowe has joined #openstack-containers | 22:01 | |
| *** ykarel has joined #openstack-containers | 22:09 | |
| *** ykarel_ has joined #openstack-containers | 22:14 | |
| *** jmlowe has quit IRC | 22:15 | |
| *** ykarel has quit IRC | 22:16 | |
| *** jmlowe has joined #openstack-containers | 22:20 | |
| *** harlowja has joined #openstack-containers | 22:30 | |
| *** vijaykc4 has quit IRC | 22:40 | |
| *** hongbin has quit IRC | 22:43 | |
| *** ykarel_ has quit IRC | 22:49 | |
| *** ykarel_ has joined #openstack-containers | 23:04 | |
| *** ykarel_ has quit IRC | 23:09 | |
| *** markguz_ has joined #openstack-containers | 23:22 | |
| *** vijaykc4 has joined #openstack-containers | 23:43 | |
| *** markguz_ has quit IRC | 23:44 | |
| *** markguz_ has joined #openstack-containers | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!