| *** trident has joined #openstack-containers | 00:03 | |
| *** ykarel|away has joined #openstack-containers | 02:50 | |
| *** ricolin has joined #openstack-containers | 03:49 | |
| *** ykarel|away has quit IRC | 04:06 | |
| *** ykarel|away has joined #openstack-containers | 04:19 | |
| *** ramishra has joined #openstack-containers | 04:20 | |
| *** udesale has joined #openstack-containers | 05:02 | |
| andrein | hi everyone, any ideas why this would happen? | 06:14 |
|---|---|---|
| andrein | W0904 20:12:14.134030 1 config.go:73] Argument --sync-config-file or --sync-configmap-name missing. Data synchronization between Keystone and Kubernetes is disabled. | 06:14 |
| andrein | E0904 20:12:14.135498 1 main.go:60] failed to initialize keystone client: open /etc/kubernetes/ca-bundle.crt: no such file or directory | 06:14 |
| andrein | the ac-bundle file indeed does not exist, but I'm not sure what's supposed to create it | 06:14 |
| *** openstackgerrit has quit IRC | 06:51 | |
| *** udesale has quit IRC | 06:56 | |
| *** janki has joined #openstack-containers | 06:58 | |
| *** trungnv has joined #openstack-containers | 07:02 | |
| *** ykarel|away is now known as ykarel | 07:04 | |
| *** sapd1_x has quit IRC | 07:10 | |
| *** trident has quit IRC | 07:21 | |
| *** ivve has joined #openstack-containers | 07:27 | |
| *** trident has joined #openstack-containers | 07:29 | |
| *** rcernin has quit IRC | 07:33 | |
| *** trident has quit IRC | 07:34 | |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| *** trident has joined #openstack-containers | 07:43 | |
| brtknr | andrein: more context please? | 08:21 |
| andrein | brtknr: I've tried to set up two cluster templates, on one with f27 and one with f29. f27 creates successfully, but all nodes are tainted. the only pods running are flannel (running) and keystoneauth (Crashloop backoff) | 08:23 |
| andrein | the error above is from keystoneauth | 08:23 |
| brtknr | andrein: can you check which daemonset is failing to create pods? | 08:24 |
| brtknr | is the openstack-controller-manager pod running in kube-system namespace? | 08:25 |
| brtknr | have you looked at the heat-container-agent logs? | 08:25 |
| andrein | I can, just give me a few minutes to fix my network node (looks like it's not booting anymore so I can't access the cluster at the moment) | 08:25 |
| brtknr | okay | 08:25 |
| brtknr | andrein: also which version of k8s are you tried you run? | 08:26 |
| brtknr | are you trying* | 08:26 |
| brtknr | 1.15 isnt supported on stein at the moment | 08:26 |
| andrein | f27, kubernetes 1.11.6, openstack stein https://www.irccloud.com/pastebin/Ve1PpCUN/ | 08:27 |
| *** ykarel|lunch is now known as ykarel | 08:28 | |
| brtknr | andrein: do you get the same outcome with calico? | 08:28 |
| andrein | that's a good question | 08:28 |
| andrein | let me create a template for calico | 08:29 |
| *** openstackgerrit has joined #openstack-containers | 08:35 | |
| openstackgerrit | pengyuesheng proposed openstack/magnum master: Blacklist eventlet 0.25.0 https://review.opendev.org/680329 | 08:35 |
| *** trident has quit IRC | 09:01 | |
| *** trident has joined #openstack-containers | 09:09 | |
| andrein | brtknr: calico fails almost the same way, except not even calico deploys | 09:18 |
| andrein | https://www.irccloud.com/pastebin/dpGG5fHj/ | 09:18 |
| brtknr | can you paste the link generated by `cat /var/log/cloud-init-output.log | nc seashells.io 1337` | 09:27 |
| brtknr | also `journalctl -u heat-container-agent | nc seashells.io 1337` | 09:27 |
| brtknr | andrein: ^ | 09:27 |
| brtknr | assuming you're happy for me to look at the logs :) | 09:28 |
| brtknr | andrein: also are all the kube* services green? | 09:29 |
| brtknr | on master | 09:29 |
| andrein | kube-* services are running (though most are throwin errors like crazy) | 09:31 |
| andrein | Sep 05 09:31:10 test-7pstxuooelrw-master-0.novalocal runc[2521]: W0905 09:31:10.639248 1 x509.go:172] x509: subject with cn=admin is not in the allowed list: [front-proxy-client kube kubernetes] | 09:31 |
| andrein | on the apiserver for example | 09:31 |
| andrein | (that's why i suspected it's somehow related to the keystone auth pod not working) | 09:32 |
| andrein | brtknr: i've sent you the logs in a private message, i hope that's ok | 09:33 |
| brtknr | thanks, got it | 09:34 |
| openstackgerrit | pengyuesheng proposed openstack/magnum master: Blacklist eventlet 0.21.0,0.23.0,0.25.0 https://review.opendev.org/680329 | 09:48 |
| brtknr | would you mind creating the cluster with `heat_container_agent_tag: train-dev`, we did some work recently on improving the output of the log message | 09:58 |
| brtknr | andrein: this is a cluster label | 09:58 |
| *** danil has joined #openstack-containers | 10:14 | |
| andrein | brtknr: doing that now | 10:16 |
| *** udesale has joined #openstack-containers | 10:20 | |
| *** udesale has quit IRC | 10:28 | |
| *** udesale has joined #openstack-containers | 10:29 | |
| *** usr2033 has joined #openstack-containers | 10:30 | |
| andrein | brtknr: how do I check if the heat_container_agent tag was updated in the new cluster? | 10:36 |
| *** ramishra has quit IRC | 10:39 | |
| *** sapd1_x has joined #openstack-containers | 10:46 | |
| brtknr | in the cluster, try running sudo atomic images list | 10:46 |
| brtknr | and see if the tag is correct? | 10:46 |
| brtknr | did you do heat_container_agent_tag or heat_container_agent | 10:46 |
| brtknr | did you apply the label as heat_container_agent_tag: train-dev or heat_container_agent: train-dev, should the the one with _tag | 10:49 |
| *** trungnv has quit IRC | 11:04 | |
| *** ramishra has joined #openstack-containers | 11:06 | |
| *** sapd1_x has quit IRC | 11:07 | |
| andrein | brtknr: > docker.io/openstackmagnum/heat-container-agent train-dev 1a4602311d43 2019-09-05 10:24 161.38 MB ostree | 11:08 |
| brtknr | andrein: looks good, could you share the output of heat container agent again please? | 11:09 |
| andrein | sure, 1 sec, just noticed this | 11:09 |
| andrein | kube-apiserver log with train-dev https://www.irccloud.com/pastebin/lVEqJX3q/ | 11:09 |
| brtknr | could you try with label kube_tag: v1.14.6 | 11:15 |
| brtknr | andrein: are you able to reach your keystone endpoint from the container? | 11:17 |
| brtknr | try curling it | 11:17 |
| brtknr | from the master* | 11:18 |
| andrein | {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}} | 11:18 |
| andrein | it's reachable | 11:18 |
| andrein | creating the new cluster now (still on f27, should I try the f29 image?) | 11:23 |
| *** lpetrut has joined #openstack-containers | 11:27 | |
| andrein | with kube_tag: v1.14.6 it looks like etcd can't start :-? | 11:37 |
| andrein | i've had a similar issue on f29 with the default heat agent where etcd couldn't be pulled from docker hub (unauthorized, although running the same command by hand worked), but this time there isn't any output in cloud-init-output.log for the etcd image pull | 11:53 |
| *** usr2033 has quit IRC | 12:23 | |
| *** mkuf_ is now known as mkuf | 12:25 | |
| *** jmlowe has quit IRC | 12:36 | |
| *** jmlowe has joined #openstack-containers | 12:59 | |
| *** ramishra has quit IRC | 13:04 | |
| *** ramishra has joined #openstack-containers | 13:04 | |
| *** jchhatbar has joined #openstack-containers | 13:05 | |
| *** janki has quit IRC | 13:06 | |
| *** jchhatba_ has joined #openstack-containers | 13:08 | |
| *** pcaruana has quit IRC | 13:10 | |
| *** jchhatbar has quit IRC | 13:11 | |
| *** pcaruana has joined #openstack-containers | 13:40 | |
| *** jchhatba_ has quit IRC | 13:44 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Trivial fix for cluster creation in master https://review.opendev.org/679987 | 13:55 |
| *** danil has quit IRC | 14:03 | |
| *** lpetrut has quit IRC | 14:05 | |
| *** pcaruana has quit IRC | 14:12 | |
| *** lpetrut has joined #openstack-containers | 14:29 | |
| *** ykarel is now known as ykarel|afk | 14:40 | |
| *** udesale has quit IRC | 14:46 | |
| *** lpetrut has quit IRC | 14:47 | |
| *** udesale has joined #openstack-containers | 14:47 | |
| *** ykarel|afk is now known as ykarel|away | 15:25 | |
| *** ykarel|away has quit IRC | 15:35 | |
| *** jmlowe has quit IRC | 15:47 | |
| *** ykarel|away has joined #openstack-containers | 15:53 | |
| *** jmlowe has joined #openstack-containers | 16:00 | |
| *** altlogbot_0 has quit IRC | 16:01 | |
| *** altlogbot_1 has joined #openstack-containers | 16:02 | |
| *** irclogbot_2 has quit IRC | 16:02 | |
| *** irclogbot_3 has joined #openstack-containers | 16:03 | |
| *** irclogbot_3 has quit IRC | 16:07 | |
| *** irclogbot_3 has joined #openstack-containers | 16:07 | |
| *** ivve has quit IRC | 16:19 | |
| *** sapd1_x has joined #openstack-containers | 16:34 | |
| *** zigo has quit IRC | 16:43 | |
| *** ramishra has quit IRC | 16:45 | |
| *** spsurya has quit IRC | 16:48 | |
| *** sapd1_x has quit IRC | 16:48 | |
| *** zigo has joined #openstack-containers | 17:09 | |
| *** udesale has quit IRC | 17:18 | |
| *** ricolin has quit IRC | 17:35 | |
| *** ykarel|away has quit IRC | 18:56 | |
| *** trident has quit IRC | 19:09 | |
| *** trident has joined #openstack-containers | 19:20 | |
| *** jmlowe has quit IRC | 19:32 | |
| *** jmlowe has joined #openstack-containers | 19:50 | |
| *** lpetrut has joined #openstack-containers | 20:34 | |
| *** lpetrut has quit IRC | 20:38 | |
| *** flwang has quit IRC | 21:00 | |
| -openstackstatus- NOTICE: Gerrit is being restarted to pick up configuration changes. Should be quick. Sorry for the interruption. | 21:13 | |
| *** rcernin has joined #openstack-containers | 23:23 | |
| *** threestrands has joined #openstack-containers | 23:30 | |
| *** rcernin is now known as rcernin|brb | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!