Sunday, 2017-05-28

*** glycerine has joined #openstack-kolla00:03
*** itlinux has joined #openstack-kolla00:06
*** itlinux has quit IRC00:07
*** Pavo has joined #openstack-kolla00:10
*** manheim has joined #openstack-kolla00:14
*** manheim has quit IRC00:19
sdakesbezverk_ looks like some cleanup jobs are consinstely failing with tis error: http://logs.openstack.org/72/468672/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/8bb4f5a/console.html#_2017-05-27_23_57_01_06094700:22
sdakeselenium is imploding00:22
sdakekfox1111 sbezverk_ ^^00:22
sdakehttp://logs.openstack.org/72/468672/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/8bb4f5a/console.html#_2017-05-27_23_54_56_08245500:22
sdakehappens on all cloud providers00:23
sbezverk_sdake: it is not code issue, it looks something with devicemapper. There was update for centos recently, maybe it is collaterral.. how about Ubuntu job?00:28
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403100:31
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403100:33
sdakethe reason ceph gates fail is because of cleanup_tests.sh00:36
sdakethat integration merged in february - so YMMV00:37
*** salv-orlando has joined #openstack-kolla00:38
sbezverk_sdake: the gate was fine until recently, but recently kernel got changed  and 3.10.0-514.21.1.el7.x86_64 now used at gate00:39
sdakethis problem is already known then?00:40
*** salv-orlando has quit IRC00:42
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Do Not Merge, kolla-kubernetes gate tests  https://review.openstack.org/46106300:43
sbezverk_sdake: similar error was reported on another version of kernel00:44
sdakethe cleanup failure of the selenium backtrace?00:44
sdakeof/or00:44
sbezverk_sdake: no nova delete getting stuck in init00:45
sdakewhat type of images do those cleanup jobs use?00:46
sbezverk_sdake: the same nova regular containers are using..00:46
sdakeis there a global describe output in the capture logs?00:49
*** manheim has joined #openstack-kolla00:51
sdakelooks like entrypoint is the first thing it hits00:52
sdakehttp://logs.openstack.org/80/422380/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/7023dc3/console.html00:52
sdakenova cleanup jobs, but no nova jobs00:53
*** emccormick has joined #openstack-kolla00:53
*** manheim has quit IRC00:55
sdakesbezverk_ cleanup waits on keystone00:57
sdakeis this a keystone service: http://logs.openstack.org/80/422380/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/7023dc3/console.html#_2017-05-27_21_02_22_39615200:57
*** Pavo has quit IRC00:57
*** Pavo has joined #openstack-kolla00:59
*** emccormick has quit IRC01:10
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Test removal of selenium from ceph gates  https://review.openstack.org/46870001:10
*** emccormick has joined #openstack-kolla01:10
*** emccormick has quit IRC01:14
sdakesbezverk_ clarkb said the images are rebiult daily from latest upstream01:14
sdakesbezverk_ but he isn't sure when new kernels come in - its whenever upstream releases a new kernel01:14
sdakethe kernel and syscall interface has all kinds of interactions between different versions01:14
sdakealthough red hat is pretty strict about not modifying the syscall ABI01:15
sdakeand by pretyt, I mean super :)01:15
*** emccormick has joined #openstack-kolla01:20
*** manheim has joined #openstack-kolla01:26
*** manheim has quit IRC01:31
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403101:31
*** hrw has quit IRC01:32
*** hrw has joined #openstack-kolla01:34
*** manheim has joined #openstack-kolla01:35
*** yingjun has joined #openstack-kolla01:39
*** yingjun has quit IRC01:39
*** manheim has quit IRC01:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Test of more gate output  https://review.openstack.org/46870101:46
*** manheim has joined #openstack-kolla01:49
kbaegisAnyone running into this? https://hastebin.com/womunobomi.sql01:53
*** manheim has quit IRC01:54
*** glycerine has quit IRC01:54
*** kbaegis has quit IRC02:02
*** manheim has joined #openstack-kolla02:07
*** Pavo has quit IRC02:14
*** manheim has quit IRC02:26
sbezverk_sdake: 6 hours ago ironic PS got merged with ALL gate jobs in green. I doubt it broke anytihing, I believe it must be something external..02:27
*** thomasem- has quit IRC02:27
*** thomasem_ has joined #openstack-kolla02:27
*** manheim has joined #openstack-kolla02:35
*** manheim has quit IRC02:39
*** jascott1 has joined #openstack-kolla02:51
*** manheim has joined #openstack-kolla02:53
*** Pavo has joined #openstack-kolla02:54
*** jascott1 has quit IRC02:55
*** klindgren_ has joined #openstack-kolla02:58
*** klindgren has quit IRC03:00
*** kbaegis has joined #openstack-kolla03:02
*** manheim has quit IRC03:07
*** manheim has joined #openstack-kolla03:16
*** manheim has quit IRC03:20
*** manheim has joined #openstack-kolla03:30
*** Pavo has quit IRC03:31
*** manheim has quit IRC03:34
*** kbaegis has quit IRC03:36
*** kbaegis has joined #openstack-kolla03:36
*** kbaegis has quit IRC03:42
*** kbaegis has joined #openstack-kolla03:42
*** emccormick has quit IRC03:45
*** manheim has joined #openstack-kolla03:48
*** kbaegis has quit IRC03:51
*** kbaegis1 has joined #openstack-kolla03:51
*** manheim has quit IRC03:52
*** manheim has joined #openstack-kolla04:06
*** manheim has quit IRC04:11
*** manheim has joined #openstack-kolla04:24
*** kbaegis1 has quit IRC04:28
*** manheim has quit IRC04:29
*** manheim has joined #openstack-kolla04:43
*** manheim has quit IRC04:47
*** manheim has joined #openstack-kolla05:01
*** salv-orlando has joined #openstack-kolla05:05
*** manheim has quit IRC05:10
*** manheim has joined #openstack-kolla05:19
*** jamesbenson has joined #openstack-kolla05:22
*** manheim has quit IRC05:23
*** jamesbenson has quit IRC05:27
*** jamesbenson has joined #openstack-kolla05:28
*** jamesbenson has quit IRC05:31
*** jamesbenson has joined #openstack-kolla05:32
*** jamesbenson has quit IRC05:33
*** jamesbenson has joined #openstack-kolla05:33
*** manheim has joined #openstack-kolla05:33
*** manheim has quit IRC05:38
*** manheim has joined #openstack-kolla05:51
*** manheim has quit IRC05:55
*** salv-orlando has quit IRC06:02
*** manheim has joined #openstack-kolla06:09
*** manheim has quit IRC06:14
*** jamesbenson has quit IRC06:20
*** jamesbenson has joined #openstack-kolla06:21
*** jamesbenson has quit IRC06:25
*** janki has joined #openstack-kolla06:27
*** manheim has joined #openstack-kolla06:28
*** manheim has quit IRC06:32
*** manheim has joined #openstack-kolla06:41
*** manheim has quit IRC06:46
*** manheim has joined #openstack-kolla07:00
*** manheim has quit IRC07:05
*** manheim has joined #openstack-kolla07:13
*** manheim has quit IRC07:18
*** manheim has joined #openstack-kolla07:25
*** janki has quit IRC07:27
*** manheim has quit IRC07:29
*** jascott1 has joined #openstack-kolla07:33
*** salv-orlando has joined #openstack-kolla07:33
*** jascott1 has quit IRC07:37
*** jascott1 has joined #openstack-kolla07:37
*** salv-orlando has quit IRC07:38
*** janki has joined #openstack-kolla07:52
*** tvignaud_pto has quit IRC08:02
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403108:03
*** jascott1 has quit IRC08:08
*** jascott1 has joined #openstack-kolla08:09
*** jascott1 has quit IRC08:13
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Revert "Remove kolla-ansible dependencies"  https://review.openstack.org/46871108:14
*** tvignaud_pto has joined #openstack-kolla08:15
*** mewald has joined #openstack-kolla08:19
*** jroll has quit IRC08:20
sdakesbezverk_ our jobs don't gate, meaning they dont run again pre-merge, so it could indeed be a regression08:20
sdakesbezverk_ or it could indeed be something external08:21
*** jascott1 has joined #openstack-kolla08:29
*** pbourke has quit IRC08:32
*** pbourke has joined #openstack-kolla08:33
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403109:01
*** janki has quit IRC10:17
*** jascott1 has quit IRC10:31
*** salv-orlando has joined #openstack-kolla10:36
*** salv-orlando has quit IRC10:41
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872011:32
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872012:08
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872012:12
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872012:13
*** jamesPR has joined #openstack-kolla12:14
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872012:47
*** manheim has joined #openstack-kolla13:19
*** Pavo has joined #openstack-kolla13:21
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872013:28
*** emccormick has joined #openstack-kolla13:33
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872013:40
*** itlinux has joined #openstack-kolla13:48
*** manheim has quit IRC13:52
*** Manheim has joined #openstack-kolla13:54
*** emccormick has quit IRC14:01
*** emccormick has joined #openstack-kolla14:04
*** kbaegis has joined #openstack-kolla14:14
*** Mannheim has joined #openstack-kolla14:17
*** Manheim has quit IRC14:17
*** manheim has joined #openstack-kolla14:23
*** emccormick has quit IRC14:24
*** emccormick has joined #openstack-kolla14:24
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872014:27
*** manheim has quit IRC14:27
sbezverk_kolla-k8s latest docker update broke come gate jobs, when I pin docker to the previous version, those jobs pass..14:28
sbezverk_s/come/some/14:28
jamesPRsbezverk_, kolla-k8s is kubernetes?14:40
kbaegisHey guys. Seem to be having some trouble with tgtd and the iscsi initiators. Anyone have experience with the LVM/iscsi setup?14:41
kbaegisHere's boot from vol: https://hastebin.com/womunobomi.sql, and attach volume on booted: https://hastebin.com/xeqoruhoqi.sql14:41
kbaegisShould we be setting iscsi_initiator_list in cinder.conf?14:42
*** emccormick has quit IRC14:42
sbezverk_jamesPR : yep14:43
sbezverk_kbaegis : what is up with iscsid/tgtd?14:43
sbezverk_kbaegis : check cinder.conf make sure lvm section has all required values set14:46
sbezverk_it looks like it is missing at least ip address of the host acting as tgtd server14:46
*** itlinux has quit IRC14:47
*** jamesPR has quit IRC14:54
*** japestinho has joined #openstack-kolla14:59
kbaegissbezverk_: Thanks, will do15:02
kbaegisI think it's set in there15:02
kbaegisbut the fqdn isn't15:02
kbaegisHere's the backend that kolla configures for cinder.conf: https://hastebin.com/uragehegac.ini @sbezverk_15:04
kbaegisSo I can create volumes, but when I go to attach them to a nova instance it fails15:04
sbezverk_kbaegis : yep it is missing ip address for sure15:05
kbaegisPertinent error ostensibly being FailedISCSITargetPortalLogin: Could not login to any iSCSI portal15:05
kbaegissbezverk_: Well we'll need to patch this :)15:05
sbezverk_at one point kolla-ansible folks decided to remove default config for cinder-volume-lvm15:05
*** eaguilar has joined #openstack-kolla15:05
kbaegissbezverk_: Oh. I'd rather use ceph honestly, but I only have 2 nodes, so it'd be a waste15:05
sbezverk_so it has to be done with "custom" config or reconfiguration15:06
kbaegisSure. So I'll figure out where I need to set that15:06
kbaegisIs the IP declared in iscsi_initiator_list?15:06
kbaegisAnd will I set that in cinder.conf or nova.conf?15:07
sbezverk_kbaegis: all config shoulkd go into cender.conf15:07
sbezverk_kbaegis you need to setup iscsi_ip_address15:08
kbaegissbezverk_: Confusing. So what do I set that to? The local node IP?15:10
sbezverk_kolla-k8s: https://review.openstack.org/468720 will restore the gates jobs15:10
kbaegissbezverk_: What if it's a group of multiple iscsi hosts? How do I reference them?15:10
sbezverk_kbaegis : ip of the host where you run tgtd15:11
sbezverk_you need to create separate section15:11
kbaegissbezverk_: Gotcha. Thanks. I can set that in the conf/ directory, but how should we patch that to pull it from the ansible scripts?15:11
sbezverk_for each host15:11
kbaegissbezverk_: That's awful15:11
kbaegiskludgy15:12
sbezverk_kbaegis you cannot, what you should do is to prepare custom cinder.conf before you install15:12
sbezverk_then installation will detect it and add merge it with generic cinder.conf15:13
kbaegissbezverk_: Sure. I get the custom config piece. I'm saying that this will break every deployment using lvm by default15:13
kbaegisAnd we have to use a kludge to fix15:13
kbaegisso for an AIO install, it would be /etc/kolla/conf/cinder/localhost/cinder.conf, correct?15:13
kbaegisotherwise it's the inventory name of the host in lieu of localhost?15:14
kbaegisJust verifying before I run reconfigure15:14
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872015:21
kbaegisYeah, setting that in cinder-volume doesn't work15:22
sdakesbezverk_ regresssion?15:24
sdakesbezverk morning btw15:25
*** iniazi has quit IRC15:25
sdakeoh pinning15:25
*** emccormick has joined #openstack-kolla15:26
kbaegisMorning sdake :)15:29
*** emccormick has quit IRC15:30
kbaegisSo it would be helpful if someone could explain which daemons are responsible for what here. If nova doesn't care about (and gets passed the iscsi target info from cinder), then the pertinent config is exclusively with cinder-volume, correct?15:31
kbaegisShould I need to touch iscsid or tgtd configs?15:31
kbaegisAm I misunderstanding anything there?15:32
kbaegisThe only thing cinder-volume's logs see is the connection created and then immediately terminated.15:34
*** iniazi has joined #openstack-kolla15:42
kbaegisAlright guys, here's the full config for cinder.conf: https://hastebin.com/jefocoloyu.ini15:42
kbaegisI've manually set all the available options for iscsi to the appropriate settings from the config guide15:43
kbaegisI can't seem to figure out how to read tgtd or iscsid logs, but they appear to see the attachment request15:44
kbaegisThe only openstack service that appears to complain is nova-compute, which returns "FailedISCSITargetPortalLogin: Could not login to any iSCSI portal" and a traceback15:45
kbaegiswhich should match tgtd: tgtd -d 1 -f --iscsi portal=192.168.254.3:326015:48
*** itlinux has joined #openstack-kolla16:09
sdakeinc0 kfox1111 gates are blocked -can this be reviewed please: https://review.openstack.org/#/c/468720/916:20
*** salv-orlando has joined #openstack-kolla16:32
inc0done sdake16:32
openstackgerritMerged openstack/kolla-kubernetes master: cell0's should be deleted only when cell0 is enabled  https://review.openstack.org/46872016:36
*** salv-orlando has quit IRC16:36
*** salv-orlando has joined #openstack-kolla16:45
*** jamesPR has joined #openstack-kolla16:46
sdakethanks inc017:14
*** kbaegis has quit IRC17:15
*** eaguilar has quit IRC17:19
*** salv-orlando has quit IRC17:25
*** japestinho has quit IRC17:29
*** mewald has quit IRC17:34
*** eaguilar has joined #openstack-kolla17:44
*** itlinux has quit IRC17:51
bmaltaisAnyone has experience with ldap domain authentication with kolla-ansible?18:07
bmaltaisI am almost there but I am running i a strange issue. I enabled "OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True" in horizon local file18:08
bmaltaisI applied the needed config to keystone.conf to enable multidomain support via config files in domains directory18:09
*** eaguilar has quit IRC18:09
bmaltaisI can see the ldap users listed in horizon if I manually go to the /identity/domains...18:09
bmaltaisThe strange thing is that the "Damains" horizon under Identity never show up. I see the domain field when I am at the login page of horizon...18:10
bmaltaisAnd any attempt to login via an ldap user result in "An error occurred authenticating. Please try again later."18:11
bmaltaisI also manually added a matching domain to openstack using "openstack domain create ducourrier"18:11
bmaltaisIf anyone use LDAP domain login in openstack let me know. I have tried mostly everything so far.18:12
*** Pavo has quit IRC18:12
*** Pavo has joined #openstack-kolla18:20
*** sambetts|afk has quit IRC18:23
*** sambetts_ has joined #openstack-kolla18:24
*** Pavo has quit IRC18:35
*** mgoddard has joined #openstack-kolla18:46
csuttleskbaegis: you can use the node ip addr attributes in ansible if you do it from the begging to generate the right iscsi parameters on install. You need to specify the ip so that tgtd knos where to present it from. When cinder-volume creates the vol, it gets exported by landing a conf in a dir watched by tgtd (for each vol), that needs to include the tgt ip addr. Nova uses iscsi to attach an18:52
csuttlesinitiator to said eported target. If your ip addr is missing, tgtd cannot export it, and nova initialtor cannot discover it, and you get the traceback you are seeing. Been down this exact same road, albeit on bare metal18:52
csuttlesI ran into this with ipv6 becuase I didn't use bracket notation (whoops). Invalid address gives the same error as undefined address, for the same reasons.18:55
csuttlesyou chouldn't need to touch iscsid or tgtd directly, just need to set up cinder.conf right18:56
csuttleshttps://docs.openstack.org/ocata/config-reference/block-storage/samples/cinder.conf.html18:58
csuttles# The IP address that the iSCSI daemon is listening on (string value)18:58
csuttles#iscsi_ip_address = $my_ip18:58
csuttlesneeds to be set in [DEFAULT]18:59
csuttlesfeel free to dm me - my reponse time is probably slow, but I'm happy to help you. This isn't really a kolla thing as much as it is a cinder config thing :)19:01
*** mgoddard has quit IRC19:09
*** jascott1 has joined #openstack-kolla19:17
*** jascott1 has quit IRC19:21
*** klindgren_ has quit IRC19:21
*** mgoddard has joined #openstack-kolla19:23
*** kbaegis has joined #openstack-kolla19:37
*** Pavo has joined #openstack-kolla19:43
*** jamesPR has quit IRC19:43
*** iniazi has quit IRC19:47
*** Pavo has quit IRC19:53
*** iniazi has joined #openstack-kolla19:53
*** Pavo has joined #openstack-kolla19:56
*** manheim has joined #openstack-kolla20:07
*** Pavo has quit IRC20:09
*** Mannheim has quit IRC20:21
*** salv-orlando has joined #openstack-kolla20:35
openstackgerritMerged openstack/kolla-kubernetes master: Conditionalize pv creation  https://review.openstack.org/46826420:51
*** mgoddard has quit IRC20:59
*** salv-orlando has quit IRC21:43
*** manheim has quit IRC22:14
*** powerd has joined #openstack-kolla22:21
openstackgerritMerged openstack/kolla-kubernetes master: Make sure pathfinder can find passwords.yml in etc/kolla  https://review.openstack.org/46727622:26
*** jamesPR has joined #openstack-kolla22:38
*** Pavo has joined #openstack-kolla22:45
*** Mech422 has quit IRC22:50
*** powerd has quit IRC22:57
jamesPRanyone?23:08
jamesPR:)23:08
jamesPR/usr/local/bin/kolla_extend_start: line 54: mysql_install_db: command not found23:09
jamesPRgot this error23:09
jamesPRany ideas?23:09
*** salv-orlando has joined #openstack-kolla23:14
*** salv-orlando has quit IRC23:19
*** masuberu has joined #openstack-kolla23:48
*** manheim has joined #openstack-kolla23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!