Thursday, 2019-07-04

*** Sravan has joined #openstack-kolla00:11
*** Sravan has quit IRC00:15
*** igordc has quit IRC00:49
*** cgrosjean has quit IRC00:49
*** schwicht has joined #openstack-kolla00:57
*** zhanglong has joined #openstack-kolla01:13
*** happyhemant has quit IRC01:46
*** unicell has joined #openstack-kolla01:49
*** schwicht has quit IRC01:49
*** lei-zh has joined #openstack-kolla01:54
*** BjoernT_ has joined #openstack-kolla01:59
*** BjoernT has quit IRC02:02
*** lei-zh has quit IRC02:05
*** lei-zh has joined #openstack-kolla02:06
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed  https://review.opendev.org/66877302:08
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed when enable_ceph  https://review.opendev.org/66877302:15
*** iclon_ has joined #openstack-kolla02:47
*** iclon has quit IRC02:50
*** BjoernT_ has quit IRC03:04
openstackgerritMerged openstack/kolla-ansible stable/stein: Use become for all docker tasks  https://review.opendev.org/66614003:30
openstackgerritMerged openstack/kolla-ansible master: Cloudkitty InfluxDB Storage backend via Kolla-ansible  https://review.opendev.org/65841803:45
*** lei-zh1 has joined #openstack-kolla03:58
*** lei-zh has quit IRC04:01
*** skramaja has joined #openstack-kolla04:07
*** Sravan has joined #openstack-kolla04:29
*** shyamb has joined #openstack-kolla04:33
*** whoami-rajat has joined #openstack-kolla04:55
*** altlogbot_0 has quit IRC04:57
*** altlogbot_3 has joined #openstack-kolla04:58
*** shyamb has quit IRC05:05
*** shyamb has joined #openstack-kolla05:05
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed when enable_ceph  https://review.opendev.org/66877305:16
*** Sravan has quit IRC05:21
*** Luzi has joined #openstack-kolla05:29
*** lei-zh1 has quit IRC05:32
*** iclon__ has joined #openstack-kolla05:46
*** iclon_ has quit IRC05:48
*** luksky11 has joined #openstack-kolla05:50
*** cah_link has joined #openstack-kolla06:13
*** factor has joined #openstack-kolla06:16
*** shyamb has quit IRC06:24
*** shyamb has joined #openstack-kolla06:35
yoctozeptomorning06:43
*** Luzi has quit IRC06:55
*** dciabrin has joined #openstack-kolla06:55
*** ivve has joined #openstack-kolla06:57
*** Luzi has joined #openstack-kolla06:57
openstackgerritwangwei proposed openstack/kolla-ansible master: Support mon and osd to be named with hostname  https://review.opendev.org/65441707:01
*** luksky11 has quit IRC07:02
*** jaicaa has quit IRC07:08
*** jaicaa has joined #openstack-kolla07:11
*** shyamb has quit IRC07:17
*** hamzaachi has joined #openstack-kolla07:22
*** rpittau|afk is now known as rpittau07:23
*** unicell has quit IRC07:25
*** zhanglong has quit IRC07:26
*** shyamb has joined #openstack-kolla07:27
*** zhanglong has joined #openstack-kolla07:27
*** pcaruana has joined #openstack-kolla07:28
*** zhanglong has quit IRC07:32
*** shyamb has quit IRC07:33
*** zhanglong has joined #openstack-kolla07:33
openstackgerritMerged openstack/kolla master: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66897107:39
openstackgerritMerged openstack/kolla-ansible master: CI: Keep stderr in ansible logs  https://review.opendev.org/66873907:45
*** heikkine has joined #openstack-kolla07:46
openstackgerritzhulingjie proposed openstack/kolla-cli master: Fixing test failures  https://review.opendev.org/65708607:47
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: CI: Keep stderr in ansible logs  https://review.opendev.org/66908607:48
mnasiadkamorning07:50
ohwhyosa[m]Morning!07:53
openstackgerritzhulingjie proposed openstack/kolla-cli master: Sync Sphinx requirement  https://review.opendev.org/57763607:53
*** luksky11 has joined #openstack-kolla07:54
*** happyhemant has joined #openstack-kolla08:16
mgoddardmorning08:18
*** shyamb has joined #openstack-kolla08:26
*** pcaruana has quit IRC08:28
*** zhanglong has quit IRC08:28
*** zhanglong has joined #openstack-kolla08:31
*** k_mouza has joined #openstack-kolla08:31
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Wait for all compute services before cell discovery  https://review.opendev.org/66862308:32
*** Wasaac has joined #openstack-kolla08:34
*** cgrosjean has joined #openstack-kolla08:34
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Fixes ceph-osd deploys with loop devices.  https://review.opendev.org/66829108:35
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Fixes ceph-osd deploys with loop devices.  https://review.opendev.org/66828608:35
yoctozeptohttp://zuul.openstack.org/status08:39
yoctozepto^ k, gating works - one queue08:39
yoctozeptoI will propose backports to the supported branches08:39
openstackgerritRadosław Piliszek proposed openstack/kolla stable/stein: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66910308:40
openstackgerritRadosław Piliszek proposed openstack/kolla stable/rocky: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66910408:41
openstackgerritRadosław Piliszek proposed openstack/kolla stable/queens: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66910508:41
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66910608:41
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66910808:41
yoctozeptohmm, queens has no gating08:45
*** k_mouza has quit IRC08:45
yoctozepto;D08:45
mgoddardyoctozepto: nope08:47
*** k_mouza has joined #openstack-kolla08:48
*** Wasaac has quit IRC08:53
openstackgerritMerged openstack/kolla-cli master: Fixing test failures  https://review.opendev.org/65708608:56
*** shyamb has quit IRC08:57
*** shyamb has joined #openstack-kolla08:57
*** Wasaac has joined #openstack-kolla08:59
yoctozeptowow, kolla-cli activity08:59
*** Wasaac has quit IRC09:00
*** Wasaac has joined #openstack-kolla09:02
*** pcaruana has joined #openstack-kolla09:04
mnasiadkayeah, saw some movement in there09:21
mnasiadkamgoddard: so did anyone volunteer to pick it up?09:21
mgoddardmnasiadka: nope09:21
mgoddardI asked oracle folks and they agreed to drop it09:21
mgoddardwe haven't made that change yet though, need to read how to do it09:21
*** shyamb has quit IRC09:24
mgoddardmnasiadka: weird ceph error during stein upgrade job: http://logs.openstack.org/14/668514/3/check/kolla-ansible-centos-source-upgrade-ceph-3/c527c65/primary/logs/ansible/upgrade09:33
mgoddardceph_argparse.JsonFormat: unknown type CephBool09:33
*** owalsh_ is now known as owalsh09:34
mnasiadkayeah, see that, hmm09:35
mnasiadkahttps://tracker.ceph.com/issues/3935509:36
yoctozeptoso we need to wait for all daemons to reregister with their new release09:37
yoctozeptogreat09:38
yoctozepto;D09:38
yoctozeptoalso "rc": 0,09:38
yoctozeptoaah, because of cut09:38
yoctozeptothough I thought bash pipeline should break in this case09:39
mnasiadkawe can force running the command on the first upgraded mon, with -m09:40
mnasiadkaand we can use some jinja filter instead of cut :)09:41
yoctozepto$ false | true09:42
yoctozepto$ echo $?09:42
yoctozepto009:42
yoctozeptoall those years in ignorance09:42
mnasiadkaignorance is a bliss09:42
yoctozeptoindeed, indeed09:43
yoctozepto> <mnasiadka> we can force running the command on the first upgraded mon, with -m09:45
yoctozeptoyup, we redirect to the first container but we don't force ceph command to use it09:45
yoctozeptogood catch09:46
mgoddardset -o pipefail09:47
yoctozeptomgoddard: felt like it was the default09:49
yoctozeptoanyway, you were right about stein being the most tested release09:49
yoctozeptoso many ceph upgrade jobs run ;D09:49
mgoddardthat would be far too sensible09:49
*** JangwonLee has quit IRC09:55
*** Wasaac has quit IRC09:56
*** Wasaac has joined #openstack-kolla09:56
*** shyamb has joined #openstack-kolla10:01
*** shyamb has quit IRC10:09
*** shyamb has joined #openstack-kolla10:09
*** Wasaac has quit IRC10:10
*** pcaruana has quit IRC10:12
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown  https://review.opendev.org/66736310:12
*** zhanglong has quit IRC10:16
*** skramaja has quit IRC10:16
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: DNM: Try importing stop.yml in handlers  https://review.opendev.org/66913110:17
openstackgerritMerged openstack/kolla-ansible stable/stein: Check for 'become' in tasks that use Docker in pep8  https://review.opendev.org/66614110:18
openstackgerritMerged openstack/kolla stable/pike: Move pike jobs to use validated RDO Trunk  https://review.opendev.org/66852210:18
openstackgerritMerged openstack/kolla-ansible master: Don't rotate keystone fernet keys during deploy  https://review.opendev.org/66688210:18
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Don't rotate keystone fernet keys during deploy  https://review.opendev.org/66913210:20
yoctozeptomgoddard: maybe we should test ironic whenever nova code is changed in kolla-ansible, hmm?10:22
mgoddardyoctozepto: makes sense10:22
*** Dubla has quit IRC10:22
yoctozeptomgoddard: ok, proposing10:22
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: Test ironic also when nova role is modified  https://review.opendev.org/66913610:26
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: Test ironic also when nova role is modified  https://review.opendev.org/66913610:26
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Pull images before upgrade  https://review.opendev.org/66688110:34
mgoddardmnasiadka: if you have a minute, https://review.opendev.org/#/c/647699/10:35
mnasiadkamgoddard: yeah, was just going through the latest changes in that10:35
mgoddardthanks10:36
*** Dubla has joined #openstack-kolla10:37
openstackgerritcaoyuan proposed openstack/kolla-cli master: Sync Sphinx requirement  https://review.opendev.org/57763610:37
mnasiadkafinished ;)10:37
openstackgerritcaoyuan proposed openstack/kolla-cli master: Update hacking version  https://review.opendev.org/62735110:37
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/queens: Restart all nova services after upgrade  https://review.opendev.org/66793710:37
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fixes for MariaDB recovery  https://review.opendev.org/66790410:46
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Test MariaDB operations in CI  https://review.opendev.org/65566310:46
*** Wasaac has joined #openstack-kolla10:48
*** Wasaac has quit IRC10:53
*** shyamb has quit IRC10:55
*** shyamb has joined #openstack-kolla10:56
yoctozeptomgoddard: seen that? something strange in rocky11:01
yoctozeptolots of issues unrelated to patch11:02
yoctozeptoand nova is failing on waiting for up11:02
openstackgerritMerged openstack/kolla-cli master: Update hacking version  https://review.opendev.org/62735111:10
*** Wasaac has joined #openstack-kolla11:21
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown  https://review.opendev.org/66736311:22
*** Wasaac has quit IRC11:26
*** Wasaac has joined #openstack-kolla11:31
*** shyamb has quit IRC11:31
mnasiadkaso guys - what's the approach with checking ceph versions and setting require-osd-ver? It would be best to run it after all MONs and OSDs are upgraded - now it's not the case11:34
mnasiadkamaybe we should just document steps that should be done after Ceph upgrade? if somebody chooses to upgrade half of Ceph MONs and OSDs - it's not the best idea to enforce higher Ceph version functionalities...11:36
openstackgerritMerged openstack/kolla-ansible stable/stein: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66910611:40
*** shyamb has joined #openstack-kolla11:42
openstackgerritMerged openstack/kolla-ansible stable/rocky: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66910811:44
*** cah_link has quit IRC11:46
yoctozeptomgoddard: replace still broken "\\1-ironic"11:48
yoctozeptocould it be that there is still too much escaping11:49
*** skramaja has joined #openstack-kolla11:49
yoctozepto{{ hosts | map('regex_replace', '^(.*)$', 'https://\\1') | list }}11:51
yoctozeptoit's an example from the ansible docs11:51
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Add ability to kolla-ansible to switch between fetchers backend  https://review.opendev.org/65960711:51
yoctozepto(groups['nova-compute-ironic'] |11:51
yoctozepto          intersect(ansible_play_batch) |11:51
yoctozepto          map('extract', hostvars, 'ansible_hostname') |11:51
yoctozepto          map('regex_replace', '^(.*)$', '\\1-ironic') |11:51
yoctozepto          list)11:51
yoctozepto^ this is what you're doing11:51
yoctozeptolooks legit11:52
yoctozeptobut nonetheless does not work11:52
*** cah_link has joined #openstack-kolla11:54
*** Wasaac has quit IRC11:56
openstackgerritMerged openstack/kolla stable/stein: Extract CI vars for use in kolla-ansible jobs  https://review.opendev.org/66881511:56
*** cah_link has quit IRC11:57
*** cah_link has joined #openstack-kolla11:57
openstackgerritcaoyuan proposed openstack/kolla-cli master: Sync Sphinx requirement  https://review.opendev.org/57763611:58
yoctozeptoin another place we have     supported_policy_files: "{{ supported_policy_format_list | map('regex_replace', '(.*)', '{{ project_name }}_\\1') | list }}"12:00
yoctozeptowhich works12:00
yoctozeptoso it seems as if doing '-' after number is problematic for him12:01
openstackgerritcaoyuan proposed openstack/kolla-cli master: change the description of the class  https://review.opendev.org/62188312:02
openstackgerritjacky06 proposed openstack/kolla-cli master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65467812:07
*** Wasaac has joined #openstack-kolla12:08
openstackgerritcaoyuan proposed openstack/kolla-cli master: Change openstack-dev to openstack-discuss  https://review.opendev.org/62195012:09
*** henriqueof has joined #openstack-kolla12:09
*** shyamb has quit IRC12:16
*** shyamb has joined #openstack-kolla12:22
openstackgerritjacky06 proposed openstack/kolla-cli master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65467812:32
*** schwicht has joined #openstack-kolla12:32
openstackgerritjacky06 proposed openstack/kolla-cli master: update the tox minversion to 2.0  https://review.opendev.org/62622512:34
*** shyamb has quit IRC12:37
yoctozepto> <yoctozepto> so it seems as if doing '-' after number is problematic for him12:40
yoctozeptojust tested - wrong12:40
yoctozeptobut I proved that in this particular context you have to do just \112:40
yoctozeptoseems like ansible bug12:40
openstackgerritMerged openstack/kolla-cli master: Change openstack-dev to openstack-discuss  https://review.opendev.org/62195012:40
yoctozeptothey have fixed the \\\\ case before 2.012:40
yoctozeptoand docs state it should always be \\112:41
yoctozeptoyet it is not12:41
yoctozeptoodd12:41
openstackgerritMerged openstack/kolla-cli master: change the description of the class  https://review.opendev.org/62188312:43
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Test ironic also when nova role is modified  https://review.opendev.org/66913612:46
mgoddardyoctozepto: just tested - you are right about \1. It's because we're using a yaml literal block scalar (>-)13:00
mgoddardjust did some testing13:00
*** Luzi has quit IRC13:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Wait for all compute services before cell discovery  https://review.opendev.org/66862313:03
openstackgerritMerged openstack/kolla-cli master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65467813:06
mgoddardyoctozepto: https://github.com/markgoddard/ansible-experiments/tree/master/03-regex-replace13:09
*** skramaja has quit IRC13:10
*** iclon__ has quit IRC13:13
*** iclon has joined #openstack-kolla13:14
*** fxpester has joined #openstack-kolla13:16
fxpesterhi all13:16
fxpestergot a strange problem, deploy kolla queens, everything ok and running13:17
*** iclon has quit IRC13:20
*** iclon has joined #openstack-kolla13:20
*** whoami-rajat has quit IRC13:23
mgoddardfxpester: that doesn't sound like a problem :)13:25
mnasiadkamgoddard: seen my comment on ceph earlier? Need some guidance :)13:29
*** henriqueof has quit IRC13:33
mgoddardmnasiadka: missed that13:35
mgoddardwe run require-osd-release at the end of the upgrade, not sure what's wrong?13:35
mgoddardis the problem that we don't wait for them to initialise?13:36
fxpesterdo second kolla reconfigure or kolla deploy , it is doing containers restart - and some files are disappearing - for sure it is "paste" configs like keystone-paste.conf13:38
fxpesterno problems with kolla configs that deployed by dump_start13:39
fxpesterreproduced on redhat 7 with docker 17.09.1-ce and 18.09.613:40
mgoddardfxpester: do you have some error logs to share?13:42
fxpester2019-07-04 16:22:15.958 7 CRITICAL nova [-] Unhandled error: ConfigNotFound: Could not find config at /etc/nova/api-paste.ini13:45
mgoddardfxpester: does the container image contain that file?13:48
*** fxpester has quit IRC13:48
mgoddardwhich container is that?13:48
*** fxpester has joined #openstack-kolla13:48
fxpesteryes files present in images if I run `docker run -ti $image_ID bash` I can see them13:49
*** iclon has quit IRC13:50
fxpesterI guess it is docker problem, but surprised I can find any information about this in google13:50
*** iclon has joined #openstack-kolla13:50
mgoddardstrange13:53
mgoddardis it a permissions issue?13:53
mgoddardcentos queens nova-compute image has this:13:53
mgoddard-rw-r----- 1 root nova 2923 Mar 24 23:12 /etc/nova/api-paste.ini13:53
mgoddardso nova group should be able to read13:54
fxpestertrying to reproduce now, just did a fresh install so -13:56
fxpesterad9a14526614        kolla/centos-binary-nova-api:queens                    "dumb-init --singl..."   4 minutes ago        Up 4 minutes                            nova_api13:56
*** fxpester has quit IRC14:00
*** iclon has quit IRC14:01
*** fxpester has joined #openstack-kolla14:01
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown  https://review.opendev.org/66736314:02
mgoddard^ ready to go14:03
fxpesterso ok, I just did docker stop nova-api and docker start, now nova - 2019-07-04 17:01:15.407 7 ERROR nova ConfigNotFound: Could not find config at /etc/nova/api-paste.ini14:03
*** iclon has joined #openstack-kolla14:03
fxpesterad9a14526614        kolla/centos-binary-nova-api:queens                    "dumb-init --singl..."   11 minutes ago      Restarting (1) 55 seconds ago                       nova_api14:03
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Test MariaDB operations in CI  https://review.opendev.org/65566314:06
*** hamzaachi has quit IRC14:07
mgoddardfxpester: can you docker export nova_api and check the file?14:08
*** hamzaachi has joined #openstack-kolla14:09
*** schwicht has quit IRC14:10
fxpestermgoddard: nova-paste.ini is not present in tar archive14:11
mgoddardfxpester: :/ so where has it gone?14:11
fxpestertrying to debug this right now, it disapper with docker restart14:11
fxpester*docker container restart*14:12
mgoddardfxpester: try running the commands in a new container manually using 'docker run'14:13
fxpesterinside keystone container right now: -rw-r----- 1 root     keystone 2493 Nov  1  2018 keystone-paste.ini14:13
fxpesterright now doing keystone restart14:13
fxpesterdid docker start and file just disappear, no old keystone config files present only new generated by kolla14:14
fxpesterdrwx------ 2 keystone keystone   6 Jul  4 17:13 domains -rw------- 1 keystone keystone 645 Jul  4 17:13 keystone.conf14:15
*** luksky11 has quit IRC14:17
yoctozepto> <mgoddard> yoctozepto: https://github.com/markgoddard/ansible-experiments/tree/master/03-regex-replace14:18
yoctozeptoyeah, testing the same14:18
yoctozeptoeach day wiser ;-)14:18
*** fxpester has quit IRC14:19
openstackgerritMerged openstack/kolla-ansible stable/stein: Don't rotate keystone fernet keys during deploy  https://review.opendev.org/66913214:19
yoctozeptomgoddard, mnasiadka: if -m would work, then we should just use it, we already delegate to one container but do not point the ceph client to it exclusively14:19
yoctozeptoanother issue is the incomplete upgrade14:20
mgoddardyoctozepto: what can we do about the incomplete upgrade?14:22
yoctozeptomgoddard: we can check all versions as previously suggested, wait some retries to increase version, and bail out if they do not ;-)14:23
*** BjoernT has joined #openstack-kolla14:23
mgoddardsounds reasonable14:23
mgoddardis it a release blocker?14:23
yoctozeptoI doubt, but let mnasiadka confirm it14:25
yoctozeptomost ceph upgrades pass now happily14:25
mgoddardwe need to get this ceph support deprecated14:25
*** fxpester has joined #openstack-kolla14:26
yoctozeptowe do, it's against the kolla spirit14:26
mgoddardtoo late to depreate for stein? :)14:26
yoctozeptoI think we can declare it deprecated as long as we support it14:26
yoctozeptoin stein still14:26
yoctozeptobut not in train14:27
yoctozeptoand provide migration path14:27
yoctozeptoto external ceph + ceph-ansible14:27
*** BjoernT_ has joined #openstack-kolla14:27
yoctozeptoor similar14:27
mgoddardthat's the dream14:27
*** BjoernT has quit IRC14:27
openstackgerritChristian Berendt proposed openstack/kolla-ansible master: Add parameters to configure number of processes and threads of WSGI in the Horizon container  https://review.opendev.org/66919314:30
*** henriqueof has joined #openstack-kolla14:30
*** hamzaachi_ has joined #openstack-kolla14:32
*** hamzaachi has quit IRC14:32
*** fxpester has quit IRC14:33
*** goldyfruit has quit IRC14:42
*** goldyfruit has joined #openstack-kolla14:42
*** ivve has quit IRC14:51
*** luksky11 has joined #openstack-kolla14:59
openstackgerritChristian Berendt proposed openstack/kolla-ansible master: Prevent access to server-statu from Horizon  https://review.opendev.org/66920415:19
*** cah_link has quit IRC15:21
openstackgerritChristian Berendt proposed openstack/kolla-ansible master: Add parameters to configure number of processes and threads of horizon  https://review.opendev.org/66919315:24
openstackgerritChristian Berendt proposed openstack/kolla-ansible master: Prevent access to server-status from horizon  https://review.opendev.org/66920415:24
*** frawg12 has joined #openstack-kolla15:26
*** Wasaac has quit IRC15:29
*** Wasaac has joined #openstack-kolla15:30
*** priteau has joined #openstack-kolla15:30
*** frawg12 has quit IRC15:32
*** frawg12 has joined #openstack-kolla15:36
*** Dubla has quit IRC15:38
*** hamzaachi_ has quit IRC15:43
*** hamzaachi_ has joined #openstack-kolla15:45
*** whoami-rajat has joined #openstack-kolla15:47
*** BjoernT_ has quit IRC15:51
frawg12Hi, I'm looking for some support to fix my kolla-ansible testbed deployment15:55
frawg12I'm deploying a multinode setup over 3 VMs running on an OVH server15:55
frawg12so with bridges with fixed IP/Mac for public access15:55
frawg12I can't even access the dashboard/APIs from the public IP (but I can SSH to it to access the controler)15:56
frawg12the dashboard is accessible from the private network but can't get it to work from public15:56
frawg12and I'm trying without haproxy15:57
mgoddardfrawg12: perhaps you need to open up a port?15:58
frawg12it seems that on the control nodes iptables is not blocking anything15:59
mlozain stable/stein, credentials-keys directory of keystone container of my 3 controller nodes has different set of keys with throws an error 'Credential could not be decrypted. Please contact the administrator: InvalidToken'16:00
mlozaShould it be same across 3 controller nodes?16:00
mgoddardfrawg12: but what about in OVH? security groups?16:06
frawg12It's on a dedicate server, so nothing is blocked16:06
frawg12and I've deployed KVM/libvirt  on top of it16:07
frawg12and can access all ports from services running inside VMs16:07
yoctozeptofrawg12: inaccessible in what way?16:08
yoctozeptohorizon is a django app and it probably dislikes being addressed using unknown name (foreign ip address)16:08
frawg12wget or web browser access to port 80 on the public IP get no page16:08
yoctozeptoif it connects, and tells you it doesn't want to work, then it might be this16:08
yoctozeptobut is this no page or connection timeout/reject?16:09
frawg12nope16:09
frawg12wget:16:09
frawg12Connecting to 54.37.216.22:80... connected.16:09
frawg12HTTP request sent, awaiting response... No data received.16:09
frawg12Retrying.16:09
frawg12oops, put the real IP16:09
frawg12and that public IP leads me appropriately (with ssh) to the control node16:11
yoctozeptotry configuring ALLOWED_HOSTS in horizon local settings16:11
yoctozeptothough I get connection refused with this ip address16:12
yoctozeptocould be some ovh firewall16:12
yoctozeptossh answers16:12
yoctozeptohttp not16:12
frawg12yes no TLS for the time being16:12
frawg12(one problem at a time)16:12
frawg12https://docs.openstack.org/kolla-ansible/rocky/reference/horizon-guide.html16:12
yoctozeptoI did not say a thing about tls :D16:12
frawg12for ALLOWED_HOSTS?16:13
yoctozeptoyup but it seems the port is not listening16:13
yoctozeptoso it's not this16:13
frawg12the VM have multiple netowrk interfaces, one private/internal, one public on the OVH bridge, and another one on the OVH bridge for neutron16:14
*** rpittau is now known as rpittau|afk16:14
frawg12and looking at the domain names/IP in openstack endpoint list the public URL are correct16:14
yoctozeptocheck ip address horizon is listening at16:15
yoctozeptomaybe it's listening on the private one only?16:15
*** Sravan has joined #openstack-kolla16:17
frawg12not sure where to get that info16:17
frawg12for sure I can access it on the private, I'm logged on it now16:17
frawg12OK, I'm loooking inside the running horizon container to see what I can find16:21
*** Sravan has quit IRC16:21
*** henriqueof has quit IRC16:21
frawg12so /etc/httpd/conf.d/horizon.conf16:22
frawg12contains only a Listen on the private IP16:22
ohwhyosa[m]and on local_settings / local_settings.py?16:28
*** Wasaac has quit IRC16:28
ohwhyosa[m]If you are using kolla I believe it's exposed on /etc/kolla/horizon/local_settings16:29
*** Wasaac has joined #openstack-kolla16:30
ohwhyosa[m]Which reminds me I have to check again if kolla-ansible reconfigure on rocky with centos images responds to changes on settings.py16:30
ohwhyosa[m]local_settings, sorry16:30
ohwhyosa[m]Hey people, how would you go about adding a service to an already deployed openstack kolla-ansible?16:31
frawg12OPENSTACK_HOST and OPENSTACK_KEYSTONE_URL point to the private IP16:32
ohwhyosa[m]Just add the service on globals and deploy again or something else?16:32
ohwhyosa[m]And is there any ALLOWED_HOSTS?16:35
frawg12yes: ALLOWED_HOSTS = ['*']16:36
ohwhyosa[m]lemme check for a sec16:36
ohwhyosa[m]Yep, look, this is an example file16:36
ohwhyosa[m]https://github.com/openstack/horizon/blob/master/openstack_dashboard/local/local_settings.py.example16:36
ohwhyosa[m]lines 36-4016:36
ohwhyosa[m]hmmm16:36
frawg12yes, my conf seems OK (at least for this line)16:37
ohwhyosa[m]okay, try16:38
ohwhyosa[m]ss -uptan | grep 8016:38
ohwhyosa[m]no 443, right?16:38
ohwhyosa[m]I mean, you have it listening with http, no tls enabled, frawg12 ?16:39
frawg12tcp    LISTEN     0      128    192.168.100.22:80                    *:*                   users:(("httpd",pid=7,fd=3))16:39
frawg12yes no TLS / 44316:39
ohwhyosa[m]Hmmm there you have it, the apache httpd process is binding only on the private ip16:40
ohwhyosa[m]could you pastebin your httpd.conf?16:40
frawg12sure16:41
frawg12https://pastebin.com/b6Ee5gW516:43
frawg12/etc/httpd/conf.d/horizon.conf16:43
frawg12again only the private IP16:43
*** unicell has joined #openstack-kolla16:44
ohwhyosa[m]Do you want to open it to the world?16:44
ohwhyosa[m]You could Listen on 0.0.0.0:80 (though it may be too much)16:44
ohwhyosa[m]and restart httpd16:44
ohwhyosa[m]?16:44
ohwhyosa[m]could you paste also the globals.yml?16:44
frawg12it looks a bit hackish to do this kind of thing, would think that ansible-kolla should provide required things16:45
ohwhyosa[m]pastebin, I mean, be careful or you can get banned on IRC16:45
frawg12I think I'm just doing something wrong with the network conf16:45
frawg12sure, will post/clean it16:45
ohwhyosa[m]I do have a fun story regarding ASCII Art, Unicorns and an office wide ban16:45
ohwhyosa[m]Cool16:45
frawg12https://pastebin.com/eSdAaBAt16:49
frawg12nothing very fancy, I tried to use priv + pub networks, and haproxy disabled (as I was having troubles with the VIP address stuff)16:51
*** k_mouza_ has joined #openstack-kolla16:55
ohwhyosa[m]is it an AIO?16:55
ohwhyosa[m]Yup, right?16:56
frawg12no, I'm having 3 nodes: control, compute and storage16:56
frawg12(3 different VMs)16:56
frawg12(and a foruth deploy host)16:57
*** k_mouza has quit IRC16:59
frawg12https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/horizon/templates/horizon.conf.j2#L216:59
*** k_mouza_ has quit IRC17:00
frawg12si it seems that this would always use the api_interface_address, that is configured to be the private one17:00
ohwhyosa[m]yup, probably it is the haproxy the one that balances that17:02
frawg12as said I'm not able to get haproxy conf to pass17:02
ohwhyosa[m]I had understood that disabling haproxy and using the host network was only for AIO deployments?17:02
ohwhyosa[m]don't remember where I read that17:02
frawg12not clear in fact17:03
frawg12but my problem is that with haproxy I'm not sure how it should operate17:03
ohwhyosa[m]gimme a sec17:03
ohwhyosa[m]Could you meanwhile ss -uptan | grep your_pub_ip17:03
ohwhyosa[m]?17:03
frawg12nothing from inside the horizon container17:04
ohwhyosa[m]from the external host17:05
ohwhyosa[m]meaning17:05
ohwhyosa[m]the host itself17:05
frawg12for haproxy I need to have kolla_external_vip_interface be a public IP, and for this to work I need to have a predefined mac17:05
ohwhyosa[m]The comment I mean, on the pastebin you shared, is right above kolla_internal_vip address17:05
yoctozeptoapi_interface should be public one because it is assumed external clients will be using it17:07
yoctozeptootherwise it is for private use17:07
frawg12nothing from the host17:07
yoctozeptoand can be accessed via vpn17:07
yoctozeptoif you set api_interface to the internal one, then only on the internal network will the services work17:07
frawg12so with this it may work without haproxy17:08
frawg12so either haproxy + private and pub IPs, or no haproxy but only pub IP17:08
yoctozeptohaproxy can be omitted when there is only one controller17:08
yoctozeptono, it's orthogonal to public/private17:08
*** cgrosjean has quit IRC17:09
ohwhyosa[m]<yoctozepto "haproxy can be omitted when ther"> Makes sense17:10
ohwhyosa[m]Anycase, the horizon dashboard should be accessible from the outside even if the rest of the apis aren't, right yoctozepto ?17:11
frawg12ok so only setting api_interface = eth2 # (my public interface)17:11
frawg12may be enough?17:11
ohwhyosa[m]well, apis17:11
ohwhyosa[m]Exposing all the api endpoints seems a bit much, depending on the use case (like, I may want my "customers" to be able to reach their horizon dashboard and stuff like console, but never to be able to reach my api endpoints and control/operate the cloud itself)17:14
ohwhyosa[m]I'm quite propably 80% wrong at least, though17:14
*** icarusfactor has joined #openstack-kolla17:16
*** unicell has quit IRC17:16
*** henriqueof has joined #openstack-kolla17:18
*** factor has quit IRC17:19
*** k_mouza has joined #openstack-kolla17:19
*** icarusfactor has quit IRC17:21
*** frawg12 has quit IRC17:24
*** frawg12 has joined #openstack-kolla17:24
*** k_mouza has quit IRC17:24
frawg12was disconnected sorry, not sur if I missed something17:24
mnasiadkamgoddard: Currently we do ceph release check on each host in serial mode, so after host1 ceph upgrade, we run it. It would make sense to run it only after all hosts are upgraded, or do something like „kolla-ansible ceph-post-upgrade”, run some checks, do what needs to be done and output some report.17:24
yoctozeptoohwhyosa[m]: control/operate is already controlled by authorization mechanisms, no need to add complexity :-)17:28
frawg12ohwhyosa[m]: I will make some more test with what you said and also try to get haproxy work with the network conf I try to use17:29
frawg12ohwhyosa[m]: thanks!17:29
*** luksky11 has quit IRC17:33
*** ivve has joined #openstack-kolla17:36
mgoddardmnasiadka: could you point me to that serial release check?17:39
ohwhyosa[m]frawg12: Nah, thank the rest of the people around here, just trying to pay forward!17:40
mnasiadkamgoddard: bottom of upgrade.yml, the one that yoctozepto fixed, it’s going serial (which makes sense in ceph upgrade) - but not the require-osd version stuff :)17:40
ohwhyosa[m]yoctozepto:  Makes sense, though there's something about having them out in the open that makes the lil sysadmin in my shake in terror17:41
ohwhyosa[m]in me*17:41
mgoddardmnasiadka: I don't see how it's serial. It has run_once and delegate_to17:41
ohwhyosa[m]Could always firewall it and restrict ips, though17:41
yoctozeptomnasiadka, mgoddard: me neither, where should I look?17:42
yoctozeptoohwhyosa[m]: sure, layers of security usually help17:42
mnasiadkamgoddard: still it somehow runs after only first host upgrade, not all of them - or I misread ara logs :)17:42
yoctozeptoor at least make us feel more secure17:42
mgoddardmnasiadka: hmm, perhaps there was an earlier failure that prevented all hosts from executing?17:43
mnasiadkamgoddard: might be, will look into that tomorrow17:44
mgoddardmnasiadka: looked at the logs again, that is what happened. Mariadb failed earlier17:46
mgoddardwe need any_errors_fatal (if only it worked)17:46
henriqueofIs kolla-ansibe based ironic deployment fully funciontal?17:47
henriqueofI've been trying for more than amont to boot a single server but it won't work.17:47
mgoddardhenriqueof: we use ironic with kolla-ansible a lot17:50
mgoddardhenriqueof: we use centos/binary, and centos/source should be fine. Haven't tried with ubuntu17:51
henriqueofmgoddard: I've been trying for more than a month ot boot a remote server using IPMI but it doesn't work.17:51
mgoddardyoctozepto: mnasiadka: what would you think about a last minute deprecation of ceph deployment in Stein?17:51
mgoddardhenriqueof: yeah it's not trivial17:52
henriqueofnovcompute-ironic says "because one or more parameters are missing from its instance_info and insufficent information is present to boot"17:52
mgoddardhenriqueof: which OS/type?17:52
mgoddardhenriqueof: does it say which parameters?17:52
henriqueofI am using centos images.17:53
mgoddardhenriqueof: ironic conductor logs are often more helpful than nova logs17:53
henriqueofmgoddard: Yep but the params are set like the docs except image_source.17:53
henriqueofIronic conductor logs are way less helpful, there's only one line when it starts and one when it fails, no aditiuonal info.17:54
henriqueofI think I am going crazy with this. hahah17:55
mgoddardhenriqueof: try enabling debug?17:55
mgoddardhenriqueof: if you want an example of a working setup, try following https://kayobe.readthedocs.io/en/latest/development/automated.html#overcloud17:57
mgoddardhenriqueof: you can use the tenks project to create some 'fake bare metal VMs'17:57
mgoddardhenriqueof: kayobe uses kolla-ansible underneath, so the config can be easily transferred17:58
yoctozeptomgoddard: already expressed my posture - deprecate now, remove right in train but support in stein at fullest so we need to fix it anyway :P17:58
henriqueofmgoddard: Will take a look, thanks!17:58
yoctozepto(well, unless nobody cares)17:58
mgoddardyoctozepto: sorry, forgot. I figure if we know we're going to remove one day we might as well make it known as early as possible, even if we don't do it any sooner17:59
*** priteau has quit IRC17:59
yoctozeptomgoddard: you read my mind18:00
*** Wasaac has quit IRC18:06
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Deprecate Ceph deployment  https://review.opendev.org/66921418:06
mgoddardyoctozepto: ^18:07
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Pull images before upgrade  https://review.opendev.org/66688118:11
yoctozeptomgoddard: seen18:16
*** hamzaachi_ has quit IRC18:16
yoctozeptoI would add that ceph, or any storage cluster, deployment is orthogonal to openstack18:18
mnasiadkamgoddard: deprecation ... I don’t know, let me test some ceph scenarios and let’s see :)18:19
mgoddardok, we don't need to rush into it18:19
*** KeithMnemonic has quit IRC18:30
kplantmgoddard: not that my opinion carries weight here but i don't think it's irresponsible throwing out the fyi notices asap18:32
kplanta new-to-kolla-ansible user may see the notice and decide to choose a different deployment method for ceph and save themselves from being forced off of kolla deployed ceph18:33
yoctozeptokplant: that's the point18:34
kplant:-)18:35
mgoddardkplant: of course your opinion carries weight!18:40
*** Wasaac has joined #openstack-kolla18:43
*** whoami-rajat has quit IRC18:47
*** Wasaac has quit IRC18:48
*** whoami-rajat has joined #openstack-kolla19:07
*** hamzaachi has joined #openstack-kolla19:36
*** frawg12 has quit IRC19:40
*** EmilienM is now known as EvilienM19:54
*** luksky11 has joined #openstack-kolla19:55
*** EvilienM is now known as EmilienM19:56
*** dciabrin has quit IRC20:09
*** Wasaac has joined #openstack-kolla20:43
*** ivve has quit IRC20:47
*** Wasaac has quit IRC20:48
henriqueofmgoddard: still no luck with Ironic.20:52
henriqueofI have an IPMI interface connected to the openstack managemente network and a pxe interface connected to the providser netowrk, cleaning works but provisioning don't.20:52
henriqueofWhat can be the problem?20:52
*** Wasaac has joined #openstack-kolla21:04
*** Wasaac has quit IRC21:10
*** whoami-rajat has quit IRC21:27
openstackgerritMerged openstack/kolla-ansible master: Simplify handler conditionals  https://review.opendev.org/64769921:34
*** cgrosjean has joined #openstack-kolla21:35
*** dciabrin has joined #openstack-kolla21:39
*** henriqueof has quit IRC21:45
*** hamzaachi has quit IRC21:48
*** cgrosjean has quit IRC21:56
*** Dubla has joined #openstack-kolla22:03
*** Wasaac has joined #openstack-kolla22:10
*** hamzaachi has joined #openstack-kolla22:10
*** Wasaac has quit IRC22:14
*** luksky11 has quit IRC22:43
*** k_mouza has joined #openstack-kolla22:58
*** k_mouza has quit IRC23:02
*** k_mouza has joined #openstack-kolla23:09
*** hamzaachi has quit IRC23:11
*** jbadiapa has quit IRC23:31

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!