Monday, 2020-03-02

« Sunday, 2020-03-01 Index Tuesday, 2020-03-03 »
*** cfontain has joined #tripleo00:04
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_expand_roles  https://review.opendev.org/70641700:14
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_check_existing  https://review.opendev.org/70704300:14
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_populate_environment  https://review.opendev.org/70729500:14
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module metalsmith_instances  https://review.opendev.org/70641800:14
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: Playbooks for overcloud node (un)provision  https://review.opendev.org/70704400:14
*** dasp has joined #tripleo00:16
*** dasp_ has quit IRC00:18
*** dsneddon has quit IRC00:48
*** jmasud has joined #tripleo00:51
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Add c8 multinode containers job  https://review.opendev.org/71017900:56
openstackgerritMerged openstack/tripleo-upgrade master: Allow to switch from parallel to serial update.  https://review.opendev.org/70293101:14
*** dsneddon has joined #tripleo01:16
*** dsneddon has quit IRC01:20
*** tkajinam has quit IRC01:24
*** tkajinam has joined #tripleo01:25
*** dsneddon has joined #tripleo01:39
openstackgerritMerged openstack/tripleo-upgrade stable/train: Add config-heat and config-resource options.  https://review.opendev.org/70997502:00
*** ramishra has joined #tripleo02:23
tkajinamI'm afraid centos-8-standalone is broken now :-( https://b711d185688da3b864bc-5593d50c131879f6a486eeedbad80e3c.ssl.cf2.rackcdn.com/710643/1/check/tripleo-ci-centos-8-standalone/91eca12/logs/undercloud/home/zuul/standalone_deploy.log02:34
*** jcoufal has joined #tripleo02:43
*** jcoufal has quit IRC02:48
*** psachin has joined #tripleo03:34
*** ykarel|away is now known as ykarel03:55
*** raukadah is now known as chandankumar03:59
*** udesale has joined #tripleo04:04
*** dmacpher has quit IRC04:04
*** dmacpher has joined #tripleo04:07
*** jmasud has quit IRC04:13
*** jmasud has joined #tripleo04:15
*** ysandeep|away is now known as ysandeep04:19
*** numans has joined #tripleo04:43
*** skramaja has joined #tripleo04:53
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ansible master: Fixed the condition for gather_domains  https://review.opendev.org/70995904:56
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Add VolumesSnapshotTestJSON test_volume_from_snapshot train skiplist  https://review.opendev.org/70972505:03
*** saneax has joined #tripleo05:23
openstackgerritMichael Davies proposed openstack/python-tripleoclient stable/train: Refactor image upload command  https://review.opendev.org/71069805:26
*** jmasud_ has joined #tripleo05:32
*** jmasud has quit IRC05:33
*** evrardjp has quit IRC05:35
*** evrardjp has joined #tripleo05:35
openstackgerritMerged openstack/tripleo-ci master: Save disk usage for containers jobs  https://review.opendev.org/71031105:50
*** ratailor has joined #tripleo06:03
*** threestrands has joined #tripleo06:04
openstackgerritMichael Davies proposed openstack/python-tripleoclient stable/train: Local file based image uploader  https://review.opendev.org/71070206:13
*** threestrands has quit IRC06:16
*** pcaruana has joined #tripleo06:16
*** sri_ has joined #tripleo06:22
openstackgerrityatin proposed openstack/ansible-role-collect-logs master: Fix container logs collections  https://review.opendev.org/71070406:24
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/stein: Use config for layers  https://review.opendev.org/71070506:27
*** lbragstad has joined #tripleo06:28
*** lbragstad_ has quit IRC06:29
*** lmiccini has joined #tripleo06:35
*** bandini has joined #tripleo06:36
*** lmiccini has quit IRC06:36
*** lmiccini has joined #tripleo06:38
*** mrunge_ is now known as mrunge06:38
*** dpawlik has joined #tripleo06:46
*** yolanda has joined #tripleo06:47
*** rcernin has quit IRC06:47
*** pcaruana has quit IRC06:49
*** jfrancoa has joined #tripleo06:52
*** yprokule has joined #tripleo06:54
*** dpawlik has quit IRC06:57
chandankumarykarel, https://review.opendev.org/#/c/709959/ take a look at this patch07:00
*** dpawlik has joined #tripleo07:08
ykarelchandankumar, is the job started passing with ansible 2.9?07:10
ykarellast i checked it used to fail with 2.907:11
ykareland commented same that time07:11
*** cylopez has joined #tripleo07:14
*** saneax has quit IRC07:17
chandankumarykarel, I rebased it, now It might use ansible 2.807:20
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: [WIP] Add CephBasePoolVars and CephKeyVars structures  https://review.opendev.org/70929707:20
ykarelchandankumar, yes it will use ansible-2.8, but still it will fail with ansible-2.9. remember the 'str' error we seen with ansible-2.9?07:21
chandankumarykarel, yes, let me see where it is coming07:22
*** fmount has joined #tripleo07:26
*** iurygregory has joined #tripleo07:28
openstackgerritHarald Jensås proposed openstack/tripleo-ansible master: [DNM] Only add host entries for role_networks  https://review.opendev.org/71057407:31
*** dciabrin has joined #tripleo07:31
*** ykarel is now known as ykarel|lunch07:38
*** lbragstad_ has joined #tripleo07:40
*** lbragstad has quit IRC07:42
*** pcaruana has joined #tripleo07:42
*** dsneddon has quit IRC07:43
openstackgerritHarald Jensås proposed openstack/tripleo-ansible master: [DNM] Only add host entries for role_networks  https://review.opendev.org/71057407:44
*** chem has quit IRC07:50
*** slaweq has joined #tripleo07:54
*** tesseract has joined #tripleo07:59
*** pcaruana has quit IRC08:04
*** chem has joined #tripleo08:04
*** pcaruana has joined #tripleo08:07
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Add config-heat and config-resource options.  https://review.opendev.org/71070908:10
openstackgerritMerged openstack/tripleo-common master: Move parameter actions functionality to utils - Part 1/2  https://review.opendev.org/70954608:11
openstackgerritMerged openstack/tripleo-common master: Move generate_fencing_parameters functionality to utils  https://review.opendev.org/71005508:11
*** cfontain has quit IRC08:11
openstackgerritMerged openstack/puppet-pacemaker master: Add force parameter for all pcmk_ types  https://review.opendev.org/70873708:12
*** ccamacho has joined #tripleo08:13
*** dsneddon has joined #tripleo08:14
*** jaosorior has joined #tripleo08:16
*** jcapitao has joined #tripleo08:17
*** rpittau|afk is now known as rpittau08:18
*** tosky has joined #tripleo08:23
*** dsneddon has quit IRC08:23
*** jpena|off is now known as jpena08:23
*** saneax has joined #tripleo08:23
*** tkajinam has quit IRC08:24
*** pcaruana has quit IRC08:30
*** lee1 has joined #tripleo08:32
*** lee1 is now known as lyarwood08:33
*** cylopez has quit IRC08:36
*** cylopez has joined #tripleo08:37
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859908:41
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/train: Multi-cells update support.  https://review.opendev.org/70786908:43
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/train: Allow to switch from parallel to serial update.  https://review.opendev.org/71058808:43
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/train: Iterate over only if variable is defined.  https://review.opendev.org/71071308:43
*** suuuper has joined #tripleo08:47
openstackgerritDaniel Bengtsson proposed openstack/python-tripleoclient master: Stop configuring install_command in tox.  https://review.opendev.org/69350208:48
*** ykarel|lunch is now known as ykarel08:50
*** gfidente has joined #tripleo08:51
*** dsneddon has joined #tripleo08:51
*** lucasagomes has joined #tripleo08:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Multi-cells update support.  https://review.opendev.org/71071508:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Iterate over only if variable is defined.  https://review.opendev.org/71071608:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Allow to switch from parallel to serial update.  https://review.opendev.org/71071708:53
*** jmasud_ has quit IRC08:55
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Ensure bc package is installed for l3 ping tests.  https://review.opendev.org/71071808:55
*** dsneddon has quit IRC08:56
*** jpich has joined #tripleo08:59
*** jcapitao has quit IRC09:03
*** jcapitao has joined #tripleo09:04
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Iterate over only if variable is defined.  https://review.opendev.org/71071609:04
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Allow to switch from parallel to serial update.  https://review.opendev.org/71071709:04
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Allow to switch from parallel to serial update.  https://review.opendev.org/71071709:06
*** paramite|clone has joined #tripleo09:07
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/rocky: Allow to switch from parallel to serial update.  https://review.opendev.org/71072409:09
*** jmasud has joined #tripleo09:11
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Remove Mistral from the plan export  https://review.opendev.org/70781209:11
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Relax ~/.config permissions  https://review.opendev.org/71016709:12
*** jbadiapa has joined #tripleo09:13
*** bogdando has joined #tripleo09:14
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/rocky: Allow to switch from parallel to serial update.  https://review.opendev.org/71072409:16
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: Allow to switch from parallel to serial update.  https://review.opendev.org/71072509:18
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add CephBasePoolVars and CephKeyVars structures  https://review.opendev.org/70929709:20
*** dmellado has quit IRC09:23
*** dmellado has joined #tripleo09:26
openstackgerrityatin proposed openstack/tripleo-quickstart-extras master: Remove extra comma when add_repos_update is unset  https://review.opendev.org/71072709:28
*** dsneddon has joined #tripleo09:28
*** apetrich has joined #tripleo09:30
*** dsneddon has quit IRC09:33
*** dmellado has quit IRC09:33
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_mgr service  https://review.opendev.org/70284609:34
*** dmellado has joined #tripleo09:35
*** jaosorior has quit IRC09:35
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove package_update workbook  https://review.opendev.org/71072809:38
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_mgr service  https://review.opendev.org/70284609:39
*** tkajinam has joined #tripleo09:39
*** derekh has joined #tripleo09:43
*** waleedm has joined #tripleo09:46
*** alexmcleod has joined #tripleo09:53
*** sshnaidm is now known as sshnaidm|pto09:59
*** pkopec has joined #tripleo10:03
openstackgerritGabriele Cerami proposed openstack/tripleo-ci master: Add centos8 support for emit_releases_script  https://review.opendev.org/71044110:05
*** dsneddon has joined #tripleo10:06
*** dtantsur|afk is now known as dtantsur10:08
*** dsneddon has quit IRC10:11
*** leanderthal has joined #tripleo10:12
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859910:17
*** tkajinam has quit IRC10:18
*** kozhukalov has quit IRC10:21
*** kozhukalov has joined #tripleo10:22
*** social has joined #tripleo10:22
*** sshnaidm|pto has quit IRC10:26
*** pcaruana has joined #tripleo10:28
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Restart ovn-dbs resource to take new VIP property.  https://review.opendev.org/71052710:35
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859910:35
ykarelchandankumar, i was referring https://bugs.launchpad.net/tripleo/+bug/1865459 issue with container updates10:36
openstackLaunchpad bug 1865459 in tripleo "[CentOS8]tripleo modify images fail when gating-repo is not present" [Undecided,New]10:36
openstackgerritArx Cruz proposed openstack/ansible-role-collect-logs master: Alter readme path for collect logs  https://review.opendev.org/71073710:41
*** suuuper has quit IRC10:42
*** sshnaidm has joined #tripleo10:42
chandankumarykarel, aye, will take a look later10:42
*** dsneddon has joined #tripleo10:43
openstackgerritGabriele Cerami proposed openstack/tripleo-ci master: Add centos8 support for emit_releases_script  https://review.opendev.org/71044110:44
*** dsneddon has quit IRC10:47
openstackgerrityatin proposed openstack/ansible-role-tripleo-modify-image master: Remove unavailable repos from update  https://review.opendev.org/71073810:54
ykarelchandankumar, trying to fix container updates with ^^10:54
chandankumarykarel, ack, adding to my list10:54
*** udesale has quit IRC10:57
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: DNM - Testing worker  https://review.opendev.org/70943811:02
openstackgerritHarald Jensås proposed openstack/tripleo-ansible master: [DNM] Only add host entries for role_networks  https://review.opendev.org/71057411:03
*** abdysn has joined #tripleo11:05
abdysnhi all, could you push to merge this patch https://review.opendev.org/#/c/703953/. thanks11:06
*** mschuppert has joined #tripleo11:07
*** sshnaidm_ has joined #tripleo11:12
*** sshnaidm has quit IRC11:12
*** sshnaidm_ is now known as sshnaidm|pto11:15
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient stable/train: Replace ansible shell with python runner  https://review.opendev.org/71074211:18
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient stable/train: Overload stdout callback plugin  https://review.opendev.org/71074311:18
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient stable/train: Whitelist the profile_tasks callback by default  https://review.opendev.org/71074411:18
*** dsneddon has joined #tripleo11:21
*** sshnaidm|pto has quit IRC11:23
*** sshnaidm has joined #tripleo11:23
*** sshnaidm is now known as sshnaidm|pto11:24
*** dsneddon has quit IRC11:26
ratailormwhahaha, re your comment on https://review.opendev.org/#/c/709112/ where should it be, if not in tripleo-ansible ?11:29
openstackgerritFrancesco Pantano proposed openstack/puppet-tripleo master: [WIP] Add Certmonger ceph_rgw class to config tls  https://review.opendev.org/70374311:31
*** rpittau is now known as rpittau|bbl11:32
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-heat-templates master: [OVN] SRIOV with native OVN DHCP server  https://review.opendev.org/71043711:41
*** ntt has joined #tripleo11:43
nttHi, I'm trying to deploy a test environment using tripleo but I do not understand how undercloud should be connected to overcloud when using IPMI. I mean, I have a dedicated ipmi port for all overcloud nodes. Should I configure a NON IPMI port on the undercloud using the same IPMI subnet?11:45
ramishraEmilienM: Hi, https://bugs.launchpad.net/tripleo/+bug/1865473. Should we increase the number of retries?11:45
openstackLaunchpad bug 1865473 in tripleo "tripleo-ci-centos-8-standalone job failing with container create aync task" [Undecided,New]11:45
EmilienMramishra: mhh we need to find the root cause rather than increasing retry11:46
*** jcapitao is now known as jcapitao|lunch11:46
EmilienMwhy did it work fine before on centos7 and rhel811:46
ramishrathe db migration is still running11:46
ramishraand there are no errors as I can see11:46
openstackgerritFrancesco Pantano proposed openstack/puppet-tripleo master: [WIP] Add Certmonger ceph_rgw class to config tls  https://review.opendev.org/70374311:46
EmilienM7 min to run a db migration is not enough?11:46
ramishraEmilienM: https://6c87970bcd1576df1a31-6b2fbb83e342ba16a98bd56da886e807.ssl.cf1.rackcdn.com/710728/1/check/tripleo-ci-centos-8-standalone/f27a279/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log11:46
*** tkajinam has joined #tripleo11:47
EmilienMyeah I looked at that log11:47
EmilienMhow do you know this is not finished and it blocks somewhere else?11:47
ramishraLooks like it's doing the 30 retries in 3 mins or so11:47
EmilienMit seems ti me it's running by alphabetic order and vmware-nsx is the last one11:47
EmilienMhttps://6c87970bcd1576df1a31-6b2fbb83e342ba16a98bd56da886e807.ssl.cf1.rackcdn.com/710728/1/check/tripleo-ci-centos-8-standalone/f27a279/logs/undercloud/var/log/extra/podman/containers/neutron_db_sync/podman_info.log11:48
EmilienMindeed for some reason the container is not finished yet11:49
EmilienMI need to step away for 10 min, br11:49
ramishraEmilienM: do you see any other errors.. I can see the container still running11:49
EmilienMbrb11:49
ramishrahttps://6c87970bcd1576df1a31-6b2fbb83e342ba16a98bd56da886e807.ssl.cf1.rackcdn.com/710728/1/check/tripleo-ci-centos-8-standalone/f27a279/logs/undercloud/var/log/extra/podman/podman_allinfo.log11:49
ramishrayep11:49
EmilienMno i see no error either11:49
EmilienMbut still i think it's weird to need more than 7 min to upgrade a neutron database11:50
EmilienMbeagles: ^ who can we ask?11:50
*** paramite has joined #tripleo11:50
EmilienMramishra: we can increase retries in the meantime but this isn't the long term solution as you can imagine11:50
EmilienMbrb11:50
ramishraEmilienM: agreed11:50
*** paramite|clone has quit IRC11:52
slaweqEmilienM: hi11:54
slaweqEmilienM: I assume that this neutron db upgrade is on fresh db, without any data inside, right?11:54
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Skeleton role, get with the decade  https://review.opendev.org/70741411:55
*** dsneddon has joined #tripleo11:55
ramishraslaweq: yeah11:56
slaweqramishra: can we somehow get logs from this db upgrade process?11:56
ramishraslaweq: sorry, I don't think it's upgrade, it's a fresh install11:57
slaweqramishra: yes, but even on fresh install You need to run neutron-db-manage script to do this alembic migration script and create all tables in db11:57
ramishraslaweq: https://6c87970bcd1576df1a31-6b2fbb83e342ba16a98bd56da886e807.ssl.cf1.rackcdn.com/710728/1/check/tripleo-ci-centos-8-standalone/f27a279/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log11:58
slaweqand looking at e.g. devstack job: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3fd/681258/8/check/neutron-tempest-plugin-api/3fd918f/job-output.txt it takes about 30 seconds in such case11:58
ramishrathat's the db_sync log and it takes 7 mins I guess11:58
slaweqramishra: is that persistent issue which You have every time?11:59
slaweqor randomly?11:59
ramishraslaweq: I've seen it few times with centos8 jobs today11:59
slaweqbecause I remember that we had such problems in functional tests jobs from time to time that db migration tests were failing due to such slow migration when node was overloaded11:59
slaweqand it looks similar here, there is no one specific step which took long time12:00
ramishraslaweq: it's a standalone job, I don't know why it should be overloaded or anything unless the nodes/vms are slow12:00
slaweqit's just that each step takes about 2-3 seconds12:00
slaweqramishra: it can be that node on which this vm is run is overloaded and there are some iops problems12:01
slaweqat least that was what we saw in case of such failures in our neutron-functional jobs12:01
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: [WIP] Optionally configure Ceph RGW listener with SSL  https://review.opendev.org/66629112:01
*** dsneddon has quit IRC12:02
ramishraslaweq: yep, possible. May be good to compare with some old centos7 jobs12:02
slaweqramishra: look at each step in "working neutron-tempest-api" job12:02
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: [WIP] Optionally configure Ceph RGW listener with SSL  https://review.opendev.org/66629112:02
slaweqstep "Running upgrade  -> kilo" took about 6 seconds12:02
slaweqand in centos8 job same step took more than 2 minutes12:03
slaweqand next one12:03
slaweq"kilo -> 354db87e3225"12:03
ramishraslaweq: it works on devstack:)12:03
slaweqin good job it took about 0.07 second12:03
slaweqand in centos8 about 0.7 second (this one wasn't very long here too)12:04
slaweqand so on12:04
ramishraslaweq: thanks!, we'll probably observe the job more and see if it's a slow node/hardware issue12:04
slaweqramishra: I'm just saying that we saw such issues from time to time in our neutron ci in functional tests12:05
slaweqand I know about those ones only because there migration was run in test so tests were failing due to such issue12:06
slaweqin other jobs, like tempest ones, it just takes longer time but will not fail so we are probably not aware of that even12:06
*** raildo has joined #tripleo12:07
ramishraslaweq: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b73/709094/2/gate/tripleo-ci-centos-7-standalone/b7322f5/logs/undercloud/home/zuul/standalone_deploy.log12:07
ramishraThat's for nova_db_sync on centos712:07
ramishraso it's possibly slow hardware or some podman thing.. Anyway I'll propose a patch to increase the retries which will possibly reduce the large number of failures happening atm12:09
slaweqramishra++ sounds good :)12:09
*** yolanda has quit IRC12:10
slaweqramishra: sorry that I can't do more about that but I already spent a lot of time trying to fix it in neutron-functional job and the only way to workaround it was to skip test in case of such timeout :/12:10
*** yolanda has joined #tripleo12:11
openstackgerritFrancesco Pantano proposed openstack/puppet-tripleo master: [WIP] Add Certmonger ceph_rgw class to config tls  https://review.opendev.org/70374312:11
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Increase number of retries for container create async task  https://review.opendev.org/71075012:11
ramishraslaweq: I understand12:11
ramishraslaweq: thanks for the help12:12
tkajinamhttps://b711d185688da3b864bc-5593d50c131879f6a486eeedbad80e3c.ssl.cf2.rackcdn.com/710643/1/check/tripleo-ci-centos-8-standalone/91eca12/logs/undercloud/home/zuul/standalone_deploy.log12:12
tkajinamseems that there exists another failure case where we have timeout in keystone endpoint creation12:12
tkajinamin centos-8 job12:12
ramishratkajinam: I increased the retries with https://review.opendev.org/71075012:13
ramishraHope it would help reduce the failures12:13
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859912:13
*** jcoufal has joined #tripleo12:14
tkajinamramishra, I see, but this failure should be fixed by another timeout... https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_keystone_resources/tasks/endpoints.yml#L42  I don't think we can reduce the possibility by raising timeout in container creation, but we might still hit other timeouts caused by slow database12:15
tkajinams/I don't think/I think/12:15
openstackgerritFrancesco Pantano proposed openstack/puppet-tripleo master: [WIP] Add Certmonger ceph_rgw class to config tls  https://review.opendev.org/70374312:16
ramishratkajinam: oh yeah! I did not see that failure ealier..12:17
ramishratkajinam: That's not podman related, so....12:18
ramishratkajinam: the assumption is if it's slow hardware, we can do very little other than recheck or try fix it with increasing timeouts12:19
ramishraI don't know any other way.. may be approach the infra team12:20
tkajinamramishra, yeah, agreed :-(12:21
*** kozhukalov has quit IRC12:23
*** kozhukalov has joined #tripleo12:24
openstackgerrityatin proposed openstack/ansible-role-tripleo-modify-image master: Remove unavailable repos from update  https://review.opendev.org/71073812:24
openstackgerritFrancesco Pantano proposed openstack/puppet-tripleo master: [WIP] Add Certmonger ceph_rgw class to config tls  https://review.opendev.org/70374312:26
*** ratailor has quit IRC12:28
*** ansmith has quit IRC12:32
*** dsneddon has joined #tripleo12:33
*** jpena is now known as jpena|lunch12:36
*** dsneddon has quit IRC12:38
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add always tags for hieradata render in external upgrade.  https://review.opendev.org/71032012:38
openstackgerritFrancesco Pantano proposed openstack/puppet-tripleo master: [WIP] Add Certmonger ceph_rgw class to config tls  https://review.opendev.org/70374312:42
*** tkajinam has quit IRC12:45
*** kozhukalov has quit IRC12:48
*** morazi has joined #tripleo12:50
sri_ntt, all you need to have is you should be able access your overcloud IPMI ip address from undercloud, To very the connectivity install the ipmitool in undercloud then run this " ipmitool -I lanplus -U username  -P password -H <ipmi ip adress> power status"12:51
*** bfournie has quit IRC12:52
*** jmasud has quit IRC12:53
*** jmasud has joined #tripleo12:56
*** udesale has joined #tripleo12:56
openstackgerritFrancesco Pantano proposed openstack/puppet-tripleo master: [WIP] Add Certmonger ceph_rgw class to config tls  https://review.opendev.org/70374312:57
*** rlandy has joined #tripleo12:57
EmilienMslaweq, ramishra : back from afk (sorry): I read "when node was overloaded" it could be podman containers executing things in // maybe and overloading the node. Though we haven't seen that problem on centos7/rhel8 testing AFIK13:01
EmilienMramishra: +2 on the interim solution ...13:02
*** jbadiapa has quit IRC13:03
nttsri_, Thank you!13:04
*** dsneddon has joined #tripleo13:04
*** jpich has quit IRC13:06
*** jpich has joined #tripleo13:06
openstackgerritEmilien Macchi proposed openstack/tripleo-operator-ansible master: Add first documentation for the project  https://review.opendev.org/71049013:08
*** dsneddon has quit IRC13:09
ykarelchandankumar, https://review.opendev.org/#/c/710727/ is also needed for container updates13:11
weshay|ruckykarel, bug or link to failing job? https://review.opendev.org/#/c/710727/113:12
ykarelweshay|ruck, https://bugs.launchpad.net/tripleo/+bug/186545913:13
openstackLaunchpad bug 1865459 in tripleo "[CentOS8]tripleo modify images fail when gating-repo is not present" [High,Triaged]13:13
ykarelweshay|ruck, job is not failing, job pass but containers are not updated13:14
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Add additional strategy optimization  https://review.opendev.org/70444313:14
*** jcapitao|lunch is now known as jcapitao13:15
*** morazi has quit IRC13:16
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deprecate Keepalived service  https://review.opendev.org/65706713:16
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Cleanup tasks for container-puppet.py  https://review.opendev.org/71050613:16
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: libvirt: remove unused cpuset_cpus  https://review.opendev.org/71030713:17
weshay|ruckykarel, so is there another patch?13:18
weshay|ruckI see the same error in the logs there13:18
weshay|ruckhttps://1d92714a860974d4302f-8075c89a724b2b68e4da309a54349f49.ssl.cf2.rackcdn.com/710727/1/check/tripleo-ci-centos-8-scenario002-standalone/ad70952/logs/undercloud/home/zuul/ansible.log13:18
ykarelweshay|ruck, needs both Fixes:- https://review.opendev.org/#/c/710727/ and https://review.opendev.org/#/c/710738/13:18
*** morazi has joined #tripleo13:19
ykarelso two issues, one is blank repo and other is gating repo13:19
weshay|ruckykarel, does https://review.opendev.org/#/c/710738/2 depend on https://review.opendev.org/#/c/710727/ ?13:19
ykarelweshay|ruck, nope :(13:19
weshay|ruckykarel, https://1d92714a860974d4302f-8075c89a724b2b68e4da309a54349f49.ssl.cf2.rackcdn.com/710727/1/check/tripleo-ci-centos-8-scenario002-standalone/ad70952/logs/undercloud/home/zuul/ansible.log13:19
weshay|ruck  rc: 013:20
weshay|ruck  start: '2020-03-02 10:06:52.942432'13:20
weshay|ruck  stderr: 'Error: Unknown repo: ''gating-repo'''13:20
weshay|ruck  stderr_lines: <omitted>13:20
weshay|ruck  stdout: No packages were found for update...13:20
weshay|ruck  stdout_lines: <omitted>13:20
*** bfournie has joined #tripleo13:20
ykarelweshay|ruck, yes in non branched project, gating-repo is not there, so ^^ is seen13:21
ykarelweshay|ruck, see https://3bebab18307fb946dd60-5e044492af15a220e29b0352bba13d3f.ssl.cf1.rackcdn.com/702846/12/check/tripleo-ci-centos-8-standalone/1c3bb55/logs/undercloud/home/zuul/ansible.log in branched project where gating-repo exist13:21
weshay|ruckshould be in either13:21
ykarelhere error is different stderr: 'Error: Unknown repo: '''''13:21
weshay|ruckykarel, hrm.. please rework the patches to have a more clear commit message w/ the bug as well..13:23
weshay|ruckykarel, or get on https://meet.google.com/oiv-geho-mai?authuser=113:23
ykarelweshay|ruck, ack joining13:23
*** bfournie has quit IRC13:24
*** ansmith has joined #tripleo13:26
*** ansmith_ has joined #tripleo13:28
weshay|ruckhttps://review.opendev.org/#/c/710738/13:28
*** ekultails has joined #tripleo13:29
*** ansmith has quit IRC13:31
*** pcaruana has quit IRC13:31
*** jpena|lunch is now known as jpena13:32
cloudnullmornings13:33
*** rpittau|bbl is now known as rpittau13:34
*** psachin has quit IRC13:35
*** dsneddon has joined #tripleo13:35
*** aedc has joined #tripleo13:37
openstackgerrityatin proposed openstack/tripleo-quickstart master: Revert "[CentOS-8] Install ansible 2.8"  https://review.opendev.org/71076913:38
*** bfournie has joined #tripleo13:38
openstackgerrityatin proposed openstack/tripleo-quickstart master: Revert "[CentOS-8] Install ansible 2.8"  https://review.opendev.org/71076913:39
cloudnullany chance I could get a couple reviews on the following https://review.opendev.org/#/q/starredby:cloudnull+status:open,n,z13:40
*** aedc_ has joined #tripleo13:42
*** aedc has quit IRC13:42
*** dsneddon has quit IRC13:43
ykarelcloudnull, hi13:45
cloudnullhi13:45
ykarelcloudnull, is log location /var/log/tripleo-container-image-prepare-ansible.log changed?13:45
ykareli see logs are now stored at /home/zuul/ansible.log13:45
cloudnullnot that im aware of13:45
cloudnullansible runner will store general ansible logs there.13:46
ykarelcloudnull, ack can u check that iirc u proposed to store tripleo-modify logs to /var/log/tripleo-container-image-prepare-ansible.log13:46
cloudnullbut "/var/log/tripleo-container-image-prepare-ansible.log" should remain unchanged13:46
ykarelcloudnull, ack so something changed that, tripleo-modify-image logs are currently stored at /home/zuul/ansible.log13:46
*** pcaruana has joined #tripleo13:46
* cloudnull will take a look 13:47
ykarelcloudnull, examples log https://1d92714a860974d4302f-8075c89a724b2b68e4da309a54349f49.ssl.cf2.rackcdn.com/710727/1/check/tripleo-ci-centos-8-scenario002-standalone/ad70952/logs/undercloud/home/zuul/ansible.log13:47
ykarelsearch tripleo-modify-image : Run yum_update.sh13:47
*** tosky_ has joined #tripleo13:47
*** tosky has quit IRC13:47
*** tosky_ is now known as tosky13:47
*** jdandrea has joined #tripleo13:47
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Replace Docker service by Podman in roles_data.  https://review.opendev.org/71077013:48
cloudnullykarel I see, do you know which playbook invokes that role ?13:48
ykarelcloudnull, the playbooks is generated dynamically in tripleo-common13:53
cloudnulldo you know what is the name of that dynamic playbook ?13:54
ykarellooking13:54
ykarelcloudnull, something like /tmp/tripleo-modify-image-playbook-p_ztd0yc/playbook.yaml13:55
ykarelsee https://1d92714a860974d4302f-8075c89a724b2b68e4da309a54349f49.ssl.cf2.rackcdn.com/710727/1/check/tripleo-ci-centos-8-scenario002-standalone/ad70952/logs/undercloud/var/log/tripleo-container-image-prepare.log13:55
cloudnullI think we just need to re-point the logs when that playbook is invoked via the client.13:55
ykarelcloudnull, https://github.com/openstack/tripleo-common/blob/62e28b2c094c12267f7a4b4c31b31c8e35648074/tripleo_common/image/image_uploader.py#L334-L347 is that code13:56
cloudnullthe use of "ansible-playbook-${PYTHON_VERSION}" is not something we do in tripleoclient anymore, hum.13:57
ykarelfrom code it seems tripleo-container-image-prepare-ansible.log is passed as log_path13:57
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799014:00
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Run tripleo-lvmfilter role to restrict block devices visible to LVM  https://review.opendev.org/69799714:01
*** jbadiapa has joined #tripleo14:01
*** waleedm has quit IRC14:01
openstackgerritGabriele Cerami proposed openstack/tripleo-ci master: Add centos8 support for emit_releases_script  https://review.opendev.org/71044114:02
ykarelcloudnull, ansible-playbook-${python_version} for this case is done in tripleo-common https://github.com/openstack/tripleo-common/blob/62e28b2c094c12267f7a4b4c31b31c8e35648074/tripleo_common/actions/ansible.py#L33614:04
cloudnullyup.14:05
* cloudnull poking around at some of that now 14:05
ykarelack14:05
* ykarel need to leave now, will followup later14:05
*** ykarel is now known as ykarel|away14:05
openstackgerritBrent Eagles proposed openstack/tripleo-quickstart-extras master: Allow full path to be specified for amphora destination  https://review.opendev.org/69285814:06
EmilienMcloudnull: if you have time, https://review.opendev.org/#/c/710490/ is ready for review14:09
*** jpich has quit IRC14:09
*** jpich has joined #tripleo14:10
*** dciabrin_ has joined #tripleo14:10
hjensascloudnull: any hints on https://review.opendev.org/710574 ? why is role_networks not defined? It's in group_vars.14:11
hjensascloudnull: we regressed here, see novacompute entries on stein: https://logserver.rdoproject.org/35/710635/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/2fea7d9/logs/overcloud-novacompute-0/etc/hosts.txt.gz, vs master: https://logserver.rdoproject.org/32/697832/9/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/71945b4/logs/overcloud-novacompute-0/etc/hosts.txt.gz14:11
* hjensas does'nts getz the anzibless ...14:12
*** mcornea has joined #tripleo14:12
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799014:13
cloudnullhjensas is it the spacing that we want to eliminate ?14:14
*** dciabrin has quit IRC14:15
hjensascloudnull: no, the entries for the networks not used by novacompute14:15
* cloudnull rubs sand out of my eyes and drinks another coffee 14:15
openstackgerritEmilien Macchi proposed openstack/tripleo-operator-ansible master: Add first documentation for the project  https://review.opendev.org/71049014:15
EmilienMcloudnull: thx14:16
cloudnullEmilienM idk if you want to fix that on a follow on ?14:16
EmilienMcloudnull: fixed14:16
hjensascloudnull: the mistake in code is simple enoug, s/role_networks | union(enabled_networks)/ role_networks | intersection(enabled_networks)/ should fix it.14:16
*** Goneri has joined #tripleo14:16
cloudnulloh.14:16
*** dsneddon has joined #tripleo14:16
*** aedc_ has quit IRC14:17
hjensascloudnull: but then role_networks is'nt defined - see: https://3e13ec3cd8121ced58f0-69e632a7aa98516f459f7791ccff0cc0.ssl.cf1.rackcdn.com/710574/15/check/tripleo-ci-centos-7-containers-multinode/91bd740/logs/subnode-1/etc/hosts14:17
* cloudnull reading 14:17
hjensascloudnull: afaict it's in group_vars - undercloud/var/lib/mistral/overcloud/group_vars/Controller14:18
*** aedc has joined #tripleo14:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix prepare-slave playbook for centos8 OVB  https://review.opendev.org/70986614:19
cloudnullhjensas I don't see role networks here https://3e13ec3cd8121ced58f0-69e632a7aa98516f459f7791ccff0cc0.ssl.cf1.rackcdn.com/710574/15/check/tripleo-ci-centos-7-containers-multinode/91bd740/logs/undercloud/var/lib/mistral/overcloud/global_vars.yaml14:21
*** dsneddon has quit IRC14:21
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Use ipc:host for cinder-backup  https://review.opendev.org/71063514:21
cloudnullenabled_networks is though14:22
cloudnullso i would have expected the other to be in that same file14:22
* cloudnull continues looking 14:22
*** aedc has quit IRC14:23
*** aedc has joined #tripleo14:23
hjensascloudnull: enabled_networks is global_vars14:23
cloudnullI see role_networks here - https://3e13ec3cd8121ced58f0-69e632a7aa98516f459f7791ccff0cc0.ssl.cf1.rackcdn.com/710574/15/check/tripleo-ci-centos-7-containers-multinode/91bd740/logs/undercloud/var/lib/mistral/overcloud/group_vars/Controller14:24
cloudnullso it should be a group var for all nodes under the "Controller" group14:24
*** bdodd has joined #tripleo14:25
cloudnulland it looks like the host file is generated on the undercloud and then copied to all available hosts. is that right ?14:27
slaweqhi, I'm trying to add neutron tripleo job based on centos 8 in https://review.opendev.org/#/c/701351/14:28
slaweqbut it's failing on building neutron rpm package14:28
slaweqdo You know where I can find logs from this package build to check what exactly was wrong with it?14:29
*** sri_ has quit IRC14:29
*** abdysn has quit IRC14:29
hjensascloudnull: oh, it has the undercloud's vars then. We need to pick up the value via hostsvar[host] in that jinja then?14:29
cloudnullmade a suggestion to pull it from a controller node - https://review.opendev.org/#/c/710574/15/tripleo_ansible/roles/tripleo_hosts_entries/tasks/main.yml14:30
cloudnullyup, that's what I think it the issue14:30
hjensascloudnull: awesome, thanks! I will continue digging.14:32
*** gfidente has quit IRC14:32
cloudnullI think once you set the fact using the controller's facts, that template will render the expected output (or at least be closer to the expected output :D)14:34
openstackgerritMerged openstack/tripleo-common stable/train: Do not create temporary swift URLS for ceph-ansible  https://review.opendev.org/71064714:34
openstackgerritMerged openstack/tripleo-heat-templates master: Remove LocalCephAnsibleFetchDirectoryBackup from standalone 001,004,010  https://review.opendev.org/71065514:35
openstackgerritHarald Jensås proposed openstack/tripleo-ansible master: [DNM] Only add host entries for role_networks  https://review.opendev.org/71057414:39
*** ade_lee has joined #tripleo14:42
abishopowalsh: pls take a look at https://review.opendev.org/70790114:42
* owalsh looks14:42
*** mcornea has quit IRC14:42
*** cdearborn has joined #tripleo14:42
*** mcornea has joined #tripleo14:43
*** zbr is now known as zbr|pto14:43
abishopowalsh: thanks!14:45
abishopfultonj: ^^14:45
owalshabishop: I wonder if cinder will move to endpoint overrides too14:46
mwhahahaslaweq: https://85243a2a53e02e239382-cf02a1947fec3ea7a230e93d6a5b1fec.ssl.cf5.rackcdn.com/701351/4/check/neutron-centos-8-tripleo-standalone/013296a/logs/delorean_logs/component/network/2e/20/2e20723baa132e7d129c1f2c094f654f71a14d49_dev/14:46
abishopowalsh: I wondered myself, but am not aware of any effort underway14:46
slaweqmwhahaha: thx a lot14:46
mwhahahaslaweq: DEBUG:     Macro expanded in comment on line 332: %{pyver}-ncclient14:46
mwhahahaslaweq: looks like the spec needs help14:46
owalshabishop: ack, should be straightforward to change if/when that happens14:47
abishopowalsh: I assume whoever does the work will follow nova's example14:47
slaweqmwhahaha: yes, I will check that14:48
*** dsneddon has joined #tripleo14:49
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Is anyone able to do a review here? I think we can call this one done if it deploys correctly, which it seems to do, unless I'm missing something, and we can handle the collectd/qdr network configuration in https://review.opendev.org/#/c/710376/  https://review.opendev.org/71078914:52
*** aedc_ has joined #tripleo14:53
*** aedc has quit IRC14:53
*** dsneddon has quit IRC14:54
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: [DNM] Don't add IpList for disabled networks  https://review.opendev.org/67935414:54
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Bump osc-lib==2.0.0 in lower constraints  https://review.opendev.org/71079214:56
*** beekneemech is now known as bnemec15:00
*** ykarel|away is now known as ykarel15:01
*** ansmith_ has quit IRC15:01
*** ansmith has joined #tripleo15:01
*** gfidente has joined #tripleo15:01
openstackgerritAlan Bishop proposed openstack/tripleo-common stable/train: Improve upload-swift-artifacts handling of json data  https://review.opendev.org/71079315:04
openstackgerritHarald Jensås proposed openstack/tripleo-ansible master: [DNM] Only add host entries for role_networks  https://review.opendev.org/71057415:05
*** skramaja has quit IRC15:05
weshay|ruckseeing jobs fail w/15:12
weshay|ruck'tripleo_keystone_resources_data_user': {'heat_stack_domain_admin': {'domain': 'heat_stack'}}, 'ansible_loop_var': 'tripleo_keystone_resources_data_user'}) => changed=false15:12
weshay|ruck2020-03-02 14:55:31 |   ansible_job_id: '137192272725.174220'15:12
weshay|ruck2020-03-02 14:55:31 |   ansible_loop_var: tripleo_keystone_resources_user_role_async_result_item15:12
weshay|ruck2020-03-02 14:55:31 |   attempts: 115:12
weshay|ruck2020-03-02 14:55:31 |   finished: 115:12
weshay|ruck2020-03-02 14:55:31 |   msg: Must specify either a domain or project15:13
weshay|ruck2020-03-02 14:55:31 |   tripleo_keystone_resources_user_role_async_result_item:15:13
weshay|ruckring any bells?15:13
weshay|ruckade_lee, ^15:13
weshay|ruckhttps://review.opendev.org/#/c/697161/15:14
weshay|ruckEmilienM, ^15:14
EmilienMyes, give me a min I'm in meeting15:14
weshay|rucknp15:14
EmilienMthe meeting you're missing btw15:15
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Skeleton role, get with the decade  https://review.opendev.org/70741415:15
openstackgerritMerged openstack/tripleo-operator-ansible master: Add first documentation for the project  https://review.opendev.org/71049015:15
weshay|ruckmwhahaha, any reviews out on "Trying out centos8 builds, undercloud is hanging on heat notifier for some reason."15:17
weshay|ruckrlandy, ^15:17
mwhahahano,have to figure out what's goign on15:17
rlandythanks15:17
mwhahahacause rerunning it works15:17
rlandymwhahaha: we find the same problem in testing undercloud/multinode jobs - the hang is sporadic.15:18
mwhahahayea15:18
mwhahahait should be a noop15:18
mwhahahaso trying to figure out what'shappenign15:18
rlandymwhahaha: thanks for looking at this - if we figure out the hang, it will spring free undercloud and multinode jobs15:19
openstackgerritGrzegorz Grasza proposed openstack/tripleo-ansible master: Make sure not to re-enroll already enrolled server  https://review.opendev.org/70440415:21
openstackgerritGrzegorz Grasza proposed openstack/tripleo-ansible master: Added files to allow cleanup when stack delete is called  https://review.opendev.org/70658815:21
openstackgerritGrzegorz Grasza proposed openstack/tripleo-ansible master: Add undercloud-ipa-install to playbooks  https://review.opendev.org/70958915:21
*** dsneddon has joined #tripleo15:23
openstackgerritMerged openstack/tripleo-heat-templates master: Check Ceph*Key value format and halt on error  https://review.opendev.org/70909415:27
*** dsneddon has quit IRC15:28
*** goneri_ has joined #tripleo15:28
*** goneri_ has quit IRC15:28
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/train: Add custom ansible-lint rule for Role Names  https://review.opendev.org/70950515:29
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/train: Add basic molecule tests for uncovered roles  https://review.opendev.org/70952715:29
openstackgerrityatin proposed openstack/ansible-role-tripleo-modify-image master: Remove unavailable repos from update  https://review.opendev.org/71073815:30
*** aedc_ has quit IRC15:32
*** aedc_ has joined #tripleo15:32
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud upgrade prepare  https://review.opendev.org/71053715:34
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud upgrade converge  https://review.opendev.org/71054215:34
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Add STF environment  https://review.opendev.org/69462315:34
openstackgerritRyan McCabe proposed openstack/tripleo-common master: Enable configuration for ComputeCollectd  https://review.opendev.org/71003515:34
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Add j2 per-role MetricsQdrNetwork  https://review.opendev.org/71037615:35
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud update converge role  https://review.opendev.org/71053615:35
*** chandankumar is now known as raukadah15:36
weshay|ruckekultails, you are ruck?15:37
weshay|ruckekultails, doc says u are :)  incoming.. https://bugs.launchpad.net/tripleo/+bug/1865329  https://bugs.launchpad.net/tripleo/+bug/186532815:38
openstackLaunchpad bug 1865329 in tripleo "'#!/usr/bin/env python' does not work on Ubuntu Focal (e.g. in 55-heat-config)" [High,Triaged]15:38
openstackLaunchpad bug 1865328 in tripleo "os-collect-config depends on python-dev but package is not available in Ubuntu Focal" [Undecided,In progress] - Assigned to Andreas Florath (ansreas)15:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/train: Check Ceph*Key value format and halt on error  https://review.opendev.org/71079915:38
ekultailsweshay: For the DF, yeah. I'll have a look.15:38
weshay|ruckekultails, also seeing jobs fail w/15:39
weshay|ruck"[08:12:57] <weshay|ruck> 2020-03-02 14:55:31 |   msg: Must specify either a domain or project"15:39
weshay|ruckscroll up for context.. may need some help w/ that15:39
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: [DNM] Don't add IpList for disabled networks  https://review.opendev.org/67935415:40
ekultailsSounds like some kind of Keystone v2 vs. v3 problem somewhere. Hmm.15:40
weshay|ruckya15:41
*** sshnaidm|pto has quit IRC15:41
EmilienMekultails, weshay|ruck : I don't think it's for DF only. We need help from Keystone folks15:43
weshay|ruckya.. I pinged ade_lee15:43
nttHi, how can I set undercloud.conf in order to use a gateway in the pxe/provisioning network with a different ip instead of using the undercloud ip? Furthermore, should I set masquerade = false ?15:43
weshay|ruckhrybacki, ^15:43
EmilienMweshay|ruck: with https://review.opendev.org/#/c/710449/ - we should get a better idea why it fails15:44
EmilienMmy patch would tell if the domain lookup failed15:44
EmilienMweshay|ruck: can you depends-on this one if you can consistently reproduce it?15:44
weshay|ruckEmilienM, I don't think the job in question is going to run there..15:44
* weshay|ruck looks15:44
*** aedc_ has quit IRC15:45
weshay|ruckEmilienM, nope15:45
* weshay|ruck adds15:45
*** aedc has joined #tripleo15:48
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: add standalone jobs to tripleo-ansible layout  https://review.opendev.org/71080715:48
weshay|ruckEmilienM, ^15:48
weshay|ruckcool w/ that?15:48
ade_leeweshay|ruck, in meeting now -- will get back to you sson15:48
weshay|ruckade_lee, np15:48
weshay|ruckthanks15:48
EmilienMweshay|ruck: no15:48
fultonjekultails: do you know anything about the "--config-download-timeout" ?15:48
weshay|ruckyou guys all sound like a bunch of managers ;)15:49
ekultailsfultonj: Yeah. We're not 100% sure why it happens. Our leading theory is that it's related to the Keystone token expiration time.15:49
EmilienMweshay|ruck: this is way too much15:49
EmilienMweshay|ruck: and redundant with what is already there15:49
fultonjekultails: is it an actual option?15:49
weshay|ruckhrm.. redundant how?15:50
* fultonj can't find it in https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/ansible_config_download.html 15:50
weshay|rucktoo much I can understand15:50
openstackgerritMerged openstack/python-tripleoclient master: Remove Mistral from the plan export  https://review.opendev.org/70781215:50
fultonjekultails: i'm looking at a bug report and someone is talking about it like it's something they passed to their deployment15:50
weshay|ruckoh.. I see what you mean15:51
EmilienMweshay|ruck: I'll comment in Gerrit15:51
weshay|ruckEmilienM, sec..15:51
ekultailsfultonj:  --config-download-timeout should be a valid option for `openstack overcloud deploy`. In Train (and earlier releases) we ran `timeout $CONFIG_DOWNLOAD_TIMEOUT_ARG ansible-playbook ...` via subprocess. I'm not sure how that has changed in master with the switch to Ansible Runner.15:52
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Tune-up package install  https://review.opendev.org/71080815:52
EmilienMweshay|ruck: commenteds15:52
fultonjekultails: ok, thanks so it's a real param and it inserts "timeout X ansible-playbook ..."15:53
ekultailsfultonj: Yeap, you got it.15:54
weshay|ruckEmilienM, ur right..  train has the overrides as well.. going to update master for centos-8 and files15:54
*** sshnaidm|pto has joined #tripleo15:55
openstackgerritMerged openstack/python-tripleoclient master: Relax ~/.config permissions  https://review.opendev.org/71016715:56
*** dmellado has quit IRC15:57
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/train: Relax ~/.config permissions  https://review.opendev.org/71080915:59
*** dmellado has joined #tripleo16:00
*** dsneddon has joined #tripleo16:00
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: update tripleo-ansible ci for centos-8  https://review.opendev.org/71081016:01
weshay|ruckcloudnull, EmilienM second attempt16:01
* cloudnull holds beer 16:02
EmilienMweshay|ruck: you're a cowboy16:02
EmilienMbut still not good16:02
EmilienMI'll comment16:02
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: update tripleo-ansible ci for centos-8  https://review.opendev.org/71081016:03
weshay|ruckEmilienM, df voted to get centos-8 in asap16:03
*** abishop_ has joined #tripleo16:04
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: update tht standalone scenario jobs for centos-8  https://review.opendev.org/71081116:05
*** abishop has quit IRC16:06
*** cylopez has quit IRC16:10
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: update tripleo-ansible ci for centos-8  https://review.opendev.org/71081016:11
*** jcapitao is now known as jcapitao_afk16:12
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Tune-up package install  https://review.opendev.org/71080816:12
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: update tht standalone scenario jobs for centos-8  https://review.opendev.org/71081116:13
weshay|ruckekultails, review out here https://review.opendev.org/#/c/710631/116:15
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Add STF environment  https://review.opendev.org/69462316:16
*** dmellado has quit IRC16:18
ekultailsweshay: I left a comment earlier. I wonder if this fix would also work long-term with newer Ubuntu releases after Focal/20.20.16:19
ekultailsOtherwise, looks good for fixing the issue immediately.16:19
weshay|ruckekultails, cool thanks16:20
*** dmellado has joined #tripleo16:20
openstackgerritRyan McCabe proposed openstack/tripleo-common master: Enable configuration for ComputeCollectd  https://review.opendev.org/71003516:22
*** ykarel is now known as ykarel|away16:23
openstackgerritwes hayutin proposed openstack/paunch master: move centos-7 jobs to centos-8 where possible  https://review.opendev.org/71081616:25
*** ntt has quit IRC16:26
jbadiapaweshay|ruck: can we have this patch https://review.opendev.org/#/c/690698/ merged, please? or do we need to rebase it again? It's the one to add the ReaR to the scenario001.16:28
*** mschuppert has quit IRC16:28
weshay|ruckjbadiapa, reviews are hard to come by eh16:29
weshay|ruckjbadiapa, come to the tripleo meetings on tuesdays.. focus is on getting timely  / adequate reviews16:29
jbadiapaweshay|ruck: ok, I'll be there, thx16:30
weshay|ruckjbadiapa, next time :)16:31
jbadiapaweshay|ruck: thanks a lot :)16:32
*** lmiccini has quit IRC16:32
openstackgerritMerged openstack/tripleo-ansible stable/train: Fix error with command ansible module and pipes  https://review.opendev.org/71054916:33
openstackgerritwes hayutin proposed openstack/python-tripleoclient master: migrate from centos7 -> centos-8 for standalone jobs  https://review.opendev.org/71081816:33
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Add NeutronDhcpAgentDnsmasqEnableAddr6List param  https://review.opendev.org/71034216:37
openstackgerritwes hayutin proposed openstack/tripleo-common master: replace centos-7 with centos-8 in scenario jobs  https://review.opendev.org/71082116:37
ekultailsmwhahaha: I wanted to know your thoughts on this one (see my latest comment) https://review.opendev.org/#/c/706962/16:39
ekultailsThe tl;dr is that current-passed-ci is the expected/real directory but most users would probably know/use current-tripleo-rdo so I want to provide a way to support either-or.16:39
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Add c8 multinode containers job  https://review.opendev.org/71017916:40
rlandyweshay|ruck: ^^ updated with non-voting16:40
ekultailsmwhahaha: I could add 'if current-tripleo-rdo; then echo "Use current-passed-ci instead."' or something similar.16:40
openstackgerritEmilien Macchi proposed openstack/paunch master: Update zuul layout to reflect recent changes in TripleO  https://review.opendev.org/71082416:41
EmilienMweshay|ruck: ^ FYI you might want to rebase your paunch patch on top of that16:41
weshay|ruckEmilienM, on rlandy's patch?16:42
weshay|ruckoh I see16:42
weshay|rucknvermind16:42
openstackgerritEmilien Macchi proposed openstack/paunch master: Update zuul layout to reflect recent changes in TripleO  https://review.opendev.org/71082416:42
weshay|ruckroger16:42
*** dmellado has quit IRC16:46
openstackgerritwes hayutin proposed openstack/paunch master: move centos-7 jobs to centos-8 where possible  https://review.opendev.org/71081616:47
weshay|ruckthanks16:47
*** jcapitao_afk is now known as jcapitao16:48
mwhahahaekultails: i don't think we need to include it at all16:48
mwhahahaunless the ci team needs it for some reason16:48
mwhahahai'd have to defer to weshay|ruck & co16:48
*** dmellado has joined #tripleo16:49
weshay|ruckekultails, /me guessing.. include ubuntu jobs? no16:49
mwhahahaweshay|ruck: no the usage of current-tripleo-rdo as a repo config16:50
ekultailshttps://review.opendev.org/#/c/706962/16:50
ekultails^16:50
weshay|ruckekultails, heh. not in use.. I think the rdo folks like that tag..16:51
weshay|ruckekultails, current-passed-ci implies something.. but it's not clear16:51
weshay|ruckekultails, as there are multiple levels of ci16:52
weshay|ruckmeh..16:52
weshay|ruckekultails, however.. I do need a collaborator to ensure tripleo-repos works well w/ the new centos-8 layout16:52
weshay|ruckyou familiar w/ it?16:52
weshay|rucknote the diff..https://trunk.rdoproject.org/centos8-master/component/16:53
ekultailsweshay: Yeah, so I was originally going to just have it use current-tripleo-rdo but the CentOS 8 Train repo doesn't have that: https://trunk.rdoproject.org/centos8-train/16:53
ekultailsIf we ensure the symlink is created from current-tripleo-rdo to current-passed-ci then we'd be good to just use current-tripleo-rdo.16:53
weshay|ruckhttps://review.opendev.org/#/c/710098/  is one thing.. needs a grooming16:54
*** jmasud has quit IRC16:54
weshay|ruckand I'd like to finally grant mwhahaha's wish of all upstream ci using tripleo-repos16:54
*** jmasud has joined #tripleo16:55
ekultails+1 for standardization on configuring repos.16:57
weshay|ruckekultails, aye.. first pass is w/o tripleo-repos.. but it's coming16:59
weshay|ruckWINTER16:59
*** rpittau is now known as rpittau|afk16:59
*** dmellado has quit IRC17:01
*** dsneddon has quit IRC17:03
openstackgerritLuke Short proposed openstack/tripleo-repos master: Add missing current-tripleo-rdo repository choice.  https://review.opendev.org/70696217:03
ekultails^ Supporting current-tripleo-rdo only now.17:04
openstackgerritwes hayutin proposed openstack/tripleo-validations master: update centos-7 to centos-8 for validations  https://review.opendev.org/71083217:04
*** dmellado has joined #tripleo17:07
openstackgerritEmilien Macchi proposed openstack/paunch master: Update zuul layout to reflect recent changes in TripleO  https://review.opendev.org/71082417:08
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/train: Restart ovn-dbs resource to take new VIP property.  https://review.opendev.org/71083517:11
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Handle race for the already existing layer  https://review.opendev.org/71083617:13
openstackgerritMerged openstack/tripleo-ansible master: Fixed the condition for gather_domains  https://review.opendev.org/70995917:16
*** tesseract has quit IRC17:16
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade stable/train: Modify overcloud upgrade run loop.  https://review.opendev.org/71083717:16
*** jpich has quit IRC17:18
openstackgerritRonelle Landy proposed openstack/tripleo-heat-templates stable/train: DNM - Testing centos-8 triggers  https://review.opendev.org/71083817:19
*** dmellado has quit IRC17:19
mwhahahaEmilienM: thoughts on the correct location for something like https://review.opendev.org/#/c/709112/17:21
rlandyweshay|ruck: ^^ search zuul ... triggers are all centos-717:21
EmilienMmwhahaha: looking17:21
mwhahahaimho is not tripleo specific17:22
openstackgerritPablo Caruana proposed openstack/tripleo-heat-templates stable/rocky: Use ipc:host for cinder-backup  https://review.opendev.org/71083917:22
weshay|ruckrlandy, perfect :)17:22
EmilienMmwhahaha: I agree; and I kind of think the same for ade_lee's patch https://review.opendev.org/#/c/69190617:23
weshay|ruckrlandy, let's see how it goes upstream :) rlandy question to #tripleo tomorrow.. voting or non-voting for containers-multinode https://etherpad.openstack.org/p/tripleo-meeting-items17:23
rlandyack17:24
*** udesale has quit IRC17:24
*** dmellado has joined #tripleo17:25
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Handle race for the already existing layer  https://review.opendev.org/71083617:26
weshay|ruckoh ade_lee /me wonders if it's related to https://review.opendev.org/#/c/709959/2/tripleo_ansible/roles/tripleo_keystone_resources/tasks/main.yml17:26
weshay|ruckekultails, ^17:27
EmilienMweshay|ruck: indeed17:28
*** matbu has quit IRC17:28
EmilienMI missed that patch17:28
EmilienMwe need to backport it17:28
EmilienMit's conflicting, let me see what's missing17:28
weshay|ruck:)17:28
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible stable/train: Fixed the condition for gather_domains  https://review.opendev.org/71084017:28
*** pkopec has quit IRC17:28
EmilienMweshay|ruck: however I'm concerned about ykarel|away's -117:29
openstackgerritMerged openstack/tripleo-ansible master: Fix octavia services restart  https://review.opendev.org/70908917:29
openstackgerritMerged openstack/tripleo-ansible stable/train: Don't fail symlinking puppet modules if directory exists  https://review.opendev.org/71051817:29
*** bogdando has quit IRC17:30
EmilienMweshay|ruck, raukadah : do you confirm with need that backport ? ^17:30
*** lucasagomes has quit IRC17:30
*** pkopec has joined #tripleo17:31
weshay|ruckEmilienM, I don't think so tbh.. not until the ansible version changes in train17:31
*** evrardjp has quit IRC17:35
ykarel|awayEmilienM, that patch should not harm currently, but it also not fixes any thing for ansible-2.9, keystone domain fails with ansible-2.9 with that patch, so as soon as ansible2.9 those tasks will be failing17:35
*** evrardjp has joined #tripleo17:35
ykarel|awayin centos8-train we have 2.9 currently, but that will go away soon, before any centos8 train jobs are up17:35
ykarel|awayin centos7 we have 2.8, and no plan at present to bump it to 2.917:37
*** dsneddon has joined #tripleo17:37
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Revert "Remove chunk size for url stream"  https://review.opendev.org/69322517:38
*** jpena is now known as jpena|off17:39
*** dsneddon has quit IRC17:42
lbragstad_weshay|ruck having some keystone issues?17:42
*** lbragstad_ is now known as lbragstad17:42
ykarel|awayEmilienM, https://logserver.rdoproject.org/29/25329/14/check/periodic-tripleo-ci-centos-8-scenario002-standalone-master/a7e33c8/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz was the error with ansible-2.9 and that patch17:42
ade_leeweshay|ruck, just got outta meetings and gotta lunch -- lbragstad is going to fix all your keystone isues17:43
ade_leeissues17:43
weshay|ruckade_lee, I think that patch I just linked... just merged will fix it17:44
*** yprokule has quit IRC17:44
lbragstadhttps://review.opendev.org/#/c/697161/ ?17:44
weshay|rucklbragstad, https://review.opendev.org/#/c/709959/2/tripleo_ansible/roles/tripleo_keystone_resources/tasks/main.yml17:44
lbragstadah - interesting17:45
lbragstadso - newer versions of tripleo-ansible will set facts based on os_keystone_domain_info?17:48
lbragstadweshay|ruck actually - i have a question regarding the rdo repos17:52
lbragstadi recently stood up an environment with tripleo-repos -b master current-tripleo, but it didn't have a few fixes i needed (one merged 11 days ago, another merged 3 weeks ago)17:53
lbragstadwe're ( ade_lee and myself) are wondering what the difference is between that repo/master combination and tripleo-repos -b master current-tripleo-dev17:54
weshay|rucklbragstad, read openstack-discuss[tripleo] and rdo-list.. everything is pinned17:54
*** jcapitao is now known as jcapitao_off17:54
mwhahahacurrent-tripleo-dev includes the latest tripleo bits (as like is run in ci)17:54
mwhahahathere's an includepkgs that happens to give you the latest tripleo bits17:54
mwhahahathis is the recommended dev setup17:54
mwhahahardo is only promoted bits (which may have older tripleo related packages)17:55
weshay|ruckmwhahaha, still everything is pined.. lbragstad master ya?17:55
lbragstadyeah17:55
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Set container build hash based on dlrn md5 checksum  https://review.opendev.org/71023017:55
mwhahahaweshay|ruck: but not tripleo17:55
weshay|ruckya.. tripleo is not17:55
mwhahahabut yea we're not getting latest openstack stuff17:55
* weshay|ruck assumes lbragstad is looking for keystone updates17:56
lbragstadi'll get specific examples17:56
lbragstadfor additional context, i'm deploying tls-e with OVB + ldap-backends using master (but without novajoin)17:56
lbragstadEmilienM proposed this a while ago https://review.opendev.org/#/c/707471/17:57
lbragstadhttps://review.opendev.org/#/c/705634/2 is also a workaround that gets tls-e without novajoin environments working17:57
lbragstadmy openstack-tripleo-heat-templates didn't seem to have those patches17:58
mwhahahashould have both of those17:59
weshay|ruckyup17:59
mwhahahacurrent-tripleo is from the 27th17:59
lbragstadhuh - interesting...17:59
lbragstadopenstack-tripleo-heat-templates-12.1.1-0.20200301213215.2b8a301.el7.noarch is what i have on my system18:00
*** matbu has joined #tripleo18:00
mwhahahayea should have those patches18:00
lbragstadok - maybe it's our issue then...18:00
*** fmount has quit IRC18:01
*** mwhahaha changes topic to "CI Status: ORANGE (tox issues for <=py35) | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs/latest/"18:01
*** saneax has quit IRC18:03
*** derekh has quit IRC18:05
*** dsneddon has joined #tripleo18:10
*** dsneddon has quit IRC18:16
*** iurygregory has quit IRC18:19
openstackgerritLuke Short proposed openstack/tripleo-docs master: [troubleshooting] Remove QEMU workaround.  https://review.opendev.org/70186218:23
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Remove support for ceph-ansible fetch directory  https://review.opendev.org/71064818:35
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799018:36
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Run tripleo_lvmfilter role to restrict block devices visible to LVM  https://review.opendev.org/69799718:36
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799018:37
*** bandini has quit IRC18:40
*** dsneddon has joined #tripleo18:44
weshay|ruckekultails, you've been struck by..18:48
weshay|ruckcmd: /usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3 pep818:48
weshay|ruckImportError: cannot import name 'ContextManager'18:48
weshay|ruckhttps://review.opendev.org/#/c/704655/118:48
weshay|ruckthat is a new error18:48
mwhahahaweshay|ruck: see mailing list18:48
* weshay|ruck looks18:49
mwhahahaVirtualenv (and Tox) broken when run under python<3.618:49
mwhahahasee topic18:49
*** jcapitao_off has quit IRC18:49
weshay|rucksee it18:49
*** dsneddon has quit IRC18:49
*** alexmcleod has quit IRC18:50
ekultailsThis probably affects all releases, right? Not just Rocky?18:51
weshay|ruckhttp://zuul.openstack.org/builds?job_name=openstack-tox-py3518:51
mwhahahastein and below18:53
mwhahahawe dropped the 35 jobs in train18:53
mwhahahatrain is probably still ok (unless py27 is also broke)18:53
openstackgerritDave Wilde (d34dh0r53) proposed openstack/tripleo-ansible master: Add undercloud-ipa-install to playbooks  https://review.opendev.org/70958918:53
weshay|ruckekultails, probably need to sit on this and see if infra provides a workaround18:54
ekultailsweshay: Yeah, it looks like we'll have to wait for this to get merged: https://review.opendev.org/#/c/710729/18:55
ekultailsOr if that works maybe it's something we have to apply to all of our TripleO repositories?18:56
*** apetrich has quit IRC18:56
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix prepare-slave playbook for centos8 OVB  https://review.opendev.org/70986618:57
weshay|ruckya.. we could add that to pre18:57
mwhahahait's a global problem, i wouldn't work around it in tripleo18:58
ekultailsRodger that18:59
weshay|rucktracking it https://bugs.launchpad.net/tripleo/+bug/186556019:00
openstackLaunchpad bug 1865560 in tripleo "tracker: Virtualenv (and Tox) broken when run under python<3.6" [Critical,Triaged]19:00
*** weshay|ruck changes topic to "CI Status: ORANGE (tox issues for <=py35 1865560) | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs/latest/"19:01
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the deploy_plan workflow  https://review.opendev.org/71047919:02
*** dtantsur is now known as dtantsur|afk19:04
*** apetrich has joined #tripleo19:05
*** ansmith_ has joined #tripleo19:09
*** ansmith has quit IRC19:12
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859919:14
*** dsneddon has joined #tripleo19:16
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Add STF environment  https://review.opendev.org/69462319:17
*** pcaruana has quit IRC19:17
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859919:17
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859919:18
openstackgerritAde Lee proposed openstack/tripleo-ansible master: Add role to add ipa services for tls-everywhere  https://review.opendev.org/69190619:19
openstackgerritAde Lee proposed openstack/tripleo-ansible master: Add code to deregister nodes and services from freeipa  https://review.opendev.org/70022619:20
openstackgerritwes hayutin proposed openstack/tripleo-ci master: add the upstream centos-8 undercloud job  https://review.opendev.org/71085619:20
openstackgerritAde Lee proposed openstack/tripleo-ansible master: Make sure not to re-enroll already enrolled server  https://review.opendev.org/70440419:20
openstackgerritAde Lee proposed openstack/tripleo-ansible master: Added files to allow cleanup when stack delete is called  https://review.opendev.org/70658819:21
*** dsneddon has quit IRC19:21
openstackgerritAde Lee proposed openstack/tripleo-ansible master: Add undercloud-ipa-install to playbooks  https://review.opendev.org/70958919:21
EmilienMade_lee: did you read my comments on ^ ?19:22
ade_leeEmilienM, you mean about the role not being in tripleo-ansible?19:23
EmilienMyes19:23
ade_leeyeah - trying to think about how to think about that19:24
ade_leeEmilienM, the role is going to central to the way we set up tls everywhere19:24
*** pbandark has joined #tripleo19:25
ade_leeEmilienM, it is essentially the replacement for novajoin and novajoin-notifier19:25
*** morazi has quit IRC19:25
openstackgerritwes hayutin proposed openstack/tripleo-ci master: add the upstream centos-8 undercloud job  https://review.opendev.org/71085619:27
mwhahahaweshay|ruck: can we cut down the logging on the overcloud_deploy? it's like 28 megs https://e36d522c3cb98e5fd761-ceb3fa3461e53fea8d32d5b9e8c24a55.ssl.cf1.rackcdn.com/710808/2/check/tripleo-ci-centos-7-containers-multinode/7ff2603/logs/undercloud/home/zuul/19:27
* weshay|ruck looks19:27
weshay|ruckmwhahaha, remove --verbose/19:28
weshay|ruck?19:28
*** ansmith_ has quit IRC19:29
mwhahahawe need to reduce something19:29
mwhahahanot certain why the container image prepare bits are being sent to stdout either19:29
ade_leeEmilienM, mwhahaha I guess my question is - what determines whether something is part of tripleo - and what isn't.19:29
openstackgerritwes hayutin proposed openstack/tripleo-ci master: add the upstream centos-8 undercloud job  https://review.opendev.org/71085619:29
ade_leethis role is all about how tripleo uses ipa for tls-e19:30
mwhahahaade_lee: if it's something configuring an external service that isn't specific tripleo, it shouldn't live in tripleo19:30
weshay|ruckI don't think the prepare bits are in the log19:30
weshay|ruckthat log19:30
mwhahahaipa is an external component but the setup/configruation of that isn't tripleo19:30
mwhahahaweshay|ruck: yea it is19:30
weshay|ruckwhat the duece19:30
weshay|ruckoh for the containers that are not in the undercloud19:31
weshay|ruckmwhahaha, we should prepare the containers at one time imho but meh19:31
mwhahahawe can't do that19:31
weshay|ruckmwhahaha, think removing verbose will do anything?19:31
mwhahahaand i'm not getting into that again19:31
weshay|ruckk19:31
mwhahahaalso not related to this19:31
weshay|ruckmwhahaha, the other option I've been talking about w/ the logging folks19:32
mwhahahai'll go dig up the config var19:32
weshay|ruckis that we could roll the logs prior to collection19:32
*** gfidente is now known as gfidente|afk19:32
ade_leemwhahaha, EmilienM the role does not configure an ipa server.  that is assumed to already be done.  what it does is configure the undercloud and overcloud nodes as ipa clients.19:32
mwhahahaade_lee: the configuraton of a client is nto tripleo either19:32
EmilienMbut this role could be used outside of tripleo right?19:32
mwhahahawe just execute19:32
weshay|ruckmwhahaha, EmilienM rlandy has some notes on this and some wip jobs19:32
weshay|ruckre: ipa19:32
mwhahahaconfiguring an ipa client against a server is a universal issue19:32
mwhahahasee also puppet-mysql, etc19:32
mwhahahawe don't techincally own those19:32
EmilienMmwhahaha++19:33
weshay|ruckright19:33
mwhahahathey are an external dependency19:33
weshay|ruckade_lee, and rlandy was writing a role for that19:33
mwhahahathe calls into the external depdency are tripleo19:33
weshay|ruckoutside of tripleo19:33
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799019:33
weshay|ruckade_lee, put it on hold until centos-8 is done :)19:34
EmilienMso now centos8 jobs are voting in check right?19:34
weshay|ruckyes19:34
rlandysome of them are19:34
EmilienMscenario002 failed on me19:34
weshay|ruckEmilienM, domain?19:34
ade_leemwhahaha, EmilienM , weshay|ruck -- ok -- so there are a few things that the role does right now -- let me enumerate them so we can determine whether it should be there or not.19:35
EmilienMweshay|ruck: yes19:35
rlandyhttps://review.opendev.org/#/c/706373/19:35
weshay|ruckEmilienM, ya.. the fix just merged 75min ago19:35
rlandy^ last of the IPA server work19:35
ade_leeI'll do this on a pastebin ..19:36
weshay|ruckade_lee, use hackmd.. :)19:36
weshay|ruckit's super declicious19:36
weshay|ruckmeh.. gave up spelling years ago19:36
weshay|ruckmwhahaha, since we're distracted should we chat logging / operators etc?19:36
mwhahahaweshay|ruck: sure19:37
rlandyEmilienM: was the scenario002 failure in the tripleo-heat-templates job?19:37
EmilienMrlandy: was in THT: https://review.opendev.org/#/c/710506/19:38
rlandythat failure usually is ... http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario002-standalone19:39
* rlandy investigates19:39
*** pkopec has quit IRC19:40
*** pkopec has joined #tripleo19:41
*** chem has quit IRC19:48
*** chem has joined #tripleo19:50
ade_leeEmilienM, weshay|ruck mwhahaha https://etherpad.openstack.org/p/tripleo_ipa_role19:53
ade_leeEmilienM, weshay|ruck mwhahaha we're doing a lot of stuff here that is very specific to how tripleo uses ipa19:54
EmilienMbut let's say I don't use tripleo19:55
EmilienMI could easily use that role in my deployment19:55
*** dsneddon has joined #tripleo19:56
EmilienMI mean nearly all these things are client stuff19:56
ade_leeEmilienM, if I wanted to register something as a ipa client -- I could just call ansible-freeipa19:57
ade_leebut I also want to read service metadata19:57
* EmilienM opens https://github.com/freeipa/ansible-freeipa/tree/master/roles19:57
ade_leewhich is generated by tripleo19:57
ade_leeand I also want to call stuff to do all the cleanup of those kinds of data19:58
EmilienMmy concern is that having too much roles in tripleo instead of independent projects makes it harder for us, to maintain20:00
*** dsneddon has quit IRC20:01
EmilienMi don't think we should threat tripleo-ansible as a free-for-all place for roles and such20:01
EmilienMfor tripleo-directly-related, yes20:01
EmilienMbut in your case I wonder why we don't call ansible-freeipa from THT directly20:01
EmilienMor have a role in tripleo-ansible which calls ansible-freeipa, like ceph-ansible20:02
EmilienMyou're already calling ansible-freeipa for certain things but there are still tasks that will be tripleo-ansible only; that's my problem20:02
ade_leeEmilienM, so -- the things that are calling tripleo-ipa are doing the things that used to be done by novajoin20:03
ade_leeEmilienM, now -- we can bring all those things directly into a tht template - but I think thats harder to maintain20:04
EmilienMbrb20:05
*** dsneddon has joined #tripleo20:06
ade_leeEmilienM, gotta go get kids from school - but I'll pick this back up when I return20:06
EmilienMok20:07
EmilienMi don't want to be picky or anything but I'm not convainced we need a 1000LOC patch in in tripleo-ansible just to add ipa services20:08
lbragstadso - our alternative would be to put everything into a template and invoke ansible through deploy_step_tasks?20:10
fultonjexternal_deploy_steps20:11
lbragstads/our alternative/one possible alternative/20:11
fultonjhttps://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/ansible_config_download.html20:12
fultonj^ ctrl-f external_deploy_steps20:12
fultonjlbragstad: ^20:13
fultonjThese tasks are run against the undercloud node only.20:13
lbragstadhere? https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/ansible_config_download.html#tags20:13
fultonjso they could tell the undercloud to call ansible-freeipa20:13
fultonjwith paramters from THT20:13
fultonjlbragstad: that's who ceph and octavia integration already work20:14
fultonjif you want to follow an existing pattern20:14
*** derekh has joined #tripleo20:14
fultonjlbragstad: true those are documented as tags20:14
fultonjlbragstad: but they're also referring to an interface config-download has aimed at running ansible for external tools20:15
openstackgerritMerged openstack/tripleo-heat-templates master: Set logging parameters for virtlogd  https://review.opendev.org/69783220:15
openstackgerritMerged openstack/tripleo-puppet-elements stable/train: Fix permissions on 51-enable-network-service  https://review.opendev.org/71046720:15
*** jfrancoa has quit IRC20:15
weshay|ruckade_lee, there are other examples to follow I think for this.. ping me as you have time20:17
fultonjade_lee: ^20:17
*** ansmith_ has joined #tripleo20:17
*** ansmith has joined #tripleo20:19
lbragstadweshay|ruck fultonj interesting, so config-download provides the data and we'd have to write the ansible in external_deploy_steps to do whatever based on input from config-download20:20
fultonjlbragstad: yeah20:21
openstackgerritMerged openstack/tripleo-heat-templates master: Override nova's glance endpoint only when necessary  https://review.opendev.org/70790120:21
*** mcornea has quit IRC20:21
fultonjlbragstad: e.g. https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_run_ansible/tasks/main.yml20:21
*** ansmith_ has quit IRC20:21
fultonjcalls ceph-ansible20:21
fultonjyou could call ansible-freeipa20:22
fultonjlbragstad: https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L607-L62120:23
*** derekh has quit IRC20:23
fultonjthis passes params from THT to ansible as facts before doing an include_role to tripleo-ansible20:23
fultonjlbragstad: but if ceph-ansible fixes a bug i inherit it20:24
fultonji don't have to fix the same bug in tripleo-ansible20:24
lbragstadsure - that makes sense20:25
*** dsneddon_ has joined #tripleo20:28
*** mcornea has joined #tripleo20:33
fultonjlbragstad: ok, great. feel free to ask me more quesitons20:34
openstackgerritMerged openstack/tripleo-quickstart-extras master: Add VolumesSnapshotTestJSON test_volume_from_snapshot train skiplist  https://review.opendev.org/70972520:47
openstackgerritMerged openstack/tripleo-heat-templates master: Adding Rear check to CI  https://review.opendev.org/69069820:47
openstackgerritMerged openstack/os-net-config master: We need to use the MAC instead of PCI Address for Mellanox cards  https://review.opendev.org/70642920:47
openstackgerritwes hayutin proposed openstack/tripleo-ci master: update tox to min version py36  https://review.opendev.org/71086920:47
*** dsneddon_ has quit IRC20:48
*** dsneddon has quit IRC20:50
*** dsneddon has joined #tripleo20:50
lbragstadfultonj this is probably a dumb question, but i'm looking at everything in /var/lib/mistral/overcloud (my overcloud config-data?)20:50
fultonjlbragstad: i don't undersatnd your question20:51
lbragstadhypothetically, if i wanted to pull out the control_virtual_ip in global_vars.yaml20:51
lbragstadthat would be an ansible fact?20:51
fultonjlbragstad: so you see in your config-download directory a fact already set in global_vars.yaml20:52
fultonjyour tripleo-ansible task will have access to it but you then need to pass that to your free-ipa task20:53
*** jmasud has quit IRC20:53
fultonjmake a disintion between the calling ansible (config-download) and the called ansible (ansible-freeipa)20:53
lbragstadok20:54
lbragstadso - it sounds like it's possible20:54
fultonjhttps://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_work_dir/tasks/prepare.yml20:54
fultonjlbragstad: i don't know how ansible-freeipa works20:55
fultonjbut ceph-ansible expects a group_vars dir20:55
*** jmasud has joined #tripleo20:55
fultonjso i create it and populate it20:55
fultonjhttps://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_work_dir/tasks/prepare.yml#L44-L4520:55
fultonjlbragstad: you're basically writing a task to do what freeipa-ansible have to do manually to get read to use it20:56
fultonjif you use that pattern20:56
openstackgerritMerged openstack/puppet-tripleo master: Remove wsgi enabled parameters in nova  https://review.opendev.org/71064220:57
openstackgerritMerged openstack/puppet-tripleo master: Add replication_probe_interval for ovsdbs  https://review.opendev.org/71042420:57
openstackgerritMerged openstack/puppet-tripleo stable/train: Enable sudo rule creation  https://review.opendev.org/70910320:57
lbragstadfultonj ok - that helps20:57
fultonjto do what freeipa-ansible _USERS_ have to do manually20:57
* fultonj left out users20:57
lbragstadi guess the main thing i'm wondering is what information present in /var/lib/mistral/overcloud is available for us to use20:57
fultonjthe calling ansible has it all20:59
fultonjthe called ansible has what you select out to pass it20:59
* lbragstad nods21:00
fultonjlbragstad: but if you're dealing with existing variables for which Heat was auhtoritative and was passed to novajoin you don't want to break those variables21:00
fultonjbreak support for them i mean21:00
lbragstadso - we'll need to provide a mapping from what's in config-download to what we need to invoke with freeipa-ansible21:00
fultonjyeah21:00
fultonjlbragstad: but look to heat21:00
fultonjnot saying you can't use ansible of course21:00
fultonjbut maybe there's stuff in heat you want to retain backwards compatability with21:01
lbragstadoh - you're saying using the variables defined already in heat templates as opposed to pulling the same thing from config-download data?21:01
fultonjlbragstad: yes, consider that21:01
lbragstadgot it21:01
fultonjlbragstad: you asked about network data21:02
fultonjhttps://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L423-L42621:02
fultonjthat public_network declration is a part of a larger map of vars21:02
fultonjhttps://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L38621:02
fultonjwhich get set in the calling ansible here21:03
fultonjhttps://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L61021:03
lbragstadaha21:03
lbragstadand you're using that here - https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_work_dir/tasks/prepare.yml#L4521:04
fultonjlbragstad: exactly21:04
fultonjand the pattern it follows is what we'd expect for any ceph-ansible user21:04
fultonjhttps://github.com/ceph/ceph-ansible/blob/master/group_vars/all.yml.sample21:04
fultonjso the interfaces are consistent for both projects21:05
*** dking_desktop has joined #tripleo21:05
lbragstadok - so we have the ability to take an approach like that, which is good, but it also looks like we can dig through the config-data if there's something else we need21:05
fultonjand you can tar up a dir and ask the project for support as if you had set those vars by hand like everyone else :)21:05
fultonjlbragstad: yeah, i don't see why not21:06
lbragstadok - cool21:06
fultonjlbragstad: i can't ensure that myself21:06
fultonjbut i can't think of a reason it wouldn't work21:06
fultonji've been doing this since mistral and before config-download21:07
openstackgerritMerged openstack/tripleo-heat-templates master: Remove outdated comment about enabled dashboard  https://review.opendev.org/71060521:09
openstackgerritMerged openstack/tripleo-heat-templates master: Run the swift_rsync container unprivileged  https://review.opendev.org/71022521:09
openstackgerritDavid Vallee Delisle proposed openstack/os-net-config stable/train: We need to use the MAC instead of PCI Address for Mellanox cards  https://review.opendev.org/71087221:09
openstackgerritwes hayutin proposed openstack/tripleo-ci master: update tox to min version py36  https://review.opendev.org/71086921:09
openstackgerritwes hayutin proposed openstack/tripleo-ci master: Add centos8 support for emit_releases_script  https://review.opendev.org/71044121:11
*** pbandark has quit IRC21:11
*** Tengu has quit IRC21:13
*** Tengu has joined #tripleo21:15
openstackgerritMerged openstack/tripleo-heat-templates master: Disable Mistral dashboard  https://review.opendev.org/71060621:16
*** fultonj has quit IRC21:30
ade_leeEmilienM, mwhahaha , fultonj , weshay|ruck -- I guess it comes down to the fact that this new role is doing a lot of stuff that was previously in a different project -- https://opendev.org/x/novajoin which was not under tripleo21:30
ade_leeand I  can understand not wanting to keep that all in tripleo-ansible ..21:31
*** ansmith has quit IRC21:36
weshay|ruckade_lee, https://review.opendev.org/#/c/700226/21:42
*** ahosam has joined #tripleo21:42
*** ahosam has quit IRC21:51
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Add centos-8 scenario000-multinode update job  https://review.opendev.org/71087621:54
*** matbu has quit IRC22:05
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: DNM: Add centos-8 scenario000-multinode update job  https://review.opendev.org/71087622:05
*** matbu has joined #tripleo22:06
weshay|ruckrlandy, thanks updated doc22:08
*** slaweq has quit IRC22:09
*** bfournie has quit IRC22:10
*** ansmith has joined #tripleo22:10
rlandyweshay|ruck: we should add tempest to work-items22:10
*** mcornea has quit IRC22:11
*** mcornea has joined #tripleo22:11
rlandyhttps://review.rdoproject.org/r/#/c/25325/ - testproject on standalone tempest22:11
*** slaweq has joined #tripleo22:11
*** slaweq has quit IRC22:16
weshay|ruckmwhahaha, afaik.. no bugs yet on the undercloud hang in centos-8 right?22:18
mwhahahaCorrect22:19
mwhahahaStill working on determining the issue22:19
mwhahahaI'm getting differy hangs22:19
mwhahahaI think it's related to python3-keyring or some related package22:19
mwhahahaBut still trying things22:19
*** jdandrea has quit IRC22:20
mwhahaha Testing an older version of python-secretstorage at the moment22:20
rlandyhttps://bugs.launchpad.net/tripleo/+bug/186557422:26
openstackLaunchpad bug 1865574 in tripleo "centos-8 multinode and undercloud jobs are hanging on the undercloud install " [Critical,Triaged]22:26
rlandyweshay|ruck: mwhahaha: ^^22:27
rlandymarked as promotion-blocker22:27
*** tkajinam has joined #tripleo22:28
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Create a pre-deploy-plan playbook  https://review.opendev.org/71047422:28
weshay|ruckrlandy, thanks22:28
*** slaweq has joined #tripleo22:33
weshay|ruckrlandy, can we chat about ovb for 2 min22:34
weshay|ruckrlandy, can you point me at the stack we create?22:34
rlandyweshay|ruck: yeah22:34
*** jbadiapa has quit IRC22:37
*** jcoufal has quit IRC22:38
*** slaweq has quit IRC22:39
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Creates container_config_data module  https://review.opendev.org/71088222:39
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Creates container_config_data module  https://review.opendev.org/71088222:40
mwhahahaweshay|ruck, rlandy: commented on Bug 1865574 with what I have found today22:48
openstackbug 1865574 in tripleo "centos-8 multinode and undercloud jobs are hanging on the undercloud install " [Critical,Triaged] https://launchpad.net/bugs/186557422:48
rlandymwhahaha: thanks22:48
*** bdodd has quit IRC22:53
*** bdodd has joined #tripleo22:54
*** jcoufal has joined #tripleo22:58
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_expand_roles  https://review.opendev.org/70641723:01
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_check_existing  https://review.opendev.org/70704323:01
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_populate_environment  https://review.opendev.org/70729523:01
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module metalsmith_instances  https://review.opendev.org/70641823:01
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: Playbooks for overcloud node (un)provision  https://review.opendev.org/70704423:01
*** rcernin has joined #tripleo23:01
*** paramite|clone has joined #tripleo23:04
*** paramite has quit IRC23:04
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Creates container_config_data module  https://review.opendev.org/71088223:05
*** jcoufal has quit IRC23:11
*** slaweq has joined #tripleo23:11
*** slaweq has quit IRC23:16
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799023:16
*** ekultails has quit IRC23:18
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Creates container_config_data module  https://review.opendev.org/71088223:18
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799023:20
*** gfidente|afk has quit IRC23:21
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Fix prepare-slave playbook for centos8 OVB  https://review.opendev.org/70986623:27
openstackgerritMerged openstack/tripleo-heat-templates master: Restart ovn-dbs resource to take new VIP property.  https://review.opendev.org/71052723:27
openstackgerritwes hayutin proposed openstack/tripleo-operator-ansible master: WIP: update undercloud_local_interface for centos-8 w/o ifnames=0  https://review.opendev.org/71088423:32
weshay|ruckrlandy, ^23:33
*** Tengu has quit IRC23:35
*** Tengu has joined #tripleo23:36
*** mcornea has quit IRC23:38
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Add alternative local_interface for centos-8 undercloud OVB  https://review.opendev.org/71088723:39
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Creates container_config_data module  https://review.opendev.org/71088223:40
rlandyweshay|ruck: ^^23:41
weshay|ruckaye23:48
EmilienMwe'll have to migrate molecule jobs to centos8 at some point as well23:52
EmilienMbut i guess it's in the backlog23:52
weshay|ruckEmilienM, sorin is on it..but he just scheduled pto too23:54
weshay|rucklolz23:54
openstackgerritMerged openstack/tripleo-ansible stable/train: filters: make get_domain_id more robust  https://review.opendev.org/71044923:56
*** tosky has quit IRC23:57
weshay|ruckmwhahaha, hrm.. maybe zuul is prepped for collections now? https://opendev.org/openstack/ansible-collections-openstack23:59
* weshay|ruck reading through it 23:59
« Sunday, 2020-03-01 Index Tuesday, 2020-03-03 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!