Tuesday, 2017-12-12

*** ameade has joined #openstack-infra00:00
*** timburke_ has quit IRC00:00
*** betherly has joined #openstack-infra00:00
*** stevebaker has quit IRC00:01
*** RuiChen has quit IRC00:01
openstackgerritPaul Belanger proposed openstack-infra/puppet-ptgbot master: Add pip as requirement  https://review.openstack.org/52727200:01
*** bobh has joined #openstack-infra00:02
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714400:02
openstackgerritJames E. Blair proposed openstack-infra/infra-specs master: Add a spec for top-level project hosting  https://review.openstack.org/52402400:02
*** melwitt has joined #openstack-infra00:02
*** timburke has joined #openstack-infra00:02
*** melwitt is now known as Guest8586900:03
jeblairttx, clarkb, frickler: ^ i think that addresses outstanding comments00:03
*** sree has joined #openstack-infra00:04
*** slaweq has quit IRC00:04
*** xarses_ has quit IRC00:04
*** tdasilva has joined #openstack-infra00:05
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Clean up mirror in tripleo-test-cloud-rh1  https://review.openstack.org/52727400:05
*** mwhahaha has joined #openstack-infra00:05
*** bobh_ has joined #openstack-infra00:07
*** simonmcc has quit IRC00:07
*** bobh has quit IRC00:07
*** simonmcc has joined #openstack-infra00:08
*** spzala has joined #openstack-infra00:10
*** fdegir has joined #openstack-infra00:11
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714400:11
*** sree has quit IRC00:11
*** DuncanT has joined #openstack-infra00:11
*** bobh_ has quit IRC00:12
*** jascott1 has quit IRC00:14
*** spzala has quit IRC00:14
*** bobh has joined #openstack-infra00:16
*** baoli_ has quit IRC00:17
*** baoli has joined #openstack-infra00:17
openstackgerritMerged openstack-infra/system-config master: status.o.o xenial upgrade  https://review.openstack.org/52697500:18
*** bobh has quit IRC00:21
*** baoli has quit IRC00:22
*** spzala has joined #openstack-infra00:23
*** iyamahat has quit IRC00:23
*** Guest85869 is now known as melwitt00:26
*** sree has joined #openstack-infra00:28
*** kjackal_ has joined #openstack-infra00:28
*** spzala has quit IRC00:28
*** ganso has quit IRC00:29
*** portdirect has joined #openstack-infra00:29
*** kjackal has quit IRC00:30
*** sree has quit IRC00:32
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714400:33
*** spzala has joined #openstack-infra00:34
openstackgerritMerged openstack-infra/puppet-ptgbot master: Add pip as requirement  https://review.openstack.org/52727200:34
*** caphrim007 has quit IRC00:34
*** caphrim007 has joined #openstack-infra00:35
*** Apoorva has quit IRC00:38
*** spzala has quit IRC00:38
*** Apoorva has joined #openstack-infra00:38
*** caphrim007 has quit IRC00:40
*** bobh has joined #openstack-infra00:41
*** spzala has joined #openstack-infra00:42
*** sdague has quit IRC00:42
*** stevebaker has joined #openstack-infra00:43
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714400:46
*** bobh has quit IRC00:46
*** spzala has quit IRC00:46
*** stevebaker has quit IRC00:56
*** salv-orlando has joined #openstack-infra00:56
*** SumitNaiksatam has quit IRC00:56
*** spzala has joined #openstack-infra00:57
*** gyee has quit IRC00:57
openstackgerritJeremy Stanley proposed openstack-infra/puppet-subunit2sql master: Don't hardcode the path to pip  https://review.openstack.org/52728001:00
*** salv-orlando has quit IRC01:00
*** spzala has quit IRC01:02
*** spzala has joined #openstack-infra01:05
*** spzala has quit IRC01:09
*** yamahata has joined #openstack-infra01:10
*** spzala has joined #openstack-infra01:10
*** rfolco has quit IRC01:15
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714401:15
*** bobh has joined #openstack-infra01:15
*** spzala has quit IRC01:20
*** bobh has quit IRC01:20
*** jascott1 has joined #openstack-infra01:21
*** yamahata has quit IRC01:21
*** spzala has joined #openstack-infra01:23
*** spzala has quit IRC01:28
*** cuongnv has joined #openstack-infra01:29
*** Apoorva has quit IRC01:30
*** spzala has joined #openstack-infra01:30
*** cshastri has joined #openstack-infra01:31
*** aeng has quit IRC01:33
*** spzala has quit IRC01:35
*** spzala has joined #openstack-infra01:38
*** bobh has joined #openstack-infra01:39
*** stevebaker has joined #openstack-infra01:39
*** yamamoto has joined #openstack-infra01:41
*** spzala has quit IRC01:42
*** bobh has quit IRC01:43
*** liujiong has joined #openstack-infra01:44
*** stevebaker has quit IRC01:44
*** caphrim007 has joined #openstack-infra01:45
*** spzala has joined #openstack-infra01:45
*** smatzek has joined #openstack-infra01:46
*** andreas_s has joined #openstack-infra01:46
*** baoli has joined #openstack-infra01:47
*** caphrim007 has quit IRC01:49
*** smatzek has quit IRC01:50
*** aeng has joined #openstack-infra01:50
*** andreas_s has quit IRC01:50
*** spzala has quit IRC01:50
*** spzala has joined #openstack-infra01:52
*** stevebaker has joined #openstack-infra01:53
*** gouthamr has joined #openstack-infra01:53
*** aeng has quit IRC01:55
*** aeng has joined #openstack-infra01:55
*** spzala has quit IRC01:56
*** bobh has joined #openstack-infra01:57
*** salv-orlando has joined #openstack-infra01:57
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714401:57
*** Swami has quit IRC01:58
*** namnh has joined #openstack-infra01:58
*** jhesketh has quit IRC01:59
*** slaweq has joined #openstack-infra02:00
*** bobh has quit IRC02:01
*** RuiChen has joined #openstack-infra02:01
*** salv-orlando has quit IRC02:02
*** spzala has joined #openstack-infra02:04
*** slaweq has quit IRC02:05
*** spzala has quit IRC02:09
*** bobh has joined #openstack-infra02:09
*** stevebaker has quit IRC02:10
*** Apoorva has joined #openstack-infra02:12
*** bobh has quit IRC02:13
*** stevebaker has joined #openstack-infra02:14
*** annp has joined #openstack-infra02:15
*** mriedem has quit IRC02:15
*** bobh has joined #openstack-infra02:17
*** spzala has joined #openstack-infra02:18
*** bobh has quit IRC02:21
*** stevebaker has quit IRC02:21
*** zhurong has joined #openstack-infra02:21
*** spzala has quit IRC02:22
*** bobh has joined #openstack-infra02:27
*** jcoufal has joined #openstack-infra02:29
*** spzala has joined #openstack-infra02:31
*** stevebaker has joined #openstack-infra02:33
*** stevebaker has quit IRC02:33
*** bobh has quit IRC02:33
*** stevebaker has joined #openstack-infra02:33
*** stevebaker has quit IRC02:38
*** bobh has joined #openstack-infra02:38
*** stevebaker has joined #openstack-infra02:39
*** vivsoni has joined #openstack-infra02:40
*** dmellado has quit IRC02:42
*** bobh has quit IRC02:42
*** stevebaker has quit IRC02:43
*** SumitNaiksatam has joined #openstack-infra02:44
*** rlandy has quit IRC02:44
*** dmellado has joined #openstack-infra02:48
*** ijw has joined #openstack-infra02:49
*** r-daneel has quit IRC02:50
*** larainema has joined #openstack-infra02:50
*** ijw has quit IRC02:54
*** baoli has quit IRC02:54
*** baoli has joined #openstack-infra02:55
vivsoniHi Team02:57
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714402:58
*** Goneri has quit IRC02:58
*** baoli has quit IRC03:00
*** armaan has quit IRC03:02
*** liujiong has quit IRC03:02
*** armaan has joined #openstack-infra03:02
*** baoli has joined #openstack-infra03:03
*** nicolasbock has quit IRC03:05
*** liujiong has joined #openstack-infra03:06
*** bobh has joined #openstack-infra03:06
*** ianychoi has joined #openstack-infra03:08
*** aarefiev has quit IRC03:10
*** bobh has quit IRC03:10
*** signed8bit_Zzz is now known as signed8bit03:13
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Remove some of legacy jobs  https://review.openstack.org/52246503:18
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Remove some of legacy jobs on master  https://review.openstack.org/52729303:18
*** bobh has joined #openstack-infra03:19
*** HenryG has quit IRC03:19
*** HenryG has joined #openstack-infra03:20
*** bobh has quit IRC03:23
*** ramishra has joined #openstack-infra03:24
*** salv-orlando has joined #openstack-infra03:27
*** signed8b_ has joined #openstack-infra03:27
*** signed8bit has quit IRC03:30
*** salv-orlando has quit IRC03:32
*** Wei_Liu has quit IRC03:34
*** dmellado has quit IRC03:37
*** dmellado has joined #openstack-infra03:38
*** mikal_ has joined #openstack-infra03:39
*** mikal has quit IRC03:42
*** zhurong has quit IRC03:47
*** bobh has joined #openstack-infra03:48
*** signed8b_ is now known as signed8bit_Zzz03:50
*** Wei_Liu has joined #openstack-infra03:51
*** armax has quit IRC03:53
*** ykarel has joined #openstack-infra03:53
*** edmondsw has joined #openstack-infra03:53
*** armax has joined #openstack-infra03:53
*** armax has quit IRC03:54
*** bobh has quit IRC03:54
*** v1k0d3n has quit IRC03:56
*** v1k0d3n has joined #openstack-infra03:57
*** udesale has joined #openstack-infra04:00
*** slaweq has joined #openstack-infra04:01
*** bobh has joined #openstack-infra04:02
*** edmondsw has quit IRC04:02
*** slaweq has quit IRC04:06
*** bobh has quit IRC04:06
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Drop local fork of GitPython for 2.1.8 release  https://review.openstack.org/52729804:07
*** bobh has joined #openstack-infra04:09
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Drop local fork of GitPython for 2.1.8 release  https://review.openstack.org/52729804:10
*** markvoelker_ has joined #openstack-infra04:12
*** markvoelker has quit IRC04:12
*** bobh has quit IRC04:14
*** jcoufal has quit IRC04:15
*** caphrim007 has joined #openstack-infra04:17
*** jascott1 has quit IRC04:19
*** bobh has joined #openstack-infra04:19
*** bobh has quit IRC04:24
*** dbecker has quit IRC04:25
*** baoli has quit IRC04:26
*** salv-orlando has joined #openstack-infra04:28
*** coolsvap has joined #openstack-infra04:29
*** salv-orlando has quit IRC04:33
*** shu-mutou has joined #openstack-infra04:34
*** bobh has joined #openstack-infra04:37
*** jascott1 has joined #openstack-infra04:37
*** dbecker has joined #openstack-infra04:39
*** bobh has quit IRC04:41
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove unused setup_tables  https://review.openstack.org/52697004:50
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Support table prefix for sql reporter  https://review.openstack.org/52683504:50
*** gouthamr has quit IRC04:53
openstackgerritDavid Moreau Simard proposed openstack-infra/system-config master: WIP: Server reinstallation playbook  https://review.openstack.org/52730104:55
openstackgerritDavid Moreau Simard proposed openstack-infra/system-config master: WIP: Server reinstallation playbook  https://review.openstack.org/52730104:57
*** janki has joined #openstack-infra04:58
openstackgerritDavid Moreau Simard proposed openstack-infra/system-config master: WIP: Server reinstallation playbook  https://review.openstack.org/52730104:58
*** bobh has joined #openstack-infra04:59
*** jascott1 has quit IRC04:59
openstackgerritIan Wienand proposed openstack-infra/puppet-openstack_health master: Use v2.3.0 for nodejs  https://review.openstack.org/52730205:02
*** bobh has quit IRC05:04
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update to work with Ubuntu Xenial  https://review.openstack.org/52714405:04
*** spzala has quit IRC05:09
*** aeng has quit IRC05:10
*** spzala has joined #openstack-infra05:11
*** pgadiya has joined #openstack-infra05:14
*** spzala has quit IRC05:16
*** bobh has joined #openstack-infra05:17
*** edmondsw has joined #openstack-infra05:18
openstackgerritDavid Moreau Simard proposed openstack-infra/system-config master: WIP: Server reinstallation playbook  https://review.openstack.org/52730105:20
dmsimardfrickler: btw ^05:20
*** bobh has quit IRC05:22
*** edmondsw has quit IRC05:23
*** spzala has joined #openstack-infra05:24
*** iyamahat has joined #openstack-infra05:25
*** iyamahat has quit IRC05:26
*** bobh has joined #openstack-infra05:26
*** spzala has quit IRC05:29
*** baoli has joined #openstack-infra05:29
*** salv-orlando has joined #openstack-infra05:29
*** spzala has joined #openstack-infra05:30
openstackgerritTristan Cacqueray proposed openstack-infra/log-classify master: Add python project structure  https://review.openstack.org/52585805:31
openstackgerritTristan Cacqueray proposed openstack-infra/log-classify master: Import classify-log.crm logstash filter  https://review.openstack.org/52730505:31
*** bobh has quit IRC05:32
openstackgerritTristan Cacqueray proposed openstack-infra/puppet-log_processor master: Replace classify-log.crm local script by logclassify  https://review.openstack.org/52730605:32
openstackgerritTristan Cacqueray proposed openstack-infra/system-config master: logstash: use new logclassify settings  https://review.openstack.org/52730705:33
*** spzala has quit IRC05:34
*** salv-orlando has quit IRC05:34
*** andreas_s has joined #openstack-infra05:38
*** bobh has joined #openstack-infra05:38
*** robcresswell has quit IRC05:39
*** bobh has quit IRC05:42
*** andreas_s has quit IRC05:42
*** fungi has quit IRC05:45
*** baoli has quit IRC05:46
*** fungi has joined #openstack-infra05:47
*** jascott1 has joined #openstack-infra05:48
*** numans has quit IRC05:54
*** numans has joined #openstack-infra05:55
*** yamahata has joined #openstack-infra05:57
*** Apoorva has quit IRC05:58
*** jascott1 has quit IRC06:01
*** ykarel has quit IRC06:03
*** ykarel has joined #openstack-infra06:03
*** bobh has joined #openstack-infra06:05
*** spzala has joined #openstack-infra06:10
*** zhurong has joined #openstack-infra06:11
*** bobh has quit IRC06:11
*** dsariel has joined #openstack-infra06:11
*** SumitNaiksatam has quit IRC06:12
*** ykarel_ has joined #openstack-infra06:12
*** xinliang has quit IRC06:13
*** spzala has quit IRC06:14
*** dhill_ has quit IRC06:15
*** dhill_ has joined #openstack-infra06:15
*** ykarel has quit IRC06:15
*** bobh has joined #openstack-infra06:16
*** rocky-shiyan has quit IRC06:18
*** rocky-shiyan has joined #openstack-infra06:19
*** bobh has quit IRC06:22
*** eumel8 has joined #openstack-infra06:22
*** ijw has joined #openstack-infra06:22
*** eumel8 has quit IRC06:22
*** xinliang has joined #openstack-infra06:26
*** xinliang has quit IRC06:26
*** xinliang has joined #openstack-infra06:26
*** ijw has quit IRC06:27
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add cloud quota handling  https://review.openstack.org/50383806:28
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded  https://review.openstack.org/50305106:28
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional  https://review.openstack.org/50428206:28
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool  https://review.openstack.org/50428306:28
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool  https://review.openstack.org/50428406:28
*** salv-orlando has joined #openstack-infra06:30
openstackgerritTristan Cacqueray proposed openstack-infra/project-config master: Add log-classify check and gate jobs  https://review.openstack.org/52731306:30
*** salv-orlando has quit IRC06:34
openstackgerritAndy McCrae proposed openstack-infra/system-config master: Add Andy McCrae to manifests/users.pp  https://review.openstack.org/50417906:36
*** baoli has joined #openstack-infra06:44
*** baoli has quit IRC06:49
*** sshnaidm|off is now known as sshnaidm|ruck06:49
snapiriHi Infra, (noob question) I see something strange in my local.conf from the zuul job. In the following http://logs.openstack.org/53/527053/1/check/dragonflow-dsvm-fullstack-etcd-zmq/373d9ca/logs/local.conf.txt.gz I see a line that says "export OS_NO_CACHE=True", shouldn't it be withou the "export" statement?06:49
*** dhajare_ has joined #openstack-infra06:51
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add cloud quota handling  https://review.openstack.org/50383806:51
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded  https://review.openstack.org/50305106:51
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional  https://review.openstack.org/50428206:51
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool  https://review.openstack.org/50428306:51
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool  https://review.openstack.org/50428406:51
*** e0ne has joined #openstack-infra06:52
*** dtantsur|afk is now known as dtantsur06:52
*** hashar has joined #openstack-infra06:52
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: [WIP] Move prepare-zanata-client to o-z-j  https://review.openstack.org/52576006:53
*** myoung|rover|bbl is now known as myoung|rover06:53
*** dkehn has quit IRC06:54
*** dkehn has joined #openstack-infra06:54
iceyis there something weird going on with gerrit? I can connect, and get the TLS connection setup, but then the browser seems to hang at "Transferring data from review.openstack.org..."06:54
iceywell, curl seems to be OK with it so I'm guessing it's Firefox, ignore me06:55
iceylooks like it was a cache issue06:55
*** andreas_s has joined #openstack-infra06:59
*** andreas_s has quit IRC07:00
*** andreas_s has joined #openstack-infra07:01
*** dsariel has quit IRC07:01
*** udesale__ has joined #openstack-infra07:03
*** udesale__ has quit IRC07:04
*** udesale__ has joined #openstack-infra07:04
*** udesale has quit IRC07:04
*** bobh has joined #openstack-infra07:06
*** edmondsw has joined #openstack-infra07:07
*** bobh has quit IRC07:11
*** edmondsw has quit IRC07:12
*** andreas_s has quit IRC07:12
*** andreas_s has joined #openstack-infra07:13
*** fanzhang has left #openstack-infra07:13
fricklericey: well, gerrit does have its slow moments when it does garbage collection, please let us know when you see repeated issues07:14
iceyfrickler: hard refresh (shift-f5)  made it work fine; and curl has no problems with it, thanks!07:15
*** andreas_s has quit IRC07:17
fricklersnapiri: I think the export makes sense, because that option isn't used directly by devstack, but passed to the osc invocations07:18
*** adisky__ has joined #openstack-infra07:20
*** rcernin has quit IRC07:21
*** dsariel has joined #openstack-infra07:23
*** ijw has joined #openstack-infra07:28
*** bobh has joined #openstack-infra07:30
*** salv-orlando has joined #openstack-infra07:31
*** armaan has quit IRC07:33
*** armaan has joined #openstack-infra07:34
*** bobh has quit IRC07:34
*** xinliang has quit IRC07:34
*** salv-orlando has quit IRC07:35
*** Daisy_ has joined #openstack-infra07:39
*** Daisy_ has quit IRC07:39
*** jpena|off is now known as jpena07:39
*** robcresswell has joined #openstack-infra07:39
*** bobh has joined #openstack-infra07:41
*** salv-orlando has joined #openstack-infra07:44
*** bobh has quit IRC07:47
*** xinliang has joined #openstack-infra07:47
*** xinliang has quit IRC07:47
*** xinliang has joined #openstack-infra07:47
*** d0ugal has quit IRC07:51
*** sree has joined #openstack-infra07:51
AJaegerconfig-core, can we move forward with releasenotes, please? https://review.openstack.org/#/c/526850/ (reviewed by frickler already) and https://review.openstack.org/#/c/526851/07:54
*** jascott1 has joined #openstack-infra07:56
*** d0ugal has joined #openstack-infra07:57
AJaegerianw: do you want to remove the WIP from title of https://review.openstack.org/#/c/525760 ?08:00
*** bobh has joined #openstack-infra08:02
*** slaweq has joined #openstack-infra08:03
*** armaan has quit IRC08:03
rm_workis the gate having issues tonight? we're getting a TON of issues trying to merge some patches, done a few rechecks, between TIMEOUT and POST_FAILURE and nodes not starting correctly...08:05
rm_worklike this one: https://review.openstack.org/#/c/525790/08:06
*** bobh has quit IRC08:06
*** slaweq has quit IRC08:08
*** ykarel_ has quit IRC08:13
*** ykarel_ has joined #openstack-infra08:14
*** fanzhang has joined #openstack-infra08:14
*** slaweq has joined #openstack-infra08:14
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix line wrapping in github docs  https://review.openstack.org/52697108:15
*** florianf has joined #openstack-infra08:17
*** slaweq has quit IRC08:19
*** tesseract has joined #openstack-infra08:22
*** armaan has joined #openstack-infra08:23
ianwAJaeger: I just wanted to make sure it worked with the cached file08:24
*** Hal has joined #openstack-infra08:25
AJaegerianw: it's using the cached file now, isn't it? IT works...08:25
*** bobh has joined #openstack-infra08:27
*** ijw has quit IRC08:27
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Move prepare-zanata-client to o-z-j  https://review.openstack.org/52576008:28
*** bobh has quit IRC08:32
*** armaan has quit IRC08:32
*** armaan has joined #openstack-infra08:32
*** salv-orlando has quit IRC08:35
openstackgerritIan Wienand proposed openstack-infra/project-config master: Use prepare-zanata-client role  https://review.openstack.org/52734008:36
*** rcernin has joined #openstack-infra08:36
*** bobh has joined #openstack-infra08:39
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Use prepare-zanata-client role  https://review.openstack.org/52734008:40
*** amoralej|off is now known as amoralej08:41
AJaegerfrickler: could you review https://review.openstack.org/#/c/525760/ and 527340, please?08:41
*** dham1 has quit IRC08:42
*** geekinutah has quit IRC08:42
*** geekinutah has joined #openstack-infra08:42
AJaegermmh, what's up with zuulv3.openstack.org?08:42
AJaegerinfra-root, looks like Zuul is dead ;(08:42
*** ykarel_ is now known as ykarel|away08:42
AJaegerhttp://grafana.openstack.org/dashboard/db/zuul-status shows no jobs launched for the last 45 minutes but other graphs show signs of life08:42
*** ijw has joined #openstack-infra08:43
*** edwarnicke has quit IRC08:43
*** patricku has quit IRC08:43
*** bobh has quit IRC08:44
*** dham1 has joined #openstack-infra08:44
*** patricku has joined #openstack-infra08:46
*** baoli has joined #openstack-infra08:46
*** edwarnicke has joined #openstack-infra08:46
AJaeger#status notice Our CI system Zuul is currently not accessible. Wait with approving changes and rechecks until it's back online. Currently waiting for an admin to investigate.08:46
openstackstatusAJaeger: sending notice08:46
*** ykarel|away has quit IRC08:47
-openstackstatus- NOTICE: Our CI system Zuul is currently not accessible. Wait with approving changes and rechecks until it's back online. Currently waiting for an admin to investigate.08:48
AJaegerinfra-root, interesting, just received an email from Zuul - so, not completely dead?08:48
*** thorre_se has joined #openstack-infra08:48
*** bobh has joined #openstack-infra08:49
*** ykarel has joined #openstack-infra08:50
AJaegerinfra-root, zuulv3.openstack.org is back - temporary network problem?08:50
*** linkmark has joined #openstack-infra08:51
*** baoli has quit IRC08:51
*** thorre has quit IRC08:51
*** thorre_se is now known as thorre08:51
*** ralonsoh has joined #openstack-infra08:52
*** rossella_s has joined #openstack-infra08:52
*** bobh has quit IRC08:53
*** ccamacho has joined #openstack-infra08:55
*** edmondsw has joined #openstack-infra08:56
ykarelIs there some issue, i am seeing TIMED_OUT in jobs result and no logs(apart form consol.log) are uploaded to logs.openstack.org08:57
ykarelExample review: https://review.openstack.org/#/c/523741/08:57
*** rossella_s has quit IRC08:58
*** priteau has joined #openstack-infra09:00
*** edmondsw has quit IRC09:00
AJaegerinfra-root, http://grafana.openstack.org/dashboard/db/zuul-status?panelId=16&fullscreen looks still wrong09:00
AJaegerykarel: seems like there were/are network problems09:00
ykarelAJaeger, Ok09:01
*** salv-orlando has joined #openstack-infra09:01
*** rossella_s has joined #openstack-infra09:02
AJaeger#status notice Zuul is back online, looks like a temporary network problem.09:07
openstackstatusAJaeger: sending notice09:07
-openstackstatus- NOTICE: Zuul is back online, looks like a temporary network problem.09:08
openstackstatusAJaeger: finished sending notice09:10
*** bobh has joined #openstack-infra09:10
*** jappleii__ has quit IRC09:10
snapiriAJaeger: I see "server identification changed" message in the logs copying for my task (that obviously failed) http://logs.openstack.org/53/527053/1/check/tempest-dsvm-dragonflow/073a930/job-output.txt.gz, should I just recheck or would someone look into it?09:10
*** shardy has joined #openstack-infra09:12
*** jpich has joined #openstack-infra09:13
*** bobh has quit IRC09:15
AJaegersnapiri: no idea about that one ;( Even if you recheck we have the logs, so recheck is ok.09:17
*** dmellado has quit IRC09:17
*** strigazi_ is now known as strigazi09:17
fricklerinfra-root: zuul seems to have started swapping quite a bit, not sure if related to other possible networking issues, though it would be strange it that affected all clouds http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63981&rra_id=all09:20
fricklerseems we might have some new memory issue after yesterdays restart looking at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all09:22
*** baoli has joined #openstack-infra09:22
*** bobh has joined #openstack-infra09:23
*** abelur_ has joined #openstack-infra09:23
*** armaan has quit IRC09:24
*** armaan has joined #openstack-infra09:25
*** baoli has quit IRC09:26
*** bobh has quit IRC09:27
fricklerShrews: not directly related, but seeing tracebacks in zuul/web.log http://paste.openstack.org/show/628691/, that logging was added in https://review.openstack.org/507663 by you09:29
fricklerhmm, lots of errors in zuul.log, kazoo.exceptions.NoNodeError and "Node is not locked"09:32
*** bobh has joined #openstack-infra09:32
*** aarefiev has joined #openstack-infra09:34
*** lucas-afk is now known as lucasagomes09:35
*** rossella_s has quit IRC09:35
*** bobh has quit IRC09:37
*** rossella_s has joined #openstack-infra09:38
*** hemna_ has quit IRC09:42
*** ramishra has quit IRC09:43
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove python-glanceclient legacy jobs  https://review.openstack.org/52504609:43
*** ramishra has joined #openstack-infra09:44
*** derekh has joined #openstack-infra09:47
*** aidin has joined #openstack-infra09:48
*** bobh has joined #openstack-infra09:48
*** dmellado has joined #openstack-infra09:48
*** AJaeger has quit IRC09:51
*** AJaeger has joined #openstack-infra09:51
*** hemna_ has joined #openstack-infra09:52
*** bobh has quit IRC09:53
*** ganso has joined #openstack-infra09:54
*** gongysh has joined #openstack-infra09:55
*** annp has quit IRC09:56
*** zhurong has quit IRC09:57
*** jascott1 has quit IRC10:03
*** jhesketh has joined #openstack-infra10:04
*** gongysh has quit IRC10:05
*** janki has quit IRC10:07
*** dmellado has quit IRC10:07
*** namnh has quit IRC10:08
*** kiennt26 has quit IRC10:09
dulekHi! Is it possible to plot zuulv3 job stats using graphite.openstack.org?10:09
*** spzala has joined #openstack-infra10:11
*** bobh has joined #openstack-infra10:11
*** cuongnv has quit IRC10:11
*** ykarel_ has joined #openstack-infra10:12
*** spzala has quit IRC10:15
*** ykarel has quit IRC10:16
*** bobh has quit IRC10:17
*** slaweq__ is now known as slaweq10:18
*** dmellado has joined #openstack-infra10:20
*** bobh has joined #openstack-infra10:22
slaweqHi, how can I search in logstash.openstack.org for some text in job-output.txt.gz file?10:22
slaweqI made example query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20test_east_west%5C%22%20AND%20(build_name%3Aneutron-tempest-plugin-scenario-linuxbridge%20OR%20build_name%3Aneutron-tempest-plugin-dvr-multinode-scenario)%20AND%20build_branch%3Amaster10:22
slaweqbut it doesn't looks like it could search in this file10:22
*** baoli has joined #openstack-infra10:23
*** rpittau has joined #openstack-infra10:26
*** ykarel__ has joined #openstack-infra10:26
*** bobh has quit IRC10:27
*** zhurong has joined #openstack-infra10:27
*** baoli has quit IRC10:27
*** sambetts|afk is now known as sambetts10:28
*** liujiong has quit IRC10:29
*** bobh has joined #openstack-infra10:29
*** armaan has quit IRC10:29
*** armaan has joined #openstack-infra10:30
*** ykarel_ has quit IRC10:30
*** aidin has quit IRC10:32
*** bobh has quit IRC10:34
*** yamamoto has quit IRC10:35
*** fbo has joined #openstack-infra10:35
*** alexchadin has joined #openstack-infra10:36
*** bobh has joined #openstack-infra10:39
*** alexchadin has quit IRC10:39
*** sree has quit IRC10:39
*** alexchadin has joined #openstack-infra10:39
*** sdague has joined #openstack-infra10:39
sshnaidm|ruckwe have a lot of lost connections in tripleo OVB jobs on RH1 cloud: http://logs.openstack.org/03/525203/2/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/b8dd4d1/job-output.txt.gz#_2017-12-12_09_52_47_66203210:40
*** alexchadin has quit IRC10:40
*** ldnunes has joined #openstack-infra10:40
sshnaidm|ruckdoes anyone have a clue what could be wrong?10:40
*** alexchadin has joined #openstack-infra10:40
sshnaidm|ruckpabelanger, bnemec ^^10:42
*** alexchadin has quit IRC10:42
*** alexchadin has joined #openstack-infra10:43
*** bobh has quit IRC10:44
odyssey4meyeah, we're getting the same - looks like something's gone wrong with the log servers? http://logs.openstack.org/99/527099/2/check/openstack-ansible-linters/99828a9/job-output.txt.gz#_2017-12-12_10_28_51_73815010:45
*** rcarrillocruz has quit IRC10:46
AJaegersshnaidm|ruck, odyssey4me : We had earlier some problems, looked network related. It seems solved now, so please recheck - and if happens for jobs *started* in the last 60 mins, please tell again. Note you can always check our status at https://wiki.openstack.org/wiki/Infrastructure_Status10:47
odyssey4meAJaeger I literally just rechecked in the last 5 mins and got that error from a job that got rechecked10:48
*** yamamoto has joined #openstack-infra10:48
odyssey4meAJaeger oh wait, wrong one - ok thanks, let me look10:48
sshnaidm|ruckAJaeger, ack, will look10:48
*** cshastri has quit IRC10:50
*** dmellado has quit IRC10:50
*** tosky has joined #openstack-infra10:51
*** rcarrillocruz has joined #openstack-infra10:52
*** stakeda has quit IRC10:54
*** rossella_s has quit IRC10:54
*** rossella_s has joined #openstack-infra10:54
*** slaweq_ has joined #openstack-infra10:56
*** bobh has joined #openstack-infra10:57
*** shu-mutou is now known as shu-mutou-AWAY10:59
*** dmellado has joined #openstack-infra10:59
*** slaweq_ has quit IRC11:00
*** bobh has quit IRC11:01
*** rcarrillocruz has quit IRC11:02
*** rossella_s has quit IRC11:03
*** pblaho has joined #openstack-infra11:04
*** jascott1 has joined #openstack-infra11:04
*** stephenfin_ is now known as stephenfin11:05
*** rossella_s has joined #openstack-infra11:06
*** MasterofJOKers has quit IRC11:08
*** jascott1 has quit IRC11:08
*** ijw has quit IRC11:08
*** aidin has joined #openstack-infra11:11
*** MasterofJOKers has joined #openstack-infra11:11
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added new Endpoint Update Speaker  https://review.openstack.org/52737211:11
openstackgerritMerged openstack-infra/openstackid-resources master: Added new Endpoint Update Speaker  https://review.openstack.org/52737211:12
*** bobh has joined #openstack-infra11:12
*** aidin has quit IRC11:12
*** gongysh has joined #openstack-infra11:13
*** dmellado has quit IRC11:16
*** bobh has quit IRC11:16
*** bobh has joined #openstack-infra11:17
*** dmellado has joined #openstack-infra11:18
*** ociuhandu has joined #openstack-infra11:19
*** ociuhandu has quit IRC11:20
*** dmellado has quit IRC11:20
*** bobh has quit IRC11:21
*** aidin has joined #openstack-infra11:23
*** baoli has joined #openstack-infra11:24
*** dmellado has joined #openstack-infra11:24
*** zhurong has quit IRC11:25
*** alexchadin has quit IRC11:28
*** e0ne has quit IRC11:28
*** baoli has quit IRC11:28
*** alexchadin has joined #openstack-infra11:28
*** nicolasbock has joined #openstack-infra11:31
*** jkilpatr has quit IRC11:32
*** dmellado has quit IRC11:32
*** dmellado has joined #openstack-infra11:33
*** wolverineav has joined #openstack-infra11:35
*** bobh has joined #openstack-infra11:37
*** katkapilatova has joined #openstack-infra11:38
*** ykarel__ is now known as ykarel11:38
openstackgerritFabien Boucher proposed openstack-infra/zuul feature/zuulv3: Git driver  https://review.openstack.org/52561411:38
*** baoli has joined #openstack-infra11:40
*** bobh has quit IRC11:42
*** baoli has quit IRC11:44
*** Hal has quit IRC11:45
*** RuiChen has quit IRC11:45
*** smatzek has joined #openstack-infra11:45
*** Hal has joined #openstack-infra11:45
*** bobh has joined #openstack-infra11:46
*** bobh has quit IRC11:51
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoint upload speaker photo  https://review.openstack.org/52737811:53
*** yamamoto has quit IRC11:54
*** gmann_afk is now known as gmann11:55
*** yamamoto has joined #openstack-infra11:58
*** bobh has joined #openstack-infra11:58
*** ijw has joined #openstack-infra12:01
*** ijw_ has joined #openstack-infra12:01
*** ijw has quit IRC12:02
*** bobh has quit IRC12:03
*** yamamoto has quit IRC12:03
openstackgerritMerged openstack-infra/openstackid-resources master: Added new endpoint upload speaker photo  https://review.openstack.org/52737812:05
*** jkilpatr has joined #openstack-infra12:05
*** bobh has joined #openstack-infra12:06
*** ethfci has quit IRC12:07
*** lucasagomes is now known as lucas-hungry12:11
*** bobh has quit IRC12:11
*** bobh has joined #openstack-infra12:13
flaper87hey folks, is there anything wrong with this code? https://github.com/openstack/ansible-role-k8s-mariadb/blob/master/tests/pre.yml#L37-L44 It used to work fine but not anymore. The error I'm getting is that src_dir and/or short_name don't exist.12:16
flaper87http://logs.openstack.org/59/527059/1/check/ansible-role-k8s-glance-kubernetes-centos/657411d/job-output.txt.gz#_2017-12-12_06_16_17_84799312:16
flaper87jeblair: mordred dmsimard ^12:16
flaper87I noticed the var is in the inventory but there's also `_projects` so now I wonder if the one I'm using is just a property accessing the private one and whether I'm supposed to use it or not12:17
flaper87pabelanger: ^12:17
*** bobh has quit IRC12:17
*** ethfci has joined #openstack-infra12:18
*** bobh has joined #openstack-infra12:19
*** spzala has joined #openstack-infra12:19
*** rfolco has joined #openstack-infra12:19
*** ethfci has quit IRC12:21
*** hashar is now known as hasharAway12:22
*** bobh has quit IRC12:23
fricklerflaper87: zuul.projects is a hash now instead of a list, that was changed last week I think. let me look for a reference12:25
*** spzala has quit IRC12:25
*** bobh has joined #openstack-infra12:25
*** cshastri has joined #openstack-infra12:25
flaper87frickler: oh, mmh, I should have noticed that from the logs12:26
flaper87T_T12:26
flaper87it was obvious after all12:26
flaper87thanks frickler12:26
fricklerflaper87: my pleasure, I guess there should have been an announcement for that, but I fail to find it currently, maybe that is still due to be sent12:29
*** bobh has quit IRC12:30
openstackgerritJavier Peña proposed openstack-infra/system-config master: Support CentOS/RHEL for AFS mirror  https://review.openstack.org/52738712:31
*** jpena is now known as jpena|lunch12:32
*** yamamoto has joined #openstack-infra12:35
*** gongysh has quit IRC12:37
*** bobh has joined #openstack-infra12:37
*** baoli has joined #openstack-infra12:41
*** dhajare_ has quit IRC12:41
*** bobh has quit IRC12:41
*** aidin has quit IRC12:45
*** baoli has quit IRC12:45
*** alexchadin has quit IRC12:56
*** slaweq_ has joined #openstack-infra12:57
*** alexchadin has joined #openstack-infra12:57
*** bobh has joined #openstack-infra12:57
*** rcarrillocruz has joined #openstack-infra12:58
*** armaan has quit IRC12:59
*** zhurong has joined #openstack-infra12:59
*** slaweq_ has quit IRC13:01
*** bobh has quit IRC13:02
*** rosmaita has joined #openstack-infra13:04
*** rhallisey has joined #openstack-infra13:06
*** jascott1 has joined #openstack-infra13:06
*** jascott1 has quit IRC13:10
*** jcoufal has joined #openstack-infra13:11
*** zhurong has quit IRC13:14
*** weshay|sickday is now known as weshay13:15
*** e0ne has joined #openstack-infra13:15
*** armaan has joined #openstack-infra13:15
*** edmondsw has joined #openstack-infra13:16
*** lucas-hungry is now known as lucasagomes13:17
*** bobh has joined #openstack-infra13:17
Shrewsmorning all13:18
*** salv-orlando has quit IRC13:20
*** bobh has quit IRC13:22
*** amoralej is now known as amoralej|lunch13:23
*** alexchadin has quit IRC13:25
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added private serializer for Speakers  https://review.openstack.org/52739713:26
Shrewsfrickler: that CancelledError likely probably means that the websocket code (in this case, looks like maybe the finger connection to the executor) just gave up. maybe a timeout13:27
*** jpena|lunch is now known as jpena13:28
rosmaitagood morning, folks ... got a bunch of timed out, post failures on https://review.openstack.org/#/c/523028/ ... should i just recheck, or is there an incident happening?13:28
openstackgerritMerged openstack-infra/openstackid-resources master: Added private serializer for Speakers  https://review.openstack.org/52739713:28
*** signed8bit_Zzz is now known as signed8b_13:30
*** signed8b_ has quit IRC13:30
Shrewsrosmaita: just arrived myself, but looks like there were network issues earlier. should have been a notice from openstackstatus bot posted in all channels13:30
*** rlandy has joined #openstack-infra13:31
*** cshastri has quit IRC13:32
rosmaitaShrews ty13:32
*** hasharAway is now known as hashar13:33
dhellmanninfra-core, could you put https://review.openstack.org/#/c/525305/ on your review queue please? it is blocking the q-2 release for blazar.13:34
*** fbo has quit IRC13:36
*** pgadiya has quit IRC13:36
*** baoli has joined #openstack-infra13:37
*** kgiusti has joined #openstack-infra13:39
*** alexchadin has joined #openstack-infra13:39
*** gongysh has joined #openstack-infra13:42
*** andreas_s has joined #openstack-infra13:44
*** sbezverk has quit IRC13:44
*** dave-mccowan has joined #openstack-infra13:45
*** salv-orlando has joined #openstack-infra13:45
*** baoli has quit IRC13:46
*** baoli has joined #openstack-infra13:46
*** kiennt26 has joined #openstack-infra13:48
*** dave-mccowan has quit IRC13:48
*** coolsvap has quit IRC13:49
*** andreas_s has quit IRC13:50
*** bobh has joined #openstack-infra13:52
dmsimardlol what is this? https://github.com/openstack/ailuropoda13:54
odyssey4meit looks like the zuul dashboard has timed out several times today, and it seems to correlate to jobs ending up timing out if they're running through the dashboard timeout13:55
odyssey4meie job starts now, 5 mins later zuul dashboard times out, 5 mins later job times out13:55
odyssey4menot sure if they actually are related, but we've had a *lot* of timed_out job results today13:56
*** dave-mccowan has joined #openstack-infra13:56
*** dprince has joined #openstack-infra13:56
*** spzala has joined #openstack-infra13:57
dmsimardodyssey4me: the dashboard shouldn't have any impact on the jobs -- it was rolled out last friday FWIW13:57
*** bobh has quit IRC13:57
dmsimardodyssey4me: can you give an example or two ?13:57
*** bh526r has joined #openstack-infra13:58
*** mihalis68 has joined #openstack-infra13:58
ttxzuul a bit busy now13:58
ttxcurrently struggling giving status13:58
dmsimardwoah, yup. We're struggling on memory, taking a look.13:59
dmsimardinfra-root: zuulv3 is maxed on RAM13:59
odyssey4medmsimard check https://review.openstack.org/#/c/516448/ as an example - PS11's zuul result at 1:21pm13:59
dulekdmsimard: https://review.openstack.org/#/c/525502 - here's an example of timed out job13:59
dulekdmsimard: I have a few more if you're interested. ;)14:00
*** dave-mccowan has quit IRC14:00
odyssey4medmsimard also https://review.openstack.org/#/c/527333/ see the last two ci results14:00
*** slaweq_ has joined #openstack-infra14:00
*** bobh has joined #openstack-infra14:01
odyssey4meand the last two for https://review.openstack.org/#/c/510654/14:01
odyssey4methe list goes on ;)14:01
*** spzala has quit IRC14:01
*** spzala has joined #openstack-infra14:02
odyssey4meI suspected that it might be out of resources. Unfortunately that seems to be causing it to lose jobs.14:02
*** mriedem has joined #openstack-infra14:03
dmsimardSo there's something going on with the scheduler in terms of memory and it's quite obvious when looking at graphs -- it was flat at ~8GB RAM then spiked http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all14:03
odyssey4meit looks like this is a common error in them too: http://logs.openstack.org/54/510654/5/check/openstack-ansible-functional-ubuntu-xenial/031df0b/job-output.txt.gz#_2017-12-12_12_40_13_06167614:04
*** amoralej|lunch is now known as amoralej14:04
odyssey4meyowzer, that mem usage is bad14:05
*** bobh has quit IRC14:05
*** slaweq_ has quit IRC14:05
*** spzala has quit IRC14:06
*** tmorin has joined #openstack-infra14:06
odyssey4meand wow, logstash is showing a ton of "SSH Error: data could not be sent to remote host" messages in the last 24 hours, starting at around the same time as that memory usage spike14:07
*** sree has joined #openstack-infra14:07
*** trown|outtypewww is now known as trown14:08
fricklerShrews: yes, but it shouldn't fail with an AttributeError trying to log the CancelledError, I'd say14:10
*** hashar has quit IRC14:10
*** toabctl has quit IRC14:10
*** toabctl has joined #openstack-infra14:11
*** Goneri has joined #openstack-infra14:12
*** sree has quit IRC14:12
*** sree has joined #openstack-infra14:15
*** smatzek has quit IRC14:17
dmsimardinfra-root: How's this ? #status notice We're currently seeing an elevated rate of timeouts in jobs and the zuulv3.openstack.org dashboard is currently unresponsive, please stand by while we troubleshoot the issues.14:18
*** bobh has joined #openstack-infra14:20
odyssey4mezuul web isn't really unresponsive, it just dips in and out so it's "intermittently unresponsive"14:20
*** salv-orlando has quit IRC14:21
*** ykarel is now known as ykarel|afk14:23
*** yamamoto has quit IRC14:24
*** felipemonteiro_ has joined #openstack-infra14:24
*** yamamoto has joined #openstack-infra14:24
*** bobh has quit IRC14:25
*** tesseract has quit IRC14:26
fungiyeah, that seems reasonable14:26
*** felipemonteiro__ has joined #openstack-infra14:26
fungiit's using ~half its swap now14:26
clarkbdid it happen after the scheduler restart? would've been ~20 hours ago I think14:27
*** ihrachys has joined #openstack-infra14:27
fungi-rw-r--r-- 1 zuul zuul 5 Dec 11 17:45 /var/run/zuul/scheduler.pid14:28
*** ykarel|afk has quit IRC14:28
fungiyeah, looks like it14:28
fungiwell, hard to tell since its memory utilization _will_ ramp back up after a restart14:28
*** hashar has joined #openstack-infra14:28
fungibut it was spiking in jumps again like we see when people push stacks of config changes14:29
fungier, increasing not spiking14:29
fungijust cliff face increases rather than a steady slope14:29
clarkbgotcha14:29
EmilienMgood morning infra, can you please review this project import? https://review.openstack.org/#/c/527229/ - thanks14:29
*** felipemonteiro_ has quit IRC14:30
fungias far as i know we still have the issue that testing concurrent configuration changes will increase the amount of memory zuul needs due to requiring speculative copies of the changed configuration14:31
clarkbfungi: ya if you push a large number of them at one time14:31
fungiso bug stacks of configuration changes could have been teh trigger14:31
fungis/bug/big/14:31
fungiat this point we really don't have a lot of choice other than to restart the scheduler14:32
fungisince python won't reduce its memory allocation14:32
*** bobh has joined #openstack-infra14:33
*** salv-orlando has joined #openstack-infra14:33
fungihopefully the status api is still responsive enough for us to grab a copy of the pipelines so we can reenqueue changes after the restart14:35
fungiyeah, looks like i got a copy of them no problem14:36
fungidid anybody want to check anything else before we restart?14:37
*** bobh has quit IRC14:37
dmsimard#status notice We're currently seeing an elevated rate of timeouts in jobs and the zuulv3.openstack.org dashboard is intermittently unresponsive, please stand by while we troubleshoot the issues.14:37
openstackstatusdmsimard: sending notice14:37
dmsimardfungi: I suppose jeblair would be interested14:38
fungilooks like we have a handful of tags in the release pipeline14:38
clarkbfungi: you might do a thread dump? I'm ot sure how helpful that was in debugging memory leaks before14:38
fungirelease jobs look a little starved for node allocations14:39
-openstackstatus- NOTICE: We're currently seeing an elevated rate of timeouts in jobs and the zuulv3.openstack.org dashboard is intermittently unresponsive, please stand by while we troubleshoot the issues.14:39
dmsimardjeblair also mentioned a tool to detect layout leaks14:39
fungii'm going to give the release team a heads up to pause approving any more so we don't get in unfortunate situation with release artifacts14:39
fungibut we probably do want to wait for the release pipeline to clear14:39
*** sree_ has joined #openstack-infra14:40
*** sree_ is now known as Guest1740614:40
*** hjensas has quit IRC14:40
* mordred waves and reports he still has whatever clarkb had14:40
openstackstatusdmsimard: finished sending notice14:40
*** alexchadin has quit IRC14:41
fungirelease team has been notified14:42
*** sree has quit IRC14:43
dmsimard"kill -USR2 6036", that does a dump ?14:44
clarkbdmsimard: yes shouldgo in the debug log14:44
fungiit does if 6036 is the pid of eth scheduler14:44
clarkb(I didn't check the pid but thats the idea at least, sigusr2 against zuul scheduler)14:45
fungi6036 doesn't seem to be a valid pid for anything on zuulv3.o.o at the moment14:45
dmsimardyeah I pulled that from the bash history14:45
dmsimardmoving to incident channel14:46
*** spzala has joined #openstack-infra14:47
*** gongysh has quit IRC14:47
*** marst has joined #openstack-infra14:50
*** dhill_ has quit IRC14:51
openstackgerritFabien Boucher proposed openstack-infra/zuul feature/zuulv3: Git driver  https://review.openstack.org/52561414:52
tmorinhi infraroot, the previous and the currently running of legacy-tempest-dsvm-networking-bgpvpn-bagpipe are blocked on devstack-gate-setup-host, not progressing --> timeout14:55
*** salv-orl_ has joined #openstack-infra14:56
tmorinis this an already known issue ?14:56
clarkbtmorin: yes, zuul is currently experiencing performance issues due to memory pressure14:56
clarkb(there was a notie about this ~18 minutes ago)14:56
*** bobh has joined #openstack-infra14:56
*** dhill_ has joined #openstack-infra14:58
*** mriedem has quit IRC14:58
*** gouthamr has joined #openstack-infra14:58
tmorinclarkb: ah yes, I see it now14:59
*** jtomasek has joined #openstack-infra14:59
*** salv-orlando has quit IRC14:59
tmorinclarkb: btwn is there a web page somewhere on which we can see the current openstackstatus ?14:59
tmorins/btwn/btw14:59
*** pbourke has quit IRC15:00
clarkbtmorin: there is. /me finds links15:00
*** rcernin has quit IRC15:00
clarkbtmorin: https://wiki.openstack.org/wiki/Infrastructure_Status and https://twitter.com/openstackinfra15:01
*** bobh has quit IRC15:01
*** armaan has quit IRC15:01
funginoticing that the wiki/twitter feed for that doesn't distinguish between log/notice/alert/ok15:02
*** pbourke has joined #openstack-infra15:02
*** armaan has joined #openstack-infra15:03
*** rbrndt has joined #openstack-infra15:06
*** mlavalle has joined #openstack-infra15:07
mlavallefungi: ping15:08
*** dave-mccowan has joined #openstack-infra15:08
*** mriedem has joined #openstack-infra15:09
tmorinclarkb: thanks!15:10
*** armaan_ has joined #openstack-infra15:10
*** armaan has quit IRC15:10
*** cshastri has joined #openstack-infra15:11
*** smatzek has joined #openstack-infra15:13
*** dprince has quit IRC15:13
fungimlavalle: not sure what "ping" accomplishes... if you have a question, please ask it and i'll try to get to it when i'm free if nobody else beats me to it15:17
openstackgerritTung Doan proposed openstack-infra/project-config master: Add Apmec project to OpenStack  https://review.openstack.org/52724115:17
mlavallefungi: thanks. Thinking through my question. I'll get back to you15:18
mlavalleI am migrating legacy jobs to the Neutron repo in this patchset: https://review.openstack.org/#/c/522931/. Please look at the latest zuul error. Question is: do I need to migrate the tripleo-ci-centos-7-nonha-multinode-oooq job?15:22
mlavalleit is not executed for all the patches in Neutron. Only some15:22
mlavallefungi, AJaeger, clarkb: ^^^15:23
*** kiennt26 has quit IRC15:23
clarkbmlavalle: I think you should port any neutron specific jobs into neutron15:24
clarkbmlavalle: the tripleo job isn't neutron specific so likely should live in tripleo somewhere then neutron can run it15:24
mlavalleclarkb: how does it get executed in Neutron?15:25
mlavalleand only sometimes15:25
*** dhajare_ has joined #openstack-infra15:25
*** hjensas has joined #openstack-infra15:25
*** hjensas has quit IRC15:25
*** hjensas has joined #openstack-infra15:25
clarkbmlavalle: the combined zuul config of all projects is in a global (to openstack) configuration. So in your pipeline config you just add it where appropriate under eg check or gate. At that same location you can apply modifiers to affect when it runs (like only when these files change)15:26
clarkbjeblair: spec updates from yesterday lgtm15:27
mlavalleclarkb: can you point me to where it gets executed now?15:27
*** bobh has joined #openstack-infra15:27
clarkbmlavalle: you are asking for where in the config specifies it should run?15:29
mlavalleclarkb: no, it is getting execute now (before migration). Where is that execution defined?15:30
clarkbmlavalle: do you have a link to a job that ran? I believe the job itself records that info15:32
*** bobh has quit IRC15:32
*** baoli has quit IRC15:32
mlavalleclarkb: https://review.openstack.org/#/c/515062/15:32
clarkbmlavalle: http://logs.openstack.org/62/515062/3/check/legacy-tripleo-ci-centos-7-nonha-multinode-oooq/fbe70cf/zuul-info/inventory.yaml under _inheritance_path it shows you where the config comes from15:33
clarkbmlavalle: openstack-infra/project-config/zuul.d/projects.yaml@master#14190 in this case15:34
clarkbwhich of course has moved at this point /me digs more15:34
*** bobh has joined #openstack-infra15:35
hwoaranghello. could someone please get https://review.openstack.org/#/c/505657/ so bindep+opensuse tumbleweed to work properly? thank you15:37
*** slaweq is now known as slaweq_15:37
clarkbmlavalle: reading git logs I think I understand now. That legacy tripleo job was moved into the tripleo-ci tree and in that move no longer runs again neutron15:38
clarkbmlavalle: the job was also renamed to drop the legacy- prefix15:39
clarkbmlavalle: you probably want to coordinate with mwhahaha and EmilienM (EmilienM appears to have written the tripleo job chnages) on how to add that back to neutron if you would like to15:39
clarkbmlavalle: I'm trying to find a link to the historic config for you15:39
mwhahahawhich project?15:40
mwhahahai thought EmilienM had proposed moving intree for everything15:40
clarkbmwhahaha: neutron15:41
*** bobh has quit IRC15:41
mwhahahalooks like he missed that one15:41
openstackgerritJens Harbott (frickler) proposed openstack-infra/puppet-ethercalc master: Update ethercalc to work with Ubuntu Xenial  https://review.openstack.org/52714415:42
*** slaweq_ has quit IRC15:42
*** xarses_ has joined #openstack-infra15:42
mlavalleclarkb: thanks. that historic config will help15:44
AJaegerclarkb: please remember https://review.openstack.org/#/q/topic:retire-puppet-apps15:45
mlavallein the meantime, I'll coordinate with EmilienM15:45
*** caphrim007 has quit IRC15:45
*** slaweq_ has joined #openstack-infra15:45
*** aidin has joined #openstack-infra15:45
clarkbmlavalle: Ia27eb293d593fde5b850796d95e7ea11a9fa0f53 is the change, my brwoser is really unhappy with the diff though15:46
clarkbmlavalle: at least I think it is the change I havne't been able to confirm yet due to brwoser being sad over it15:46
*** bobh has joined #openstack-infra15:47
*** Guest17406 has quit IRC15:47
*** sree has joined #openstack-infra15:48
*** aidin has quit IRC15:50
mlavalleEmilienM: you around? I have a question about legacy-tripleo-ci-centos-7-nonha-multinode-oooq and Neutron15:50
*** slaweq_ has quit IRC15:50
*** e0ne_ has joined #openstack-infra15:51
*** bobh has quit IRC15:51
*** e0ne has quit IRC15:52
mlavalleEmilienM: I am migrating Neutron legacy jobs to the Neutron repo. I understand that I don't need to migrate the tripleo job. But currently it gets executed for some Neutron patches in the check queue. Do I need to add its execution to my new .zuul.yaml in Neutron?15:53
*** sree has quit IRC15:53
*** bobh has joined #openstack-infra15:56
clarkbmlavalle: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml?id=9795a88981810641d1f535401aa3ec303b445c1a#n12744 and https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml?id=9795a88981810641d1f535401aa3ec303b445c1a#n12887 is your historical config I Think15:56
*** yamahata has quit IRC15:58
*** sree has joined #openstack-infra16:00
*** bobh has quit IRC16:00
*** bobh has joined #openstack-infra16:01
EmilienMmlavalle: yeah, it would be great. In experimental pipeline at least, please16:02
*** dave-mccowan has quit IRC16:02
mlavalleclarkb: yes, it looks like it16:02
*** jascott1 has joined #openstack-infra16:08
openstackgerritPaul Belanger proposed openstack-infra/puppet-meetbot master: Support ubuntu xenial plugins directory  https://review.openstack.org/52744716:08
*** slaweq_ has joined #openstack-infra16:09
Jeffrey4lseems like zuulv3.o.o can not response now.16:09
dmsimardJeffrey4l: we are aware and are looking into it16:09
Jeffrey4lgot. thanks.16:10
*** armaan_ has quit IRC16:12
openstackgerritPaul Belanger proposed openstack-infra/puppet-meetbot master: Support ubuntu xenial plugins directory  https://review.openstack.org/52744716:13
*** cshastri has quit IRC16:13
*** jascott1 has quit IRC16:13
*** slaweq_ has quit IRC16:14
*** Apoorva has joined #openstack-infra16:15
*** sree has quit IRC16:16
mlavalleclarkb: thanks for help and time :-)16:18
clarkbmlavalle: no problem16:20
openstackgerritTung Doan proposed openstack-infra/project-config master: Add Apmec project to OpenStack  https://review.openstack.org/52724116:20
*** jtomasek has quit IRC16:23
*** jtomasek has joined #openstack-infra16:24
*** ijw_ has quit IRC16:25
*** armax has joined #openstack-infra16:26
*** slaweq_ has joined #openstack-infra16:26
*** dtantsur is now known as dtantsur|afk16:28
EmilienMdo we need PTL to approve new repos? if yes, clarkb you might want to see https://review.openstack.org/#/c/527229 - thanks16:30
*** slaweq_ has quit IRC16:31
*** sree has joined #openstack-infra16:31
*** felipemonteiro__ has quit IRC16:32
*** dsariel has quit IRC16:34
AJaegerEmilienM: for new repos that are part of an existing official project, the project PTL has to approve. Note that we have a long backlog of reviews...16:34
EmilienMAJaeger: yeah sorry16:34
EmilienMAJaeger: no PTL here16:34
*** sree has quit IRC16:36
*** e0ne_ has quit IRC16:37
*** armaan has joined #openstack-infra16:37
*** e0ne has joined #openstack-infra16:38
*** ijw has joined #openstack-infra16:38
*** e0ne has quit IRC16:38
*** e0ne has joined #openstack-infra16:38
*** e0ne has quit IRC16:39
*** e0ne has joined #openstack-infra16:40
*** e0ne has quit IRC16:40
*** ijw has quit IRC16:41
*** ijw_ has joined #openstack-infra16:41
*** e0ne has joined #openstack-infra16:41
*** andreas_s has joined #openstack-infra16:42
*** dsariel has joined #openstack-infra16:42
*** e0ne has quit IRC16:42
*** hamzy has quit IRC16:43
*** e0ne has joined #openstack-infra16:43
*** e0ne has quit IRC16:43
ttxoops looks like mordred's \u2026 is not really working for the Twitter bot16:43
ttxhttps://twitter.com/openstackinfra/status/94059226914868428816:44
*** jpena is now known as jpena|off16:44
*** r-daneel has joined #openstack-infra16:44
fungiunclean!16:44
fungi(unicode-unclean that is)16:45
*** slaweq_ has joined #openstack-infra16:45
*** ijw has joined #openstack-infra16:45
*** ijw_ has quit IRC16:45
*** Apoorva has quit IRC16:46
*** andreas_s has quit IRC16:47
*** mihalis68 has quit IRC16:47
*** dave-mccowan has joined #openstack-infra16:49
*** slaweq_ has quit IRC16:49
ttxfungi: I guess statusbot runs on python2 ?16:49
*** ijw has quit IRC16:50
*** gyee has joined #openstack-infra16:50
*** ijw has joined #openstack-infra16:50
fungilikely so16:51
*** pkovar has joined #openstack-infra16:51
fungicould also be some limitation/quirk of whatever the library interfacing with the twitter api is16:52
ttxno I looked up the code and it sends message + continuation to twitter16:52
ttxif continuation='\u2026' then it will gladly append '\\u2026' to your message16:53
ttxAs much as I want to blame Twitter for everything, this is probably not their fault16:53
*** pvaneck has joined #openstack-infra16:54
ttx(I mean it will append '\\u2026' if you run Python216:54
thiagolibHello, I have installed the os-loganalize project on my C.I log server but when I access a log.gz I'm getting the file not found error16:54
fungihah16:54
ttxand '…' if you run Python 316:54
*** ijw has quit IRC16:55
fungiyup, classic python 2.8 unicode handling. needs u'\u2026' probably16:55
fungier, 2.716:55
*** ijw has joined #openstack-infra16:55
fungiwhich will also work in py3k just fine16:56
*** ijw has quit IRC16:56
*** ijw has joined #openstack-infra16:56
ttxchecking and proposing patch16:56
fungii tested it with latest python2.7 and python3.3 up through 3.7 (beta) and print(u'\u2026') seems to behave the same on all of them16:57
ttxyes, that's what I get too16:57
fungithiagolib: you may be missing a relevant bit of apache configuration. just a sec and i'll get you a link to our config16:58
*** ijw has quit IRC16:59
*** caphrim007 has joined #openstack-infra17:00
*** ijw has joined #openstack-infra17:00
*** caphrim007 has quit IRC17:00
*** hashar has quit IRC17:00
fungithiagolib: have a look and see if you're missing anything from https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/templates/logs.vhost.erb17:00
*** caphrim007 has joined #openstack-infra17:00
*** ijw has quit IRC17:00
fungithere's a fair amount of type forcing and rewriting happening around gz stuff in there17:01
*** ijw has joined #openstack-infra17:01
openstackgerritThierry Carrez proposed openstack-infra/statusbot master: Fix continuation character under Python 2  https://review.openstack.org/52746117:01
*** ramishra has quit IRC17:02
*** rossella_s has quit IRC17:03
*** ijw has quit IRC17:04
*** ijw has joined #openstack-infra17:05
*** dave-mccowan has quit IRC17:05
*** ijw has quit IRC17:05
*** rossella_s has joined #openstack-infra17:05
*** ijw has joined #openstack-infra17:06
openstackgerritJeremy Stanley proposed openstack-infra/puppet-subunit2sql master: Don't hardcode the path to pip  https://review.openstack.org/52728017:06
*** ijw has quit IRC17:06
*** edmondsw has quit IRC17:06
*** ijw has joined #openstack-infra17:07
*** lucasagomes is now known as lucas-afk17:07
*** ijw has quit IRC17:09
*** ijw has joined #openstack-infra17:09
*** ijw has quit IRC17:11
*** ijw has joined #openstack-infra17:11
*** Apoorva has joined #openstack-infra17:12
*** ijw has quit IRC17:12
*** ijw_ has joined #openstack-infra17:12
*** ijw_ has quit IRC17:13
*** ijw has joined #openstack-infra17:13
*** jcoufal has quit IRC17:13
*** hamzy has joined #openstack-infra17:14
*** ijw has quit IRC17:14
*** ijw has joined #openstack-infra17:15
*** hjensas has quit IRC17:15
*** ijw has quit IRC17:15
*** ijw has joined #openstack-infra17:15
openstackgerritMichael Turek proposed openstack/diskimage-builder master: Add iscsi-boot element  https://review.openstack.org/51149417:15
*** Apoorva has quit IRC17:16
*** ijw has quit IRC17:16
*** ijw_ has joined #openstack-infra17:17
*** bh526r has quit IRC17:18
toskycan someone fix the channel topic on #openstack-meeting-3? It is stuck to "Open Discussion (Meeting topic: sahara)", even if the meeting (last Thursday) was closed correctly17:19
toskyprobably the issue was caused by the netsplit that followed shortly after that meeting17:19
*** jtomasek has quit IRC17:20
clarkbtosky: is there a meeting in progress? ( don't want to overwrite anything they have going on during a meeting )17:20
toskyclarkb: looking at the logs, no17:21
*** ijw_ has quit IRC17:21
clarkbok give me a sec17:21
*** tmorin has quit IRC17:21
fungii have it queued up already17:21
toskysure, no rush :)17:21
toskythanks17:21
fungifixed17:21
fungii fix them often enough they're in my server buffer history17:22
toskyeh eh17:22
clarkbheh we both changed it :)17:22
pabelangerha, I was looking for syntax myself17:22
pabelangerbut not in buffer like fungi17:22
openstackgerritEmilien Macchi proposed openstack-infra/openstack-zuul-jobs master: Revert "Add in branch case for puppet-ceph stable/jewel"  https://review.openstack.org/52746717:22
openstackgerritEmilien Macchi proposed openstack-infra/openstack-zuul-jobs master: Revert "Add in branch case for puppet-ceph stable/jewel"  https://review.openstack.org/52746717:22
fungi /msg chanserv topic #openstack-meeting-3 OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/17:23
fungipabelanger: ^17:23
pabelangerdanke!17:23
*** slaweq_ has joined #openstack-infra17:24
toskydoesn't the bot have some "fix-the-topic" command?17:24
*** dsariel has quit IRC17:24
fungiin a sense, yes. it's designed to assume whatever the topic was before a meeting started is what it should be changed back to17:25
*** ijw has joined #openstack-infra17:25
fungihowever, if it gets restarted mid-meeting that doesn't work out too well since its state is all in memory17:25
fungimore often though, we end up on the wrong side of a network services split during a meeting, and when chanserv comes back later _it_ resets the topic to the last one it saw before the split17:26
*** ijw has quit IRC17:26
*** ijw has joined #openstack-infra17:26
toskyI suspect that's what happened here17:26
fungifreenode's been doing maintenance on its service bots too, so might be similar behavior but not involving any netsplit17:27
*** baoli has joined #openstack-infra17:27
fungilike a chanserv restart straddling a #endmeeting or something17:27
*** slaweq_ has quit IRC17:28
*** ijw_ has joined #openstack-infra17:29
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Update files node to support numeric hostnames  https://review.openstack.org/52746917:29
*** ijw_ has quit IRC17:29
*** ijw_ has joined #openstack-infra17:30
*** Hal has quit IRC17:30
*** ijw_ has quit IRC17:30
*** ijw_ has joined #openstack-infra17:31
*** aspiers has quit IRC17:31
*** ijw has quit IRC17:31
openstackgerritEmilien Macchi proposed openstack-infra/openstack-zuul-jobs master: puppet-unit-4.8: override ZUUL_BRANCH after puppet-ceph/jewel checkout  https://review.openstack.org/52747417:31
*** vhosakot has joined #openstack-infra17:32
*** ijw has joined #openstack-infra17:33
*** ijw has quit IRC17:33
*** ijw has joined #openstack-infra17:33
*** ijw_ has quit IRC17:35
*** ijw has quit IRC17:35
*** ijw__ has joined #openstack-infra17:35
*** aspiers has joined #openstack-infra17:35
*** dhajare_ has quit IRC17:36
*** ijw has joined #openstack-infra17:36
*** dave-mccowan has joined #openstack-infra17:38
*** ralonsoh has quit IRC17:39
smcginnisIs zuul still being trouble shot?17:39
*** ijw__ has quit IRC17:40
*** david-lyle has joined #openstack-infra17:40
clarkbsmcginnis: yes I think jeblair is building an object graph via the back door repl in order to debug the memory leak17:40
clarkb(I think he has decided that there is indeed a memory leak)17:40
smcginnisclarkb: OK, thanks.17:40
* smcginnis goes back to coffee17:40
*** verdurin has quit IRC17:40
*** ijw has quit IRC17:41
jeblairyep, sorry.  it's still in progress.  as soon as i capture enough data, i'll restart it.17:41
*** ijw has joined #openstack-infra17:41
smcginnisjeblair: No worries, just wanted to make sure we didn't forget to send out an all clear.17:41
*** ijw has quit IRC17:42
*** kjackal_ has quit IRC17:42
*** ijw has joined #openstack-infra17:42
*** ijw has quit IRC17:42
*** ijw has joined #openstack-infra17:43
*** ijw has quit IRC17:44
*** verdurin has joined #openstack-infra17:44
*** ijw has joined #openstack-infra17:44
*** Apoorva has joined #openstack-infra17:44
*** ijw has quit IRC17:45
*** ijw has joined #openstack-infra17:45
*** ijw has quit IRC17:46
*** ijw has joined #openstack-infra17:46
*** ijw has quit IRC17:47
*** ijw_ has joined #openstack-infra17:47
*** ijw_ has quit IRC17:48
*** Apoorva has quit IRC17:49
*** Apoorva has joined #openstack-infra17:50
*** ijw has joined #openstack-infra17:50
*** Hal has joined #openstack-infra17:51
*** ijw has quit IRC17:51
*** ijw has joined #openstack-infra17:51
*** katkapilatova has left #openstack-infra17:52
*** katkapilatova has quit IRC17:52
*** ijw has quit IRC17:52
*** ijw has joined #openstack-infra17:52
*** andreas_s has joined #openstack-infra17:53
*** jpich has quit IRC17:53
*** ijw has quit IRC17:53
*** ijw_ has joined #openstack-infra17:53
*** ijw_ has quit IRC17:54
openstackgerritFrank Kloeker proposed openstack-infra/system-config master: Add Andy McCrae to manifests/users.pp  https://review.openstack.org/50417917:54
openstackgerritFrank Kloeker proposed openstack-infra/system-config master: Installation of i18n-checksite.openstack.org  https://review.openstack.org/48793617:54
*** ijw has joined #openstack-infra17:54
*** slaweq_ has joined #openstack-infra17:55
*** ijw has quit IRC17:55
*** ijw has joined #openstack-infra17:55
*** edmondsw has joined #openstack-infra17:56
*** ijw has quit IRC17:56
*** jascott1 has joined #openstack-infra17:56
*** ijw has joined #openstack-infra17:58
thiagolibhttps://www.irccloud.com/pastebin/Jo32mFTP/17:58
*** andreas_s has quit IRC17:58
*** ijw_ has joined #openstack-infra17:59
*** ijw has quit IRC17:59
*** slaweq_ has quit IRC17:59
*** ijw_ has quit IRC17:59
*** ijw has joined #openstack-infra17:59
*** Hal has quit IRC18:00
*** ijw has quit IRC18:00
*** vhosakot has quit IRC18:00
thiagolib@fungi  looking at the link you sent me the only different line is this18:00
*** ijw has joined #openstack-infra18:00
thiagolib<Directory /var/www/html/ci-logs/logs/periodic-internal/*>18:01
thiagolib     IndexOrderDefault Descending Date18:01
thiagolib  </Directory>18:01
*** shardy has quit IRC18:01
*** ijw has quit IRC18:01
*** ijw has joined #openstack-infra18:01
*** ijw has quit IRC18:02
fungithiagolib: okay, then that's probably not the issue18:02
*** sambetts is now known as sambetts|afk18:02
*** vhosakot has joined #openstack-infra18:02
clarkbfile permissions maybe?18:02
*** ijw has joined #openstack-infra18:02
fungiyou'd think the apache error would say permission/access denied rather than file not found in that case18:03
*** ijw has quit IRC18:03
fungia 401 rather than a 40418:03
*** ijw has joined #openstack-infra18:03
*** trown is now known as trown|lunch18:04
*** ijw has quit IRC18:04
*** rossella_s has quit IRC18:04
*** ijw has joined #openstack-infra18:04
*** edmondsw has quit IRC18:04
*** rossella_s has joined #openstack-infra18:04
*** ijw has quit IRC18:05
*** ijw has joined #openstack-infra18:05
*** ijw has quit IRC18:06
*** ijw has joined #openstack-infra18:06
*** ijw has quit IRC18:07
*** hamzy has quit IRC18:07
*** ijw has joined #openstack-infra18:07
*** ijw has quit IRC18:08
*** nicolasbock has quit IRC18:08
*** ccamacho has quit IRC18:08
*** ijw has joined #openstack-infra18:08
*** ijw has quit IRC18:09
*** ijw has joined #openstack-infra18:09
*** hjensas has joined #openstack-infra18:10
*** hjensas has quit IRC18:10
*** hjensas has joined #openstack-infra18:10
*** ijw has quit IRC18:10
*** ijw has joined #openstack-infra18:10
*** ijw has quit IRC18:11
*** ijw has joined #openstack-infra18:11
*** ijw has quit IRC18:12
*** ijw has joined #openstack-infra18:12
*** ijw has quit IRC18:13
*** ijw has joined #openstack-infra18:13
*** jascott1 has quit IRC18:13
zxiiroAnyone here familiar with Heat? We noticed if you spin up a flavor in Nova with 0 Disk it creates an emphemeral disk with the same size as the image. However with Heat it forces you to create a Volume. Wondering if anyone's aware of a way to force heat to work like Nova does?18:13
*** ijw has quit IRC18:13
*** ijw has joined #openstack-infra18:14
clarkbzxiiro: nope we've never really used heat beacuse it was region specific and many clouds didn't deploy it (so our many cloud regions in many clouds meant it was significant effort to use). You may have better luck in the heat channel or on the mailing list?18:14
*** ijw has quit IRC18:14
zxiiroclarkb: ok thanks18:14
fungihttps://wiki.openstack.org/wiki/IRC says #heat is their channel, btw18:15
fungi(no openstack- prefix)18:15
*** ijw has joined #openstack-infra18:15
*** ijw has quit IRC18:15
*** ijw has joined #openstack-infra18:16
*** ijw has quit IRC18:16
*** ijw has joined #openstack-infra18:17
*** udesale__ has quit IRC18:17
*** ijw has quit IRC18:18
*** ijw has joined #openstack-infra18:18
*** ijw has quit IRC18:18
*** yamahata has joined #openstack-infra18:18
*** ijw has joined #openstack-infra18:19
*** ijw has quit IRC18:19
*** ijw has joined #openstack-infra18:20
*** ijw has quit IRC18:20
*** electrofelix has quit IRC18:20
*** ijw has joined #openstack-infra18:20
*** ijw has quit IRC18:21
*** ijw has joined #openstack-infra18:22
*** raissa has joined #openstack-infra18:22
*** ijw has quit IRC18:23
*** pkovar has left #openstack-infra18:23
*** salv-orl_ has quit IRC18:23
*** ijw has joined #openstack-infra18:23
*** ijw has quit IRC18:24
*** ijw has joined #openstack-infra18:24
*** jascott1 has joined #openstack-infra18:25
*** ijw_ has joined #openstack-infra18:26
*** ijw has quit IRC18:26
*** ijw_ has quit IRC18:26
*** ijw has joined #openstack-infra18:27
*** sree has joined #openstack-infra18:27
*** ijw has quit IRC18:27
*** ijw has joined #openstack-infra18:28
*** ijw has quit IRC18:28
*** ijw has joined #openstack-infra18:28
*** ijw has quit IRC18:29
*** ijw has joined #openstack-infra18:29
*** ijw has quit IRC18:30
*** ijw has joined #openstack-infra18:30
*** ijw has quit IRC18:31
*** slaweq_ has joined #openstack-infra18:31
*** ijw has joined #openstack-infra18:31
*** ijw has quit IRC18:32
*** sree has quit IRC18:32
*** ijw has joined #openstack-infra18:32
*** bbc1__ has quit IRC18:32
*** rbrndt has quit IRC18:32
*** ijw has quit IRC18:33
*** ijw has joined #openstack-infra18:33
*** ijw has quit IRC18:33
*** hjensas has quit IRC18:34
*** ijw has joined #openstack-infra18:34
*** rlandy is now known as rlandy|brb18:34
*** bbc1__ has joined #openstack-infra18:34
*** ijw has quit IRC18:35
*** ijw has joined #openstack-infra18:35
*** tosky has quit IRC18:35
*** slaweq_ has quit IRC18:35
*** ijw has quit IRC18:36
*** Wei_Liu has quit IRC18:36
*** ijw has joined #openstack-infra18:36
*** ijw has quit IRC18:37
*** ijw has joined #openstack-infra18:37
*** ijw has quit IRC18:38
*** ijw has joined #openstack-infra18:38
*** ijw has quit IRC18:39
*** ijw has joined #openstack-infra18:40
pabelangerzxiiro: fungi: could also reach out to tripleo, they use it in deployments18:41
*** ijw has quit IRC18:41
*** Swami has joined #openstack-infra18:41
*** ijw has joined #openstack-infra18:41
zxiiropabelanger:  ok thanks for the heads up18:41
*** salv-orlando has joined #openstack-infra18:42
*** ijw has quit IRC18:42
*** ijw has joined #openstack-infra18:42
*** andreas_s has joined #openstack-infra18:42
*** jascott1 has quit IRC18:43
*** ijw has quit IRC18:43
*** yamahata has quit IRC18:43
*** ijw has joined #openstack-infra18:43
*** ijw has quit IRC18:44
*** ijw has joined #openstack-infra18:44
*** ijw has quit IRC18:45
*** ijw has joined #openstack-infra18:45
*** ijw has quit IRC18:45
*** yamahata has joined #openstack-infra18:46
*** jascott1 has joined #openstack-infra18:46
*** ijw has joined #openstack-infra18:46
*** slaweq_ has joined #openstack-infra18:46
*** ijw has quit IRC18:46
*** ijw has joined #openstack-infra18:47
*** raissa has quit IRC18:47
*** andreas_s has quit IRC18:47
*** ijw has quit IRC18:47
*** ijw has joined #openstack-infra18:48
*** sree has joined #openstack-infra18:48
*** ijw has quit IRC18:48
*** rossella_s has quit IRC18:49
pabelangerclarkb: travel to PTG confirmed18:50
*** ijw_ has joined #openstack-infra18:50
*** sbezverk has joined #openstack-infra18:50
clarkbwoot18:50
*** hjensas has joined #openstack-infra18:50
*** hjensas has quit IRC18:50
*** hjensas has joined #openstack-infra18:50
fungiboffo!18:50
*** ijw_ has quit IRC18:50
*** adisky__ has quit IRC18:50
*** yamahata has quit IRC18:51
*** ijw has joined #openstack-infra18:51
*** rossella_s has joined #openstack-infra18:51
*** sree has quit IRC18:52
*** ijw has quit IRC18:53
*** ijw has joined #openstack-infra18:53
*** ijw_ has joined #openstack-infra18:54
*** ijw has quit IRC18:54
*** ijw_ has quit IRC18:55
*** ijw has joined #openstack-infra18:55
*** ijw has quit IRC18:56
*** ijw has joined #openstack-infra18:56
*** wolverineav has quit IRC18:57
*** ijw has quit IRC18:57
*** ijw_ has joined #openstack-infra18:57
*** wolverineav has joined #openstack-infra18:57
*** ijw_ has quit IRC18:57
*** ijw has joined #openstack-infra18:58
*** Wei_Liu has joined #openstack-infra18:58
clarkbInfra meeting in just a few minutes over in #openstack-meeting18:58
*** ijw has quit IRC18:58
*** myoung|rover is now known as myoung|rover|bia18:58
*** ijw has joined #openstack-infra18:59
*** edmondsw has joined #openstack-infra18:59
*** ijw has quit IRC18:59
*** armaan has quit IRC18:59
*** ijw has joined #openstack-infra19:00
*** myoung|rover|bia is now known as myoung|rover|bbl19:00
*** armaan has joined #openstack-infra19:00
*** ijw has quit IRC19:00
openstackgerritJames E. Blair proposed openstack-infra/puppet-exim master: Move user-supplied routers to the end  https://review.openstack.org/52749819:00
openstackgerritJames E. Blair proposed openstack-infra/puppet-mailman master: Add multi-site support  https://review.openstack.org/52648319:00
*** ijw has joined #openstack-infra19:00
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Make lists.openstack.org multisite  https://review.openstack.org/52749919:01
*** baoli has quit IRC19:01
*** baoli has joined #openstack-infra19:01
*** ijw has quit IRC19:01
*** ijw has joined #openstack-infra19:01
*** pvaneck has quit IRC19:02
*** ijw has quit IRC19:02
*** ijw has joined #openstack-infra19:02
*** armaan has quit IRC19:03
*** ijw has quit IRC19:03
*** ijw_ has joined #openstack-infra19:03
*** armaan has joined #openstack-infra19:04
*** ijw_ has quit IRC19:04
*** rossella_s has quit IRC19:05
*** ijw has joined #openstack-infra19:06
*** trown|lunch is now known as trown19:06
*** pramodrj07 has joined #openstack-infra19:06
*** ijw has quit IRC19:06
*** ijw has joined #openstack-infra19:07
*** sree has joined #openstack-infra19:07
openstackgerritDavid Moreau Simard proposed openstack-infra/infra-specs master: Implement and provide an infrastructure continuous deployment dashboard  https://review.openstack.org/52750019:07
*** slaweq__ has joined #openstack-infra19:07
*** aviau has quit IRC19:07
*** rossella_s has joined #openstack-infra19:07
*** aviau has joined #openstack-infra19:07
*** ijw has quit IRC19:08
*** ijw has joined #openstack-infra19:08
openstackgerritDavid Moreau Simard proposed openstack-infra/infra-specs master: Implement and provide an infrastructure continuous deployment dashboard  https://review.openstack.org/52750019:08
*** ijw has quit IRC19:08
*** ijw has joined #openstack-infra19:09
*** ijw has quit IRC19:09
*** ijw has joined #openstack-infra19:10
*** ijw has quit IRC19:10
*** ijw has joined #openstack-infra19:10
*** ijw has quit IRC19:11
*** ijw has joined #openstack-infra19:11
*** sree has quit IRC19:11
*** slaweq__ has quit IRC19:12
*** ijw has quit IRC19:12
*** ijw has joined #openstack-infra19:12
*** ijw has quit IRC19:13
*** ijw has joined #openstack-infra19:13
*** ijw has quit IRC19:14
*** ijw has joined #openstack-infra19:14
*** ijw has quit IRC19:15
*** ijw has joined #openstack-infra19:15
*** rossella_s has quit IRC19:15
*** ijw has quit IRC19:15
*** ijw has joined #openstack-infra19:16
*** rossella_s has joined #openstack-infra19:16
openstackgerritDavid Moreau Simard proposed openstack-infra/infra-specs master: Implement an infrastructure continuous deployment dashboard  https://review.openstack.org/52750019:16
*** ijw has quit IRC19:16
*** ijw has joined #openstack-infra19:17
*** ijw has quit IRC19:17
*** ijw has joined #openstack-infra19:18
*** ijw has quit IRC19:18
*** ijw has joined #openstack-infra19:18
*** ijw has quit IRC19:19
*** Goneri has quit IRC19:19
*** ijw has joined #openstack-infra19:19
*** ijw has quit IRC19:20
*** ijw has joined #openstack-infra19:20
openstackgerritJames E. Blair proposed openstack-infra/puppet-mailman master: Add multi-site support  https://review.openstack.org/52648319:21
*** ijw has quit IRC19:21
*** ijw has joined #openstack-infra19:21
*** ijw has quit IRC19:22
*** ijw has joined #openstack-infra19:22
*** ijw has quit IRC19:23
*** ijw has joined #openstack-infra19:23
*** ijw has quit IRC19:24
*** ijw has joined #openstack-infra19:24
*** ijw has quit IRC19:25
*** ijw has joined #openstack-infra19:25
*** rossella_s has quit IRC19:25
*** ijw has quit IRC19:27
*** sree has joined #openstack-infra19:27
*** pblaho has quit IRC19:27
*** pblaho has joined #openstack-infra19:28
*** ijw has joined #openstack-infra19:28
*** ijw has quit IRC19:30
*** hamzy has joined #openstack-infra19:30
*** rbrndt has joined #openstack-infra19:30
*** rlandy|brb is now known as rlandy19:31
*** ijw has joined #openstack-infra19:31
*** sree has quit IRC19:31
*** ijw has quit IRC19:32
*** ijw has joined #openstack-infra19:32
pabelangerdmsimard: mordred: when you have review power, I'd love some feedback on https://review.openstack.org/526708/ as it relateds to shared ansible_host settings in zuulv3 now19:33
*** ijw has quit IRC19:33
*** ijw has joined #openstack-infra19:33
*** ijw has quit IRC19:35
*** pvaneck has joined #openstack-infra19:36
*** ijw has joined #openstack-infra19:37
*** ijw has quit IRC19:37
*** pvaneck_ has joined #openstack-infra19:37
*** ijw has joined #openstack-infra19:37
*** ijw_ has joined #openstack-infra19:39
*** ijw_ has quit IRC19:40
*** ijw_ has joined #openstack-infra19:40
*** pvaneck has quit IRC19:40
*** ijw_ has quit IRC19:40
*** ijw_ has joined #openstack-infra19:41
*** ijw_ has quit IRC19:41
*** ijw has quit IRC19:42
*** baoli has quit IRC19:42
*** slaweq__ has joined #openstack-infra19:43
*** salv-orlando has quit IRC19:46
*** leyal has quit IRC19:48
*** slaweq__ has quit IRC19:48
*** masuberu has joined #openstack-infra19:48
*** leyal has joined #openstack-infra19:49
*** esberglu has quit IRC19:49
*** esberglu has joined #openstack-infra19:50
*** masber has quit IRC19:51
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Remove duplicate required-projects  https://review.openstack.org/52750719:52
*** tonyb_ has quit IRC19:52
*** tosky has joined #openstack-infra19:52
*** tonyb has joined #openstack-infra19:52
ianwflaper87: sorry about the .projects change ... we just realised in the meeting that i missed it via codesearch due to that not updating.  do you need help with it?19:53
*** baoli has joined #openstack-infra19:56
*** dave-mccowan has quit IRC19:56
*** Apoorva has quit IRC20:00
dmsimardjeblair: ^ could we have prevented that from merging or does it end up just getting mashed together and not causing an actual error ?20:01
jeblairdmsimard: we will prevent changes like that from merging as soon as we eliminate all the changes like that which have already merged20:01
fungidmsimard: it causes an actual error now20:01
dmsimardok, thanks20:02
AJaegerdmsimard: but jeblair had to revert it to restart zuul20:02
fungiright, it's thanks to https://review.openstack.org/52723120:02
AJaegerdmsimard: so, next start should have it (if jeblair applies it manually again)20:02
clarkboh gotcha revert was in zuul beacuse config was broken with that in20:04
dmsimardWell, that's what we call a cliff in the graphs: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all20:04
jeblairi just did a local manual revert20:04
dmsimardI'll keep an eye on it ..20:04
*** myoung|rover|bbl is now known as myoung|rover20:05
*** salv-orlando has joined #openstack-infra20:05
flaper87ianw: no worries, I think I got it sorted: https://review.openstack.org/527325 in case you want to take a look20:06
flaper87still waiting for CI, though. might need a recheck or something20:06
*** sree has joined #openstack-infra20:07
*** rfolco has quit IRC20:07
AJaegerflaper87: no recheck needed, we just restarted zuul. I see it running20:07
jeblairi have to run -- can someone send a status all clear?20:08
flaper87AJaeger: cool, thanks20:08
*** wolverineav has quit IRC20:10
*** sree has quit IRC20:11
*** wolverineav has joined #openstack-infra20:11
fungi#status notice The zuul scheduler has been restarted after lengthy troubleshooting for a memory consumption issue; earlier changes have been reenqueued but if you notice jobs not running for a new or approved change you may want to leave a recheck comment or a new approval vote20:14
openstackstatusfungi: sending notice20:14
openstackgerritMerged openstack-infra/project-config master: Remove duplicate required-projects  https://review.openstack.org/52750720:15
-openstackstatus- NOTICE: The zuul scheduler has been restarted after lengthy troubleshooting for a memory consumption issue; earlier changes have been reenqueued but if you notice jobs not running for a new or approved change you may want to leave a recheck comment or a new approval vote20:15
openstackgerritFabien Boucher proposed openstack-infra/zuul feature/zuulv3: Git driver  https://review.openstack.org/52561420:17
*** slaweq__ has joined #openstack-infra20:17
openstackgerritMerged openstack-infra/zuul-jobs master: releasenotes: Remove package install  https://review.openstack.org/52685020:19
AJaegerteam, just a heads up - we're not installing the package itself for releasenotes anymore ^20:20
*** slaweq__ has quit IRC20:22
fungithat's a pretty major change, thanks for the reminder!20:23
AJaegerso far only merged the check/gate change - the post one will be done tomorrow.20:25
AJaegerNote that some repos might fail - they just need to merge my change https://review.openstack.org/#/q/status:open++topic:releasenotes-version but have ignored it so far ;(20:25
AJaeger(will only fail if they build releasenotes)20:26
*** ldnunes has quit IRC20:30
SpamapSinteresting, neither PyYAML nor rust-yaml enforce "mapping keys must be unique" from the YAML spec20:30
SpamapSthat's annoying20:30
AJaegerdmsimard: will you update https://review.openstack.org/526666 , please ?20:30
SpamapSI wonder how may things would explode in a hail of melting circuit boards if they started asserting that20:30
*** rbrndt_ has joined #openstack-infra20:32
*** bobh_ has joined #openstack-infra20:32
*** dmellado has quit IRC20:32
*** salv-orl_ has joined #openstack-infra20:32
*** e0ne has joined #openstack-infra20:33
*** umbarkar has joined #openstack-infra20:33
*** jkilpatr has quit IRC20:33
*** openstackgerrit has quit IRC20:34
*** alex_xu has quit IRC20:34
*** notmyname has quit IRC20:34
*** lathiat has quit IRC20:34
*** lathiat has joined #openstack-infra20:34
*** dmellado has joined #openstack-infra20:34
*** hughhalf has quit IRC20:34
*** jbernard has quit IRC20:34
*** bswartz has quit IRC20:34
*** salv-orlando has quit IRC20:35
*** armaan has quit IRC20:35
*** bobh has quit IRC20:35
*** joadavis has quit IRC20:35
*** kaisers has quit IRC20:35
*** patriciadomin has quit IRC20:35
*** sileht has quit IRC20:35
*** mika has quit IRC20:35
*** jmccrory has quit IRC20:35
*** armaan has joined #openstack-infra20:35
*** rbrndt has quit IRC20:35
*** tumbarka has quit IRC20:35
*** _ari_ has quit IRC20:35
*** myoung|rover has quit IRC20:35
*** mdrabe has quit IRC20:35
*** benj_ has quit IRC20:35
*** dteselkin has quit IRC20:35
*** masayukig has quit IRC20:35
*** abelur_ has quit IRC20:35
*** vkmc has quit IRC20:36
*** pahuang has quit IRC20:36
*** StevenK has quit IRC20:36
*** lihi has quit IRC20:37
*** pramodrj07 has quit IRC20:38
jeblairSpamapS: and the spec goes on at length about how they have improved upon json by requiring that in the spec20:38
*** jbernard has joined #openstack-infra20:39
*** hughhalf has joined #openstack-infra20:39
*** abelur_ has joined #openstack-infra20:39
*** alex_xu has joined #openstack-infra20:39
*** myoung has joined #openstack-infra20:39
SpamapSjeblair: indeed, it actually would be great if implementations refused to parse duplicate keys. I've shot myself in the foot a few times exactly as this one did.20:39
SpamapSAnsible has started enforcing it.20:39
*** sileht has joined #openstack-infra20:39
fungiSpamapS: in the old trac bug report i believe we were arguing to just make it a loader parameter so it could be selectively enforced20:40
*** pahuang has joined #openstack-infra20:40
*** lihi has joined #openstack-infra20:40
SpamapSfungi: yeah that would be a good compromise.20:40
*** kaisers has joined #openstack-infra20:40
*** dteselkin has joined #openstack-infra20:40
*** patriciadomin has joined #openstack-infra20:40
*** jmccrory has joined #openstack-infra20:40
*** _ari_ has joined #openstack-infra20:40
*** masayukig has joined #openstack-infra20:40
fungiheck, leave it non-default if worried about breaking backward compat in the lib20:40
*** bswartz has joined #openstack-infra20:41
*** joadavis has joined #openstack-infra20:41
*** notmyname has joined #openstack-infra20:41
*** openstackgerrit has joined #openstack-infra20:41
openstackgerritMerged openstack-infra/system-config master: Support xenial on files  https://review.openstack.org/52718620:41
*** benj_ has joined #openstack-infra20:41
openstackgerritMerged openstack-infra/system-config master: Clean up mirror in tripleo-test-cloud-rh1  https://review.openstack.org/52727420:41
*** mdrabe has joined #openstack-infra20:42
*** mika has joined #openstack-infra20:42
*** mika has joined #openstack-infra20:42
*** StevenK has joined #openstack-infra20:44
*** greghaynes has quit IRC20:45
*** asilenkov has quit IRC20:45
*** rakhmerov has quit IRC20:45
*** adam_g has quit IRC20:45
*** kmalloc has quit IRC20:45
*** clarkb has quit IRC20:45
*** andymccr has quit IRC20:45
*** mancdaz has quit IRC20:45
*** jistr has quit IRC20:45
*** slaweq__ has joined #openstack-infra20:45
*** asilenkov has joined #openstack-infra20:45
*** mancdaz has joined #openstack-infra20:45
*** jistr has joined #openstack-infra20:45
*** clarkb has joined #openstack-infra20:45
*** vkmc has joined #openstack-infra20:45
*** greghaynes has joined #openstack-infra20:45
*** andymccr has joined #openstack-infra20:45
*** rakhmerov has joined #openstack-infra20:45
*** kmalloc has joined #openstack-infra20:45
*** vkmc has quit IRC20:46
*** vkmc has joined #openstack-infra20:46
*** adam_g has joined #openstack-infra20:46
*** myoung is now known as myoung|rover20:47
*** patriciadomin has quit IRC20:47
*** gaelL has quit IRC20:48
openstackgerritMerged openstack-infra/puppet-subunit2sql master: Don't hardcode the path to pip  https://review.openstack.org/52728020:48
*** slaweq__ has quit IRC20:49
*** patriciadomin has joined #openstack-infra20:50
*** gaelL has joined #openstack-infra20:50
*** vhosakot has quit IRC20:52
*** amoralej is now known as amoralej|off20:54
*** dave-mccowan has joined #openstack-infra20:54
*** slaweq__ has joined #openstack-infra20:55
*** Goneri has joined #openstack-infra20:55
openstackgerritMerged openstack-infra/system-config master: Comment out server in puppet.conf  https://review.openstack.org/50726620:56
*** apetrich has quit IRC20:58
*** abelur_ has quit IRC20:59
*** dave-mcc_ has joined #openstack-infra20:59
*** apetrich has joined #openstack-infra20:59
*** dave-mccowan has quit IRC20:59
openstackgerritJeremy Stanley proposed openstack-infra/zuul-base-jobs master: Add generic base and base-test jobs/playbooks  https://review.openstack.org/52614021:00
*** slaweq__ has quit IRC21:00
*** abelur_ has joined #openstack-infra21:00
*** Apoorva has joined #openstack-infra21:00
*** pblaho has quit IRC21:05
*** Apoorva has quit IRC21:05
*** Douhet has quit IRC21:07
*** sree has joined #openstack-infra21:07
openstackgerritMerged openstack-infra/project-config master: Import ansible-role-redhat-subscription (Part 1)  https://review.openstack.org/52722921:07
*** Apoorva has joined #openstack-infra21:08
*** Douhet has joined #openstack-infra21:08
*** wolverineav has quit IRC21:09
*** wolverineav has joined #openstack-infra21:11
*** sree has quit IRC21:11
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Remove files01.o.o from hiera  https://review.openstack.org/52751921:12
*** bobh_ has quit IRC21:13
*** bobh has joined #openstack-infra21:17
*** Apoorva_ has joined #openstack-infra21:18
*** rhallisey has quit IRC21:20
*** linkmark has quit IRC21:20
*** Apoorva has quit IRC21:21
*** bobh has quit IRC21:22
*** jappleii__ has joined #openstack-infra21:23
*** jappleii__ has quit IRC21:24
*** jappleii__ has joined #openstack-infra21:25
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Make lists.openstack.org multisite  https://review.openstack.org/52749921:25
*** jappleii__ has quit IRC21:26
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Make lists.openstack.org multisite  https://review.openstack.org/52749921:26
openstackgerritJames E. Blair proposed openstack-infra/puppet-exim master: Move user-supplied routers to the end  https://review.openstack.org/52749821:26
*** jappleii__ has joined #openstack-infra21:27
openstackgerritJames E. Blair proposed openstack-infra/puppet-mailman master: Add multi-site support  https://review.openstack.org/52648321:27
*** sree has joined #openstack-infra21:27
*** jappleii__ has quit IRC21:27
*** bobh has joined #openstack-infra21:28
*** jappleii__ has joined #openstack-infra21:28
*** ccamacho has joined #openstack-infra21:28
*** ccamacho has quit IRC21:28
*** kgiusti has left #openstack-infra21:28
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add dns servers  https://review.openstack.org/52752621:29
clarkbjeblair: semi related to ^ I'm noticing that a significant chunk of our cron spam is https://tronche.com/wiki/Exim_keep_environment21:29
*** smatzek has quit IRC21:29
clarkbjeblair: would you be willing to take a look at that and formulate an opinion if we can set keep_environment to an empty value?21:29
jeblairclarkb: yep21:29
clarkbfrom my understanding of how we use exim I think its likely safe esepcially everywhere but lists.o.o21:30
*** slaweq__ has joined #openstack-infra21:30
*** sree has quit IRC21:32
*** bobh has quit IRC21:32
*** rbrndt_ has quit IRC21:32
openstackgerritJames E. Blair proposed openstack-infra/puppet-exim master: Set keep_environment to empty to silence warning  https://review.openstack.org/52752821:35
*** slaweq__ has quit IRC21:35
*** rbrndt has joined #openstack-infra21:36
jeblairclarkb: i believe it's safe everywhere ^21:36
clarkbjeblair: thanks21:36
jeblairi think the warning is telling us that it is clearing env variables, so if something breaks, check that.  i don't think anything is broken for us now.  setting it to the default value explicitly silences the warning.21:37
clarkbThat was my understanding too. I figured the only place we'd really make use of stuff like that would be lists.o.o if anywhere21:37
*** trown is now known as trown|outtypewww21:38
*** bobh has joined #openstack-infra21:38
openstackgerritMerged openstack-infra/system-config master: Support xenial on lists  https://review.openstack.org/52717521:41
*** bobh has quit IRC21:43
*** felipemonteiro_ has joined #openstack-infra21:46
*** hamzy has quit IRC21:46
dmsimardpabelanger: oh wow, we submitted puppet runs to puppetdb directly without a puppetmaster ? I didn't know that21:47
*** sree has joined #openstack-infra21:47
*** dklyle has joined #openstack-infra21:48
*** david-lyle has quit IRC21:48
pabelangerdmsimard: yup, we'd pull reports back to puppetmaster.o.o, then push them into puppetdb. Because nodes don't talk directly to puppetdb21:49
clarkbinfra-root can we get reviews on https://review.openstack.org/#/c/527528/1 to cut down on the amount of cronspam we are receiving?21:49
pabelangerlooking21:49
*** bobh has joined #openstack-infra21:49
clarkbdmsimard: and it wasn't working properly and I think the api chagned with newer puppetdb, it ended up being a not so easy thing to keep working21:50
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: Import ansible-role-redhat-subscription (Part 2)  https://review.openstack.org/52722621:50
pabelangerclarkb: dmsimard: Yah, the main issue, it didn't report failed things. Only success, which made it hard to see what was failing21:51
pabelangerclarkb: +321:51
*** sree has quit IRC21:51
dmsimardthanks, I'll let a few reviews roll in and draft a new version21:52
*** bobh has quit IRC21:54
*** Goneri has quit IRC21:55
fungiclarkb: i believe i've found some issues with the approach taken in 516740 but trying to figure out how to solve it in puppet... i don't need three levels of quoting in the command if we can drop the explicit /bin/bash -c and am curious why you added it (the command line is itself posixly fine and works under dash for me)21:56
clarkbfungi: it is because puppet just forks and execs that without a shell aiui21:58
*** rcernin has joined #openstack-infra21:58
clarkbfungi: and since I was using the output of multiple commands I had to glue that together with a shell and pulled that in with bash -c21:59
fungioh, got it, so we need shell replacement there21:59
fungiany idea how to wrap the parameters to test in quoting?21:59
clarkbfungi: I think we may be able to use \" around them?21:59
fungii'll give it a try in puppet-subunit2sql22:00
fungithough it'll have to be \\" in the puppet string i think22:00
fungiwhich i seem to recall freaks puppet out somehow, but let's give it a shot22:00
*** slaweq_ has quit IRC22:00
openstackgerritClark Boylan proposed openstack-infra/system-config master: Reduce cacti updates cron spam  https://review.openstack.org/52753122:01
clarkbanother cronspam cleanup ^22:01
*** bobh has joined #openstack-infra22:01
* clarkb goes back to reviewing sprint changes22:02
*** slaweq_ has joined #openstack-infra22:02
*** e0ne has quit IRC22:03
fungiif it doesn't work, we can switch to the old hack of prepending both strings with a constant like x22:03
openstackgerritJeremy Stanley proposed openstack-infra/puppet-subunit2sql master: Quote arguments to the test command  https://review.openstack.org/52753222:03
fungiclarkb: ^ trying to see if this will work for puppet22:04
*** jkilpatr has joined #openstack-infra22:05
*** slaweq_ has quit IRC22:06
*** flwang has quit IRC22:06
*** bobh has quit IRC22:07
*** sree has joined #openstack-infra22:07
*** bobh has joined #openstack-infra22:08
*** slaweq__ has joined #openstack-infra22:08
*** ijw has joined #openstack-infra22:08
*** wolverineav has quit IRC22:09
*** rlandy is now known as rlandy|afk22:10
*** ijw has quit IRC22:10
*** ijw has joined #openstack-infra22:11
*** ijw has quit IRC22:11
*** ijw has joined #openstack-infra22:12
*** sree has quit IRC22:12
*** bobh has quit IRC22:12
*** slaweq__ has quit IRC22:13
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Create zuul-ci.org zone repo (1/3)  https://review.openstack.org/52753322:13
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Create zuul-ci.org zone repo (2/3)  https://review.openstack.org/52753422:13
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Create zuul-ci.org zone repo (3/3)  https://review.openstack.org/52753522:13
clarkbjeblair: thinking out loud. Wouldnt it be simpler to have a single repo for all the zones?22:14
*** ijw has quit IRC22:15
*** ijw_ has joined #openstack-infra22:15
jeblairclarkb: i don't think it'll make a big difference in terms of the plumbing.  i think there are advantages in repo-level acls -- it's easy to delegate management of a zone that way.22:16
*** ijw_ has quit IRC22:16
*** ijw has joined #openstack-infra22:17
clarkbhrm22:17
*** ijw has quit IRC22:17
clarkb(fwiw I've really been finding the lack fo a global git grep on my local machine to be a major pain recently)22:17
*** gouthamr has quit IRC22:18
*** ijw has joined #openstack-infra22:18
clarkbthough at least zones are fairly tightly scoped so maybe all the repos there won't be a problem22:18
jeblairclarkb: could help if we agree to call them all "zone-*"22:19
*** flwang has joined #openstack-infra22:19
*** bobh has joined #openstack-infra22:19
dmsimardout of curiosity, did we plan on upgrading gerrit to 2.14 in the near future or was there a reason why it hasn't been done yet ?22:19
*** ijw has quit IRC22:19
clarkbdmsimard: we have. The reason it hasn't been done yet is we only just got to 2.13 and now its kidn of a bad time in the cycle to take a large downtime needed to do the upgrade22:19
*** ganso has quit IRC22:19
jeblairalso fixing codesearch will help22:19
clarkbdmsimard: there was a recent thread on the gerrit mailing list about upgrading from 2.13 to 2.15 that I've been meaning to read beacuse that might actually be us depending on timing22:20
clarkbdmsimard: but that upgrade is on the infra todo board and we can do review-dev without impacting production to test things even if in the middle of release cycle22:20
*** vhosakot has joined #openstack-infra22:20
dmsimardclarkb: thanks22:21
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update ethercalc to work with Ubuntu Xenial  https://review.openstack.org/52714422:21
clarkbjeblair: ya codesearch helps, though I find its presentation of the data to sometimes be too much (hard to search through visually with lots of results)22:21
dmsimardknock on wood but memory usage is pretty flat on zuulv3, so far so good.22:22
clarkbfungi: are you testing 527532?22:22
dmsimardAJaeger: thanks for reminding me about that22:23
dmsimardAJaeger: I ended up finding the issue and know of a fix, I'll try and push it22:23
*** bobh has quit IRC22:24
*** pvaneck_ has quit IRC22:25
openstackgerritIan Wienand proposed openstack-infra/puppet-ethercalc master: Update ethercalc to work with Ubuntu Xenial  https://review.openstack.org/52714422:26
*** sree has joined #openstack-infra22:27
*** bobh has joined #openstack-infra22:27
fungiclarkb: well, first i was waiting to see if puppet could even parse it22:29
*** ijw has joined #openstack-infra22:29
clarkbah22:29
funginot quite sure how to pass a customized local copy of a module to the launch-node.py script though22:29
*** ijw has quit IRC22:30
fungii mean, i tested the command being run by the onlyif22:30
fungiand the altered version of it22:30
clarkbfungi: you can do it by putting that module update on the host in question then run puppet apply like ansible does22:31
fungiso if puppet doesn't have any trouble parsing the manifest, then i think the only question is whether the \\" will translate properly to \" in the command-line22:31
clarkbI don't know what launch-node has a way to plumb that though probably have to use --keep22:31
*** ijw has joined #openstack-infra22:31
*** sree has quit IRC22:31
*** ijw has quit IRC22:32
*** ijw has joined #openstack-infra22:32
*** ijw has quit IRC22:32
*** bobh has quit IRC22:33
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Create paste hiera group  https://review.openstack.org/52753622:33
*** ijw has joined #openstack-infra22:34
*** ijw has quit IRC22:34
openstackgerritPaul Belanger proposed openstack-infra/puppet-meetbot master: Support ubuntu xenial plugins directory  https://review.openstack.org/52744722:34
*** ijw has joined #openstack-infra22:35
*** ijw has quit IRC22:36
*** ijw has joined #openstack-infra22:36
*** bobh has joined #openstack-infra22:36
*** ijw has quit IRC22:36
*** ijw has joined #openstack-infra22:37
*** ijw has quit IRC22:37
*** ijw has joined #openstack-infra22:38
*** ijw has quit IRC22:38
*** ijw has joined #openstack-infra22:39
*** ijw has quit IRC22:39
*** ijw has joined #openstack-infra22:41
*** ijw has quit IRC22:41
*** ijw has joined #openstack-infra22:42
*** bobh has quit IRC22:42
*** ijw has quit IRC22:43
*** andreas_s has joined #openstack-infra22:44
*** ijw has joined #openstack-infra22:47
*** spzala has quit IRC22:48
*** ijw has quit IRC22:48
*** ijw_ has joined #openstack-infra22:48
*** spzala has joined #openstack-infra22:48
*** ijw_ has quit IRC22:48
openstackgerritAlex Schultz proposed openstack-infra/openstack-zuul-jobs master: puppet-unit-4.8: override ZUUL_BRANCH after puppet-ceph/jewel checkout  https://review.openstack.org/52747422:49
*** ijw has joined #openstack-infra22:49
*** andreas_s has quit IRC22:49
*** ijw has quit IRC22:49
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Revert "Revert "Add sphinx_python variable to sphinx role and job""  https://review.openstack.org/52666622:49
*** ijw has joined #openstack-infra22:50
*** ijw has quit IRC22:50
ianwclarkb / fungi: re our codesearch discussion this morning, presumably the index taking things offline is why it's in the emergency file?22:50
dmsimardpabelanger might know22:51
clarkbianw: that I don't know22:51
ianwi feel like we can solve this fairly easily ... have a small script that sees if the config file would change, if it does, put up a "indexing" static page, run the reindex, bring it back to live.  run that from cron22:52
ianwof course we could get fancier with this via post jobs, etc, but kiss22:52
*** spzala has quit IRC22:53
ianwafaict, the service restart was/is tied to project-config updates which i imagine would cause the issues22:54
clarkbdmsimard: ok I've caught up on the gerrit thread for 2.13 to 2.15. TLDR is you can't do it22:54
clarkbthere is an issue with the way db migration work that make it not viable. So we will have to go from 2.13 to 2.14 then 2.1522:54
clarkbianw: ya22:55
clarkbalso ++ to kiss22:55
*** ijw has joined #openstack-infra22:56
*** ijw_ has joined #openstack-infra22:57
*** ijw has quit IRC22:57
*** ijw_ has quit IRC22:57
*** bobh has joined #openstack-infra22:57
*** ijw has joined #openstack-infra22:58
openstackgerritPaul Belanger proposed openstack-infra/puppet-meetbot master: Support ubuntu xenial plugins directory  https://review.openstack.org/52744722:58
*** esberglu has quit IRC22:58
*** ijw has quit IRC22:58
*** ijw has joined #openstack-infra22:59
*** esberglu has joined #openstack-infra22:59
pabelangerianw: yah, we don't run puppet on codesearch.o.o, so it's always been in emergency file22:59
pabelangerianw: IIRC, everytime puppet runs, it kicks the restart process of hound, so we'd need to solve that22:59
*** ijw has quit IRC23:00
ianwpabelanger: yep, working on it ;)23:00
pabelangerianw: I cannot remember what kicks the restart, just that it happens every 15mins23:00
clarkbthe way gerrit diffs pabelanger's last patchset is really weird23:00
clarkbthanks gerrit23:00
*** rbrndt has quit IRC23:00
ianwpabelanger: looking at the puppet, the service gets notified by project-config updates, so yeah, that would be constant23:00
*** ijw has joined #openstack-infra23:01
pabelangerianw: yah, that sounds right23:01
*** ijw has quit IRC23:01
fungiianw: yeah, not sure why it's in the emergency file. if it's the one under revision control i'd git blame... otherwise best we can do is check our status log or worse, grep the infra channel history23:01
*** edmondsw has quit IRC23:02
*** ijw has joined #openstack-infra23:02
*** bobh has quit IRC23:03
*** ijw has quit IRC23:03
*** armaan has quit IRC23:03
*** esberglu has quit IRC23:03
*** rbrndt has joined #openstack-infra23:04
*** armaan has joined #openstack-infra23:04
fungiodds are i put it in there and don't remember why ;)23:04
*** ijw has joined #openstack-infra23:05
*** edmondsw has joined #openstack-infra23:05
*** ijw has quit IRC23:05
*** ijw has joined #openstack-infra23:06
*** ijw_ has joined #openstack-infra23:07
*** ijw has quit IRC23:07
clarkbfungi: isn't it fun when you git blame something and you are like what was I thinking23:08
*** ijw_ has quit IRC23:08
clarkbor "I don't remember this at all"23:08
*** marst has quit IRC23:08
clarkbgit though never forgets23:08
*** ijw has joined #openstack-infra23:08
*** jklare has quit IRC23:08
*** ijw has quit IRC23:08
*** calbers has quit IRC23:08
*** ijw has joined #openstack-infra23:09
*** ijw has quit IRC23:10
*** ijw_ has joined #openstack-infra23:10
*** edmondsw has quit IRC23:10
*** ijw_ has quit IRC23:10
*** ijw has joined #openstack-infra23:11
*** ijw has quit IRC23:12
*** bobh has joined #openstack-infra23:12
*** esberglu has joined #openstack-infra23:13
*** calbers has joined #openstack-infra23:13
*** ijw has joined #openstack-infra23:13
*** ijw has quit IRC23:13
*** ijw has joined #openstack-infra23:14
*** slaweq__ has joined #openstack-infra23:14
*** ijw has quit IRC23:14
*** ijw has joined #openstack-infra23:15
*** priteau has quit IRC23:15
*** esberglu has quit IRC23:15
*** ijw has quit IRC23:15
*** ijw has joined #openstack-infra23:16
*** bobh has quit IRC23:16
*** ijw has quit IRC23:16
*** ijw has joined #openstack-infra23:17
*** jklare has joined #openstack-infra23:17
*** slaweq__ has quit IRC23:19
*** ijw has quit IRC23:21
*** bobh has joined #openstack-infra23:21
fungiyeah, i frequently just end up cursing my past self for being so stupid23:22
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Add node requests graph to zuul dashboard  https://review.openstack.org/52754023:22
fungibut occasionally i find notes i left to my future self which make it worthwhile23:22
*** chandankumar has quit IRC23:22
fungithough also a fair number of "# TODO(mordred) make this actually work"23:23
*** chandankumar has joined #openstack-infra23:23
fungiwhich are often quite nice, since they tend to include some detail on what the thing would do if it were to exist23:24
jeblair527450 does two *very* important things -- it adds a piece of missing information AND it balances out the symmetry of that section of the page23:24
fungisymmetry++!23:24
fungiit's how humans differentiate our creations from those of lesser species23:25
*** abelur_ has quit IRC23:25
clarkbfungi: care to review https://review.openstack.org/#/c/527531/ as a second cronspam cleanup change?23:25
fungisure23:25
*** baoli has quit IRC23:25
jeblairfungi: i aspire to the greater species, such as bees.23:25
fungii have a couple minutes 'till the over timer dings and i have to rush back into the kitchen23:25
clarkbjeblair: dancing grafana pages next?23:26
fungijeblair: bees use trinary symmetry!23:26
*** bobh has quit IRC23:26
jeblairnow i want hexagonal grafana graphs23:26
fungii have some pads of hexagonal graph paper. maybe we can start there23:26
*** gaurangt has quit IRC23:27
*** ijw has joined #openstack-infra23:29
*** chandankumar has quit IRC23:29
clarkbI've updated the infra board story for gerrit upgrade with the information I found earlier (basically switched it to a more concrete upgrade to 2.14)23:31
*** ijw_ has joined #openstack-infra23:31
*** ijw has quit IRC23:33
*** chandankumar has joined #openstack-infra23:34
*** gaurangt has joined #openstack-infra23:35
*** ijw_ has quit IRC23:35
*** bobh has joined #openstack-infra23:36
*** stakeda has joined #openstack-infra23:36
*** ijw has joined #openstack-infra23:37
*** andreas_s has joined #openstack-infra23:38
*** ijw has quit IRC23:38
*** ijw has joined #openstack-infra23:39
openstackgerritMerged openstack-infra/system-config master: Support xenial on subunit-worker  https://review.openstack.org/52718123:39
*** abelur_ has joined #openstack-infra23:40
*** dprince has joined #openstack-infra23:40
*** ijw_ has joined #openstack-infra23:41
*** bobh has quit IRC23:41
*** andreas_s has quit IRC23:43
*** dingyichen has joined #openstack-infra23:43
*** ijw has quit IRC23:43
*** bobh has joined #openstack-infra23:43
*** edmondsw has joined #openstack-infra23:44
*** abelur_ has quit IRC23:44
*** ijw has joined #openstack-infra23:44
*** ijw_ has quit IRC23:45
*** bobh has quit IRC23:46
*** bobh has joined #openstack-infra23:47
*** slaweq__ has joined #openstack-infra23:47
*** ijw_ has joined #openstack-infra23:48
*** edmondsw has quit IRC23:48
*** rbrndt has quit IRC23:48
*** ijw has quit IRC23:48
*** dprince has quit IRC23:49
*** bobh has quit IRC23:51
*** slaweq__ has quit IRC23:51
*** ijw_ has quit IRC23:52
*** dprince has joined #openstack-infra23:53
*** ijw has joined #openstack-infra23:55
*** bobh has joined #openstack-infra23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!