Friday, 2018-08-31

*** anteaya has joined #openstack-infra00:07
*** mtreinish has quit IRC00:07
*** mtreinish has joined #openstack-infra00:09
*** longkb has joined #openstack-infra00:10
*** lbragstad has quit IRC00:12
*** signed8b_ is now known as signed8bit_Zzz00:12
*** lbragstad has joined #openstack-infra00:14
notmynamedhellmann: nguyenhai_: something's screwy https://review.openstack.org/#/q/owner:nguyentrihai93%2540gmail.com+AND+(project:openstack/swift+OR+project:openstack/python-swiftclient)+status:open00:16
imacdonnurgh .. https://review.openstack.org/#/c/591527/ timed out in the gate ... any objection to me requesting a recheck ?00:16
notmynamedhellmann: nguyenhai_: http://logs.openstack.org/14/596214/6/gate/legacy-tempest-dsvm-neutron-src/7a5e573/job-output.txt.gz#_2018-08-30_23_54_15_08084900:17
clarkbno maybe ianw can reenqueue to the gate too?00:17
ianwclarkb: i can ... sorry just looking.  this is important?00:18
imacdonnanything that uses devstack-gate is broken until that goes through, so kinda00:19
ianwoh right, ok00:19
ianwis it force-merge worthy?00:20
imacdonnpossibly? it seems fairly straightforward, and it's broken anyway, so......00:22
fungii'd be hesitant to force merge a d-g change. you could requeue it into the gate and promote it to the top of the queue to get it hopefully merged sooner00:23
ianwok, will requeue00:24
fungii mean, are we confident ansible 2.5 hasn't changed any of the ways we were relying on 2.3 in d-g?00:24
fungithey don't have the best track record when it comes to backward-compatibility00:24
notmynameclarkb: oh, is this why I just saw a bunch of gate jobs fail? (links above)00:25
clarkbnotmyname: likely00:25
notmynameah, ok. then I won't worry about it. thanks :-)00:25
ianwhrm ... sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=gate --project=openstack-infra/devstack-gate --ref=refs/heads/master --newrev=87b3df316938d346feed6186465e6862c0a377eb00:26
clarkbianw: not enqueue ref, just enqueue iirc00:27
ianwohhh, yeah00:27
clarkbchange vs already merged ref00:27
fungiright. enqueue is used for changes, enqueue-ref for other objects (explicit commits, branch states, tags...)00:28
ianwsorry, yeah i was working on the wheel periodic jobs yesterday so that that in my window already00:29
fungiif you used my reenqueue.py script, it probably doesn't grok that distinction00:29
fungiit could be made to00:29
fungialso if people find that useful i could propose it somewhere (tools subdir of the zuul repo?)00:30
imacdonnIt's running again .. thanks, guys!00:30
ianwok it's promoted00:31
ianwplease join with me off-list to congratulate 591527,3 on it's promotion00:32
fungiit's probably a fine time for me to go ahead with the etherpad-lite restart on etherpad.o.o now too00:32
fungii'll go ahead and do that00:32
fungiianw: will punch and pie be served?00:32
imacdonnmmm, pie00:32
* fungi is a sucker for free refreshments at such functions00:33
ianwyes but 15 minutes only, then everybody back to work!00:33
*** harlowja has quit IRC00:34
*** ijw has joined #openstack-infra00:34
fungi#status log restarted etherpad-lite service on etherpad.openstack.org to start making use of the 1.7.0 release in the wake of https://review.openstack.org/59754400:34
openstackstatusfungi: finished logging00:34
scasmore people will come if they think we'll have punch and pie!00:39
*** ijw has quit IRC00:39
*** bobh has joined #openstack-infra00:43
fungithanks, cartman00:48
*** anteaya has quit IRC00:56
*** yamamoto has joined #openstack-infra00:58
*** hongbin has joined #openstack-infra01:01
bkero...I was promised there would be punch and pie.01:01
*** mriedem_lawnboy is now known as mriedem01:04
*** mriedem has quit IRC01:10
*** bobh has quit IRC01:14
*** dingyichen has quit IRC01:18
*** imacdonn has quit IRC01:19
*** imacdonn has joined #openstack-infra01:19
openstackgerritMerged openstack-infra/project-config master: adding readthedocs to networking-ansible  https://review.openstack.org/59719401:21
*** rlandy has quit IRC01:34
*** bdodd has quit IRC01:34
*** bdodd has joined #openstack-infra01:37
*** bobh has joined #openstack-infra01:38
*** bobh has quit IRC01:48
*** yamamoto has quit IRC01:49
*** bobh has joined #openstack-infra01:51
*** ramishra has quit IRC01:54
*** Emine has quit IRC01:55
*** bobh has quit IRC01:56
*** bobh has joined #openstack-infra02:04
*** bobh has quit IRC02:09
*** apetrich has quit IRC02:14
*** armax has quit IRC02:17
*** dpawlik has joined #openstack-infra02:18
*** anteaya has joined #openstack-infra02:20
*** bobh has joined #openstack-infra02:21
*** dpawlik has quit IRC02:22
*** jamesmcarthur has joined #openstack-infra02:23
*** bobh has quit IRC02:25
*** jamesmcarthur has quit IRC02:27
*** ramishra has joined #openstack-infra02:28
*** jamesmcarthur has joined #openstack-infra02:32
*** anteaya has quit IRC02:33
*** bobh has joined #openstack-infra02:33
*** jamesmcarthur has quit IRC02:36
*** bobh has quit IRC02:38
*** bobh has joined #openstack-infra02:41
*** bobh has quit IRC02:46
tonybI'm seeing grenade jobs fail with: ERROR! Unexpected Exception: No module named manager02:50
tonybany ideas? http://logs.openstack.org/69/595569/4/check/requirements-integration/345f7a4/job-output.txt.gz#_2018-08-31_02_18_46_06669802:50
tonybsorry that's a different job with the same error: http://logs.openstack.org/69/595569/4/check/neutron-grenade/bc6b45a/job-output.txt.gz#_2018-08-31_02_18_15_281785 is the grenade job02:51
clarkbtonyb: its due to new release of ara not working with older ansible. There is a devstack gate fix that ianw enqueued to the gate earlier02:51
*** bobh has joined #openstack-infra02:51
*** annp has joined #openstack-infra02:51
tonybclarkb: Ahh okay02:51
* tonyb cools hsi heals02:51
clarkb59152702:51
tonybclarkb: thanks02:52
clarkblooks like it failed again :(02:53
*** psachin has joined #openstack-infra02:55
*** bobh has quit IRC02:55
clarkbianw if still around try enqueue it again? otherwise tonyb you should recheck it02:56
ianwclarkb: just done that.  have to run out for a thing with kids for a bit.  i think, maybe if it still fails, a force-merge might be appropriate (of course, why the d-g is unstable is another matter ... but one thing at a time)02:57
tonybclarkb: It looks like it's back in the gate pipeline?02:57
tonybianw: Oh thanks02:57
ianwok, am afk but will check in later around when it would finish02:58
tonybianw: I'll watch it too and possibly recheck02:59
clarkbya its put kids to bed time for me here03:00
*** bobh has joined #openstack-infra03:01
*** bobh has quit IRC03:06
*** bobh has joined #openstack-infra03:12
*** bobh has quit IRC03:16
*** bobh has joined #openstack-infra03:25
*** ramishra has quit IRC03:29
*** bobh has quit IRC03:30
*** nicolasbock has quit IRC03:33
*** bobh has joined #openstack-infra03:33
*** signed8bit_Zzz is now known as signed8b_03:37
*** bobh has quit IRC03:38
*** bobh has joined #openstack-infra03:42
*** bobh has quit IRC03:47
*** bobh has joined #openstack-infra03:55
*** udesale has joined #openstack-infra03:59
johnsomYeah, this is breaking a bunch of gates. A notification when we can recheck would be appreciated.03:59
*** bobh has quit IRC03:59
*** dave-mccowan has quit IRC04:05
*** ramishra has joined #openstack-infra04:06
*** bobh has joined #openstack-infra04:07
*** bobh has quit IRC04:12
*** signed8b_ has quit IRC04:13
*** hamzy has joined #openstack-infra04:15
*** Bhujay has joined #openstack-infra04:18
*** bobh has joined #openstack-infra04:20
*** bobh has quit IRC04:25
openstackgerritMerged openstack-infra/devstack-gate master: Bump ansible to latest 2.5 version  https://review.openstack.org/59152704:36
tonyb^^ \o/ ^^04:36
*** bobh has joined #openstack-infra04:41
openstackgerritTrinh Nguyen proposed openstack-infra/irc-meetings master: Update Searchlight meeting time  https://review.openstack.org/59842204:42
*** bobh has quit IRC04:45
*** jamesmcarthur has joined #openstack-infra04:46
openstackgerritTrinh Nguyen proposed openstack-infra/irc-meetings master: Update Searchlight meeting time  https://review.openstack.org/59842204:47
*** jchhatbar has joined #openstack-infra04:49
*** jamesmcarthur has quit IRC04:50
*** bobh has joined #openstack-infra04:51
*** yamamoto has joined #openstack-infra04:54
*** bobh has quit IRC04:56
*** jchhatbar has quit IRC05:03
*** bobh has joined #openstack-infra05:04
*** bobh has quit IRC05:09
*** janki has joined #openstack-infra05:09
*** hongbin has quit IRC05:12
*** ramishra has quit IRC05:20
openstackgerritMerged openstack-infra/project-config master: wheel-index: only create index once  https://review.openstack.org/59799705:22
*** janki has quit IRC05:23
AJaegerianw, tonyb ^05:23
tonybAJaeger: thanks!05:24
tonybAJaeger: I just saw (in my email) your +W I didn't realise it'd merge so fast ;P05:24
openstackgerritMerged openstack-infra/irc-meetings master: Update Searchlight meeting time  https://review.openstack.org/59842205:27
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: mirror-info: add variable for CBS proxy  https://review.openstack.org/59825005:28
*** bobh has joined #openstack-infra05:30
openstackgerritMerged openstack-infra/project-config master: Create an openstack/placement project  https://review.openstack.org/59836205:32
*** bobh has quit IRC05:34
*** ramishra has joined #openstack-infra05:37
*** bhavikdbavishi has joined #openstack-infra05:42
*** bobh has joined #openstack-infra05:43
*** kaiokmo has quit IRC05:45
*** kaiokmo has joined #openstack-infra05:46
*** bobh has quit IRC05:47
*** yamamoto has quit IRC05:51
*** bhavikdbavishi has quit IRC05:58
*** psachin has quit IRC06:00
*** jbadiapa has joined #openstack-infra06:01
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/59842406:04
*** bobh has joined #openstack-infra06:12
*** bobh has quit IRC06:17
*** dpawlik has joined #openstack-infra06:18
*** holser_ has joined #openstack-infra06:20
*** Bhujay has quit IRC06:20
*** dpawlik has quit IRC06:22
*** apetrich has joined #openstack-infra06:24
*** janki has joined #openstack-infra06:27
*** holser_ has quit IRC06:29
*** bobh has joined #openstack-infra06:29
*** aojea has joined #openstack-infra06:32
*** bobh has quit IRC06:33
openstackgerritSimon Westphahl proposed openstack-infra/nodepool master: Update static nodes in Zookeeper on config change  https://review.openstack.org/59815606:37
*** ccamacho has joined #openstack-infra06:43
*** ccamacho has quit IRC06:44
*** alexchadin has joined #openstack-infra06:46
*** e0ne has joined #openstack-infra06:50
*** ccamacho has joined #openstack-infra06:52
*** e0ne has quit IRC06:52
*** hashar has joined #openstack-infra06:53
*** ccamacho has quit IRC06:57
*** ccamacho has joined #openstack-infra06:58
*** bobh has joined #openstack-infra06:59
*** olivierb has joined #openstack-infra07:00
*** rcernin has quit IRC07:00
*** bobh has quit IRC07:01
*** bobh has joined #openstack-infra07:01
*** bobh has quit IRC07:06
*** bobh has joined #openstack-infra07:13
*** yamamoto has joined #openstack-infra07:14
*** shardy has joined #openstack-infra07:16
*** bobh has quit IRC07:17
*** pcaruana has joined #openstack-infra07:19
*** janki has quit IRC07:25
*** dpawlik has joined #openstack-infra07:28
*** holser_ has joined #openstack-infra07:29
*** holser_ has quit IRC07:30
*** holser_ has joined #openstack-infra07:30
*** jpena|off is now known as jpena07:31
*** shardy_ has joined #openstack-infra07:31
*** janki has joined #openstack-infra07:32
*** dpawlik has quit IRC07:32
*** shardy has quit IRC07:34
*** jamesmcarthur has joined #openstack-infra07:34
*** dpawlik has joined #openstack-infra07:38
*** jamesmcarthur has quit IRC07:38
*** shardy_ is now known as shardy07:39
openstackgerritSorin Sbarnea proposed openstack-infra/zuul master: Display correct cursor when hovering over patchset header  https://review.openstack.org/59822207:41
*** Bhujay has joined #openstack-infra07:41
*** jpich has joined #openstack-infra07:41
*** gema has joined #openstack-infra07:42
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/59842407:44
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove #openstack-product  https://review.openstack.org/59847707:44
openstackgerritOlivier Bourdon proposed openstack/diskimage-builder master: Fix CentOS image build failure when dib runs on debian based system  https://review.openstack.org/55948507:45
*** dpawlik has quit IRC07:46
*** dpawlik_ has joined #openstack-infra07:46
*** bobh has joined #openstack-infra07:53
*** jchhatbar has joined #openstack-infra07:55
*** janki has quit IRC07:57
*** alexchadin has quit IRC07:57
*** bobh has quit IRC07:58
*** florianf has joined #openstack-infra07:59
*** dbecker has quit IRC08:04
*** dbecker has joined #openstack-infra08:05
*** alexchadin has joined #openstack-infra08:07
*** bobh has joined #openstack-infra08:07
openstackgerritSorin Sbarnea proposed openstack-infra/devstack-gate master: Avoid breaking python module dependencies with pip install  https://review.openstack.org/59848608:08
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Add development-proposals to spec index  https://review.openstack.org/59848708:08
*** electrofelix has joined #openstack-infra08:08
AJaegerinfra-root, could you review the remaining changes for the retirement of the 18 repos I proposed, please? Full list at https://review.openstack.org/#/q/status:open+++topic:infra-retire08:10
*** bobh has quit IRC08:12
*** bobh has joined #openstack-infra08:16
*** Bhujay has quit IRC08:17
*** pbourke has quit IRC08:17
*** pbourke has joined #openstack-infra08:19
*** bobh has quit IRC08:21
*** bobh has joined #openstack-infra08:25
*** bobh has quit IRC08:29
ttxfungi + infra-core: this should fix the ptgbot currently not joining the IRC channel : https://review.openstack.org/#/c/598155/ <-- please review asap so we can test everything is ok there08:29
*** bobh has joined #openstack-infra08:38
*** olivierb_ has joined #openstack-infra08:42
*** bobh has quit IRC08:43
*** olivierb has quit IRC08:43
*** bobh has joined #openstack-infra08:43
*** bobh has quit IRC08:47
*** bobh has joined #openstack-infra08:55
ianwttx: LGTM but probably fungi can babysit it tomorrow08:55
ttx++ thx08:56
*** bobh has quit IRC08:59
*** e0ne has joined #openstack-infra09:00
*** alexchadin has quit IRC09:01
*** bobh has joined #openstack-infra09:02
*** dpawlik_ has quit IRC09:04
*** bobh has quit IRC09:07
*** dtantsur|afk is now known as dtantsur09:07
*** Sigyn has joined #openstack-infra09:08
*** dpawlik has joined #openstack-infra09:11
*** sthussey has quit IRC09:12
*** ramishra has quit IRC09:13
*** bobh has joined #openstack-infra09:14
*** dbecker has quit IRC09:14
*** alexchadin has joined #openstack-infra09:15
dtantsurhi infra! is this failure already known? http://logs.openstack.org/49/596249/2/gate/ironic-lib-tempest-partition-agent_ipmitool/735774f/job-output.txt.gz#_2018-08-30_20_37_03_28184209:15
dtantsurreading the scrollback, it can be rechecked after ansible bump to 2.5, right? maybe worth a notice?09:17
*** dbecker has joined #openstack-infra09:17
*** Sigyn has left #openstack-infra09:17
*** bobh has quit IRC09:18
*** bobh has joined #openstack-infra09:22
*** bobh has quit IRC09:27
*** kaiokmo has quit IRC09:27
*** gmann has quit IRC09:27
*** bobh has joined #openstack-infra09:31
*** bobh has quit IRC09:35
*** hamzy_ has joined #openstack-infra09:37
*** hamzy has quit IRC09:37
*** yamamoto has quit IRC09:38
*** stakeda has quit IRC09:41
*** gmann has joined #openstack-infra09:43
*** bobh has joined #openstack-infra09:47
*** ramishra has joined #openstack-infra09:48
*** bobh has quit IRC09:52
*** priteau has joined #openstack-infra09:53
*** alexchadin has quit IRC09:53
AJaegerdtantsur: indeed09:54
*** bobh has joined #openstack-infra09:54
AJaeger#status notice Jobs using devstack-gate (legacy devstack jobs) have been failing due to an ara update. We use now a newer ansible version, it's safe to recheck if you see "ImportError: No module named manager" in the logs.09:55
openstackstatusAJaeger: sending notice09:55
*** dpawlik has quit IRC09:56
*** dpawlik_ has joined #openstack-infra09:56
-openstackstatus- NOTICE: Jobs using devstack-gate (legacy devstack jobs) have been failing due to an ara update. We use now a newer ansible version, it's safe to recheck if you see "ImportError: No module named manager" in the logs.09:57
*** shabeeb has joined #openstack-infra09:57
*** shabeeb has quit IRC09:57
*** shabeeb has joined #openstack-infra09:58
*** bobh has quit IRC09:58
openstackstatusAJaeger: finished sending notice09:59
*** bobh has joined #openstack-infra10:03
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add project templates for openstack-tox-lower-constraints  https://review.openstack.org/59810310:03
*** alexchadin has joined #openstack-infra10:07
*** bobh has quit IRC10:07
dtantsurthanks AJaeger10:11
*** bobh has joined #openstack-infra10:13
*** bobh has quit IRC10:17
*** Bhujay has joined #openstack-infra10:21
*** shabeeb has quit IRC10:23
*** srp has joined #openstack-infra10:23
*** Bhujay has quit IRC10:26
*** ramishra has quit IRC10:27
*** ramishra has joined #openstack-infra10:28
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update descriptions for some templates  https://review.openstack.org/59859110:28
*** dave-mccowan has joined #openstack-infra10:33
*** nicolasbock has joined #openstack-infra10:33
*** bobh has joined #openstack-infra10:39
*** annp has quit IRC10:40
*** bobh has quit IRC10:41
*** bobh has joined #openstack-infra10:41
openstackgerritSorin Sbarnea proposed openstack-infra/devstack-gate master: Avoid breaking python module dependencies with pip install  https://review.openstack.org/59848610:42
*** bobh has quit IRC10:46
*** stephenfin is now known as finucannot10:46
*** cgoncalves has quit IRC10:48
ssbarneacan someone have a look at ^^, AJaeger? You are already fully aware about the ara issue and this should prevent future similar problems.10:48
ssbarneai wonder which one of the check jobs is really using the "devstack-vm-gate-wrap.sh" script as I was not able to identify it yet.10:50
*** cgoncalves has joined #openstack-infra10:50
*** alexchadin has quit IRC10:53
*** bobh has joined #openstack-infra10:55
*** alexchadin has joined #openstack-infra10:57
*** bobh has quit IRC10:59
openstackgerritHarry Kominos proposed openstack/diskimage-builder master: Fix bootloader packages for aarch64  https://review.openstack.org/59859511:06
*** udesale has quit IRC11:12
*** gfidente has joined #openstack-infra11:12
*** ginopc has joined #openstack-infra11:15
openstackgerritSorin Sbarnea proposed openstack-infra/openstack-zuul-jobs master: Silence syncronize on post to avoid console spam  https://review.openstack.org/59859911:16
*** longkb has quit IRC11:28
*** alexchadin has quit IRC11:28
*** jpena is now known as jpena|lunch11:28
openstackgerritChris Dent proposed openstack-infra/project-config master: Add openstack/placement as a repo on #openstack-placement  https://review.openstack.org/59860311:29
openstackgerritGhanshyam Mann proposed openstack-infra/project-config master: Switch to tempest-multinode-full job  https://review.openstack.org/59861011:35
*** bobh has joined #openstack-infra11:36
gmannslaweq: switching the usage of neutron-tempest-multinode-full - https://review.openstack.org/#/c/598610/11:37
*** dbecker has quit IRC11:38
*** alexchadin has joined #openstack-infra11:39
*** alexchadin has quit IRC11:39
*** bobh has quit IRC11:39
*** alexchadin has joined #openstack-infra11:40
*** alexchadin has quit IRC11:40
*** bobh has joined #openstack-infra11:40
gmannAJaeger: seems no core team for tempest-stress repo - https://review.openstack.org/#/admin/groups/1860,members. can you add me there and later i will add respective members. This repo was added here- https://review.openstack.org/#/c/533025/11:43
*** bobh has quit IRC11:44
*** e0ne has quit IRC11:44
*** tpsilva has joined #openstack-infra11:49
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: remove job settings for Chef OpenStack repositories  https://review.openstack.org/59861411:51
*** cdent has joined #openstack-infra11:55
cdentGood morniing infra people, thanks for creating openstack/placement.11:56
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: remove job settings for adjutant repositories  https://review.openstack.org/59862011:56
slaweqgmann: thx, I will update my patch to be Depends-On on Yours :)11:57
cdentWhen someone gets the chance can the initial membership of the placement-core group be set to inherit nova-core please?11:57
*** srp has quit IRC11:57
*** srp has joined #openstack-infra11:58
*** alexchadin has joined #openstack-infra11:58
*** ramishra has quit IRC11:58
*** kaiokmo has joined #openstack-infra11:59
*** tonyb has quit IRC12:02
*** trown|outtypewww is now known as trown12:07
*** rh-jelabarre has quit IRC12:08
*** boden has joined #openstack-infra12:12
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: remove job settings for Telemetry repositories  https://review.openstack.org/59867112:13
*** ansmith has joined #openstack-infra12:15
*** tonyb has joined #openstack-infra12:19
*** giblet has joined #openstack-infra12:22
*** jpena|lunch is now known as jpena12:23
*** shardy has quit IRC12:35
*** rlandy has joined #openstack-infra12:40
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: remove job settings for Puppet OpenStack repositories  https://review.openstack.org/59870912:51
*** cdent has quit IRC12:55
*** e0ne has joined #openstack-infra12:55
*** EmilienM is now known as EvilienM12:57
*** ansmith has quit IRC13:01
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: remove job settings for cloudkitty repositories  https://review.openstack.org/59892913:02
*** tosky has joined #openstack-infra13:02
*** jcoufal has joined #openstack-infra13:03
fricklercdent: done (though you don't seem to be online currently)13:04
*** ansmith has joined #openstack-infra13:06
fricklergmann: done, too (AJaeger doesn't have access for that I think)13:06
*** alexchadin has quit IRC13:09
*** yamamoto has joined #openstack-infra13:12
gmannfrickler: thanks.13:13
AJaegerdhellmann: 2500 nodes are needed to run all the current pushed changes - good that it's Friday ;)13:15
dhellmannAJaeger : oof. I figured the Friday after release and before a US holiday would be OK but I wasn't counting on the number of nodes being so high13:16
* AJaeger is glad to see how nice Zuul scales - 330 changes in check are not a problem anymore ;)13:16
AJaegerdhellmann: no worries, most of US is not at work yet - and in an hour or two we look much better...13:16
dhellmannyeah, I also figured doing it early would be better13:17
dhellmannthat's a pretty sharp spike in the grapfana graph13:17
dhellmann /me has to step out13:18
AJaegerdirk: is rpm packaging ready? See http://lists.openstack.org/pipermail/openstack-dev/2018-August/134082.html13:18
AJaegereumel8: what about i18n? ^13:19
*** kgiusti has joined #openstack-infra13:20
AJaegerclarkb: what are we doing with infra and python3-first? do we want to have dhellmann migrate everything - or keep it central? With our many repos I'm fine either way, it's basically the same reviewing anyhow...13:20
AJaegerdhellmann: if we do infra, please wait until we have merged https://review.openstack.org/#/q/status:open+++topic:infra-retire - saves us 18 repos to touch...13:20
*** eharney has joined #openstack-infra13:23
dirkAJaeger: sigh, [rpm} is not a tag that I filter for..13:26
*** anteaya has joined #openstack-infra13:27
*** bnemec is now known as beekneemech13:31
*** alexchadin has joined #openstack-infra13:34
*** alexchadin has quit IRC13:34
*** alexchadin has joined #openstack-infra13:35
*** alexchadin has quit IRC13:35
*** alexchadin has joined #openstack-infra13:35
*** alexchadin has quit IRC13:36
*** rascasoft is now known as raoh13:36
*** alexchadin has joined #openstack-infra13:36
*** alexchadin has quit IRC13:36
*** alexchadin has joined #openstack-infra13:37
*** lbragstad is now known as elbragstad13:37
*** hashar is now known as hasharAway13:38
*** efried is now known as fried_rice13:38
*** yamamoto has quit IRC13:43
*** shabeeb has joined #openstack-infra13:43
*** mriedem has joined #openstack-infra13:44
mriedemwas a bug ever opened for this ansible failure that started last night? http://logs.openstack.org/65/598365/1/check/nova-next/e8cae4c/ara-report/result/6ac39717-e1f4-4a82-9f41-45a01621f67f/13:45
mriedemi just want to e-r fingerprint it13:45
*** srp has quit IRC13:46
*** sthussey has joined #openstack-infra13:47
openstackgerritMerged openstack-infra/elastic-recheck master: Drop mox usage  https://review.openstack.org/59464213:49
*** raoh is now known as rascasoft13:52
*** Seb-Solon has quit IRC13:52
*** anteaya has quit IRC13:52
dhellmannAJaeger : yeah, I'm planning to wait for that one13:55
dhellmanndirk : sorry :-/13:55
*** giblet is now known as giblet_off13:57
*** rh-jelabarre has joined #openstack-infra14:01
mriedemwell if not https://bugs.launchpad.net/openstack-gate/+bug/179015214:03
openstackLaunchpad bug 1790152 in OpenStack-Gate "jobs failing with ansible "ImportError: No module named manager" since 8/30" [Undecided,New]14:03
fricklermriedem: I don't think there was a bug. https://review.openstack.org/591527 is the patch that fixed it in case you hadn't seen that14:03
mriedemfrickler: thanks14:04
*** jamesmcarthur has joined #openstack-infra14:08
dirkdhellmann: no problem, I have AJaeger that reads every email for me and notifies me when I'm likely not going to respond otherwise ;)14:08
dhellmanndirk : oh, nice, where can I find a helper like that? :-)14:08
AJaegerLol ;)14:09
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: remove job settings for Packaging-rpm repositories  https://review.openstack.org/59897414:12
*** mriedem has quit IRC14:14
*** mriedem has joined #openstack-infra14:15
openstackgerritDean Troyer proposed openstack-infra/project-config master: Add stx-distcloud and stx-distcloud-client repos  https://review.openstack.org/59838314:15
dirkdhellmann: thats a good question, I'm not sure why I deserve the honor myself ;)14:17
*** rh-jelabarre has quit IRC14:22
*** anteaya has joined #openstack-infra14:25
*** hamzy_ is now known as hamzy14:26
*** Seb-Solon has joined #openstack-infra14:27
*** rh-jelabarre has joined #openstack-infra14:31
*** zzzeek has quit IRC14:33
*** zzzeek has joined #openstack-infra14:34
elbragstadmriedem are you still seeing failures for https://bugs.launchpad.net/openstack-gate/+bug/1790152 ?14:34
openstackLaunchpad bug 1790152 in OpenStack-Gate "jobs failing with ansible "ImportError: No module named manager" since 8/30" [Undecided,Fix released]14:34
elbragstadi just rechecked a change that was failing with that last night and it apparently tripped on it again14:34
elbragstadhttp://logs.openstack.org/46/598346/8/check/legacy-tempest-dsvm-neutron-src/a96f6de/job-output.txt.gz#_2018-08-30_22_32_12_72142414:35
*** srp has joined #openstack-infra14:35
*** armax has joined #openstack-infra14:37
*** shabeeb has quit IRC14:37
mriedemlogstash was still showing stuff14:39
elbragstadmmm14:40
*** shabeeb has joined #openstack-infra14:40
mriedemhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22from%20ansible.config.manager%20import%20find_ini_config_file%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=2d14:40
*** zzzeek has quit IRC14:40
mriedemwell nevermind it looks like that's tailed off14:40
mriedemso it should be ok14:40
elbragstadhmmm14:40
mriedemMMMMM HHMM M MM MAHHMHMH HM HHM14:41
mriedemelbragstad: http://logs.openstack.org/46/598346/8/check/legacy-tempest-dsvm-neutron-src/a96f6de/job-output.txt.gz#_2018-08-30_22_32_12_721424 is from last night14:41
mriedemnot this morning14:41
*** alexchadin has quit IRC14:41
mriedem598346 is queued14:42
mriedemso you don't have results yet14:42
* elbragstad facepalm14:42
mriedemhmm indeed14:42
*** zzzeek has joined #openstack-infra14:42
mriedemcheck queue is large14:42
mriedemso get your sitting pants on14:42
elbragstadmriedem sorry - i'm apparently _way_ too excited to see the results of that patch14:42
mriedemobviously14:42
*** jpena is now known as jpena|off14:42
elbragstadlike a toddler stuck in a van14:43
*** jpena|off is now known as jpena14:43
*** jaosorior has quit IRC14:43
elbragstadheading to six flags for the first time EVAR!14:43
*** srp has quit IRC14:43
mriedemi immediately thought of something much much worse than that14:43
elbragstadoh... oh no14:44
*** shabeeb has quit IRC14:44
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Add query for ansible bug 1790152  https://review.openstack.org/59898714:45
openstackbug 1790152 in OpenStack-Gate "jobs failing with ansible "ImportError: No module named manager" since 8/30" [Undecided,Fix released] https://launchpad.net/bugs/179015214:45
*** kiennt26 has joined #openstack-infra14:46
*** zzzeek has quit IRC14:46
*** zzzeek has joined #openstack-infra14:47
*** anteaya has quit IRC14:52
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add a run-nodepool job  https://review.openstack.org/59840314:52
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add testinfra tests for bridge  https://review.openstack.org/59840414:52
openstackgerritHarry Kominos proposed openstack/diskimage-builder master: Fix bootloader packages for aarch64  https://review.openstack.org/59859515:02
mriedemcan we get an easy +W on https://review.openstack.org/#/c/598603/ please?15:03
mriedemjust adds the new openstack/placement repo to the channel list for #openstack-placement15:03
mriedem*repo list for the channel...15:03
mriedemfungi: thanks15:03
fungigo magic irc bot go15:04
openstackgerritMerged openstack/ptgbot master: Identify with SASL  https://review.openstack.org/59815515:11
*** ccamacho has quit IRC15:12
*** e0ne has quit IRC15:14
*** anteaya has joined #openstack-infra15:19
openstackgerritMerged openstack-infra/project-config master: Add openstack/placement as a repo on #openstack-placement  https://review.openstack.org/59860315:26
*** pcaruana has quit IRC15:26
AJaegerfungi, could you review https://review.openstack.org/597399 and https://review.openstack.org/597442 for removing obsolete content from system-config, please? This came up in the repo retirement...15:27
*** jchhatbar has quit IRC15:27
fungisure15:27
AJaegerthanks15:27
fungiinfra-root: just reported in the releaseteam meeting, a bullet line got its authorship color reassigned (and this is after i restarted the service around utc midnight). checking to see if the upgrade failed to deploy or something, though the correct version was definitely checked out in the vcsrepo path before i did the restart15:29
*** jlvillal is now known as jlvilla-viva15:30
fungipossible this could also be browser-side javascript caching at work15:32
*** Dobroslaw has quit IRC15:35
openstackgerritMarkus Hosch proposed openstack-infra/zuul master: Add a sanity check for all refs returned by Gerrit  https://review.openstack.org/59901115:36
fungilooks like the commit which was intended to fix it alters src/static/js/AttributeManager.js and src/static/js/ace2_inner.js so not sure if those are frontend or backend15:39
fungilooks like everything under src/static is served to browsers, so i think it may be caching related behavior15:41
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete legacy-tempest-dsvm-full-kafka  https://review.openstack.org/59901315:43
*** dpawlik_ has quit IRC15:44
*** e0ne has joined #openstack-infra15:45
*** ginopc has quit IRC15:47
*** electrofelix has quit IRC15:47
AJaegerconfig-core, https://review.openstack.org/#/c/598614/ is ready - tiny python3-first change.15:50
AJaegerplease review ^15:50
*** kiennt26 has quit IRC15:51
TheJuliaAny chance https://review.openstack.org/#/c/596844/ could get another set of eyes along with someone with sufficient permissions to add ironic-core to the new group?15:54
*** e0ne has quit IRC15:56
*** jpich has quit IRC15:59
*** r-daneel has joined #openstack-infra16:01
*** jpena is now known as jpena|off16:03
*** openstackgerrit has quit IRC16:06
*** aojea has quit IRC16:07
*** holser_ has quit IRC16:07
*** openstackgerrit has joined #openstack-infra16:10
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update descriptions for some templates  https://review.openstack.org/59859116:10
*** anteaya has quit IRC16:18
AJaegerfungi, could you help TheJulia with https://review.openstack.org/#/c/596844/  and setting up the ACL, please? ^16:19
clarkbAJaeger: TheJulia I'm taking a look now16:20
*** mriedem is now known as mriedem_away16:21
clarkbAJaeger: as for migrating infra configs into individual repos, I think we can probably do that at our own pace? the big thing is its the same group of reviewers in current and potential future location so not a major change in that regard16:21
clarkbone benefit would be to make things self testing pre merge if out of project-config16:21
fungithanks clarkb!16:23
fungi(for picking up the virtualbmc-core request that is)16:24
clarkbbut I think we can probably go through that on our own time unless we have volunteers to shepherd it through? (mostly thinking we are going through a transition away from puppet so many repos will get fewer eyeballs on them, and also there is a lot of effort required to do that)16:24
*** dkehn has joined #openstack-infra16:26
*** fried_rice is now known as fried_rolls16:27
scasi'm retiring a repo that has no centralized jobs. i presume the retirement process is the same, just slightly different, than currently documented16:27
clarkbfungi: is there a change we need to review to update the gpg secret in the zuul configs for the stein key?16:27
clarkbscas: you'll apply the noop jobs to your project config and not the central config16:28
clarkbscas: otherwise I think it should be the same16:28
scasit used to be the source of jobs. it does't currently do anything16:28
clarkbsure, to merge changes to a repo you must have check and gate jobs which is why we apply noop to the repo first, then merge in the retirement change16:29
clarkbif it doesn't do anything at all currently it may already have noop jobs applied to it?16:29
scasah, i did noop it16:29
*** dkehn_ has quit IRC16:30
fungiclarkb: i haven't written them yet. there'll be one with a zuul secret replacement in project-config and another with a documentation update to the releases repo... working on that here in a bit since we decided in the releaseteam meeting we can swap those any time now16:30
scasokay. i noticed i was going off-script, and just needed a sanity check16:31
clarkbfungi: cool, just making sure I wasn't avoiding some important reviews there :) let me know when they are up and I'll put them near top of the list16:31
fungigladly!16:31
fungistill trying to catch up on e-mail unfortunately16:32
clarkbscas: no problem, we'll likely have to make minor tweaks to docs like that as we push more of that config into each repository and stop managing so much centrally16:32
*** trown is now known as trown|lunch16:34
*** ssbarnea|ruck has joined #openstack-infra16:36
*** anteaya has joined #openstack-infra16:36
*** r-daneel has quit IRC16:38
*** r-daneel has joined #openstack-infra16:38
clarkbis the crazy zuul demand today catch up from devstack-gate + ara + ansible being unhappy?16:39
clarkbianw: thank you for getting that fix in16:39
clarkbssbarnea: out of curiousity why does devstack-gate need to use the same version of ansible as tripleo?16:39
clarkb(I think it is fine if they do but I don't think they are coupled such that being out of sync warrants a -1)16:40
TheJuliaclarkb: thanks16:40
clarkbin fact they've been out of sync all along as far as I can tell16:40
clarkbTheJulia: I've scribbled a note to swing around after things merge to update group membership too16:40
TheJuliaansible versions have literally always been out of sync across the community for as long as we've used ansible in the community16:40
clarkbTheJulia: indeed16:40
TheJuliaclarkb: thanks again :)16:40
*** olivierb_ has quit IRC16:41
ssbarnea|ruckclarkb i don't think there is a clear *must*, but clearly we did had a version conflict.16:42
TheJuliathere was a discussion about it... ?three? years ago, and there were some legitimate reasons to not force a version range to lockstep with the community16:42
clarkbssbarnea|ruck: the conflict was between ara and ansible though, no tripleo involved16:42
TheJuliamainly I think around playbook syntax changes16:42
clarkbTheJulia: which continue to be an issue I think16:43
TheJuliaIndeed16:43
TheJuliaAlso, ops folks don't always necessarily have the latest and greatest ansible on hand.16:43
ssbarnea|rucki am in favor of ranges myself, for compatibility reasons. always mentioning minimal and capping only when you know that the newer version does not work (not before)16:43
ssbarnea|ruckbut is not up to me to decide that.16:44
clarkbssbarnea|ruck: this case is somewhat special in that its a virtualenv for devstack gate just for devstack gate16:44
clarkbnothing else is using this installation of ansible16:44
clarkbprobably what we should do as a followup now, is pin ara too16:44
clarkbthen the ansible and ara in that virtualenv should continue working as long as no one deletes those versiosn from pypi16:44
TheJuliaclarkb: ++ seems reasonable to me, and projects that need specific versions can put that in place for their jobs to download and leverage it16:45
TheJulia(which we did before zuul playbooks)16:45
clarkbTheJulia: this is the old pre zuul playbooks tooling fwiw, so we never really stopped, eventually we will when we stop using devstack-gate though16:45
TheJuliaahh!16:46
openstackgerritFabien Boucher proposed openstack-infra/zuul-jobs master: ensure-sphinx: do not attempt a gettext install if exists  https://review.openstack.org/59902816:47
TheJuliaAnyway, back to the code mines16:47
*** Swami has joined #openstack-infra16:48
openstackgerritMerged openstack-infra/system-config master: Remove obsolete pypi-mirror  https://review.openstack.org/59739916:49
openstackgerritMerged openstack-infra/system-config master: Remove releasestatus configuration  https://review.openstack.org/59744216:49
ssbarnea|ruckclarkb since we are here I would like to ask something about making syncronize (logs) tasks quiet in order to avoid console spam. Example way to silence it: https://review.openstack.org/#/c/598599/1/playbooks/legacy/tempest-dsvm-neutron-full-centos-7/post.yaml16:49
ssbarnea|rucknow the stuggle is that there are tons of post.yml files, one for each legacy job, seems like a nightmare to make this change to all of them. Any ideas on how this can be achieve in a better way?16:50
AJaegerclarkb: I agree, we should do it at our pace - where it's beneficial.16:52
clarkbssbarnea|ruck: I think this was one of the motivations of the new zuulv3 job config layout, where roles could be reconsumed across jobs16:52
AJaegerscas: still the same process - merge noop-jobs in project-config first to be able to remove everything.16:52
clarkbssbarnea|ruck: may be best to put the effort into converting the jobs?16:52
scasAJaeger: i already have noop in-repo and nothing in project-config16:52
AJaegerclarkb: the large Zuul load is python3-first jobs...16:53
clarkbAJaeger: ah16:53
AJaegerscas: you still need noop-jobs in project-config to gate it - since your retirement change will remove your zuul file. Try it ;)16:54
ssbarnea|ruckclarkb: ok, there is already effort being done in this direction. i wanted to know if anyone has something against silencing rsync. i personally hate having to scroll tons of screens generated by post / collection to spot the real error.16:54
*** anteaya has quit IRC16:55
scasAJaeger: good point16:55
AJaegerssbarnea|ruck: let's get rid of legacy jobs ;)16:55
AJaegerconfig-core, next python3-first change - please review: https://review.openstack.org/#/c/598614/16:56
openstackgerritMerged openstack-infra/project-config master: Add virtualbmc-core group  https://review.openstack.org/59684416:57
ssbarnea|ruckmwhahaha https://review.openstack.org/#/c/598322/ is you do not mind, please.17:02
*** anteaya has joined #openstack-infra17:12
openstackgerritFabien Boucher proposed openstack-infra/zuul-jobs master: ensure-sphinx: do not attempt a gettext install if exists  https://review.openstack.org/59902817:20
*** jcoufal has quit IRC17:24
openstackgerritTrevor Bramwell proposed openstack-infra/zuul master: Fix typo in documentation chown commands  https://review.openstack.org/59904017:28
*** jamesmcarthur has quit IRC17:30
AJaegerclarkb: regarding job moves into infra repos - want to put this up as topic on next infra meeting, discuss here some more - or send directly an email to dhellmann to remove us from his todo list? I'm fine with the last option, just wondering whether there is more discussion needed17:33
dhellmannAJaeger , clarkb : I'm fine doing whatever you all decide to do.17:34
*** trown|lunch is now known as trown17:34
*** jamesmcarthur has joined #openstack-infra17:34
*** jcoufal has joined #openstack-infra17:36
*** eernst has joined #openstack-infra17:36
clarkbAJaeger: we can probably put it to the infra list and if there is no objections go with it17:37
*** sshnaidm|off has quit IRC17:37
*** sshnaidm|off has joined #openstack-infra17:38
*** sshnaidm|off has quit IRC17:41
*** sshnaidm|off has joined #openstack-infra17:41
AJaegerclarkb: do you want to write something or shall I?17:43
AJaegerclarkb: also, could you put https://review.openstack.org/#/c/598614/ on your review queue, please? That's a tiny change for python3-first - moves the final chef repo.17:44
*** jamesmcarthur has quit IRC17:45
dtroyerWe are looking to set up docs.starlingx.io modeled after docs.o.o, is there a process to get that done?  I'm not finding anything…17:46
clarkbAJaeger: I can write it when I'm done with reviees17:46
AJaegerthanks, clarkb17:46
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Improve keystorage migration  https://review.openstack.org/59904317:48
corvusdtroyer: the process isn't documented yet, but here are some clues: http://specs.openstack.org/openstack-infra/infra-specs/specs/project-hosting.html#websites17:50
*** mriedem_away is now known as mriedem17:50
dtroyercorvus: thanks!17:51
corvusdtroyer: you can look at the setup for zuul-ci.org too17:51
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ptgbot master: Use port 6697 for IRC over SSL/TLS  https://review.openstack.org/59904617:51
fungiinfra-puppet-core: that ^ would be nice to get in before monday17:52
corvusdtroyer: (zuul publishes the root of its website in the same way as docs, and docs get published to /docs.  either thing is possible, it's just a matter of setting the roots appropriately)17:52
corvusfungi: you mean monday next, yeah? :)17:52
dtroyerwe'll probably be closer to zuul-ci17:53
fungicorvus: i mean this coming monday so people can prepopulate some schedule claims and whatnot17:53
corvusdtroyer: the zuul-ci.org docs jobs are defined in project-config17:53
corvusfungi: oh i see17:53
fungiptgbot is currently broken otherwise17:53
AJaegerdtroyer: and the server setup is AFAIK is system-config17:54
fungicorvus: and i know we're avoiding making intrusive changes to our existing bots, but wonder if we want to consider porting the sasl change to statusbot too as i suspect (haven't looked yet) that it's probably failing to join at least a few of its configured channels while it races the identify17:54
corvusAJaeger, dtroyer: yep -- there's some vhost stuff in system-config.  we'll just need to create an afs volume before that lands, but it's easy, any infra-root should be able to17:55
corvusfungi: those are exactly the sort of urgent maintenance changes i'd be happy to review :)17:56
fungithanks, i'll take a look in that case17:56
corvusdtroyer: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/zuul-jobs.yaml17:57
*** dtantsur is now known as dtantsur|afk17:59
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for ansible bug 1790152  https://review.openstack.org/59898718:05
openstackbug 1790152 in OpenStack-Gate "jobs failing with ansible "ImportError: No module named manager" since 8/30" [Undecided,Fix released] https://launchpad.net/bugs/179015218:05
dtroyerSo it look like I add a stanza for docs.starlingx.io next to zuul-ci.org in maanifests/site.pp?18:06
corvusdtroyer: probably so :)18:07
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update descriptions for templates  https://review.openstack.org/59859118:08
AJaegerif any infra-root, likes to help with retiring the remaing repos, please see topic:infra-retire - or https://review.openstack.org/#/q/status:open+++topic:infra-retire . clarkb gave his +2 already, so I need a second reviewer... thanks!18:09
openstackgerritDean Troyer proposed openstack-infra/system-config master: Add docs.starlingx.io vhost  https://review.openstack.org/59904818:16
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient master: Refactor Project Group editing to match other objects  https://review.openstack.org/59840718:18
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient master: Support linking to Project Groups by name  https://review.openstack.org/59840818:18
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient master: Link to Project Groups by name  https://review.openstack.org/59840918:18
*** priteau has quit IRC18:18
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient master: Refactor Project Group editing to match other objects  https://review.openstack.org/59840718:21
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient master: Support linking to Project Groups by name  https://review.openstack.org/59840818:21
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient master: Link to Project Groups by name  https://review.openstack.org/59840918:21
openstackgerritMerged openstack-infra/project-config master: remove job settings for Chef OpenStack repositories  https://review.openstack.org/59861418:22
openstackgerritJeremy Stanley proposed openstack-infra/statusbot master: Identify with SASL  https://review.openstack.org/59904918:41
fungithat's not been tested yet, gonna wip18:41
clarkbAJaeger: ok email sent18:42
*** gfidente has quit IRC18:43
*** anteaya has quit IRC18:44
*** eernst has quit IRC18:47
*** eernst has joined #openstack-infra18:49
*** eernst has quit IRC18:50
*** eernst_ has joined #openstack-infra18:50
*** tosky has quit IRC18:52
fungiftr, statusbot does look like it's missing from a few channels (#ara, #congress and #edeploy as of the 2018-08-29 02:59 restart)18:54
*** eernst_ has quit IRC18:54
*** openstackstatus has quit IRC18:58
*** eernst has joined #openstack-infra19:03
openstackgerritDean Troyer proposed openstack-infra/project-config master: Set up stx-docs and stx-specs publish jobs  https://review.openstack.org/59905419:03
*** jmorgan1 has quit IRC19:04
clarkbcorvus: http://logs.openstack.org/03/598403/3/check/system-config-run-nodepool/2a3f470/job-output.txt.gz#_2018-08-31_17_39_22_264566 seems to be why the nodepool test is failing19:08
clarkbcorvus: do we apply the same hostnames to groups mapping in the testinfra tests that we do in production?19:08
clarkbI'm guessing we need to copy that file into /etc/ansible/hosts like we do with the group and host vars19:09
corvusclarkb: we should -- current test behavior relies on it actually.  it's in inventory/groups.yaml19:11
clarkbhuh I wonder why nl01 and nb02 don't become part of the nodepool-launcher and nodepool-builder groups then19:12
corvusclarkb: this confirms that: http://logs.openstack.org/03/598403/3/check/system-config-run-nodepool/2a3f470/job-output.txt.gz#_2018-08-31_17_37_50_13308619:13
corvusthey show up in nodepool, which is due to the same file19:13
corvusi see the error19:14
corvusthe test caught a real bug :)19:14
clarkbfwiw http://logs.openstack.org/03/598403/3/check/system-config-run-nodepool/2a3f470/ansible/ is where I would expect to find the config doing that mapping but I don't see it in those dirs19:14
corvusclarkb: it's in the repo itself19:14
corvusif you look in ansible.cfg in the link you just pasted, you'll see the inventory line points directly into /opt/system-config19:15
corvuswhich is actually what production does too19:15
clarkbinventory = /etc/ansible/hosts/inventory.yaml,/opt/system-config/inventory/groups.yaml got it19:15
*** rh-jelabarre has quit IRC19:15
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add a run-nodepool job  https://review.openstack.org/59840319:16
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add testinfra tests for bridge  https://review.openstack.org/59840419:16
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Fix regex in nodepool ansible group matcher  https://review.openstack.org/59905519:16
clarkbaha, that should be squashed down into the first change in the stack I think19:16
clarkbI guess we can confirm it works first19:16
corvusclarkb: probably so; want me to go ahead and squash?19:17
clarkbya I think we should otherwise it will be broken if half the stack merges19:17
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Install limestone CA on hosts using openstacksdk  https://review.openstack.org/59302919:17
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Manage clouds.yaml files in ansible  https://review.openstack.org/59832919:17
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add a run-nodepool job  https://review.openstack.org/59840319:17
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add testinfra tests for bridge  https://review.openstack.org/59840419:17
openstackgerritJeremy Stanley proposed openstack-infra/statusbot master: Identify with SASL  https://review.openstack.org/59904919:21
*** fried_rolls is now known as fried_rice19:23
clarkbpabelanger: corvus the executor's accepting jobs graph right now is interesting. I think it is telling us that we may need another executor if we want to run jobs at full capacity?19:33
clarkb(I ended up here by way of "when will we get results on the nodepool test above and wow look at that backlog, oh we have ready nodeswhy aren't they used")19:34
clarkbit does look a lot better than it used to fwiw so new kernel and extra node is helping19:34
*** e0ne has joined #openstack-infra19:36
*** openstackstatus has joined #openstack-infra19:37
*** ChanServ sets mode: +v openstackstatus19:37
clarkbwe have 1.154k max nodes according to logstash which does line up roughly with 100 test nodes per executor19:39
clarkbinterseting how stable that number has been over time19:39
corvusclarkb: there's a lot of hysteresis right now due to the large number of dynamic config changes + large number of config changes merging, so i think the workload is slightly abnormal.  the hysteresis causes jobs to be launched in bursts which causes the executor governors to kick in19:39
clarkbah19:39
clarkbI was hoping we would be able to merge the clouds.yaml stack today, but based on job result throughput that might be a monday thing (monday is ah oliday too)19:41
corvusclarkb: further evidence of this -- i think the main indicator for "we need more executors" is the "queued jobs" metric in the executor queue graph.  that's trending back down to zero as the number of changes in pipelines is reduced and the hysteresis is reduced.  at the same time, we still have a 1k node request backlog, so we're still running all-out.19:42
corvus(adding another executor would help with the burstiness, but probably isn't needed strictly based on the system load alone)19:43
clarkband to improve the hyseresis we would need to make zuul config loading quicker right?19:46
corvusclarkb: yep19:47
openstackgerritDean Troyer proposed openstack-infra/project-config master: Set up stx-docs and stx-specs publish jobs  https://review.openstack.org/59905419:48
*** ansmith has quit IRC19:52
*** hasharAway is now known as hashar19:53
*** eernst has quit IRC19:54
*** openstackstatus has quit IRC19:55
*** eernst has joined #openstack-infra19:56
*** eharney has quit IRC19:57
*** openstackstatus has joined #openstack-infra19:58
*** ChanServ sets mode: +v openstackstatus19:58
openstackgerritJeremy Stanley proposed openstack-infra/statusbot master: Identify with SASL  https://review.openstack.org/59904920:01
*** jcoufal has quit IRC20:02
*** kgiusti has left #openstack-infra20:03
*** e0ne has quit IRC20:06
openstackgerritJeremy Stanley proposed openstack-infra/puppet-statusbot master: Use port 6697 for IRC over SSL/TLS  https://review.openstack.org/59906120:07
openstackgerritJeremy Stanley proposed openstack-infra/statusbot master: Identify with SASL  https://review.openstack.org/59904920:08
*** openstackstatus has quit IRC20:11
*** openstackstatus has joined #openstack-infra20:12
*** ChanServ sets mode: +v openstackstatus20:12
fungicorvus: okay, i think those ^ are good now20:12
fungiporting was a little fiddly as ptgbot and statusbot had a few dissimilarities20:13
openstackgerritNicolas Hicher proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS  https://review.openstack.org/52873920:13
clarkbspeaking of sasl and irc bots, can we drop +r at this point?20:15
*** nicolasbock has quit IRC20:16
clarkbinfra-root ^ thoughts on doing that oever the weekend in this cahnnel to see if it is quiet? (I'm not in any channels that are not +r so unsure if attack persists)20:16
openstackgerritMerged openstack-infra/zuul master: Fix typo in documentation chown commands  https://review.openstack.org/59904020:16
corvusclarkb: no objection; i also don't know the latest20:17
clarkbI seem to remember that was a manual change due to how chanserv works? or am I misremembering?20:18
clarkbianw did update accessbo20:18
AJaegerclarkb: I think ianw updated accessbot, so it's now automated20:20
clarkbya I reviewed those changes and the merged config file seems to indicate this is the case20:20
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Serve project SSH keys and document  https://review.openstack.org/59906320:20
*** boden has quit IRC20:21
*** jmorgan1 has joined #openstack-infra20:23
*** holser_ has joined #openstack-infra20:23
openstackgerritMerged openstack-infra/puppet-ptgbot master: Use port 6697 for IRC over SSL/TLS  https://review.openstack.org/59904620:23
openstackgerritClark Boylan proposed openstack-infra/project-config master: Disable +r on #openstack-infra  https://review.openstack.org/59906520:24
clarkbAJaeger: corvus fungi ^ I think I got that right20:24
clarkbhrm maybe I don't need the dest channel to remove the +f20:25
clarkbchanserv help output at least implies -f is fine without a channel specifier20:25
* clarkb updates20:25
openstackgerritClark Boylan proposed openstack-infra/project-config master: Disable +r on #openstack-infra  https://review.openstack.org/59906520:25
fungihrm, stuck on the remaining signing subkey steps... gnupg 2.1 doesn't seem to want to let me set an empty passphrase on the signing subkey20:29
*** anteaya has joined #openstack-infra20:31
clarkbfungi: looking at the statusbot change doesn't freenode sasl auth require a cert?20:32
clarkbor am I thinking of some other ircd20:32
*** niceplace has joined #openstack-infra20:32
fungii've never used a cert with it20:33
fungijust a password20:33
clarkbok I'm probably thinking of some other ircd then20:33
clarkbhttps://www.weechat.org/files/doc/stable/weechat_user.en.html#irc_sasl_ecdsa_nist256p_challenge is the method I'm thinking of I think20:34
clarkbanyways change to statusbot lgtm20:34
AJaegerclarkb: Idea looks fine - I babysit when it is merged...20:34
AJaegerclarkb: wanted to say : I suggest to babysit...20:35
clarkbAJaeger: ya I may also manually run the mlock before approving it to test it20:35
*** trown is now known as trown|outtypewww20:36
*** openstackstatus has quit IRC20:36
clarkbbut need reviews first :)20:36
AJaegerclarkb: you get my vote of confidence ;)20:36
*** openstackstatus has joined #openstack-infra20:37
*** ChanServ sets mode: +v openstackstatus20:37
AJaegerregarding reviewing - I took some time to update descriptions for all "new" templates in openstack-zuul-jobs. If anybody wants to review descriptions, please go over https://review.openstack.org/59859120:38
* AJaeger waves good night and wishes everybody a great - and for those with public holiday on Monday: and long - weekend20:38
*** harlowja has joined #openstack-infra20:46
*** imacdonn has quit IRC20:48
*** e0ne has joined #openstack-infra20:56
EvilienMinfra: just a heads up about the new centos CBS proxy: it works super fine, thanks again20:56
*** cmurphy|vacation is now known as cmurphy20:58
*** holser_ has quit IRC20:58
*** anteaya has quit IRC20:59
*** e0ne has quit IRC21:00
*** imacdonn has joined #openstack-infra21:01
*** hashar has quit IRC21:02
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: WIP: add-build-sshkey: interact with ssh agent directly  https://review.openstack.org/59907321:03
clarkbcorvus: http://logs.openstack.org/03/598403/5/check/system-config-run-nodepool/2e7e80a/job-output.txt.gz#_2018-08-31_20_23_49_183839 this split config mgmt thing is fun :/ problem now is that the nodepool user doesn't exist on the test nodes because puppet does that21:17
clarkbwhich is a good catch because we run this before puppet on a fresh host21:17
clarkbI Guess we go ahead and ensure the group and user in ansible too for now?21:18
corvusclarkb: yeah, i think we should do that21:20
*** eernst has quit IRC21:32
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add a run-nodepool job  https://review.openstack.org/59840321:33
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add testinfra tests for bridge  https://review.openstack.org/59840421:33
clarkbcorvus: ^ that should do it21:33
ianwclarkb: accessbot should handle mode changes now; see also https://review.openstack.org/#/c/589001/ for a cleanup on that21:35
clarkbianw: does https://review.openstack.org/#/c/599065/ look right to you?21:36
ianwLGTM, but testing was all pretty manual, just me manually running on a reduced config on unimportant channels21:40
clarkbianw: that refactor appears to need to be coordinated with an update to our config? we use global.alumni currently21:40
clarkbwe also set global.mode21:41
ianwclarkb: oh, no sorry it should be safe.  that means remove the global-alumni and global-mode keys, and put them into the "global" variable21:41
ianwconfusing, sorry21:41
clarkbah21:41
clarkbimplicit global vs explicit global21:42
ianwthey don't need to be separate keys.  i didn't realise quite how it was hanging together when i first wrote it during spamageddon21:42
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: add-build-sshkey: Remove only the master key  https://review.openstack.org/59907321:43
*** jamesmcarthur has joined #openstack-infra21:44
corvusclarkb: we probably could just add tasks like that to playbooks rather than making a role.21:45
corvusclarkb: (notaminusone)21:46
*** florianf has quit IRC21:48
clarkbianw: left a comment on the refactor21:49
clarkbcorvus: so that they dont appear fully functional? I did it to avoid making the playbook huge21:50
openstackgerritIan Wienand proposed openstack-infra/puppet-accessbot master: Small refactor, remove global alumni and mode keys  https://review.openstack.org/58900121:57
openstackgerritIan Wienand proposed openstack-infra/puppet-accessbot master: Small refactor, remove global alumni and mode keys  https://review.openstack.org/58900121:58
corvusclarkb: i was thinking at some point we should just farm out to ethor playbooks.  so maybe "includ_playbook: nodepool" in base, then add stuff like that in there.  but right now, i'd say lets try lots of things and sort it out at the ptg :)21:58
corvuss/ethor/other/21:58
clarkbwf,21:59
clarkber wfm21:59
*** elbragstad has quit IRC22:02
*** EvilienM is now known as EmilienM22:04
*** ijw has joined #openstack-infra22:06
*** tosky has joined #openstack-infra22:09
*** ijw has quit IRC22:11
*** rlandy has quit IRC22:16
*** harlowja has quit IRC22:24
*** mriedem has quit IRC22:32
*** ijw has joined #openstack-infra22:34
*** markvoelker has joined #openstack-infra22:40
openstackgerritClark Boylan proposed openstack-infra/system-config master: Manage clouds.yaml files in ansible  https://review.openstack.org/59832922:50
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add a run-nodepool job  https://review.openstack.org/59840322:50
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add testinfra tests for bridge  https://review.openstack.org/59840422:50
clarkbcorvus: it might actually apss now I hope22:50
corvusclarkb: what was the latest bug?22:51
clarkbcorvus: nodepool clouds.yaml and bridge clouds.yaml have different secrets names and layout22:51
clarkbcorvus: I had to update the playbooks/zuul/templates/group_vars/nodepool_clouds.yaml.j2 contents22:51
clarkbthis is hold over from the old files, I don't want to change them as part of this as it would require a hiera/ansible group_vars update to do that22:51
clarkbwe can do that later if we want22:52
corvusclarkb: "neat"22:52
*** jrist has quit IRC22:58
*** tpsilva has quit IRC22:58
*** jamesmcarthur has quit IRC23:01
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: add-build-sshkey: Remove only the master key  https://review.openstack.org/59907323:03
clarkbarg I think it fails on a new thing now23:07
clarkbbut yay for testing23:07
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: add-build-sshkey: Remove only the master key  https://review.openstack.org/59907323:08
*** Swami has quit IRC23:08
*** jamesmcarthur has joined #openstack-infra23:10
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add a run-nodepool job  https://review.openstack.org/59840323:11
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add testinfra tests for bridge  https://review.openstack.org/59840423:11
clarkbcorvus: ^ the type of the content is bytes not str apparently23:11
corvusoh, huh.23:12
clarkband with that I need to pop out. Hopefully that is ready for us on monday/tuesday23:12
corvus++23:12
*** ijw has quit IRC23:12
*** tosky has quit IRC23:14
*** jamesmcarthur has quit IRC23:15
*** eernst has joined #openstack-infra23:22
*** ijw has joined #openstack-infra23:22
*** lamt has joined #openstack-infra23:23
openstackgerritJeremy Stanley proposed openstack-infra/project-config master: Add Freezer to StoryBoard  https://review.openstack.org/59691823:31
*** jamesmcarthur has joined #openstack-infra23:31
*** ijw has quit IRC23:32
openstackgerritMerged openstack-infra/project-config master: Set up slogging in Storyboard  https://review.openstack.org/59187123:36
*** r-daneel has quit IRC23:36
openstackgerritMerged openstack-infra/project-config master: Enable Searchlight to Use StoryBoard  https://review.openstack.org/59691623:39
*** jamesmcarthur has quit IRC23:40
openstackgerritMerged openstack-infra/project-config master: Add Freezer to StoryBoard  https://review.openstack.org/59691823:48
*** ijw has joined #openstack-infra23:51
*** jamesmcarthur has joined #openstack-infra23:53
*** ijw has quit IRC23:57
*** jamesmcarthur has quit IRC23:57
*** jamesmcarthur has joined #openstack-infra23:58
*** jamesmcarthur has quit IRC23:58
*** jamesmcarthur has joined #openstack-infra23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!