Sunday, 2017-06-18

*** wolverineav has joined #openstack-infra00:00
*** wolverineav has quit IRC00:05
*** gmann has quit IRC00:05
*** gmann has joined #openstack-infra00:05
*** Guest16483 has quit IRC00:06
*** annegentle has joined #openstack-infra00:08
*** annegentle is now known as Guest279900:09
*** wolverineav has joined #openstack-infra00:12
*** hongbin_ has joined #openstack-infra00:15
*** hongbin has quit IRC00:15
*** wolverineav has quit IRC00:17
*** wolverineav has joined #openstack-infra00:18
*** Guest2799 has quit IRC00:22
*** zz_dimtruck is now known as dimtruck00:26
*** hongbin has joined #openstack-infra00:37
*** hongbin_ has quit IRC00:37
*** wolverineav has quit IRC00:38
openstackgerritAnne Gentle proposed openstack-infra/reviewstats master: Adds api-ref, docs, or installdocs statistics measures  https://review.openstack.org/46128000:39
*** baoli has joined #openstack-infra00:43
*** baoli has quit IRC00:43
*** alex_xu has quit IRC00:49
openstackgerritRosario Di Somma proposed openstack-infra/shade master: Don't remove top-container element for user and project REST API calls  https://review.openstack.org/47515000:49
*** hongbin has quit IRC00:52
*** alex_xu has joined #openstack-infra00:54
*** lbragstad has joined #openstack-infra01:08
*** cshastri has joined #openstack-infra01:11
*** ihrachys has joined #openstack-infra01:11
*** jamesmcarthur has joined #openstack-infra01:12
*** jamesmcarthur has quit IRC01:13
*** jamesmcarthur has joined #openstack-infra01:13
*** ihrachys has quit IRC01:13
*** Julien-zte has quit IRC01:16
*** jamesmcarthur has quit IRC01:18
*** SumitNaiksatam has quit IRC01:18
*** SumitNaiksatam has joined #openstack-infra01:19
openstackgerritAnne Gentle proposed openstack-infra/reviewstats master: Adds api-ref, docs, or installdocs statistics measures  https://review.openstack.org/46128001:23
*** baoli has joined #openstack-infra01:30
*** wolverineav has joined #openstack-infra01:30
*** baoli has quit IRC01:34
*** wolverineav has quit IRC01:35
*** ykarel_ has joined #openstack-infra01:36
*** xinliang has quit IRC01:36
*** markvoelker has joined #openstack-infra01:45
*** bkero has quit IRC01:47
*** xinliang has joined #openstack-infra01:48
*** xinliang has quit IRC01:48
*** xinliang has joined #openstack-infra01:48
*** thorst has joined #openstack-infra01:49
*** lbragstad has quit IRC01:51
*** thorst has quit IRC01:53
*** Julien-zte has joined #openstack-infra01:58
*** Julien-zte has quit IRC02:06
*** ykarel_ has quit IRC02:08
*** gomarivera has joined #openstack-infra02:10
*** gomarivera has quit IRC02:12
*** gomarivera has joined #openstack-infra02:12
*** Goneri has quit IRC02:13
*** gomarivera_ has joined #openstack-infra02:16
*** gomarivera has quit IRC02:16
*** darvon has quit IRC02:18
*** markvoelker has quit IRC02:18
*** darvon has joined #openstack-infra02:20
*** yamahata_ has joined #openstack-infra02:44
*** thorst has joined #openstack-infra02:44
*** thorst has quit IRC02:44
*** Julien-zte has joined #openstack-infra02:54
*** bkero has joined #openstack-infra03:10
*** bkero has quit IRC03:11
*** bkero has joined #openstack-infra03:11
*** yamamoto has joined #openstack-infra03:11
*** Julien-zte has quit IRC03:23
*** yamahata_ has quit IRC03:24
*** wolverineav has joined #openstack-infra03:30
*** pradk has quit IRC03:34
*** wolverineav has quit IRC03:35
*** wolverineav has joined #openstack-infra03:37
*** wolverineav has quit IRC03:39
*** wolverineav has joined #openstack-infra03:39
*** Goneri has joined #openstack-infra03:41
*** isaacb has joined #openstack-infra03:45
*** isaacb has quit IRC03:51
*** yamamoto has quit IRC03:52
*** pradk has joined #openstack-infra03:54
*** peterstac has joined #openstack-infra04:00
*** dhinesh has joined #openstack-infra04:01
*** vgnbkr has joined #openstack-infra04:01
*** ianychoi has quit IRC04:02
*** schang has joined #openstack-infra04:03
*** dougshelley66 has joined #openstack-infra04:03
*** watersoul_ has quit IRC04:03
*** watersoul has joined #openstack-infra04:04
*** ramishra has joined #openstack-infra04:05
*** markvoelker has joined #openstack-infra04:15
*** isaacb has joined #openstack-infra04:17
*** yamamoto has joined #openstack-infra04:19
*** isaacb has quit IRC04:21
*** yamamoto has quit IRC04:29
*** thorst has joined #openstack-infra04:45
*** yamamoto has joined #openstack-infra04:46
*** markvoelker has quit IRC04:50
*** thorst has quit IRC04:50
*** salv-orl_ has quit IRC04:50
*** salv-orlando has joined #openstack-infra04:51
*** yamamoto has quit IRC04:51
*** wolverineav has quit IRC05:01
*** isaacb has joined #openstack-infra05:02
*** ykarel_ has joined #openstack-infra05:03
*** ramishra has quit IRC05:11
*** salv-orl_ has joined #openstack-infra05:15
*** salv-orlando has quit IRC05:18
*** yamamoto has joined #openstack-infra05:25
*** Dinesh_Bhor has quit IRC05:25
*** Goneri has quit IRC05:26
*** yamamoto has quit IRC05:26
*** jascott1 has quit IRC05:28
*** yamamoto has joined #openstack-infra05:28
*** yamamoto has quit IRC06:23
*** links has joined #openstack-infra06:37
*** hieulq_ has joined #openstack-infra06:40
*** dhinesh has quit IRC06:43
*** thorst has joined #openstack-infra06:46
*** markvoelker has joined #openstack-infra06:47
*** rcernin has joined #openstack-infra06:49
*** thorst has quit IRC06:51
*** hieulq__ has joined #openstack-infra06:54
*** hieulq_ has quit IRC06:55
*** rcernin has quit IRC07:02
*** rcernin has joined #openstack-infra07:02
*** aviau has quit IRC07:02
*** aviau has joined #openstack-infra07:03
*** pradk has quit IRC07:10
*** tesseract has joined #openstack-infra07:12
*** isaacb has quit IRC07:14
*** hieulq__ has quit IRC07:15
*** markvoelker has quit IRC07:19
*** yamamoto has joined #openstack-infra07:24
*** yamamoto has quit IRC07:30
*** tosky has joined #openstack-infra07:32
*** yamamoto has joined #openstack-infra07:32
*** yamamoto has quit IRC07:37
*** jascott1 has joined #openstack-infra07:44
openstackgerritMerged openstack-infra/project-config master: [magnum] Add experimental dcos job  https://review.openstack.org/47177607:58
*** andreww has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Re-enable beaker tests for puppet-midonet  https://review.openstack.org/47021008:01
*** zzzeek has joined #openstack-infra08:02
*** xarses has joined #openstack-infra08:02
*** xarses has quit IRC08:02
*** xarses has joined #openstack-infra08:03
*** andreww has quit IRC08:04
*** jamesmcarthur has joined #openstack-infra08:14
*** markvoelker has joined #openstack-infra08:16
openstackgerritMerged openstack-infra/project-config master: Re-enable beaker tests for puppet-midonet  https://review.openstack.org/47021008:17
*** HeOS has joined #openstack-infra08:17
*** HeOS has quit IRC08:18
*** jamesmcarthur has quit IRC08:18
*** HeOS has joined #openstack-infra08:18
*** lifeless has quit IRC08:20
openstackgerritMerged openstack-infra/project-config master: jenkins: scripts: propose_update: Pass project directory as first argument  https://review.openstack.org/47178508:20
*** lifeless has joined #openstack-infra08:20
openstackgerritMerged openstack-infra/project-config master: Add etcd3 to networking-generic-switch job  https://review.openstack.org/47234308:23
*** yamamoto has joined #openstack-infra08:24
*** yamamoto has quit IRC08:26
*** yolanda has quit IRC08:38
*** yolanda has joined #openstack-infra08:41
*** thorst has joined #openstack-infra08:47
*** yamamoto has joined #openstack-infra08:49
mtreinishfungi: so I can now confirm that mqtt/firehose isn't being blocked in china (at least on my hotel's wifi) we might not need the websockets interface after all08:50
mtreinishalthough it's probably still something useful to have08:50
*** markvoelker has quit IRC08:50
*** yamamoto has quit IRC08:51
*** thorst has quit IRC08:52
*** links has quit IRC08:54
*** yamamoto has joined #openstack-infra08:56
*** yamamoto has quit IRC09:00
*** yamamoto has joined #openstack-infra09:01
*** yamamoto has quit IRC09:06
openstackgerritAkihiro Motoki proposed openstack-infra/project-config master: Add neutron-fwaas-dashboard project  https://review.openstack.org/47516309:07
openstackgerritAkihiro Motoki proposed openstack-infra/project-config master: Add neutron-vpnaas-dashboard project  https://review.openstack.org/47516509:12
*** Julien-zte has joined #openstack-infra09:21
openstackgerritAkihiro Motoki proposed openstack-infra/project-config master: Add neutron-vpnaas-dashboard project  https://review.openstack.org/47516509:32
openstackgerritAkihiro Motoki proposed openstack-infra/project-config master: Add neutron-vpnaas-dashboard project  https://review.openstack.org/47516509:34
openstackgerritAkihiro Motoki proposed openstack-infra/project-config master: Add neutron-fwaas-dashboard project  https://review.openstack.org/47516309:36
*** yamamoto has joined #openstack-infra09:38
*** yamamoto has quit IRC09:43
*** ykarel_ has quit IRC09:47
*** thorst has joined #openstack-infra09:48
*** armax has quit IRC09:52
*** hieulq_ has joined #openstack-infra09:52
*** thorst has quit IRC09:53
*** armax has joined #openstack-infra09:53
*** armax has quit IRC09:53
*** armax has joined #openstack-infra09:53
*** armax has quit IRC09:54
*** armax has joined #openstack-infra09:54
*** armax has quit IRC09:55
*** armax has joined #openstack-infra09:55
*** sshnaidm has quit IRC09:55
*** armax has quit IRC09:55
*** armax has joined #openstack-infra09:56
*** armax has quit IRC09:56
*** armax has joined #openstack-infra09:57
*** armax has quit IRC09:57
*** armax has joined #openstack-infra09:57
*** armax has quit IRC09:58
*** Julien-zte has quit IRC10:00
*** cshastri has quit IRC10:05
*** gouthamr has quit IRC10:05
*** wfoster has quit IRC10:07
*** wfoster has joined #openstack-infra10:08
*** jamesmcarthur has joined #openstack-infra10:15
*** jamesmcarthur has quit IRC10:19
*** dhajare has joined #openstack-infra10:24
*** HeOS has quit IRC10:41
*** yamamoto has joined #openstack-infra10:43
*** yamamoto has quit IRC10:46
*** yamamoto has joined #openstack-infra10:46
*** markvoelker has joined #openstack-infra10:48
*** sshnaidm has joined #openstack-infra10:48
openstackgerritSagi Shnaidman proposed openstack-infra/project-config master: migrate periodic jobs on stable branches to oooq  https://review.openstack.org/47473610:50
*** ykarel_ has joined #openstack-infra10:56
*** yamamoto has quit IRC11:12
*** gomarivera_ has quit IRC11:13
*** dhajare has joined #openstack-infra11:14
*** salv-orlando has joined #openstack-infra11:15
*** salv-orl_ has quit IRC11:18
*** markvoelker has quit IRC11:21
*** hieulq_ has quit IRC11:21
*** hieulq_ has joined #openstack-infra11:22
*** yamamoto has joined #openstack-infra11:24
*** hieulq_ has quit IRC11:27
*** Julien-zte has joined #openstack-infra11:40
*** ykarel_ has quit IRC11:40
*** thorst has joined #openstack-infra11:49
*** thorst has quit IRC11:53
*** jkilpatr has joined #openstack-infra12:00
*** thorst has joined #openstack-infra12:03
*** thorst has quit IRC12:03
*** gomarivera has joined #openstack-infra12:10
fungimtreinish: i agree with you on both points12:13
*** yamamoto has quit IRC12:14
*** jamesmcarthur has joined #openstack-infra12:15
fungii'm not so much concerned about the great wall but rather all the lesser walls various corporate overlords impose upon their employees who then need months of negotiating with the security team there to get egress allowed to some new specific port (and often, it only gets allowed by specific destination address, which makes replacing servers that much harder)12:15
*** gomarivera has quit IRC12:16
*** markvoelker has joined #openstack-infra12:18
*** jamesmcarthur has quit IRC12:19
mordredfungi: ++12:23
*** abramley has joined #openstack-infra12:34
*** Julien-zte has quit IRC12:43
*** abramley has quit IRC12:46
*** Julien-zte has joined #openstack-infra12:47
*** e0ne has joined #openstack-infra12:48
openstackgerritMerged openstack-infra/shade master: Don't remove top-container element for user and project REST API calls  https://review.openstack.org/47515012:48
openstackgerritMerged openstack-infra/shade master: Convert keypairs tests to requests_mock  https://review.openstack.org/47509012:48
openstackgerritMerged openstack-infra/shade master: Add normalization and functional tests for keypairs  https://review.openstack.org/47513412:48
*** markvoelker has quit IRC12:51
openstackgerritMerged openstack-infra/shade master: Convert keypairs calls to REST  https://review.openstack.org/47509112:52
*** e0ne has quit IRC12:54
*** gomarivera has joined #openstack-infra13:11
*** jamesmcarthur has joined #openstack-infra13:14
*** yamamoto has joined #openstack-infra13:15
*** gomarivera has quit IRC13:16
*** jamesmcarthur has quit IRC13:19
*** yamamoto has quit IRC13:22
*** dimtruck is now known as zz_dimtruck13:27
*** dhajare has quit IRC13:38
*** zz_dimtruck is now known as dimtruck13:46
*** markvoelker has joined #openstack-infra13:49
*** rcernin has quit IRC13:53
*** dizquierdo has joined #openstack-infra14:02
*** thorst has joined #openstack-infra14:04
*** thorst has quit IRC14:09
*** gomarivera has joined #openstack-infra14:12
*** jascott1 has quit IRC14:16
*** gomarivera has quit IRC14:17
*** markvoelker has quit IRC14:22
*** Julien-zte has quit IRC14:27
*** Julien-zte has joined #openstack-infra14:29
*** Julien-zte has joined #openstack-infra14:30
*** Julien-zte has quit IRC14:30
*** Julien-zte has joined #openstack-infra14:31
*** Julien-zte has quit IRC14:32
*** sshnaidm has quit IRC14:33
*** Julien-zte has joined #openstack-infra14:33
*** Julien-zte has quit IRC14:34
*** lbragstad has joined #openstack-infra14:36
*** jascott1 has joined #openstack-infra14:45
*** anthonyper has quit IRC14:45
*** anthonyper has joined #openstack-infra14:45
*** andreww has joined #openstack-infra14:46
*** lbragstad has quit IRC14:46
*** xarses has quit IRC14:49
*** tumbarka has joined #openstack-infra14:50
*** andreww has quit IRC14:52
*** umbarkar has quit IRC14:53
fungiinfra-root: apparently the virtualenv on proposal.slave.o.o was creating python3.4-based environments by default, probably owing to the pip vs pip3 mishap from a few weeks ago15:05
fungii just fixed it by forcibly reinstalling virtualenv, but spot checking i find that release.slave.o.o is similarly affected so this may be more widespread than i expected15:05
fungifixed that one just now as well15:06
fungisigning01.ci is also affected15:07
fungifixed too15:07
fungii need to step away to tackle some chores, but if anybody finds more a quick `sudo pip install -U --force virtualenv` seems to take care of it15:08
*** gomarivera has joined #openstack-infra15:13
openstackgerritMonty Taylor proposed openstack-infra/shade master: Remove use of FakeServer from tests  https://review.openstack.org/47517715:17
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert FakeSecGroup to dict  https://review.openstack.org/47517815:17
*** gomarivera has quit IRC15:17
*** markvoelker has joined #openstack-infra15:19
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Zun] Turn dsvm job to voting  https://review.openstack.org/47184215:22
*** hongbin has joined #openstack-infra15:22
*** sshnaidm has joined #openstack-infra15:26
*** bobh has joined #openstack-infra15:28
*** hongbin_ has joined #openstack-infra15:31
odyssey4mefungi should we still be seeing DNS resolution failures in tests?15:33
*** e0ne has joined #openstack-infra15:33
odyssey4meI've been having to do a *lot* of rechecks over the last 36 hours.15:33
*** hongbin has quit IRC15:34
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert server group tests to requests_mock  https://review.openstack.org/47518115:41
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert Server Groups to REST  https://review.openstack.org/47518215:41
*** hongbin has joined #openstack-infra15:41
*** hongbin_ has quit IRC15:43
fungiodyssey4me: clarkb worked out that one of teh .org tld servers is intermittently returning bad results for lots of names and so recursion ends up negative caching some failures as a result. not sure there's much more we can do to work around that if it's an issue globally15:46
*** e0ne has quit IRC15:49
*** baoli has joined #openstack-infra15:50
*** markvoelker has quit IRC15:52
fungi2001:500:c::1 a.k.a. b0.org.afilias-nst.org is the one he mentioned being an issue15:53
fungiodyssey4me: we also merged a change of clarkb's to add error logging for unbound which should have made it into today's images15:55
fungiso maybe we can narrow down the issue a little better15:55
clarkbya next step will be grabbing that log file in jobs15:55
clarkbor just dumping its contents or something15:56
clarkb/var/log/unbound.log15:56
*** e0ne has joined #openstack-infra15:56
clarkbif we can pin it to that host in our job logs there then next step is file a bug with the nameserver I think15:58
*** baoli has quit IRC15:59
openstackgerritTom Barron proposed openstack-infra/project-config master: Promote manila lvm centos 7 job to voting  https://review.openstack.org/47230215:59
funginothing of note in recent nanog posts at least16:03
*** thorst has joined #openstack-infra16:05
*** hongbin_ has joined #openstack-infra16:10
*** ykarel_ has joined #openstack-infra16:11
*** hongbin has quit IRC16:11
*** thorst has quit IRC16:12
*** gomarivera has joined #openstack-infra16:14
*** hongbin_ has quit IRC16:15
*** hongbin has joined #openstack-infra16:16
*** gomarivera has quit IRC16:18
*** e0ne has quit IRC16:20
*** e0ne has joined #openstack-infra16:21
*** e0ne has quit IRC16:21
*** e0ne has joined #openstack-infra16:24
*** bobh has quit IRC16:27
*** ykarel_ has quit IRC16:33
*** gomarivera has joined #openstack-infra16:34
*** armax has joined #openstack-infra16:36
*** tesseract has quit IRC16:38
*** nicolasbock has joined #openstack-infra16:41
*** armax has quit IRC16:44
*** markvoelker has joined #openstack-infra16:50
*** hongbin_ has joined #openstack-infra16:58
openstackgerritEgor Borisov proposed openstack-infra/jenkins-job-builder master: Adds integration with Allure Report plugin for Jenkins into the `publishers` section.  https://review.openstack.org/47518516:59
*** hongbin has quit IRC17:00
*** hongbin_ has quit IRC17:03
*** gomarivera has quit IRC17:04
*** gomarivera has joined #openstack-infra17:05
*** e0ne has quit IRC17:05
*** e0ne has joined #openstack-infra17:05
*** hongbin has joined #openstack-infra17:06
*** e0ne has quit IRC17:06
*** e0ne has joined #openstack-infra17:06
*** e0ne has quit IRC17:07
*** e0ne has joined #openstack-infra17:07
*** e0ne has quit IRC17:07
*** e0ne has joined #openstack-infra17:08
*** e0ne has quit IRC17:08
*** superdan is now known as dansmith17:08
*** e0ne has joined #openstack-infra17:09
*** e0ne has quit IRC17:09
*** gomarivera has quit IRC17:09
*** hongbin has quit IRC17:10
*** hongbin has joined #openstack-infra17:11
*** salv-orl_ has joined #openstack-infra17:16
*** gomarivera has joined #openstack-infra17:18
*** salv-orlando has quit IRC17:18
*** markvoelker has quit IRC17:23
*** gomarivera has quit IRC17:30
*** gomarivera has joined #openstack-infra17:31
*** hongbin has quit IRC17:32
*** gomarivera has quit IRC17:35
*** bobh has joined #openstack-infra17:39
*** bobh has quit IRC17:42
*** gomarivera has joined #openstack-infra17:43
*** AJaeger has joined #openstack-infra17:45
AJaegerinfra-root, I see two project-config changes in the gate queue - waiting for a job since over 30 mins. https://review.openstack.org/#/c/472499/ is the first one - waiting for ubuntu-xenial node. we should have 19 ready nodes...17:47
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert hypervisor test to requests_mock  https://review.openstack.org/47518618:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert hypervisor list to REST  https://review.openstack.org/47518718:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert host aggregate tests to requests_mock  https://review.openstack.org/47518818:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert host aggregates calls to REST  https://review.openstack.org/47518918:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Convert remaining nova tests to requests_mock  https://review.openstack.org/47519018:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Translate final nova calls to REST  https://review.openstack.org/47519118:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Remove novaclient from shade's dependencies  https://review.openstack.org/47519218:00
*** thorst has joined #openstack-infra18:01
*** thorst has quit IRC18:02
fungiAJaeger: i count a total of 6 ubuntu-xenial in a ready state, all for just a few seconds except one which looks like it's been sitting stale for nearly a month18:07
fungii'm going to delete that one just to keep tidy18:07
*** dkehn has joined #openstack-infra18:08
fungii wonder if the mergers are having a problem18:13
*** kzaitsev1pi has joined #openstack-infra18:18
*** kzaitsev_pi has quit IRC18:18
fungi2017-06-18 17:17:39,321 DEBUG zuul.Repo: CreateZuulRef master/Z3f73e4f638184f898b1f08ff55fcf9b1 at 7b6e516128071203bf760314f029614e9622199e on <git.Repo "/var/lib/zuul/git/openstack-infra/project-config/.git">18:19
fungithat looks okay18:19
*** markvoelker has joined #openstack-infra18:20
mordredfungi: maybe the configure-mirrors nodepool script combined with the ongoing dns issues is just causing nodepool to be slowering getting working nodes out there?18:23
fungiwell, it seems to be handing nodes out to other (lower priority) jobs18:24
fungi2017-06-18 17:06:34,447 INFO zuul.Gearman: Launch job gate-project-config-layout (uuid: dc445a8f1380470b87aa9daf067e4c9b) for change <Change 0x7fd3fcb22e90 472499,1> with dependent changes []18:29
fungiso the scheduler at least got that far18:29
fungi2017-06-18 17:06:34,449 DEBUG zuul.DependentPipelineManager: Adding build <Build dc445a8f1380470b87aa9daf067e4c9b of gate-project-config-layout on <Worker Unknown>> of job gate-project-config-layout to item <QueueItem 0x7fd37ec23750 for <Change 0x7fd3fcb22e90 472499,1> in gate>18:30
fungii see no further reference to Build dc445a8f1380470b87aa9daf067e4c9b in the scheduler's debug log18:31
fungigearman shows two outstanding jobs for build:gate-project-config-layout which are i think the ones in the gate18:35
fungiAJaeger: wild goose chase there... gate-project-config-layout uses ubuntu-trusty nodes, not ubuntu-xenial18:37
fungiwe have zero ready ubuntu-trusty nodes18:38
fungionly building and delete18:38
fungino used either18:38
fungiprobably we just put new images into rotation and the trusty ones aren't booting to completion18:39
fungiyeah, they all got uploaded around 2.5-3 hours ago18:40
mordredfungi: ahhh. yah -that would do it18:40
fungii can go ahead and delete these for now so we build from yesterday's images18:40
fungioh, though we need to pause rebuilding for ubuntu-trusty to do that, right?18:41
*** dhinesh has joined #openstack-infra18:42
fungii need to go attend to the grill for a few minutes, but can work out how to do that when i get back if nobody else beats me to it18:42
*** links has joined #openstack-infra18:49
*** markvoelker has quit IRC18:52
*** eranrom has quit IRC19:01
fungiclarkb: looks from the debug output like we've somehow tanked unbound on ubuntu-trusty. consistent ready node script failures attempting to lookup git.openstack.org: "connection timed out; no servers could be reached"19:02
*** armax has joined #openstack-infra19:03
*** yamahata_ has joined #openstack-infra19:05
openstackgerritJeremy Stanley proposed openstack-infra/project-config master: Pause rebuilding ubuntu-trusty images  https://review.openstack.org/47519519:05
fungiif someone wants to quick-approve that ^ then i'll delete the latest ubuntu-trusty images once it lands19:06
mordredfungi: done19:10
*** eranrom has joined #openstack-infra19:10
fungimuch appreciated19:11
mordredfungi: yay for sundays!19:11
*** HeOS has joined #openstack-infra19:11
fungioh, right, that change is never going to pass testing because it will wait indefinitely for a node19:18
*** mhickey has joined #openstack-infra19:19
*** eranrom has quit IRC19:19
fungirather than figure out which of the various nodepool servers i need to add to the emergency list and manually apply that change on, i'm just going to bypass gating19:19
openstackgerritMerged openstack-infra/project-config master: Pause rebuilding ubuntu-trusty images  https://review.openstack.org/47519519:21
*** gomarivera has quit IRC19:25
*** gomarivera has joined #openstack-infra19:26
*** thorst has joined #openstack-infra19:34
*** thorst has quit IRC19:34
*** armax has quit IRC19:37
*** links has quit IRC19:38
*** yamahata_ has quit IRC19:44
fungivoluptuous.error.MultipleInvalid: extra keys not allowed @ data['labels'][1]['pause']19:47
openstackgerritJeremy Stanley proposed openstack-infra/project-config master: Move ubuntu-trusty image pause to correct section  https://review.openstack.org/47519619:49
fungiself-approving and manually merging that ^ now19:49
*** markvoelker has joined #openstack-infra19:50
openstackgerritMerged openstack-infra/project-config master: Move ubuntu-trusty image pause to correct section  https://review.openstack.org/47519619:50
openstackgerritMerged openstack-infra/shade master: Remove use of FakeServer from tests  https://review.openstack.org/47517719:51
openstackgerritMerged openstack-infra/shade master: Convert FakeSecGroup to dict  https://review.openstack.org/47517819:51
openstackgerritMerged openstack-infra/shade master: Convert server group tests to requests_mock  https://review.openstack.org/47518119:52
openstackgerritMerged openstack-infra/shade master: Convert Server Groups to REST  https://review.openstack.org/47518219:52
openstackgerritMerged openstack-infra/shade master: Convert hypervisor test to requests_mock  https://review.openstack.org/47518619:52
*** nicolasbock has quit IRC19:53
openstackgerritMerged openstack-infra/shade master: Convert hypervisor list to REST  https://review.openstack.org/47518719:53
*** gouthamr has joined #openstack-infra19:57
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Add script parsing status multinode periodic jobs  https://review.openstack.org/47519720:03
*** andreww has joined #openstack-infra20:20
*** xarses_ has quit IRC20:23
*** markvoelker has quit IRC20:24
fungigrr, pausing seems to just prevent building new images, not reuploading already built existing one. i guess i need to delete the built copy too20:28
fungistarting to think the increased complexity warrants a `nodepool image-rollback` subcommand or something20:28
fungido i have to dib-image-delete on the builder where it built, or is the scheduler smart enough to negotiate that through zk?20:30
fungii guess i'll find out20:30
*** markvoelker has joined #openstack-infra20:30
fungiahh, nice, that seems to work20:30
fungithough now i need to wait for these reuploads to complete before i can delete them again20:33
fungiCannot delete because image upload in progress20:33
openstackgerritMerged openstack-infra/project-config master: Add tempest OOOQ job to experimental for tripleo  https://review.openstack.org/47249920:34
openstackgerritMerged openstack-infra/project-config master: Run gate-tempest-dsvm-neutron-multinode-full in devstack check  https://review.openstack.org/47101620:35
fungiat least project-config is dislodged again20:36
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Add multinode jobs to promotion  https://review.openstack.org/47520020:44
*** yamamoto has joined #openstack-infra20:50
*** yamamoto has quit IRC20:54
*** wolverineav has joined #openstack-infra20:55
*** xarses_ has joined #openstack-infra21:01
*** xarses has joined #openstack-infra21:04
*** xarses_ has quit IRC21:06
*** jascott1 has quit IRC21:07
*** arxcruz|off is now known as arxcruz21:09
*** xarses has quit IRC21:21
*** askb has joined #openstack-infra21:26
fungihah, so it uploaded them all but auto-deleted them as soon as each one was uploaded21:28
fungi#status log Image builds for ubuntu-trusty are paused and have been rolled back to yesterday until DNS issues can be unraveled21:29
openstackstatusfungi: finished logging21:29
*** aeng has joined #openstack-infra21:32
*** thorst has joined #openstack-infra21:35
*** wolverineav has quit IRC21:39
*** thorst has quit IRC21:40
*** funzo has quit IRC21:42
openstackgerritMerged openstack-infra/shade master: Convert host aggregate tests to requests_mock  https://review.openstack.org/47518821:48
*** salv-orl_ has quit IRC22:01
*** jkilpatr has quit IRC22:04
*** dhinesh has quit IRC22:07
*** dizquierdo has quit IRC22:14
*** Julien-zte has joined #openstack-infra22:18
*** Julien-zte has quit IRC22:30
*** Julien-zte has joined #openstack-infra22:30
*** hongbin has joined #openstack-infra22:39
*** mhickey has quit IRC22:44
*** tosky has quit IRC22:49
*** SamYaple has quit IRC23:02
*** SamYaple has joined #openstack-infra23:03
openstackgerritMerged openstack-infra/shade master: Convert host aggregates calls to REST  https://review.openstack.org/47518923:13
clarkb fungi doea trust unbound not like that config?23:14
fungiclarkb: that's what i'm expecting based on purely circumstantial evidence right now. i haven't tested it, just saw that our latest trusty images had no working dns resolution23:18
fungifigure it's not the end of the world if trusty images are a day or two stale for now, so long as jobs keep flowing23:19
clarkbkk23:20
*** dhinesh has joined #openstack-infra23:20
*** jascott1 has joined #openstack-infra23:22
*** dimtruck is now known as zz_dimtruck23:32
*** thorst has joined #openstack-infra23:36
*** armax has joined #openstack-infra23:37
*** Julien-zte has quit IRC23:39
*** thorst has quit IRC23:41
*** hongbin has quit IRC23:44
*** hongbin has joined #openstack-infra23:45
*** armax has quit IRC23:53
*** lbragstad has joined #openstack-infra23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!