Saturday, 2016-08-13

*** roxanaghe has quit IRC00:02
*** tqtran has joined #openstack-infra00:06
*** zz_dimtruck is now known as dimtruck00:10
*** tqtran has quit IRC00:10
*** hparekh has joined #openstack-infra00:11
*** amitgandhinz has joined #openstack-infra00:12
*** Darkstorm has joined #openstack-infra00:12
*** gouthamr has quit IRC00:13
*** jcoufal has quit IRC00:16
*** amitgandhinz has quit IRC00:16
*** gyee has quit IRC00:18
*** bswartz has quit IRC00:18
*** matrohon has quit IRC00:19
*** spzala has quit IRC00:23
*** spzala has joined #openstack-infra00:23
*** Darkstorm has quit IRC00:23
*** yamahata has joined #openstack-infra00:26
*** adrian_otto has quit IRC00:26
*** adrian_otto has joined #openstack-infra00:27
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add bindep.txt support to zuul layout  https://review.openstack.org/35511600:28
*** spzala has quit IRC00:29
*** nwkarsten has joined #openstack-infra00:29
*** adrian_otto has quit IRC00:32
*** nwkarsten has quit IRC00:34
*** Apoorva_ has joined #openstack-infra00:35
*** bswartz has joined #openstack-infra00:35
*** sc` has quit IRC00:35
*** Apoorva has quit IRC00:38
*** fguillot_ has joined #openstack-infra00:39
*** Apoorva_ has quit IRC00:39
*** adrian_otto has joined #openstack-infra00:49
*** tkelsey has joined #openstack-infra00:51
*** sdake has quit IRC00:51
*** thorst_ has joined #openstack-infra00:53
*** raunak has joined #openstack-infra00:54
*** tphummel has quit IRC00:54
*** markvoelker has joined #openstack-infra00:55
*** tkelsey has quit IRC00:55
*** pvaneck has quit IRC00:57
*** oomichi has quit IRC00:58
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Make ZK fixture more robust  https://review.openstack.org/35513100:58
*** armax has quit IRC00:58
*** markvoelker has quit IRC01:00
*** thorst_ has quit IRC01:01
*** raunak has quit IRC01:04
*** thorst_ has joined #openstack-infra01:04
*** spzala has joined #openstack-infra01:05
*** thorst_ has quit IRC01:07
*** mriedem has joined #openstack-infra01:11
*** amitgandhinz has joined #openstack-infra01:13
openstackgerritMerged openstack-infra/zuul: Fix addFailTest  https://review.openstack.org/35263701:14
openstackgerritMerged openstack-infra/zuul: Re-enable test_connection tests  https://review.openstack.org/35194901:14
openstackgerritMerged openstack-infra/zuul: Tests: update references from worker to launch_server  https://review.openstack.org/35376801:14
openstackgerritMerged openstack-infra/zuul: Tests: add an extra debug line when stalled  https://review.openstack.org/35376901:14
openstackgerritMerged openstack-infra/shade: Deal with clouds that don't have fips betterer  https://review.openstack.org/35508301:14
*** baoli has joined #openstack-infra01:18
*** amitgandhinz has quit IRC01:18
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Make the kolla-kubernetes relate jobs non-voting  https://review.openstack.org/35513201:20
cloudnullis there a known issue with the infra-mirrors from earlier today? I have a bunch of errors in my gate jobs with the following in various regions: "HTTPError: 404 Client Error: Not Found for url: http://mirror.iad.rax.openstack.org/wheel/ubuntu-16.04-x86_64/pip/"?01:21
cloudnullhttp://logs.openstack.org/52/354952/1/check/gate-openstack-ansible-os_horizon-ansible-func-ubuntu-xenial-nv/4ffb6b3/console.html#_2016-08-12_22_58_25_53070101:22
cloudnullthat URL is still 404, so idk if its something with the mirror or something we're doing wrong.01:22
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: add post-send script option  https://review.openstack.org/35513501:26
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova schedule bug 1612875  https://review.openstack.org/35513601:29
openstackbug 1612875 in OpenStack Compute (nova) "FixedIPsTestJson fails server build with "was re-scheduled: operation failed: filter 'nova-no-nd-reflection' already exists with uuid"" [Undecided,New] https://launchpad.net/bugs/161287501:29
Jeffrey4ldoes anyone have ideas why the CI result not shown on the top of the Patch Set page.01:29
*** yamamoto has joined #openstack-infra01:32
*** tphummel has joined #openstack-infra01:34
*** adrian_otto has quit IRC01:39
*** adrian_otto has joined #openstack-infra01:39
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: use base_email_create to customize email flexible  https://review.openstack.org/35513901:45
*** armax has joined #openstack-infra01:47
openstackgerritMerged openstack-infra/elastic-recheck: Add query for nova schedule bug 1612875  https://review.openstack.org/35513601:49
openstackbug 1612875 in OpenStack Compute (nova) "FixedIPsTestJson fails server build with "was re-scheduled: operation failed: filter 'nova-no-nd-reflection' already exists with uuid"" [Undecided,New] https://launchpad.net/bugs/161287501:49
*** thorst_ has joined #openstack-infra01:50
*** rbuzatu has joined #openstack-infra01:56
*** markvoelker has joined #openstack-infra01:59
*** tphummel has quit IRC02:00
*** rbuzatu has quit IRC02:01
*** baoli has quit IRC02:01
*** yamamoto has quit IRC02:02
*** adrian_otto has quit IRC02:02
*** baoli has joined #openstack-infra02:02
*** yamamoto has joined #openstack-infra02:02
*** baoli has quit IRC02:03
*** baoli has joined #openstack-infra02:04
*** docaedo has quit IRC02:05
*** eharney has quit IRC02:05
*** yamamoto has quit IRC02:07
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: customize abort/failure/success/fixed email notification  https://review.openstack.org/35514502:07
*** baoli has quit IRC02:09
*** baoli has joined #openstack-infra02:10
*** julim has quit IRC02:12
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: use base_email_create to customize email flexible  https://review.openstack.org/35513902:13
*** amitgandhinz has joined #openstack-infra02:14
*** rajinir has quit IRC02:15
*** burgerk has joined #openstack-infra02:17
*** thorst_ has quit IRC02:18
*** amitgandhinz has quit IRC02:20
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Add gate-novaclient-dsvm-functional-neutron-nv job  https://review.openstack.org/35514802:22
*** yamahata has quit IRC02:23
*** burgerk has quit IRC02:23
*** thorst_ has joined #openstack-infra02:24
*** thorst_ has quit IRC02:25
*** adrian_otto has joined #openstack-infra02:26
*** raunak has joined #openstack-infra02:31
*** elo has joined #openstack-infra02:37
*** eric_lopez has joined #openstack-infra02:40
*** elo has quit IRC02:41
*** thorst_ has joined #openstack-infra02:45
*** burgerk has joined #openstack-infra02:45
*** baoli has quit IRC02:46
*** adrian_otto has quit IRC02:46
*** baoli has joined #openstack-infra02:46
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: use base_email_create to customize email flexible  https://review.openstack.org/35513902:47
*** thorst_ has quit IRC02:48
*** burgerk has quit IRC02:50
*** sdake has joined #openstack-infra02:50
*** baoli has quit IRC02:51
*** tkelsey has joined #openstack-infra02:53
*** mriedem has quit IRC02:53
Jeffrey4lwhat does the `dsvm` stand for in jobs name?02:56
*** thorst_ has joined #openstack-infra02:57
*** tkelsey has quit IRC02:58
*** mriedem has joined #openstack-infra02:58
*** thorst_ has quit IRC02:59
fungiJeffrey4l: "DevStack Virtual Machine" (now somewhat of a misnomer as they no longer run on different virtual machines but we haven't renamed them since)02:59
Jeffrey4lfungi, so, i can remove this word in the job, right?   Kolla's job have a long name, i want to shorten them.03:01
fungiJeffrey4l: ci result table display is broken by the gerrit upgrade earlier today, but we're hoping someone will whip up a javascript patch this weekend to make it work with the slightly changed hyperlink format in our ci comments03:02
*** yamamoto has joined #openstack-infra03:02
Jeffrey4lThanks for the infor.03:03
fungiJeffrey4l: yes, that's probably a fine part of the name to remove. for example a lot of projects have functional test jobs that use devstack-gate but don't say dsvm in their names03:03
Jeffrey4l*info03:03
Jeffrey4lcool. I will try to remove them.03:03
*** links has joined #openstack-infra03:04
*** yamamoto has quit IRC03:05
*** yamamoto has joined #openstack-infra03:06
*** yamamoto has quit IRC03:06
*** yamamoto has joined #openstack-infra03:06
*** yamamoto has quit IRC03:07
*** sdake has quit IRC03:09
fungiJeffrey4l: the history is that back when we had separate, e.g., bare-trusty and devstack-trusty job nodes we did a pattern match on dsvm in job names to decide which ones ran on devstack-specific images. now that we've switched to installing distro packages during job runtime for all jobs we've been able to reduce down to using common job nodes for all job types03:10
fungiso the "dsvm" in some job names is vestigial03:11
Jeffrey4lroger that. thanks.03:11
fungisimilar to the "check-" or "gate-" prefixes on lots of job names03:11
Jeffrey4lso check- gate- is useless, right?03:11
fungiwe used to use those to be able to run check pipeline and gate pipeline jobs on different providers03:11
funginow we run the same jobs everywhere03:12
Jeffrey4lgreat. i will remove these two word :D03:12
fungiso, yes, you could likely drop "gate-" from the start of your job names if they're already too long03:12
Jeffrey4li will. it is really long now. like `gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv`03:13
*** yamamoto has joined #openstack-infra03:15
*** yamamoto has quit IRC03:16
*** thorst_ has joined #openstack-infra03:16
*** amitgandhinz has joined #openstack-infra03:16
*** yamamoto has joined #openstack-infra03:17
*** raunak has quit IRC03:18
*** thorst_ has quit IRC03:19
*** thorst_ has joined #openstack-infra03:20
*** amitgandhinz has quit IRC03:21
*** yamamoto has quit IRC03:21
*** thorst_ has quit IRC03:21
*** eric_lopez has quit IRC03:22
*** thorst_ has joined #openstack-infra03:22
*** spzala has quit IRC03:27
*** spzala has joined #openstack-infra03:28
*** elo has joined #openstack-infra03:28
*** rbuzatu has joined #openstack-infra03:28
*** thorst_ has quit IRC03:29
*** ansiwen has quit IRC03:30
*** mdbooth has quit IRC03:30
*** mdbooth has joined #openstack-infra03:31
*** ansiwen has joined #openstack-infra03:32
*** spzala has quit IRC03:32
*** raunak has joined #openstack-infra03:32
*** rbuzatu has quit IRC03:33
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: customize abort/failure/success/fixed email notification  https://review.openstack.org/35514503:33
*** sdake has joined #openstack-infra03:34
openstackgerritEmilien Macchi proposed openstack-infra/project-config: fix typo in comment  https://review.openstack.org/35515303:39
EmilienManteaya: ^03:39
EmilienMfixed :)03:39
*** Julien-zte has quit IRC03:41
*** timl0101 has quit IRC03:43
EmilienMsomething changed in Gerrit, isn't? I we don't see CI jobs results under the vote, and jobs colors don't work also03:48
*** ramishra_ has quit IRC03:49
*** ramishra has joined #openstack-infra03:51
*** sdake has quit IRC03:57
*** sdake has joined #openstack-infra04:04
*** tqtran has joined #openstack-infra04:08
*** tqtran has quit IRC04:12
*** dkehn has quit IRC04:14
*** dkehn_ has quit IRC04:15
*** dkehn has joined #openstack-infra04:16
*** amitgandhinz has joined #openstack-infra04:17
*** yamamoto has joined #openstack-infra04:19
*** kzaitsev_mb has joined #openstack-infra04:19
*** adrian_otto has joined #openstack-infra04:20
*** amitgandhinz has quit IRC04:21
*** csomerville has quit IRC04:22
*** markvoelker has quit IRC04:23
*** cody-somerville has joined #openstack-infra04:23
*** cody-somerville has joined #openstack-infra04:23
*** yamamoto has quit IRC04:24
*** thorst_ has joined #openstack-infra04:25
*** dkehn_ has joined #openstack-infra04:26
*** gouthamr has joined #openstack-infra04:30
*** thorst_ has quit IRC04:33
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: customize abort/failure/success/fixed email notification  https://review.openstack.org/35514504:34
*** fguillot_ has quit IRC04:34
*** yamamoto has joined #openstack-infra04:35
*** elo has quit IRC04:35
*** elo has joined #openstack-infra04:36
*** yamamoto has quit IRC04:38
*** sdake has quit IRC04:39
*** yamamoto has joined #openstack-infra04:39
*** elo has quit IRC04:41
*** elo has joined #openstack-infra04:41
*** eric_lopez has joined #openstack-infra04:45
*** yamamoto has quit IRC04:45
*** elo has quit IRC04:46
*** IlyaG has quit IRC04:48
*** eric_lopez has quit IRC04:49
*** itisha has quit IRC04:50
*** adrian_otto has quit IRC04:51
*** elo has joined #openstack-infra04:52
*** sarob has joined #openstack-infra04:53
*** sdake has joined #openstack-infra04:55
*** pt_15 has quit IRC04:56
*** sdake has quit IRC04:56
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: customize abort/failure/success/fixed email notification  https://review.openstack.org/35514504:57
*** gouthamr has quit IRC04:59
*** adrian_otto has joined #openstack-infra04:59
*** dkehn has quit IRC05:03
*** dkehn_ has quit IRC05:03
*** baoli has joined #openstack-infra05:03
*** elo has quit IRC05:04
*** elo has joined #openstack-infra05:04
*** baoli has quit IRC05:08
*** adrian_otto has quit IRC05:09
*** eric_lopez has joined #openstack-infra05:09
*** elo has quit IRC05:09
*** adrian_otto has joined #openstack-infra05:09
*** dkehn has joined #openstack-infra05:10
*** apetrich has joined #openstack-infra05:11
*** armax has quit IRC05:12
*** sdake has joined #openstack-infra05:13
*** adrian_otto has quit IRC05:13
*** eric_lopez has quit IRC05:13
*** eric_lopez has joined #openstack-infra05:14
*** adrian_otto has joined #openstack-infra05:14
*** dkehn_ has joined #openstack-infra05:15
*** sarob has quit IRC05:17
*** sarob has joined #openstack-infra05:17
*** amitgandhinz has joined #openstack-infra05:18
*** eric_lopez has quit IRC05:21
*** amitgandhinz has quit IRC05:22
*** markvoelker has joined #openstack-infra05:24
*** raunak has quit IRC05:25
*** IlyaG has joined #openstack-infra05:25
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: customize abort/failure/success/fixed email notification  https://review.openstack.org/35514505:27
*** elo has joined #openstack-infra05:28
*** _nadya_ has joined #openstack-infra05:28
*** markvoelker has quit IRC05:28
*** rbuzatu has joined #openstack-infra05:29
*** thorst_ has joined #openstack-infra05:31
*** dimtruck is now known as zz_dimtruck05:31
*** yamamoto has joined #openstack-infra05:33
*** thorst_ has quit IRC05:38
*** tonytan4ever has quit IRC05:39
*** sarob has quit IRC05:44
*** sdake has quit IRC05:46
*** r-mibu has quit IRC05:46
*** r-mibu has joined #openstack-infra05:46
*** adrian_otto has quit IRC05:47
*** adrian_otto has joined #openstack-infra05:48
*** adrian_otto has quit IRC05:50
*** yamamoto has quit IRC05:51
*** adrian_otto has joined #openstack-infra05:52
*** yamamoto has joined #openstack-infra05:52
*** tkelsey has joined #openstack-infra05:55
*** adrian_otto has quit IRC05:56
*** psachin has joined #openstack-infra05:56
*** elo has quit IRC05:57
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: customize abort/failure/success/fixed email notification  https://review.openstack.org/35514506:00
*** tkelsey has quit IRC06:00
*** tonytan4ever has joined #openstack-infra06:01
*** psachin has quit IRC06:02
*** elo has joined #openstack-infra06:02
*** yamamoto has quit IRC06:08
*** kzaitsev_mb has quit IRC06:12
*** adrian_otto has joined #openstack-infra06:14
*** yamamoto has joined #openstack-infra06:14
*** senk_ has joined #openstack-infra06:16
*** elo has quit IRC06:18
*** ihrachys has joined #openstack-infra06:18
*** adrian_otto has quit IRC06:18
*** amitgandhinz has joined #openstack-infra06:18
*** adrian_otto has joined #openstack-infra06:19
*** amitgandhinz has quit IRC06:23
*** sdake has joined #openstack-infra06:29
*** sdake has quit IRC06:29
*** elo has joined #openstack-infra06:29
*** adrian_otto has quit IRC06:33
*** elo has quit IRC06:36
*** thorst_ has joined #openstack-infra06:36
*** zhurong has joined #openstack-infra06:41
*** elo has joined #openstack-infra06:41
*** thorst_ has quit IRC06:44
*** rwsu has quit IRC06:45
*** eric_lopez has joined #openstack-infra06:45
openstackgerritLuz Cazares proposed openstack/os-testr: Save subunit results file when using --no-discover  https://review.openstack.org/33918906:47
*** elo has quit IRC06:47
*** eric_lopez has quit IRC06:51
*** eric_lopez has joined #openstack-infra06:51
*** yamamoto has quit IRC06:52
*** ihrachys has quit IRC06:52
*** Hal has quit IRC06:53
*** Hal has joined #openstack-infra06:53
*** eric_lopez has quit IRC06:57
*** rbuzatu has quit IRC06:59
*** rbuzatu has joined #openstack-infra07:01
*** yamahata has joined #openstack-infra07:06
*** eranrom_ has joined #openstack-infra07:08
*** kzaitsev_mb has joined #openstack-infra07:10
*** eranrom has quit IRC07:10
*** elo has joined #openstack-infra07:10
*** _nadya_ has quit IRC07:13
*** hparekh has quit IRC07:14
*** baoli has joined #openstack-infra07:16
*** elo has quit IRC07:16
*** amitgandhinz has joined #openstack-infra07:19
*** baoli has quit IRC07:20
*** hieulq_ has joined #openstack-infra07:21
*** elo has joined #openstack-infra07:22
*** amitgandhinz has quit IRC07:24
*** rbuzatu has quit IRC07:26
*** ihrachys has joined #openstack-infra07:26
*** hparekh has joined #openstack-infra07:27
*** hparekh_ has joined #openstack-infra07:27
*** rbuzatu has joined #openstack-infra07:27
*** hparekh_ has quit IRC07:27
*** dkehn has quit IRC07:27
*** dkehn_ has quit IRC07:28
*** rbuzatu has quit IRC07:31
*** rbuzatu has joined #openstack-infra07:31
*** tonytan4ever has quit IRC07:32
*** dkehn has joined #openstack-infra07:35
*** IlyaG has quit IRC07:37
*** ihrachys has quit IRC07:41
*** thorst_ has joined #openstack-infra07:42
*** kus has joined #openstack-infra07:43
*** kzaitsev_mb has quit IRC07:45
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Shorten the job name of Kolla project  https://review.openstack.org/35516707:45
*** dkehn_ has joined #openstack-infra07:47
*** sdake has joined #openstack-infra07:49
*** thorst_ has quit IRC07:49
*** yamamoto has joined #openstack-infra07:52
*** rbuzatu has quit IRC07:54
*** yamamoto has quit IRC07:58
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** tqtran has joined #openstack-infra08:09
*** tqtran has quit IRC08:14
*** shashank_hegde has joined #openstack-infra08:19
*** amitgandhinz has joined #openstack-infra08:20
*** mikelk has joined #openstack-infra08:24
*** amitgandhinz has quit IRC08:25
*** sdake has quit IRC08:30
*** mikelk has quit IRC08:31
*** mikelk has joined #openstack-infra08:31
*** mikelk has quit IRC08:31
*** mikelk has joined #openstack-infra08:32
*** tonytan4ever has joined #openstack-infra08:32
*** tonytan4ever has quit IRC08:37
*** ihrachys has joined #openstack-infra08:42
*** hieulq_ has quit IRC08:44
*** thorst_ has joined #openstack-infra08:46
*** pcaruana has joined #openstack-infra08:48
*** thorst_ has quit IRC08:54
*** hieulq_ has joined #openstack-infra08:57
*** senk_ has quit IRC08:58
*** shashank_hegde has quit IRC08:58
*** hieulq_ has quit IRC09:05
*** _nadya_ has joined #openstack-infra09:05
*** mikelk has quit IRC09:06
*** cody-somerville has quit IRC09:08
*** ihrachys has quit IRC09:19
*** sdake has joined #openstack-infra09:20
*** amitgandhinz has joined #openstack-infra09:21
*** zhurong has quit IRC09:24
*** raunak has joined #openstack-infra09:25
*** amitgandhinz has quit IRC09:25
*** markvoelker has joined #openstack-infra09:27
*** raunak_ has joined #openstack-infra09:29
*** raunak has quit IRC09:30
*** raunak_ is now known as raunak09:30
*** markvoelker has quit IRC09:32
*** sdake has quit IRC09:38
openstackgerritguo yunxian proposed openstack/os-testr: Delete openstack/common in flake8 exclude list  https://review.openstack.org/35517509:42
*** senk_ has joined #openstack-infra09:42
*** Hal has quit IRC09:43
*** raunak has quit IRC09:50
*** thorst_ has joined #openstack-infra09:51
*** kus has quit IRC09:55
*** thorst_ has quit IRC09:58
*** yamahata has quit IRC09:59
*** senk_ has quit IRC10:21
*** amitgandhinz has joined #openstack-infra10:22
*** amitgandhinz has quit IRC10:26
*** markvoelker has joined #openstack-infra10:28
*** markvoelker has quit IRC10:32
*** tonytan4ever has joined #openstack-infra10:34
*** ayoung has quit IRC10:37
*** tonytan4ever has quit IRC10:38
*** ayoung has joined #openstack-infra10:49
*** thorst_ has joined #openstack-infra10:56
*** tkelsey has joined #openstack-infra10:59
*** tkelsey has quit IRC11:03
*** thorst_ has quit IRC11:04
*** _nadya_ has quit IRC11:11
*** yamamoto has joined #openstack-infra11:12
*** yamamoto has quit IRC11:17
*** yamamoto has joined #openstack-infra11:20
*** amitgandhinz has joined #openstack-infra11:22
*** Mmike has quit IRC11:23
*** Mmike has joined #openstack-infra11:25
*** amitgandhinz has quit IRC11:27
*** markvoelker has joined #openstack-infra11:29
*** markvoelker has quit IRC11:33
*** Julien-zte has joined #openstack-infra11:34
*** yamamoto has quit IRC11:35
*** kzaitsev_mb has joined #openstack-infra11:44
fungiEmilienM: yes, a recent gerrit patch (in yesterday's upgrade) that makes it a less attractive target for spammers slightly altered the hyperlink format in comments, so our custom javascript to build that test results table needs some minor, unforeseen tweaking now11:47
fungihopefully someone will find time to whip that up over the next day or two11:47
mordredfungi: I believe in the javascript people11:47
mordredthey're also not as scary as crab people11:48
funginot to mention the mole people11:48
*** kzaitsev_mb has quit IRC11:48
mordredugh. mole people11:49
mordredfungi: ansible should now be fixed-fixed on puppetmaster11:51
*** senk_ has joined #openstack-infra11:52
mordredfungi: I have forced an inventory cache refresh by hand and verify that all of the hosts are actually there11:52
*** zhurong has joined #openstack-infra11:54
fungimordred: thanks!12:00
*** senk_ has quit IRC12:00
*** thorst_ has joined #openstack-infra12:02
openstackgerritMonty Taylor proposed openstack-infra/system-config: Stop expanding hostvars in ansible inventory  https://review.openstack.org/35518212:02
mordredfungi: review.o.o is down12:09
*** thorst_ has quit IRC12:09
mordredcom.google.inject.ProvisionException: Unable to provision, see the following errors:12:12
mordred1) No index versions ready; run Reindex12:12
mordredinfra-root: if anyone else happens to be up/around ... ^^12:12
*** yamamoto has joined #openstack-infra12:13
mordred#status alert gerrit is having issues ... it is being working, no ETA at the moment12:20
openstackstatusmordred: sending alert12:20
*** links has quit IRC12:20
-openstackstatus- NOTICE: gerrit is having issues ... it is being working, no ETA at the moment12:23
*** ChanServ changes topic to "gerrit is having issues ... it is being working, no ETA at the moment"12:23
*** amitgandhinz has joined #openstack-infra12:23
mordredfungi: I am running the reindex, as the gerrit start error message seems to indicate it does not want to start without one12:25
*** zhurong has quit IRC12:25
*** gongysh has joined #openstack-infra12:25
openstackstatusmordred: finished sending alert12:26
*** zhurong has joined #openstack-infra12:26
mordredReindexing changes: projects: 55% (640/1149), 48% (171808/352375) (|)12:28
*** amitgandhinz has quit IRC12:28
*** markvoelker has joined #openstack-infra12:29
*** markvoelker has quit IRC12:34
mordredReindexing changes: projects: 76% (879/1149), 68% (242640/352375) (/)12:34
*** sdague has joined #openstack-infra12:46
mordredReindexing changes: projects: 99% (1141/1149), 89% (316208/352375) (\)12:47
*** mriedem1 has joined #openstack-infra13:01
*** mriedem has quit IRC13:04
*** thorst_ has joined #openstack-infra13:05
*** ihrachys has joined #openstack-infra13:10
*** mriedem1 is now known as mriedem13:11
*** thorst_ has quit IRC13:13
*** senk_ has joined #openstack-infra13:13
*** gongysh has quit IRC13:21
mordredReindexing changes: projects: 99% (1147/1149), 97% (342222/352375) (|)t13:22
*** ayoung has quit IRC13:23
*** amitgandhinz has joined #openstack-infra13:24
*** Na3iL has joined #openstack-infra13:25
*** thorst_ has joined #openstack-infra13:28
*** amitgandhinz has quit IRC13:28
*** markvoelker has joined #openstack-infra13:30
*** rossella_s has joined #openstack-infra13:31
mordredHOLY CRAP the last 3% really sucks13:32
*** markvoelker has quit IRC13:35
*** baoli has joined #openstack-infra13:35
*** baoli has quit IRC13:37
*** baoli has joined #openstack-infra13:38
*** baoli has quit IRC13:39
*** zz_dimtruck is now known as dimtruck13:40
*** ihrachys has quit IRC13:40
*** kzaitsev_mb has joined #openstack-infra13:45
*** hockeynut has joined #openstack-infra13:45
*** hockeynut has quit IRC13:47
*** kzaitsev_mb has quit IRC13:50
*** senk_ has quit IRC13:55
*** sdake has joined #openstack-infra13:55
mordredReindexing changes: projects: 99% (1148/1149), 97% (345148/352375) (|)13:56
*** TRUMP-PENCE has quit IRC13:57
*** hieulq_ has joined #openstack-infra13:59
mordredwe're at 98!13:59
mordredReindexing changes: projects: 99% (1148/1149), 98% (345338/352375)13:59
*** TRUMP-PENCE has joined #openstack-infra13:59
*** TRUMP-PENCE has quit IRC13:59
*** TRUMP-PENCE has joined #openstack-infra14:01
*** TRUMP-PENCE has quit IRC14:01
*** TRUMP-PENCE has joined #openstack-infra14:06
*** TRUMP-PENCE has quit IRC14:06
*** TRUMP-PENCE has joined #openstack-infra14:08
*** TRUMP-PENCE has quit IRC14:08
*** TRUMP-PENCE has joined #openstack-infra14:10
*** TRUMP-PENCE has quit IRC14:10
*** chlong|mtg has joined #openstack-infra14:11
*** tqtran has joined #openstack-infra14:11
*** zhurong has quit IRC14:12
*** TRUMP-PENCE has joined #openstack-infra14:14
*** tqtran has quit IRC14:16
*** rbuzatu has joined #openstack-infra14:17
*** thorst_ has quit IRC14:20
*** thorst_ has joined #openstack-infra14:21
*** rbuzatu has quit IRC14:22
*** electrofelix has joined #openstack-infra14:24
*** amitgandhinz has joined #openstack-infra14:25
*** hockeynut has joined #openstack-infra14:26
*** rbuzatu has joined #openstack-infra14:27
*** hockeynut has quit IRC14:28
*** sdake has quit IRC14:28
*** thorst_ has quit IRC14:29
*** amitgandhinz has quit IRC14:29
*** markvoelker has joined #openstack-infra14:31
*** rossella_s has quit IRC14:34
*** markvoelker has quit IRC14:35
*** arif-ali has quit IRC14:36
*** rcernin has joined #openstack-infra14:39
*** hockeynut has joined #openstack-infra14:43
mordredReindexing changes: projects: 99% (1148/1149), 98% (348046/352375)14:43
mordredhave I mentioned this is my least favorite thing ever?14:44
*** _nadya_ has joined #openstack-infra14:44
*** hockeynut has quit IRC14:45
kragnizlooks grim14:45
*** kzaitsev_mb has joined #openstack-infra14:46
*** asselin_ has joined #openstack-infra14:47
*** sdague has quit IRC14:48
*** tphummel has joined #openstack-infra14:48
*** thorst_ has joined #openstack-infra14:48
*** thorst_ has quit IRC14:48
*** asselin has quit IRC14:50
*** adrian_otto has joined #openstack-infra14:50
*** kzaitsev_mb has quit IRC14:51
*** hieulq_ has quit IRC14:53
mordredReindexing changes: projects: 99% (1148/1149), 98% (348728/352375)14:55
jeblairo/14:57
fungimordred: oh, when you got puppet working again did that retrigger a restart and an offline reindex?14:57
fungithe version of gerrit we have installed now is supposed to support online reindexing14:58
mordredfungi: yah14:58
mordredfungi: well, it triggered a restart14:58
mordredwhen I first looked at things, there were no javas running14:58
*** adrian_otto has quit IRC14:59
*** rfolco has joined #openstack-infra14:59
fungiapparently the reason it spends so long on the last percent is that it is one thread per repo, and runs through them in alpha order, so takes a while before it gets to nova and then nova is running on one thread for a couple hours after that point14:59
fungii think zaro has submitted a patch upstream to switch that scheduler to descending order by size, which should more efficiently us ethe available threads15:00
mordred++15:00
*** fguillot_ has joined #openstack-infra15:00
fungis/size/change count/15:00
*** _nadya_ has quit IRC15:01
mordredso I'm not sure what action puppet triggered that caused gerrit to think a full offline reindex was required15:01
mordredbut the init script refused to start gerrit otherwise :(15:01
fungimordred: i probably didn't correctly fake the upgrade when i did it manually15:01
mordredpossibly so?15:01
fungii downloaded into the gerrit-wars directory and then copied that to review_site/bin/gerrit.war but i may have overlooked something15:02
mordredyah - I think there were things in the puppet logs about gerrit.war - but it didn't look like it installed a new version15:02
* mordred glares at puppet15:03
fungibut i ran the same init from the puppet exec after that, and then started the service from the initscript and it ran fine15:03
mordredooh. we're at 9915:03
mordredReindexing changes: projects: 99% (1148/1149), 99% (349180/35237515:03
fungii think zaro has proposed (or was going to propose) a change to set secondary_index=false in the openstack_project::review class so that we no longer fire an offline reindex on upgrade, but it sounds like it thought one was necessary for some reason?15:04
jeblairyeah, puppet 'upgraded' gerrit again15:04
*** ihrachys has joined #openstack-infra15:05
fungii thought it only did that if the filename it was going to download wasn't already present in the gerrit-wars fir15:05
fungidir15:05
mordredjeblair: well, while waiting for this to finish, I've updated the shade changes we need for ipv6 with tests and release notes - as well as wrote tests for the change I landed yesterday15:06
mordredso - yay for that I guess :)15:06
fungialso, wiki upgrade with newer captcha implementation is still not preventing all spammers. got three more overnight15:06
mordredblerg15:06
mordredwell, at least we've upgraded/modernized so iterating on new solutions should be easier15:07
jeblairfungi: it did some stuff related to the libraries (bouncy castle, mysql).... it also changed some ownership/perms of gerrit.war....15:07
jeblairAug 13 12:03:10 review puppet-user[48695]: (/Stage[main]/Gerrit/Exec[gerrit-initial-init]) Triggered 'refresh' from 2 events15:07
fungioh, i wonder if i downloaded it as the review user instead of root15:07
*** rbuzatu has quit IRC15:07
jeblairso two (2) things made it want to init.15:07
jeblairwe get to guess which15:08
jeblairwhich eventually led to Aug 13 12:08:48 review puppet-user[48695]: (/Stage[main]/Gerrit/Exec[gerrit-reindex]) Failed to call refresh: Command exceeded timeout15:08
jeblairwhich is the current sad15:08
*** hockeynut has joined #openstack-infra15:09
fungigot it. so it thought it needed to upgrade again, which triggered a reindex, which got killed by the timeout, which left it unable to start because of a partially complete index15:09
jeblairya, that's what it looks like to me15:09
fungiif it hadn't triggered the offline reindex exec then it would have just been a momentary blip for a gerrit restart i guess15:10
mordredI think definitely removing the auto-offline-reindex from puppet would be good15:11
mordredsince there are pretty much no cases in which we want puppet to be the one doing that15:11
jeblairyep15:11
*** annegentle has joined #openstack-infra15:11
fungiyeah, and also we're supposed to be able to reindex online since yesterday's upgrade15:11
mordred\o/15:12
fungiat least you caught this pretty quickly once puppet started working again15:12
fungisorry i wasn't around to help troubleshoot, i only just got back from breakfast with the inlaws15:13
*** arif-ali has joined #openstack-infra15:13
mordredno worries at all - it hasn't been terrible - just boring waiting on the reindex15:14
mordredbreakfast with the inlaws is, they tell me, important15:14
fungihopefully this will be the last one ever15:14
mordred+10015:14
fungii'll see if zaro already has that change proposed, or write one myself15:14
fungiand then i'll see if i can figure out the hideci.js patch we need for handling the new href tag format15:15
*** electrofelix has quit IRC15:15
*** armax has joined #openstack-infra15:20
*** tonytan4ever has joined #openstack-infra15:20
*** megm has quit IRC15:21
fungiyeah, didn't find a change for it yet in my gertty cache, so i've got one ready to push as soon as gerrit is running again15:23
*** annegentle has quit IRC15:23
mordredwoot15:24
mordredReindexing changes: projects: 99% (1148/1149), 99% (350462/352375)15:24
fungiworking on a hideci.js patch now15:24
*** amitgandhinz has joined #openstack-infra15:25
fungijavascript is a dark art15:26
* fungi prepares a gate seal15:26
jeblair1913 patches to index remain, 1913 patches to index; take one patch, index it down, 1912 patches to index remain!15:26
fungias the old saying goes, never cast a summoning if you haven't also memorized the corresponding binding15:26
fungier, that would have been better as "my grandmother always said...15:27
jeblairor even your eternally young great, great, great grandmother.15:28
*** rfolco has quit IRC15:28
fungiheh15:28
fungiseems like i haven't been up to dunwich to visit for _aeons_15:29
*** shashank_hegde has joined #openstack-infra15:29
*** gouthamr has joined #openstack-infra15:30
*** amitgandhinz has quit IRC15:30
mordreddone!15:30
mordredok15:30
mordredso - /etc/init.d/gerrit start isn't going to do anything evil, right?15:30
fungior i just use sudo service gerrit start15:31
fungibut yeah, 'tis fine15:31
*** hockeynut has quit IRC15:32
mordredI've done that - it says ok - but the web app seems unhappy15:32
*** markvoelker has joined #openstack-infra15:32
mordredshift-reload got it15:32
fungitakes at least a minute to start15:32
fungidon't panic yet15:32
openstackgerritMonty Taylor proposed openstack-infra/shade: Base auto_ip on interface_ip not public_v4  https://review.openstack.org/35504815:32
openstackgerritMonty Taylor proposed openstack-infra/shade: Add tests to show IP inference in missed conditions  https://review.openstack.org/35518315:32
fungiif you see the war running in the process list15:33
fungialso, gertty knows better than we do15:33
mordredyah. it's working15:33
fungias does gerritbot apparently15:33
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Stop triggering offline Gerrit reindexing  https://review.openstack.org/35518415:33
fungijeblair: mordred: ^ with fire15:33
*** markvoelker has quit IRC15:36
*** mriedem has quit IRC15:36
fungiwell, i _thought_ the hideci.js issue was due to rel="nofollow" now being in the href tags, but i'm not seeing an overly strict regex in use by the results parser to account for it15:37
mordredmight be a dom restructure due to GWT then15:41
fungioh, yep, comparing an already cached tab, the verified comments from the jenkins user were formatted differently than they are now15:43
*** megm has joined #openstack-infra15:43
fungibefore the job name was hyperlinked to the log. now it's the job name followed by a raw url and only the latter is hyperlinked15:44
fungiand the result lines have bullets by them now15:45
mordreddid they get rid of the ability to post url comments?15:46
jeblairif they have bullets, then our commentlink regex trick is failing15:47
fungiaha, that would 'splain it (and would also confuse hideci.js)15:47
jeblairwe post them as '* list items' and then the commentlink regex manipulates that into the nicer formatting15:47
*** kzaitsev_mb has joined #openstack-infra15:47
fungii guess that's only directly in the gerrit puppet module, not in openstack_project15:48
jeblairit's in modules/openstack_project/manifests/review.pp15:49
fungioh, in the manifest15:49
jeblair        match => '<li>([^ ]+) <a href=\"[^\"]+\" target=\"_blank\">([^<]+)</a> : ([^ ]+)([^<]*)</li>',15:49
jeblair        html  => '<li class=\"comment_test\"><span class=\"comment_test_name\"><a href=\"$2\">$1</a></span> <span class=\"comment_test_result\"><span class=\"result_$3\">$3</span>$4</span></li>',15:49
fungiright, for some reason i though we had put it directl yin the config template15:49
fungiso that's actually straightforward to fix i think15:50
jeblairyeah, prolly just need to add the nofollow bit to that first line15:50
mordred++15:50
jeblairi'll do it15:50
*** fguillot_ is now known as fguillot_afk15:50
fungiim about to push it, just writing the commit message15:51
jeblairok you're ahead of me :)15:51
*** fguillot_afk has quit IRC15:51
jeblairby a good 5 seconds15:51
*** shashank_hegde has quit IRC15:51
*** baoli has joined #openstack-infra15:51
*** ihrachys has quit IRC15:51
openstackgerritMonty Taylor proposed openstack-infra/shade: Base auto_ip on interface_ip not public_v4  https://review.openstack.org/35504815:52
*** kzaitsev_mb has quit IRC15:52
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Update Gerrit CI comment regex for rel="nofollow"  https://review.openstack.org/35518615:53
fungijeblair: mordred: there ^ complete with upstream change reference15:53
fungishould i carry rel="nofollow" into the results table too?15:54
fungithat might be wise to thwart spammers15:54
mordredyah - I think so15:54
fungiadding now15:54
mordredalso, are target and rel always in that order?15:54
jeblairheh, yeah, otherwise i reckon we would have created a (very subtle) loophole15:54
jeblairmordred: they appear to be on inspection15:55
mordredcool15:55
jeblairoh wait15:55
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Update Gerrit CI comment regex for rel="nofollow"  https://review.openstack.org/35518615:55
*** mriedem has joined #openstack-infra15:55
jeblairthat's not the order i see.... gimme a sec to make sure i'm looking right15:55
*** baoli has quit IRC15:56
fungithat was the order https://gerrit-review.googlesource.com/#/c/80320/4/gerrit-gwtexpui/src/main/java/com/google/gwtexpui/safehtml/client/SafeHtml.java showed15:56
fungibut maybe java can reorder them or something?15:56
*** armax has quit IRC15:57
pabelangermorning15:58
jeblairweird.  firebug has them as "<a rel="nofollow" target...."15:59
jeblairbut i don't know if that's firebug or gerrit15:59
pabelangermordred: wow, osic is using ipv6 I think15:59
fungipabelanger: they switched it on yesterday16:00
jeblairfungi: your order is in what's in our repo too: http://git.openstack.org/cgit/openstack-infra/gerrit/commit/?h=openstack/2.11.4&id=b50155fc803cd2b5124b37f8dbe470752152774a16:00
pabelangerfungi: right, but mordred said it might now work because of shade bug16:00
jeblairfungi: why don't we experiment on review-dev?16:00
pabelangerfungi: and had a revert up16:00
fungipatch on the way16:00
pabelangerchecking logs to see if things are happy now16:01
mordredpabelanger: so - using ipv6 is one part ...16:01
mordredpabelanger: the question is did nodepool also allocate an ipv4 FIP needlessly?16:01
jeblairfungi: 'because gerrit is not running on review-dev' would be the answer to that question.16:01
jeblairfungi: meh.  i vote we land your patch, and if it doesn't work, move it to my order.  :)16:01
mordredjeblair: ++16:01
*** dimtruck is now known as zz_dimtruck16:02
*** zz_dimtruck is now known as dimtruck16:02
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Test Gerrit CI comment regex for rel="nofollow"  https://review.openstack.org/35518816:03
pabelangermordred: http://paste.openstack.org/show/556611/16:03
fungiwell, there's the corresponding change for review-dev too16:03
pabelangerI do see 1 ipv4 address16:03
*** thorst_ has joined #openstack-infra16:03
pabelangerbut it is private16:03
fungii'm fine landing them in either order16:03
fungii can also hand-patch it in the gerrit config and do a quick restart if we're concerned16:04
mordredpabelanger: great16:04
mordredpabelanger: does osic have fips?16:04
fungiyeah, gerrit currently refuses to start on review-dev16:05
pabelangermordred: we don't use them any more. They allocated us ipv4 address a while back16:05
fungi"No index versions ready; run Reindex"16:06
fungiso we probably triggered a similar brokenness on review-dev16:06
*** yaume has joined #openstack-infra16:06
fungiand since zaro imported the nova repo into it to be able to test a sufficiently involved online reindex, it now runs longer than our timeout there too16:06
fungii'll start fixing it16:06
mordredpabelanger: awesome.16:06
*** thorst_ has quit IRC16:07
mordredpabelanger: so - I'm going to peek a little more to see why this worked16:07
mordred:)16:07
fungii've got an offline reindex running on review-dev now16:08
fungithat will probably take an hour or so to complete16:08
*** shashank_hegde has joined #openstack-infra16:09
fungi#status ok The Gerrit service on review.openstack.org is online again16:11
openstackstatusfungi: sending ok16:11
*** dimtruck is now known as zz_dimtruck16:12
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"16:13
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is online again16:13
pabelangerjeblair: do we need to update zuul_console for ipv6 support?16:13
openstackstatusfungi: finished sending ok16:16
pabelangertcp        0      0 0.0.0.0:19885           0.0.0.0:*               LISTEN16:17
pabelangerlook like we are only binding to ipv416:17
fungicould presumably just change that to ::1988516:20
fungii haven't looked into whether we're using socket.socket() for that or what16:20
pabelangerYa, if I under stand the code in zuul_console, it should trying both ipv4/ipv6.  But ipv4 seems to take priority16:21
fungifor a listening socket to bind to all available interfaces and address families, i don't think you need to try/order anything16:21
*** adrian_otto has joined #openstack-infra16:22
*** tonytan4ever has quit IRC16:22
fungimost linux distros' have been setting the kernel to dual-stack and support unified family sockets for years afaik16:22
pabelangerlooks like we are using socket.getaddrinfo()16:23
pabelangerhttps://docs.python.org/2/library/socket.html has an example, which I am trying16:23
fungiat least at a syscall level i think you have to set the socket option IPV6_V6ONLY on an AF_INET6 socket if you don't want it also listening on v4 addresses16:24
*** amitgandhinz has joined #openstack-infra16:26
jeblairpabelanger: oh wow, i love that the example is not for the thing you should actually do16:27
jeblair"The server side will listen to the first address family available (it should listen to both instead)."16:27
pabelangerya16:28
pabelanger>>> socket.getaddrinfo(None, 80,socket.AF_UNSPEC,socket.SOCK_STREAM,0,socket.AI_PASSIVE)16:28
pabelanger[(2, 1, 6, '', ('0.0.0.0', 80)), (10, 1, 6, '', ('::', 80, 0, 0))]16:28
pabelangerwe are only binding to ipv416:28
pabelangersince it is returned first16:28
*** yaume has quit IRC16:28
jeblairso if i'm following, we actually need to listen on *every* item returned16:29
jeblairso we'll need to listen on and store N sockets (where N=2)16:29
pabelangerLooks that way16:29
jeblairthiss will complicate the logic somewhat; i think we'll need 2 accept threads.16:30
jeblair(right now, the accept thread is the main thread).16:31
*** amitgandhinz has quit IRC16:31
*** fguillot_ has joined #openstack-infra16:32
jeblairanyway, it shouldn't be too bad.  i'd probably just have the socket creation spawn an accept thread for each socket it binds, then have the main thread wait for those threads to join.16:32
pabelangerAside of the lack of console, it does look like 2001:4800:1ae1:18:f816:3eff:fe55:8977 is working correctly16:32
fungiin other projects i've just done something along the lines of socket.socket(socket.AF_INET6, socket.SOCK_STREAM).bind(('::', port)) and it accepts on both its ipv4 and ipv6 addresses16:32
*** hieulq_ has joined #openstack-infra16:33
*** markvoelker has joined #openstack-infra16:33
jeblairoh that works?16:33
pabelangercloudnull: good news, it looks like osic-cloud1 is working correctly with ipv6. Change when live a few hours ago16:34
fungiyeah, as long as the kernel doesn't have forced separation for v4 and v6only sockets, which has been passe for ages16:34
fungiexcept on *bsds still i thinkl16:34
fungiyou can still set linux to act that way, but it's not typical in modern distros16:35
*** thorst_ has joined #openstack-infra16:37
*** markvoelker has quit IRC16:37
fungifor example, here's a mud engine i did it in: http://mudpy.org/gitweb?p=mudpy.git;a=blob;f=lib/mudpy/misc.py;h=ea18ffb;hb=HEAD#l45816:38
fungiin my configuration i set the parameter that ends up in the "host" variable to "::" and can connect on its ipv4 or ipv6 addresses, yet netstat -lnt only shows the port listening on ::16:39
zarolooks like i'm missing the fun16:39
jeblairpabelanger: i like fungi's idea :)16:40
mordredpabelanger: ok - so - there is another bug (which isn't breaking anytihng but which I'll fix now that I see it)16:40
fungijeblair: all the best server ideas come from muds after all16:40
mordredpabelanger: that bug has happily caused the ipv6 on osic to work when it should have broken16:40
mordredpabelanger: so we should still land my fix for that16:40
zarofungi: -1 for your change to stop triggering offline reindex https://review.openstack.org/#/c/35518416:40
jeblairfungi: yeah, if it works for a mud, it'll work for us.16:40
fungizaro: thanks, looking16:40
*** thorst_ has quit IRC16:41
pabelangerjeblair: fungi: let me see if I can get it working. Might need some hand holding on this16:41
mordred(for some reason shade is detecting the private v4 address on osic as a public address ... I have not yet figured out why)16:41
zarosorry, i was playing with review-dev to verify rel=nofollow issue yesterday but didn't restore it16:41
pabelangermordred: do we need to revert OSIC to ipv4 or can we keep moving forward?16:41
mordrednope16:41
mordredit's all great16:41
mordredit's a happy accident for now :)16:41
*** senk_ has joined #openstack-infra16:41
pabelangeryay16:41
jeblairpabelanger: cool, i'm going to go $saturday, but can check back in on things later.16:42
pabelangerjeblair: ack16:42
openstackgerritMonty Taylor proposed openstack-infra/system-config: Tell shade that OSIC doesn't need floating ips  https://review.openstack.org/35519116:42
mordredpabelanger: this will cut down on the number of API calls we do to accomplish things though16:42
*** yamamoto has quit IRC16:43
fungizaro: ooh, great catch. i totally forgot to check whether that variable was used for anything else16:43
pabelangermordred: great16:43
zarofungi: looks like review-dev is in same state as review now?  we can test fixes on now?16:43
fungizaro: yeah, i manually ran an offline reindex on review-dev and was able to start gerrit again after that16:44
fungibasically same thing happened there as happened on review.o.o16:44
openstackgerritMerged openstack-infra/system-config: Update Gerrit CI comment regex for rel="nofollow"  https://review.openstack.org/35518616:45
fungiafter that ^ is showing up on review.o.o i'll restart gerrit there again quickly16:46
zarofungi: are you working on https://review.openstack.org/#/c/355184  i can help there if needed16:47
*** shashank_hegde has quit IRC16:48
fungizaro: i haven't started revising it yet, so if you want to take it over i'll be happy to review!16:48
zarofungi: ok. new PS coming soon16:48
*** kzaitsev_mb has joined #openstack-infra16:48
fungii have some weekend stuff i need to get done not in here too16:48
fungibut want to make sure everything is settled properly first16:49
*** chlong|mtg has quit IRC16:52
*** kzaitsev_mb has quit IRC16:53
*** IlyaG has joined #openstack-infra16:53
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Add an offline reindex parameter  https://review.openstack.org/35519416:54
*** mlekkas has quit IRC16:54
*** ihrachys has joined #openstack-infra16:54
zarofungi: ^16:55
*** Na3iL has quit IRC16:56
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Add an offline reindex parameter  https://review.openstack.org/35519416:57
*** mriedem has quit IRC17:00
*** IlyaG has quit IRC17:00
*** psachin has joined #openstack-infra17:03
*** raunak has joined #openstack-infra17:07
*** adrian_otto has quit IRC17:07
openstackgerritPaul Belanger proposed openstack-infra/zuul: Simplify zuul_console port binding logic  https://review.openstack.org/35519717:09
*** rhallisey has joined #openstack-infra17:09
pabelangerfungi: jeblair: ^ I believe that should work for ipv4/ipv6. I tested the code locally and seems to work. Let me know what you think17:09
pabelangermordred: also, did we still need to restart nodepool.o.o or have you done that already to pickup latest shade release?17:10
*** IlyaG has joined #openstack-infra17:11
*** thorst_ has joined #openstack-infra17:12
*** tphummel has quit IRC17:14
*** thorst_ has quit IRC17:15
*** IlyaG has quit IRC17:15
*** thorst_ has joined #openstack-infra17:15
*** ihrachys has quit IRC17:16
mordredpabelanger: I have not restarted it17:16
*** tphummel has joined #openstack-infra17:16
*** hieulq_ has quit IRC17:17
mordredpabelanger: I'd like to land that system-config change though before we do - should cut down on the number of API calls we're making17:17
pabelangermordred: WFM17:18
zaromordred: turn off offline reindex https://review.openstack.org/#/c/355194/17:19
*** thorst_ has quit IRC17:20
fungirestarting gerrit on review.o.o again now for the commentlink patch17:21
mordredzaro: +A - thanks!17:21
*** sdake has joined #openstack-infra17:21
fungithat seems to have worked. ci somments are back to our expected format now and the results table has returned17:24
fungii'm disappearing to saturday some stuff17:24
fungiback later maybe17:24
mordredwoot!17:24
Krenairhow do you subscribe to things in storyboard?17:27
*** amitgandhinz has joined #openstack-infra17:27
Krenairif you make a comment are you automatically emailed about any further comments?17:28
*** amitgandhinz has quit IRC17:31
*** thorst_ has joined #openstack-infra17:31
*** zhenguo has quit IRC17:32
SotKKrenair: you'll have to subscribe to the story you commented on for that at the moment17:32
Krenairthat's what I'd like to do, but how?17:33
SotKThere is a star next to the story title, which when clicked will subscribe you to the story17:33
*** markvoelker has joined #openstack-infra17:33
Krenairaha, thanks17:33
*** thorst_ has quit IRC17:35
*** markvoelker has quit IRC17:38
*** e0ne has joined #openstack-infra17:39
*** tphummel has quit IRC17:39
*** bswartz has quit IRC17:50
*** raunak has quit IRC17:50
openstackgerritMerged openstack-infra/puppet-gerrit: Add an offline reindex parameter  https://review.openstack.org/35519417:54
*** rbuzatu has joined #openstack-infra17:54
*** senk_ has quit IRC17:55
*** senk_ has joined #openstack-infra17:56
*** sdake has quit IRC18:03
*** sdake has joined #openstack-infra18:05
*** yaume has joined #openstack-infra18:06
*** gouthamr has quit IRC18:07
*** tqtran has joined #openstack-infra18:13
*** rbuzatu has quit IRC18:16
*** tqtran has quit IRC18:17
*** senk_ has quit IRC18:17
*** yamamoto has joined #openstack-infra18:17
*** yamamoto has quit IRC18:22
*** tonytan4ever has joined #openstack-infra18:23
*** rcernin has quit IRC18:27
*** hockeynut has joined #openstack-infra18:27
*** tonytan4ever has quit IRC18:27
*** rbuzatu has joined #openstack-infra18:27
*** amitgandhinz has joined #openstack-infra18:28
*** kzaitsev_mb has joined #openstack-infra18:29
*** fguillot_ is now known as fguillot_afk18:31
*** rbuzatu has quit IRC18:32
*** amitgandhinz has quit IRC18:32
*** itisha has joined #openstack-infra18:34
*** tphummel has joined #openstack-infra18:34
*** markvoelker has joined #openstack-infra18:34
*** pt_15 has joined #openstack-infra18:34
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Make the kolla-kubernetes jobs non-voting and experimental  https://review.openstack.org/35519918:36
*** sdake has quit IRC18:38
*** markvoelker has quit IRC18:38
*** hockeynut has quit IRC18:40
*** sdake has joined #openstack-infra18:54
*** e0ne has quit IRC18:55
*** e0ne has joined #openstack-infra18:55
*** fguillot_ has joined #openstack-infra18:56
*** fguillot_ is now known as fguillot_afk18:58
*** fguillot_afk has quit IRC18:58
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Make the kolla-kubernetes jobs non-voting and experimental  https://review.openstack.org/35519919:03
*** timello has quit IRC19:04
*** timello has joined #openstack-infra19:06
*** fifieldt has quit IRC19:07
*** timello has quit IRC19:07
*** fguillot_ has joined #openstack-infra19:09
*** fguillot_ is now known as fguillot_afk19:10
*** fguillot_afk has quit IRC19:11
*** yaume has quit IRC19:11
*** timello has joined #openstack-infra19:12
*** sdake has quit IRC19:13
*** tonytan4ever has joined #openstack-infra19:17
*** fifieldt has joined #openstack-infra19:18
*** tesseract- has joined #openstack-infra19:20
*** fguillot_ has joined #openstack-infra19:20
*** tesseract- has quit IRC19:21
*** eric_lopez has joined #openstack-infra19:27
*** psachin has quit IRC19:27
*** amitgandhinz has joined #openstack-infra19:29
*** elo has quit IRC19:29
*** amitgandhinz has quit IRC19:33
*** markvoelker has joined #openstack-infra19:35
*** markvoelker has quit IRC19:39
openstackgerritAlex Schultz proposed openstack-infra/project-config: Update puppet integration jobs to use bindep  https://review.openstack.org/35520319:50
openstackgerritRyan Hallisey proposed openstack-infra/project-config: Make the kolla-kubernetes jobs non-voting and experimental  https://review.openstack.org/35519919:54
*** rfolco has joined #openstack-infra19:57
*** eric_lopez has quit IRC19:59
*** elo has joined #openstack-infra19:59
*** e0ne has quit IRC20:01
*** rfolco has quit IRC20:08
*** pfallenop has quit IRC20:11
*** shashank_hegde has joined #openstack-infra20:11
*** pfallenop has joined #openstack-infra20:17
*** thorst_ has joined #openstack-infra20:20
*** tonytan4ever has quit IRC20:28
*** amitgandhinz has joined #openstack-infra20:29
*** kzaitsev_mb has quit IRC20:31
*** pfallenop has quit IRC20:31
*** pfallenop has joined #openstack-infra20:32
*** Jeffrey4l_ has joined #openstack-infra20:34
*** amitgandhinz has quit IRC20:34
*** Jeffrey4l has quit IRC20:35
*** markvoelker has joined #openstack-infra20:36
*** raunak has joined #openstack-infra20:36
*** markvoelker has quit IRC20:40
*** thorst_ has quit IRC20:43
*** pfallenop has quit IRC20:49
*** gibi has quit IRC20:56
*** rhallisey has quit IRC20:58
*** rhallisey has joined #openstack-infra20:58
*** rhallisey_ has joined #openstack-infra21:02
*** rhallisey_ has quit IRC21:03
*** pfallenop has joined #openstack-infra21:05
openstackgerritJoost van der Griendt proposed openstack-infra/jenkins-job-builder: adding extra properties to sonar publisher  https://review.openstack.org/35520621:06
*** thorst_ has joined #openstack-infra21:16
*** thorst_ has quit IRC21:16
openstackgerritJoost van der Griendt proposed openstack-infra/jenkins-job-builder: add credentials-id to svn_tags plugin  https://review.openstack.org/35520721:24
*** tonytan4ever has joined #openstack-infra21:29
*** amitgandhinz has joined #openstack-infra21:30
*** mriedem has joined #openstack-infra21:32
*** thorst_ has joined #openstack-infra21:34
*** thorst_ has quit IRC21:34
*** tonytan4ever has quit IRC21:34
*** thorst_ has joined #openstack-infra21:34
*** amitgandhinz has quit IRC21:34
*** markvoelker has joined #openstack-infra21:36
*** yamahata has joined #openstack-infra21:37
*** thorst_ has quit IRC21:39
*** markvoelker has quit IRC21:41
openstackgerritJoost van der Griendt proposed openstack-infra/jenkins-job-builder: Adding support for Hidden parameter plugin  https://review.openstack.org/35520921:42
*** sdake has joined #openstack-infra21:45
cloudnullpabelanger: woot!!22:00
*** tqtran has joined #openstack-infra22:14
openstackgerritJoost van der Griendt proposed openstack-infra/jenkins-job-builder: Add support for stash-pullrequest-builder plugin Although the application has now been renamed/merge to BitBucket, it is still sensible to keep the Stash name for now. As there are already plugins named BitBucket, which are purely targeting the cloud solu  https://review.openstack.org/35521122:15
*** tqtran has quit IRC22:19
*** rbuzatu has joined #openstack-infra22:24
*** rhallisey_ has joined #openstack-infra22:24
*** rhallisey_ has quit IRC22:24
*** sdake has quit IRC22:24
*** rbuzatu has quit IRC22:24
*** rbuzatu has joined #openstack-infra22:24
*** amitgandhinz has joined #openstack-infra22:31
*** amitgandhinz has quit IRC22:35
*** markvoelker has joined #openstack-infra22:37
*** markvoelker has quit IRC22:41
*** pt_15 has quit IRC22:42
*** apetrich has quit IRC22:42
*** pt_15 has joined #openstack-infra22:44
*** tonytan4ever has joined #openstack-infra22:46
*** gouthamr has joined #openstack-infra22:46
openstackgerritMerged openstack-infra/grafyaml: Remove discover from test-requirements  https://review.openstack.org/34579822:46
*** tonytan4ever has quit IRC22:50
*** harlowja has quit IRC22:50
*** rbuzatu has quit IRC22:55
openstackgerritMonty Taylor proposed openstack-infra/jenkins-job-builder: Add support for stash-pullrequest-builder plugin  https://review.openstack.org/35521122:57
cloudnullidk if more infra cores are about but I'd love to bump the quota on the osic usage https://review.openstack.org/#/c/35490022:58
*** superdan is now known as dansmith22:58
*** rbuzatu has joined #openstack-infra22:59
cloudnullthe tenant quota is all ready to go, to up things to 256 but I'd like to guage the environment in steps if at all possible23:00
clarkbcloudnull: does that depend on ipv6 working? I have been out ao not sure what the bigger picture is now23:01
cloudnullit does23:01
cloudnullin checking the cloud usage it seems like openstackci is running on the ipv6 instances23:02
cloudnullhttps://snag.gy/Ccbtzj.jpg23:02
openstackgerritMerged openstack-infra/system-config: Stop expanding hostvars in ansible inventory  https://review.openstack.org/35518223:03
clarkboh we did set ipv6 preferred in there cool23:03
pleia2clarkb: welcome to this morning :) pabelanger et al got all the wheels churning with ipv623:03
clarkbI assume this means we fixed dns resolvers \o/23:03
clarkbneat23:04
* clarkb approves23:04
openstackgerritMerged openstack-infra/system-config: Tell shade that OSIC doesn't need floating ips  https://review.openstack.org/35519123:04
cloudnullonce that runs for a bit, maybe sometime mid next week, I'd like to push it up even further .23:05
clarkbthat is great to hear. thank you for getting this going23:05
cloudnullthanks for working with me on it :)23:05
*** rbuzatu has quit IRC23:05
cloudnullsorry it took so long23:05
pabelangerclarkb: actually, we haven't updated DNS yet23:05
clarkbpleia2: mirning? you must be somewhere west of the west coast :)23:06
clarkbpabelanger: oh?23:06
clarkbpabelanger: does that mean osic cant resolve stuff or is it nating out to google from the private ip?23:06
clarkbI am guessing it NATs and neutron magically makes this work for outbound stuff?23:06
pabelangerclarkb: ya, we have ipv6-ipv4 right now23:06
pabelangerthat is what I assue23:06
cloudnullyes.23:07
pabelangerhowever, we should fix dns for sure23:07
cloudnullwe have v4 plugged into a neutron router.23:07
clarkbya but less urgent than I thought. cool23:07
cloudnullso neutron magic should make v4 dns happy23:07
pleia2clarkb: I meant, it happened this morning23:08
clarkbpleia2: ah I see23:08
cloudnullif we start seeing gate jobs die in a fire due to that we can make some adjustments on our side to bypass the neutron router.23:08
clarkbI have been pretty properly checked out. turns out traveling with two 14 month olds is exhausting23:08
pleia2I bet :\23:09
*** yamamoto has joined #openstack-infra23:09
clarkbcloudnull: the only differnce with that compared to the old fip aituation is nat on a shared IP right? so more constrained NAT resources?23:09
mordredclarkb: I don't follow your last sentence?23:15
cloudnullits nat on a shared IP, but no need to grab a FIP or any of that tomfoolery.23:16
mordredyay!23:16
mordredgrabbing FIPs is the worst tomfoolery23:16
cloudnullrouter is has a gate way of our public V4 network, with the v4 subnet as an interface.23:16
clarkbmordred: nat is contrained by port availability iirc23:16
mordredcloudnull: although I did just land a config patch to prevent us from hitting your FIP endpoint a billion times just to determine that we don't need any :)23:17
clarkbmordred: when you 1:1 nat thats not an issue23:17
clarkbwhen you many to one nat it can be23:17
mordredah - gotcha. nod23:17
cloudnullclarkb: ++23:17
cloudnullwe're running on the router for now, but if it becomes an issue we'll take care of it on the F523:17
mordredcloudnull: btw - the new FIP workaround patch is now live in shade ... so when we restart nodepool next we should see an improvement in bluebox23:17
cloudnull++ my API will be a lot happier,23:18
mordredgah23:18
* cloudnull hopes23:18
mordredclarkb: ^^ that fip patch was for you23:18
mordredcloudnull: yes - I've been looking at a bunch of API call sequences to osic to try to make sure we're not hitting a ton of needless stuff23:18
mordredalso - for reasons I don't understand yet - shade sees the private ip I get from osic as public23:19
mordredonce I figure out why, I'll let you know23:19
cloudnullif its something i need to change let me know.23:19
mordred(it's not a problem for us -just a weird that demands more investigation)23:19
cloudnull^ those are the types of problems I spin on for the longest :)23:20
mordredyah. "this isn't breaking anything, but it's wrong, so god knows what it's going to break at 2am in 6 months"23:20
clarkbits only not breaking us due to the iov6 preferred flag23:21
openstackgerritMerged openstack-infra/project-config: Increase the quota in the OSIC cloud1  https://review.openstack.org/35490023:21
*** yamamoto has quit IRC23:21
cloudnullhahaha! the 2am breakage w/out context is my favorite :D23:21
cloudnullwoot!23:22
mordredclarkb: no, there's another part that _should_ be breaking us even with that flag23:22
mordredclarkb: but whatever if causing the "private-is-public" thing is accidentally causing us not to be broken23:22
mordredwe've landed a patch to fix the breakage that's hidden by the other bug23:23
mordredbut I think fixing the other bug too is a thing we should do23:23
mordredclarkb: when do you back to portlands?23:24
clarkbyesterday23:25
clarkbbut was completely dead after overnight flight with kids23:25
SpamapSyeah, that's definitely a deadly activity23:25
clarkbturns out they sleep but you dont23:25
mordredoy23:25
SpamapSmore ambien next time23:25
*** raunak has quit IRC23:30
*** amitgandhinz has joined #openstack-infra23:32
*** amitgandhinz has quit IRC23:36
*** thorst_ has joined #openstack-infra23:36
*** markvoelker has joined #openstack-infra23:38
cloudnullclarkb: do we know when the limit will be raised for OSIC "test nodes"? is that something that just needs to be pushed out which will get picked up in the next iteration?23:39
*** markvoelker has quit IRC23:42
*** chem|off has quit IRC23:46
*** sdake has joined #openstack-infra23:51
prometheanfirecloudnull: hi :D23:52
prometheanfirecloudnull: did you break osic?23:53
cloudnullno.23:53
cloudnullwhy?23:53
cloudnulleverything looks greem23:55
cloudnull*green23:55
clarkbcloudnull: goes out on the next puppet run23:55
prometheanfirecloudnull: just making fun of you23:55
cloudnullcool, no worries.23:55
cloudnullprometheanfire: ah.23:55
* cloudnull was in a panic for a moment23:55
cloudnull:P23:55
*** karthik__ has joined #openstack-infra23:56
pabelangerkeeps refreshing grafana23:57
pabelangerhttp://grafana.openstack.org/dashboard/db/nodepool-osic23:57
cloudnullf523:57
cloudnullf523:57
cloudnullf523:57
cloudnull:)23:57
sbezverkevening folks, any infra cores around?23:57
sbezverkwe have a trouble with kolla kuberntes gates, some help would be nice.23:58
sbezverkhttps://review.openstack.org/35519923:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!