Monday, 2017-01-02

*** sdake_ is now known as sdake00:21
*** ianychoi has joined #openstack-infra00:30
*** winggundamth has quit IRC00:31
*** baoli has quit IRC00:34
*** salv-orlando has joined #openstack-infra00:43
*** markvoelker has joined #openstack-infra00:46
*** rbuzatu has joined #openstack-infra00:48
*** salv-orlando has quit IRC00:48
*** markvoelker has quit IRC00:51
*** baoli has joined #openstack-infra01:03
*** baoli has quit IRC01:04
*** rbuzatu has quit IRC01:14
*** rbuzatu has joined #openstack-infra01:15
*** baoli has joined #openstack-infra01:16
*** baoli has quit IRC01:16
*** askb has quit IRC01:27
*** hongbin has quit IRC01:42
*** salv-orlando has joined #openstack-infra01:44
*** markvoelker has joined #openstack-infra01:47
*** salv-orlando has quit IRC01:48
*** benj_ has quit IRC01:50
*** baoli has joined #openstack-infra01:51
*** markvoelker has quit IRC01:51
*** baoli has quit IRC01:51
*** sdake_ has joined #openstack-infra01:52
*** benj_ has joined #openstack-infra01:52
*** sdake has quit IRC01:54
*** benj_ has quit IRC02:02
*** baoli has joined #openstack-infra02:04
*** benj_ has joined #openstack-infra02:04
*** sdake_ has quit IRC02:08
*** jeckersb_gone is now known as jeckersb02:10
*** baoli has quit IRC02:12
*** jeckersb is now known as jeckersb_gone02:12
*** baoli has joined #openstack-infra02:13
*** baoli has quit IRC02:13
*** hongbin has joined #openstack-infra02:17
*** benj_ has quit IRC02:17
*** sdake has joined #openstack-infra02:18
*** benj_ has joined #openstack-infra02:19
*** wolverineav has joined #openstack-infra02:19
*** wolverineav has quit IRC02:24
*** benj_ has quit IRC02:30
*** david-lyle has joined #openstack-infra02:35
*** benj_ has joined #openstack-infra02:35
*** david-lyle has quit IRC02:39
*** Julien-zte has joined #openstack-infra02:44
*** salv-orlando has joined #openstack-infra02:44
*** halprin has left #openstack-infra02:47
*** markvoelker has joined #openstack-infra02:48
*** salv-orlando has quit IRC02:49
*** markvoelker has quit IRC02:52
*** baoli has joined #openstack-infra02:58
*** baoli has quit IRC02:58
*** Jeffrey4l__ has joined #openstack-infra03:05
*** Jeffrey4l_ has quit IRC03:09
*** hongbin has quit IRC03:11
*** ramishra has joined #openstack-infra03:13
*** rbuzatu has quit IRC03:28
*** links has joined #openstack-infra03:36
*** vikrant has joined #openstack-infra03:38
*** baoli has joined #openstack-infra03:40
*** salv-orlando has joined #openstack-infra03:45
*** markvoelker has joined #openstack-infra03:48
*** psachin has joined #openstack-infra03:49
*** salv-orlando has quit IRC03:50
*** markvoelker has quit IRC03:53
*** sdake has quit IRC04:07
*** zz_dimtruck is now known as dimtruck04:09
*** hongbin has joined #openstack-infra04:18
*** dimtruck is now known as zz_dimtruck04:28
*** zz_dimtruck is now known as dimtruck04:28
*** rbuzatu has joined #openstack-infra04:29
timrc_Happy New Year, everyone!04:31
*** sdake has joined #openstack-infra04:33
*** sdake_ has joined #openstack-infra04:36
*** david-lyle has joined #openstack-infra04:36
*** dimtruck is now known as zz_dimtruck04:37
*** baoli has quit IRC04:39
*** sdake has quit IRC04:40
*** david-lyle has quit IRC04:41
*** salv-orlando has joined #openstack-infra04:46
*** markvoelker has joined #openstack-infra04:49
*** snarwade has joined #openstack-infra04:50
*** salv-orlando has quit IRC04:51
*** markvoelker has quit IRC04:53
*** baoli has joined #openstack-infra04:53
*** rbuzatu has quit IRC05:03
*** hongbin has quit IRC05:03
*** gus has quit IRC05:05
*** baoli has quit IRC05:11
*** lindycoder has joined #openstack-infra05:15
*** lindycoder has quit IRC05:19
*** winggundamth has joined #openstack-infra05:23
*** coolsvap has joined #openstack-infra05:24
*** zz_dimtruck is now known as dimtruck05:28
*** dimtruck is now known as zz_dimtruck05:38
*** salv-orlando has joined #openstack-infra05:47
*** sdake has joined #openstack-infra05:47
*** markvoelker has joined #openstack-infra05:50
*** sdake_ has quit IRC05:51
*** salv-orlando has quit IRC05:51
*** markvoelker has quit IRC05:54
*** gouthamr has joined #openstack-infra05:55
*** Ravikiran_K has joined #openstack-infra05:59
*** rbuzatu has joined #openstack-infra06:00
*** abregman has joined #openstack-infra06:12
*** gouthamr has quit IRC06:19
*** winggundamth has quit IRC06:28
*** zz_dimtruck is now known as dimtruck06:29
*** salv-orlando has joined #openstack-infra06:29
*** sdake has quit IRC06:31
*** rbuzatu has quit IRC06:34
*** sdake has joined #openstack-infra06:38
*** dimtruck is now known as zz_dimtruck06:39
*** winggundamth has joined #openstack-infra06:40
*** nadya has joined #openstack-infra06:57
*** hurgleburgler has quit IRC07:02
*** zz_dimtruck is now known as dimtruck07:09
*** Guest71130 has quit IRC07:10
*** salv-orl_ has joined #openstack-infra07:13
*** salv-orlando has quit IRC07:16
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Refactor how puppet sets Gerrit's gitweb configs  https://review.openstack.org/41601507:18
*** dimtruck is now known as zz_dimtruck07:19
clarkbBecause I just noticed but its too late at night for me to deal wiht it right this moment, it looks like journald on our d-g runs may be dropping older messages07:21
clarkbbasically job starts at 1700 ish but syslog starts at 1800 ish07:21
*** nadya has quit IRC07:21
clarkbI do not think this is related to any of the ansibly changes07:22
*** pcaruana has joined #openstack-infra07:23
*** pgadiya has joined #openstack-infra07:23
*** slaweq_ has joined #openstack-infra07:27
openstackgerritKhai Do proposed openstack-infra/system-config: Define Gerrit's gitweb configs as a dict  https://review.openstack.org/41601807:27
*** sdake has quit IRC07:28
*** nadya has joined #openstack-infra07:30
*** rbuzatu has joined #openstack-infra07:31
*** cemason1 is now known as cemason07:33
*** yuli_s has quit IRC07:33
*** florianf has joined #openstack-infra07:36
flip214_Hi. The gate-tempest-dsvm-full-drbd-devstack-ubuntu-xenial-nv CI job has been failing for some time, and the only thing I could see in the log files was07:36
flip214_hat "lvcreate" and similar commands took much too long (eg. 90 seconds).07:36
flip214_here's no kern.log archived any more (with Trusty it was saved, too); so I'd like to ask for SSH access to a node running this check, to find out what's going wrong.07:36
flip214_My SSH pubkey is available on request.07:36
*** sdake has joined #openstack-infra07:37
*** rcernin has joined #openstack-infra07:37
*** rcernin has quit IRC07:39
*** coolsvap has quit IRC07:39
clarkbflip214_: there is no kern.log because we dump all the messages from journald since the time the job started07:41
clarkbyou'll find all of those messages in the single syslog file now (though if the issue above that I mention is prevalent you may not get them from the beginning of jobs)07:41
*** rcernin has joined #openstack-infra07:41
clarkbI'd start by checking that log file though07:41
openstackgerritKhai Do proposed openstack-infra/system-config: Fix gitweb configuration on review-dev.o.o  https://review.openstack.org/41602007:42
*** jtomasek has joined #openstack-infra07:44
*** nadya has quit IRC07:47
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Refactor how puppet sets Gerrit's gitweb configs  https://review.openstack.org/41601507:51
*** rcernin has quit IRC07:51
*** rcernin has joined #openstack-infra07:52
flip214_clarkb: I looked at that one, of course.07:58
flip214_but there's stuff missing, as far as I can say.07:58
clarkbya I think journald may be circularly buffering and dropping older messages07:59
clarkbwill have to dig into that tomorrow07:59
*** zz_dimtruck is now known as dimtruck08:00
*** bhavik1 has joined #openstack-infra08:01
*** yuval has joined #openstack-infra08:02
*** rbuzatu has quit IRC08:04
*** bhavik1 has quit IRC08:07
*** markus_z has joined #openstack-infra08:08
*** dimtruck is now known as zz_dimtruck08:10
*** Hal has joined #openstack-infra08:12
*** Hal is now known as Guest7527408:12
openstackgerritSteven Dake proposed openstack-infra/reviewday: Add kolla deliverables to reviewday  https://review.openstack.org/37668508:16
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Refactor how puppet sets Gerrit's gitweb configs  https://review.openstack.org/41601508:17
*** rbuzatu has joined #openstack-infra08:19
*** abregman has quit IRC08:27
*** abregman has joined #openstack-infra08:30
*** salv-orl_ has quit IRC08:30
*** nmagnezi has joined #openstack-infra08:33
*** sdake has quit IRC08:38
*** nadya has joined #openstack-infra08:41
*** nmagnezi has quit IRC08:43
*** rbuzatu has quit IRC08:46
*** nmagnezi has joined #openstack-infra08:46
*** arxcruz has joined #openstack-infra08:51
*** flepied has joined #openstack-infra08:51
*** Guest75274 has quit IRC08:58
*** Hal has joined #openstack-infra08:58
*** Hal is now known as Guest8235108:59
*** zzzeek has quit IRC09:00
*** zzzeek has joined #openstack-infra09:00
*** pgadiya has quit IRC09:00
*** Guest82351 has quit IRC09:01
*** zz_dimtruck is now known as dimtruck09:01
*** rbuzatu has joined #openstack-infra09:01
*** salv-orlando has joined #openstack-infra09:02
*** pgadiya has joined #openstack-infra09:05
*** abregman has quit IRC09:05
*** hashar has joined #openstack-infra09:09
*** dimtruck is now known as zz_dimtruck09:11
*** d0ugal has joined #openstack-infra09:14
*** abregman has joined #openstack-infra09:14
*** pgadiya has quit IRC09:20
*** dtantsur|afk is now known as dtantsur09:26
*** pgadiya has joined #openstack-infra09:26
*** mat128_ has quit IRC09:28
*** aviau has quit IRC09:29
*** zul has quit IRC09:30
*** cschwede has quit IRC09:30
*** rbuzatu has quit IRC09:30
*** mat128 has joined #openstack-infra09:31
*** cschwede has joined #openstack-infra09:31
*** aviau has joined #openstack-infra09:31
*** SotK_ has quit IRC09:31
*** zul has joined #openstack-infra09:31
*** SotK has joined #openstack-infra09:32
*** askb has joined #openstack-infra09:37
flip214_clarkb: I don't think it's older messages; I believe that not all kernel log levels are being stored somehow.09:40
*** fbo has joined #openstack-infra09:52
*** zz_dimtruck is now known as dimtruck10:01
*** d0ugal has quit IRC10:06
*** dimtruck is now known as zz_dimtruck10:13
*** tosky has joined #openstack-infra10:27
*** Ravikiran_K has quit IRC10:36
*** dargains has joined #openstack-infra10:47
*** dmellado has joined #openstack-infra10:56
*** zz_dimtruck is now known as dimtruck11:04
*** dargains has quit IRC11:07
*** askb has quit IRC11:11
*** mptacekx has joined #openstack-infra11:11
*** nadya has quit IRC11:13
*** dimtruck is now known as zz_dimtruck11:14
mptacekxHi, happy new year :-) Can someone from devstack-gate core's take a look on https://review.openstack.org/#/c/407424/ please ...11:14
*** dargains has joined #openstack-infra11:14
*** Hal has joined #openstack-infra11:15
*** Hal is now known as Guest5470211:16
*** rossella_s has joined #openstack-infra11:19
*** lseki has joined #openstack-infra11:28
*** cdent has joined #openstack-infra11:32
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Fix ordering in the ironic-inspector grenade job definition  https://review.openstack.org/41603511:40
flip214_clarkb: http://logs.openstack.org/56/415956/1/check/gate-tempest-dsvm-full-drbd-devstack-ubuntu-xenial-nv/d05f4ca/logs/syslog.txt.gz#_Jan_01_12_55_3011:45
flip214_shows that something strange happens with the loop device that is used as a VG11:45
flip214_and then the worker gets killed: http://logs.openstack.org/56/415956/1/check/gate-tempest-dsvm-full-drbd-devstack-ubuntu-xenial-nv/d05f4ca/logs/syslog.txt.gz#_Jan_01_13_29_0511:46
flip214_but no kernel messages with more details about that11:47
dtantsurhi! can somebody please check https://review.openstack.org/#/c/414109/? it will hopefully fix our grenade finally..11:48
*** rossella_s has quit IRC11:54
*** wolverineav has joined #openstack-infra11:57
*** bhavik1 has joined #openstack-infra11:57
*** ccamacho is now known as ccamacho|lunch11:58
*** wolverin_ has joined #openstack-infra11:59
*** bhavik1 has quit IRC12:00
*** wolverineav has quit IRC12:03
*** florianf has quit IRC12:03
*** salv-orlando has quit IRC12:04
*** zz_dimtruck is now known as dimtruck12:05
*** dargains has quit IRC12:06
*** asettle has joined #openstack-infra12:07
*** dargains has joined #openstack-infra12:09
*** hashar is now known as hasharLunch12:09
*** pgadiya has quit IRC12:10
*** dimtruck is now known as zz_dimtruck12:14
*** asettle has quit IRC12:17
*** florianf has joined #openstack-infra12:19
*** asettle has joined #openstack-infra12:21
*** dargains has quit IRC12:23
*** yamamoto has joined #openstack-infra12:32
*** snarwade has quit IRC12:38
*** vikrant has quit IRC12:40
*** vikrant has joined #openstack-infra12:41
*** salv-orlando has joined #openstack-infra12:41
*** vikrant has quit IRC12:42
*** jaypipes has joined #openstack-infra12:44
*** rbuzatu has joined #openstack-infra12:49
*** chem has joined #openstack-infra12:52
*** rfolco has joined #openstack-infra12:54
*** markvoelker has joined #openstack-infra12:55
*** markvoelker has quit IRC13:00
*** pcaruana has quit IRC13:00
*** pcaruana has joined #openstack-infra13:04
*** zz_dimtruck is now known as dimtruck13:05
*** oomichi has quit IRC13:07
openstackgerritMerged openstack-infra/openstackid: Removed TLD validation  https://review.openstack.org/41558213:08
*** oomichi has joined #openstack-infra13:08
*** oomichi has quit IRC13:13
*** nadya has joined #openstack-infra13:13
*** salv-orl_ has joined #openstack-infra13:13
*** baoli has joined #openstack-infra13:13
*** baoli has quit IRC13:14
*** oomichi has joined #openstack-infra13:14
*** hasharLunch is now known as hashar13:14
openstackgerritSimon proposed openstack-infra/jenkins-job-builder: Prevent creation of empty parameters for Repo  https://review.openstack.org/41605013:14
*** salv-orl_ has quit IRC13:15
*** dimtruck is now known as zz_dimtruck13:15
*** salv-orlando has quit IRC13:16
*** yamamoto has quit IRC13:18
*** nadya has quit IRC13:18
*** rbuzatu has quit IRC13:19
*** oomichi has quit IRC13:19
*** shu-mutou-OFF has quit IRC13:20
*** oomichi has joined #openstack-infra13:20
*** oomichi has quit IRC13:22
*** rbuzatu has joined #openstack-infra13:23
*** oomichi has joined #openstack-infra13:25
*** dargains has joined #openstack-infra13:26
*** oomichi has quit IRC13:27
*** oomichi has joined #openstack-infra13:30
*** sdake has joined #openstack-infra13:33
*** rbuzatu has quit IRC13:35
*** ccamacho|lunch is now known as ccamacho13:37
openstackgerritDmitry Tantsur proposed openstack-infra/tripleo-ci: Run automated cleaning in the non-HA job on Ocata and newer  https://review.openstack.org/40801013:38
*** rbuzatu has joined #openstack-infra13:40
*** tphummel has joined #openstack-infra13:43
*** david-lyle has joined #openstack-infra13:43
*** baoli has joined #openstack-infra13:46
*** zul has quit IRC13:47
*** david-lyle has quit IRC13:47
*** zul has joined #openstack-infra13:51
*** markvoelker has joined #openstack-infra13:56
openstackgerritSteven Dake proposed openstack-infra/reviewday: Fix pep8 check gate fails completely  https://review.openstack.org/41605713:57
*** bhavik1 has joined #openstack-infra13:57
sdakehey folks if anyone is around - reviewday has failed to build for going on 6 weeks :)13:59
sdakehttps://review.openstack.org/#/c/400227/13:59
*** rlandy has joined #openstack-infra13:59
sdakerather the gate has been blocked14:00
*** markvoelker has quit IRC14:00
*** baoli has quit IRC14:02
*** baoli has joined #openstack-infra14:06
*** zz_dimtruck is now known as dimtruck14:06
*** links has quit IRC14:09
*** rcernin has quit IRC14:09
*** rcernin has joined #openstack-infra14:11
*** bhavik1 has quit IRC14:11
*** baoli has quit IRC14:12
*** rbuzatu has quit IRC14:13
*** e0ne has joined #openstack-infra14:14
*** rbuzatu has joined #openstack-infra14:14
*** salv-orlando has joined #openstack-infra14:15
*** dimtruck is now known as zz_dimtruck14:16
*** dtantsur is now known as dtantsur|bbl14:18
*** yamamoto has joined #openstack-infra14:19
*** e0ne has quit IRC14:20
*** salv-orlando has quit IRC14:21
*** psachin has quit IRC14:21
*** e0ne has joined #openstack-infra14:24
*** e0ne has quit IRC14:26
*** yamamoto has quit IRC14:29
*** e0ne has joined #openstack-infra14:30
*** berendt has joined #openstack-infra14:32
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE: test periodic with swap  https://review.openstack.org/41606514:33
berendtWhen running gate jobs, is it possible to exclude a specific cloud provider?14:34
*** e0ne has quit IRC14:34
*** e0ne has joined #openstack-infra14:38
*** e0ne has quit IRC14:40
*** ansiwen_ is now known as ansiwen14:50
openstackgerritMerged openstack-infra/storyboard: H803 hacking have been deprecated  https://review.openstack.org/41356014:53
*** baoli has joined #openstack-infra14:56
*** baoli has quit IRC14:56
*** markvoelker has joined #openstack-infra14:57
AJaegerberendt: no.14:57
AJaegerberendt: see also http://docs.openstack.org/infra/manual/testing.html14:58
*** markvoelker has quit IRC15:01
*** baoli has joined #openstack-infra15:01
*** baoli has quit IRC15:01
*** asettle has quit IRC15:06
*** zz_dimtruck is now known as dimtruck15:07
*** dargains has quit IRC15:11
openstackgerritZara proposed openstack-infra/storyboard: Make it impossible to create a userless private story  https://review.openstack.org/41607015:11
*** hurgleburgler has joined #openstack-infra15:14
*** links has joined #openstack-infra15:16
*** salv-orlando has joined #openstack-infra15:17
*** rbuzatu has quit IRC15:18
*** rbuzatu has joined #openstack-infra15:18
*** rbuzatu has quit IRC15:18
*** rbuzatu has joined #openstack-infra15:18
*** dargains has joined #openstack-infra15:21
*** salv-orlando has quit IRC15:21
*** baoli has joined #openstack-infra15:23
*** baoli has quit IRC15:25
*** dargains has quit IRC15:26
*** mptacekx has quit IRC15:29
openstackgerritMerged openstack-infra/project-config: Fix Inspector grenade: enable devstack plugin once  https://review.openstack.org/41410915:34
*** baoli has joined #openstack-infra15:38
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Enable ask-staging.openstack.org puppet run  https://review.openstack.org/27665615:41
*** salv-orlando has joined #openstack-infra15:41
*** baoli has quit IRC15:43
*** links has quit IRC15:44
mrmartinhi fungi15:44
*** david-lyle has joined #openstack-infra15:45
mrmartinthe stackalytics.openstack.org not answering the http requests, is it happens due a maintenance, or just simply gone away?15:47
fungimrmartin: hey there. i was about to check in with you and find out if you're okay with me approving 408657 (i didn't want to approve that and disappear for days on end, but now that i'm around i'm happy to keep an eye on it)15:48
*** nmagnezi has quit IRC15:48
fungimrmartin: not sure about stackalyics.o.o, i know there are some funky restart operations that happen when it updates which take it offline for long periods. maybe one of them got stuck15:48
fungii'll take a look15:48
*** david-lyle has quit IRC15:49
AJaegerhappy new year, infra!15:49
fungimrmartin: since that site hasn't gone into production yet, i doubt anyone's been keeping a close watch on it15:50
mrmartinI'm around today and tomorrow. May I get an ssh access, so I can help if something is going wrong? I did the tests, and the patchset is running well in my local dev instance, but who knows, askbot upgrades can be tricky.15:50
fungihappy new year AJaeger!15:50
AJaegerfungi, clarkb, pabelanger : There're quite a few open requests for project-config that have one +2 - if either of you have a few minutes to review this week, it would be great. Also, I have two changes that need review love: https://review.openstack.org/415285 and https://review.openstack.org/411469 . thanks!15:51
AJaegerthanks, fungi !15:51
fungimrmartin: propose a change adding ssh access for your account to the ask.o.o server and i'll see if i can convince the team, but seem reasonable15:51
mrmartinok. let's do it as a first step before the approval.15:51
fungiworks for me15:52
dmelladoAJaeger: happy new year ;)15:53
AJaegerfungi, infra-root, regarding docs.o.o (see http://lists.openstack.org/pipermail/openstack-docs/2016-December/009430.html ) - do you want to move docs.o.o to AFS published docs this week while it's still a bit silent?15:53
AJaegerdmellado: thanks15:53
AJaegerOr shall I put it on the infra team agenda for tomorrow?15:54
fungiAJaeger: i haven't finished catching up on e-mail yet, but it seems like an appropriate topic for tomorrow's meeting since it's a major step for one of our priority efforts15:55
AJaegerfungi: ok, will put it up later - just got called away15:55
openstackgerritMerged openstack-infra/project-config: Add scenario test for neutron-dynamic-routing  https://review.openstack.org/40777815:55
*** markvoelker has joined #openstack-infra15:58
*** rhallisey has joined #openstack-infra15:59
*** rbuzatu has quit IRC15:59
*** dtantsur|bbl is now known as dtantsur16:00
*** rbuzatu has joined #openstack-infra16:00
openstackgerritMarton Kiss proposed openstack-infra/system-config: Adding user 'mkiss' to ask.o.o  https://review.openstack.org/41607216:02
*** markvoelker has quit IRC16:02
mrmartinfungi , this one ^^16:02
fungithanks, mrmartin!16:03
*** yuval has quit IRC16:05
fungiinfra-root: https://review.openstack.org/416072 would grant mrmartin a shell account on ask.openstack.org (production). we want to perform a major upgrade and while some production-like upgrade testing has been done in staging, it's a fragile process and getting him direct access to view logs and inspect the state on disk would be very helpful if this second upgrade attempt also fails16:06
*** hongbin has joined #openstack-infra16:08
*** panda is now known as panda|pto16:13
*** dimtruck is now known as zz_dimtruck16:13
*** asettle has joined #openstack-infra16:13
*** matrohon has joined #openstack-infra16:15
*** rcernin has quit IRC16:16
* fungi sighs at "nokogiri-1.7.0 requires ruby version >= 2.1.0, which is incompatible with the current version, ruby 2.0.0p648"16:16
fungigate-openstackci-beaker-centos-7 seems to be broken by a dep16:16
funginokogiri tagged v1.7.0 a week ago16:18
*** abregman has quit IRC16:20
*** sdake has quit IRC16:23
AJaegerfungi, there's a change up for that...16:25
fungioh, awesome, i'll stop writing mine16:25
AJaegerfungi, https://review.openstack.org/41571016:25
fungithanks! approved16:26
fungiinteresting we don't run gate-openstackci-beaker-.* jobs against that repo16:27
AJaegerfungi, should I remove the agenda items from last time from the infra team meeting agenda for tomorrow - or do you want to do it yourself?16:27
fungiAJaeger: i'll clean it up just to make sure i don't accidentally remove any new stuff, just add your sub-bullet in "Docs Publishing via AFS"16:27
AJaegerok16:28
*** liusheng has quit IRC16:29
*** liusheng has joined #openstack-infra16:29
openstackgerritMerged openstack-infra/puppet-openstack_infra_spec_helper: Pin nokogiri to fix centos 7 jobs  https://review.openstack.org/41571016:30
*** dargains has joined #openstack-infra16:35
*** rcernin has joined #openstack-infra16:39
*** baoli has joined #openstack-infra16:39
*** dargains has quit IRC16:40
*** markus_z has quit IRC16:41
*** baoli has quit IRC16:44
*** sdake has joined #openstack-infra16:50
*** dave-mccowan has joined #openstack-infra16:53
*** pcrews has quit IRC16:54
*** cdent has quit IRC16:54
*** sdake has quit IRC16:55
clarkbto pick up from last night journald's default is to only persistent logs if /var/log/journald exists or if /etc/systemd/journald.conf has storage set to persistent16:56
clarkbnow I need to check if our images do either one of these things16:56
clarkbflip214_: journald by default is supposed to print kernel messages16:56
clarkbflip214_: if the time range is captured then I think its possible that nothing is being logged there?16:57
*** markvoelker has joined #openstack-infra16:58
*** jamesdenton has joined #openstack-infra17:03
*** markvoelker has quit IRC17:03
clarkbflip214_: perhaps 'Jan 01 13:29:05 ubuntu-xenial-rax-ord-6436518 systemd-journald[312]: Suppressed 247 messages from /system.slice/dbus.service' is where this would have been logged?17:06
*** dargains has joined #openstack-infra17:11
*** baoli has joined #openstack-infra17:12
*** baoli has quit IRC17:12
clarkbmrmartin: fungi I have rechecked that ask.o.o access' change now that the infra spec helper change has merged17:12
*** dargains has quit IRC17:15
*** bhavik1 has joined #openstack-infra17:18
*** dmellado has quit IRC17:19
*** dtantsur is now known as dtantsur|afk17:19
*** zz_dimtruck is now known as dimtruck17:21
*** rattboi_ has quit IRC17:31
*** rattboi has joined #openstack-infra17:31
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Don't remove swap file  https://review.openstack.org/41607817:32
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Don't remove swap file  https://review.openstack.org/41607817:33
*** tinwood has quit IRC17:39
fungithanks clarkb17:39
*** hashar has quit IRC17:39
clarkbI have confirmed that our xenial iamges do not have a /var/log/journald so will not have their logs stored on disk17:40
clarkbwhich probably explains why on longer running jobs we get that circular buffer behavior17:40
*** d0ugal has joined #openstack-infra17:40
*** d0ugal has quit IRC17:40
*** d0ugal has joined #openstack-infra17:40
*** rossella_s has joined #openstack-infra17:41
clarkbfedora has a /var/log/journal but centos does not17:43
clarkbI thought system was going to fix all of these problems17:43
*** dargains has joined #openstack-infra17:43
fungiclearly these are systemic problems ;)17:44
clarkbtranslate-dev.o.o which was booted on a rax xenial image also does not have a /var/log/journal17:45
clarkband neither does git01.o.o booted on the rax centos 7 image17:45
*** rossella_s has quit IRC17:47
*** d0ugal has quit IRC17:49
*** dougwig has quit IRC17:51
*** andrewbogott has quit IRC17:51
*** MarkAtwood has quit IRC17:51
*** tdurakov has quit IRC17:51
*** briancurtin has quit IRC17:51
*** fdegir has quit IRC17:52
*** ildikov has quit IRC17:52
*** Ng has quit IRC17:52
*** tomhambleton has quit IRC17:52
*** mgkwill has quit IRC17:52
*** johnsom has quit IRC17:52
*** vdrok has quit IRC17:52
*** dirk has quit IRC17:52
*** ogelbukh has quit IRC17:52
*** puiterwijk|cld has quit IRC17:52
*** jraim has quit IRC17:52
*** ewindisch has quit IRC17:52
*** betherly has quit IRC17:53
*** Kevin_Zheng has quit IRC17:53
*** gmann has quit IRC17:53
*** TheJulia has quit IRC17:53
*** sripriya has quit IRC17:53
*** iremizov has quit IRC17:53
*** boris-42 has quit IRC17:53
*** rpodolyaka has quit IRC17:53
*** rguillebert has quit IRC17:53
*** nikhil has quit IRC17:53
*** csatari has quit IRC17:53
*** scottda has quit IRC17:53
*** samueldmq has quit IRC17:54
*** tphummel has quit IRC17:54
*** tomhambleton has joined #openstack-infra17:55
*** baoli has joined #openstack-infra17:55
*** MarkAtwood has joined #openstack-infra17:55
*** andrewbogott has joined #openstack-infra17:55
*** dougwig has joined #openstack-infra17:55
*** baoli has quit IRC17:55
flip214_clarkb: possibly. I would like to know what happens during the time when the dbus communication is blocked.17:55
*** jamesdenton has quit IRC17:56
*** tphummel has joined #openstack-infra17:56
*** baoli has joined #openstack-infra17:56
*** ildikov has joined #openstack-infra17:56
*** ewindisch has joined #openstack-infra17:56
*** mgkwill has joined #openstack-infra17:56
*** Kevin_Zheng has joined #openstack-infra17:56
*** sripriya has joined #openstack-infra17:56
*** csatari has joined #openstack-infra17:56
*** bhavik1 has quit IRC17:56
*** ogelbukh has joined #openstack-infra17:57
*** boris-42 has joined #openstack-infra17:57
*** iremizov has joined #openstack-infra17:57
flip214_it seems that lvchange and other commands are blocked during that time17:57
flip214_and I'd have hoped for the kernel log to explain why17:57
*** samueldmq has joined #openstack-infra17:57
*** nikhil has joined #openstack-infra17:57
*** dave-mccowan has quit IRC17:57
clarkbflip214_: right, and the kernel log is that file17:57
clarkbso either 1) there isn't anything logging or 2) its hidden in that suppression17:58
*** vdrok has joined #openstack-infra17:58
clarkbor 3) its logged to some level that we don't print by default for some reason17:58
*** rguillebert has joined #openstack-infra17:58
*** betherly has joined #openstack-infra17:58
*** TheJulia has joined #openstack-infra17:58
*** rpodolyaka has joined #openstack-infra17:59
*** markvoelker has joined #openstack-infra17:59
*** fdegir has joined #openstack-infra17:59
*** johnsom has joined #openstack-infra17:59
*** gmann has joined #openstack-infra18:00
*** scottda has joined #openstack-infra18:00
*** tdurakov has joined #openstack-infra18:00
*** jraim has joined #openstack-infra18:00
*** markvoelker has quit IRC18:04
*** salv-orlando has quit IRC18:04
*** salv-orlando has joined #openstack-infra18:05
*** tphummel has quit IRC18:06
*** srobert has joined #openstack-infra18:07
*** srobert has quit IRC18:07
*** briancurtin has joined #openstack-infra18:07
*** srobert has joined #openstack-infra18:07
openstackgerritClark Boylan proposed openstack-infra/project-config: Make journald log collection persistent  https://review.openstack.org/41608018:11
clarkbdib cores may have better ideas for ^18:11
clarkbianw ^18:11
EmilienMhi! can someone review https://review.openstack.org/#/c/414606/ please? (project-config)18:14
EmilienMand review on https://review.openstack.org/#/c/414585/18:14
EmilienM(and happy new year infra folks :-))18:15
*** pono has quit IRC18:21
*** dargains has quit IRC18:23
*** gouthamr has joined #openstack-infra18:27
*** asettle has quit IRC18:28
*** asettle has joined #openstack-infra18:28
*** asettle has quit IRC18:33
*** pcrews has joined #openstack-infra18:37
*** hongbin has quit IRC18:39
*** pcrews has quit IRC18:39
*** tphummel has joined #openstack-infra18:47
*** dave-mccowan has joined #openstack-infra18:51
*** gouthamr has quit IRC18:53
*** flepied has quit IRC18:56
*** tphummel has quit IRC18:57
*** markvoelker has joined #openstack-infra19:00
AJaegerfungi, just reading your reply to my email on docs.o.o. Do you have a few minutes to chat?19:02
fungiAJaeger: sure19:02
AJaegergreat.19:03
AJaegerdocs.o.o has some historic data and we (=docs team) fear that we might need some data later. That's why the question for an archive came up.19:04
*** markvoelker has quit IRC19:05
AJaegerAn alternative I envison would be to download all files and create an extra infra server docs-archive.o.o that we would keep for some months up.19:05
AJaegerOr just download the files to some location so that we can access them if needed.19:05
fungiAJaeger: yep, that's an option i was considering as well, though we probably need to have more people involved in that discussion (which is why tomorrow's meeting is probably a good time to bring it up)19:05
fungidid you have any numbers on the size of that archive?19:06
AJaegerfungi: according to line 22 of https://etherpad.openstack.org/p/afs-docs ianw has done such an archive already - not sure where19:07
AJaegerah, it's at /afs/.openstack.org/docs-old/19:08
clarkbAJaeger: you should be able to drop the . and get public read access19:09
AJaegerclarkb: http://files.openstack.org/docs-old/ works as well19:09
AJaegermy local archive from August was 10 GB.19:09
*** pcrews has joined #openstack-infra19:10
AJaegerSo, we could resync to that location and add a public IP to it.19:10
AJaegerso, with that we should be fine - I forgot about that ;(19:11
AJaegerwill answer now to that email...19:12
*** salv-orl_ has joined #openstack-infra19:14
*** ianychoi has quit IRC19:15
fungiAJaeger: cool, yes i have no issue with the idea of serving that archived docs tree for as long as we can reasonably manage to do so. i'm also not opposed to refreshing it but that may be fairly time intensive due to lack of rsync support on the cloudsites end (we likely have to crawl and retransfer a lot of files via ftp)19:16
*** salv-orlando has quit IRC19:17
clarkbhow did we do it the first time around?19:17
clarkbI guess first time you copy everything anwyays so didn't matter19:17
*** rbuzatu has quit IRC19:18
AJaegerclarkb: see line 50 in the etherpad19:18
AJaegerrecursive lftp19:18
*** rbuzatu has joined #openstack-infra19:18
AJaegerfungi, yes, it will run over a day I fear.19:18
clarkbif we use lftp mirror -n it should do an incremental update based on timestamps looks like19:20
clarkbwhich should be quicker?19:20
AJaegerclarkb: worth trying ;)19:20
fungiyeah, not as fast as rsync but hopefully way faster than starting from scratch19:20
* AJaeger just removed the WIP from https://review.openstack.org/#/c/415506 - the change to stop publishing to the old developer site19:22
*** dimtruck is now known as zz_dimtruck19:22
clarkbugh I should figure out why logstash/ES are unhappy19:22
clarkbseems like they do that more often now19:22
*** rbuzatu has quit IRC19:23
AJaegerok, answered via email.19:23
AJaegertalk to you again tomorrow!19:23
fungithanks AJaeger!19:24
*** rbuzatu has joined #openstack-infra19:24
*** salv-orl_ has quit IRC19:25
AJaegerfungi, clarkb if any of you has time to rerun the mirror, it would be nice - then we can switch sooner...19:25
clarkblooks like the workers are blocking sending data to logstash which implies logstash is unhapppy but nothing in the lgostash logs since novemeber19:25
AJaeger;(19:25
* AJaeger signs off19:25
clarkbfungi: if you have a moment to review https://review.openstack.org/#/c/388122/1 I can get services restarted today. Otherwise I may just self approve (jeblair has reviewed it so its not unreviewed)19:27
clarkbI do see exceptions about the crm114 processor so its possible ^ is biting us19:27
clarkbfungi: and I think I see an issue with that change... its actually a noop19:34
clarkbI WIP'd it19:34
fungiunapproved19:34
clarkbwe generate new_filters for every log event so it won't carry over any "bad" state between events19:35
clarkbES looks happy (its actually not doing much because all the old indexes have timed out and been deleted as no new data is going into the lcuster...)19:37
*** e0ne has joined #openstack-infra19:37
clarkbso going to start with restarting logstash daemons then logstash worker daemons (if necessary)19:37
*** rbuzatu has quit IRC19:39
*** e0ne has quit IRC19:39
openstackgerritMerged openstack-infra/system-config: Enable ask-staging.openstack.org puppet run  https://review.openstack.org/27665619:39
clarkbok restarting the logstash daemon on logstash-worker02 made stuff start happening19:41
clarkbwill do the otehrs now19:41
clarkb(also had to be kill -9'd so likely in a bad way for some reason)19:42
*** e0ne has joined #openstack-infra19:43
*** nadya has joined #openstack-infra19:44
*** e0ne has quit IRC19:45
clarkbprobably worht investigating taking the laep to elasticsearch 2.x and updating logstash to whatever is current19:47
clarkbwill have to look at what the API changes are in 2.x19:47
*** e0ne has joined #openstack-infra19:50
*** nadya has quit IRC19:50
*** e0ne has quit IRC19:52
*** dave-mccowan has quit IRC19:58
*** florianf has quit IRC19:59
*** pcaruana has quit IRC20:00
fungimrmartin: reviewing the logs on stackalytics.o.o, apache was complaining about "AH00485: scoreboard is full, not at MaxRequestWorkers" going back as far as september. an apache restart seems to have solved it20:00
*** cemason has quit IRC20:00
clarkbThings seem to be moving now. Will check in on e-r status pages to make sure it is happy againtoo20:00
*** markvoelker has joined #openstack-infra20:01
fungimrmartin: looks like it may be an occurrence of https://bz.apache.org/bugzilla/show_bug.cgi?id=5355520:01
openstackbz.apache.org bug 53555 in mpm_event "Scoreboard full error with event/ssl" [Major,Resolved: fixed] - Assigned to bugs20:01
mrmartinsounds nice20:01
clarkbheh except we are going to try and index the last weeks worth of logs :/20:02
mrmartinI'm just trying to deploy stackalytics into a dev instance.20:02
clarkbload drops over the holidays so gonna let it run for a bit and see if it catches up at all20:02
clarkboh wait even better20:02
clarkbits indexing things it thinks are from december 23rd 2017...20:03
*** Ng has joined #openstack-infra20:03
*** dirk has joined #openstack-infra20:03
clarkbIs it syslog that doesn't record the year?20:03
fungiclarkb: yes, looks like it20:03
*** puiterwijk|cld has joined #openstack-infra20:04
fungiugh. recent vim upgrade seems to have unexpectedly enabled x11 pointer integration20:04
clarkb:set mouse=""20:05
*** markvoelker has quit IRC20:06
clarkbor maybe its mouse= ?20:06
fungithe latter seems to have worked, thanks20:06
fungii suspect this was due to the debian package for it starting to ship a global default vimrc20:07
*** yamahata has joined #openstack-infra20:07
fungiaha, yes it's only happening when i `sudo vi ...` or similar, so "...now ships with a defaults.vim file which, when the user has no vimrc, enables some options..." in /usr/share/doc/vim/NEWS.Debian.gz looks likely20:10
fungikicking in because i have no ~root/.vimrc apparently20:11
*** jamesdenton has joined #openstack-infra20:12
fungistarted in vim 8.0 apparently20:12
*** rbuzatu has joined #openstack-infra20:13
*** asettle has joined #openstack-infra20:14
*** asettle has quit IRC20:14
*** srobert has quit IRC20:15
*** asettle has joined #openstack-infra20:15
*** asettle has quit IRC20:15
*** asettle has joined #openstack-infra20:15
*** asettle has quit IRC20:15
*** asettle has joined #openstack-infra20:16
*** asettle has quit IRC20:16
*** asettle has joined #openstack-infra20:17
*** asettle has quit IRC20:17
*** e0ne has joined #openstack-infra20:22
*** e0ne has quit IRC20:24
*** rfolco has quit IRC20:26
yankcrimeif in doubt, :smile20:27
*** spligak has joined #openstack-infra20:28
*** yamahata has quit IRC20:29
fungi(and don't panic!)20:34
*** david-lyle has joined #openstack-infra20:48
*** david-lyle has quit IRC20:52
*** rbuzatu_ has joined #openstack-infra20:54
*** asettle has joined #openstack-infra20:54
*** rbuzatu has quit IRC20:55
*** salv-orlando has joined #openstack-infra20:55
*** flepied has joined #openstack-infra20:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove docs publishing from networking-calico  https://review.openstack.org/41608720:56
*** pcaruana has joined #openstack-infra21:00
*** salv-orlando has quit IRC21:00
*** asettle has quit IRC21:01
*** salv-orlando has joined #openstack-infra21:01
*** markvoelker has joined #openstack-infra21:01
*** rcernin has quit IRC21:02
*** markvoelker has quit IRC21:06
*** pcrews has quit IRC21:06
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs  https://review.openstack.org/35921521:08
*** asettle has joined #openstack-infra21:09
*** asettle has quit IRC21:10
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove docs publishing from legacy repos  https://review.openstack.org/41608721:11
*** rbuzatu_ has quit IRC21:13
*** pcaruana has quit IRC21:20
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Fixes dib-init-system for Debian Jessie and Debian Stretch  https://review.openstack.org/41609121:22
clarkb#status log logstash daemons were 'stuck' and have been restarted on logstash-worker0X.o.o hosts. Events are being processed and indexed again as a result. Should probably look into upgrading logstash install (and possibly elasticsearch21:30
openstackstatusclarkb: finished logging21:30
*** Jeffrey4l__ has quit IRC21:34
*** Jeffrey4l__ has joined #openstack-infra21:36
clarkbI think we should upgrade ES to https://www.elastic.co/downloads/past-releases/elasticsearch-1-7-6 and logstash to https://www.elastic.co/downloads/past-releases/logstash-2-4-121:40
clarkbI am going to start testing the logstash 2.4.1 upgrade as its a bit easier to do21:40
*** ianychoi has joined #openstack-infra21:41
*** pcaruana has joined #openstack-infra21:42
clarkb(I went ahead and updated logstash-worker01 to 2.4.1 as it was already disabled in puppet due to the mqtt issues. If it holds up I will push a change later this week to update the rest of the logstash workers)21:45
*** sdake has joined #openstack-infra21:51
*** aviau has quit IRC21:53
*** aviau has joined #openstack-infra21:53
*** sdake has quit IRC21:56
jheskethMorning21:59
*** markvoelker has joined #openstack-infra22:02
*** markvoelker has quit IRC22:07
openstackgerritDirk Mueller proposed openstack-dev/pbr: Don't raise exception on missing man pages  https://review.openstack.org/41379522:09
*** pcaruana has quit IRC22:13
*** hurgleburgler has quit IRC22:18
*** zz_dimtruck is now known as dimtruck22:20
*** rbuzatu has joined #openstack-infra22:27
*** umeshksingla has joined #openstack-infra22:29
*** rbuzatu has quit IRC22:29
*** wolverin_ has quit IRC22:46
*** wolverineav has joined #openstack-infra22:47
*** wolverineav has quit IRC22:49
*** david-lyle has joined #openstack-infra22:49
*** askb has joined #openstack-infra22:49
*** wolverineav has joined #openstack-infra22:49
*** jerryz has joined #openstack-infra22:53
*** david-lyle has quit IRC22:54
*** wolverineav has quit IRC22:54
*** jerryz has quit IRC22:54
*** rbuzatu has joined #openstack-infra22:54
*** wolverineav has joined #openstack-infra22:55
*** hurgleburgler has joined #openstack-infra22:57
*** dimtruck is now known as zz_dimtruck22:58
*** zz_dimtruck is now known as dimtruck22:58
*** wolverineav has quit IRC23:00
*** hurgleburgler has quit IRC23:00
*** markvoelker has joined #openstack-infra23:03
*** pahuang has joined #openstack-infra23:07
*** dimtruck is now known as zz_dimtruck23:07
*** markvoelker has quit IRC23:08
*** hurgleburgler has joined #openstack-infra23:12
*** hurgleburgler has quit IRC23:17
*** umeshksingla has quit IRC23:23
*** umeshksingla has joined #openstack-infra23:23
*** Julien-zte has quit IRC23:24
*** jaypipes has quit IRC23:27
*** salv-orlando has quit IRC23:29
*** lseki has quit IRC23:30
*** pabelanger has quit IRC23:31
*** migi has quit IRC23:32
*** pabelanger has joined #openstack-infra23:32
*** migi has joined #openstack-infra23:33
*** gnuoy has quit IRC23:38
*** rbuzatu has quit IRC23:38
*** rbuzatu has joined #openstack-infra23:39
*** rbuzatu has quit IRC23:41
*** rbuzatu has joined #openstack-infra23:41
*** rfolco has joined #openstack-infra23:44
*** umeshksingla has left #openstack-infra23:50
*** umeshksingla has joined #openstack-infra23:50
*** adreznec has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!