Saturday, 2018-04-14

*** tosky has quit IRC00:02
openstackgerritMerged openstack-infra/system-config master: Re-enable the websockets tls port  https://review.openstack.org/56133600:03
*** d0ugal has quit IRC00:05
*** d0ugal has joined #openstack-infra00:06
*** slaweq has joined #openstack-infra00:11
*** slaweq has quit IRC00:16
*** lbragstad has quit IRC00:30
*** sambetts|afk has quit IRC00:32
*** sambetts_ has joined #openstack-infra00:34
fungii suppose if any infra-root wants to approve https://review.openstack.org/561342 then i'll be happy to restart gerrit for it over the weekend00:37
fungiotherwise it can wait for a suitable time next week00:37
*** anteaya has quit IRC00:40
*** harlowja has quit IRC00:41
*** wolverin_ has quit IRC00:42
*** e0ne has joined #openstack-infra00:49
*** e0ne has quit IRC00:50
*** germs has joined #openstack-infra00:51
*** gyee has quit IRC00:55
*** germs has quit IRC00:57
mtreinishhmm, it looks like the port got opened on firehose01.o.o but puppet didn't restart mosquitto to take the config change00:58
*** mriedem has quit IRC01:09
*** rh-jelabarre has quit IRC01:11
*** cshastri has joined #openstack-infra01:25
*** harlowja has joined #openstack-infra01:25
*** sambetts_ has quit IRC01:31
*** sambetts_ has joined #openstack-infra01:35
*** sambetts_ has quit IRC01:39
*** ihrachys has quit IRC01:40
*** sambetts_ has joined #openstack-infra01:40
*** salv-orl_ has joined #openstack-infra01:47
*** salv-orlando has quit IRC01:50
*** yamahata has quit IRC01:50
*** rpioso is now known as rpioso|afk01:51
*** shoogz has quit IRC02:06
*** sambetts_ has quit IRC02:07
*** sambetts_ has joined #openstack-infra02:07
*** shoogz has joined #openstack-infra02:08
fungiwe probably don't have the service subscribe/notify to the config?02:10
fungii'll give it a kick in the bits02:10
fungimtreinish: mmm, no dice. first the config hasn't been updated yet (last modified in february)02:20
*** migi has quit IRC02:20
*** sai1 has quit IRC02:20
*** sai_ has joined #openstack-infra02:20
*** ssbarnea has quit IRC02:21
*** _ari_ has quit IRC02:21
*** migi has joined #openstack-infra02:21
mtreinishfungi: hmm, that's odd. I would have assumed the puppet run which opened the port in the firewall would have pulled in the new config02:21
*** _ari_ has joined #openstack-infra02:21
*** haleyb has quit IRC02:21
mtreinishwell it's too late to debug it now. I guess it'll have to wait until later02:22
*** ssbarnea has joined #openstack-infra02:23
fungiyeah, i can dig into it further later but am packing it in for tonight02:23
*** eharney has quit IRC02:25
*** haleyb has joined #openstack-infra02:25
*** rfolco|off has quit IRC02:40
*** harlowja has quit IRC02:49
*** jamesmcarthur has joined #openstack-infra02:52
*** edmondsw has quit IRC02:56
*** jamesmcarthur has quit IRC02:56
*** germs has joined #openstack-infra02:58
*** germs has quit IRC02:58
*** germs has joined #openstack-infra02:58
*** sambetts_ has quit IRC03:02
ianychoiWould there be somebody who can kick an attacking bot user in #openstack-swift IRC channel?03:03
*** germs has quit IRC03:03
ianychoiThe bot has gone - thank u03:04
ianychoiUnlike other bots, a bot was alive for several minutes and called people a lot...03:05
*** dhajare has joined #openstack-infra03:07
*** sambetts_ has joined #openstack-infra03:08
*** imacdonn has quit IRC03:09
*** imacdonn has joined #openstack-infra03:09
*** dhill_ has quit IRC03:13
*** dhill_ has joined #openstack-infra03:14
*** dpawlik has joined #openstack-infra03:19
*** adreznec has quit IRC03:22
*** dpawlik has quit IRC03:23
*** adreznec has joined #openstack-infra03:24
*** fried_rice has quit IRC03:27
*** fried_rice has joined #openstack-infra03:27
*** yamahata has joined #openstack-infra03:57
*** pbourke_ has joined #openstack-infra04:04
*** pbourke has quit IRC04:05
*** harlowja has joined #openstack-infra04:09
*** slaweq has joined #openstack-infra04:13
*** nicolasbock has quit IRC04:13
*** dhajare has quit IRC04:17
*** slaweq has quit IRC04:18
*** witek has quit IRC04:33
*** witek has joined #openstack-infra04:37
*** dhajare has joined #openstack-infra04:38
*** kei-ichi has quit IRC04:39
*** kei-ichi has joined #openstack-infra04:40
*** hongbin has joined #openstack-infra04:48
*** hongbin has quit IRC04:50
*** germs has joined #openstack-infra04:59
*** germs has quit IRC04:59
*** germs has joined #openstack-infra04:59
*** harlowja has quit IRC05:01
*** caphrim007 has joined #openstack-infra05:01
*** germs has quit IRC05:03
*** dhajare has quit IRC05:12
*** chkumar|off is now known as chandankumar05:19
*** sree_ has joined #openstack-infra05:33
*** sree_ is now known as Guest5151705:33
*** dhajare has joined #openstack-infra05:37
*** lpetrut has joined #openstack-infra05:38
*** dhajare has quit IRC05:58
*** dklyle has quit IRC06:04
*** dklyle has joined #openstack-infra06:05
*** psachin has joined #openstack-infra06:06
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/56139206:10
*** iranzo has joined #openstack-infra06:11
*** harlowja has joined #openstack-infra06:13
*** armaan has joined #openstack-infra06:15
openstackgerritAbbas Yazdanpanah proposed openstack-infra/git-review master: `work-in-progress` and `private` workflow in Gerrit 2.15  https://review.openstack.org/55948806:15
*** Guest51517 has quit IRC06:19
*** harlowja has quit IRC06:28
*** lpetrut has quit IRC06:28
*** dbecker has quit IRC06:31
*** claudiub|2 has joined #openstack-infra06:34
*** lpetrut has joined #openstack-infra06:37
*** dbecker has joined #openstack-infra06:44
*** jogo has quit IRC06:52
*** dhajare has joined #openstack-infra06:52
*** gongysh has joined #openstack-infra06:53
*** gongysh has quit IRC07:05
*** armaan has quit IRC07:10
*** armaan has joined #openstack-infra07:11
*** dpawlik has joined #openstack-infra07:15
*** jogo has joined #openstack-infra07:16
*** jogo has joined #openstack-infra07:16
*** dpawlik has quit IRC07:17
*** gongysh has joined #openstack-infra07:17
*** lpetrut has quit IRC07:20
*** gongysh has quit IRC07:26
*** rajinir has quit IRC07:36
*** neiloy_ has joined #openstack-infra07:37
*** psachin has quit IRC07:38
*** dsariel has quit IRC07:48
*** claudiub|2 has quit IRC07:49
*** adam_g has quit IRC07:52
*** adam_g has joined #openstack-infra07:53
*** armaan has quit IRC08:08
*** sree_ has joined #openstack-infra08:09
*** sree_ is now known as Guest8879008:09
*** jtomasek has quit IRC08:11
*** sambetts_ has quit IRC08:12
*** sambetts_ has joined #openstack-infra08:13
*** dsariel has joined #openstack-infra08:25
*** armaan has joined #openstack-infra08:30
*** e0ne has joined #openstack-infra08:33
*** Guest88790 has quit IRC08:48
*** sree_ has joined #openstack-infra08:48
*** sree_ is now known as Guest4636808:49
*** agopi_ has quit IRC08:51
*** e0ne has quit IRC08:52
*** Guest46368 has quit IRC08:53
openstackgerritXing Zhang proposed openstack-infra/puppet-etherpad_lite master: Set legacy_debian_symlinks to true  https://review.openstack.org/56139808:57
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/56139209:16
*** neiloy_ has quit IRC09:24
*** xarses has quit IRC09:40
*** xarses has joined #openstack-infra09:40
*** sambetts_ has quit IRC09:47
*** sambetts_ has joined #openstack-infra09:50
*** cshastri has quit IRC10:12
*** janki has joined #openstack-infra10:18
*** tosky has joined #openstack-infra10:19
*** armaan_ has joined #openstack-infra10:26
*** armaan has quit IRC10:27
*** armaan_ has quit IRC10:39
*** armaan has joined #openstack-infra10:40
*** e0ne has joined #openstack-infra10:57
*** cshastri has joined #openstack-infra11:10
*** claudiub|2 has joined #openstack-infra11:10
*** e0ne has quit IRC11:16
*** slaweq has joined #openstack-infra11:19
*** slaweq has quit IRC11:21
*** jchhatbar has joined #openstack-infra11:30
*** janki has quit IRC11:33
*** adam_g has quit IRC11:45
*** mriedem has joined #openstack-infra11:45
*** adam_g has joined #openstack-infra11:45
*** sree has joined #openstack-infra11:48
*** jtomasek has joined #openstack-infra11:53
*** sree has quit IRC11:53
fungiianychoi: yeah, this round of freenode spam hit roughly a dozen channels i'm in with highlight spam and also integrated some privmsg spam. looks like they probably hit a significant proportion of the channels on freenode from numerous random nicks before freenode staff could block them12:10
*** cshastri has quit IRC12:10
*** dsariel has quit IRC12:13
*** neith has quit IRC12:43
*** dpawlik has joined #openstack-infra12:45
*** dpawlik has quit IRC12:50
*** e0ne has joined #openstack-infra12:57
*** e0ne has quit IRC13:00
*** sree has joined #openstack-infra13:00
openstackgerritMerged openstack-infra/system-config master: Comment on linked stories for review state changes  https://review.openstack.org/56134213:02
*** e0ne has joined #openstack-infra13:02
*** sree has quit IRC13:05
*** yamamoto has joined #openstack-infra13:08
fungii'll restart gerrit on review-dev01.o.o and review.o.o for that ^ later today13:09
*** e0ne has quit IRC13:11
fungiinfra-root: our doom has arrived (pip 10 now released) https://mail.python.org/pipermail/distutils-sig/2018-April/032175.html13:12
*** yamamoto has quit IRC13:13
*** e0ne has joined #openstack-infra13:13
*** e0ne has quit IRC13:16
* fungi flees in panic (to run saturday morning errands)13:21
*** yamamoto has joined #openstack-infra13:33
*** sree has joined #openstack-infra13:41
*** yamamoto has quit IRC13:45
*** pcichy has joined #openstack-infra13:49
*** yamamoto has joined #openstack-infra13:50
*** yamamoto has quit IRC13:51
*** rfolco|off has joined #openstack-infra14:07
*** sree has quit IRC14:11
*** sree has joined #openstack-infra14:11
*** sree has quit IRC14:16
*** kiennt26 has joined #openstack-infra14:18
*** dhajare has quit IRC14:28
*** agopi_ has joined #openstack-infra14:37
*** yamamoto has joined #openstack-infra14:40
*** claudiub|2 has quit IRC14:56
*** lbragstad has joined #openstack-infra15:05
*** bobh has joined #openstack-infra15:10
*** xarses has quit IRC15:15
*** xarses has joined #openstack-infra15:15
*** slaweq has joined #openstack-infra15:20
*** slaweq has quit IRC15:25
*** bobh has quit IRC15:26
*** kiennt26 has quit IRC15:39
*** dhajare has joined #openstack-infra15:40
*** nicolasbock has joined #openstack-infra15:40
*** rfolco|off has quit IRC15:41
*** lbragstad has quit IRC15:43
*** bobh has joined #openstack-infra15:52
mordredfungi: yup. and it's breaking neutron-grenade15:58
mordredinfra-root: http://logs.openstack.org/34/557934/2/gate/neutron-grenade/716a2e1/logs/grenade.sh.txt.gz#_2018-04-14_15_04_51_75615:59
mordredneutron-grenade just failed for me - which means I'm *pretty* sure it's going to fail for literally everyone in openstack15:59
mordredAJaeger, andreaf ^^16:02
mordredclarkb: I know you've got patches up - I'm not sure what the status of them is or what we should do to move forward16:03
*** fried_rice is now known as efried16:05
*** tosky has quit IRC16:06
clarkbmordred: topic:venv_support iirc. Its kind of a mess. I know I break ironic with it. I think we have to add the feature to every devstack branch first but not enable it. Then enable it on every grenade branch then enable it on every devstacl branch. The problem is nothing works anyways so that sort of coordinated branch series is not going to work16:08
clarkbwe can also attempt to just pin pip short term16:08
*** tosky has joined #openstack-infra16:09
*** jchhatbar has quit IRC16:09
*** lpetrut has joined #openstack-infra16:25
*** AR2018 has joined #openstack-infra16:26
*** Goneri has quit IRC16:30
fungirestarted gerrit on review-dev01.o.o and retested commenting and it still works: https://storyboard-dev.openstack.org/#!/story/176053216:37
fungii guess i can restart gerrit on review.o.o and time now... objections?16:37
fungier, any time now16:39
*** dhajare has quit IRC16:42
*** jtomasek has quit IRC16:45
*** armaan has quit IRC16:46
*** armaan has joined #openstack-infra16:46
fungi#status notice The Gerrit service at https://review.openstack.org/ will be offline for a minute while it is restarted to pick up a configuration change allowing it to start commenting on stories in StoryBoard, and will return to service momentarily16:49
openstackstatusfungi: sending notice16:49
-openstackstatus- NOTICE: The Gerrit service at https://review.openstack.org/ will be offline for a minute while it is restarted to pick up a configuration change allowing it to start commenting on stories in StoryBoard, and will return to service momentarily16:51
*** rfolco|off has joined #openstack-infra16:51
*** AR2018 has quit IRC16:52
openstackstatusfungi: finished sending notice16:52
fungiand it's back up again now... webui loads for me16:53
mnaserspinning slowly for me but i guess its warming up16:53
fungithat seems to take more like 3 minutes in production to start lately16:54
mnaserloading16:55
*** e0ne has joined #openstack-infra17:03
*** lpetrut has quit IRC17:08
*** Goneri has joined #openstack-infra17:09
*** iranzo has quit IRC17:24
*** e0ne has quit IRC17:29
*** r-daneel has quit IRC17:30
*** lpetrut has joined #openstack-infra17:40
*** dhajare has joined #openstack-infra17:40
*** Goneri has quit IRC17:47
*** claudiub|2 has joined #openstack-infra17:52
*** pcichy has quit IRC17:54
*** armaan has quit IRC18:05
*** yamamoto has quit IRC18:05
johnsomUgh, pip 10 is out18:07
johnsomOur DSVM gates do not like it18:07
johnsomhttp://logs.openstack.org/16/559416/7/check/octavia-v1-dsvm-scenario/4f048db/job-output.txt.gz#_2018-04-14_18_04_52_33564118:08
johnsomCannot uninstall 'psutil'.18:08
johnsomLooks like devstack is installing python-psutil for dstat18:11
johnsomhttp://git.openstack.org/cgit/openstack-dev/devstack/tree/files/debs/dstat18:11
*** r-daneel has joined #openstack-infra18:17
*** r-daneel_ has joined #openstack-infra18:20
*** r-daneel has quit IRC18:21
*** r-daneel_ is now known as r-daneel18:22
*** Hunner has quit IRC18:30
*** Hunner has joined #openstack-infra18:31
*** Hunner has quit IRC18:31
*** Hunner has joined #openstack-infra18:31
mnaserthats a common issue i think johnsom18:32
mnaseri'm sure it'll get slowly ironed out18:33
mnaserunfortunately it was released on a weekend18:33
johnsomYeah, I took a guess with a patch for devstack.  It's just unpleasant as every dsvm job is going to be toast until we iron these out.18:33
clarkbthe issue is you can't just fix psutil because then the next system library will be a problem18:35
clarkbpip10 has effectively stated no more global installs18:35
clarkbat least until distros figure out how to stop using distutils18:36
*** bobh has quit IRC18:40
fungiwell, not _all_ distro-packaged libs... just the ones built using distutils (i.e., setup.py install)18:45
*** rfolco|off has quit IRC18:45
fungidistro packages built via setuptools or with another framework providing similar metadata aren't an issue18:46
*** r-daneel has quit IRC18:47
fungibut pypa aren't overly fond of supporting workarounds for it: https://github.com/pypa/pip/pull/506618:49
fungiso i'm not holding out hope that we can expect pip to alter its behavior in this regard unless there's a minor revolt across the python packaging ecosystem18:50
fungiwhich, honestly, would take a week or longer to sort out anyway, so we're better off just sucking it up and finding solutions on our end18:51
*** slaweq has joined #openstack-infra18:56
*** r-daneel has joined #openstack-infra19:00
*** slaweq has quit IRC19:00
*** hongbin has joined #openstack-infra19:05
*** yamamoto has joined #openstack-infra19:06
*** slaweq has joined #openstack-infra19:06
*** yamamoto has quit IRC19:12
*** dhajare has quit IRC19:13
*** dpawlik has joined #openstack-infra19:17
*** slagle has quit IRC19:20
*** krenczew1 has quit IRC19:26
*** krenczewski has joined #openstack-infra19:26
*** toabctl has quit IRC19:29
*** toabctl has joined #openstack-infra19:32
*** dpawlik has quit IRC19:47
pabelanger2018-04-14 19:46:23.614 | [repo-update|http://mirror.dfw.rax.openstack.org/opensuse/update/tumbleweed/] Valid metadata not found at specified URL19:49
pabelangerdirk: clarkb: ^seeing that on nodepool-builder for tumbleweed images19:49
pabelangernot sure why that URL is getting used19:50
dirkpabelanger: argh19:50
dirkpabelanger: we recently changed to build the images with the mirror url set19:51
dirkSo that was also being used for the update repo.. so I guess we need to mirror it19:51
dirkI'll submit a review for that19:51
dirkpabelanger: is it causing the build to fail?19:52
pabelangerdirk: yah, see http://nb01.openstack.org/opensuse-tumbleweed-0000008398.log19:52
pabelangerthey we're looping19:52
pabelangerso, we could revert the mirror change for now, until we figure out how to mirror updates19:52
pabelangeror pause builds19:52
*** nicolasbock has quit IRC19:52
dirkIt's not difficult to fix the mirroring19:53
pabelangerwfm19:53
*** Goneri has joined #openstack-infra19:53
*** tosky has quit IRC19:58
*** armaan has joined #openstack-infra19:58
*** tosky has joined #openstack-infra19:58
openstackgerritDirk Mueller proposed openstack-infra/system-config master: Mirror opensuse emergency update channel  https://review.openstack.org/56143120:02
dirkpabelanger: ^^20:02
*** Goneri has quit IRC20:07
pabelangerdirk: left questions20:08
*** yamamoto has joined #openstack-infra20:08
fungiyou're far more observant than i am20:08
*** r-daneel has quit IRC20:08
*** r-daneel has joined #openstack-infra20:09
*** r-daneel has quit IRC20:14
*** yamamoto has quit IRC20:14
mnaserheads up20:19
mnaserpip-and-virtualenv is broken because of pip 1020:19
mnaserwell sorry, thats inaccurate20:19
mnaserpip-and-virtualenv does pip -U so it upgrades to pip 10 which later breaks things20:20
mnaser:(20:20
*** Goneri has joined #openstack-infra20:22
mnaserlooks like the `svc-map` element installing python-yaml and then building the an image (lbaas amphora in this case) needs it own version of pyyaml which conflicts and causes this20:24
mnaserugh20:24
pabelangermnaser: yah, I suspected we might have issues with that element20:24
mnaserpyyaml in our upper-constraints.txt is 3.12, ubuntu has 3.1120:27
mnaserim gonna work around it by yum remove python-yaml :\20:28
dirkpabelanger: answered20:30
pabelangerdirk: +3, lets give it a try20:35
fungimnaser: if it helps (i'm sure it doesn't!) the upstream answer from pip maintainers is that you should get your distro to fix their packaging, or switch to pip installing into virtualenvs and not into the system context20:45
pabelangeryah, not really user friendly to force the issue. I'd be curious is they (upstream pip maintainers) have offer to help distro packaging fix things20:46
*** dpawlik has joined #openstack-infra20:46
openstackgerritMerged openstack-infra/system-config master: Mirror opensuse emergency update channel  https://review.openstack.org/56143120:48
*** dpawlik has quit IRC20:50
*** pcaruana has quit IRC20:56
*** jogo has quit IRC20:58
*** e0ne has joined #openstack-infra21:08
*** yamamoto has joined #openstack-infra21:10
*** jogo has joined #openstack-infra21:14
*** jogo has joined #openstack-infra21:14
*** yamamoto has quit IRC21:14
*** sambetts_ has quit IRC21:21
*** sambetts_ has joined #openstack-infra21:25
*** e0ne has quit IRC21:35
*** lpetrut has quit IRC21:37
*** wolverineav has joined #openstack-infra21:47
*** dpawlik has joined #openstack-infra21:50
*** slaweq has quit IRC21:50
*** armaan has quit IRC21:56
*** wolverineav has quit IRC22:00
*** wolverineav has joined #openstack-infra22:00
*** Goneri has quit IRC22:02
*** wolverineav has quit IRC22:05
*** yamamoto has joined #openstack-infra22:10
*** jamesmcarthur has joined #openstack-infra22:11
*** yamamoto has quit IRC22:15
hongbinfolks, this patch unblock our gate immediately https://review.openstack.org/#/c/561427/22:16
*** eernst has joined #openstack-infra22:28
fungihongbin: might make more sense to discuss devstack changes in #openstack-qa where the devstack maintainers hang out22:35
*** jamesmcarthur has quit IRC22:36
hongbinsure22:36
clarkbI'm pretty sure that wont pass gremade tests22:41
*** spiffxp has joined #openstack-infra22:54
*** vipul has joined #openstack-infra22:55
*** eernst has quit IRC22:58
*** eernst has joined #openstack-infra22:59
*** yamamoto has joined #openstack-infra23:12
*** yamamoto has quit IRC23:17
pabelangertumbleweed DIBs look to be building again, thanks dirk23:21
*** HenryG_ is now known as HenryG23:28
*** njohnston has joined #openstack-infra23:33
pabelangermtreinish: how do projects manage stestr? With os-testr added that into our DIB images, but don't see that with stestr. I'm trying to see if we can finally stop doing that venv with os-testr23:33
*** HenryG has quit IRC23:34
pabelangereg: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/install.d/91-venv-os-testr23:34
*** HenryG has joined #openstack-infra23:36
*** slaweq has joined #openstack-infra23:51
*** slaweq has quit IRC23:56
*** eernst has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!