| ianw | yeah, was good to learn a bit about that as i followed along. previously i had some spring experience but on tomcat | 00:00 |
|---|---|---|
| *** Swami__ has joined #openstack-infra | 00:00 | |
| *** jcoufal has joined #openstack-infra | 00:01 | |
| *** vaidy has joined #openstack-infra | 00:01 | |
| *** thorst_ has joined #openstack-infra | 00:02 | |
| *** jamielennox_ has joined #openstack-infra | 00:02 | |
| *** thorst_ has quit IRC | 00:02 | |
| ianw | fungi: i mentioned before, but i think it went into a black hole - i wonder if https://docs.openstack.org/user-guide/search.html?q=neutron was broken before the https switch? | 00:03 |
| ianw | just clicking i noticed the release notes have mixed content -> https://review.openstack.org/#/c/433962/ but didn't notice anything else after quick poke | 00:03 |
| *** sflanigan has quit IRC | 00:03 | |
| *** Swami has quit IRC | 00:04 | |
| *** Swami_ has quit IRC | 00:04 | |
| *** pahuang has quit IRC | 00:04 | |
| *** jak-jak has joined #openstack-infra | 00:04 | |
| *** Swami has joined #openstack-infra | 00:04 | |
| *** jamielennox_ is now known as jamielennox | 00:06 | |
| *** hyakuhei has joined #openstack-infra | 00:06 | |
| fungi | are you getting the same "please enable javascript" error i am there then? | 00:07 |
| ianw | yep | 00:07 |
| fungi | my guess is it's something to do with the gcse.src conditional there | 00:07 |
| ianw | Load denied by X-Frame-Options: https://cse.google.com/cse_v2/ads?q=neutron&... blah | 00:07 |
| fungi | oh, i see. so it's using https correctly just not being allowed | 00:08 |
| *** annegentle has quit IRC | 00:09 | |
| ianw | also a 404 for https://docs.openstack.org/user-guide/common/js/doc-characters.json | 00:09 |
| fungi | https://github.com/google/google-api-javascript-client/issues/209 maybe? | 00:10 |
| *** dingyichen has joined #openstack-infra | 00:10 | |
| *** rm_work| is now known as rm_work | 00:11 | |
| *** rm_work has joined #openstack-infra | 00:11 | |
| ianw | hmm, interesting, in chrome, it pops up a search box | 00:12 |
| ianw | not in firefox | 00:13 |
| *** jascott1_ has joined #openstack-infra | 00:13 | |
| *** Goneri has quit IRC | 00:13 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: TEST: Don't override worker counts https://review.openstack.org/433992 | 00:14 |
| *** andreww has quit IRC | 00:16 | |
| *** EricGonc_ has quit IRC | 00:16 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface https://review.openstack.org/385562 | 00:17 |
| fungi | ianw: should probably bring this to AJaeger's attention once he wakes up, and feel free to emergency revert 432334 for now if needed, we can always revert the revert later once this is fixed | 00:18 |
| *** ijw has joined #openstack-infra | 00:18 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Revert "Revert "toci_gate_test: allow to run upgrade jobs for scenarios"" https://review.openstack.org/433715 | 00:19 |
| ianw | fungi: yeah, i'm not 100% sure it's related? and it mostly works with chrome (although you still get that odd no js error). so yeah, just a bug to be handled in due course i'd say | 00:19 |
| *** rbuzatu has quit IRC | 00:20 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Drop ceph workaround for mitaka https://review.openstack.org/425754 | 00:20 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Remove rdo-qemu-ev workaround https://review.openstack.org/425753 | 00:20 |
| *** sflanigan has joined #openstack-infra | 00:21 | |
| fungi | as long as our infra liaison for the docs team is aware, i'm cool | 00:21 |
| fungi | we can treat it as severely as they need | 00:21 |
| *** pahuang has joined #openstack-infra | 00:21 | |
| *** mtanino has quit IRC | 00:23 | |
| *** jcoufal has quit IRC | 00:24 | |
| *** xarses_ has joined #openstack-infra | 00:25 | |
| *** rbuzatu has joined #openstack-infra | 00:25 | |
| *** john-davidge has joined #openstack-infra | 00:28 | |
| *** Julien-zte has joined #openstack-infra | 00:29 | |
| *** rbuzatu has quit IRC | 00:30 | |
| *** xarses_ has quit IRC | 00:31 | |
| *** john-davidge has quit IRC | 00:33 | |
| *** xarses_ has joined #openstack-infra | 00:33 | |
| *** xarses_ has quit IRC | 00:33 | |
| *** xarses_ has joined #openstack-infra | 00:34 | |
| *** dimtruck is now known as zz_dimtruck | 00:36 | |
| *** hongbin has quit IRC | 00:37 | |
| *** EricGonczer_ has joined #openstack-infra | 00:38 | |
| *** EricGonczer_ has quit IRC | 00:38 | |
| *** xarses_ has quit IRC | 00:42 | |
| *** rbuzatu has joined #openstack-infra | 00:43 | |
| *** Guest3904 is now known as medberry | 00:43 | |
| *** medberry has quit IRC | 00:43 | |
| *** medberry has joined #openstack-infra | 00:43 | |
| *** medberry is now known as med_ | 00:43 | |
| *** annegentle has joined #openstack-infra | 00:44 | |
| *** denisra has quit IRC | 00:44 | |
| ianw | clarkb: if you got a minute, wouldn't mind some thoughts on capturing the coverage output for openstack-health as suggested by fungi. | 00:47 |
| ianw | i'm thinking we have to shoe-horn the stats into the subunit output, maybe | 00:47 |
| *** sree has joined #openstack-infra | 00:47 | |
| *** tlian has quit IRC | 00:50 | |
| *** sree has quit IRC | 00:52 | |
| clarkb | ianw: I think that is what would be required if we don't want to change openstack-health at all | 00:52 |
| ianw | yeah, first step i'm just reading up on subunit streams etc to see if we can extract the overall coverage number and insert it | 00:54 |
| clarkb | ianw: we could potentially modify the coverage command that recombines all the sub coverage to emit subunit | 00:54 |
| clarkb | or even just have our own little wrapper around coverage that did it | 00:54 |
| ianw | yeah, i think i'm coming to the latter, to start with anyway | 00:54 |
| ianw | i guess putting coverage in post and tracking there does make sense to get that "moving average" type overview | 00:56 |
| *** tlian has joined #openstack-infra | 00:59 | |
| *** gongysh has joined #openstack-infra | 00:59 | |
| *** tphummel has quit IRC | 01:04 | |
| *** xarses_ has joined #openstack-infra | 01:09 | |
| *** thorst_ has joined #openstack-infra | 01:10 | |
| *** andreww has joined #openstack-infra | 01:10 | |
| *** thorst_ has quit IRC | 01:10 | |
| *** andreww has quit IRC | 01:10 | |
| fungi | that's sort of what i was thinking as well. if we can turn coverage values into subunit somehow... | 01:10 |
| *** andreww has joined #openstack-infra | 01:10 | |
| *** thiagolib has joined #openstack-infra | 01:11 | |
| *** smarcet has joined #openstack-infra | 01:12 | |
| *** xarses_ has quit IRC | 01:13 | |
| *** gongysh has quit IRC | 01:14 | |
| *** shu-mutou-biz has joined #openstack-infra | 01:14 | |
| *** wolverineav has quit IRC | 01:16 | |
| *** Apoorva has quit IRC | 01:17 | |
| *** zz_dimtruck is now known as dimtruck | 01:18 | |
| *** smarcet has quit IRC | 01:19 | |
| *** liujiong has joined #openstack-infra | 01:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 01:24 |
| *** yanyanhu has joined #openstack-infra | 01:24 | |
| *** pvaneck has quit IRC | 01:26 | |
| *** MasterOfBugs has quit IRC | 01:26 | |
| *** sdague has joined #openstack-infra | 01:31 | |
| *** wolverineav has joined #openstack-infra | 01:32 | |
| *** wolverineav has quit IRC | 01:37 | |
| *** sdague has quit IRC | 01:40 | |
| *** annegentle has quit IRC | 01:43 | |
| bkero | fungi: https://www.indiegogo.com/projects/gpd-pocket-7-0-umpc-laptop-ubuntu-or-win-10-os-laptop--2/x/5727346#/ | 01:43 |
| *** thorst_ has joined #openstack-infra | 01:44 | |
| *** andreww has quit IRC | 01:45 | |
| *** sree has joined #openstack-infra | 01:46 | |
| *** rbuzatu has quit IRC | 01:47 | |
| *** rlandy has quit IRC | 01:47 | |
| *** rbuzatu has joined #openstack-infra | 01:48 | |
| *** erikmwilson has joined #openstack-infra | 01:49 | |
| *** sree has quit IRC | 01:51 | |
| *** shu-mutou-biz has quit IRC | 01:51 | |
| *** liujiong has quit IRC | 01:51 | |
| *** liujiong has joined #openstack-infra | 01:51 | |
| *** rbuzatu has quit IRC | 01:52 | |
| *** xarses_ has joined #openstack-infra | 01:56 | |
| *** mdrabe has joined #openstack-infra | 01:57 | |
| *** thorst_ has quit IRC | 02:03 | |
| *** gongysh has joined #openstack-infra | 02:05 | |
| *** baoli has joined #openstack-infra | 02:08 | |
| *** rbuzatu has joined #openstack-infra | 02:08 | |
| *** jascott1_ has quit IRC | 02:11 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 02:11 |
| *** jascott1_ has joined #openstack-infra | 02:12 | |
| mriedem | https://review.openstack.org/#/c/433685/ has been running in tripleo-experimental for nearly 11.5 hours | 02:12 |
| mriedem | seems something is off the rails there | 02:12 |
| *** zara_the_lemur__ has quit IRC | 02:12 | |
| *** rbuzatu has quit IRC | 02:13 | |
| *** wolverineav has joined #openstack-infra | 02:14 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 02:15 |
| *** jascott1_ has quit IRC | 02:16 | |
| *** dimtruck is now known as zz_dimtruck | 02:16 | |
| openstackgerrit | Brian Haley proposed openstack-infra/devstack-gate master: Support an IPv6 underlay network https://review.openstack.org/343041 | 02:18 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 02:18 |
| *** sflanigan has quit IRC | 02:20 | |
| *** thorst_ has joined #openstack-infra | 02:24 | |
| *** thorst_ has quit IRC | 02:24 | |
| *** thorst_ has joined #openstack-infra | 02:25 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/tripleo-ci master: Fix to use correct config options for ssh user https://review.openstack.org/434026 | 02:26 |
| *** thorst_ has quit IRC | 02:29 | |
| *** thorst_ has joined #openstack-infra | 02:31 | |
| *** sflanigan has joined #openstack-infra | 02:31 | |
| *** wolverineav has quit IRC | 02:34 | |
| *** xarses_ has quit IRC | 02:34 | |
| *** zz_dimtruck is now known as dimtruck | 02:34 | |
| *** thorst_ has quit IRC | 02:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo/scenario004: add manila files https://review.openstack.org/434031 | 02:40 |
| *** Sukhdev has quit IRC | 02:40 | |
| *** gouthamr has quit IRC | 02:45 | |
| *** tqtran has quit IRC | 02:45 | |
| *** gouthamr has joined #openstack-infra | 02:46 | |
| *** rbuzatu has joined #openstack-infra | 02:49 | |
| *** baoli has quit IRC | 02:53 | |
| *** rossella_s has quit IRC | 02:53 | |
| *** ramishra has joined #openstack-infra | 02:53 | |
| *** baoli has joined #openstack-infra | 02:53 | |
| *** rossella_s has joined #openstack-infra | 02:53 | |
| *** rbuzatu has quit IRC | 02:54 | |
| *** yuanying has quit IRC | 02:55 | |
| *** john-davidge has joined #openstack-infra | 02:55 | |
| *** john-davidge has quit IRC | 02:59 | |
| *** yamahata has quit IRC | 03:00 | |
| *** markvoelker_ has joined #openstack-infra | 03:02 | |
| *** amotoki has joined #openstack-infra | 03:02 | |
| *** rkukura_ has joined #openstack-infra | 03:03 | |
| *** baoli has quit IRC | 03:03 | |
| *** masayukig_ has joined #openstack-infra | 03:04 | |
| *** ewindisch_ has joined #openstack-infra | 03:04 | |
| *** dhinesh has quit IRC | 03:04 | |
| *** electrical_ has joined #openstack-infra | 03:04 | |
| *** robcresswell_ has joined #openstack-infra | 03:04 | |
| *** boris-42_ has joined #openstack-infra | 03:04 | |
| *** fmccrthy_ has joined #openstack-infra | 03:05 | |
| *** nikhil_ has joined #openstack-infra | 03:05 | |
| *** nikhil_ is now known as Guest16200 | 03:05 | |
| *** ansiwen_ has joined #openstack-infra | 03:05 | |
| *** vdrok_ has joined #openstack-infra | 03:05 | |
| *** mdbooth_ has joined #openstack-infra | 03:05 | |
| *** szaher_ has joined #openstack-infra | 03:05 | |
| *** serverascode_ has joined #openstack-infra | 03:06 | |
| *** jbernard_ has joined #openstack-infra | 03:06 | |
| *** jamespag` has joined #openstack-infra | 03:06 | |
| *** tonyb_ has joined #openstack-infra | 03:06 | |
| *** mnencia_ has joined #openstack-infra | 03:06 | |
| *** slagle_ has joined #openstack-infra | 03:06 | |
| *** dmellado_ has joined #openstack-infra | 03:06 | |
| *** jamespag` has quit IRC | 03:06 | |
| *** jamespag` has joined #openstack-infra | 03:06 | |
| *** bradm_ has joined #openstack-infra | 03:06 | |
| *** njohnston_ has joined #openstack-infra | 03:06 | |
| *** _d34dh0r53_ has joined #openstack-infra | 03:06 | |
| *** Zara_ has joined #openstack-infra | 03:06 | |
| *** adam_g_ has joined #openstack-infra | 03:06 | |
| *** reed_ has joined #openstack-infra | 03:06 | |
| *** clayg_ has joined #openstack-infra | 03:06 | |
| *** admcleod_ has joined #openstack-infra | 03:06 | |
| *** nibz has joined #openstack-infra | 03:06 | |
| *** coreycb` has joined #openstack-infra | 03:06 | |
| *** Gorian- has joined #openstack-infra | 03:06 | |
| *** rosmaita_ has joined #openstack-infra | 03:07 | |
| *** cinerama` has joined #openstack-infra | 03:07 | |
| *** ramishra_ has joined #openstack-infra | 03:07 | |
| *** arif-ali_ has joined #openstack-infra | 03:07 | |
| *** mugsie|a1t has joined #openstack-infra | 03:07 | |
| *** phealy has quit IRC | 03:07 | |
| *** docaedo_ has joined #openstack-infra | 03:07 | |
| *** Hunner_ has joined #openstack-infra | 03:07 | |
| *** Hunner_ has quit IRC | 03:07 | |
| *** Hunner_ has joined #openstack-infra | 03:07 | |
| *** phealy has joined #openstack-infra | 03:07 | |
| *** njohnston has quit IRC | 03:07 | |
| *** nikhil has quit IRC | 03:07 | |
| *** vdrok has quit IRC | 03:07 | |
| *** mnencia has quit IRC | 03:07 | |
| *** yonglihe has quit IRC | 03:07 | |
| *** fdegir has quit IRC | 03:07 | |
| *** szaher has quit IRC | 03:07 | |
| *** dmellado has quit IRC | 03:07 | |
| *** arif-ali has quit IRC | 03:07 | |
| *** serverascode has quit IRC | 03:07 | |
| *** jbernard has quit IRC | 03:07 | |
| *** ewindisch has quit IRC | 03:07 | |
| *** fmccrthy has quit IRC | 03:07 | |
| *** electrical has quit IRC | 03:07 | |
| *** admcleod has quit IRC | 03:07 | |
| *** markvoelker has quit IRC | 03:07 | |
| *** boris-42 has quit IRC | 03:07 | |
| *** jamespage has quit IRC | 03:07 | |
| *** calebb has quit IRC | 03:07 | |
| *** smcginnis has quit IRC | 03:07 | |
| *** Zara has quit IRC | 03:07 | |
| *** slagle has quit IRC | 03:07 | |
| *** verdurin has quit IRC | 03:07 | |
| *** reed has quit IRC | 03:07 | |
| *** ramishra has quit IRC | 03:07 | |
| *** knangia has quit IRC | 03:07 | |
| *** tonyb has quit IRC | 03:07 | |
| *** rkukura has quit IRC | 03:07 | |
| *** adam_g has quit IRC | 03:07 | |
| *** Cibo has quit IRC | 03:07 | |
| *** apetrich has quit IRC | 03:07 | |
| *** rosmaita has quit IRC | 03:07 | |
| *** ansiwen has quit IRC | 03:07 | |
| *** bswartz has quit IRC | 03:07 | |
| *** coreycb has quit IRC | 03:07 | |
| *** robcresswell has quit IRC | 03:07 | |
| *** cinerama has quit IRC | 03:07 | |
| *** clayg has quit IRC | 03:07 | |
| *** nibalizer has quit IRC | 03:07 | |
| *** docaedo has quit IRC | 03:07 | |
| *** bradm has quit IRC | 03:07 | |
| *** masayukig has quit IRC | 03:07 | |
| *** gcb has quit IRC | 03:07 | |
| *** mdbooth has quit IRC | 03:07 | |
| *** Hunner has quit IRC | 03:07 | |
| *** lbragstad has quit IRC | 03:07 | |
| *** Gorian has quit IRC | 03:07 | |
| *** shadower has quit IRC | 03:07 | |
| *** d34dh0r53 has quit IRC | 03:07 | |
| *** pcrews has quit IRC | 03:07 | |
| *** mugsie|alt has quit IRC | 03:07 | |
| *** mnencia_ is now known as mnencia | 03:07 | |
| *** rkukura_ is now known as rkukura | 03:07 | |
| *** adam_g_ is now known as adam_g | 03:07 | |
| *** arif-ali_ is now known as arif-ali | 03:07 | |
| *** smcginnis_ has joined #openstack-infra | 03:07 | |
| *** smcginnis_ is now known as smcginnis | 03:07 | |
| *** gcb has joined #openstack-infra | 03:08 | |
| *** reed_ is now known as reed | 03:08 | |
| *** ijw has quit IRC | 03:08 | |
| *** apetrich has joined #openstack-infra | 03:08 | |
| *** lbragstad has joined #openstack-infra | 03:08 | |
| *** verdurin has joined #openstack-infra | 03:08 | |
| *** ewindisch_ is now known as ewindisch | 03:08 | |
| *** fmccrthy_ is now known as fmccrthy | 03:08 | |
| *** calebb has joined #openstack-infra | 03:08 | |
| *** pcrews_ has joined #openstack-infra | 03:08 | |
| *** Cibo has joined #openstack-infra | 03:08 | |
| *** vdrok_ is now known as vdrok | 03:09 | |
| *** robcresswell_ is now known as robcresswell | 03:09 | |
| *** yonglihe has joined #openstack-infra | 03:09 | |
| *** electrical_ is now known as electrical | 03:09 | |
| *** Guest16200 is now known as nikhil | 03:09 | |
| *** thorst_ has joined #openstack-infra | 03:10 | |
| *** rbuzatu has joined #openstack-infra | 03:11 | |
| *** fdegir has joined #openstack-infra | 03:11 | |
| *** thorst_ has quit IRC | 03:11 | |
| *** gildub has joined #openstack-infra | 03:13 | |
| *** jascott1_ has joined #openstack-infra | 03:13 | |
| *** dchen has joined #openstack-infra | 03:15 | |
| *** isq has quit IRC | 03:15 | |
| *** rbuzatu has quit IRC | 03:16 | |
| *** jascott1_ has quit IRC | 03:18 | |
| *** rosmaita_ is now known as rosmaita | 03:18 | |
| *** serverascode_ is now known as serverascode | 03:18 | |
| *** amotoki has quit IRC | 03:19 | |
| *** jaosorior has joined #openstack-infra | 03:19 | |
| *** tonyb_ is now known as tonyb | 03:21 | |
| *** masayukig_ is now known as masayukig | 03:21 | |
| *** ihrachys has joined #openstack-infra | 03:23 | |
| *** isq has joined #openstack-infra | 03:29 | |
| *** wolverineav has joined #openstack-infra | 03:31 | |
| *** wolverineav has quit IRC | 03:35 | |
| *** sflanigan has quit IRC | 03:35 | |
| *** thorst_ has joined #openstack-infra | 03:37 | |
| *** thorst_ has quit IRC | 03:37 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 03:41 |
| *** Jeffrey4l_ has quit IRC | 03:43 | |
| *** Jeffrey4l_ has joined #openstack-infra | 03:43 | |
| *** wolverineav has joined #openstack-infra | 03:43 | |
| *** gildub has quit IRC | 03:44 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job https://review.openstack.org/427481 | 03:45 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 03:45 |
| *** ihrachys has quit IRC | 03:45 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 03:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 03:46 |
| *** wolverineav has quit IRC | 03:47 | |
| *** rbuzatu has joined #openstack-infra | 03:48 | |
| *** sflanigan has joined #openstack-infra | 03:48 | |
| *** armax has quit IRC | 03:52 | |
| *** rbuzatu has quit IRC | 03:52 | |
| *** cshastri has joined #openstack-infra | 03:53 | |
| *** ihrachys has joined #openstack-infra | 03:54 | |
| *** sree has joined #openstack-infra | 03:55 | |
| *** docaedo_ is now known as docaedo | 03:59 | |
| *** yonglihe has quit IRC | 03:59 | |
| *** ihrachys has quit IRC | 04:01 | |
| *** gouthamr has quit IRC | 04:01 | |
| *** nmathew has joined #openstack-infra | 04:02 | |
| *** nicolasbock has quit IRC | 04:06 | |
| *** yuanying has joined #openstack-infra | 04:08 | |
| *** rbuzatu has joined #openstack-infra | 04:09 | |
| *** YorikSar has quit IRC | 04:11 | |
| *** thiagolib has quit IRC | 04:11 | |
| *** YorikSar has joined #openstack-infra | 04:12 | |
| *** rbuzatu has quit IRC | 04:13 | |
| *** dhinesh has joined #openstack-infra | 04:14 | |
| *** amotoki has joined #openstack-infra | 04:22 | |
| *** sbezverk has left #openstack-infra | 04:30 | |
| *** rbuzatu has joined #openstack-infra | 04:30 | |
| *** YorikSar has quit IRC | 04:30 | |
| *** yamahata has joined #openstack-infra | 04:31 | |
| *** YorikSar has joined #openstack-infra | 04:32 | |
| *** dimtruck is now known as zz_dimtruck | 04:32 | |
| *** links has joined #openstack-infra | 04:34 | |
| *** rbuzatu has quit IRC | 04:34 | |
| *** tphummel has joined #openstack-infra | 04:34 | |
| *** thorst_ has joined #openstack-infra | 04:38 | |
| *** jascott1_ has joined #openstack-infra | 04:43 | |
| *** thorst_ has quit IRC | 04:43 | |
| *** jascott1_ has quit IRC | 04:48 | |
| *** dhinesh has quit IRC | 04:50 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 04:56 | |
| *** masayukig is now known as masayukig_ | 04:58 | |
| *** Sukhdev has joined #openstack-infra | 04:58 | |
| *** wolverineav has joined #openstack-infra | 05:00 | |
| *** wolverineav has quit IRC | 05:05 | |
| *** adriant has quit IRC | 05:07 | |
| *** rbuzatu has joined #openstack-infra | 05:11 | |
| *** Sukhdev has quit IRC | 05:12 | |
| *** mdrabe has quit IRC | 05:13 | |
| *** tphummel has quit IRC | 05:13 | |
| *** rbuzatu has quit IRC | 05:16 | |
| *** wolverineav has joined #openstack-infra | 05:16 | |
| *** knangia has joined #openstack-infra | 05:18 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/pbr master: Remove testr entry point https://review.openstack.org/422379 | 05:22 |
| *** john-davidge has joined #openstack-infra | 05:22 | |
| *** nadya has joined #openstack-infra | 05:23 | |
| *** zz_dimtruck is now known as dimtruck | 05:23 | |
| *** wolverineav has quit IRC | 05:24 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/pbr master: Also generate XML coverage report https://review.openstack.org/434071 | 05:24 |
| *** nadya has quit IRC | 05:24 | |
| ianw | lifeless: ^^ i think that changelog now better describes what the heck is going on with testr/test and PBR | 05:25 |
| *** amotoki has quit IRC | 05:26 | |
| *** wolverineav has joined #openstack-infra | 05:26 | |
| *** john-davidge has quit IRC | 05:27 | |
| *** faizy has joined #openstack-infra | 05:28 | |
| *** wolverineav has quit IRC | 05:30 | |
| *** dimtruck is now known as zz_dimtruck | 05:34 | |
| jianghuaw | hi, is the zookeeper service mandatory for nodepoo-0.3.1? | 05:35 |
| *** amotoki has joined #openstack-infra | 05:36 | |
| *** sandanar has joined #openstack-infra | 05:36 | |
| *** sflanigan has quit IRC | 05:40 | |
| *** bradm_ is now known as bradm | 05:40 | |
| *** dingyichen has quit IRC | 05:40 | |
| *** tqtran has joined #openstack-infra | 05:44 | |
| *** sree has quit IRC | 05:45 | |
| *** sree has joined #openstack-infra | 05:46 | |
| *** tqtran has quit IRC | 05:49 | |
| *** wolverineav has joined #openstack-infra | 05:49 | |
| *** vsaienko has quit IRC | 05:49 | |
| *** rbuzatu has joined #openstack-infra | 05:50 | |
| *** sflanigan has joined #openstack-infra | 05:52 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/426863 | 05:53 |
| *** amotoki has quit IRC | 05:53 | |
| *** dingyichen has joined #openstack-infra | 05:53 | |
| *** wolverineav has quit IRC | 05:53 | |
| *** krtaylor has joined #openstack-infra | 05:55 | |
| *** gyee has quit IRC | 05:58 | |
| *** ramishra_ has quit IRC | 05:58 | |
| *** ramishra has joined #openstack-infra | 06:00 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job with RabbitMQ https://review.openstack.org/426307 | 06:00 |
| *** hamzy has quit IRC | 06:01 | |
| *** amotoki has joined #openstack-infra | 06:02 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job with RabbitMQ https://review.openstack.org/426307 | 06:04 |
| *** nadya has joined #openstack-infra | 06:06 | |
| *** amotoki has quit IRC | 06:10 | |
| *** salv-orlando has joined #openstack-infra | 06:10 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Enable coverage jobs for monasca projects https://review.openstack.org/432845 | 06:11 |
| *** pgadiya has joined #openstack-infra | 06:13 | |
| *** rbuzatu has quit IRC | 06:13 | |
| *** sree has quit IRC | 06:14 | |
| *** pcaruana has joined #openstack-infra | 06:20 | |
| *** amotoki has joined #openstack-infra | 06:21 | |
| *** krtaylor has quit IRC | 06:24 | |
| *** hamzy has joined #openstack-infra | 06:24 | |
| *** claudiub has joined #openstack-infra | 06:30 | |
| *** Jeffrey4l__ has joined #openstack-infra | 06:30 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix missing mutex release when aborting builds https://review.openstack.org/432211 | 06:31 |
| *** psachin has joined #openstack-infra | 06:32 | |
| *** Jeffrey4l_ has quit IRC | 06:34 | |
| *** e0ne has joined #openstack-infra | 06:36 | |
| *** amotoki_ has joined #openstack-infra | 06:37 | |
| *** thorst_ has joined #openstack-infra | 06:39 | |
| *** krtaylor has joined #openstack-infra | 06:40 | |
| *** amotoki has quit IRC | 06:40 | |
| *** pahuang has quit IRC | 06:42 | |
| *** eliqiao has quit IRC | 06:42 | |
| *** thorst_ has quit IRC | 06:43 | |
| *** eliqiao has joined #openstack-infra | 06:46 | |
| *** aviau has quit IRC | 06:47 | |
| *** aviau has joined #openstack-infra | 06:48 | |
| *** skraynev has quit IRC | 06:51 | |
| *** rossella_s has quit IRC | 06:53 | |
| *** Swami__ has quit IRC | 06:53 | |
| *** Swami has quit IRC | 06:53 | |
| *** rossella_s has joined #openstack-infra | 06:53 | |
| *** nadya has quit IRC | 06:54 | |
| *** jascott1_ has joined #openstack-infra | 06:55 | |
| *** pahuang has joined #openstack-infra | 06:55 | |
| *** admcleod_ has quit IRC | 06:57 | |
| *** jascott1_ has quit IRC | 07:00 | |
| *** salv-orl_ has joined #openstack-infra | 07:01 | |
| *** salv-orlando has quit IRC | 07:03 | |
| *** anilvenkata has joined #openstack-infra | 07:04 | |
| *** rcernin has joined #openstack-infra | 07:07 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add release notes jobs for blazar https://review.openstack.org/431274 | 07:07 |
| *** skraynev has joined #openstack-infra | 07:08 | |
| AJaeger | ianw, fungi: Searching is somehow broken - and was before. NO need for a revert. Let me send an email to the list. And check the json file... | 07:09 |
| AJaeger | mmh, https://docs.openstack.org/user-guide/_static/js/ shows doc-characters-json | 07:11 |
| AJaeger | Why common/js ? | 07:11 |
| ianw | AJaeger: cool, i thought so, didn't seem ssl related | 07:11 |
| *** kaisers has joined #openstack-infra | 07:12 | |
| *** psachin has quit IRC | 07:13 | |
| *** rbuzatu has joined #openstack-infra | 07:14 | |
| ianw | AJaeger: also, i've got a bit of a plan to make coverage graphs ... i'm thinking something like getting pbr to generate coverage xml reports (https://review.openstack.org/#/c/434071/) and then a "fake" test to get that into subunit output. | 07:15 |
| ianw | that should get results into subunit2sql at which point openstack-health could query them | 07:15 |
| ianw | if that was run from post jobs, you'd get a moving view of code coverage | 07:15 |
| ianw | that's the theory, anyway, i'll see how i go with it | 07:15 |
| *** zz_dimtruck is now known as dimtruck | 07:17 | |
| AJaeger | ianw: interesting idea, look forward to it. | 07:20 |
| *** jascott1_ has joined #openstack-infra | 07:20 | |
| *** markus_z has joined #openstack-infra | 07:23 | |
| *** isaacb has joined #openstack-infra | 07:25 | |
| *** jtomasek has joined #openstack-infra | 07:27 | |
| *** monester has joined #openstack-infra | 07:27 | |
| *** jtomasek_ has joined #openstack-infra | 07:27 | |
| *** dimtruck is now known as zz_dimtruck | 07:27 | |
| *** andreas_s has joined #openstack-infra | 07:28 | |
| *** e0ne has quit IRC | 07:29 | |
| *** knangia has quit IRC | 07:30 | |
| *** jascott1_ has quit IRC | 07:31 | |
| *** jascott1_ has joined #openstack-infra | 07:31 | |
| *** tesseract has joined #openstack-infra | 07:31 | |
| *** jtomasek has quit IRC | 07:32 | |
| *** jascott1_ has quit IRC | 07:35 | |
| *** yamamoto has quit IRC | 07:38 | |
| *** saibarspeis has joined #openstack-infra | 07:39 | |
| *** jtomasek_ is now known as jtomasek | 07:45 | |
| *** tqtran has joined #openstack-infra | 07:46 | |
| *** rbuzatu has quit IRC | 07:48 | |
| yolanda | good morning | 07:49 |
| *** tqtran has quit IRC | 07:50 | |
| *** isaacb has quit IRC | 07:51 | |
| *** rbuzatu has joined #openstack-infra | 07:52 | |
| *** rbuzatu has quit IRC | 07:56 | |
| *** yuval has joined #openstack-infra | 07:57 | |
| *** anilvenkata has quit IRC | 07:59 | |
| *** isaacb has joined #openstack-infra | 08:00 | |
| *** monester has quit IRC | 08:01 | |
| *** salv-orl_ has quit IRC | 08:10 | |
| *** Hal has joined #openstack-infra | 08:10 | |
| *** salv-orlando has joined #openstack-infra | 08:10 | |
| *** nmathew has quit IRC | 08:10 | |
| *** amotoki_ has quit IRC | 08:14 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-openstackci master: Pass Java arguments to Jenkins master https://review.openstack.org/263021 | 08:14 |
| *** sree has joined #openstack-infra | 08:15 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/434130 | 08:15 |
| *** anilvenkata has joined #openstack-infra | 08:15 | |
| *** zz_dimtruck is now known as dimtruck | 08:18 | |
| *** sree has quit IRC | 08:19 | |
| *** aviau has quit IRC | 08:19 | |
| *** aviau has joined #openstack-infra | 08:19 | |
| *** ralonsoh has joined #openstack-infra | 08:20 | |
| *** ccamacho|Zzz is now known as ccamacho | 08:21 | |
| *** amotoki has joined #openstack-infra | 08:24 | |
| *** shardy has joined #openstack-infra | 08:24 | |
| *** martinkopec has joined #openstack-infra | 08:25 | |
| *** nadya has joined #openstack-infra | 08:27 | |
| *** nadya has quit IRC | 08:27 | |
| *** nadya has joined #openstack-infra | 08:27 | |
| *** dimtruck is now known as zz_dimtruck | 08:28 | |
| *** nmathew has joined #openstack-infra | 08:32 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues https://review.openstack.org/425459 | 08:32 |
| *** sshnaidm|off is now known as sshnaidm | 08:32 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues https://review.openstack.org/425459 | 08:34 |
| *** tuanluong has joined #openstack-infra | 08:35 | |
| *** esikachev has joined #openstack-infra | 08:36 | |
| *** links has quit IRC | 08:36 | |
| *** matrohon has joined #openstack-infra | 08:37 | |
| *** csomerville has quit IRC | 08:38 | |
| *** e0ne has joined #openstack-infra | 08:38 | |
| *** cody-somerville has joined #openstack-infra | 08:38 | |
| *** thorst_ has joined #openstack-infra | 08:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/434130 | 08:41 |
| *** dingyichen has quit IRC | 08:42 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/426863 | 08:42 |
| *** dchen has quit IRC | 08:42 | |
| *** yonglihe has joined #openstack-infra | 08:42 | |
| *** matrohon has quit IRC | 08:43 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Compress logs to tar.gz archive https://review.openstack.org/434143 | 08:43 |
| *** thorst_ has quit IRC | 08:44 | |
| *** yuanying has quit IRC | 08:47 | |
| *** yuanying has joined #openstack-infra | 08:47 | |
| *** yuanying has quit IRC | 08:47 | |
| *** jpich has joined #openstack-infra | 08:48 | |
| *** priteau has joined #openstack-infra | 08:49 | |
| *** matrohon has joined #openstack-infra | 08:52 | |
| *** links has joined #openstack-infra | 08:53 | |
| *** jascott1_ has joined #openstack-infra | 08:54 | |
| *** admcleod has joined #openstack-infra | 08:58 | |
| *** jascott1_ has quit IRC | 08:59 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** zzzeek has joined #openstack-infra | 09:00 | |
| *** amotoki has quit IRC | 09:05 | |
| *** radez has quit IRC | 09:05 | |
| *** hashar has joined #openstack-infra | 09:08 | |
| *** lucas-afk is now known as lucasagomes | 09:10 | |
| *** panda|zZ is now known as panda | 09:11 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 09:13 |
| openstackgerrit | Merged openstack-infra/project-config master: telemetry: run tripleo/scenario001 jobs as -nv https://review.openstack.org/433734 | 09:14 |
| *** cmurphy has quit IRC | 09:16 | |
| *** cmurphy has joined #openstack-infra | 09:16 | |
| *** bhavik1 has joined #openstack-infra | 09:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Puppet OpenStack beakers jobs on Ubuntu Trusty https://review.openstack.org/433775 | 09:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Install Guide Jobs to Designate https://review.openstack.org/341671 | 09:18 |
| *** zz_dimtruck is now known as dimtruck | 09:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: mistral: switch tripleo job to scenario003 https://review.openstack.org/433754 | 09:20 |
| *** rbuzatu has joined #openstack-infra | 09:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move log-api tempest gate to voting stage https://review.openstack.org/433018 | 09:21 |
| *** psachin has joined #openstack-infra | 09:21 | |
| *** yamamoto has joined #openstack-infra | 09:23 | |
| *** salv-orlando has quit IRC | 09:23 | |
| *** mhickey has joined #openstack-infra | 09:24 | |
| *** yamamoto_ has joined #openstack-infra | 09:24 | |
| *** e0ne has quit IRC | 09:24 | |
| *** aarefiev_afk is now known as aarefiev | 09:25 | |
| *** askb has quit IRC | 09:25 | |
| *** yamamoto_ has quit IRC | 09:25 | |
| *** yamamoto_ has joined #openstack-infra | 09:26 | |
| *** yamamoto_ has quit IRC | 09:26 | |
| *** links has quit IRC | 09:26 | |
| *** yuanying has joined #openstack-infra | 09:26 | |
| *** gildub has joined #openstack-infra | 09:27 | |
| *** yamamoto has quit IRC | 09:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Don't run nova dvsm tests for .git repo changes https://review.openstack.org/366933 | 09:28 |
| *** dimtruck is now known as zz_dimtruck | 09:29 | |
| *** gcb has quit IRC | 09:31 | |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** ssbarnea has joined #openstack-infra | 09:33 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:33 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 09:35 |
| *** nmathew has quit IRC | 09:39 | |
| *** nmathew has joined #openstack-infra | 09:39 | |
| *** rfolco has joined #openstack-infra | 09:40 | |
| *** e0ne has joined #openstack-infra | 09:40 | |
| *** links has joined #openstack-infra | 09:43 | |
| *** Julien-zte has quit IRC | 09:50 | |
| *** wolverineav has joined #openstack-infra | 09:50 | |
| *** wolverin_ has joined #openstack-infra | 09:51 | |
| *** wolverineav has quit IRC | 09:51 | |
| *** Zara_ is now known as Zara | 09:51 | |
| *** jordanP has joined #openstack-infra | 09:54 | |
| *** salv-orlando has joined #openstack-infra | 09:54 | |
| *** nadya has quit IRC | 09:59 | |
| *** anilvenkata has quit IRC | 10:01 | |
| *** john-davidge has joined #openstack-infra | 10:01 | |
| *** hyakuhei has quit IRC | 10:01 | |
| *** hyakuhei has joined #openstack-infra | 10:01 | |
| *** hyakuhei has quit IRC | 10:01 | |
| *** hyakuhei has joined #openstack-infra | 10:01 | |
| *** rzetikx has joined #openstack-infra | 10:03 | |
| rzetikx | Hi, I would like to ask you what kind of statistics graph you are using for zuul status. I just know Grafana at the moment. Thanks | 10:06 |
| openstackgerrit | Jeremy Liu proposed openstack-infra/project-config master: Setup translation job for cloudkitty-dashboard https://review.openstack.org/434192 | 10:08 |
| *** dtantsur is now known as dtantsur|bbl | 10:10 | |
| *** rbuzatu has quit IRC | 10:14 | |
| *** isaacb has quit IRC | 10:14 | |
| *** tuanluong has quit IRC | 10:14 | |
| *** isaacb has joined #openstack-infra | 10:15 | |
| *** rbuzatu has joined #openstack-infra | 10:15 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Move OSA upgrade job from experimental queue to periodic https://review.openstack.org/419517 | 10:18 |
| odyssey4me | AJaeger yolanda ^ For that patch, what I'm failing to see is where I define the branches and times that the periodic job should execute? | 10:19 |
| odyssey4me | It looks like the timings are defined in the pipeline spec? | 10:19 |
| *** mptacekx has joined #openstack-infra | 10:19 | |
| *** sambetts|afk is now known as sambetts | 10:19 | |
| *** denisra has joined #openstack-infra | 10:20 | |
| *** zz_dimtruck is now known as dimtruck | 10:20 | |
| *** yamahata has quit IRC | 10:20 | |
| *** liujiong has quit IRC | 10:21 | |
| *** isaacb has quit IRC | 10:25 | |
| *** denisra_ has joined #openstack-infra | 10:26 | |
| *** ociuhandu has joined #openstack-infra | 10:28 | |
| *** denisra has quit IRC | 10:30 | |
| *** dimtruck is now known as zz_dimtruck | 10:30 | |
| *** salv-orlando has quit IRC | 10:31 | |
| *** bhavik1 has quit IRC | 10:33 | |
| *** yanyanhu has quit IRC | 10:34 | |
| *** nadya has joined #openstack-infra | 10:37 | |
| *** thorst_ has joined #openstack-infra | 10:41 | |
| *** ldnunes has joined #openstack-infra | 10:41 | |
| *** nadya has quit IRC | 10:42 | |
| *** gongysh has quit IRC | 10:42 | |
| *** tosky has joined #openstack-infra | 10:43 | |
| *** wolverin_ has quit IRC | 10:43 | |
| *** wolverineav has joined #openstack-infra | 10:44 | |
| *** thorst_ has quit IRC | 10:46 | |
| *** masayukig_ is now known as masayukig | 10:49 | |
| *** masayukig is now known as masayukig_ | 10:50 | |
| *** nadya has joined #openstack-infra | 10:50 | |
| *** masayukig_ is now known as masayukig | 10:50 | |
| *** ssbarnea has quit IRC | 10:52 | |
| *** wolverineav has quit IRC | 10:52 | |
| *** rossella_s has quit IRC | 10:53 | |
| *** rossella_s has joined #openstack-infra | 10:53 | |
| *** martinkopec has quit IRC | 10:55 | |
| *** ssbarnea has joined #openstack-infra | 10:56 | |
| *** nmathew has quit IRC | 10:56 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 10:56 |
| *** rbuzatu has quit IRC | 10:57 | |
| *** martinkopec has joined #openstack-infra | 10:58 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 11:00 |
| *** nicolasbock has joined #openstack-infra | 11:02 | |
| *** dtantsur|bbl is now known as dtantsur | 11:05 | |
| AJaeger | odyssey4me: see how other periodic jobs are defined, You need to set the branch explicitely. Sorry, no time today to give exact instructions | 11:10 |
| odyssey4me | I'll try to puzzle it out again later, thanks AJaeger | 11:11 |
| *** salv-orlando has joined #openstack-infra | 11:13 | |
| AJaeger | odyssey4me: https://review.openstack.org/#/c/426308/1/jenkins/jobs/projects.yaml - see the branch-override there as one example. | 11:15 |
| AJaeger | bbl | 11:15 |
| odyssey4me | awesome, thanks AJaeger | 11:15 |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config master: Notify #openstack-dns of stable/ocata changes https://review.openstack.org/434221 | 11:16 |
| *** masayukig is now known as masayukig_ | 11:20 | |
| *** zz_dimtruck is now known as dimtruck | 11:22 | |
| *** Jeffrey4l__ is now known as Jeffrey4l | 11:22 | |
| *** isaacb has joined #openstack-infra | 11:22 | |
| *** wolverineav has joined #openstack-infra | 11:23 | |
| *** smarcet has joined #openstack-infra | 11:25 | |
| *** wolverineav has quit IRC | 11:28 | |
| *** dimtruck is now known as zz_dimtruck | 11:31 | |
| *** martinkopec has quit IRC | 11:32 | |
| *** martinkopec has joined #openstack-infra | 11:32 | |
| *** sdague has joined #openstack-infra | 11:34 | |
| *** jkilpatr has quit IRC | 11:35 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/434232 | 11:35 |
| *** ldnunes has quit IRC | 11:36 | |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| *** rcrit has quit IRC | 11:38 | |
| *** ociuhandu has quit IRC | 11:42 | |
| AJaeger | odyssey4me: but that only works with devstack-gate jobs | 11:45 |
| AJaeger | odyssey4me: for others, see in project-config e.g. jenkins/jobs/python-bitrot-jobs.yaml | 11:46 |
| AJaeger | the zuul-git-branch-prep-upper-constraints is the magic - use branch: stable/X to select the stable/X branch | 11:47 |
| *** salv-orlando has quit IRC | 11:48 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job https://review.openstack.org/427481 | 11:49 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 11:50 |
| *** yamamoto has joined #openstack-infra | 11:53 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/elastic-recheck master: Add query for OOM issue on RAX 1664953 https://review.openstack.org/434238 | 11:53 |
| *** jkilpatr has joined #openstack-infra | 11:53 | |
| *** mdorman has quit IRC | 11:53 | |
| *** mdorman has joined #openstack-infra | 11:55 | |
| *** ociuhandu has joined #openstack-infra | 11:55 | |
| *** thiagolib has joined #openstack-infra | 11:57 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues https://review.openstack.org/425459 | 12:00 |
| *** rbuzatu has joined #openstack-infra | 12:00 | |
| *** gildub has quit IRC | 12:01 | |
| *** raildo has joined #openstack-infra | 12:02 | |
| *** lucasagomes is now known as lucas-hungry | 12:04 | |
| *** sree has joined #openstack-infra | 12:04 | |
| *** wolverineav has joined #openstack-infra | 12:06 | |
| *** ldnunes has quit IRC | 12:07 | |
| *** rbuzatu has quit IRC | 12:08 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/project-config master: tripleo-ci move multinode-upgrades-nv to check queue https://review.openstack.org/434247 | 12:09 |
| *** wolverineav has quit IRC | 12:10 | |
| *** isaacb has quit IRC | 12:11 | |
| *** salv-orlando has joined #openstack-infra | 12:11 | |
| *** Mmike has left #openstack-infra | 12:12 | |
| *** masayukig_ is now known as masayukig | 12:19 | |
| *** rbuzatu has joined #openstack-infra | 12:19 | |
| *** ldnunes has joined #openstack-infra | 12:21 | |
| *** zz_dimtruck is now known as dimtruck | 12:21 | |
| *** sree has quit IRC | 12:24 | |
| *** hashar has quit IRC | 12:25 | |
| *** Qiming has quit IRC | 12:28 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci master: Add converge update to major upgrade test https://review.openstack.org/434250 | 12:29 |
| *** Qiming has joined #openstack-infra | 12:30 | |
| *** dimtruck is now known as zz_dimtruck | 12:31 | |
| *** ricardoas has joined #openstack-infra | 12:34 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/434232 | 12:35 |
| *** thorst_ has joined #openstack-infra | 12:38 | |
| *** faizy has quit IRC | 12:39 | |
| *** Julien-zte has joined #openstack-infra | 12:42 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** panda is now known as panda|lunch | 12:44 | |
| *** wolverineav has joined #openstack-infra | 12:45 | |
| *** Julien-zte has quit IRC | 12:47 | |
| *** Julien-z_ has joined #openstack-infra | 12:47 | |
| *** ociuhandu has quit IRC | 12:48 | |
| *** wolverineav has quit IRC | 12:49 | |
| *** yamamoto has joined #openstack-infra | 12:50 | |
| *** yamamoto has quit IRC | 12:51 | |
| *** coreycb` is now known as coreycb | 12:51 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues https://review.openstack.org/425459 | 12:52 |
| *** coreycb has quit IRC | 12:52 | |
| *** eliqiao has quit IRC | 12:52 | |
| *** coreycb has joined #openstack-infra | 12:53 | |
| *** fandi has joined #openstack-infra | 12:53 | |
| *** eliqiao has joined #openstack-infra | 12:55 | |
| *** smarcet has quit IRC | 12:58 | |
| *** hashar has joined #openstack-infra | 12:58 | |
| *** EricGonczer_ has joined #openstack-infra | 13:01 | |
| *** isaacb has joined #openstack-infra | 13:02 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove TODO comment that seems to be done https://review.openstack.org/433961 | 13:03 |
| *** nadya has quit IRC | 13:03 | |
| *** trown|outtypewww is now known as trown | 13:04 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_footer_message https://review.openstack.org/430486 | 13:04 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Drop test_node_label https://review.openstack.org/430473 | 13:04 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_client_enqueue_ref test https://review.openstack.org/393887 | 13:05 |
| *** slagle_ is now known as slagle | 13:06 | |
| *** kzaitsev_mb has quit IRC | 13:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:07 | |
| *** rbuzatu has quit IRC | 13:08 | |
| *** lucas-hungry is now known as lucasagomes | 13:09 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 13:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 13:10 |
| *** amotoki has joined #openstack-infra | 13:10 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 13:13 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 13:13 |
| *** gouthamr has joined #openstack-infra | 13:14 | |
| *** eantyshev has quit IRC | 13:14 | |
| *** pgadiya has quit IRC | 13:14 | |
| *** faizy has joined #openstack-infra | 13:15 | |
| *** wolverineav has joined #openstack-infra | 13:15 | |
| *** jcoufal has joined #openstack-infra | 13:18 | |
| *** wolverineav has quit IRC | 13:20 | |
| *** zz_dimtruck is now known as dimtruck | 13:22 | |
| *** eharney has joined #openstack-infra | 13:26 | |
| *** panda|lunch is now known as panda | 13:27 | |
| *** jheroux has joined #openstack-infra | 13:28 | |
| *** rbuzatu has joined #openstack-infra | 13:28 | |
| *** eantyshev has joined #openstack-infra | 13:28 | |
| *** gordc has joined #openstack-infra | 13:29 | |
| *** dimtruck is now known as zz_dimtruck | 13:32 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job https://review.openstack.org/431649 | 13:32 |
| *** salv-orlando has quit IRC | 13:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Wait for volumes to detach before deleting them https://review.openstack.org/432770 | 13:35 |
| *** EricGonczer_ has quit IRC | 13:35 | |
| *** bswartz has joined #openstack-infra | 13:39 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job https://review.openstack.org/431649 | 13:39 |
| *** rlandy has joined #openstack-infra | 13:40 | |
| *** wolverineav has joined #openstack-infra | 13:43 | |
| *** fandi has quit IRC | 13:45 | |
| *** rhallisey has joined #openstack-infra | 13:46 | |
| *** fguillot has joined #openstack-infra | 13:46 | |
| *** wolverineav has quit IRC | 13:47 | |
| *** ociuhandu has joined #openstack-infra | 13:51 | |
| *** rhallisey has quit IRC | 13:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Move make_local_copy to utils https://review.openstack.org/433848 | 13:53 |
| *** tongli has joined #openstack-infra | 13:54 | |
| pabelanger | jianghuaw: yes, zookeeper is a hard dependency for nodepool and zuulv3 now | 13:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job https://review.openstack.org/431649 | 13:56 |
| *** nadya has joined #openstack-infra | 13:57 | |
| *** wolverineav has joined #openstack-infra | 14:03 | |
| *** ihrachys has joined #openstack-infra | 14:04 | |
| *** isaacb has quit IRC | 14:05 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Set Node image_id and launcher attributes https://review.openstack.org/433242 | 14:05 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add helper scripts to print version discovery info https://review.openstack.org/431720 | 14:05 |
| *** isaacb has joined #openstack-infra | 14:06 | |
| *** cshastri has quit IRC | 14:07 | |
| *** Goneri has joined #openstack-infra | 14:08 | |
| *** rhallisey has joined #openstack-infra | 14:08 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add generator API method for node iteration https://review.openstack.org/433252 | 14:08 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Disconnect from ZooKeeper at shutdown https://review.openstack.org/433919 | 14:08 |
| *** wolverineav has quit IRC | 14:08 | |
| *** trown is now known as trown|brb | 14:08 | |
| *** mdrabe has joined #openstack-infra | 14:09 | |
| *** kgiusti has joined #openstack-infra | 14:10 | |
| *** jcoufal_ has joined #openstack-infra | 14:10 | |
| *** jcoufal has quit IRC | 14:11 | |
| jianghuaw | pabelanger, thanks. Could you suggest the reference doc for zookeeper setup? | 14:12 |
| *** jcoufal has joined #openstack-infra | 14:12 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix missing mutex release when aborting builds https://review.openstack.org/432211 | 14:13 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove subnodes from nodepool https://review.openstack.org/432403 | 14:13 |
| *** wolverineav has joined #openstack-infra | 14:13 | |
| *** jcoufal_ has quit IRC | 14:14 | |
| *** matt-borland has joined #openstack-infra | 14:15 | |
| *** jamesdenton has joined #openstack-infra | 14:15 | |
| pabelanger | jianghuaw: we are currently using v0.5.5 of https://github.com/deric/puppet-zookeeper to configure zookeeper | 14:16 |
| *** esikachev has quit IRC | 14:17 | |
| *** wolverineav has quit IRC | 14:17 | |
| *** salv-orlando has joined #openstack-infra | 14:18 | |
| *** salv-orlando has quit IRC | 14:18 | |
| *** salv-orlando has joined #openstack-infra | 14:18 | |
| jianghuaw | pabelanger, Beside of zookeeper, is there any update with zmq-publishers? | 14:19 |
| jianghuaw | I redeployed a CI env which was setup one year ago. But after the redeployment, it seems the zmq-publisher not running well. | 14:19 |
| jianghuaw | the default port 8888 is not listened actually. | 14:19 |
| openstackgerrit | Merged openstack-infra/shade master: Add accessor method to pull URLs from the catalog https://review.openstack.org/432358 | 14:20 |
| pabelanger | mriedem: | 14:21 |
| pabelanger | mriedem: sorry | 14:21 |
| pabelanger | jianghuaw: next version of nodepool will remove zmq-publishers, it will be 100% zookeeper moving forward | 14:21 |
| openstackgerrit | Paul Carver proposed openstack-dev/hacking master: Disambiguate the hacking rules for third party imports https://review.openstack.org/434306 | 14:22 |
| *** jamesdenton has quit IRC | 14:22 | |
| *** jsheeren has joined #openstack-infra | 14:22 | |
| *** zz_dimtruck is now known as dimtruck | 14:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job https://review.openstack.org/431649 | 14:24 |
| jianghuaw | pabelanger, ah. The information is important. thanks. | 14:24 |
| *** trown|brb is now known as trown | 14:24 | |
| *** jamesdenton has joined #openstack-infra | 14:25 | |
| jianghuaw | I've installed nodepool 0.3.1 in my env, could it fully work if I configure nodepool with zookeeper by without zmq-publishers? | 14:26 |
| *** john-davidge has quit IRC | 14:26 | |
| *** john-davidge has joined #openstack-infra | 14:27 | |
| *** dprince has joined #openstack-infra | 14:27 | |
| *** dmellado_ is now known as dmellado | 14:27 | |
| pabelanger | jianghuaw: no, 0.3.1 will still need zmq-publishers, out next releases (I don't know the version number ATM) will drop support. | 14:29 |
| pabelanger | jianghuaw: it is the feature/zuulv3 branch in git, if you want to review the code | 14:29 |
| Shrews | jianghuaw: nodepool-builder is the only piece that uses ZooKeeper in 0.3.1. nodepoold still needs zmq | 14:29 |
| pabelanger | jianghuaw: we also have #zuul now, a dedicated channel for development, if you want to idle there too | 14:29 |
| *** Krenair has quit IRC | 14:30 | |
| *** baoli has joined #openstack-infra | 14:30 | |
| *** links has quit IRC | 14:31 | |
| *** mriedem has quit IRC | 14:31 | |
| jianghuaw | pabelanger and Shrews: Ok. So the test node scheduling for jobs still depends on zmq? | 14:32 |
| *** Julien-z_ has quit IRC | 14:32 | |
| *** Julien-zte has joined #openstack-infra | 14:32 | |
| *** dimtruck is now known as zz_dimtruck | 14:33 | |
| *** wolverineav has joined #openstack-infra | 14:33 | |
| AJaeger | pabelanger: could you review https://review.openstack.org/#/c/432436/ , please? dependencies merged for fuel-web python-db | 14:33 |
| pabelanger | jianghuaw: right | 14:33 |
| Shrews | yeah. and i guess technically nodepoold still needs zookeeper to get at image info, too | 14:33 |
| Shrews | but zmq and gearman are both going away in the next version | 14:34 |
| mordred | and mysql | 14:35 |
| jianghuaw | Shrews, does the nest version mean 0.4 or the following new version? | 14:35 |
| jianghuaw | s/nest/next/ | 14:35 |
| Shrews | jianghuaw: not sure | 14:35 |
| *** Krenair has joined #openstack-infra | 14:37 | |
| *** chlong has quit IRC | 14:37 | |
| pabelanger | I thought jeblair was saying 1.0... but not 100% | 14:37 |
| *** wolverineav has quit IRC | 14:37 | |
| jianghuaw | Shrews and pabelanger, Thanks for your replying and explanation. | 14:39 |
| *** tongli has quit IRC | 14:40 | |
| jianghuaw | Shrews and pabelanger, Could you suggest any reference doc(if any) for me to make nodepool-0.3.1 working in my env easier? :-) | 14:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add reporter for Federated Message Bus (fedmsg) https://review.openstack.org/426861 | 14:43 |
| Shrews | jianghuaw: oh, 0.4.0 is the zookeeper release. see http://lists.openstack.org/pipermail/openstack-infra/2017-January/005037.html | 14:43 |
| openstackgerrit | Kyrylo Shatskyy proposed openstack-infra/jenkins-job-builder master: Add support for 'Groovy Label Assignment Plugin' https://review.openstack.org/428419 | 14:43 |
| dhellmann | did we have a zuul or gerrit blip earlier today? I have a patch that is approved and has run its gate jobs but that is not merged. It has no dependencies and doesn't seem to be tied to any other commits. https://review.openstack.org/#/c/433767/ | 14:43 |
| *** wolverineav has joined #openstack-infra | 14:44 | |
| dhellmann | last update about 40 minutes ago | 14:44 |
| dhellmann | fungi : ^^ | 14:44 |
| *** Swami has joined #openstack-infra | 14:44 | |
| jianghuaw | Shrews, cool. thanks. | 14:45 |
| andreykurilin | hi folks! Can anyone explain me how I can add here custom dashboards https://review.openstack.org/#/admin/projects/openstack/rally,dashboards ? | 14:48 |
| andreykurilin | It is possible? | 14:48 |
| *** ociuhandu has quit IRC | 14:51 | |
| *** jcoufal_ has joined #openstack-infra | 14:51 | |
| *** esikachev has joined #openstack-infra | 14:52 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job https://review.openstack.org/431649 | 14:52 |
| *** rossella_s has quit IRC | 14:53 | |
| *** hongbin has joined #openstack-infra | 14:53 | |
| pabelanger | dhellmann: it looks fine now? It is possible zuul was still processing events | 14:53 |
| *** tphummel has joined #openstack-infra | 14:53 | |
| *** rossella_s has joined #openstack-infra | 14:54 | |
| *** jcoufal has quit IRC | 14:54 | |
| dhellmann | pabelanger : we've approved others since then and they merged, so I was worried about an issue. I added another W+1 to it and it is regating, which seems like an odd response since there was already a +2 from Jenkins, but maybe that's normal. | 14:56 |
| pabelanger | dhellmann: Ya, it is possible there might have been a network blip. I can go look into the logs and see what happened | 14:57 |
| dhellmann | pabelanger : cool, thanks. Don't put a ton of time into it, I guess unless it's part of a bigger pattern. | 14:57 |
| * dhellmann steps afk for a bit | 14:57 | |
| pabelanger | ack | 14:57 |
| *** wolverin_ has joined #openstack-infra | 14:58 | |
| *** isaacb has quit IRC | 14:59 | |
| *** mriedem has joined #openstack-infra | 14:59 | |
| *** adrian_otto has joined #openstack-infra | 15:00 | |
| openstackgerrit | Jeremy Liu proposed openstack-infra/project-config master: Setup translation job for cloudkitty-dashboard https://review.openstack.org/434192 | 15:00 |
| *** wolverineav has quit IRC | 15:01 | |
| *** zz_dimtruck is now known as dimtruck | 15:03 | |
| *** rbrndt has joined #openstack-infra | 15:04 | |
| *** jamesdenton has quit IRC | 15:05 | |
| AJaeger | pabelanger, mordred : could you review https://review.openstack.org/#/c/432436/ , please? dependencies merged for fuel-web python-db change | 15:06 |
| *** jamesdenton has joined #openstack-infra | 15:07 | |
| fungi | pabelanger: dhellmann: yeah, i hope we've got something in the main zuul log from around 14:06 indicating why it didn't manage to merge it in gerrit and yet also didn't leave a comment saying so | 15:07 |
| *** jaosorior has quit IRC | 15:09 | |
| fungi | aha, raised an exception. i'll put the tb on paste.o.o in a sec | 15:10 |
| *** jaosorior has joined #openstack-infra | 15:10 | |
| *** eroux has quit IRC | 15:10 | |
| *** isaacb has joined #openstack-infra | 15:10 | |
| *** lejit has quit IRC | 15:13 | |
| fungi | dhellmann: pabelanger: jeblair: http://paste.openstack.org/show/599050/ coupled with the fact that the succeeded comment and verify +2 appeared on https://review.openstack.org/433767 at 14:06 suggests that zuul hit a 2-minute timeout waiting on a response from gerrit's api... do you concur? | 15:15 |
| fungi | oh, and ignore that stray logline about the ironic job. it's unrelated but i missed trimming it from my paste | 15:16 |
| *** jamesden_ has joined #openstack-infra | 15:16 | |
| *** mtanino has joined #openstack-infra | 15:16 | |
| fungi | i'm digging in the gerrit error log now to see if i can spot any relevant needles in the haystack at that end | 15:17 |
| *** jamesdenton has quit IRC | 15:18 | |
| *** EricGonczer_ has joined #openstack-infra | 15:18 | |
| *** armax has joined #openstack-infra | 15:18 | |
| *** burgerk has joined #openstack-infra | 15:19 | |
| *** eroux has joined #openstack-infra | 15:19 | |
| fungi | ick, looks like db communication issues? http://paste.openstack.org/show/599052/ | 15:21 |
| pabelanger | fungi: thanks for looking, got pulled away | 15:22 |
| mordred | fungi: wasn't there some email about something trove something something for rackspace? | 15:22 |
| mordred | did we maybe just hit that? | 15:22 |
| fungi | so looks more like zuul did probably get an error back, and the two-minute timeout was probably on gerrit's end trying to perform a database lookup | 15:22 |
| fungi | mordred: that's the next place i was going to look ;) | 15:22 |
| *** saibarspeis has quit IRC | 15:22 | |
| *** andymaier has joined #openstack-infra | 15:23 | |
| *** claudiub|2 has joined #openstack-infra | 15:23 | |
| *** radez has joined #openstack-infra | 15:23 | |
| *** jsheeren has quit IRC | 15:24 | |
| *** saibarspeis has joined #openstack-infra | 15:24 | |
| fungi | no alerts or maintenance for https://status.rackspace.com/ and no trove-related trouble tickets for that tenant. pending maintenances we're being alerted to are host maintenance which will impact elasticsearch02 on the 19th and block storage maintenance impacting static and wiki-dev on the 28th | 15:25 |
| *** claudiub has quit IRC | 15:26 | |
| fungi | no unusual spikes on the cacti graphs for review.o.o around that timeframe either | 15:27 |
| *** adrian_otto has quit IRC | 15:27 | |
| *** adrian_otto has joined #openstack-infra | 15:28 | |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=34&rra_id=all shows a little bit of an inbound network traffic spike on the trove-facing interface, so maybe that was it | 15:28 |
| *** nadya has quit IRC | 15:28 | |
| fungi | but it's barely a third the size of the spikes we get at db backup time | 15:29 |
| fungi | i guess it could have been particularly unlucky timing | 15:29 |
| *** mriedem1 has joined #openstack-infra | 15:31 | |
| dhellmann | fungi, pabelanger : it seems likely that it was bad timing; I'll let you know if it shows up again | 15:31 |
| fungi | thanks, dhellmann | 15:31 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add support for overriding mistral service type https://review.openstack.org/434344 | 15:32 |
| *** mriedem has quit IRC | 15:33 | |
| fungi | probably the best we could do for future-proofing is to either properly report back an error to gerrit and/or check the resulting state and retry outstanding operations. but this seems infrequent enough that can probably wait for zuul v3 (and zuul pushing its own merge commits probably makes this even less likely still) | 15:33 |
| *** yolanda has quit IRC | 15:34 | |
| *** nadya has joined #openstack-infra | 15:35 | |
| jeblair | fungi: well, the 'report back an error to gerrit' is tricky if gerrit can't update its db... | 15:35 |
| *** yolanda has joined #openstack-infra | 15:35 | |
| fungi | jeblair: agreed | 15:36 |
| fungi | the catch-22 there dawned on me right after i said it | 15:36 |
| *** mriedem1 is now known as mriedem | 15:37 | |
| jeblair | i think maybe a 'try 3 times then give up' algorithm would be okay -- but part of the intent with the current code is not to let gerrit get zuul completely stuck. | 15:37 |
| fungi | apparently in this case the db problem didn't prevent zuul from commenting/voting so a followup error comment likely would have shown up | 15:37 |
| jeblair | ah yeah, then even better for the try-3-times approach | 15:38 |
| fungi | though not necessarily the case of course, and i can certainly envision situations where whatever caused the action to fail would have also prevented followup comments | 15:38 |
| fungi | but yes, a likely improvement at least | 15:39 |
| jeblair | yep, like "this change is broken in the gerrit db" | 15:39 |
| mordred | "this change is cut in half" | 15:40 |
| jeblair | mordred: did you cut this change in half? | 15:40 |
| fungi | eric the half a change | 15:41 |
| mordred | jeblair: this change is cut in half. I'd like to exchange it for one that is not ... cut in half | 15:41 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job https://review.openstack.org/431649 | 15:44 |
| *** mriedem has quit IRC | 15:52 | |
| *** amotoki has quit IRC | 15:55 | |
| *** sree has joined #openstack-infra | 15:56 | |
| *** sree has quit IRC | 15:56 | |
| *** abregman has joined #openstack-infra | 15:56 | |
| *** sree has joined #openstack-infra | 15:56 | |
| *** mriedem has joined #openstack-infra | 15:56 | |
| *** abregman has quit IRC | 15:56 | |
| *** Julien-zte has quit IRC | 15:57 | |
| *** abregman has joined #openstack-infra | 15:58 | |
| *** knangia has joined #openstack-infra | 15:58 | |
| *** jascott1_ has joined #openstack-infra | 15:58 | |
| *** psachin has quit IRC | 15:58 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup https://review.openstack.org/433736 | 16:00 |
| *** vhosakot has joined #openstack-infra | 16:00 | |
| *** xarses_ has joined #openstack-infra | 16:00 | |
| *** kzaitsev_mb has quit IRC | 16:01 | |
| *** marst has joined #openstack-infra | 16:01 | |
| *** yuval has quit IRC | 16:01 | |
| *** andreww has joined #openstack-infra | 16:02 | |
| *** chlong has joined #openstack-infra | 16:04 | |
| *** ccamacho has left #openstack-infra | 16:04 | |
| *** nadya has quit IRC | 16:05 | |
| *** xarses_ has quit IRC | 16:06 | |
| *** xarses_ has joined #openstack-infra | 16:06 | |
| *** flepied has quit IRC | 16:06 | |
| *** andreww has quit IRC | 16:06 | |
| *** andreww has joined #openstack-infra | 16:07 | |
| *** nadya has joined #openstack-infra | 16:07 | |
| *** yamahata has joined #openstack-infra | 16:07 | |
| *** thorst_ has quit IRC | 16:08 | |
| *** ricardoas has quit IRC | 16:08 | |
| *** ihrachys has quit IRC | 16:08 | |
| *** thorst_ has joined #openstack-infra | 16:09 | |
| *** isaacb has quit IRC | 16:09 | |
| *** morgan_ is now known as morgan | 16:10 | |
| *** xarses_ has quit IRC | 16:10 | |
| *** xarses_ has joined #openstack-infra | 16:11 | |
| *** xarses_ has quit IRC | 16:11 | |
| *** andreww has quit IRC | 16:12 | |
| *** xarses_ has joined #openstack-infra | 16:12 | |
| *** erhudy has joined #openstack-infra | 16:14 | |
| *** erikmwilson has quit IRC | 16:15 | |
| *** nadya has quit IRC | 16:15 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job https://review.openstack.org/431649 | 16:15 |
| *** EricGonczer_ has quit IRC | 16:15 | |
| *** erikmwilson has joined #openstack-infra | 16:15 | |
| *** aarefiev is now known as aarefiev_afk | 16:15 | |
| *** andreww has joined #openstack-infra | 16:16 | |
| *** xarses_ has quit IRC | 16:20 | |
| *** nadya has joined #openstack-infra | 16:20 | |
| *** mptacekx has quit IRC | 16:21 | |
| *** flepied has joined #openstack-infra | 16:22 | |
| openstackgerrit | Béla Vancsics proposed openstack-dev/hacking master: Add a check to make sure the right assert* method is used https://review.openstack.org/434366 | 16:22 |
| *** cinerama` is now known as cinerama | 16:23 | |
| *** saibarspeis has quit IRC | 16:24 | |
| *** rossella_s has quit IRC | 16:25 | |
| *** rzetikx has quit IRC | 16:25 | |
| *** EricGonczer_ has joined #openstack-infra | 16:25 | |
| AJaeger | fungi, pabelanger, mordred : could you put https://review.openstack.org/#/c/432436/ on your review queue, please? dependencies merged for fuel-web python-db change... | 16:27 |
| *** jcoufal has joined #openstack-infra | 16:28 | |
| pabelanger | done | 16:28 |
| AJaeger | thanks | 16:28 |
| *** faizy has quit IRC | 16:28 | |
| *** baoli has quit IRC | 16:30 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Remove the keystoneclient auth fallback https://review.openstack.org/434372 | 16:30 |
| *** baoli has joined #openstack-infra | 16:31 | |
| *** jcoufal_ has quit IRC | 16:31 | |
| *** jcoufal__ has joined #openstack-infra | 16:31 | |
| *** ihrachys has joined #openstack-infra | 16:32 | |
| *** jcoufal has quit IRC | 16:32 | |
| *** jcoufal_ has joined #openstack-infra | 16:33 | |
| *** annegentle has joined #openstack-infra | 16:33 | |
| *** vhosakot has quit IRC | 16:33 | |
| *** martinkopec has quit IRC | 16:34 | |
| *** jcoufal__ has quit IRC | 16:36 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 16:36 |
| *** sandanar has quit IRC | 16:36 | |
| *** annegentle has quit IRC | 16:37 | |
| *** annegentle has joined #openstack-infra | 16:38 | |
| *** thorst_ has quit IRC | 16:38 | |
| *** adrian_otto has quit IRC | 16:39 | |
| *** mdrabe has quit IRC | 16:39 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci master: Add converge update to major upgrade test https://review.openstack.org/434250 | 16:39 |
| mriedem | ianw: did you find out if nodepool is running devstack on trusty? https://review.openstack.org/#/c/433274/ | 16:39 |
| fungi | mriedem: as discussed in our meeting yesterday, it's xenial | 16:40 |
| fungi | (also ianw is probably sound asleep by now) | 16:40 |
| mriedem | ok good to know | 16:41 |
| fungi | the risk to us is that we're running nodepool on trusty servers and doing integration testing of changes to it on xenial so there's the potential for us to break production, but the solution to that is to upgrade our prod servers | 16:41 |
| *** sree has quit IRC | 16:41 | |
| clarkb | which is straightforward for the riskiest servers (image builders) and planned for post ptg now | 16:42 |
| *** jcoufal has joined #openstack-infra | 16:42 | |
| *** terryw is now known as otherwiseguy | 16:42 | |
| *** sree has joined #openstack-infra | 16:42 | |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate https://review.openstack.org/434384 | 16:42 |
| *** sree has quit IRC | 16:43 | |
| *** sree has joined #openstack-infra | 16:43 | |
| *** rcernin has quit IRC | 16:43 | |
| *** masayukig is now known as masayukig_ | 16:45 | |
| *** masayukig_ is now known as masayukig | 16:45 | |
| clarkb | pabelanger et al, are there any other items that I can help with for ptg prep? I got most of my TODOs sorted yesterday with translate-dev upgraded and elasticsearch green again. So let me know if I can help somewhere | 16:45 |
| *** jcoufal_ has quit IRC | 16:45 | |
| clarkb | otherwise will likely spend time prepping the how to debug test failures thing | 16:45 |
| *** jaypipes has joined #openstack-infra | 16:45 | |
| *** tosky has quit IRC | 16:46 | |
| pabelanger | clarkb: I was planning on starting nl01.o.o today, if you want to look at zuulv3-dev.o.o, I am okay with that | 16:49 |
| *** jamesden_ has quit IRC | 16:49 | |
| *** tosky has joined #openstack-infra | 16:49 | |
| *** hieulq_ has joined #openstack-infra | 16:49 | |
| pabelanger | clarkb: I was thinking of just standing up a plan server, so we can then iterate on zuulv3 patches as we land them | 16:49 |
| *** abregman has quit IRC | 16:49 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job https://review.openstack.org/427481 | 16:49 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin) https://review.openstack.org/427482 | 16:49 |
| *** thorst_ has joined #openstack-infra | 16:49 | |
| clarkb | pabelanger: I reiviewed the zuulv3-dev patch too I found one issue with the ansible grouping | 16:49 |
| clarkb | pabelanger: but that was yesterday so maybe I need to go look at new patches? | 16:49 |
| pabelanger | clarkb: ya, I haven't uploaded anything new yet | 16:50 |
| clarkb | its the ansible grouping thing that always trips me up so now I know to look for it :) | 16:51 |
| *** kfarr has joined #openstack-infra | 16:52 | |
| *** andymaier has quit IRC | 16:52 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add IRC notifications from Gerrit for #craton https://review.openstack.org/434391 | 16:53 |
| *** tqtran has joined #openstack-infra | 16:55 | |
| sigmavirus | Hey, so #craton has a tox env that we run separately from the standard `py35` environment because it contains slower functional tests. Is there a good way to update our check/gate jobs to run that as well? | 16:56 |
| * sigmavirus suspects we have to define a custom job for that to work, but was hoping that wasn't the case | 16:56 | |
| AJaeger | sigmavirus: there are many examples in project-config on how to do that ;) | 16:57 |
| *** mdrabe has joined #openstack-infra | 16:57 | |
| AJaeger | sigmavirus: search for gate-{name}-tox-{envlist}-{node} in jenkins/jobs/projects.yaml | 16:57 |
| AJaeger | envlist is your environment | 16:57 |
| sigmavirus | ah | 16:58 |
| AJaeger | sigmavirus: that template will run "tox -e envlist" for you | 16:58 |
| sigmavirus | I was looking in the wrong place | 16:58 |
| sigmavirus | Thanks AJaeger ! | 16:58 |
| *** tesseract has quit IRC | 16:58 | |
| *** jamesdenton has joined #openstack-infra | 16:58 | |
| AJaeger | sigmavirus: and then add it to your repo in zuul/layout.yaml as well to trigger when to run | 16:58 |
| *** jascott1_ has quit IRC | 16:58 | |
| *** adrian_otto has joined #openstack-infra | 16:59 | |
| *** jascott1_ has joined #openstack-infra | 16:59 | |
| *** ramishra has quit IRC | 17:01 | |
| *** jascott1_ has quit IRC | 17:03 | |
| *** trown is now known as trown|lunch | 17:04 | |
| *** jordanP has quit IRC | 17:04 | |
| *** erikmwilson has quit IRC | 17:05 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add lock state to node listing https://review.openstack.org/434397 | 17:05 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 17:06 |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| annegentle | Huh. I coulda sworn there was a Launchpad for logging bugs against the www.openstack site. Where'd it go? | 17:06 |
| AJaeger | annegentle: https://bugs.launchpad.net/openstack-org ? | 17:07 |
| clarkb | annegentle: openstack-org | 17:07 |
| annegentle | there it is! | 17:07 |
| annegentle | LOL thanks to you both | 17:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Produce some debug details about nova usage objects https://review.openstack.org/434401 | 17:08 |
| *** erikmwilson has joined #openstack-infra | 17:08 | |
| *** faizy has joined #openstack-infra | 17:09 | |
| *** tqtran has quit IRC | 17:11 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: grafana: track new rally py35 job for neutron https://review.openstack.org/434402 | 17:11 |
| *** e0ne has quit IRC | 17:11 | |
| *** andreas_s has quit IRC | 17:11 | |
| clarkb | AJaeger: wins with the proper url :) | 17:12 |
| AJaeger | clarkb: this is not about winning ;) | 17:13 |
| annegentle | snerk | 17:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_diskimage_build_only test https://review.openstack.org/433265 | 17:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_dib_upload_fail test https://review.openstack.org/433235 | 17:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_image_upload_fail test https://review.openstack.org/433270 | 17:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable working test_builder.py tests https://review.openstack.org/433262 | 17:14 |
| annegentle | Bug logged: https://bugs.launchpad.net/openstack-org/+bug/1665041 for a summit voting login problem | 17:14 |
| openstack | Launchpad bug 1665041 in openstack-org "Vote for speakers link login seems broken" [Undecided,New] | 17:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Try to capture novaclient debug logging https://review.openstack.org/434404 | 17:14 |
| *** d0ugal has quit IRC | 17:16 | |
| *** beekneemech is now known as bnemec | 17:16 | |
| *** csomerville has joined #openstack-infra | 17:21 | |
| *** priteau has quit IRC | 17:21 | |
| *** markus_z has quit IRC | 17:21 | |
| *** EricGonczer_ has quit IRC | 17:22 | |
| *** jamesdenton has quit IRC | 17:22 | |
| *** baoli has quit IRC | 17:22 | |
| *** jbernard_ is now known as jbernard | 17:23 | |
| *** jascott1_ has joined #openstack-infra | 17:24 | |
| *** jascott1_ has quit IRC | 17:24 | |
| *** chlong has quit IRC | 17:24 | |
| *** cody-somerville has quit IRC | 17:24 | |
| *** Apoorva has joined #openstack-infra | 17:25 | |
| *** annegentle has quit IRC | 17:25 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 17:25 |
| *** EricGonczer_ has joined #openstack-infra | 17:28 | |
| *** d0ugal has joined #openstack-infra | 17:30 | |
| *** d0ugal has quit IRC | 17:30 | |
| *** d0ugal has joined #openstack-infra | 17:30 | |
| *** sabayonuser has joined #openstack-infra | 17:30 | |
| *** thiagolib has quit IRC | 17:31 | |
| *** ralonsoh has quit IRC | 17:31 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job https://review.openstack.org/431777 | 17:33 |
| *** ZZelle_ has joined #openstack-infra | 17:33 | |
| *** jaosorior has quit IRC | 17:35 | |
| *** annegentle has joined #openstack-infra | 17:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Force nova microversion to 2.0 https://review.openstack.org/434416 | 17:36 |
| *** lucasagomes is now known as lucas-afk | 17:38 | |
| *** mriedem has quit IRC | 17:40 | |
| *** jpich has quit IRC | 17:40 | |
| *** baoli has joined #openstack-infra | 17:41 | |
| *** d0ugal has quit IRC | 17:42 | |
| *** faizy has quit IRC | 17:45 | |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate https://review.openstack.org/434384 | 17:47 |
| kfarr | sdague ping! I have a question about setting a nova conf value in the gate | 17:47 |
| kfarr | You had a few comments in https://review.openstack.org/#/c/383293/ about using post-config | 17:48 |
| *** rossella_s has joined #openstack-infra | 17:48 | |
| kfarr | and I think I configured it correctly here, but was hoping you could take a look: https://review.openstack.org/#/c/434384/ | 17:48 |
| *** mhickey has quit IRC | 17:51 | |
| *** yamahata has quit IRC | 17:51 | |
| *** hashar has quit IRC | 17:52 | |
| pabelanger | jeblair: do you mind +3 430329 and 430339 | 17:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add image list to debug output on failure https://review.openstack.org/434427 | 17:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there https://review.openstack.org/434428 | 17:52 |
| *** vhosakot has joined #openstack-infra | 17:53 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 17:54 |
| *** rossella_s has quit IRC | 17:54 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: Put neutron config at the top of the local.conf file https://review.openstack.org/434430 | 17:55 |
| *** vsaienko has joined #openstack-infra | 17:56 | |
| *** jcoufal has quit IRC | 17:59 | |
| *** EricGonczer_ has quit IRC | 17:59 | |
| *** tphummel has quit IRC | 18:01 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck master: Add query for bug 1665053 https://review.openstack.org/434436 | 18:01 |
| openstack | bug 1665053 in tempest "test_stamp_pattern times out waiting for the device to show up on the guest" [Undecided,New] https://launchpad.net/bugs/1665053 | 18:01 |
| harlowja | zxiiro hi there, do u mind looking over https://review.openstack.org/#/c/403940/ | 18:01 |
| harlowja | it'd be nice to merge that so that at godaddy we can use it :) | 18:02 |
| harlowja | i don't want to go to the UI anymore to make this stuff, lol | 18:02 |
| *** hieulq_ has quit IRC | 18:02 | |
| *** ihrachys_ has joined #openstack-infra | 18:02 | |
| *** ihrachys has quit IRC | 18:02 | |
| *** panda is now known as panda|bbl | 18:04 | |
| *** sambetts is now known as sambetts|afk | 18:04 | |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job https://review.openstack.org/434439 | 18:04 |
| *** tonytan4ever has joined #openstack-infra | 18:05 | |
| *** Goneri has quit IRC | 18:09 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci master: Remove test-environments/multinode_major_upgrade.yaml https://review.openstack.org/426375 | 18:12 |
| *** tqtran has joined #openstack-infra | 18:13 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck master: Fix spaces in 2 queries https://review.openstack.org/434445 | 18:13 |
| *** jamielennox is now known as jamielennox|away | 18:13 | |
| zxiiro | harlowja: hey, I'm currently traveling at a conference. If you cc me on it, it will go into my mail queue which I'll get to as soon as I'm back from travel | 18:13 |
| harlowja | kk | 18:14 |
| jeblair | pabelanger: +3 | 18:14 |
| *** kfarr has quit IRC | 18:14 | |
| *** Goneri has joined #openstack-infra | 18:14 | |
| *** esikachev has quit IRC | 18:15 | |
| pabelanger | danke | 18:15 |
| Shrews | c 2 | 18:15 |
| Shrews | ugh | 18:16 |
| pabelanger | ug ho | 18:16 |
| pabelanger | logs.o.o is full again | 18:16 |
| *** spzala has joined #openstack-infra | 18:16 | |
| pabelanger | /dev/mapper/main-logs 12682583968 12682567584 0 100% /srv/static/logs | 18:16 |
| * fungi sighs | 18:17 | |
| *** Swami_ has joined #openstack-infra | 18:17 | |
| bkero | again? | 18:17 |
| *** tphummel has joined #openstack-infra | 18:17 | |
| clarkb | last time was due to many gigabyte job logs | 18:17 |
| *** tqtran has quit IRC | 18:17 | |
| *** adrian_otto has quit IRC | 18:17 | |
| clarkb | we probably need to do an audit again and see if we have that going on | 18:17 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=717&rra_id=4&view_type=&graph_start=1454129464&graph_end=1487182648 | 18:17 |
| fungi | current retention is 60 days | 18:18 |
| *** flepied has quit IRC | 18:18 | |
| *** esikachev has joined #openstack-infra | 18:18 | |
| fungi | we generate 12tb of test logs (compressed) in two months | 18:18 |
| fungi | shall i shrink that to 45 days for now to get us some breathing room so people can see what's being logged too excessively? | 18:19 |
| jeblair | fungi: i like that plan | 18:19 |
| pabelanger | ++ | 18:19 |
| mtreinish | fungi: that's a lot of logs | 18:20 |
| *** Swami has quit IRC | 18:20 | |
| jeblair | (also, when we write the zuul v3 log copying playbook, maybe we should see about logging the size of the logs we copy somewhere, so we can more easily track down large jobs in the future) | 18:20 |
| pabelanger | Ya, that is a good idea | 18:20 |
| mtreinish | fungi: you might want to put that on the ML too | 18:20 |
| fungi | there's a log_archive_maintenance.sh script which has been running for the past 4 days | 18:21 |
| pabelanger | stuck? | 18:21 |
| jeblair | Process 20918 attached | 18:21 |
| jeblair | wait4(-1, | 18:21 |
| fungi | strace -p 25422 (the child process) looks active | 18:22 |
| * fungi waits for his terminal to catch up with the sigint he sent | 18:22 | |
| *** esikachev has quit IRC | 18:22 | |
| *** jkilpatr has quit IRC | 18:22 | |
| jeblair | yeah, it's waiting on find | 18:22 |
| jeblair | which is running | 18:22 |
| fungi | i'm going to abort that since it's likely bogged down trying to do compression, and run a modified version which just handles deletion | 18:22 |
| fungi | to get us back on track faster | 18:23 |
| fungi | then we can resume the normal one | 18:23 |
| fungi | any objections? | 18:23 |
| jeblair | we don't know how long that script normally takes do we? | 18:23 |
| fungi | can probably infer from cpu graphs in cacti? | 18:24 |
| fungi | i have it queued up in a root screen session on static.o.o ready to go | 18:24 |
| pabelanger | I want to say over 24 hours the last time I ran it in screen | 18:24 |
| jeblair | fungi: no objection | 18:25 |
| *** esikachev has joined #openstack-infra | 18:26 | |
| fungi | previous finds/maint script killed, modified one which just does deletions is running now | 18:27 |
| *** jheroux has quit IRC | 18:27 | |
| fungi | 337M available already | 18:28 |
| fungi | jobs should no longer be hemmoraging | 18:28 |
| fungi | heh, and then something got uploaded and we're back to 123M available ;) | 18:28 |
| *** jkilpatr has joined #openstack-infra | 18:29 | |
| pabelanger | clarkb: do you have a script you used to audit the logs folder? | 18:29 |
| clarkb | pabelanger: no I didn't do anything sophisicated just ran du | 18:30 |
| pabelanger | k | 18:30 |
| clarkb | pabelanger: I ran it a little bit into the fs tree so that it wasn't hitting everything but that was diverse enough to see oh wow these 3 changes are all multigigabyte | 18:30 |
| clarkb | oh and they are all part of $project | 18:30 |
| clarkb | and go from there | 18:30 |
| fungi | looks like we're slowly making traction, deleting more than is being uploaded | 18:30 |
| * dims peeks | 18:31 | |
| *** esikachev has quit IRC | 18:31 | |
| *** tqtran has joined #openstack-infra | 18:31 | |
| * dims spots "<pabelanger>logs.o.o is full again" | 18:31 | |
| *** yamahata has joined #openstack-infra | 18:31 | |
| fungi | six more weeks of winter? | 18:32 |
| dims | lol | 18:32 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job https://review.openstack.org/431777 | 18:33 |
| jlvillal | Anyone else noticing "POST_FAILURE" errors? | 18:34 |
| *** flepied has joined #openstack-infra | 18:34 | |
| jlvillal | In Zuul | 18:34 |
| pabelanger | yes | 18:34 |
| pabelanger | logs.o.o is full, we are deleting content | 18:34 |
| jlvillal | Ah, thanks! | 18:34 |
| *** ihrachys_ has quit IRC | 18:34 | |
| fungi | (some?) jobs should be succeeding now | 18:35 |
| *** adrian_otto has joined #openstack-infra | 18:35 | |
| fungi | i keep seeing us get up to nearly 1gb free and then drop back down to almost none | 18:35 |
| *** ihrachys has joined #openstack-infra | 18:35 | |
| fungi | yep, now we're at 100K available :/ | 18:35 |
| *** Swami_ has quit IRC | 18:37 | |
| *** dhinesh has joined #openstack-infra | 18:37 | |
| pabelanger | I see we don't compress journal logs | 18:38 |
| clarkb | pabelanger: looks like I did something like: du -hs /srv/static/logs/35* | sort -h | 18:38 |
| fungi | something tells me it's going to take some time for find to happen across enough 45-60 day old logs to delete | 18:38 |
| pabelanger | found some in tripleo jobs, around 30MB uncompressed | 18:38 |
| clarkb | then 36* eetc until I found some patterns then attacked those rather than bulk | 18:39 |
| fungi | right now it's not removing them fast enough to offset the incoming volume | 18:39 |
| clarkb | pabelanger: sorry 35/* I think | 18:39 |
| fungi | i'm going to guess that the reason the log maintenance script was still running 4 days in was that up until yesterday it was competing with mlocate update scans | 18:40 |
| clarkb | pabelanger: 30MB uncompressed is still pretty small in grand scheme of things, previouslly it was single jobs with 2GB of logs | 18:40 |
| *** adrian_otto has quit IRC | 18:40 | |
| pabelanger | clarkb: ya, these are 250MB each, with multiple rechecks | 18:40 |
| *** ihrachys has quit IRC | 18:41 | |
| *** mriedem has joined #openstack-infra | 18:42 | |
| *** esikachev has joined #openstack-infra | 18:42 | |
| *** ihrachys has joined #openstack-infra | 18:43 | |
| *** trown|lunch is now known as trown | 18:44 | |
| *** salv-orlando has quit IRC | 18:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there https://review.openstack.org/434428 | 18:44 |
| *** MasterOfBugs has joined #openstack-infra | 18:45 | |
| pabelanger | clarkb: tripleo check jobs, are doing 603M right now. Looking to see what is going on | 18:45 |
| *** Goneri has quit IRC | 18:45 | |
| *** esikachev has quit IRC | 18:46 | |
| clarkb | my du is still being really slow :( | 18:47 |
| fungi | it's probably competing with my find -exec rm | 18:48 |
| pabelanger | clarkb: So, where do I update things to start compressing a new log file? | 18:48 |
| fungi | competing for i/o bandwidth i mean | 18:48 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 18:48 |
| clarkb | pabelanger: ideally jobs would compress as much as sane, then we have our fallback cron that sweeps through and picks up the leftovers | 18:48 |
| pabelanger | k | 18:48 |
| clarkb | pabelanger: that distributes the load and reduces network bw needed to do copies | 18:49 |
| fungi | what filename pattern are these uncompressed files? maybe we can use this to make some quick traction | 18:49 |
| *** zara_the_lemur__ has joined #openstack-infra | 18:49 | |
| pabelanger | /srv/static/logs/01/422801/1/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/9018e9c/logs/var/log/journal/ef56f1b7283141aea559c3525915a766 | 18:49 |
| pabelanger | for example | 18:49 |
| *** saibarspeis has joined #openstack-infra | 18:49 | |
| pabelanger | are 40GB uncompressed | 18:49 |
| fungi | so logs/var/log/journal/* | 18:50 |
| *** ankur-gupta-f4 has joined #openstack-infra | 18:50 | |
| pabelanger | ya | 18:50 |
| fungi | what are they? | 18:50 |
| pabelanger | system.journal and user-3000.journal | 18:50 |
| pabelanger | they are actually binary | 18:50 |
| mordred | sounds like systemd logs yeah? | 18:50 |
| fungi | oic, those are directory names | 18:50 |
| pabelanger | so, not sure it makes sense to archive them? | 18:51 |
| clarkb | pabelanger: devstack-gate archives the compressed text output of journalctl instead fwiw | 18:51 |
| *** sdague has quit IRC | 18:51 | |
| pabelanger | clarkb: let me see how tripleo is doing it | 18:51 |
| fungi | `du -sh /srv/static/logs/01/422801/1/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/9018e9c/logs/var/log/journal/ef56f1b7283141aea559c3525915a766` says that's only 33mib | 18:52 |
| fungi | not 40gib | 18:52 |
| pabelanger | fungi: typo on my side, I meant MB | 18:52 |
| fungi | oh, that's tiny | 18:52 |
| fungi | comparatively speaking | 18:52 |
| mordred | yah | 18:52 |
| pabelanger | multiple rechecks | 18:52 |
| mordred | 40GB of systemd logs is huge - and also I didn't blink at all thinking that might be the size of the logs :) | 18:52 |
| pabelanger | and multi nodes | 18:52 |
| pabelanger | so, it adds up | 18:53 |
| openstackgerrit | Victor Morales proposed openstack-dev/pbr master: Add memory tracer argument https://review.openstack.org/433947 | 18:53 |
| pabelanger | 1.2G /srv/static/logs/01/422801 | 18:54 |
| *** sdague has joined #openstack-infra | 18:55 | |
| *** jamesdenton has joined #openstack-infra | 18:55 | |
| fungi | which is about 0.01% of the current volume | 18:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there https://review.openstack.org/434428 | 18:55 |
| fungi | so not tiny, but for all jobs of all rechecks of a change it's not giant either | 18:56 |
| pabelanger | fungi: right, but other logs are only a few MB | 18:56 |
| *** sindhu has joined #openstack-infra | 18:56 | |
| clarkb | if my du ever finishes hopefully we can get a bit more insight at a higher level | 18:56 |
| clarkb | pabelanger: devstack-gate jobs were running inthe ~600MB range before iirc | 18:56 |
| clarkb | so I don't think thats true | 18:56 |
| clarkb | (but higher level du should confirm if it ever finishes) | 18:56 |
| * fungi picks a random recent change for an example | 18:56 | |
| *** tosky has quit IRC | 18:57 | |
| pabelanger | 7.0G ./414601 | 18:57 |
| pabelanger | another tripleo job | 18:57 |
| clarkb | pabelanger: thats not a job thats a change | 18:57 |
| pabelanger | mybad | 18:57 |
| *** tnovacik has joined #openstack-infra | 18:57 | |
| clarkb | so the change in total used 7GB which isn't tiny, but is there a specific job acting up? in the past it was literally one jobs made 2GB of logs | 18:58 |
| *** esikachev has joined #openstack-infra | 18:58 | |
| clarkb | then run that job a bunch and single change could be using 100 GB | 18:58 |
| pabelanger | this one is about 30 different patch uploads | 18:58 |
| fungi | hrm... /srv/static/logs/16/434316/1/check/gate-grenade-dsvm-neutron-ubuntu-xenial/1ca37b1 ran for one hour and succeeded, du -sh says total volume is 26mib | 18:58 |
| fungi | oh, wait, grenade | 18:58 |
| pabelanger | and multiple rechecks it seems | 18:58 |
| clarkb | (its good info regardless but important to distinguish between job cost vs total aggregate change cost) | 18:58 |
| fungi | but /srv/static/logs/16/434316/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7adca73 is 55mib | 18:59 |
| clarkb | ok mabe it was 600MB uncompressed | 19:00 |
| clarkb | and I have my numbers mixed up | 19:00 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 19:00 |
| pabelanger | ya, with /srv/static/logs/01/414601/24/check/gate-tripleo-ci-centos-7-3nodes-multinode-nv/3012361 it is 101M, but 75MB of that is the journal logs | 19:00 |
| fungi | we've made back >1gib of traction so maybe we're finally overcoming the inbound volume | 19:02 |
| *** dharinic is now known as dharinic|lunch | 19:02 | |
| *** pcaruana has quit IRC | 19:03 | |
| *** esikachev has quit IRC | 19:04 | |
| clarkb | I've restarted du with a smaller set of changes to hopefully get back something in a reasonable amount of time | 19:06 |
| *** MasterOfBugs has quit IRC | 19:07 | |
| clarkb | doing a few one offs for changes ~300MB seems to be normalish | 19:07 |
| *** MasterOfBugs has joined #openstack-infra | 19:07 | |
| clarkb | but thats with only a few patchsets and no rechecks looks like | 19:07 |
| *** adrian_otto has joined #openstack-infra | 19:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci master: Exclude /var/log/journal from devstack-logs https://review.openstack.org/434467 | 19:08 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci master: Add converge update to major upgrade test https://review.openstack.org/434250 | 19:09 |
| *** esikachev has joined #openstack-infra | 19:09 | |
| *** EricGonczer_ has joined #openstack-infra | 19:09 | |
| fungi | we've been consistently over 1gib available for the past 10 minutes or so now (just climbed to 2gib), so hopefully the worst is behind us for now | 19:09 |
| *** shardy has quit IRC | 19:10 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck master: Add query for bug 1664793 https://review.openstack.org/434436 | 19:11 |
| openstack | bug 1664793 in OpenStack Compute (nova) "test_stamp_pattern failing in ceph jobs" [Undecided,Confirmed] https://launchpad.net/bugs/1664793 | 19:11 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 19:11 |
| *** ihrachys_ has joined #openstack-infra | 19:13 | |
| *** ihrachys has quit IRC | 19:15 | |
| *** esikachev has quit IRC | 19:17 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Fix spaces in 2 queries https://review.openstack.org/434445 | 19:18 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 19:18 |
| *** EricGonczer_ has quit IRC | 19:18 | |
| clarkb | I fail at du'ing current one which should'ev been relatively quick is still going. Maybe there is a massive change dir int here? | 19:19 |
| clarkb | pabelanger: fungi http://paste.openstack.org/show/599100/ one of these is not like the others | 19:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci master: Exclude /var/log/journal from devstack-logs https://review.openstack.org/434467 | 19:21 |
| fungi | yeah, we just lost our traction. back to 0 bytes available again | 19:22 |
| bkero | :'( logs | 19:22 |
| *** sdake_ has joined #openstack-infra | 19:22 | |
| clarkb | super preliminary data says kolla likely our problem | 19:22 |
| clarkb | however that seems mostly bundled up in a single patchset 15/22GB on ps13 | 19:23 |
| clarkb | so maybe this is a one off | 19:23 |
| fungi | high volume of changes/rechecks, or large logs for a single job? | 19:23 |
| clarkb | fungi: not sure yet, du still slow | 19:23 |
| clarkb | will dig in though | 19:23 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool master: Create nodepool::launcher https://review.openstack.org/430329 | 19:23 |
| fungi | yeah, i have a feeling the current deletion work is slowing down your du, and vice versa | 19:23 |
| clarkb | should I hold off while deletion goes? | 19:24 |
| clarkb | as that is what makes current jobs happy? | 19:24 |
| fungi | maybe, though there's also the possibility you'll find something we can delete directly to free up more space faster | 19:25 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Add support for nodepool-launcher https://review.openstack.org/430339 | 19:25 |
| clarkb | like patchset 13 of change 404435 :) | 19:25 |
| *** sdake has quit IRC | 19:26 | |
| sdague | ok, anyone understand this pbr issue? | 19:26 |
| sdague | http://paste.openstack.org/show/599102/ | 19:26 |
| sdague | this is a 'pip3 install devstack-tools' on one of my devstack machines | 19:27 |
| *** sdake has joined #openstack-infra | 19:27 | |
| clarkb | fungi: pabelanger appears to be related to kolla's experiemntal jobs | 19:28 |
| clarkb | http://paste.openstack.org/show/599103/ probably a good idea to get them to reign that in if possible | 19:28 |
| pabelanger | I wonder if they are also copying images | 19:29 |
| sdague | clarkb: also, we need https://review.openstack.org/#/c/433093 for ironic grenade jobs to work again, because they need a feature that's broken in the current release | 19:29 |
| pabelanger | yup | 19:30 |
| pabelanger | -rw-r--r-- 1 jenkins jenkins 3.1G Dec 14 09:40 centos-binary-registry.tar.gz | 19:30 |
| pabelanger | that is the tarball they are coping to tarball.o.o | 19:30 |
| *** sdake_ has quit IRC | 19:30 | |
| pabelanger | let me check there jobs and see what is going on | 19:30 |
| fungi | fun, so they're also saving a copies of it to the logserver over and over? | 19:31 |
| clarkb | fungi: apeprs so | 19:31 |
| *** rbrndt has quit IRC | 19:31 | |
| fungi | we can probably do a surgical find to delete all those and free up a bunch | 19:31 |
| *** e0ne has joined #openstack-infra | 19:33 | |
| clarkb | sdague: the problem appears to be that devstack-tools installs as lib/devstack but your version info is at lib/devstack_tools-version.dist-info | 19:33 |
| *** e0ne has quit IRC | 19:33 | |
| clarkb | sdague: so they don't line up, you may need to edit the pcakge name? | 19:33 |
| sdague | clarkb: hmmm... this used to work | 19:33 |
| clarkb | sdague: I installed ina local venv then ls'd lib/python3.5/site-packages to see that | 19:34 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Enable mlock_tracker in gate https://review.openstack.org/434474 | 19:34 |
| sdague | clarkb: so I need to redo the directory structure? | 19:35 |
| clarkb | sdague: I think that would be one way to fix it yes | 19:36 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Enable mlock_tracker in gate https://review.openstack.org/434474 | 19:36 |
| clarkb | sdague: should be easy enough to test if you mv then pip install ./repo/dir | 19:36 |
| sdague | clarkb: so I'm super confused, because this totally was working, and is even used in the gate | 19:36 |
| *** chlong has joined #openstack-infra | 19:36 | |
| sdague | mostly confused as to what changed to make it stop working | 19:37 |
| *** john-davidge has quit IRC | 19:37 | |
| clarkb | I don't know that pbr has changed | 19:37 |
| clarkb | maybe setuptools changed? | 19:37 |
| sdague | maybe? | 19:37 |
| clarkb | ther ewas a thread about releasing pbr and dhellmann said wait for ocata first, but maybe someone jumped the gun? | 19:38 |
| sdague | pbr didn't seem to be released recently | 19:38 |
| sdague | but setuptools did 2 days ago | 19:38 |
| *** EricGonczer_ has joined #openstack-infra | 19:38 | |
| mordred | \o/ | 19:38 |
| fungi | worth testing with an older setuptools to see if that was where the behavior change crept in | 19:39 |
| fungi | the changelog implies there were changes to finding/reading dist-info metadata | 19:40 |
| *** e0ne has joined #openstack-infra | 19:40 | |
| mordred | I'm not sure how this worked - this behavior (needing to match what's in __init__.py with what the name is in setup.cfg is not new at al | 19:42 |
| mordred | and has tripped up people before | 19:42 |
| mordred | I mean, that said - clearly it worked and now doesn't | 19:42 |
| mordred | but it's a common mistake | 19:42 |
| sdague | ug | 19:42 |
| sdague | still didn't fix it | 19:43 |
| sdague | oh, maybe I missed one reference | 19:43 |
| mordred | sdague: https://review.openstack.org/434478 | 19:43 |
| *** esikachev has joined #openstack-infra | 19:44 | |
| *** hashar has joined #openstack-infra | 19:44 | |
| mordred | maybe it worked before because something else installed a python package as "devstack" and that was what was being reported in __init__.py but nobody noticed they were off | 19:44 |
| mordred | and now nothing is instlaling anything as "devstack" so there is no entry in the pkg_resources system | 19:45 |
| sdague | mordred: ah.... | 19:45 |
| sdague | mordred: interesting | 19:45 |
| mordred | sdague: did devstack itself used to install a thing and recently stopped? | 19:45 |
| sdague | mordred: nope | 19:45 |
| mordred | hr | 19:45 |
| mordred | hrm | 19:45 |
| sdague | this is that thing, for the first time | 19:45 |
| mordred | well - no clue how it ever worked but this bug has been latent in this repo since commit 1 | 19:46 |
| sdague | mordred: yep, weird though, because this totally is being run in the grenade job and working | 19:46 |
| sdague | anyway, thank you for the fix | 19:46 |
| sdague | well get that out shortly | 19:47 |
| clarkb | sdague: with python2 right? I wonder if its a 2 vs 3 weirdness somehow | 19:47 |
| sdague | clarkb: I was pretty sure we were only doing python3 | 19:47 |
| mordred | it's not 2 vs 3 | 19:47 |
| sdague | but, maybe | 19:47 |
| gordc | mtreinish: just to verify, https://review.openstack.org/#/c/338435 is only being applied to master, ocata, newton? | 19:48 |
| gordc | it broke our gate because we look for uec suffix. just need to know which branches i need to backport to | 19:48 |
| fungi | clarkb: pabelanger: were one of you already in the process of syncing up with kolla liaisons? | 19:48 |
| mtreinish | gordc: and mitaka, https://review.openstack.org/367049 | 19:48 |
| fungi | i'm trying to get a count right now of how many centos-binary-registry.tar.gz copies we have on the logs volume to see what deleting them will free up | 19:49 |
| clarkb | nope happnes with python2 for me as well | 19:49 |
| pabelanger | fungi: clarkb: actually, I don't think it is an issue any more. The experimental job has moved to post, but I no longer see us coping the tarball for the registry | 19:49 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job https://review.openstack.org/431777 | 19:49 |
| clarkb | pabelanger: ok so possibly a one off? | 19:49 |
| gordc | mtreinish: ack. we're backporting because of grenade? | 19:49 |
| pabelanger | clarkb: fungi http://logs.openstack.org/40/40130b2e7a591931ddac366dc6f8d085e1f83e37/post/kolla-dsvm-build-ubuntu-source-ubuntu-xenial/997bce3/logs/ is the job in question | 19:49 |
| clarkb | pabelanger: fungi though depending on how often they run the experiemnental and post jobs may still be an issue? | 19:49 |
| fungi | pabelanger: okay, so odds are if we delete these they won't care (however there may not be enough of them to matter) | 19:49 |
| mtreinish | gordc: actually because we want to remove the uec support from tempest (it's been deprecated for 2 or 3 cycles) | 19:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix several concurrent shade gate issues https://review.openstack.org/434428 | 19:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data https://review.openstack.org/434481 | 19:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: DNM - Patch to verify 2.0 usage payload behavior https://review.openstack.org/434482 | 19:49 |
| *** tonytan4ever has quit IRC | 19:49 | |
| pabelanger | clarkb: I think it was an issue when we did testing in experimental, how it moved to post around Jan 3 | 19:50 |
| mtreinish | gordc: https://review.openstack.org/#/c/338377/ | 19:50 |
| pabelanger | so the job doesn't exist any more | 19:50 |
| *** tonytan4ever has joined #openstack-infra | 19:50 | |
| gordc | mtreinish: kk, thanks for background. will try to match on something more generic in our gate | 19:50 |
| clarkb | pabelanger: right just thinking if they ran the experimental jobs say 30 times thats roughly half a terabyte on its own | 19:51 |
| *** chlong has quit IRC | 19:51 | |
| clarkb | so may still be the problem | 19:51 |
| pabelanger | clarkb: yes, that is possible | 19:52 |
| pabelanger | https://review.openstack.org/#/c/404435/ would be the review to focus on | 19:52 |
| pabelanger | let me see if I can find it | 19:52 |
| clarkb | pabelanger: well thats just one instance of it | 19:52 |
| clarkb | and resulted in 15GB of logs on patchset 13 | 19:52 |
| clarkb | so we'd need to look for other instances | 19:52 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate https://review.openstack.org/434399 | 19:53 |
| pabelanger | clarkb: understood, 404435 was the main patch for testing experimental IIRC | 19:53 |
| clarkb | pabelanger: oh, except they only had massive logs the one time | 19:53 |
| clarkb | and it wasn't in the last patchset either | 19:53 |
| pabelanger | right, I think something changed to stop the copy, but haven't figured out what | 19:54 |
| jeblair | clarkb, fungi, mordred: in addition to logging the size of things we copy to logs.o.o in zuulv3, we could probably *limit* the size -- ie, copy from worker to launcher; du; then bail if it is too large. if we think we're likely to go that way in zuulv3, we could probably add that as a feature to v2.5 if we thought it necessary. | 19:54 |
| mordred | gordc: that broke the shade gate too - for the exact same reason | 19:56 |
| clarkb | pabelanger: gotcha | 19:56 |
| clarkb | I'm running another du scan to see if anything else like that pops up | 19:57 |
| clarkb | and will probably continue to run them for a bit | 19:57 |
| clarkb | since I can't really take a full world view look in reasonable amount of time | 19:57 |
| *** mdurrant_ is now known as mdurrant | 19:57 | |
| gordc | mordred: i used GLANCE_IMAGE_NAME=$(openstack image list | awk '/ cirros.* /{print $4; exit}') as workaround. not sure if that's the same issue you had. | 19:58 |
| *** tonytan4ever has quit IRC | 19:58 | |
| mordred | gordc: yah - similar - you may have problems with that on old clouds (because of ami/aki/ari existing - which is, I'm sure, why you were looking for the -uec string like we were) - but if the image change is backported everywhere, then I think your thing is golden | 19:59 |
| *** tonytan4ever has joined #openstack-infra | 19:59 | |
| mordred | (our lookup is in python, so I just look for startswtih('cirros') after I fail finding something that starts with cirros and ends with uec) | 20:00 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/puppet-elastic_recheck master: Enable recheck irc bot for neutron https://review.openstack.org/433828 | 20:00 |
| *** Gorian- is now known as Gorian | 20:00 | |
| mtreinish | ihrachys_: ^^^ so I was thinking about that a bit, ideally what would you like e-r to report in channel? | 20:01 |
| gordc | mordred: hope so... else i need to google more bash stuff. | 20:01 |
| mtreinish | ihrachys_: because the bot configuration might not be sufficient | 20:02 |
| *** dharinic|lunch is now known as dharinic | 20:02 | |
| fungi | jeblair: that sounds like an excellent option for zuul | 20:02 |
| mtreinish | as in I might need to improve how we configure it in e-r | 20:02 |
| pabelanger | clarkb: Hmm, tripleo is copying etc/alternatives | 20:03 |
| pabelanger | which is about 911MB | 20:03 |
| clarkb | wat | 20:04 |
| mordred | maybe they shoudl copy the sylinks and not the destinations | 20:04 |
| clarkb | pabelanger: for every host on every job? | 20:04 |
| mordred | are they using tar to copy? | 20:04 |
| pabelanger | clarkb: not sure, just checking | 20:04 |
| pabelanger | /srv/static/logs/29/431029/ | 20:04 |
| pabelanger | was most recent | 20:04 |
| fungi | well, /etc/alternatives is symlinks to stuff in /usr/bin generally, so presumably the copying is dereferencing links as it goes | 20:04 |
| *** esikachev has quit IRC | 20:04 | |
| mordred | I could see copying /etc/alternatives to see what the symlinks point to - but copying the dereferneced links == bad | 20:04 |
| *** erikmwilson has quit IRC | 20:04 | |
| pabelanger | it is some java things | 20:05 |
| mordred | pabelanger: that'll be a nice thing to clean up | 20:05 |
| mordred | yup | 20:05 |
| mordred | java alternatives on nodes | 20:05 |
| mordred | will eat some space | 20:05 |
| fungi | ouch, definitely | 20:05 |
| mordred | maybe we shoudl "alias cp 'cp -d'" :) | 20:06 |
| *** erikmwilson has joined #openstack-infra | 20:06 | |
| clarkb | pabelanger: have a full path? I am not finding it through logs.o.o web browsing | 20:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Introducing Omni https://review.openstack.org/433363 | 20:07 |
| mordred | clarkb: isn't the full path /srv/static/logs/29/431029/ ? | 20:07 |
| clarkb | mordred: no there is not etc/alternatives there | 20:07 |
| clarkb | check/ and check-tripleo/ then the individual jobs | 20:07 |
| *** Sukhdev has joined #openstack-infra | 20:08 | |
| mordred | nod | 20:08 |
| pabelanger | /srv/static/logs/29/431029/26/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3ca8868/logs/undercloud/etc/alternatives | 20:08 |
| mordred | clarkb: /srv/static/logs/29/431029/24/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ocata/c6090c9/logs/undercloud/etc/alternatives | 20:08 |
| clarkb | I guess I could switch to sshing onto static and use find but I like using the user accessible version as much as possible | 20:08 |
| clarkb | tyty | 20:08 |
| mordred | yah - things like /srv/static/logs/29/431029/24/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ocata/c6090c9/logs/undercloud/etc/alternatives/jre_1.7.0/lib/rhino.jar.gz are unhappy in logs :) | 20:08 |
| mordred | pabelanger: nice find | 20:08 |
| pabelanger | The issue is, tripleo-ci jobs will grab everything, then exclude things | 20:08 |
| pabelanger | it should be the inverted, exclude everything, copy what is needed | 20:09 |
| mordred | pabelanger: well, we should at the very least add an exclude for /etc/alternatives | 20:09 |
| pabelanger | yes | 20:09 |
| pabelanger | will patch | 20:09 |
| mordred | while we can get it swapped to what you said :) | 20:09 |
| pabelanger | https://review.openstack.org/#/c/434467/ | 20:09 |
| pabelanger | is for /var/log/journal | 20:09 |
| *** jkilpatr_ has joined #openstack-infra | 20:09 | |
| mordred | ++ | 20:09 |
| clarkb | my suggestion would be to specifically whitelist what you need out of /etc | 20:09 |
| clarkb | this is what devstack-gate does and works well | 20:10 |
| *** tonytan4ever has quit IRC | 20:10 | |
| fungi | or if you feel strongly that /etc should only contain small files, then don't follow symlinks you find in it | 20:10 |
| *** tonytan4ever has joined #openstack-infra | 20:11 | |
| clarkb | also this is why I keep thinking it would be nice to have an /etc/openstack but there are still some projects that that wouldn't work with aiui | 20:11 |
| *** Goneri has joined #openstack-infra | 20:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Use new ara generate html command https://review.openstack.org/434492 | 20:11 |
| pabelanger | clarkb: yes! | 20:11 |
| mordred | dmsimard: ^^ that's accurate right? | 20:11 |
| clarkb | so /etc/openstack/nova /etc/openstack/neutron et al then just tar /etc/openstack and call it a day | 20:11 |
| * dmsimard looks | 20:11 | |
| clarkb | pabelanger: ya I tried to do this a couple years ago and discovered that like half our projects had hardcoded config path requirements | 20:12 |
| clarkb | pabelanger: and promptly had a sad and did something else more productive than fixing that | 20:12 |
| dmsimard | mordred: I had submitted https://review.openstack.org/#/c/433207/ already | 20:12 |
| *** Sukhdev has quit IRC | 20:12 | |
| mordred | oh! | 20:12 |
| mordred | I will abandon mine | 20:12 |
| mordred | clarkb: ^^ wanna +A dmsimard 's patch? | 20:13 |
| *** jkilpatr has quit IRC | 20:13 | |
| * clarkb looks | 20:14 | |
| *** ldnunes has quit IRC | 20:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci master: Switch logs for TARCMD to include only specific logs https://review.openstack.org/434496 | 20:17 |
| *** esikachev has joined #openstack-infra | 20:17 | |
| dmsimard | mordred: when I deprecated it, I searched for usage in openstack land with codesearch.openstack.org and submitted patches everywhere | 20:17 |
| dmsimard | <3 codesearch.openstack.org | 20:17 |
| mordred | pabelanger: find /srv/static/logs -wholename '*etc/alternatives*' -exec 'rm -rf {}' would be fun :) | 20:17 |
| mordred | dmsimard: yay! | 20:17 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/puppet-elastic_recheck master: Enable recheck irc bot for neutron https://review.openstack.org/433828 | 20:17 |
| mordred | dmsimard: there are times when I really want an API to that so that I use it like find in a script :) | 20:18 |
| dmsimard | yeah, like "openstackgrep" or something | 20:18 |
| openstackgerrit | Merged openstack-infra/project-config master: combine vitrage dsvm jobs as one pattern https://review.openstack.org/432990 | 20:18 |
| mordred | dmsimard: I mean - I'm 100% certain there is an API we could call | 20:19 |
| mordred | dmsimard: the website is all client-side javascript calling a go backend | 20:19 |
| mordred | iirc | 20:19 |
| fungi | mordred: pabelanger: we might want to reduce the number of recursive scans we're performing on that filesystem. right now in have one deleting files older than 45 days and another that's been running for some 30 minutes now just trying to `find /srv/static/logs/ -name centos-binary-registry.tar.gz` which has system load up around 8 now | 20:20 |
| mordred | dmsimard: https://github.com/etsy/hound/blob/master/api/api.go ... /api/v1/search seems to be a thing | 20:20 |
| mordred | fungi: I will step away from logs.o.o | 20:21 |
| dmsimard | mordred: now you just need a client :) | 20:21 |
| mordred | dmsimard: I feel like we're each trying to nerd-snipe the other into being the one to write it :) | 20:21 |
| *** esikachev has quit IRC | 20:21 | |
| *** rbrndt has joined #openstack-infra | 20:22 | |
| fungi | as it stands, the one find removing files >45 days still isn't able to delete data as fast as jobs are writing it | 20:22 |
| *** salv-orlando has joined #openstack-infra | 20:22 | |
| fungi | so i'm going to kill the other one i have trying to locate kolla's registry tarballs | 20:23 |
| fungi | in hopes that may speed things up a bit | 20:23 |
| fungi | looks like that's bringing system load back down a bit now | 20:25 |
| fungi | and the 45-day expiration find seems to maybe be picking up steam again, hard to be sure | 20:26 |
| *** salv-orlando has quit IRC | 20:26 | |
| *** nadya has quit IRC | 20:26 | |
| *** esikachev has joined #openstack-infra | 20:27 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 20:30 |
| mordred | dmsimard: import requests ; s=requests.get('http://codesearch.openstack.org/api/v1/search', params=dict(q='ara generate', stats='fosho', i='nope', repos='*', rng=':20', files=None)).json()['Results'] | 20:30 |
| mordred | dmsimard: please enjoy that true and false are "fosho" and "nope" :( | 20:31 |
| dmsimard | lol | 20:31 |
| clarkb | https://review.openstack.org/#/c/410835/ was another change that had quite a bit of disk use. ~5GB | 20:34 |
| clarkb | but appears to be lots of small cuts there and oslo.messaging just runs a ton of jobs | 20:34 |
| *** rbuzatu has quit IRC | 20:35 | |
| *** ankur-gupta-f4 has left #openstack-infra | 20:36 | |
| *** esikachev has quit IRC | 20:36 | |
| *** jamielennox|away is now known as jamielennox | 20:36 | |
| sigmavirus | mordred: why use dict(...)? | 20:37 |
| fungi | load average is down to ~2 now, but unfortunately the 45-day expiration find is still not deleting files fast enough to keep pace with uploads | 20:37 |
| sigmavirus | Why do you hate dictionary literals?! | 20:37 |
| sigmavirus | =P | 20:37 |
| *** ihrachys_ has quit IRC | 20:37 | |
| *** Sukhdev has joined #openstack-infra | 20:38 | |
| *** john-davidge has joined #openstack-infra | 20:38 | |
| fungi | and load's spiked back up to ~5 again, so i guess it's not steady | 20:38 |
| *** ihrachys has joined #openstack-infra | 20:39 | |
| *** rbuzatu has joined #openstack-infra | 20:39 | |
| mordred | sigmavirus: too much typing :) | 20:39 |
| openstackgerrit | Ian Wienand proposed openstack-dev/pbr master: Remove testr entry point https://review.openstack.org/422379 | 20:40 |
| mordred | sigmavirus: but mostly just to troll you I think | 20:40 |
| openstackgerrit | Ian Wienand proposed openstack-dev/pbr master: Also generate XML coverage report https://review.openstack.org/434071 | 20:40 |
| fungi | yeah, dict() lets you skip some quoting ;) | 20:40 |
| *** sdake has quit IRC | 20:41 | |
| fungi | if you have >3 keys, it's fewer characters overall | 20:41 |
| *** e0ne has quit IRC | 20:41 | |
| pabelanger | fungi: clarkb: mordred: https://review.openstack.org/#/c/431029/ is the problem patch right now | 20:42 |
| *** esikachev has joined #openstack-infra | 20:42 | |
| pabelanger | it is what keeps burning through our HDD in logs.o.o | 20:42 |
| *** john-davidge has quit IRC | 20:43 | |
| *** aviau has quit IRC | 20:43 | |
| *** rbuzatu has quit IRC | 20:43 | |
| *** aviau has joined #openstack-infra | 20:43 | |
| fungi | does it generate an inordinate amount of additional logging, or is it just because it has 27 patchsets over the course of a week? | 20:44 |
| *** abompard has quit IRC | 20:44 | |
| fungi | anyway, to stop the bleeding, i could see deleting logs for all patchsets prior to the most recent one | 20:44 |
| clarkb | looks like ps24 was worth 18GB though the otehrs are much smaller | 20:44 |
| clarkb | heh check 21MB, check-tripleo 18GB | 20:45 |
| fungi | that may get us dar enough ahead of the curve that the expiration work will keep us above water until job volume trails off for the day | 20:45 |
| fungi | s/dar/far/ | 20:45 |
| pabelanger | yes | 20:46 |
| smcginnis | Are the POST_FAILURES I'm seeing from a full driver or something? | 20:46 |
| pabelanger | http://logs.openstack.org/29/431029/26/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3ca8868/logs/undercloud/etc/ | 20:46 |
| pabelanger | tripleo is now copying the full contents of /etc from their undercloud | 20:46 |
| clarkb | it is interesting that if I du on check-tripleo it says 18GB | 20:46 |
| clarkb | if I du on the individual jobs under there its in megabytes | 20:46 |
| clarkb | so there is something funny going on in disk usage too I think | 20:47 |
| mordred | smcginnis: yah - we have some jobs uploading giant amounts of data and we can't delete things quickly enough to keep up | 20:47 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for bug 1664793 https://review.openstack.org/434436 | 20:47 |
| openstack | bug 1664793 in OpenStack Compute (nova) "test_stamp_pattern timing out waiting for attached device to show up in guest" [Undecided,Confirmed] https://launchpad.net/bugs/1664793 | 20:47 |
| *** r1chardj0n3s has joined #openstack-infra | 20:47 | |
| smcginnis | mordred: Joy! | 20:47 |
| *** sdake has joined #openstack-infra | 20:47 | |
| *** sdake has quit IRC | 20:47 | |
| smcginnis | We're good at DOS'ing ourselves. | 20:47 |
| clarkb | oh wait I am derp nevermind | 20:47 |
| clarkb | cd'd into the wrong dir | 20:47 |
| mordred | smcginnis: maybe if cinder had a magical "please delete the files we don't want but not the ones we do want and do it in violation of the speed of light" feature? | 20:48 |
| mordred | smcginnis: being well aware that cinder doesn't do filesystems of course ... | 20:48 |
| smcginnis | mordred: I'll get a spec started for that right away! :) | 20:48 |
| mordred | smcginnis: thanks! | 20:48 |
| r1chardj0n3s | hey folks, any idea what's goimg on with https://review.openstack.org/#/c/427125 and the POST_FAILUREs of doom? | 20:48 |
| AJaeger | smcginnis: please include intelligent compression and removal of not needed files in the spec ;) | 20:48 |
| smcginnis | r1chardj0n3s: Just asked too. Log volumes are getting full. | 20:49 |
| *** vsaienko has quit IRC | 20:49 | |
| jeblair | fungi: maybe time for a #status notice? | 20:49 |
| r1chardj0n3s | smcginnis: ah ok, thanks | 20:49 |
| smcginnis | jeblair: +1 | 20:49 |
| fungi | r1chardj0n3s: we ran out of space to save logs from jobs and are in the process of blowing away anything older than 6 weeks to clear out enough space for current jobs | 20:49 |
| r1chardj0n3s | fungi: thanks | 20:49 |
| smcginnis | AJaeger: Maybe we can have it rewrite the blocks to change your data to what you actually want too. ;) | 20:49 |
| *** ssbarnea has quit IRC | 20:49 | |
| fungi | r1chardj0n3s: though the volume is so high that we're not currently deleting as fast as new logs are uploaded. looking for other ways to pick up steam there | 20:49 |
| *** sdake has joined #openstack-infra | 20:50 | |
| *** vsaienko has joined #openstack-infra | 20:50 | |
| mordred | fungi: maybe a single recursive find looking for and deleting entire etc/alternatives dirs or kolla registry tarballs would get us larger chunks of files deleted at once? | 20:50 |
| pabelanger | clarkb: ya, 18GB for all jobs, about 4GB for each undercloud | 20:50 |
| *** esikachev has quit IRC | 20:51 | |
| pabelanger | clarkb: from what I understand, it is a recent change for to help tripleo debug failures | 20:51 |
| clarkb | pabelanger: unfortuniately it is now causing failures :P | 20:51 |
| mordred | which would explain why we're out of space all of a sudden | 20:51 |
| pabelanger | indeed | 20:52 |
| *** abompard has joined #openstack-infra | 20:52 | |
| *** abompard has joined #openstack-infra | 20:52 | |
| fungi | status notice We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear. | 20:52 |
| fungi | something like that? ^ | 20:52 |
| pabelanger | yes | 20:52 |
| jeblair | fungi: lgtm | 20:52 |
| *** zara_the_lemur__ has left #openstack-infra | 20:53 | |
| pabelanger | we could maybe set max server to 0 for tripleo-test-cloud-rh1, until we get a large chunk of data restored | 20:53 |
| mordred | pabelanger: I thought they were running a bunch of things in normal gate now? | 20:53 |
| mordred | fungi: ++ | 20:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo-ui: add openstack-server-release-jobs https://review.openstack.org/434510 | 20:54 |
| clarkb | pabelanger: we are and those are in few MB range of log data | 20:54 |
| clarkb | er mordred ^ | 20:54 |
| pabelanger | mordred: they are, but this looks limited to ovb jobs, which run on tripleo-test-cloud-rh1 | 20:54 |
| mordred | ah | 20:54 |
| mordred | yah - I think dropping max server to 0 for a sec while we clean things should help us not fight new uploads maybe? | 20:54 |
| jeblair | yeah, we limit log uploads from tox runs to 50MB. that's kind of what we considered a 'large amount of data' at one point. | 20:54 |
| mordred | jeblair: nobody would ever need more than 64K of ram you know | 20:55 |
| fungi | #status notice We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear. | 20:55 |
| openstackstatus | fungi: sending notice | 20:55 |
| mordred | that's 64 THOUSAND bytes!!! | 20:55 |
| pabelanger | maybe time to update rsync to limit data too? | 20:55 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo-ui: add openstack-server-release-jobs https://review.openstack.org/434510 | 20:55 |
| jeblair | mordred: that's more than 400 tweets | 20:55 |
| *** sdake has quit IRC | 20:56 | |
| -openstackstatus- NOTICE: We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear. | 20:56 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Enable peakmem_tracker in gate https://review.openstack.org/434511 | 20:57 |
| *** saibarspeis has quit IRC | 20:57 | |
| *** tnovacik has quit IRC | 20:57 | |
| EmilienM | pabelanger: sorry for the problem about logs :-/ we'll make our possible to reduce the number of logs we're saving and are open to changes | 20:57 |
| *** rfolco has quit IRC | 20:57 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 cloud https://review.openstack.org/434512 | 20:58 |
| pabelanger | mordred: fungi: ^ the extreme option | 20:58 |
| openstackstatus | fungi: finished sending notice | 20:58 |
| pabelanger | EmilienM: it appears limited to OVB jobs | 20:59 |
| EmilienM | pabelanger: wait, what is the impact of that? | 20:59 |
| EmilienM | pabelanger: can we have a quick workaround in tripleo-ci in the meantime? | 20:59 |
| *** vsaienko has quit IRC | 20:59 | |
| *** adriant has joined #openstack-infra | 20:59 | |
| EmilienM | slagle, bnemec ^ fyi | 20:59 |
| pabelanger | EmilienM: nothing decided yet, but it would stop OVB jobs from running | 21:00 |
| pabelanger | until we can reclaim HDD space | 21:00 |
| EmilienM | ok, let's try to find a short term solution for now | 21:00 |
| mordred | EmilienM: currently we're producing about 4G of logs (and other files) for each undercloud | 21:01 |
| EmilienM | I know I know about the number | 21:01 |
| EmilienM | I know it's terrible etc | 21:01 |
| mordred | kk. sorry - just trying to tl;dr if it was busy scrollback | 21:01 |
| *** mdorman has left #openstack-infra | 21:01 | |
| jeblair | we could land that and revert it quickly as soon as we have something better. | 21:02 |
| fungi | EmilienM: a major cause is apparently that all of /etc is being copied, and symlinks (including stuff in /etc/alternatives) are dereferenced when doing so resulting in copies of all sorts of stuff from /usr you really don't need | 21:02 |
| EmilienM | pabelanger: ok so https://review.openstack.org/#/c/434467 is ready to land once we have the OVB jobs results | 21:03 |
| *** erikmwilson has quit IRC | 21:04 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert use of .register_uri to .register_uris https://review.openstack.org/434516 | 21:04 |
| EmilienM | pabelanger: if we exclude /etc/alternatives for now, is it a good start? | 21:07 |
| *** erikmwilson has joined #openstack-infra | 21:08 | |
| pabelanger | EmilienM: we need to figure out what changed to start uploading: http://logs.openstack.org/29/431029/24/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ocata/c6090c9/logs/undercloud/etc/ | 21:09 |
| *** mhickey has joined #openstack-infra | 21:09 | |
| clarkb | EmilienM: yse, however my suggestion earlier was you need to whitelist what you want out of /etc | 21:09 |
| pabelanger | because that is 4GB of data each patch | 21:09 |
| mordred | EmilienM: yah. I think that and also the /var/log/journal patch | 21:09 |
| clarkb | don't copy everything just because | 21:09 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck master: Add support to irc bot for filtering messages by repo https://review.openstack.org/434519 | 21:09 |
| mordred | I think just blacklisting those two while you work up the correct whitelist | 21:09 |
| mordred | will be very helpful | 21:09 |
| *** jamesdenton has quit IRC | 21:09 | |
| mtreinish | ihrachys: ^^^ I think that'll do it | 21:09 |
| mtreinish | ihrachys: heh, I also just noticed you're not in openstack-qa :) | 21:10 |
| EmilienM | clarkb: I agree, but it might take some time to do it | 21:10 |
| clarkb | I mean | 21:10 |
| EmilienM | mordred: ok | 21:10 |
| clarkb | there is already a whitelist in devstack/devstack-gate | 21:10 |
| clarkb | just reuse that? | 21:10 |
| EmilienM | mordred: on it | 21:10 |
| EmilienM | clarkb: we'll consider this option seriously | 21:10 |
| ihrachys | mtreinish: should I? :) lemme add it to my list. | 21:10 |
| EmilienM | clarkb: let me grab some people and estinguish this fire. We'll fix the root cause later, this week | 21:11 |
| *** raildo has quit IRC | 21:11 | |
| pabelanger | I'm going to start purging http://logs.openstack.org/29/431029 for some HDD space | 21:11 |
| fungi | thanks pabelanger! that's already making a significant difference | 21:12 |
| fungi | 17gib free and climbing | 21:12 |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate https://review.openstack.org/434384 | 21:13 |
| *** ijw has joined #openstack-infra | 21:14 | |
| pabelanger | fungi: ya | 21:14 |
| jeblair | i approved the disable tripleo-test-cloud-rh1 disable; if we want to stop that, we should act fast. :) | 21:16 |
| *** clayg_ is now known as clayg | 21:16 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck master: Add support to irc bot for filtering messages by repo https://review.openstack.org/434519 | 21:16 |
| fungi | fingers crossed that this surge in available space will carry us through until the 45-day expiration pass is able to scan further through the filesystem | 21:17 |
| *** Apoorva has quit IRC | 21:17 | |
| pabelanger | okay, deleted 431029 | 21:17 |
| pabelanger | got us 24GB | 21:18 |
| clarkb | did we also manage to delete the 15GB of kolla images? | 21:18 |
| pabelanger | I didn't, but can check again | 21:18 |
| clarkb | even if we just delete that one set of experimental job logs it will free up ~15GB so may be worth it for headroom | 21:19 |
| fungi | clarkb: logs from a single tripleo change | 21:19 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci master: Exclude etc/alternatives https://review.openstack.org/434523 | 21:19 |
| pabelanger | k, doing kolla logs now | 21:19 |
| fungi | but yes, we should do a separate find to remove those kolla repository tarballs from their experimental job logs | 21:20 |
| fungi | i gave up after 30 minutes of waiting for it to try to find them because i didn't want to slow down the 45-day expiration pass even more | 21:20 |
| *** ijw has quit IRC | 21:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 cloud https://review.openstack.org/434512 | 21:22 |
| *** vsaienko has joined #openstack-infra | 21:22 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck master: Add support to irc bot for filtering messages by repo https://review.openstack.org/434519 | 21:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackci master: Reduce job log retention to 45 days https://review.openstack.org/434524 | 21:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Disable tripleo-test-cloud-rh1 cloud" https://review.openstack.org/434525 | 21:23 |
| pabelanger | when we are ready^ | 21:23 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul master: Expose commitMessage as a Change attribute https://review.openstack.org/222791 | 21:23 |
| *** ijw has joined #openstack-infra | 21:24 | |
| *** annegentle has quit IRC | 21:24 | |
| *** gildub has joined #openstack-infra | 21:25 | |
| *** kgiusti has quit IRC | 21:26 | |
| *** askb has joined #openstack-infra | 21:27 | |
| clarkb | fungi: ya was just thinking doing the specific one we know about would be useful | 21:28 |
| pabelanger | up to 45GB now | 21:28 |
| *** Apoorva has joined #openstack-infra | 21:29 | |
| *** esikachev has joined #openstack-infra | 21:29 | |
| fungi | clarkb: oh, yep sure | 21:31 |
| EmilienM | tbh, I don't know what to do now. It seems like the /etc/alternatives thing was done in a single patch: https://review.openstack.org/#/c/431029/ | 21:32 |
| EmilienM | slagle is making sure we don't have this dir in multinode jobs but we think it won't do anything since we didn't have the dir before | 21:33 |
| EmilienM | https://review.openstack.org/#/c/434523/ | 21:33 |
| EmilienM | and pabelanger is exclusing the journal file: https://review.openstack.org/#/c/434467/ | 21:33 |
| EmilienM | what else we can do on short term? | 21:33 |
| EmilienM | except blocking this patch: https://review.openstack.org/#/c/431029/ | 21:33 |
| *** Jeffrey4l has quit IRC | 21:34 | |
| EmilienM | AJaeger: can you look https://review.openstack.org/#/c/434510 please? I replied, I need your guidance | 21:34 |
| *** Jeffrey4l has joined #openstack-infra | 21:35 | |
| clarkb | EmilienM: I think once fungi's expiration pass is complete we will be in a better place to audit all the things more quickly and accurately | 21:35 |
| fungi | EmilienM: all that sounds great to me | 21:35 |
| *** kzaitsev_mb has joined #openstack-infra | 21:35 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Upgrade devstack-tools to 0.2.0 https://review.openstack.org/433093 | 21:35 |
| clarkb | then hopefulyl we will have more feedback for tripleo and others | 21:35 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Upgrade devstack-tools to 0.2.1 https://review.openstack.org/433093 | 21:35 |
| *** jkilpatr_ has quit IRC | 21:35 | |
| EmilienM | clarkb: ok | 21:36 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Replace "ara generate" by "ara generate html" https://review.openstack.org/433207 | 21:36 |
| fungi | we're past 0.3% free now! | 21:38 |
| *** burgerk has quit IRC | 21:38 | |
| *** rbuzatu has joined #openstack-infra | 21:40 | |
| *** esikachev has quit IRC | 21:41 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add nl01.openstack.org server https://review.openstack.org/430324 | 21:41 |
| *** dimtruck is now known as zz_dimtruck | 21:42 | |
| EmilienM | do we have alerts (email or something) that says when we reach more than 80%, etc? | 21:43 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Run zuul v3 launcher https://review.openstack.org/433347 | 21:44 |
| fungi | EmilienM: no, because we're nearly always there | 21:45 |
| *** sdake has joined #openstack-infra | 21:46 | |
| fungi | our main recourse is continually shrinking log retention | 21:46 |
| EmilienM | fungi: ok, got it | 21:46 |
| pabelanger | EmilienM: http://logs.openstack.org/00/420100/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-oooq-nv/b3365b3/logs/undercloud/var/log/oooq/ is another entry, using a large amount of data | 21:46 |
| pabelanger | EmilienM: looks to be a lot of duplicate data | 21:46 |
| EmilienM | sshnaidm: ^ | 21:46 |
| fungi | used to be 6 months, now we're only able to keep 6 weeks :/ | 21:46 |
| sshnaidm | EmilienM, yeah, I found the problem | 21:46 |
| sshnaidm | EmilienM, it's related to one of patches merged to extras today | 21:47 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Inherit playbooks and modify job variance https://review.openstack.org/430483 | 21:47 |
| pabelanger | /srv/static/logs/00/420100 alone is 6.1GB over 6 patchsets | 21:47 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Split merger and launcher git roots https://review.openstack.org/430456 | 21:47 |
| *** annegentle has joined #openstack-infra | 21:49 | |
| *** esikachev has joined #openstack-infra | 21:49 | |
| sdague | pabelanger: yeh, it looks like the whole filesystem is getting uploaded | 21:49 |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config master: networking-ovn: Reorganize CI jobs. https://review.openstack.org/394684 | 21:49 |
| sdague | on every run | 21:50 |
| mordred | related to our earlier conversation about putting an rsync limit in - I think we can safely say that 1G of log upload for a single patch is _definitely_ too much | 21:50 |
| sdague | mordred: ++ | 21:50 |
| *** eharney_ has joined #openstack-infra | 21:50 | |
| *** eharney has quit IRC | 21:50 | |
| mordred | I'm pretty sure the limit wants to be lower than that | 21:50 |
| *** eharney_ is now known as eharney | 21:50 | |
| pabelanger | interesting, stackviz is about 20MB too | 21:51 |
| pabelanger | 20M ./stackviz | 21:51 |
| pabelanger | that is from gate-tempest-dsvm-neutron-full-ubuntu-xenial | 21:51 |
| EmilienM | pabelanger: giving you a bit of background here. The issue was not directly related to regular OVB & multinode jobs but to TripleO Quickstart transition. I apologize for that, but this tooling is still experimental. The "stable" tooling doesn't consume much storage for logs, unless someone proves it the opposite | 21:51 |
| jeblair | mordred: so, somewhere between 64k and 1gb, you're saying? | 21:51 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert test_object to use .register_uris https://review.openstack.org/434532 | 21:51 |
| fungi | mordred: trying to think of how that should work to avoid hamstringing projects, maybe if there's to much then instead only upload a list of the files and their sizes (in a machine-readable format for easier investigation)? | 21:51 |
| fungi | s/to much/too much/ | 21:52 |
| mordred | fungi: ++ | 21:52 |
| jeblair | fungi: that's a good idea. and of course, always the console log. | 21:52 |
| mordred | fungi: I tihnk that's a great idea | 21:52 |
| mordred | jeblair: yah :) | 21:52 |
| fungi | right, always the console log | 21:52 |
| EmilienM | pabelanger: in other words, we'll work with TripleO QuickStart folks to improve log collection but the current OVB / multinode jobs that don't use quickstart seems to work fine and don't collect the world. | 21:52 |
| EmilienM | fungi, clarkb, mordred: I found useful to share that ^ | 21:53 |
| fungi | thanks EmilienM! | 21:53 |
| *** esikachev has quit IRC | 21:54 | |
| mordred | EmilienM: nod | 21:54 |
| EmilienM | I wouldn't blame TripleO CI as a whole, but I would just help folks working on Quickstart to get log collection better and probably re-use what has been done by devstack gate | 21:54 |
| jhesketh | Morning | 21:54 |
| fungi | looks like we're no longer losing the battle for free space, so can probably turn the tripleo cloud back on whenever? | 21:54 |
| EmilienM | we've found out that the main root cause was: https://review.openstack.org/#/c/431029/ | 21:54 |
| *** eroux has quit IRC | 21:54 | |
| *** esikachev has joined #openstack-infra | 21:55 | |
| pabelanger | fungi: 434525 is the revert | 21:56 |
| *** zz_dimtruck is now known as dimtruck | 21:56 | |
| pabelanger | it's been about 20mins since tripleo-test-cloud-rh1 has been disabled | 21:56 |
| *** dprince has quit IRC | 21:57 | |
| *** annegentle has quit IRC | 21:57 | |
| *** panda|bbl is now known as panda | 21:58 | |
| *** claudiub has joined #openstack-infra | 21:58 | |
| *** tonytan4ever has quit IRC | 21:59 | |
| *** esikachev has quit IRC | 21:59 | |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate https://review.openstack.org/434384 | 22:01 |
| *** john-davidge has joined #openstack-infra | 22:01 | |
| *** claudiub|2 has quit IRC | 22:01 | |
| *** esikachev has joined #openstack-infra | 22:02 | |
| *** Krenair has quit IRC | 22:02 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_queue_names https://review.openstack.org/429883 | 22:03 |
| *** ihrachys_ has joined #openstack-infra | 22:04 | |
| *** rwsu has quit IRC | 22:04 | |
| *** panda_ has joined #openstack-infra | 22:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove misleading log statements https://review.openstack.org/433970 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Squelch ref-replication gerrit warnings https://review.openstack.org/433959 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add Git driver https://review.openstack.org/433942 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove gearman settled check https://review.openstack.org/433943 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add (minimal) support for topic-changed event https://review.openstack.org/433966 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename zuul-server zuul-scheduler https://review.openstack.org/433980 | 22:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct getGitwebUrl https://review.openstack.org/433964 | 22:05 |
| *** panda is now known as Guest45303 | 22:05 | |
| *** ihrachys has quit IRC | 22:05 | |
| *** Guest45303 has quit IRC | 22:05 | |
| *** Goneri has quit IRC | 22:05 | |
| *** panda_ is now known as panda | 22:05 | |
| *** trown is now known as trown|outtypewww | 22:06 | |
| *** john-davidge has quit IRC | 22:06 | |
| clarkb | ok last chance to veto my ptg dinner location/time/seatcount selections | 22:06 |
| *** esikachev has quit IRC | 22:06 | |
| clarkb | I'm gonna grab a late lunch then give restaurant(s) a call | 22:06 |
| *** esikachev has joined #openstack-infra | 22:07 | |
| *** vsaienko has quit IRC | 22:08 | |
| *** Goneri has joined #openstack-infra | 22:09 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 22:09 |
| *** ihrachys_ has quit IRC | 22:10 | |
| *** esikachev has quit IRC | 22:11 | |
| *** jkilpatr_ has joined #openstack-infra | 22:12 | |
| *** EricGonczer_ has quit IRC | 22:14 | |
| *** rbuzatu has quit IRC | 22:14 | |
| *** EricGonczer_ has joined #openstack-infra | 22:14 | |
| *** ihrachys has joined #openstack-infra | 22:15 | |
| *** thorst_ has quit IRC | 22:16 | |
| r1chardj0n3s | thanks again for keeping the gates going, #infra! | 22:16 |
| *** r1chardj0n3s has left #openstack-infra | 22:16 | |
| *** gouthamr has quit IRC | 22:22 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 22:22 |
| *** ihrachys has quit IRC | 22:23 | |
| *** salv-orlando has joined #openstack-infra | 22:23 | |
| *** EricGonczer_ has quit IRC | 22:24 | |
| *** esikachev has joined #openstack-infra | 22:26 | |
| *** salv-orlando has quit IRC | 22:28 | |
| *** esikachev has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add Git driver https://review.openstack.org/433942 | 22:30 |
| *** rbuzatu has joined #openstack-infra | 22:31 | |
| *** hashar has quit IRC | 22:32 | |
| *** rhallisey has quit IRC | 22:32 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove gearman settled check https://review.openstack.org/433943 | 22:32 |
| *** masayukig is now known as masayukig_ | 22:33 | |
| *** sshnaidm is now known as sshnaidm|off | 22:34 | |
| *** vsaienko has joined #openstack-infra | 22:35 | |
| *** rbuzatu has quit IRC | 22:36 | |
| *** Krenair has joined #openstack-infra | 22:36 | |
| *** matt-borland has quit IRC | 22:43 | |
| panda | pabelanger: you there ? | 22:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data https://review.openstack.org/434481 | 22:44 |
| mordred | clarkb: when you get back from restaurants - could you do me a friendly and look at https://review.openstack.org/#/c/434428/ ? I've been trying to not bother people with shade patches - but this one is fixing gate breaks from today | 22:45 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/elastic-recheck master: Add query for OOM issue on RAX 1664953 https://review.openstack.org/434238 | 22:45 |
| mordred | (it's not a scary patch) | 22:45 |
| *** esikachev has joined #openstack-infra | 22:46 | |
| clarkb | mordred: I can, also discovering restaurants are hard | 22:46 |
| clarkb | mordred: first option says we need one check for group that big. So if you want to load up your corp card... :P | 22:46 |
| clarkb | currently trying to find phone number for other restaurant but its in a hotel so this is harder than it should be | 22:47 |
| mordred | clarkb: yah - I don't think it'll be a problem (what's our headcount?) | 22:47 |
| clarkb | mordred: 14 | 22:47 |
| mordred | yah - happy shouldn't be a problem | 22:47 |
| pabelanger | panda: some what | 22:48 |
| panda | fungi: thanks | 22:49 |
| panda | pabelanger: oh, well sorry for the mess, I just wanted to say you can wipe all the data relative to those patches | 22:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable tripleo-test-cloud-rh1 cloud" https://review.openstack.org/434525 | 22:49 |
| *** eharney has quit IRC | 22:49 | |
| pabelanger | panda: np! Thanks for looking into the issue | 22:50 |
| *** esikachev has quit IRC | 22:50 | |
| fungi | once the 45-day expiration completes (which i anticipate being sometime tomorrow) i'll do a dev ml post with a redux and current state on retention vs quantity | 22:52 |
| *** dave-mccowan has quit IRC | 22:53 | |
| *** gouthamr has joined #openstack-infra | 22:54 | |
| *** adrian_otto1 has joined #openstack-infra | 22:59 | |
| *** vsaienko has quit IRC | 22:59 | |
| clarkb | fungi: it looks like we have steady stated around 42GB free | 23:00 |
| fungi | yep | 23:00 |
| smcginnis | Is logging still an issue, or is it safe to recheck some things? | 23:00 |
| fungi | ought to grow wider as the ci load lessens for the day | 23:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Allow nl01.o.o to access zookeeper https://review.openstack.org/434550 | 23:01 |
| *** rbuzatu has joined #openstack-infra | 23:01 | |
| *** adrian_otto has quit IRC | 23:01 | |
| pabelanger | clarkb: fungi: jeblair: ^ firewall rule to allow nl01.o.o to talk to zookeeper | 23:01 |
| pabelanger | also, nl01.o.o is online | 23:01 |
| pabelanger | just waiting for puppet to run and confirm in cacti | 23:01 |
| fungi | smcginnis: not out of the woods yet, but the errors should have subsided an hour or so ago | 23:01 |
| *** salv-orlando has joined #openstack-infra | 23:02 | |
| jeblair | pabelanger: lgtm | 23:04 |
| smcginnis | fungi: Great, thanks! | 23:05 |
| *** rbuzatu has quit IRC | 23:05 | |
| *** esikachev has joined #openstack-infra | 23:07 | |
| *** _d34dh0r53_ is now known as d34dh0r53 | 23:09 | |
| *** mdrabe has quit IRC | 23:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove misleading log statements https://review.openstack.org/433970 | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Squelch ref-replication gerrit warnings https://review.openstack.org/433959 | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add (minimal) support for topic-changed event https://review.openstack.org/433966 | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename zuul-server zuul-scheduler https://review.openstack.org/433980 | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct getGitwebUrl https://review.openstack.org/433964 | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily disable test_dependent_behind_dequeue https://review.openstack.org/434553 | 23:11 |
| clarkb | mordred: shade fixes lgtm | 23:12 |
| *** esikachev has quit IRC | 23:12 | |
| mordred | clarkb: thanks! | 23:13 |
| *** spzala has quit IRC | 23:17 | |
| *** adrian_otto1 has quit IRC | 23:17 | |
| *** thorst_ has joined #openstack-infra | 23:17 | |
| *** gyee has joined #openstack-infra | 23:18 | |
| pabelanger | #status log nl01.openstack.org (nodepool-launcher) is now online. Nodepool services are disabled. | 23:18 |
| openstackstatus | pabelanger: finished logging | 23:18 |
| pabelanger | infra-root:^ | 23:18 |
| *** adrian_otto has joined #openstack-infra | 23:19 | |
| *** mriedem has quit IRC | 23:19 | |
| *** tphummel has quit IRC | 23:20 | |
| *** ihrachys has joined #openstack-infra | 23:21 | |
| *** wolverin_ has quit IRC | 23:21 | |
| *** thorst_ has quit IRC | 23:21 | |
| *** wolverineav has joined #openstack-infra | 23:22 | |
| *** ihrachys has quit IRC | 23:22 | |
| *** rbuzatu has joined #openstack-infra | 23:23 | |
| *** baoli has quit IRC | 23:23 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Temporarily disable test_dependent_behind_dequeue https://review.openstack.org/434553 | 23:23 |
| *** esikachev has joined #openstack-infra | 23:24 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Squelch ref-replication gerrit warnings https://review.openstack.org/433959 | 23:24 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Correct getGitwebUrl https://review.openstack.org/433964 | 23:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add (minimal) support for topic-changed event https://review.openstack.org/433966 | 23:25 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable test_dib_upload_fail test https://review.openstack.org/433235 | 23:26 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove misleading log statements https://review.openstack.org/433970 | 23:26 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename zuul-server zuul-scheduler https://review.openstack.org/433980 | 23:26 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable working test_builder.py tests https://review.openstack.org/433262 | 23:27 |
| *** thorst_ has joined #openstack-infra | 23:27 | |
| *** rbuzatu has quit IRC | 23:27 | |
| *** wolverineav has quit IRC | 23:28 | |
| *** esikachev has quit IRC | 23:28 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable test_diskimage_build_only test https://review.openstack.org/433265 | 23:28 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable test_image_upload_fail test https://review.openstack.org/433270 | 23:28 |
| *** masayukig_ is now known as masayukig | 23:28 | |
| *** panda is now known as panda|zZ | 23:29 | |
| *** gordc has quit IRC | 23:29 | |
| *** marst has quit IRC | 23:29 | |
| *** marst has joined #openstack-infra | 23:29 | |
| *** denisra_ has quit IRC | 23:30 | |
| *** Sukhdev has quit IRC | 23:31 | |
| *** thorst_ has quit IRC | 23:32 | |
| *** thorst_ has joined #openstack-infra | 23:32 | |
| pabelanger | still not seeing snmp requests to nl01.o.o | 23:33 |
| pabelanger | not sure why that is ATM | 23:33 |
| *** marst has quit IRC | 23:34 | |
| clarkb | did it get added as a host in cacti? | 23:34 |
| pabelanger | yup | 23:34 |
| pabelanger | I can also see nl01.o.o in /var/lib/cacti/devices | 23:35 |
| pabelanger | guess a waiting game now | 23:35 |
| *** dingyichen has joined #openstack-infra | 23:35 | |
| *** thorst_ has quit IRC | 23:37 | |
| *** masayukig is now known as masayukig_ | 23:38 | |
| clarkb | up to 47GB free | 23:38 |
| pabelanger | okay, working now | 23:38 |
| clarkb | I guess the tapering off must be happening | 23:38 |
| pabelanger | didn't wait long enough | 23:39 |
| pabelanger | clarkb: I think tomorrow I'll see about adding more space to tarballs.o.o | 23:40 |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1055&rra_id=all | 23:40 |
| pabelanger | just to be safe | 23:40 |
| clarkb | pabelanger: we can't | 23:40 |
| pabelanger | Oh | 23:40 |
| pabelanger | why is that? | 23:40 |
| clarkb | we are at our limit of attached devices and every volume is max size for individual volume | 23:41 |
| mordred | yup | 23:41 |
| clarkb | we'd have to host tarballs on a different host then move the volume(s) | 23:41 |
| pabelanger | OIC | 23:41 |
| pabelanger | k, maybe we should figure that out at PTG | 23:41 |
| mordred | mechanisms by which we could grow the storage are to move to an AFS solutoin with multiple fileservers - or to run something liek ceph on top of multiple fileservers | 23:41 |
| clarkb | pabelanger: Xen has limit of 16 devices, 2 are taken up by base VM (one is / and one is swap iirc) and then we have 14 1TB cinder volumes with lvm over the top | 23:42 |
| mordred | each of those increase the level of complexity, as we'd be putting complex network things over top of cloud complex network things | 23:42 |
| *** marst has joined #openstack-infra | 23:42 | |
| clarkb | I guess another possible fix is beg cloud provider to give us bigger volumes | 23:44 |
| clarkb | but I think they are size restricted for raisins | 23:44 |
| mordred | if we did multiple fileservers behind an AFS cell - we'd still have to allocate storage to locatoins, and that might get tedious with our dir hashing structure | 23:44 |
| clarkb | and not just arbitrarily done that way | 23:44 |
| *** sdake has quit IRC | 23:45 | |
| *** tphummel has joined #openstack-infra | 23:45 | |
| mordred | but it would have the benefit that we already know how to publish to those - if we did ceph there are more moving parts for us to deal with, but it has the benefit of presenting as single contiguous region of storage that we don't have to split between a bunch of directories | 23:45 |
| mordred | both seem like a non-zero amount of work | 23:46 |
| clarkb | mordred: do you know if ceph allows for online resize? | 23:46 |
| mordred | BUT - I agree with pabelanger - we should at least talk about it at the PTG | 23:46 |
| clarkb | seems like it should considering its block device is built on object storage | 23:46 |
| mordred | clarkb: I do not know - but I _think_ so | 23:47 |
| *** salv-orlando has quit IRC | 23:47 | |
| *** esikachev has joined #openstack-infra | 23:47 | |
| jeblair | yeah, i imagine if we used afs, we'd do something like make 100 volumes, and write a script to manage them, balancing/moving them as needed to other fileservers | 23:47 |
| mordred | jeblair: yup | 23:47 |
| jeblair | (afs is meant to scale to tens or hundreds of thousands of volumes, so that's still peanuts) | 23:48 |
| mordred | jeblair: you know - russ albery wrote that program in fortran to calculate afs volume rebalances | 23:48 |
| jeblair | mordred: i imagine it is correct. | 23:48 |
| clarkb | jeblair: just all within a millisecond of each other :) | 23:48 |
| mordred | jeblair: I cannot imagine it is incorrect | 23:48 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix missing mutex release when aborting builds https://review.openstack.org/384980 | 23:49 |
| mordred | clarkb: the nice thing about _this_ application is that it has a much more resilient consumption demands than our mirror volumes | 23:49 |
| mordred | and we won't make copies of files into caches until they're actually looked at, which tails off pretty rapidly | 23:50 |
| clarkb | ya | 23:50 |
| mordred | but - yeah - there are a bunch of pros and cons in a bunch of different directions to discuss | 23:50 |
| jeblair | i'll note that "talk about this in more detail" is pretty exactly the feedback jhesketh gave on https://review.openstack.org/269928 | 23:50 |
| clarkb | in an ideal world I think I'd ask manilla for 50TB and then when that fills up consider spring cleaning and if that isn't feasible ask it to grow to 100TB | 23:51 |
| *** esikachev has quit IRC | 23:51 | |
| mordred | clarkb: I keep waiting for the day when we live in an ideal world :) | 23:51 |
| mordred | jeblair: and yah | 23:51 |
| clarkb | I need to record norah saying "Turtle" then when I think of turtles all the way down I can play that and laugh | 23:52 |
| mordred | the nice thing is I think we're all in a better position to discuss that spec in more detail now | 23:52 |
| mordred | clarkb: hahaha | 23:52 |
| clarkb | she like rolls her els and its turtlertlertlertletle when she says it | 23:53 |
| *** spzala has joined #openstack-infra | 23:55 | |
| fungi | pabelanger: make sure to put tarballs.o.o discussion on our ptg etherpad if you haven't already | 23:55 |
| pabelanger | fungi: roger | 23:55 |
| mordred | ++ | 23:55 |
| *** rbrndt has quit IRC | 23:57 | |
| *** mhickey has quit IRC | 23:58 | |
| *** spzala has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!