| *** ijw has joined #openstack-infra | 00:01 | |
| *** itsuugo has quit IRC | 00:02 | |
| *** itsuugo has joined #openstack-infra | 00:03 | |
| *** Jeffrey4l has joined #openstack-infra | 00:03 | |
| *** esikachev has joined #openstack-infra | 00:05 | |
| *** rajinir has quit IRC | 00:05 | |
| openstackgerrit | zhurong proposed openstack-infra/project-config: Add translation jobs to karbor-dashboard https://review.openstack.org/374827 | 00:05 |
|---|---|---|
| *** sflanigan has joined #openstack-infra | 00:08 | |
| *** esikachev has quit IRC | 00:09 | |
| *** itsuugo has quit IRC | 00:09 | |
| *** yamahata has quit IRC | 00:09 | |
| *** zhurong has quit IRC | 00:10 | |
| *** itsuugo has joined #openstack-infra | 00:10 | |
| *** rkukura has quit IRC | 00:13 | |
| *** AnarchyAo has joined #openstack-infra | 00:14 | |
| *** armax has joined #openstack-infra | 00:14 | |
| *** spzala has joined #openstack-infra | 00:14 | |
| *** drawsmcgraw has joined #openstack-infra | 00:17 | |
| *** sc` has quit IRC | 00:19 | |
| *** spzala has quit IRC | 00:19 | |
| *** shadowfku has quit IRC | 00:19 | |
| anteaya | tonyb: you about? | 00:20 |
| anteaya | tonyb: could you review https://review.openstack.org/#/c/375008/1 | 00:20 |
| anteaya | tonyb: it is a patch I offered to make the proposal bot job not fail on monasca-statsd for stable/newton | 00:20 |
| *** sflanigan has quit IRC | 00:20 | |
| anteaya | tonyb: looks like someone on monasca is willing to do the work to keep it in requirements | 00:21 |
| tonyb | anteaya: Sure. | 00:21 |
| anteaya | could you advise them on their approach | 00:21 |
| anteaya | they are looking for advise and I don't want to steer them wrong | 00:21 |
| *** gildub has joined #openstack-infra | 00:21 | |
| anteaya | thanks tonyb | 00:21 |
| *** kaisers has joined #openstack-infra | 00:22 | |
| *** xarses has quit IRC | 00:22 | |
| *** kaisers2 has quit IRC | 00:22 | |
| *** kaisers1 has quit IRC | 00:23 | |
| *** tuanla has joined #openstack-infra | 00:23 | |
| *** kaisers1 has joined #openstack-infra | 00:23 | |
| *** spzala has joined #openstack-infra | 00:23 | |
| anteaya | I asked him to expedite so this didn't drag on for months and to his credit his is expediting | 00:24 |
| anteaya | s/his is/he is | 00:24 |
| anteaya | I don't actually type anymore, I just kind of mash the keyboard and hope for the best | 00:25 |
| *** amitgandhinz has joined #openstack-infra | 00:25 | |
| *** marst has joined #openstack-infra | 00:29 | |
| *** itsuugo has quit IRC | 00:30 | |
| *** amitgandhinz has quit IRC | 00:30 | |
| *** caowei has joined #openstack-infra | 00:30 | |
| *** itsuugo has joined #openstack-infra | 00:31 | |
| *** nickola has joined #openstack-infra | 00:34 | |
| fungi | that sounds like me | 00:34 |
| anteaya | :) | 00:34 |
| fungi | if i smash my face into the keyboard a bunch, sometimes words happen | 00:34 |
| anteaya | is it hard on your nose much? | 00:34 |
| fungi | not any more | 00:34 |
| anteaya | ha ha ha | 00:34 |
| anteaya | I put my face in the snow | 00:35 |
| anteaya | about a foot or so of soft powder I just stick my face in | 00:35 |
| anteaya | so quiet | 00:35 |
| *** Julien-zte has joined #openstack-infra | 00:35 | |
| anteaya | gets warm in there too | 00:35 |
| *** Apoorva_ has joined #openstack-infra | 00:36 | |
| *** Apoorva_ has quit IRC | 00:36 | |
| *** sflanigan has joined #openstack-infra | 00:37 | |
| *** sflanigan has quit IRC | 00:37 | |
| *** sflanigan has joined #openstack-infra | 00:37 | |
| anteaya | I think I've had just about as much fun as I can handle today | 00:38 |
| *** nexusz99 has joined #openstack-infra | 00:38 | |
| asselin__ | does tox -e zuul in project-config supposed to take a very long time? I see it's 11 minutes long job? | 00:38 |
| anteaya | time for some offline things | 00:38 |
| anteaya | g'night | 00:39 |
| *** Apoorva has quit IRC | 00:40 | |
| *** itsuugo has quit IRC | 00:43 | |
| fungi | asselin__: yeah, it's using jjb to parse the entire set of jobs, and then making sure all jobs listed in the zuul layout.yaml are actually defined | 00:44 |
| fungi | it's not a quick check | 00:45 |
| asselin__ | fungi, yeah...I'm used to it taking ~2 minutes...I guess upstream has a lot more jobs | 00:45 |
| fungi | last i looked it was in the vicinity of 10k job definitions | 00:45 |
| *** itsuugo has joined #openstack-infra | 00:45 | |
| *** senk has quit IRC | 00:46 | |
| asselin__ | at 2 minutes I don't mind to run it before I submit...at 10+ mins, I'll wait for the result. Probably needs some updates to make it scale better. | 00:46 |
| asselin__ | wait for CI to give me the result* | 00:46 |
| fungi | yep, i think we're mostly just living with it until zuul v3 | 00:47 |
| fungi | job definition and scheduling are more intertwined, and the same check may not be needed (or even possible) | 00:47 |
| asselin__ | ok cool..good to know this problem will just 'go away' | 00:47 |
| *** senk has joined #openstack-infra | 00:48 | |
| *** xarses has joined #openstack-infra | 00:51 | |
| *** marst has quit IRC | 00:52 | |
| *** mtanin___ has joined #openstack-infra | 00:52 | |
| *** itsuugo has quit IRC | 00:52 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Add elastic-recheck linters job https://review.openstack.org/375155 | 00:53 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Make elastic-recheck linters job voting https://review.openstack.org/375156 | 00:53 |
| *** mtanino has quit IRC | 00:53 | |
| *** itsuugo has joined #openstack-infra | 00:54 | |
| *** markvoelker has joined #openstack-infra | 00:55 | |
| *** ijw has quit IRC | 00:57 | |
| *** itsuugo has quit IRC | 00:59 | |
| *** itsuugo has joined #openstack-infra | 00:59 | |
| *** gyee has quit IRC | 01:01 | |
| *** senk has quit IRC | 01:02 | |
| *** senk__ has joined #openstack-infra | 01:02 | |
| *** jamielennox is now known as jamielennox|away | 01:03 | |
| *** esikachev has joined #openstack-infra | 01:06 | |
| *** ddieterly has joined #openstack-infra | 01:07 | |
| *** esikachev has quit IRC | 01:10 | |
| *** itsuugo has quit IRC | 01:11 | |
| *** nexusz99 has quit IRC | 01:11 | |
| *** spzala has quit IRC | 01:12 | |
| *** vinaypotluri has quit IRC | 01:12 | |
| *** itsuugo has joined #openstack-infra | 01:12 | |
| *** ddieterly has quit IRC | 01:13 | |
| *** ijw has joined #openstack-infra | 01:14 | |
| *** thcipriani is now known as thcipriani|afk | 01:15 | |
| *** zhurong has joined #openstack-infra | 01:17 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Enable py35 voting on project designate https://review.openstack.org/375186 | 01:18 |
| *** rossella_ has quit IRC | 01:19 | |
| *** rossella_ has joined #openstack-infra | 01:19 | |
| *** jamesdenton has joined #openstack-infra | 01:21 | |
| *** pahuang has quit IRC | 01:21 | |
| *** dimtruck is now known as zz_dimtruck | 01:21 | |
| *** dprince has joined #openstack-infra | 01:22 | |
| *** jamesden_ has joined #openstack-infra | 01:22 | |
| *** dprince has quit IRC | 01:22 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Enable py35 voting on project designate https://review.openstack.org/375186 | 01:22 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 01:22 |
| *** jamielennox|away is now known as jamielennox | 01:22 | |
| *** kjorgensen has quit IRC | 01:23 | |
| *** roxanaghe has joined #openstack-infra | 01:24 | |
| *** jamesdenton has quit IRC | 01:25 | |
| *** kjorgensen has joined #openstack-infra | 01:26 | |
| *** amitgandhinz has joined #openstack-infra | 01:26 | |
| *** itsuugo has quit IRC | 01:27 | |
| *** kjorgensen has quit IRC | 01:27 | |
| *** kjorgensen has joined #openstack-infra | 01:27 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Enable py35 voting on project designate https://review.openstack.org/375186 | 01:28 |
| *** itsuugo has joined #openstack-infra | 01:28 | |
| *** ijw has quit IRC | 01:28 | |
| *** amitgandhinz has quit IRC | 01:31 | |
| *** kjorgensen has quit IRC | 01:32 | |
| *** itsuugo has quit IRC | 01:33 | |
| *** Julien-zte has quit IRC | 01:34 | |
| *** tqtran has quit IRC | 01:34 | |
| *** itsuugo has joined #openstack-infra | 01:35 | |
| *** roxanaghe has quit IRC | 01:35 | |
| *** Sukhdev has joined #openstack-infra | 01:37 | |
| *** roxanaghe has joined #openstack-infra | 01:37 | |
| *** yanyanhu has joined #openstack-infra | 01:39 | |
| *** armax has quit IRC | 01:39 | |
| *** itsuugo has quit IRC | 01:39 | |
| *** itsuugo has joined #openstack-infra | 01:41 | |
| *** pvaneck has quit IRC | 01:41 | |
| *** salv-orl_ has joined #openstack-infra | 01:42 | |
| *** salv-orlando has quit IRC | 01:45 | |
| *** itsuugo has quit IRC | 01:48 | |
| *** itsuugo has joined #openstack-infra | 01:49 | |
| *** mriedem has quit IRC | 01:53 | |
| *** signed8bit_Zzz is now known as signed8bit | 01:53 | |
| *** signed8bit has quit IRC | 01:54 | |
| *** itsuugo has quit IRC | 01:56 | |
| *** itsuugo has joined #openstack-infra | 01:57 | |
| *** calq has joined #openstack-infra | 02:00 | |
| *** itsuugo has quit IRC | 02:02 | |
| *** itsuugo has joined #openstack-infra | 02:04 | |
| *** esikachev has joined #openstack-infra | 02:06 | |
| *** esikachev has quit IRC | 02:11 | |
| *** thorst has joined #openstack-infra | 02:11 | |
| *** itsuugo has quit IRC | 02:15 | |
| *** itsuugo has joined #openstack-infra | 02:16 | |
| *** mtanino has joined #openstack-infra | 02:16 | |
| *** thorst has quit IRC | 02:16 | |
| *** mtanin___ has quit IRC | 02:17 | |
| *** Sukhdev has quit IRC | 02:19 | |
| *** claudiub has quit IRC | 02:21 | |
| *** mtanino has quit IRC | 02:22 | |
| *** zz_dimtruck is now known as dimtruck | 02:24 | |
| *** rfolco has quit IRC | 02:24 | |
| *** itsuugo has quit IRC | 02:24 | |
| *** itsuugo has joined #openstack-infra | 02:26 | |
| *** amitgandhinz has joined #openstack-infra | 02:27 | |
| *** Sukhdev has joined #openstack-infra | 02:28 | |
| *** jamesden_ has quit IRC | 02:30 | |
| *** jamesdenton has joined #openstack-infra | 02:30 | |
| *** itsuugo has quit IRC | 02:31 | |
| *** itsuugo has joined #openstack-infra | 02:32 | |
| *** rfolco has joined #openstack-infra | 02:32 | |
| *** amitgandhinz has quit IRC | 02:33 | |
| *** mugsie has quit IRC | 02:33 | |
| *** mugsie has joined #openstack-infra | 02:35 | |
| *** roxanaghe has quit IRC | 02:36 | |
| *** jamielennox is now known as jamielennox|away | 02:37 | |
| *** EricGonczer_ has joined #openstack-infra | 02:37 | |
| *** Sukhdev has quit IRC | 02:38 | |
| *** itsuugo has quit IRC | 02:38 | |
| *** itsuugo has joined #openstack-infra | 02:40 | |
| *** gothicmindfood has quit IRC | 02:41 | |
| *** amotoki has quit IRC | 02:41 | |
| *** kaisers_ has joined #openstack-infra | 02:42 | |
| *** EricGonczer_ has quit IRC | 02:42 | |
| *** rfolco has quit IRC | 02:42 | |
| *** itsuugo has quit IRC | 02:45 | |
| *** sflanigan is now known as sflanigan_afk | 02:46 | |
| *** gothicmindfood has joined #openstack-infra | 02:46 | |
| *** kaisers_ has quit IRC | 02:46 | |
| *** itsuugo has joined #openstack-infra | 02:46 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use low-memory-usage.yaml in ci https://review.openstack.org/374931 | 02:47 |
| *** jamielennox|away is now known as jamielennox | 02:51 | |
| *** amotoki has joined #openstack-infra | 02:53 | |
| *** itsuugo has quit IRC | 02:53 | |
| *** itsuugo has joined #openstack-infra | 02:54 | |
| *** rajinir has joined #openstack-infra | 02:56 | |
| *** itsuugo has quit IRC | 02:58 | |
| *** itsuugo has joined #openstack-infra | 02:59 | |
| *** andrewbogott has quit IRC | 03:01 | |
| *** andrewbogott has joined #openstack-infra | 03:01 | |
| *** sflanigan_afk is now known as sflanigan | 03:01 | |
| *** itsuugo has quit IRC | 03:04 | |
| *** thorst has joined #openstack-infra | 03:04 | |
| *** david-lyle has quit IRC | 03:04 | |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Further fix for Senlin integration test https://review.openstack.org/372199 | 03:04 |
| *** itsuugo has joined #openstack-infra | 03:05 | |
| *** senk has joined #openstack-infra | 03:05 | |
| *** senk__ has quit IRC | 03:06 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Add elastic-recheck linters job https://review.openstack.org/375155 | 03:07 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Make elastic-recheck linters job voting https://review.openstack.org/375156 | 03:07 |
| *** esikachev has joined #openstack-infra | 03:07 | |
| *** drifterza has quit IRC | 03:09 | |
| *** amotoki has quit IRC | 03:09 | |
| *** itsuugo has quit IRC | 03:10 | |
| *** itsuugo has joined #openstack-infra | 03:10 | |
| *** esikachev has quit IRC | 03:12 | |
| *** itsuugo has quit IRC | 03:17 | |
| *** amotoki has joined #openstack-infra | 03:19 | |
| *** itsuugo has joined #openstack-infra | 03:19 | |
| *** itsuugo has quit IRC | 03:24 | |
| *** itsuugo has joined #openstack-infra | 03:24 | |
| *** mtanino has joined #openstack-infra | 03:26 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Undercloud install with tripleo-quickstart https://review.openstack.org/358919 | 03:26 |
| *** armax has joined #openstack-infra | 03:27 | |
| *** rfolco has joined #openstack-infra | 03:27 | |
| *** rfolco has quit IRC | 03:27 | |
| *** amitgandhinz has joined #openstack-infra | 03:28 | |
| *** itsuugo has quit IRC | 03:29 | |
| *** itsuugo has joined #openstack-infra | 03:30 | |
| *** dimtruck is now known as zz_dimtruck | 03:31 | |
| *** nickola has quit IRC | 03:32 | |
| *** amitgandhinz has quit IRC | 03:33 | |
| *** roxanaghe has joined #openstack-infra | 03:35 | |
| *** AnarchyAo has quit IRC | 03:37 | |
| *** itsuugo has quit IRC | 03:37 | |
| *** vikrant has joined #openstack-infra | 03:38 | |
| *** itsuugo has joined #openstack-infra | 03:39 | |
| *** amotoki has quit IRC | 03:39 | |
| *** mtanino has quit IRC | 03:41 | |
| *** yamamoto_ has joined #openstack-infra | 03:42 | |
| *** gouthamr has joined #openstack-infra | 03:43 | |
| *** itsuugo has quit IRC | 03:44 | |
| *** itsuugo has joined #openstack-infra | 03:45 | |
| *** Sukhdev has joined #openstack-infra | 03:46 | |
| *** rkukura has joined #openstack-infra | 03:48 | |
| *** gouthamr has quit IRC | 03:49 | |
| *** amotoki has joined #openstack-infra | 03:49 | |
| *** itsuugo has quit IRC | 03:51 | |
| *** itsuugo has joined #openstack-infra | 03:52 | |
| *** drifterza has joined #openstack-infra | 03:53 | |
| *** EricGonczer_ has joined #openstack-infra | 03:54 | |
| *** baoli_ has joined #openstack-infra | 03:54 | |
| *** ijw has joined #openstack-infra | 03:56 | |
| *** itsuugo has quit IRC | 03:57 | |
| *** cody-somerville has quit IRC | 03:58 | |
| *** itsuugo has joined #openstack-infra | 03:58 | |
| *** baoli has quit IRC | 03:58 | |
| *** EricGonczer_ has quit IRC | 04:01 | |
| *** EricGonc_ has joined #openstack-infra | 04:01 | |
| *** ijw has quit IRC | 04:01 | |
| *** sdake has quit IRC | 04:02 | |
| *** esberglu has joined #openstack-infra | 04:02 | |
| *** cody-somerville has joined #openstack-infra | 04:03 | |
| *** Julien-zte has joined #openstack-infra | 04:03 | |
| *** EricGonc_ has quit IRC | 04:03 | |
| *** marst has joined #openstack-infra | 04:04 | |
| *** drifterza has quit IRC | 04:04 | |
| *** aeng has quit IRC | 04:05 | |
| *** pgadiya has joined #openstack-infra | 04:06 | |
| *** cody-somerville has quit IRC | 04:07 | |
| *** Julien-zte has quit IRC | 04:08 | |
| *** itsuugo has quit IRC | 04:08 | |
| *** esikachev has joined #openstack-infra | 04:08 | |
| *** coolsvap has joined #openstack-infra | 04:08 | |
| *** thorst has quit IRC | 04:09 | |
| *** itsuugo has joined #openstack-infra | 04:09 | |
| *** thorst has joined #openstack-infra | 04:12 | |
| *** esberglu has quit IRC | 04:12 | |
| *** esikachev has quit IRC | 04:13 | |
| *** baoli has joined #openstack-infra | 04:14 | |
| *** itsuugo has quit IRC | 04:14 | |
| *** itsuugo has joined #openstack-infra | 04:15 | |
| *** drifterza has joined #openstack-infra | 04:15 | |
| *** baoli_ has quit IRC | 04:17 | |
| *** ijw has joined #openstack-infra | 04:17 | |
| *** ijw has quit IRC | 04:19 | |
| *** pahuang has joined #openstack-infra | 04:21 | |
| *** aeng has joined #openstack-infra | 04:22 | |
| *** thorst has quit IRC | 04:23 | |
| *** thorst has joined #openstack-infra | 04:24 | |
| *** AlexxNica has quit IRC | 04:27 | |
| *** thorst has quit IRC | 04:29 | |
| *** amitgandhinz has joined #openstack-infra | 04:29 | |
| *** baoli has quit IRC | 04:29 | |
| *** itsuugo has quit IRC | 04:30 | |
| *** itsuugo has joined #openstack-infra | 04:31 | |
| *** rkukura has quit IRC | 04:31 | |
| *** tqtran has joined #openstack-infra | 04:32 | |
| *** isviridov_away has quit IRC | 04:33 | |
| *** amitgandhinz has quit IRC | 04:34 | |
| *** vaidy has quit IRC | 04:34 | |
| *** pahuang has quit IRC | 04:35 | |
| *** Julien-zte has joined #openstack-infra | 04:36 | |
| *** tqtran has quit IRC | 04:37 | |
| *** itsuugo has quit IRC | 04:41 | |
| *** Julien-zte has quit IRC | 04:41 | |
| *** Julien-zte has joined #openstack-infra | 04:41 | |
| *** itsuugo has joined #openstack-infra | 04:42 | |
| *** narayrak has joined #openstack-infra | 04:46 | |
| *** pahuang has joined #openstack-infra | 04:52 | |
| *** tphummel has quit IRC | 04:52 | |
| *** itsuugo has quit IRC | 04:54 | |
| *** itsuugo has joined #openstack-infra | 04:55 | |
| *** thorst has joined #openstack-infra | 04:56 | |
| *** vinaypotluri has joined #openstack-infra | 04:57 | |
| *** roxanaghe has quit IRC | 04:58 | |
| *** itsuugo has quit IRC | 05:00 | |
| *** thorst has quit IRC | 05:01 | |
| *** itsuugo has joined #openstack-infra | 05:01 | |
| *** rajinir has quit IRC | 05:05 | |
| *** itsuugo has quit IRC | 05:06 | |
| *** edtubill has joined #openstack-infra | 05:06 | |
| *** itsuugo has joined #openstack-infra | 05:07 | |
| *** edtubill has quit IRC | 05:07 | |
| *** edtubill has joined #openstack-infra | 05:08 | |
| *** esikachev has joined #openstack-infra | 05:09 | |
| *** jaosorior has joined #openstack-infra | 05:10 | |
| *** senk has quit IRC | 05:11 | |
| *** edtubill has quit IRC | 05:12 | |
| *** esikachev has quit IRC | 05:13 | |
| *** sabari has quit IRC | 05:17 | |
| *** itsuugo has quit IRC | 05:19 | |
| *** edtubill has joined #openstack-infra | 05:20 | |
| *** itsuugo has joined #openstack-infra | 05:20 | |
| *** rkukura has joined #openstack-infra | 05:26 | |
| *** thorst has joined #openstack-infra | 05:27 | |
| *** itsuugo has quit IRC | 05:28 | |
| *** GordonTX has joined #openstack-infra | 05:28 | |
| *** itsuugo has joined #openstack-infra | 05:28 | |
| *** armax has quit IRC | 05:29 | |
| *** amitgandhinz has joined #openstack-infra | 05:30 | |
| *** thorst has quit IRC | 05:31 | |
| *** hashar has joined #openstack-infra | 05:32 | |
| *** armax has joined #openstack-infra | 05:32 | |
| *** armax has quit IRC | 05:34 | |
| *** aeng has quit IRC | 05:34 | |
| *** itsuugo has quit IRC | 05:34 | |
| *** itsuugo has joined #openstack-infra | 05:34 | |
| *** adriant has quit IRC | 05:35 | |
| *** amitgandhinz has quit IRC | 05:35 | |
| *** armax has joined #openstack-infra | 05:36 | |
| *** sabari has joined #openstack-infra | 05:36 | |
| *** armax has quit IRC | 05:38 | |
| *** kaisers_ has joined #openstack-infra | 05:40 | |
| *** itsuugo has quit IRC | 05:41 | |
| *** itsuugo has joined #openstack-infra | 05:42 | |
| *** thorst has joined #openstack-infra | 05:43 | |
| *** itsuugo has quit IRC | 05:47 | |
| *** itsuugo has joined #openstack-infra | 05:47 | |
| *** aeng has joined #openstack-infra | 05:48 | |
| *** thorst has quit IRC | 05:48 | |
| *** senk has joined #openstack-infra | 05:48 | |
| *** armax has joined #openstack-infra | 05:50 | |
| *** edtubill has quit IRC | 05:52 | |
| *** salv-orl_ has quit IRC | 05:57 | |
| *** itsuugo has quit IRC | 05:57 | |
| *** itsuugo has joined #openstack-infra | 05:58 | |
| *** roxanaghe has joined #openstack-infra | 05:59 | |
| *** isaacb has joined #openstack-infra | 05:59 | |
| *** thorst has joined #openstack-infra | 05:59 | |
| *** ianw is now known as ianw_pto | 06:02 | |
| *** thorst has quit IRC | 06:03 | |
| *** roxanaghe has quit IRC | 06:04 | |
| *** itsuugo has quit IRC | 06:05 | |
| *** itsuugo has joined #openstack-infra | 06:07 | |
| *** yamahata has joined #openstack-infra | 06:09 | |
| *** YorikSar has quit IRC | 06:10 | |
| *** itsuugo has quit IRC | 06:12 | |
| *** itsuugo has joined #openstack-infra | 06:12 | |
| *** thorst has joined #openstack-infra | 06:15 | |
| *** rcernin has joined #openstack-infra | 06:15 | |
| *** itsuugo has quit IRC | 06:17 | |
| yolanda | good morning | 06:18 |
| *** itsuugo has joined #openstack-infra | 06:18 | |
| AJaeger | good morning, yolanda. Did you arrive home safely? | 06:18 |
| yolanda | hi AJaeger . Yes, it went smoothly, I arrived yesterday afternoon | 06:19 |
| *** aeng has quit IRC | 06:19 | |
| yolanda | how was the talk? | 06:19 |
| *** thorst has quit IRC | 06:19 | |
| AJaeger | yolanda: full room, good questions - now they need to start not only using it but also contriburting ;) | 06:19 |
| *** claudiub has joined #openstack-infra | 06:20 | |
| yolanda | yay! | 06:21 |
| *** jamespage has quit IRC | 06:21 | |
| *** gnuoy` has quit IRC | 06:21 | |
| *** gnuoy`` has joined #openstack-infra | 06:21 | |
| *** isaacb has quit IRC | 06:21 | |
| *** jamespag` has joined #openstack-infra | 06:22 | |
| *** itsuugo has quit IRC | 06:23 | |
| *** rlu has joined #openstack-infra | 06:24 | |
| *** itsuugo has joined #openstack-infra | 06:25 | |
| *** andreas_s has joined #openstack-infra | 06:25 | |
| *** rlu has quit IRC | 06:28 | |
| *** rlu has joined #openstack-infra | 06:28 | |
| *** itsuugo has quit IRC | 06:30 | |
| *** thorst has joined #openstack-infra | 06:31 | |
| *** itsuugo has joined #openstack-infra | 06:31 | |
| *** amitgandhinz has joined #openstack-infra | 06:31 | |
| AJaeger | yolanda: could you review https://review.openstack.org/#/c/375111 , please? This is for stable/newton work | 06:34 |
| yolanda | i need to step out for school and I'll start reviews when i come back | 06:35 |
| *** itsuugo has quit IRC | 06:36 | |
| AJaeger | no problem. Have a safe trip | 06:36 |
| *** thorst has quit IRC | 06:36 | |
| *** amitgandhinz has quit IRC | 06:36 | |
| *** itsuugo has joined #openstack-infra | 06:36 | |
| openstackgerrit | Merged openstack-infra/project-config: add stable-compat jobs for tempest and devstack-gate vs. newton https://review.openstack.org/375110 | 06:37 |
| *** woodster_ has quit IRC | 06:40 | |
| *** florianf has joined #openstack-infra | 06:41 | |
| *** camunoz has quit IRC | 06:41 | |
| *** itsuugo has quit IRC | 06:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add periodic newton jobs https://review.openstack.org/375271 | 06:42 |
| *** rkukura has quit IRC | 06:42 | |
| *** itsuugo has joined #openstack-infra | 06:43 | |
| *** nherciu_ has joined #openstack-infra | 06:44 | |
| *** rkukura has joined #openstack-infra | 06:44 | |
| *** liusheng has left #openstack-infra | 06:46 | |
| openstackgerrit | Merged openstack/diskimage-builder: Allow ramdisk-create to run without $USER set https://review.openstack.org/357985 | 06:46 |
| *** thorst has joined #openstack-infra | 06:47 | |
| *** esikachev has joined #openstack-infra | 06:49 | |
| *** Ravikiran_K has joined #openstack-infra | 06:50 | |
| *** thorst has quit IRC | 06:51 | |
| *** GordonTX has quit IRC | 06:52 | |
| *** matrohon has joined #openstack-infra | 06:54 | |
| *** jlanoux has joined #openstack-infra | 06:58 | |
| *** Hal1 has joined #openstack-infra | 06:58 | |
| *** itsuugo has quit IRC | 07:00 | |
| *** itsuugo has joined #openstack-infra | 07:01 | |
| *** thorst has joined #openstack-infra | 07:02 | |
| *** salv-orlando has joined #openstack-infra | 07:03 | |
| *** martinkopec has joined #openstack-infra | 07:03 | |
| *** salv-orlando has quit IRC | 07:03 | |
| *** andymaier has joined #openstack-infra | 07:04 | |
| *** salv-orlando has joined #openstack-infra | 07:05 | |
| *** thorst has quit IRC | 07:07 | |
| *** salv-orlando has quit IRC | 07:08 | |
| *** itsuugo has quit IRC | 07:10 | |
| *** Sukhdev has quit IRC | 07:11 | |
| *** itsuugo has joined #openstack-infra | 07:11 | |
| *** esikachev has quit IRC | 07:15 | |
| *** panda|zZ is now known as panda | 07:16 | |
| *** thorst has joined #openstack-infra | 07:18 | |
| *** bauzas is now known as bauwser | 07:18 | |
| *** jlanoux_ has joined #openstack-infra | 07:20 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add network config for puppet-infracloud on hiera https://review.openstack.org/371607 | 07:21 |
| *** ihrachys has joined #openstack-infra | 07:21 | |
| *** ihrachys has quit IRC | 07:22 | |
| *** ihrachys has joined #openstack-infra | 07:22 | |
| *** jlanoux has quit IRC | 07:23 | |
| *** itsuugo has quit IRC | 07:23 | |
| *** thorst has quit IRC | 07:23 | |
| *** itsuugo has joined #openstack-infra | 07:23 | |
| *** flepied has quit IRC | 07:24 | |
| *** berendt has joined #openstack-infra | 07:24 | |
| *** itsuugo has quit IRC | 07:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Cleanup partial translated files https://review.openstack.org/375285 | 07:31 |
| AJaeger | yolanda: hope this is in your queue as well ^ - thanks! | 07:31 |
| *** itsuugo has joined #openstack-infra | 07:32 | |
| yolanda | it will benow | 07:32 |
| AJaeger | ;) | 07:32 |
| *** amitgandhinz has joined #openstack-infra | 07:32 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** amoralej|off is now known as amoralej | 07:36 | |
| *** itsuugo has quit IRC | 07:36 | |
| *** jpich has joined #openstack-infra | 07:37 | |
| *** amitgandhinz has quit IRC | 07:37 | |
| *** yaume has joined #openstack-infra | 07:38 | |
| openstackgerrit | Merged openstack-infra/project-config: add newton series jobs to the stable-compat template https://review.openstack.org/375111 | 07:38 |
| *** itsuugo has joined #openstack-infra | 07:39 | |
| *** pahuang has quit IRC | 07:42 | |
| *** itsuugo has quit IRC | 07:43 | |
| *** caowei has quit IRC | 07:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Add searchlight-ui-core to searchlight-ui ACL https://review.openstack.org/375148 | 07:44 |
| *** itsuugo has joined #openstack-infra | 07:44 | |
| *** yolanda has quit IRC | 07:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Add elastic-recheck linters job https://review.openstack.org/375155 | 07:44 |
| openstackgerrit | Merged openstack-infra/project-config: Further fix for Senlin integration test https://review.openstack.org/372199 | 07:45 |
| openstackgerrit | Merged openstack-infra/project-config: Add translation jobs to karbor-dashboard https://review.openstack.org/374827 | 07:45 |
| openstackgerrit | Merged openstack-infra/system-config: Use smartSummarize on zuul status graphs https://review.openstack.org/373384 | 07:47 |
| *** yolanda has joined #openstack-infra | 07:47 | |
| *** notnownikki has joined #openstack-infra | 07:48 | |
| openstackgerrit | Merged openstack-infra/puppet-mosquitto: Add more logging https://review.openstack.org/374154 | 07:48 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Remove not used element https://review.openstack.org/373886 | 07:48 |
| *** itsuugo has quit IRC | 07:49 | |
| *** shardy has joined #openstack-infra | 07:49 | |
| *** itsuugo has joined #openstack-infra | 07:50 | |
| *** thorst has joined #openstack-infra | 07:50 | |
| *** thorst has quit IRC | 07:54 | |
| *** makowals_ has joined #openstack-infra | 07:55 | |
| *** itsuugo has quit IRC | 07:57 | |
| *** makowals_ has quit IRC | 07:58 | |
| *** makowals has quit IRC | 07:58 | |
| *** makowals has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** itsuugo has joined #openstack-infra | 08:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Move Puppet OpenStack gate to Puppet 4 voting https://review.openstack.org/374395 | 08:00 |
| openstackgerrit | Merged openstack-infra/project-config: Demote OSA multi-region swift job from voting to non-voting https://review.openstack.org/374762 | 08:00 |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** amotoki_ has joined #openstack-infra | 08:00 | |
| *** tqtran has joined #openstack-infra | 08:01 | |
| openstackgerrit | Merged openstack-infra/project-config: set gnocchi 2.2 and 3.0 to correct branches https://review.openstack.org/374780 | 08:01 |
| *** makowals has quit IRC | 08:01 | |
| *** vinaypotluri has quit IRC | 08:02 | |
| *** asettle has joined #openstack-infra | 08:02 | |
| *** amotoki has quit IRC | 08:03 | |
| *** pilgrimstack has joined #openstack-infra | 08:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Move ironic-inspector tempest discovery job to the check pipeline https://review.openstack.org/374020 | 08:05 |
| *** tqtran has quit IRC | 08:05 | |
| *** asettle has quit IRC | 08:07 | |
| *** asettle has joined #openstack-infra | 08:09 | |
| *** asettle has quit IRC | 08:09 | |
| *** asettle has joined #openstack-infra | 08:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Move tripleo upgrades job to OVB cloud https://review.openstack.org/373406 | 08:10 |
| openstackgerrit | Merged openstack-infra/project-config: Promote OSA os_glance repo CentOS7 job to voting https://review.openstack.org/373458 | 08:11 |
| openstackgerrit | Merged openstack-infra/project-config: Implement functional tests for OSA plugins repo https://review.openstack.org/373434 | 08:12 |
| openstackgerrit | Merged openstack-infra/project-config: ooi: add functional testing to the gate https://review.openstack.org/372975 | 08:12 |
| openstackgerrit | Merged openstack-infra/project-config: Move PDNS4 job to check https://review.openstack.org/373337 | 08:13 |
| openstackgerrit | Merged openstack-infra/project-config: Change Zuul Jobs for Gluon Project https://review.openstack.org/374393 | 08:15 |
| *** GordonTX has joined #openstack-infra | 08:19 | |
| *** Ravikiran_K has quit IRC | 08:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix typos in openstack_functions.py & common_translation_update.sh https://review.openstack.org/374491 | 08:20 |
| openstackgerrit | Merged openstack-infra/project-config: Skip Senlin gate jobs for doc-only changes https://review.openstack.org/373753 | 08:20 |
| openstackgerrit | Merged openstack-infra/project-config: Bump rh1 back up to 60 max-servers https://review.openstack.org/373447 | 08:20 |
| *** thorst has joined #openstack-infra | 08:21 | |
| openstackgerrit | Merged openstack-infra/project-config: mistral: add tripleo multinode job https://review.openstack.org/370423 | 08:22 |
| openstackgerrit | Merged openstack-infra/project-config: Notify #openstack-containers for newton patches https://review.openstack.org/373398 | 08:22 |
| *** GordonTX has quit IRC | 08:24 | |
| *** amotoki_ is now known as amotoki | 08:24 | |
| *** mhickey has joined #openstack-infra | 08:25 | |
| *** Ravikiran_K has joined #openstack-infra | 08:26 | |
| *** _degorenko|afk is now known as degorenko | 08:26 | |
| *** thorst has quit IRC | 08:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Ceph Charm Shared Lib https://review.openstack.org/366842 | 08:27 |
| *** flepied has joined #openstack-infra | 08:28 | |
| *** gildub has quit IRC | 08:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable devstack job for a doc-only change for Zun https://review.openstack.org/369565 | 08:29 |
| *** jlanoux_ has quit IRC | 08:32 | |
| *** ihrachys has quit IRC | 08:33 | |
| *** isviridov_away has joined #openstack-infra | 08:33 | |
| openstackgerrit | Merged openstack-infra/project-config: add reno jobs for Magnum-UI https://review.openstack.org/372902 | 08:33 |
| *** amitgandhinz has joined #openstack-infra | 08:33 | |
| *** zhurong has quit IRC | 08:34 | |
| *** acoles_ is now known as acoles | 08:35 | |
| *** narayrak has quit IRC | 08:36 | |
| *** narayrak has joined #openstack-infra | 08:38 | |
| *** thorst has joined #openstack-infra | 08:38 | |
| *** jordanP has joined #openstack-infra | 08:38 | |
| *** amitgandhinz has quit IRC | 08:38 | |
| *** vaidy has joined #openstack-infra | 08:38 | |
| *** florianf has quit IRC | 08:41 | |
| *** thorst has quit IRC | 08:42 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:42 | |
| *** chem has joined #openstack-infra | 08:43 | |
| *** derekh has joined #openstack-infra | 08:44 | |
| *** dtantsur|afk is now known as dtantsur | 08:45 | |
| *** chem has quit IRC | 08:46 | |
| *** chem has joined #openstack-infra | 08:46 | |
| *** gnuoy`` has quit IRC | 08:46 | |
| *** esikachev has joined #openstack-infra | 08:47 | |
| *** florianf has joined #openstack-infra | 08:47 | |
| openstackgerrit | Merged openstack/diskimage-builder: elements: opensuse: Add support for openSUSE Leap https://review.openstack.org/371602 | 08:48 |
| *** asettle has quit IRC | 08:49 | |
| *** yaume has quit IRC | 08:52 | |
| *** asettle has joined #openstack-infra | 08:53 | |
| *** dizquierdo has joined #openstack-infra | 08:53 | |
| *** asettle has quit IRC | 08:54 | |
| *** asettle has joined #openstack-infra | 08:54 | |
| *** esikachev has quit IRC | 08:55 | |
| *** dtardivel has joined #openstack-infra | 08:57 | |
| *** sflanigan is now known as sflanigan_afk | 08:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/375329 | 08:59 |
| *** roxanaghe has joined #openstack-infra | 09:00 | |
| *** kaisers_ has quit IRC | 09:02 | |
| *** jamespag` is now known as jamespage | 09:02 | |
| *** ihrachys has joined #openstack-infra | 09:03 | |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** yamahata has quit IRC | 09:03 | |
| *** openstackgerrit has joined #openstack-infra | 09:03 | |
| *** roxanaghe has quit IRC | 09:05 | |
| *** makowals has joined #openstack-infra | 09:07 | |
| *** kaisers_ has joined #openstack-infra | 09:12 | |
| dtantsur | AJaeger, FYI we've got our first dsvm job successfully running on xenial, so we're going to move more jobs soon | 09:13 |
| *** yanyanhu has quit IRC | 09:13 | |
| *** esikachev has joined #openstack-infra | 09:15 | |
| *** narayrak has quit IRC | 09:15 | |
| *** narayrak has joined #openstack-infra | 09:15 | |
| openstackgerrit | Nikolay Mahotkin proposed openstack-infra/project-config: [k8s-docker] Come back noop jobs https://review.openstack.org/375341 | 09:15 |
| *** nmakhotkin has joined #openstack-infra | 09:16 | |
| *** rossella_ has quit IRC | 09:19 | |
| AJaeger | dtantsur: Woot! | 09:19 |
| *** itsuugo has quit IRC | 09:19 | |
| AJaeger | Is Nikolay Mahotkin around? | 09:19 |
| *** rossella_ has joined #openstack-infra | 09:19 | |
| *** GordonTX has joined #openstack-infra | 09:20 | |
| nmakhotkin | AJaeger: I'm here | 09:21 |
| *** itsuugo has joined #openstack-infra | 09:21 | |
| *** HeOS has joined #openstack-infra | 09:22 | |
| openstackgerrit | Denis Laxalde proposed openstack-infra/jenkins-job-builder: Fix documentation about "allow-empty-results" parameter for junit publisher https://review.openstack.org/375347 | 09:22 |
| AJaeger | nmakhotkin: thanks! Looking at your change 375341, this is no go. it will break Zuul ;( | 09:23 |
| nmakhotkin | AJaeger: Could you please explain in details what is wrong with it? | 09:23 |
| AJaeger | So, you could run a simple test - like pep8 - on these branches. Or even a real dummy test... | 09:23 |
| AJaeger | The problem is that repos that share a job, run together. So, if you submit a change and somebody else submits one which has a noop, then both will be serialized. The unrelated change will wait for yours. | 09:24 |
| AJaeger | This is intented with testing of nova, neutron etc. | 09:24 |
| AJaeger | But you don't want to have everybody else wait for any of your changes. noop means: No waiting. | 09:25 |
| AJaeger | You cannot have in one queue both a noop job and another one. That seriously hurts all repos that have noop jobs. | 09:25 |
| *** GordonTX has quit IRC | 09:25 | |
| nmakhotkin | AJaeger: ok, I thing I get it | 09:26 |
| *** itsuugo has quit IRC | 09:26 | |
| *** itsuugo has joined #openstack-infra | 09:27 | |
| nmakhotkin | then I'll define a simple job which will be run on stable branches | 09:27 |
| AJaeger | thanks! | 09:27 |
| AJaeger | make it voting, please - you need it for both queues | 09:27 |
| *** kaisers_ has quit IRC | 09:28 | |
| *** Hal1 has quit IRC | 09:29 | |
| *** Julien-zte has quit IRC | 09:30 | |
| jhesketh | Any other infra-root around or anybody know about logstash not having logs for the last few days? | 09:31 |
| *** YorikSar has joined #openstack-infra | 09:32 | |
| *** amotoki has quit IRC | 09:32 | |
| *** jlanoux has joined #openstack-infra | 09:33 | |
| *** itsuugo has quit IRC | 09:34 | |
| *** amitgandhinz has joined #openstack-infra | 09:34 | |
| *** esikachev has quit IRC | 09:34 | |
| *** itsuugo has joined #openstack-infra | 09:35 | |
| *** amitgandhinz has quit IRC | 09:40 | |
| *** kaisers_ has joined #openstack-infra | 09:42 | |
| *** panda is now known as panda|afk | 09:43 | |
| *** kaisers_ has quit IRC | 09:46 | |
| *** itsuugo has quit IRC | 09:51 | |
| *** drifterza has quit IRC | 09:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/375329 | 09:52 |
| *** gouthamr has joined #openstack-infra | 09:53 | |
| *** itsuugo has joined #openstack-infra | 09:53 | |
| *** zynzel has quit IRC | 09:53 | |
| *** amotoki has joined #openstack-infra | 09:54 | |
| *** amotoki has quit IRC | 09:55 | |
| *** tosky has joined #openstack-infra | 09:56 | |
| *** florianf has quit IRC | 09:58 | |
| *** esikachev has joined #openstack-infra | 09:59 | |
| *** kaisers_ has joined #openstack-infra | 09:59 | |
| *** itsuugo has quit IRC | 10:00 | |
| *** kaisers_ has joined #openstack-infra | 10:00 | |
| *** bmjen has joined #openstack-infra | 10:02 | |
| *** itsuugo has joined #openstack-infra | 10:02 | |
| *** kaisers__ has joined #openstack-infra | 10:03 | |
| *** Hunner has joined #openstack-infra | 10:03 | |
| *** Hunner has quit IRC | 10:03 | |
| *** Hunner has joined #openstack-infra | 10:03 | |
| *** kaisers_ has quit IRC | 10:06 | |
| jhesketh | turns out we're out of diskspace on logstash... I'm cleaning it up now | 10:06 |
| *** itsuugo has quit IRC | 10:07 | |
| *** itsuugo has joined #openstack-infra | 10:09 | |
| *** yolanda has quit IRC | 10:09 | |
| *** florianf has joined #openstack-infra | 10:10 | |
| *** sdague has joined #openstack-infra | 10:11 | |
| AJaeger | thanks, jhesketh ! | 10:12 |
| *** Hal1 has joined #openstack-infra | 10:13 | |
| *** tuanla has quit IRC | 10:15 | |
| nmakhotkin | AJaeger: what if a job on stable branch would be non-voting? | 10:18 |
| nmakhotkin | is it ok? | 10:18 |
| *** panda|afk is now known as panda | 10:22 | |
| *** kaisers__ has quit IRC | 10:23 | |
| *** itsuugo has quit IRC | 10:23 | |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 10:23 |
| *** itsuugo has joined #openstack-infra | 10:24 | |
| *** Hal1 has quit IRC | 10:24 | |
| *** Hal1 has joined #openstack-infra | 10:26 | |
| *** itsuugo has quit IRC | 10:29 | |
| *** yolanda has joined #openstack-infra | 10:30 | |
| *** itsuugo has joined #openstack-infra | 10:30 | |
| *** dtantsur is now known as dtantsur|bbl | 10:34 | |
| *** amitgandhinz has joined #openstack-infra | 10:35 | |
| *** itsuugo has quit IRC | 10:35 | |
| *** vaidy has quit IRC | 10:36 | |
| *** isviridov_away has quit IRC | 10:37 | |
| *** itsuugo has joined #openstack-infra | 10:37 | |
| *** Ravikiran_K has quit IRC | 10:39 | |
| AJaeger | it could be non-voting - but we don't have non-voting jobs in hte gate queue. So, you still would need a voting job there. Or we break best practice ;( | 10:40 |
| *** amitgandhinz has quit IRC | 10:40 | |
| *** itsuugo has quit IRC | 10:41 | |
| *** itsuugo has joined #openstack-infra | 10:42 | |
| nmakhotkin | AJaeger: I finally realized that | 10:42 |
| nmakhotkin | and then added docs job instead | 10:43 |
| nmakhotkin | thanks! | 10:43 |
| *** yamamoto_ has quit IRC | 10:43 | |
| AJaeger | ;) | 10:44 |
| *** yamamoto has joined #openstack-infra | 10:46 | |
| *** kaisers_ has joined #openstack-infra | 10:48 | |
| *** roxanaghe has joined #openstack-infra | 10:49 | |
| *** hferenc has joined #openstack-infra | 10:53 | |
| *** askb has quit IRC | 10:54 | |
| *** roxanaghe has quit IRC | 10:54 | |
| AJaeger | berendt: looking at the kolla failure: You have a bindep.txt file that installs packages that you need. That list does not include the openssl dev package | 10:54 |
| AJaeger | Was there a release yesterday that now needs those libs? | 10:54 |
| berendt | AJaeger thanks, i will have a look on it | 10:54 |
| AJaeger | berendt: let me do quickly a patch... | 10:55 |
| *** yamamoto has quit IRC | 10:55 | |
| berendt | AJaeger https://review.openstack.org/#/c/375228/ should fix this, I have not seen this patch | 10:56 |
| AJaeger | berendt: indeed, that should fix it. I'll abandon my quick one ;) | 10:56 |
| berendt | AJaeger thanks for the quick help | 10:57 |
| AJaeger | berendt: you're welcome | 10:57 |
| *** dprince has joined #openstack-infra | 10:58 | |
| *** dragonmaster has joined #openstack-infra | 10:59 | |
| *** jkilpatr has quit IRC | 11:00 | |
| *** itsuugo has quit IRC | 11:02 | |
| *** itsuugo has joined #openstack-infra | 11:03 | |
| *** lucas-afk is now known as lucasagomes | 11:05 | |
| *** itsuugo has quit IRC | 11:11 | |
| *** ccamacho|afk is now known as ccamacho | 11:11 | |
| *** itsuugo has joined #openstack-infra | 11:12 | |
| *** zhurong has joined #openstack-infra | 11:13 | |
| *** zhurong has quit IRC | 11:14 | |
| nmakhotkin | yolanda: hi! | 11:16 |
| *** itsuugo has quit IRC | 11:17 | |
| *** kaisers_ has quit IRC | 11:18 | |
| yolanda | nmakhotkin, hi | 11:18 |
| nmakhotkin | could you please help with https://review.openstack.org/#/c/375341/2 ? It is a serious blocker for us now | 11:18 |
| *** itsuugo has joined #openstack-infra | 11:19 | |
| yolanda | checking | 11:19 |
| *** dtantsur|bbl is now known as dtantsur | 11:20 | |
| nmakhotkin | yolanda: thank you so much! | 11:20 |
| *** drifterza has joined #openstack-infra | 11:22 | |
| *** rtheis has joined #openstack-infra | 11:22 | |
| *** dizquierdo has quit IRC | 11:27 | |
| *** kaisers_ has joined #openstack-infra | 11:27 | |
| *** ldnunes has joined #openstack-infra | 11:28 | |
| *** kaisers__ has joined #openstack-infra | 11:28 | |
| *** nexusz99 has joined #openstack-infra | 11:30 | |
| openstackgerrit | Merged openstack-infra/project-config: [k8s-docker] Docs job now is running on < Newton https://review.openstack.org/375341 | 11:32 |
| *** kaisers_ has quit IRC | 11:32 | |
| *** yolanda has quit IRC | 11:33 | |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 11:34 |
| *** yolanda has joined #openstack-infra | 11:34 | |
| *** amitgandhinz has joined #openstack-infra | 11:36 | |
| *** jkilpatr has joined #openstack-infra | 11:38 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 11:38 |
| *** amotoki has joined #openstack-infra | 11:38 | |
| *** asselin_ has joined #openstack-infra | 11:38 | |
| lucasagomes | hi all, the images we use for the gate testing (more specifically the ubuntu xenial) is available for download somewhere ? | 11:39 |
| *** EricGonczer_ has joined #openstack-infra | 11:40 | |
| *** amitgandhinz has quit IRC | 11:40 | |
| *** itsuugo has quit IRC | 11:41 | |
| *** asselin__ has quit IRC | 11:41 | |
| *** itsuugo has joined #openstack-infra | 11:42 | |
| *** gouthamr has quit IRC | 11:44 | |
| rcarrillocruz | lucasagomes: we don't but you can build them | 11:46 |
| rcarrillocruz | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/README.rst | 11:46 |
| lucasagomes | rcarrillocruz, that's great help! Thank you | 11:47 |
| *** itsuugo has quit IRC | 11:47 | |
| rcarrillocruz | np | 11:47 |
| *** zhurong has joined #openstack-infra | 11:47 | |
| *** itsuugo has joined #openstack-infra | 11:47 | |
| *** andreas_s has quit IRC | 11:49 | |
| *** rhallisey has joined #openstack-infra | 11:50 | |
| dhellmann | infra-core: it would be great to get this d-g change that sets up stable/newton merged today if we can: https://review.openstack.org/#/c/362438/ | 11:53 |
| openstackgerrit | Tang Chen proposed openstack-infra/project-config: Enable ironic functional test https://review.openstack.org/356233 | 11:55 |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job https://review.openstack.org/375416 | 11:56 |
| pabelanger | morning! | 11:56 |
| *** sabari has quit IRC | 11:56 | |
| *** jamesdenton has quit IRC | 11:56 | |
| pabelanger | I've given up on backscroll over the last few days | 11:56 |
| AJaeger | morning, pabelanger ! Hope you're feeling better than Tuesday? | 11:58 |
| pabelanger | AJaeger: Yup, I was better by the morning. I think the apple strudel was the fix :) | 11:59 |
| AJaeger | in combination with the Ginger Ale ;) | 12:00 |
| AJaeger | dhellmann: I gave a -1 on your change https://review.openstack.org/375092 - and pushed this one: https://review.openstack.org/375271 . Hope that concludes the changes needed | 12:00 |
| *** kaisers__ has quit IRC | 12:01 | |
| *** zhurong has quit IRC | 12:01 | |
| *** ccamacho is now known as ccamacho|lunch | 12:02 | |
| *** thorst has joined #openstack-infra | 12:03 | |
| *** jpena is now known as jpena|lunch | 12:03 | |
| Zara | nibalizer: thanks for the review on the usage patch! I'll try to fix that later tonight; might be a while before I can get to it. | 12:05 |
| *** yamamoto has quit IRC | 12:05 | |
| *** zul has quit IRC | 12:07 | |
| *** thorst has quit IRC | 12:08 | |
| *** psilvad has joined #openstack-infra | 12:08 | |
| *** raildo has joined #openstack-infra | 12:09 | |
| dhellmann | AJaeger : does your patch replace mine, or should I fix mine? | 12:09 |
| *** psilvad has joined #openstack-infra | 12:09 | |
| *** jenk_s has joined #openstack-infra | 12:10 | |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 12:10 |
| *** gouthamr has joined #openstack-infra | 12:11 | |
| *** jenk_s_ has joined #openstack-infra | 12:11 | |
| pabelanger | AJaeger: Also, I'm now using guten morgen when I now come online! | 12:11 |
| *** jenk_s_ has quit IRC | 12:12 | |
| AJaeger | dhellmann: mine complements yours - you need to update yours | 12:12 |
| dhellmann | AJaeger : ack | 12:12 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add newton bitrot jobs https://review.openstack.org/375092 | 12:13 |
| dhellmann | AJaeger : ^^ | 12:13 |
| *** asettle has quit IRC | 12:13 | |
| *** jenk_s has quit IRC | 12:14 | |
| openstackgerrit | Jiri Stransky proposed openstack/diskimage-builder: Fix grub installation for RHEL https://review.openstack.org/375430 | 12:15 |
| *** vikrant has quit IRC | 12:17 | |
| *** sc68cal_ is now known as sc68cal | 12:18 | |
| AJaeger | dhellmann: LGTM. pabelanger, do you want to +2A https://review.openstack.org/#/c/375092, please? | 12:18 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 12:18 |
| AJaeger | dhellmann: if https://review.openstack.org/375271 is fine, we can ask pabelanger to review it as well - if he has time... | 12:18 |
| *** thorst has joined #openstack-infra | 12:19 | |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/puppet-apps_site: Glare support for app-catalog https://review.openstack.org/359029 | 12:20 |
| *** mriedem has joined #openstack-infra | 12:22 | |
| *** sabari has joined #openstack-infra | 12:23 | |
| *** thorst has quit IRC | 12:24 | |
| *** esberglu has joined #openstack-infra | 12:25 | |
| mriedem | clarkb: you want to get this in now? https://review.openstack.org/#/c/362438/ | 12:26 |
| sdague | dhellmann: are you also handling branching on devstack / grenade? | 12:27 |
| mnaser | maybe not so infra related but perhaps folks here have experience - anyone worked with using jenkins (or any other ci tool) to build images? | 12:28 |
| mnaser | we're thinking of automating this by downloading vendor images (centos/ubuntu/etc), adding them, running a few tempest tests with the images and if everything is ok, mark it as public | 12:29 |
| *** thorst has joined #openstack-infra | 12:29 | |
| *** esberglu has quit IRC | 12:30 | |
| *** yamamoto has joined #openstack-infra | 12:30 | |
| mnaser | (and also, is jenkins job builder master pretty stable or better to stick with tagged releases? | 12:32 |
| *** zul has joined #openstack-infra | 12:32 | |
| *** amoralej is now known as amoralej|lunch | 12:33 | |
| pabelanger | mnaser: today we build images from scratch, but orignially, we'd download them from upstream with diskimage-builder, add some bits to let jenkins work, then run tempest on them. However, we'd mark them public first. I guess you could simply release them with our the jenkins bits once the job passed | 12:33 |
| *** pgadiya has quit IRC | 12:34 | |
| dhellmann | sdague : I can handle those, but need to know which commit to branch from | 12:35 |
| *** itsuugo has quit IRC | 12:35 | |
| sdague | dhellmann: honestly, HEAD on master is fine | 12:35 |
| sdague | for both | 12:35 |
| sdague | I'll do any fix ups after branches | 12:35 |
| *** itsuugo has joined #openstack-infra | 12:35 | |
| *** esberglu has joined #openstack-infra | 12:36 | |
| *** esberglu has quit IRC | 12:36 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job https://review.openstack.org/338088 | 12:36 |
| dhellmann | sdague : ok, if you have a few minutes to look over http://git.openstack.org/cgit/openstack/releases/tree/PROCESS.rst and suggest any additions that are missing that would be good. I'm trying to get all of this stuff written down in one place this time | 12:36 |
| sdague | I moved my +2 on the d-g change back to a +1, because I think the branches should go first | 12:36 |
| *** nickola has joined #openstack-infra | 12:36 | |
| *** roxanaghe has joined #openstack-infra | 12:36 | |
| *** amitgandhinz has joined #openstack-infra | 12:37 | |
| sdague | dhellmann: ah, yeh, so these were on that list | 12:37 |
| sdague | L156 | 12:37 |
| sdague | so yeh, devstack -> grenade -> d-g is the order we wrote down before, which I think is the right sequence | 12:38 |
| *** rcernin has quit IRC | 12:38 | |
| dhellmann | sdague : grenade done | 12:38 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job https://review.openstack.org/338088 | 12:39 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Make the ovb-updates job work again https://review.openstack.org/374406 | 12:39 |
| dhellmann | sdague : devstack done | 12:39 |
| sdague | dhellmann: great | 12:39 |
| sdague | ok, +2 on d-g change agin | 12:40 |
| *** nmakhotkin has left #openstack-infra | 12:40 | |
| *** daemontool has joined #openstack-infra | 12:40 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 12:40 |
| *** roxanaghe has quit IRC | 12:41 | |
| dhellmann | so I skipped a couple of steps it seems, ok. I'll verify the checklist I'm actually using includes all of these things | 12:41 |
| *** nexusz99 has quit IRC | 12:42 | |
| *** amitgandhinz has quit IRC | 12:42 | |
| *** nexusz99 has joined #openstack-infra | 12:42 | |
| *** itsuugo has quit IRC | 12:42 | |
| *** asettle has joined #openstack-infra | 12:43 | |
| *** pcaruana|afk| has joined #openstack-infra | 12:44 | |
| *** itsuugo has joined #openstack-infra | 12:44 | |
| *** kgiusti has joined #openstack-infra | 12:44 | |
| mnaser | thanks for that pabelanger -- we'll do a few things here and there and see where we go | 12:45 |
| *** drawsmcgraw has left #openstack-infra | 12:46 | |
| *** thorst has quit IRC | 12:46 | |
| *** nexusz99 has quit IRC | 12:47 | |
| *** sdake has joined #openstack-infra | 12:48 | |
| *** armax has quit IRC | 12:49 | |
| *** pgadiya has joined #openstack-infra | 12:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: [WIP] Don't upload DIB if provider is disabled https://review.openstack.org/375448 | 12:50 |
| *** binbincong has quit IRC | 12:51 | |
| zigo | pabelanger: nova 14.0.0~rc1-1 was rejected by the FTP masters because of missing copyright holders in debian/copyright, I've fixed it, though the Gerrit repository already has a 14.0.0_rc1-1 tag, can it be deleted so I can push a new one? | 12:51 |
| zigo | I mean, tag name is: debian/14.0.0_rc1-1 | 12:51 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: do not run puppet3 xenial jobs on stable mitaka/liberty/hammer https://review.openstack.org/375450 | 12:51 |
| openstackgerrit | Merged openstack-infra/project-config: add newton bitrot jobs https://review.openstack.org/375092 | 12:51 |
| pabelanger | zigo: we don't usually delete tags, since we cannot delete them everywhere. We usually ask projects to just increment to the next tag | 12:53 |
| zigo | pabelanger: The issue is that tags must match an upload to Debian. | 12:53 |
| zigo | Here, it wont ... | 12:53 |
| zigo | pabelanger: "everywhere" means what in this case? | 12:53 |
| zigo | pabelanger: I can retag, not a problem, just trying to understand what the issue is. | 12:54 |
| pabelanger | zigo: right, could you not re-upload the package into debian as a new tag? not and existing? | 12:54 |
| zigo | pabelanger: Tags are matching the debian version of the package. | 12:54 |
| zigo | debian/14.0.0_rc1-1 | 12:54 |
| zigo | in this case. | 12:54 |
| zigo | I can do: debian/14.0.0_rc1-1-correct-tag | 12:54 |
| zigo | Something like that ... :P | 12:54 |
| *** mriedem has quit IRC | 12:55 | |
| pabelanger | zigo: The tag in git, if we delete the tag now, by local version of the repo will still have it | 12:55 |
| zigo | pabelanger: "local" means what? | 12:55 |
| pabelanger | zigo: on my computer | 12:55 |
| zigo | It should be only me for the moment ... | 12:55 |
| zigo | I don't think anyone touched the deb-nova repo since I uploaded this morning. | 12:55 |
| zigo | Those who did can probably delete their local tag. | 12:56 |
| zigo | It wont do any "git pull" error anyway (which IMO is a flaw in Git) | 12:56 |
| zigo | I know it doesn't sound the right thing to delete the tag though. | 12:56 |
| *** david-lyle has joined #openstack-infra | 12:56 | |
| zigo | I'd understand if you completely refused. | 12:56 |
| zigo | debian/14.0.0_rc1-1-the-real-one will do! :) | 12:57 |
| *** matt-borland has joined #openstack-infra | 12:57 | |
| zigo | It's kind of my fault, I shouldn't tag something that goes through the NEW queue. | 12:57 |
| pabelanger | Right, we don't want to get into the issue with 2 tags debian/14.0.0_rc1-1 with different sha1 info. Even if you are the only one, we cannot be 100% sure. So, we just ask, increment your tag to the next possible version, and say debian/14.0.0_rc1-1 was a failed release (due to build issues) | 12:58 |
| pabelanger | zigo: great | 12:58 |
| *** pilgrimstack has quit IRC | 12:59 | |
| *** thorst has joined #openstack-infra | 12:59 | |
| zigo | pabelanger: The main issue here, is that the build script will *NOT* publish 14.0.0_rc1-1 in infra, because of that tag. | 13:00 |
| zigo | pabelanger: I'm not sure if you realize it... | 13:00 |
| zigo | It's not a so big deal, it's just a debian/copyright fix though... | 13:00 |
| *** pcaruana|afk| has quit IRC | 13:00 | |
| *** kaisers_ has joined #openstack-infra | 13:01 | |
| *** pcaruana has quit IRC | 13:01 | |
| *** pcaruana has joined #openstack-infra | 13:02 | |
| *** amitgandhinz has joined #openstack-infra | 13:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: [WIP] Don't upload DIB if provider is disabled https://review.openstack.org/375448 | 13:04 |
| *** thorst has quit IRC | 13:04 | |
| *** rtheis has quit IRC | 13:05 | |
| *** rcernin has joined #openstack-infra | 13:05 | |
| *** itsuugo has quit IRC | 13:06 | |
| *** rtheis has joined #openstack-infra | 13:06 | |
| *** itsuugo has joined #openstack-infra | 13:07 | |
| zigo | pabelanger: I thought about adding an sbuild DIB element, and now, I'm not so sure it's desirable to do it before we have a real Debian mirror with ftp-sync (because the sbuild-createchroot could fail to download some binaries, then the image would be failed). | 13:07 |
| zigo | pabelanger: Your thoughts? | 13:07 |
| zigo | pabelanger: Should I wait, or it's going to be fine to have a failure every odd days? | 13:08 |
| *** pilgrimstack has joined #openstack-infra | 13:08 | |
| *** alaski is now known as lascii | 13:09 | |
| zigo | pabelanger: And then, we'd be using the Jessie image from the previous day, or something? | 13:10 |
| pabelanger | zigo: Ya, we don't want to break our image builds in nodepool, I did have something in puppet a while back, I should find that again | 13:10 |
| pabelanger | zigo: So, here is how you could do it today | 13:11 |
| *** e0ne has joined #openstack-infra | 13:11 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: do not run puppet3 xenial jobs on stable mitaka/liberty/hammer https://review.openstack.org/375450 | 13:11 |
| zigo | Ok, I'll wait for the ftp-sync then (but maybe attempt to write the script still). | 13:11 |
| pabelanger | zigo: we could create a new periodic job to create the chroots, which uploads them to tarballs.o.o | 13:11 |
| pabelanger | then, when you do a patch build, just download the chroot from tarballs.o.o, update if needed, and go | 13:12 |
| pabelanger | that would help with build times, and least amount of changes to our base images | 13:12 |
| *** jpena|lunch is now known as jpena | 13:13 | |
| *** jaosorior has quit IRC | 13:13 | |
| *** e0ne has quit IRC | 13:14 | |
| *** jaosorior has joined #openstack-infra | 13:14 | |
| *** jamesdenton has joined #openstack-infra | 13:14 | |
| *** amitgandhinz has quit IRC | 13:14 | |
| *** amitgandhinz has joined #openstack-infra | 13:15 | |
| *** dragonmaster has quit IRC | 13:16 | |
| zigo | pabelanger: Ok, thanks for the tip. | 13:17 |
| *** inc0 has joined #openstack-infra | 13:18 | |
| *** rossella_ has quit IRC | 13:19 | |
| *** mtanino has joined #openstack-infra | 13:19 | |
| *** rossella_ has joined #openstack-infra | 13:19 | |
| *** xyang1 has joined #openstack-infra | 13:21 | |
| *** jamesdenton has quit IRC | 13:21 | |
| *** e0ne has joined #openstack-infra | 13:22 | |
| *** eharney has joined #openstack-infra | 13:22 | |
| *** jamesdenton has joined #openstack-infra | 13:23 | |
| *** pilgrimstack has quit IRC | 13:24 | |
| *** pilgrimstack has joined #openstack-infra | 13:24 | |
| *** mdrabe has joined #openstack-infra | 13:25 | |
| *** rcernin has quit IRC | 13:26 | |
| *** amitgandhinz has quit IRC | 13:26 | |
| EmilienM | yolanda, AJaeger: hey, in my patch for puppet4 yesterday, I broke stable branch CI (not critical now) and I have a fix, when you have time https://review.openstack.org/#/c/375450/ | 13:27 |
| *** amitgandhinz has joined #openstack-infra | 13:27 | |
| pleia2 | jhesketh: if you're still around... what filled up on the logstash server? (I'm now a little worried that our mqtt patch is filling the disks with error logs now since the change we added doesn't seem to work | 13:29 |
| *** amoralej|lunch is now known as amoralej | 13:30 | |
| *** acoles has quit IRC | 13:30 | |
| jhesketh | pleia2: the log-client-debug.log was filled up... it's still 25GB... (I removed an old one) | 13:31 |
| *** EricGonc_ has joined #openstack-infra | 13:31 | |
| jhesketh | it's in log-rotate so I'm not sure why it wasn't rotated | 13:31 |
| * pleia2 nods | 13:31 | |
| jhesketh | I'm going to poke at that next | 13:31 |
| pleia2 | thanks, if it is the mqtt stuff it's an easy patch to shut it off until we can debug further | 13:31 |
| *** esberglu has joined #openstack-infra | 13:32 | |
| *** sflanigan_afk has quit IRC | 13:32 | |
| *** EricGonczer_ has quit IRC | 13:32 | |
| jhesketh | I don't think so, but it looks like logs still aren't being indexed.. so I'm looking into that still | 13:33 |
| jhesketh | had hoped they'd come back as the gearman queue was huge and appears to have gone down, but results aren't coming n | 13:33 |
| pabelanger | zigo: do you have a 1 liner to create the chroot? | 13:33 |
| *** Goneri has joined #openstack-infra | 13:34 | |
| *** amitgandhinz has quit IRC | 13:35 | |
| *** amitgandhinz has joined #openstack-infra | 13:35 | |
| *** tongli has joined #openstack-infra | 13:36 | |
| *** njohnston has quit IRC | 13:36 | |
| *** ekarlso_ has joined #openstack-infra | 13:37 | |
| *** senk has quit IRC | 13:37 | |
| *** njohnston has joined #openstack-infra | 13:39 | |
| *** annegentle has joined #openstack-infra | 13:39 | |
| *** gordc has joined #openstack-infra | 13:40 | |
| *** njohnston has quit IRC | 13:40 | |
| *** tuannguyen has joined #openstack-infra | 13:40 | |
| *** njohnston has joined #openstack-infra | 13:40 | |
| *** amitgandhinz has quit IRC | 13:41 | |
| *** amitgandhinz has joined #openstack-infra | 13:42 | |
| *** tuannguyen has quit IRC | 13:43 | |
| *** tuannguyen has joined #openstack-infra | 13:44 | |
| *** baoli has joined #openstack-infra | 13:45 | |
| *** rbrndt has joined #openstack-infra | 13:45 | |
| *** baoli_ has joined #openstack-infra | 13:46 | |
| *** kaisers_ has quit IRC | 13:46 | |
| *** cardeois has joined #openstack-infra | 13:47 | |
| *** akshai has joined #openstack-infra | 13:47 | |
| *** kragniz has quit IRC | 13:48 | |
| *** kragniz has joined #openstack-infra | 13:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: [WIP] Don't upload DIB if provider is disabled https://review.openstack.org/375448 | 13:49 |
| *** baoli has quit IRC | 13:49 | |
| *** bin_ has joined #openstack-infra | 13:49 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Always configure ipv6 address with net-iso https://review.openstack.org/374922 | 13:51 |
| *** rcernin has joined #openstack-infra | 13:52 | |
| *** e0ne has quit IRC | 13:53 | |
| *** AlexxNica has joined #openstack-infra | 13:53 | |
| *** Ravikiran_K has joined #openstack-infra | 13:54 | |
| *** woodster_ has joined #openstack-infra | 13:54 | |
| *** dansmith is now known as superdan | 13:55 | |
| *** hongbin has joined #openstack-infra | 13:55 | |
| openstackgerrit | Liuqing Jing proposed openstack-infra/project-config: Use nodejs4-jobs for senlin https://review.openstack.org/375500 | 13:56 |
| *** spzala has joined #openstack-infra | 13:56 | |
| *** srobert has joined #openstack-infra | 13:57 | |
| *** GordonTX has joined #openstack-infra | 13:59 | |
| *** itsuugo has quit IRC | 14:00 | |
| *** itsuugo has joined #openstack-infra | 14:01 | |
| *** GordonTX has quit IRC | 14:01 | |
| *** amitgandhinz has quit IRC | 14:01 | |
| *** jheroux has joined #openstack-infra | 14:02 | |
| openstackgerrit | Liuqing Jing proposed openstack-infra/project-config: Use nodejs4-jobs for senlin-dashboard https://review.openstack.org/375500 | 14:02 |
| *** GordonTX has joined #openstack-infra | 14:02 | |
| *** mriedem has joined #openstack-infra | 14:02 | |
| *** thorst has joined #openstack-infra | 14:04 | |
| *** senk has joined #openstack-infra | 14:04 | |
| *** ramishra has quit IRC | 14:05 | |
| *** xarses has quit IRC | 14:05 | |
| *** ramishra has joined #openstack-infra | 14:06 | |
| *** nexusz99 has joined #openstack-infra | 14:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: [WIP] Don't upload DIB if provider is disabled https://review.openstack.org/375448 | 14:08 |
| *** sandanar has joined #openstack-infra | 14:08 | |
| *** EricGonc_ has quit IRC | 14:09 | |
| *** yamahata has joined #openstack-infra | 14:10 | |
| *** daemontool has quit IRC | 14:12 | |
| *** daemontool_ has joined #openstack-infra | 14:12 | |
| *** EricGonczer_ has joined #openstack-infra | 14:12 | |
| *** daemontool__ has joined #openstack-infra | 14:13 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/jenkins-job-builder: Allow to configure workflow job in Groovy file https://review.openstack.org/323744 | 14:13 |
| *** pgadiya has quit IRC | 14:14 | |
| *** sjmc7 has joined #openstack-infra | 14:16 | |
| *** rtheis has quit IRC | 14:16 | |
| *** daemontool_ has quit IRC | 14:17 | |
| *** Goneri has quit IRC | 14:17 | |
| *** Liuqing has joined #openstack-infra | 14:19 | |
| *** rlu has quit IRC | 14:20 | |
| *** ccamacho|lunch is now known as ccamacho | 14:20 | |
| openstackgerrit | Merged openstack/diskimage-builder: Shorten DHCP timeout in dhcp-all-interfaces https://review.openstack.org/375073 | 14:20 |
| openstackgerrit | Lukas Bednar proposed openstack-infra/jenkins-job-builder: properties: cover lockable-resources plugin https://review.openstack.org/371292 | 14:20 |
| david-lyle | fungi, do have time for me to ask some nodepool questions? fairly high leve | 14:22 |
| david-lyle | *level | 14:22 |
| *** itsuugo has quit IRC | 14:23 | |
| *** rlu has joined #openstack-infra | 14:23 | |
| *** rlu has quit IRC | 14:23 | |
| openstackgerrit | Lukas Bednar proposed openstack-infra/jenkins-job-builder: properties: cover lockable-resources plugin https://review.openstack.org/371292 | 14:24 |
| *** itsuugo has joined #openstack-infra | 14:24 | |
| sjmc7 | good morning. i just had a new ACL group created - could somebody add me to the members list please? group is searchlight-ui-core https://review.openstack.org/#/admin/groups/1584,members added in review https://review.openstack.org/#/c/375148/ | 14:25 |
| *** edtubill has joined #openstack-infra | 14:25 | |
| *** thorst has quit IRC | 14:26 | |
| *** thorst has joined #openstack-infra | 14:26 | |
| david-lyle | or anyone knowledgeable about nodepool and providing openstack clouds for testing? I just have some API requirements questions | 14:27 |
| nibalizer | david-lyle: i can try to answer, whats up? | 14:28 |
| david-lyle | nibalizer: I was reading through the requirements here http://docs.openstack.org/infra/system-config/contribute-cloud.html | 14:29 |
| david-lyle | and the source code for nodepool | 14:29 |
| david-lyle | I was curious what the volume requirement was used for | 14:30 |
| david-lyle | and a couple other questions | 14:30 |
| *** thorst has quit IRC | 14:31 | |
| david-lyle | so the volume is used to store a mirror of PyPI? | 14:32 |
| fungi | david-lyle: the ability to attach a cinder volume? we could probably get by without it if absolutely needed. we use that to give us more control over the filesystem used for our afs mirrors in each provider | 14:32 |
| *** marst has quit IRC | 14:32 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 14:32 |
| *** tuannguyen has quit IRC | 14:33 | |
| fungi | but we expected the afs caches to be larger than they turned out being | 14:33 |
| david-lyle | fungi is it just the basic CRUD around volumes then? | 14:33 |
| *** tuannguyen has joined #openstack-infra | 14:33 | |
| pabelanger | clarkb: fungi: I'll be testing ^ today for our ssh-keygen issues on debuntu | 14:33 |
| fungi | david-lyle: yeah, in fact nodepool does nothing at all with creating/attaching/detaching volumes for workers | 14:33 |
| fungi | we just manually or semi-manually create a volume to hold the afs cache when we construct the mirror server | 14:34 |
| david-lyle | ok | 14:34 |
| david-lyle | another question I see image_snapshot upload is in the nodepool code, but can't quite figure out how it's used, is that as requirement? | 14:35 |
| fungi | and like i said, we could probably just stick that on the rootfs of the mirror server instead given it doesn't grow like we thought it mightt | 14:35 |
| fungi | though having it on a separate volume does make replacing the mirror a little snappier since we don't have to run with a cold cache after | 14:36 |
| *** rtheis has joined #openstack-infra | 14:36 | |
| *** nemish__ has joined #openstack-infra | 14:38 | |
| fungi | david-lyle: we used to build a template vm, snapshot it and then boot workers from that | 14:38 |
| fungi | david-lyle: but now we build images from scratch elsewhere and upload them into glance | 14:38 |
| fungi | the old snapshot/template support is still in there for now, but will likely be dropped fairly soon | 14:38 |
| david-lyle | fungi: are the images just general OS images plus some that are reused, or one per node? | 14:39 |
| sjmc7 | AJaeger: when you get a sec could you add me as the initial member of https://review.openstack.org/#/admin/groups/1584,members please? | 14:39 |
| david-lyle | *OS image (plus some additions), | 14:39 |
| fungi | david-lyle: they're minimal images built via debootstrap, rpmstrap, et cetera using diskimage-builder | 14:40 |
| fungi | david-lyle: there are around half a dozen images total, one for each distro/version we test on | 14:41 |
| fungi | so ubuntu-trusty, ubuntu-xenial, centos-7, fedora-24, debian-jessie, et cetera | 14:42 |
| fungi | we rebuild and replace them (or try to) once a day | 14:42 |
| AJaeger | sjmc7: I can't - but others in the channel might as time permits | 14:42 |
| david-lyle | fungi: ok, that's what I assumed, wanted to make sure | 14:42 |
| sjmc7 | ok, thanks | 14:43 |
| david-lyle | fungi, last question, is there a requirement for neutron? or just a public IP per node? what I read indicates a public IP, but wanted to confirm | 14:43 |
| *** senk has quit IRC | 14:43 | |
| jhesketh | infra-root: I've been debugging logstash but haven't made a lot of progress sadly | 14:44 |
| *** sflanigan_afk has joined #openstack-infra | 14:44 | |
| jhesketh | it's possible half the services just need restarting but I was trying to better understand the problem | 14:45 |
| fungi | david-lyle: yes, public ip per node (which could be ipv6, we have some providers with no global ipv4 addresses on the workers and we use v6 there just fine). i'm pretty sure we still work with nova-net but i don't know how recently that's been tested | 14:45 |
| *** florianf has quit IRC | 14:45 | |
| jhesketh | infra-root: I've documented what I can remember here as I have to leave in a minute: https://etherpad.openstack.org/p/logstash-debugging | 14:45 |
| jhesketh | help fixing/debugging it would be appreciated | 14:45 |
| david-lyle | fungi, ok thanks for all the answers I appreciate your time | 14:46 |
| *** asettle has quit IRC | 14:46 | |
| *** acoles has joined #openstack-infra | 14:47 | |
| *** eclaire has joined #openstack-infra | 14:47 | |
| fungi | david-lyle: any time | 14:47 |
| anteaya | jhesketh: thank you | 14:47 |
| fungi | jhesketh: i'll try to pick it up in a bit if nobody else finds time | 14:49 |
| fungi | thanks for digging into it | 14:49 |
| jhesketh | fungi: cool, thanks | 14:50 |
| jhesketh | no worries... sorry I didn't get further | 14:50 |
| *** tonytan_brb has joined #openstack-infra | 14:50 | |
| *** zz_dimtruck is now known as dimtruck | 14:50 | |
| jhesketh | I'll try and be around to answer any questions etc. | 14:50 |
| anteaya | jhesketh: logstash is tricky | 14:53 |
| pleia2 | jhesketh: thanks for getting that started | 14:53 |
| *** tonytan4ever has quit IRC | 14:53 | |
| anteaya | dhellmann: you +2'd a patch that I'm being asked to abandon, would you consider it an afront if I abandon https://review.openstack.org/#/c/375008/ | 14:53 |
| pleia2 | I'm exhausted and busy with trip turnaround today (fly out again tomorrow), so I might not get to it myself, but I'll try | 14:53 |
| anteaya | rest | 14:53 |
| * clarkb can help debug but probably a good thing if others continue to do it so that more people are familiar with it | 14:54 | |
| dhellmann | anteaya : if it's not needed, it's not needed | 14:54 |
| anteaya | post travel you must rest, lest you get post travel illness | 14:54 |
| anteaya | dhellmann: very good thank you I will abandon | 14:54 |
| dhellmann | anteaya, AJaeger : is there an explanation of why it's not needed somewhere? | 14:54 |
| *** martinkopec has quit IRC | 14:54 | |
| *** florianf has joined #openstack-infra | 14:54 | |
| anteaya | tonyb gave one in his comment | 14:54 |
| dhellmann | ah, ok | 14:55 |
| jhesketh | pleia2: you should indeed rest | 14:55 |
| dhellmann | anteaya : thanks for pinging me | 14:55 |
| anteaya | dhellmann: mostly I wanted the bot run to work without errors tonyb says his test confirms | 14:55 |
| pleia2 | jhesketh: well, I feel bad since it could be the change that mtreinish and I landed that gummed things up... | 14:55 |
| jhesketh | fungi: if you or any other infra-root do take a look, if you could please add your comments to the etherpad that'd be appreciated for my learning experience | 14:55 |
| anteaya | dhellmann: sure | 14:55 |
| fungi | jhesketh: absolutely | 14:55 |
| jhesketh | ta | 14:56 |
| jhesketh | pleia2: I don't think so, but don't feel bad either way :-) | 14:56 |
| pleia2 | hehe, k | 14:57 |
| anteaya | clarkb yeah spread the logstash guruship around a bit | 14:57 |
| AJaeger | dhellmann: see the comment in the review: the monasca team backported changes to the branch so that their requirements are clean | 14:57 |
| jhesketh | looking at cacti the logstash disk has been full for some time... maybe that wasn't related: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=422&rra_id=all | 14:57 |
| dhellmann | AJaeger : got it, thanks | 14:57 |
| openstackgerrit | Merged openstack-infra/project-config: puppet: do not run puppet3 xenial jobs on stable mitaka/liberty/hammer https://review.openstack.org/375450 | 14:57 |
| AJaeger | anteaya: thanks for the change, it triggered a fast action ;) | 14:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote CentOS7 job for OSA os_cinder repo to voting https://review.openstack.org/375556 | 14:58 |
| *** amotoki has quit IRC | 14:58 | |
| AJaeger | clarkb: could you review https://review.openstack.org/365650 , please? That's for python3.5 and mirrors a change in requirements repo. | 14:58 |
| anteaya | AJaeger: it did so, I actually was really impressed | 14:59 |
| *** annegentle has quit IRC | 14:59 | |
| *** burgerk has joined #openstack-infra | 14:59 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 15:00 |
| *** annegentle has joined #openstack-infra | 15:00 | |
| AJaeger | anteaya: so was I | 15:01 |
| *** nexusz99 has joined #openstack-infra | 15:01 | |
| *** rossella_ has quit IRC | 15:01 | |
| clarkb | AJaeger: done. we will also need to update the test node for that job but this should happen first as its forward and backward compatbile | 15:02 |
| AJaeger | clarkb: that is on the proposal slave - puppet should update it. | 15:03 |
| *** jistr is now known as jistr|call | 15:03 | |
| AJaeger | clarkb: or what do you mean? | 15:04 |
| AJaeger | here's another newton change, reviews are welcome: https://review.openstack.org/375271 | 15:04 |
| clarkb | AJaeger: we need to upgrade the proposal slave to xenial before it will use python3.5 | 15:05 |
| *** EricGonczer_ has quit IRC | 15:05 | |
| fungi | clarkb: if you feel like venting, you may want to reply to http://lists.openstack.org/pipermail/openstack-dev/2016-September/104420.html | 15:05 |
| *** EricGonczer_ has joined #openstack-infra | 15:05 | |
| anteaya | does clarkb vent? | 15:06 |
| fungi | i guess we'll find outt | 15:07 |
| anteaya | I thought you were too busy shipping babies around to vent | 15:07 |
| clarkb | I made my case to dansmith already. I don't entirely like the answer but I am tiold its EHARD but nova recognizes the use case | 15:07 |
| clarkb | also that sentence was very badly constructed | 15:07 |
| *** rvasilets___ has joined #openstack-infra | 15:07 | |
| *** mat128 has quit IRC | 15:08 | |
| anteaya | you are jet lagged | 15:08 |
| AJaeger | clarkb: really? with using python3 instead of 3.5, it should work with both versions. Shouldn't it? | 15:08 |
| anteaya | you have some poorly constructed sentence points to use up | 15:08 |
| anteaya | so that was one | 15:08 |
| *** mat128 has joined #openstack-infra | 15:08 | |
| clarkb | AJaeger: it should but we don't have python3.5 on proposal.slave.o.o right now (because its trusty iirc). So just mentioning we have an additional step to do if we want that | 15:09 |
| clarkb | AJaeger: the change is good because it enables us to make that transition without breaking | 15:09 |
| *** EricGonczer_ has quit IRC | 15:10 | |
| *** thcipriani|afk is now known as thcipriani | 15:11 | |
| *** ijw has joined #openstack-infra | 15:11 | |
| *** nickola has quit IRC | 15:13 | |
| *** burgerk has quit IRC | 15:13 | |
| AJaeger | clarkb: Ah, now I get it. | 15:13 |
| *** jheroux has quit IRC | 15:14 | |
| *** amotoki has joined #openstack-infra | 15:14 | |
| *** Na3iL has joined #openstack-infra | 15:15 | |
| *** kambiz has joined #openstack-infra | 15:15 | |
| clarkb | EmilienM: question for you, are the puppet module scenario tests deploying openstack with ssl? | 15:15 |
| *** amitgandhinz has joined #openstack-infra | 15:16 | |
| EmilienM | of course | 15:16 |
| AJaeger | project-config cores, could you review https://review.openstack.org/365650 and https://review.openstack.org/375271 , please? | 15:16 |
| EmilienM | and ipv6 | 15:16 |
| clarkb | EmilienM: if so are you not running the full tempest suite? because tempest currently does not work with ssl and yet your jobs do pass | 15:16 |
| EmilienM | we run smoke | 15:16 |
| EmilienM | + some other tests | 15:16 |
| clarkb | ah ok that explains it. thanks | 15:16 |
| AJaeger | mugsie: could you review https://review.openstack.org/375186 , please? | 15:16 |
| EmilienM | https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L192-L209 | 15:16 |
| *** ijw has quit IRC | 15:16 | |
| EmilienM | clarkb: ^ | 15:16 |
| clarkb | EmilienM: we are very close to making the regular full set of jobs working with ssl though :) | 15:16 |
| *** ijw has joined #openstack-infra | 15:16 | |
| *** AlexxNica has quit IRC | 15:17 | |
| EmilienM | clarkb: what is the bug? | 15:17 |
| *** amotoki has quit IRC | 15:17 | |
| clarkb | EmilienM: current one is tempest asserts things about http headers in a swift test that are not valid to assert according to http rfc | 15:17 |
| anteaya | mtreinish: this adds neutron xenial periodic stable jobs to tempest, are you available for a +1? https://review.openstack.org/#/c/375271/1 | 15:18 |
| clarkb | EmilienM: so just need to stop tempest from doing that since the proxy terminating ssl is cleaning up the responses for us | 15:18 |
| *** mamitchl has joined #openstack-infra | 15:18 | |
| EmilienM | interesting | 15:18 |
| openstackgerrit | Merged openstack-infra/devstack-gate: update grenade settings for stable/newton https://review.openstack.org/362438 | 15:18 |
| AJaeger | armax, dougwig: Here're some neutron infra liason reviews needed, could either of you look at them, please? https://review.openstack.org/373534 https://review.openstack.org/373534 https://review.openstack.org/368945 https://review.openstack.org/363356 | 15:18 |
| AJaeger | anteaya: thanks! | 15:19 |
| clarkb | EmilienM: what service do you use to terminate ssl? haproxy or apache or nginx or ? | 15:19 |
| *** Liuqing has quit IRC | 15:19 | |
| anteaya | AJaeger: thank you | 15:19 |
| clarkb | EmilienM: may also be related to how intelligent your proxy is when it comes to cleaning up invalid http | 15:19 |
| *** jheroux has joined #openstack-infra | 15:19 | |
| *** rcernin has quit IRC | 15:19 | |
| *** xarses has joined #openstack-infra | 15:19 | |
| EmilienM | clarkb: we don't terminate SSL | 15:19 |
| AJaeger | A review for https://review.openstack.org/375285 would be appreciated as well. Thanks | 15:20 |
| EmilienM | clarkb: we run all services with native SSL support from oslo service | 15:20 |
| *** lucasagomes is now known as lucas-hungry | 15:20 | |
| clarkb | gotcha | 15:20 |
| *** esikachev has quit IRC | 15:20 | |
| EmilienM | I know some services are broken or we didn't do it yet in puppet | 15:20 |
| EmilienM | but really most of them do it | 15:20 |
| *** tonytan_brb is now known as tonytan4ever | 15:21 | |
| clarkb | I'm not completely sure why devstack decided to go with the proxy route. Probably because thats how people are actually deploying it in the real world? | 15:22 |
| *** srobert has quit IRC | 15:23 | |
| *** mhickey has quit IRC | 15:24 | |
| *** nexusz99 has quit IRC | 15:25 | |
| *** mdrabe has quit IRC | 15:25 | |
| pabelanger | clarkb: stepping out for some lunch, but once back going to be testing DIBs in infracloud for our ssh-keygen fixes | 15:26 |
| *** jistr|call is now known as jistr | 15:26 | |
| *** mhickey has joined #openstack-infra | 15:26 | |
| clarkb | pabelanger: great. did you get a trusty fix too? | 15:26 |
| *** roxanaghe has joined #openstack-infra | 15:26 | |
| *** mhickey has quit IRC | 15:26 | |
| * clarkb grumps about debian being sillty | 15:26 | |
| pabelanger | clarkb: Yup, once systemd is happy, I'll get something for upstart | 15:27 |
| pabelanger | land it in the same patchset | 15:27 |
| *** AlexxNica has joined #openstack-infra | 15:27 | |
| *** jistr is now known as jistr|biab | 15:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Promote CentOS7 job for OSA os_cinder repo to voting https://review.openstack.org/375556 | 15:29 |
| *** tphummel has joined #openstack-infra | 15:30 | |
| *** mdrabe has joined #openstack-infra | 15:31 | |
| openstackgerrit | Merged openstack-infra/project-config: requirement-constraints: Update for python3.5 https://review.openstack.org/365650 | 15:31 |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup partial translated files https://review.openstack.org/375285 | 15:31 |
| *** rajinir has joined #openstack-infra | 15:33 | |
| *** chem` has joined #openstack-infra | 15:33 | |
| *** hashar is now known as idoine | 15:33 | |
| *** senk has joined #openstack-infra | 15:33 | |
| *** chem has quit IRC | 15:34 | |
| EmilienM | clarkb: the proxy route is also fine. We do it in TripleO. In Puppet OpenStack we thought it would be more challenging to go with oslo service route | 15:34 |
| EmilienM | clarkb: and indeed, we found some bugs, reported it and they are fixed | 15:34 |
| *** roxanaghe has quit IRC | 15:34 | |
| EmilienM | clarkb: either way, puppet openstack support both | 15:35 |
| *** roxanaghe has joined #openstack-infra | 15:35 | |
| clarkb | cool. I was mostly just curious if you had somehow corrected these issues in tempest some other way but I think you mostly avopid them by running smoke (and possibly because you don't proxy_ | 15:35 |
| *** jistr|biab is now known as jistr | 15:36 | |
| *** dimtruck is now known as zz_dimtruck | 15:37 | |
| *** thorst has joined #openstack-infra | 15:39 | |
| *** thorst has quit IRC | 15:41 | |
| *** thorst has joined #openstack-infra | 15:41 | |
| *** roxanagh_ has joined #openstack-infra | 15:43 | |
| *** sandanar has quit IRC | 15:44 | |
| *** thorst has quit IRC | 15:45 | |
| *** roxanaghe has quit IRC | 15:45 | |
| *** vinaypotluri has joined #openstack-infra | 15:45 | |
| *** chem`` has joined #openstack-infra | 15:45 | |
| *** thorst has joined #openstack-infra | 15:46 | |
| *** chem` has quit IRC | 15:46 | |
| *** narayrak has quit IRC | 15:46 | |
| *** edmondsw has quit IRC | 15:46 | |
| AJaeger | anteaya: mtreinish might still be travelling. The jobs in https://review.openstack.org/375271 are the usual jobs, this is the same pattern we do for mitaka and liberty as well. | 15:47 |
| * AJaeger will be back later | 15:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run newton to master/ocata grenade on xenial https://review.openstack.org/375582 | 15:48 |
| clarkb | fungi: dhellmann ^ there is that change. Passed locally for me. Will work on PROCESS update too | 15:48 |
| nibalizer | fungi: fyi http://paste.openstack.org/show/582779/ looks like bluebox is gonna do some non-workload-impacting maintenance stuffs | 15:48 |
| *** yamahata has quit IRC | 15:49 | |
| *** tuannguyen has quit IRC | 15:49 | |
| *** flepied has quit IRC | 15:49 | |
| *** edmondsw has joined #openstack-infra | 15:49 | |
| mriedem | we don't run trusty nodes for any ocata (master) patches now right? | 15:50 |
| mriedem | clarkb: ^? | 15:50 |
| *** dtantsur is now known as dtantsur|afk | 15:50 | |
| clarkb | mriedem: it depends on the job since we decided to do the migration on a per job basis | 15:51 |
| dhellmann | clarkb : I'm having trouble reconciling the comment you have there with the regexes | 15:51 |
| *** chem`` has quit IRC | 15:51 | |
| clarkb | nova swift glance keystone are all very xenialed | 15:51 |
| clarkb | neutron cinder not completely because they have a lot more random jorbs | 15:51 |
| mriedem | ok | 15:51 |
| clarkb | dhellmann: I was trying to explain what we want there. Its the "new" side that determines the instance type because that grenade job runs against the new side if that makes sense | 15:52 |
| clarkb | dhellmann: so if the job is against newton that means we are doing mitaka to newton and should run on trusty | 15:52 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 15:53 |
| dhellmann | clarkb : it looks like the regex on line 1928 is saying that stable/newton should not match, and therefore not run the xenial job. Which part of that am I misreading? (I trust your intent and not my interpretation of the config settings) | 15:53 |
| dhellmann | ah! | 15:53 |
| dhellmann | yes, ok, that's the bit I wasn't getting | 15:53 |
| dhellmann | I was thinking old, not new | 15:53 |
| dhellmann | maybe add that detail to the comment? | 15:53 |
| *** thorst has quit IRC | 15:53 | |
| clarkb | sure | 15:54 |
| dhellmann | so right now we don't run any grenade jobs on xenial? | 15:54 |
| dhellmann | oh, well, we do master, don't we | 15:54 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run newton to master/ocata grenade on xenial https://review.openstack.org/375582 | 15:55 |
| *** thorst has joined #openstack-infra | 15:55 | |
| clarkb | dhellmann: we don't until we merge this change | 15:55 |
| dhellmann | alright, I think I understand this now | 15:55 |
| clarkb | dhellmann: I hope that comment update makes it easier to udnerstand? | 15:55 |
| dhellmann | clarkb : it does. you could also say explicitly "the regex pattern matches against the branch grenade considers 'new'" | 15:56 |
| openstackgerrit | Merged openstack-infra/project-config: Enable py35 voting on project designate https://review.openstack.org/375186 | 15:56 |
| *** thorst has quit IRC | 15:56 | |
| *** thorst has joined #openstack-infra | 15:57 | |
| *** bnemec is now known as beekneemech | 15:58 | |
| *** jpena is now known as jpena|off | 15:58 | |
| *** nexusz99 has joined #openstack-infra | 15:58 | |
| EmilienM | pabelanger: is the rubygem mirror still in our radar ? https://review.openstack.org/#/c/253616/ | 15:58 |
| *** matrohon has quit IRC | 15:58 | |
| *** Goneri has joined #openstack-infra | 15:58 | |
| *** jpena|off is now known as jpena | 15:59 | |
| *** senk has quit IRC | 15:59 | |
| *** mdrabe has quit IRC | 16:01 | |
| *** spzala has quit IRC | 16:01 | |
| *** mdrabe has joined #openstack-infra | 16:01 | |
| *** kjorgensen has joined #openstack-infra | 16:02 | |
| *** senk has joined #openstack-infra | 16:02 | |
| EmilienM | clarkb: question for you now. What version of puppet to you use to deploy Infra Cloud ? | 16:02 |
| clarkb | EmilienM: puppet 3. something | 16:02 |
| anteaya | AJaeger: okay, I do like to ask though | 16:02 |
| *** gongysh has joined #openstack-infra | 16:03 | |
| *** thorst has quit IRC | 16:03 | |
| EmilienM | clarkb: I'm asking because starting from stable/newton and beyond, we gate on puppet4 | 16:03 |
| clarkb | ya I saw that but your email said you would continue to also test puppet 3 | 16:03 |
| EmilienM | we support puppet3 for stable/mitaka | 16:03 |
| EmilienM | yes | 16:03 |
| fungi | nibalizer: thanks, so nothing that'll impact nodepool as i read it. good deal | 16:03 |
| clarkb | the problem is puppet 3 to 4 is a major transition iirc | 16:03 |
| clarkb | and we have more than just infra cloud that we puppet | 16:03 |
| EmilienM | but in Ocata, we'll remove puppet3 support I think | 16:03 |
| EmilienM | we need to move forward one day | 16:03 |
| clarkb | sure | 16:04 |
| clarkb | when is puppet end of lifing puppet 3? | 16:04 |
| *** tqtran has joined #openstack-infra | 16:04 | |
| EmilienM | I don't think we are in a hurry of anything but I want to know what is blocking people to upgrade | 16:04 |
| *** thorst has joined #openstack-infra | 16:04 | |
| *** thorst has quit IRC | 16:04 | |
| EmilienM | clarkb: I'm not sure. Hunner ^ ? | 16:04 |
| clarkb | EmilienM: I think what is blocking us is a requirement to rewrite most of our puppet | 16:04 |
| EmilienM | have you tried to deploy infra with puppet4? | 16:05 |
| EmilienM | we started with experimental jobs and iterate on it until it passed CI | 16:05 |
| *** dprince has quit IRC | 16:05 | |
| fungi | there's a lot of conditionals in there that boil down to "if '' {" | 16:06 |
| *** spzala has joined #openstack-infra | 16:06 | |
| nibalizer | fungi: yea sounds like they're just taking their management layer thing offline for a bit | 16:06 |
| EmilienM | I'm sure your manifests will work with puppet4 module some little changes | 16:06 |
| *** yamamoto has quit IRC | 16:06 | |
| fungi | EmilienM: so we'll at least need refactoring in places to deal with the change in boolean casting for empty strings | 16:06 |
| clarkb | EmilienM: the problem is we can't choose to run with different versions | 16:06 |
| clarkb | so literaly everythjing hasd to be updated first | 16:06 |
| nibalizer | i want to say pabelanger told me he'd validated puppet4 | 16:06 |
| clarkb | its a major pita and makes me grumpy puppet did this too us after 2 to 3 | 16:06 |
| *** Apoorva has joined #openstack-infra | 16:07 | |
| clarkb | nibalizer: the slave modules work | 16:07 |
| *** thorst has joined #openstack-infra | 16:07 | |
| fungi | we need to rewrite to make everything work the same in both puppet 3 and puppet 4, and then switch it all from 3 to 4 in a big bang | 16:07 |
| clarkb | sincewe puppet 4 on fedora | 16:07 |
| nibalizer | im not super sure we'd need to big bang now, since we have puppet apply | 16:07 |
| nibalizer | the openstack_project module, yes | 16:07 |
| EmilienM | clarkb: I agree | 16:07 |
| fungi | the openstack_project module is basically our entrypoint to all the other modules though | 16:07 |
| fungi | so we need them all working correctly with puppet 3 and 4 before we can switch openstack_project from 3 to 4 | 16:08 |
| nibalizer | but it is included on the slaves I think, so if pabelanger is saying slaves work on puppet4, then we have that working | 16:08 |
| EmilienM | if infra needs help on 3 to 4 transition, I would be happy to spend little time on it | 16:08 |
| clarkb | nibalizer: we do have that working | 16:08 |
| *** tqtran has quit IRC | 16:08 | |
| nibalizer | so we could do it piece by piece | 16:08 |
| clarkb | nibalizer: no | 16:08 |
| EmilienM | clarkb: do you have syntax/unit puppet4 jobs running on infra modules? | 16:08 |
| EmilienM | we started by that :) | 16:09 |
| clarkb | nibalizer: you still have to switch everything all at once | 16:09 |
| nibalizer | clarkb: why? | 16:09 |
| fungi | nibalizer: well, there are a handful of classes in openstack_project that are used in building node images, so that much of openstack_project is working at least, yes | 16:09 |
| *** yamamoto has joined #openstack-infra | 16:09 | |
| clarkb | nibalizer: you can patch piece by piece but we arent going to make review.o.o 3 and zuul.o.o 4 | 16:09 |
| nibalizer | yah thats all im saying | 16:09 |
| *** yamamoto has quit IRC | 16:09 | |
| fungi | we'd need a fair amount of additional logic to run different puppet versions on different servers arbitrarily | 16:10 |
| nibalizer | im not sure we would fungi | 16:10 |
| nibalizer | id like to think we could get away with just ansible runs whatever puppet it finds on the filesystem | 16:10 |
| clarkb | yes | 16:10 |
| *** senk has quit IRC | 16:10 | |
| EmilienM | what version of OpenStack is running Infra Cloud ? IIRC it was Liberty but I'm not sure | 16:10 |
| sigmavirus | Is it still okay to include the meeting bot in team channels for when they want to have impromptu meetings of subteams? | 16:10 |
| fungi | at a start, we'd need markers in the global site manifest for which puppet version to use on which nodes, so that we can test them correctly | 16:10 |
| clarkb | nibalizer: but you need something to pick that version.. | 16:10 |
| clarkb | EmilienM: mitaka now iirc | 16:11 |
| clarkb | sigmavirus: we have in the past said no so that meetings are discoverable | 16:11 |
| EmilienM | ok | 16:11 |
| *** jlanoux has quit IRC | 16:11 | |
| fungi | nibalizer: yes we could cowboy it without much added logic, i'm saying to acrually test that we won't break we need a lot more logic in our current jobs | 16:11 |
| EmilienM | clarkb: and do you know when ~ upgrade to newton is targeted? | 16:11 |
| clarkb | EmilienM: I don't | 16:11 |
| nibalizer | its an annoying amount of work to upgrade to puppet4 | 16:12 |
| nibalizer | for like, zero value | 16:12 |
| nibalizer | which is why I haven't pushed for it at all | 16:13 |
| *** AlexxNica has quit IRC | 16:13 | |
| anteaya | sigmavirus: also any bot maintenance takes our meeting channels into account, not meetings we can't know about | 16:13 |
| anteaya | like restarts and such | 16:13 |
| *** thorst has quit IRC | 16:13 | |
| EmilienM | nibalizer: zero value? | 16:14 |
| sigmavirus | anteaya: clarkb thank you. I thought this was okay in the past but my memory is clearly wrong :) | 16:14 |
| anteaya | sigmavirus: people have done it against our advice in the past | 16:14 |
| *** thorst has joined #openstack-infra | 16:14 | |
| sigmavirus | =( | 16:14 |
| mtreinish | fungi, pleia2, nibalizer, jhesketh: ugh, looks like mosquitto died again... | 16:14 |
| EmilienM | have we evaluated the amount of work by testing it and listing all puppet failures? | 16:15 |
| *** Swami has joined #openstack-infra | 16:15 | |
| nibalizer | EmilienM: https://pbs.twimg.com/media/CrDsHi0UIAA9-nU.jpg | 16:15 |
| nibalizer | mtreinish: womp womp | 16:16 |
| nibalizer | mtreinish: firehose01.openstack.org? | 16:16 |
| mtreinish | yeah | 16:16 |
| clarkb | mtreinish: can yopu review https://review.openstack.org/#/c/374328/3 ? it makes tempest work with ssl | 16:16 |
| mtreinish | clarkb: I can, one sec | 16:16 |
| EmilienM | nibalizer: I see but what happens if OpenStack community wants to include puppet4 features into Puppet OpenStack modules? | 16:17 |
| *** AlexxNica has joined #openstack-infra | 16:17 | |
| mtreinish | clarkb, greghaynes: +2 | 16:17 |
| greghaynes | :D | 16:17 |
| *** cody-somerville has joined #openstack-infra | 16:18 | |
| *** armax has joined #openstack-infra | 16:18 | |
| *** jistr is now known as jistr|afk | 16:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Add periodic newton jobs https://review.openstack.org/375271 | 16:18 |
| anteaya | mtreinish: if you can kill mostiquos you would be a hero round my area | 16:18 |
| anteaya | take you on every camping trip I would | 16:19 |
| anteaya | who's that guy? | 16:19 |
| anteaya | he is killing mostiquos | 16:19 |
| anteaya | leave him be and take him his dinner | 16:19 |
| mtreinish | anteaya: haha, well this one seems to be suicidal. We can't keep it running for more than a day | 16:19 |
| anteaya | ha ha ha | 16:19 |
| mtreinish | it must be because I announced the firehose on the ML | 16:20 |
| anteaya | that's what we need suicial mostiquos | 16:20 |
| anteaya | stage fright | 16:20 |
| anteaya | teach it breathing exercises | 16:20 |
| *** salv-orlando has joined #openstack-infra | 16:20 | |
| *** jaosorior has quit IRC | 16:21 | |
| fungi | mtreinish: i was going to ask about that... i was streaming from it, disconnected for a moment to run something else on the terminal, and then tried to reconnect and got connection refusal thereafter | 16:21 |
| *** AlexxNica has quit IRC | 16:21 | |
| *** nexusz99 has quit IRC | 16:21 | |
| mtreinish | fungi: yeah, pleia2 and I were looking at that in walldorf. We never found an answer, since the logging is at a pretty consistent level (meaning it's not helpful) | 16:22 |
| *** spzala has quit IRC | 16:22 | |
| fungi | constantly unhelpful logging is the best logging | 16:22 |
| fungi | or, in this case, only logging | 16:22 |
| mtreinish | fungi: I did push a patch to add all the log levels to maybe provide some insight: https://review.openstack.org/#/c/374154/ | 16:23 |
| mtreinish | but I wasn't holding my breath | 16:23 |
| *** yamamoto has joined #openstack-infra | 16:24 | |
| *** nherciu_ has quit IRC | 16:24 | |
| fungi | mtreinish: i think it may have been killed when logrotate reloaded it? | 16:24 |
| fungi | mtreinish: at least there's a mosquitto.log.1.gz which ends with "1474611901: Reloading config." | 16:25 |
| mtreinish | fungi: oh, that could be, and would be super annoying | 16:25 |
| fungi | mtreinish: and no /var/log/mosquitto/mosquitto.log | 16:25 |
| *** pvaneck has joined #openstack-infra | 16:26 | |
| clarkb | you can tell logrotate to copy the file over then truncate the log | 16:26 |
| clarkb | probably what you need there | 16:26 |
| fungi | which suggests to me that it died while reloading its configuration after receiving sighup, and didn't get as far as opening a new inode for the log | 16:26 |
| mtreinish | fungi: I think /var/log/mosquitto/mosquitto.log was there before when pleia2 and I looked at the earlier crashes. But I'm not sure | 16:26 |
| fungi | mtreinish: there's an old crufty /var/log/mosquitto.log from august, before we updated the log path | 16:27 |
| mtreinish | yeah, we probably should delete that one :) | 16:27 |
| fungi | which i'll go ahead and delete so nobody gets confused | 16:27 |
| fungi | it's hard enough to read epoch time | 16:27 |
| nibalizer | mtreinish: i have restarted mosquitto | 16:29 |
| fungi | thanks nibalizer | 16:29 |
| *** yamamoto has quit IRC | 16:29 | |
| nibalizer | and it seems to be still running | 16:29 |
| *** lucas-hungry is now known as lucasagomes | 16:29 | |
| *** spzala has joined #openstack-infra | 16:29 | |
| *** cardeois has quit IRC | 16:29 | |
| *** yamahata has joined #openstack-infra | 16:29 | |
| nibalizer | yah the unix seconds logging is annoying | 16:29 |
| Hunner | clarkb: puppet 3 is eol end of December I think? | 16:30 |
| fungi | i saw we keep an eye on this, and if the future failure causes have consistent symptoms we either try to figure out how to improve mosquitto to handle log rotation better, or as clarkb suggests change to a hackier logrotate method | 16:30 |
| fungi | s/saw/say/ | 16:30 |
| mtreinish | nibalizer: cool, thanks | 16:30 |
| fungi | but at least this time around it looks like it choked during log rotation | 16:30 |
| *** thorst has quit IRC | 16:31 | |
| mtreinish | fungi, clarkb: I can push a patch to something to flip the logrotate setting if you point me in the right direction | 16:31 |
| *** thorst has joined #openstack-infra | 16:31 | |
| *** pilgrimstack has quit IRC | 16:31 | |
| clarkb | mtreinish: I think logstash may use the setting look for copytruncatr iirc | 16:32 |
| clarkb | er copytruncate | 16:32 |
| nibalizer | ssl errors in the logs now | 16:32 |
| nibalizer | Hunner: link to announce? | 16:32 |
| *** gongysh has quit IRC | 16:32 | |
| mtreinish | nibalizer: hmm, maybe from the logstash workers? | 16:32 |
| fungi | mtreinish: clarkb: i would have expected to find the logrotate module in use in puppet-mosquitto, but maybe the distro package is setting that up on its own? | 16:33 |
| mtreinish | which might explain why I wasn't seeing any messages from there before we left | 16:33 |
| *** thorst has quit IRC | 16:33 | |
| mtreinish | fungi: yeah, I assumed it was ubuntu | 16:33 |
| nibalizer | http://paste.openstack.org/show/582792/ | 16:33 |
| *** berendt has quit IRC | 16:33 | |
| clarkb | fungi: our puppet stuff is set per file | 16:33 |
| nibalizer | fungi: missed you in germany! | 16:33 |
| mtreinish | I know I didn't explicitly set anything up with logrotate | 16:33 |
| Hunner | nibalizer: https://puppet.com/misc/puppet-enterprise-lifecycle/ | 16:33 |
| nibalizer | mtreinish: clarkb had safe and uneventful returns? | 16:33 |
| clarkb | so you'd need a new puppet define for eachfile to rotate | 16:33 |
| fungi | nibalizer: thanks! hope you had a few liters in my honor | 16:33 |
| *** sc` has joined #openstack-infra | 16:33 | |
| clarkb | nibalizer: yup just tired | 16:34 |
| *** spzala has quit IRC | 16:34 | |
| nibalizer | Hunner: thats all puppet enterprise | 16:34 |
| Hunner | nibalizer: I think the foss lifecycle matches it | 16:34 |
| mtreinish | nibalizer: yeah, it was a straightforward trip back. Definitely glad I paid for premium economy though. It's so nice being able to fit in the seats | 16:35 |
| * fungi wonders what the puppet 3 fork that everyone ends up using after puppet 3 reaches eol will be named | 16:35 | |
| *** spzala has joined #openstack-infra | 16:35 | |
| mtreinish | nibalizer: oh and new top speed on the drive back to the airport, 234 kph | 16:35 |
| nibalizer | noice | 16:35 |
| nibalizer | autobahhhhn | 16:35 |
| *** makowals has quit IRC | 16:36 | |
| fungi | mtreinish: i have to say, while my height isn't advantageous for much, it's nice to still be reasonably comfortable in the modern micro-coach seating most airlines have switched to | 16:36 |
| *** jordanP has quit IRC | 16:36 | |
| *** thorst has joined #openstack-infra | 16:36 | |
| *** Hal1 has quit IRC | 16:36 | |
| nibalizer | mtreinish: yea looks like ssl errors related to the logstash workers? or its just saying its connected to the logstash workers, and also getting ssl errors from something else unrelatedly | 16:37 |
| Hunner | nibalizer: https://groups.google.com/forum/#!topic/puppet-dev/jkcCNSk9w4w | 16:37 |
| rajinir | Can I manually re-run ./safe-devstack-vm-gate-wrap.sh to debug a failed build? Pre_test_hook is failing and I want to debug it | 16:37 |
| mtreinish | nibalizer: it's probably a config issue (or a bug in the mqtt plugin) on the logstash workers | 16:37 |
| mtreinish | nibalizer: they are configured to use ssl so we don't send the service user password in plain text | 16:37 |
| mtreinish | but it looks like the ssl handshake is failing | 16:38 |
| *** gyee has joined #openstack-infra | 16:38 | |
| mtreinish | which is why there aren't any logstash messages on the firehose | 16:38 |
| mtreinish | pleia2: ^^^ that explains the problem from wed. | 16:38 |
| fungi | rajinir: you _might_ be able to, but we don't test that it works correctly. we delete and use a fresh vm every time we run | 16:38 |
| *** sjmc7 has left #openstack-infra | 16:38 | |
| clarkb | rajinir: you can if you copy all the setup code before it. we do however piblish a reproduce.sh script for every devstack gate run which makes this much eaiser | 16:39 |
| clarkb | I would just copy and run that | 16:39 |
| nibalizer | Hunner: fun | 16:39 |
| rajinir | clarkb:fungi: Just found the reproduce.sh | 16:39 |
| wznoinsk | hi infra | 16:39 |
| *** spzala has quit IRC | 16:39 | |
| rajinir | clarkb:fungi: yep, seems to work ...thanks | 16:39 |
| Hunner | nibalizer: It would be nice if it mentioned that foss matches on that page... | 16:39 |
| anteaya | dhellmann: can you read the last comment on this patch? https://review.openstack.org/#/c/375008/ | 16:39 |
| mtreinish | nibalizer: I wouldn't expect anyone else to be using ssl, since you have to give the CA cert to the client explicitly and I haven't written the docs on how to do that yet | 16:39 |
| fungi | mtreinish: pleia2: is it configured to use the correct port number? does it trust the server cert? | 16:40 |
| pabelanger | EmilienM: On the radar yes, I want to get it online for barcelona | 16:40 |
| anteaya | dhellmann: I wanted to get some activity, seems I might have created a panic | 16:40 |
| wznoinsk | could someone help me understand what happened here -> https://review.openstack.org/#/c/374970/ <-, I see +2s despite changing the files and rebasing after I got them | 16:40 |
| anteaya | dhellmann: panic was not my intent | 16:40 |
| EmilienM | pabelanger: ++ | 16:40 |
| EmilienM | let me know if I can help | 16:40 |
| nibalizer | clarkb: Hunner so dec 31 looks like | 16:41 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 16:41 |
| clarkb | is this the new puppet business model? | 16:41 |
| rajinir | clarkb:fungi: spoke too soon, looks like I have to copy setup code, which is a bit complicated | 16:41 |
| mtreinish | fungi: it should be the right port number: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/logstash_worker.pp | 16:41 |
| clarkb | make backward incompatible changes then eol aggressively? | 16:41 |
| nibalizer | i wonder what the plan is here | 16:41 |
| mtreinish | fungi: port 8883 is the ssl'd mqtt port | 16:41 |
| nibalizer | http://packages.ubuntu.com/xenial/puppet | 16:41 |
| clarkb | rajinir: you shouldnt need to with reproduce.sh | 16:41 |
| mtreinish | fungi: but I'm not sure about the cert trusting. | 16:41 |
| nibalizer | maybe ubuntu will keep a patched 3.x around for us | 16:41 |
| mtreinish | I've got to step away for a little while though | 16:42 |
| mtreinish | I'll bbl | 16:42 |
| clarkb | rajinir: I have been using reproduce.sh a lot lately and it mostly just works.make sure you run it with sudo ability for the user | 16:42 |
| anteaya | mtreinish: go kill more mostquitos | 16:42 |
| clarkb | I think the comment at the beginning of the file documents that | 16:42 |
| rajinir | clarkb: Its in the logs folder. so is that right one? | 16:42 |
| *** pvaneck_ has joined #openstack-infra | 16:43 | |
| clarkb | yes there is only one | 16:43 |
| *** pvaneck has quit IRC | 16:44 | |
| *** amitgandhinz has quit IRC | 16:44 | |
| fungi | pabelanger: when you're caught up, slagle has revived the "status of tripleo-test-cloud-rh1" thread from august, and it looks like we now have some options for redundant tripleo testing environments, which means instead of switching to third-party it could operate in a full upstream mode (no longer needing separate pipelines, with actual voting/gating) | 16:45 |
| fungi | i'm personally in favor of any method that ends the years of limbo it's been in | 16:46 |
| fungi | whether that's third-party ci or finally being fully integrated | 16:46 |
| *** thorst has quit IRC | 16:47 | |
| pabelanger | fungi: right, I still want to push for 3rd party CI. Which I think we are talking about next week or so. I don't think we should bring it into a full upstream mode, as we should do that with RDO cloud (coming online in the next few months) | 16:47 |
| pabelanger | fungi: but yes, I'm hoping to have a good plan in place for Barcelona and start working on it | 16:48 |
| *** thorst has joined #openstack-infra | 16:48 | |
| *** pvaneck has joined #openstack-infra | 16:48 | |
| slagle | pabelanger: is there solid timing on the RDO cloud? | 16:48 |
| anteaya | is RDO cloud a potential new host cloud? | 16:48 |
| slagle | i've kind of pushed that to the back of the mind b/c the schedule keeps getting pushed out, so i dont feel we can plan reliably around it | 16:48 |
| pabelanger | slagle: I've heard rumblings for the next few months | 16:48 |
| fungi | pabelanger: the main concern with bringing any provider with specialized support for specific jobs online in nodepool is that there needs to be multi-environment redundancy so that the primary ci pipelines don't back up when one provider is offline due to maintenance or network et cetera failures | 16:48 |
| *** pvaneck_ has quit IRC | 16:48 | |
| pabelanger | anteaya: potential, nothing official yet | 16:49 |
| *** ccamacho is now known as ccamacho|out | 16:49 | |
| fungi | pabelanger: so if rdo cloud will be more than one region/cloud then sounds like a cool option. though the tripleo cloud also sounds like it's got that option today | 16:49 |
| anteaya | pabelanger: very good, thank you | 16:50 |
| pabelanger | What I have heard, is tripleo cloud hardware is eventually being absorbed into the new cloud, but I need to see if that is happening or not | 16:50 |
| slagle | pabelanger: eventually that is supposed to happen | 16:51 |
| *** spzala has joined #openstack-infra | 16:51 | |
| slagle | the timing around that is very much up in the air though, and i've been told now we have the existing hardware for as long as we say we need/want it | 16:51 |
| fungi | pabelanger: the other wrinkle is that we're looking at a governance change to move the tripleo-ci deliverable from the infra team to the tripleo team, and i was hoping to first make sure there's a plan for what is actually happening with the ci before pushing forward on that front | 16:51 |
| pabelanger | fungi: right, I would say another 2 weeks before we need to move on that. Some of the people are away next week for a conference. | 16:52 |
| *** pvaneck has quit IRC | 16:52 | |
| fungi | a lot of us will be away for a conference not long after that ;) | 16:53 |
| pabelanger | Yar | 16:53 |
| pabelanger | The good news, is I think everybody is on the same page. We just need to validate that, and start making plans | 16:53 |
| *** thorst has quit IRC | 16:54 | |
| *** jpena is now known as jpena|off | 16:54 | |
| *** cardeois has joined #openstack-infra | 16:54 | |
| pabelanger | I still need to read and comment on the post slagle did this week. | 16:54 |
| *** nexusz99 has joined #openstack-infra | 16:54 | |
| *** thorst has joined #openstack-infra | 16:55 | |
| *** spzala has quit IRC | 16:55 | |
| *** derekh has quit IRC | 16:56 | |
| *** abregman has joined #openstack-infra | 16:56 | |
| fungi | pabelanger: okay, thanks. i'm open to good arguments in either direction, but since you've been a de-facto liaison for us on this i wanted to wait for you to weigh in first | 16:56 |
| pabelanger | fungi: sure, I'll ready today and get comments in by first of next week | 16:56 |
| *** spzala has joined #openstack-infra | 16:56 | |
| *** AlexxNica has joined #openstack-infra | 16:57 | |
| fungi | slagle: ^ i didn't want you to think i'm ignoring your ml post, just didn't want to jump into it while most of the team is in transit/sleeping off their sprint travel | 16:57 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Add libselinux-python as a dependency on RHEL https://review.openstack.org/373401 | 16:57 |
| slagle | fungi: yea, np :) | 16:58 |
| slagle | i will be travelling next week myself, but will make time to keep the conversation going | 16:58 |
| *** vhosakot has joined #openstack-infra | 16:58 | |
| *** jistr|afk is now known as jistr | 16:59 | |
| fungi | thanks! | 16:59 |
| *** bethwhite_ has quit IRC | 16:59 | |
| *** amitgandhinz has joined #openstack-infra | 16:59 | |
| *** makowals has joined #openstack-infra | 17:00 | |
| *** spzala has quit IRC | 17:01 | |
| *** markvoelker has quit IRC | 17:02 | |
| *** markvoelker has joined #openstack-infra | 17:02 | |
| *** spzala has joined #openstack-infra | 17:02 | |
| *** AnarchyAo has joined #openstack-infra | 17:04 | |
| *** inc0 has quit IRC | 17:04 | |
| *** ccamacho|out has quit IRC | 17:05 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on SREG extension https://review.openstack.org/375627 | 17:06 |
| *** trown is now known as trown|lunch | 17:06 | |
| *** spzala has quit IRC | 17:07 | |
| *** makowals has quit IRC | 17:07 | |
| *** inc0 has joined #openstack-infra | 17:08 | |
| *** thorst has quit IRC | 17:10 | |
| *** thorst has joined #openstack-infra | 17:10 | |
| *** kaisers_ has joined #openstack-infra | 17:11 | |
| *** kaisers_ has quit IRC | 17:11 | |
| *** kaisers_ has joined #openstack-infra | 17:11 | |
| *** edtubill has quit IRC | 17:12 | |
| *** kzaitsev_mb has quit IRC | 17:13 | |
| *** esikachev has joined #openstack-infra | 17:13 | |
| *** ihrachys has quit IRC | 17:13 | |
| *** thorst has quit IRC | 17:15 | |
| *** thorst has joined #openstack-infra | 17:16 | |
| *** thorst has quit IRC | 17:16 | |
| *** nexusz99 has quit IRC | 17:17 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on SREG extension https://review.openstack.org/375627 | 17:17 |
| *** degorenko is now known as _degorenko|afk | 17:19 | |
| *** pvaneck has joined #openstack-infra | 17:20 | |
| *** spzala has joined #openstack-infra | 17:20 | |
| *** thorst has joined #openstack-infra | 17:21 | |
| *** daemontool__ has quit IRC | 17:21 | |
| *** Apoorva has quit IRC | 17:21 | |
| *** senk has joined #openstack-infra | 17:22 | |
| *** thorst has quit IRC | 17:23 | |
| *** thorst has joined #openstack-infra | 17:23 | |
| *** spzala has quit IRC | 17:24 | |
| *** Rockyg has quit IRC | 17:25 | |
| *** ihrachys has joined #openstack-infra | 17:25 | |
| *** spzala has joined #openstack-infra | 17:26 | |
| *** yamamoto has joined #openstack-infra | 17:26 | |
| dhellmann | anteaya : I don't understand why these 2 things are related at all. I need to get roland on irc so we can discuss it. | 17:27 |
| *** rvasilets___ has quit IRC | 17:28 | |
| anteaya | he is rhochmuth | 17:28 |
| anteaya | he pm'd me not sure what channels he is in, he isn't in here | 17:28 |
| dhellmann | I hunted for him earlier and didn't find him online | 17:28 |
| *** jpich has quit IRC | 17:29 | |
| anteaya | yeah so my motivation was cleanup, this project wasnt' in the form to get its bit passing in the proposal bot job | 17:29 |
| anteaya | so I offered the patch | 17:29 |
| anteaya | I think he thinks that the repo might be removed from newton or something | 17:29 |
| anteaya | rather than removed from requirements | 17:29 |
| anteaya | which is useless for it to be in there at this point since it isn't serving any purpose | 17:29 |
| anteaya | should I send him to -release if I find him? | 17:30 |
| *** senk has quit IRC | 17:30 | |
| *** spzala has quit IRC | 17:30 | |
| *** spzala has joined #openstack-infra | 17:32 | |
| *** yamamoto has quit IRC | 17:32 | |
| fungi | put another way, repos shouldn't be added to the projects.txt file in the requirements repo _until_ they have requirements check jobs running and passing. otherwise the requirements proposal job will fail some basic sanity checks and never propose a requirements update to them (and we'll see the job listed as failing, though in actuality that's just there for signaling the issue and is mostly benign | 17:32 |
| fungi | as the job skips repos it can't propose to and still does the rest) | 17:32 |
| clarkb | dtantsur|afk: jroll http://logs.openstack.org/74/372374/15/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ssh-tinyipa-nv/7dae090/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-22_19_59_43_498 ran against a devstack deployed with ssl in front of glance | 17:33 |
| anteaya | fungi: agreed | 17:33 |
| clarkb | dtantsur|afk: jroll the normal nova job worked fine which makes me think maybe something in the ironic driver there is unhappy with ssl? any chance you can poke at that? (feel free to ask me questions about the job setup) | 17:33 |
| *** thorst has quit IRC | 17:34 | |
| clarkb | also might try to see if you can make the error be a bit mroe verbose. Could not connect to glance but what url was attempted and how did it fail? | 17:34 |
| *** vhosakot has quit IRC | 17:34 | |
| *** thorst has joined #openstack-infra | 17:34 | |
| *** tosky has quit IRC | 17:35 | |
| *** akshai has quit IRC | 17:35 | |
| *** tuannguyen has joined #openstack-infra | 17:35 | |
| *** ihrachys has quit IRC | 17:36 | |
| *** florianf has quit IRC | 17:36 | |
| *** gouthamr has quit IRC | 17:36 | |
| *** gouthamr_ has joined #openstack-infra | 17:36 | |
| *** spzala has quit IRC | 17:36 | |
| *** akshai has joined #openstack-infra | 17:37 | |
| *** thorst has quit IRC | 17:38 | |
| *** thorst has joined #openstack-infra | 17:39 | |
| *** gouthamr has joined #openstack-infra | 17:40 | |
| *** vhosakot has joined #openstack-infra | 17:40 | |
| pabelanger | think we might be having some networking issues with infracloud-vanilla | 17:40 |
| pabelanger | unable to ssh into controller00.vanilla.ic.openstack.org | 17:41 |
| pabelanger | rcarrillocruz: ^ | 17:41 |
| *** pvaneck has quit IRC | 17:41 | |
| *** gouthamr_ has quit IRC | 17:41 | |
| *** csomerville has joined #openstack-infra | 17:43 | |
| *** spzala has joined #openstack-infra | 17:44 | |
| *** thorst has quit IRC | 17:45 | |
| *** artmr has quit IRC | 17:45 | |
| *** cody-somerville has quit IRC | 17:46 | |
| *** thorst has joined #openstack-infra | 17:46 | |
| clarkb | dtantsur|afk: jroll ok I think I have tracked this down to ironic setting urls for services in its config instead of using the keystone catalog... | 17:47 |
| *** gouthamr has quit IRC | 17:47 | |
| clarkb | dtantsur|afk: jroll I am going to attempt to hack around that in devstack but maybe ironic should ask keystone instead? | 17:47 |
| *** florianf has joined #openstack-infra | 17:47 | |
| *** gouthamr_ has joined #openstack-infra | 17:48 | |
| *** cardeois has quit IRC | 17:49 | |
| *** cardeois_ has joined #openstack-infra | 17:49 | |
| *** gouthamr_ has quit IRC | 17:49 | |
| *** gouthamr has joined #openstack-infra | 17:50 | |
| *** nexusz99 has joined #openstack-infra | 17:50 | |
| *** tqtran has joined #openstack-infra | 17:50 | |
| *** gyee has quit IRC | 17:50 | |
| pleia2 | pabelanger: just got an email that seems to confirm that (email was general, but I suspect we're impacted) | 17:52 |
| *** spzala has quit IRC | 17:52 | |
| *** amoralej is now known as amoralej|off | 17:52 | |
| *** thorst has quit IRC | 17:53 | |
| *** esikachev has quit IRC | 17:54 | |
| fungi | that's unfortunate | 17:54 |
| fungi | pleia2: planned or unplanned (from their perspective)? | 17:54 |
| *** spzala has joined #openstack-infra | 17:54 | |
| *** thorst has joined #openstack-infra | 17:54 | |
| *** notnownikki has quit IRC | 17:54 | |
| *** nexusz99 has quit IRC | 17:55 | |
| pleia2 | fungi: unplanned, and unclear (they're investigating) | 17:55 |
| fungi | pleia2: thanks | 17:56 |
| *** psachin has joined #openstack-infra | 17:56 | |
| *** bin_ has quit IRC | 17:57 | |
| *** dtardivel has quit IRC | 17:57 | |
| clarkb | sdague: I am discovering that devstack plugins are going to make ssl tricky :) | 17:59 |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Update 'usage' https://review.openstack.org/372454 | 17:59 |
| clarkb | sdague: basically its on the plugin to do all this stuff correctly when we enable ssl in devstack proper. Not a bad thing but will require churn in a bunch of places likely | 17:59 |
| Zara | nibalizer ^ hopefully that takes care of your concern | 18:00 |
| *** thorst has quit IRC | 18:01 | |
| clarkb | zaro: looks like when you use the built in editor in gerrit if you change the commit message it shows those chagnes in the main change page before you publish the edit | 18:01 |
| *** krtaylor has quit IRC | 18:01 | |
| *** thorst has joined #openstack-infra | 18:01 | |
| clarkb | zaro: not sure if it does that for all browser sessions or just the one that edited ti. Do you know if that is an issue in newer gerrit? | 18:01 |
| *** flepied has joined #openstack-infra | 18:02 | |
| *** itsuugo has quit IRC | 18:03 | |
| *** thorst has quit IRC | 18:04 | |
| *** itsuugo has joined #openstack-infra | 18:04 | |
| *** trown|lunch is now known as trown | 18:06 | |
| *** pvaneck has joined #openstack-infra | 18:06 | |
| *** thorst has joined #openstack-infra | 18:07 | |
| fungi | clarkb: my guess would be it's a state being presented only to the editing client (similar to draft comments not yet saved) | 18:07 |
| *** rhallisey has quit IRC | 18:08 | |
| fungi | though maybe the ui could do a better job of indicating it's doing that | 18:08 |
| jroll | clarkb: we landed support for that in newton but haven't updated devstack yet: https://github.com/openstack/ironic/commit/f9ea26ebf33118cfc179cc183588df2a829db4b6 | 18:08 |
| clarkb | ya I think it should. But not sure if this is already fixed or not | 18:08 |
| clarkb | considering all the work on ui things there | 18:08 |
| clarkb | jroll: ya I just pushed a fix to the devstack plugin that I think will work | 18:09 |
| clarkb | jroll: see https://review.openstack.org/#/c/375649/1/devstack/lib/ironic and https://review.openstack.org/372374 's ironic job will test it | 18:09 |
| *** kzaitsev_mb has joined #openstack-infra | 18:09 | |
| *** acoles is now known as acoles_ | 18:09 | |
| anteaya | AJaeger: my patch remains in the abandoned state | 18:09 |
| AJaeger | anteaya: as native speaker, do you have any comment on https://review.openstack.org/374614 , please (nothing urgent) ? | 18:09 |
| AJaeger | anteaya: great! ;) | 18:09 |
| *** salv-orlando has quit IRC | 18:10 | |
| *** HeOS has quit IRC | 18:10 | |
| *** notnownikki has joined #openstack-infra | 18:10 | |
| anteaya | AJaeger: are you kidding me | 18:10 |
| anteaya | adds added imports imported | 18:11 |
| *** psachin has quit IRC | 18:11 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 18:11 |
| anteaya | as long as what you do are doing did do is accurate that is the important thing | 18:11 |
| jroll | clarkb: cool, we plan to start using the catalog in ocata | 18:11 |
| jroll | thanks for the fix | 18:11 |
| *** florianf has quit IRC | 18:12 | |
| anteaya | AJaeger: my mind boggles at why someone thinks this is important | 18:12 |
| AJaeger | ;) | 18:12 |
| AJaeger | anteaya: I wondered whether a native speaker would say that this is far better or refuse to participate in a bikeshet ;) | 18:14 |
| AJaeger | shed? | 18:14 |
| anteaya | put me down for refuse to participate | 18:14 |
| *** armax_ has joined #openstack-infra | 18:14 | |
| AJaeger | then I'll leave a comment and let's see what happens... | 18:14 |
| anteaya | have fun | 18:15 |
| *** armax has quit IRC | 18:15 | |
| *** armax_ is now known as armax | 18:15 | |
| anteaya | I can't bring myself to comment personally | 18:15 |
| anteaya | can't even look at that a second time | 18:15 |
| *** thorst has quit IRC | 18:16 | |
| *** armax has quit IRC | 18:16 | |
| *** rkukura has quit IRC | 18:16 | |
| *** itsuugo has quit IRC | 18:16 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add experimental networking-ovn multinode job https://review.openstack.org/373528 | 18:16 |
| *** thorst has joined #openstack-infra | 18:16 | |
| AJaeger | sorry, anteaya | 18:17 |
| AJaeger | thanks for your comment, that helped me. | 18:17 |
| anteaya | AJaeger: no no no, don't apologize | 18:17 |
| anteaya | yup, I understand why you asked me and it is valid | 18:17 |
| anteaya | I'm just stating that some folks remain a mystery to me | 18:18 |
| anteaya | and will continue to do so | 18:18 |
| rcarrillocruz | pabelanger: yes indeed, i cannot even login to HipChat | 18:18 |
| *** itsuugo has joined #openstack-infra | 18:18 | |
| rcarrillocruz | just saw the email about the Houston lab outage | 18:18 |
| AJaeger | anteaya: ;) | 18:18 |
| clarkb | jroll: also we are really close to having tls-proxy work for the services in devstack itself. You may want ot look at using devstack/lib/tls to deploy ironic that way too | 18:19 |
| anteaya | AJaeger: :) | 18:19 |
| *** sflanigan_afk has quit IRC | 18:20 | |
| *** tphummel has quit IRC | 18:20 | |
| *** vhosakot has quit IRC | 18:21 | |
| jroll | clarkb: does that go through apache I guess? that's been on our list | 18:22 |
| pabelanger | rcarrillocruz: pleia2: looks like infracloud is back online | 18:22 |
| *** thorst has quit IRC | 18:22 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run newton to master/ocata grenade on xenial https://review.openstack.org/375582 | 18:23 |
| clarkb | AJaeger: ^ now with more ) | 18:23 |
| clarkb | jroll: yup it used to use stud and was never tested then stud became abandon ware so greghaynes just switched it to apache which is already used all over devstack | 18:23 |
| jroll | ah nice | 18:23 |
| AJaeger | clarkb: let's try it ;) | 18:24 |
| clarkb | (my personal agenda here is a few weeks ago osc + neutron stopped working against https endpoints so I really want to make ssl our defautl so that if osc breaks we know before we release) | 18:24 |
| *** Ravikiran_K has quit IRC | 18:25 | |
| *** tuannguyen has quit IRC | 18:27 | |
| *** thorst has joined #openstack-infra | 18:28 | |
| *** zz_dimtruck is now known as dimtruck | 18:28 | |
| *** inc0 has quit IRC | 18:29 | |
| *** e0ne has joined #openstack-infra | 18:29 | |
| *** thorst has quit IRC | 18:31 | |
| *** itsuugo has quit IRC | 18:31 | |
| *** thorst has joined #openstack-infra | 18:32 | |
| *** itsuugo has joined #openstack-infra | 18:32 | |
| AJaeger | clarkb: enough ")" this time ;) | 18:33 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 18:33 |
| anteaya | yay enough ")"s | 18:34 |
| AJaeger | just for the fun of it, I'll share another one here ;) | 18:35 |
| sigmavirus | AJaeger: I think that review to change the verbs in the commit messages is because of OpenStack's commit message style guide | 18:35 |
| sigmavirus | But few enough projects follow that for it not to be a defacto standard anymore | 18:35 |
| mtreinish | pleia2, fungi, nibalizer: I think germqtt and lpmqtt also crashed because mosquitto was down for long enough the retry connection stuff failed | 18:35 |
| *** inc0 has joined #openstack-infra | 18:35 | |
| AJaeger | sigmavirus: I would never -1 a change for that ;) | 18:36 |
| sigmavirus | AJaeger: there are projects that do | 18:36 |
| pleia2 | mtreinish: yeah, just brought them both back | 18:36 |
| sigmavirus | Which I suspect is their prerogative because it's in their HACKING files but still | 18:36 |
| mtreinish | pleia2: thanks | 18:36 |
| sigmavirus | AJaeger: that's my guess at least | 18:36 |
| anteaya | sigmavirus: thanks, I'm still not reviewing that patch | 18:37 |
| *** amitgandhinz has quit IRC | 18:37 | |
| *** itsuugo has quit IRC | 18:37 | |
| sigmavirus | anteaya: yeah, that's why I didn't mention you directly. Didn't want to tag you back into this conversation since you didn't seem to want to be part of it. | 18:37 |
| *** andymaier has quit IRC | 18:38 | |
| AJaeger | sigmavirus: i left a comment out of respect for the submission - I'm not opposed to it but neither care enough to push it through. | 18:38 |
| sigmavirus | AJaeger: if you want someone else's opinion on it I can comment, but I don't know how to vote | 18:38 |
| sigmavirus | AJaeger: right, I can understand why some projects might care, but I don't think enough do to warrant it | 18:38 |
| *** itsuugo has joined #openstack-infra | 18:38 | |
| sigmavirus | Which I guess makes me -1 | 18:38 |
| AJaeger | sigmavirus: feel free to leave a comment without voting - like I did. | 18:38 |
| *** tphummel has joined #openstack-infra | 18:39 | |
| AJaeger | sigmavirus: but -1s or +1s work as well ;) | 18:39 |
| *** thorst has quit IRC | 18:40 | |
| *** rcarrillocruz has quit IRC | 18:40 | |
| *** tuannguyen has joined #openstack-infra | 18:40 | |
| *** amitgandhinz has joined #openstack-infra | 18:40 | |
| sigmavirus | AJaeger: and done | 18:41 |
| harlowja | fungi before a twinkle, nice | 18:43 |
| harlowja | lol | 18:43 |
| *** e0ne has quit IRC | 18:43 | |
| *** edtubill has joined #openstack-infra | 18:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Run newton to master/ocata grenade on xenial https://review.openstack.org/375582 | 18:44 |
| fungi | harlowja: discussing in an etherpad how to read an ml is an interesting friday afternoon diversion | 18:44 |
| *** rcarrillocruz has joined #openstack-infra | 18:45 | |
| openstackgerrit | Merged openstack-infra/project-config: gnocchi: add upgrade testing jobs https://review.openstack.org/369164 | 18:45 |
| clarkb | jroll: hrm still failed http://logs.openstack.org/74/372374/16/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ssh-tinyipa-nv/dbfb3f0/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-23_18_41_14_202 probably something else I need to do but the lack of debug output from nova ironic driver/ironic client making this hard | 18:46 |
| *** e0ne has joined #openstack-infra | 18:47 | |
| * AJaeger wishes everybody a sunny weekend | 18:48 | |
| clarkb | I wonder if glance_protocol is at fault. the default is http and I don't think I see it reset to https anywhere /me tries | 18:49 |
| *** cardeois_ has quit IRC | 18:49 | |
| clarkb | (should also be set by the catalog imo) | 18:49 |
| harlowja | fungi its interesting like archeology kind of stuff | 18:51 |
| harlowja | ML-archaelogy , ha | 18:51 |
| harlowja | (which will be a new course that I will teach someday) | 18:51 |
| anteaya | AJaeger: thank you too | 18:51 |
| harlowja | along with openstack-archaelogy 201 | 18:51 |
| harlowja | lol | 18:51 |
| harlowja | (which vish & jay will teach the initial part of) | 18:51 |
| *** sdague has quit IRC | 18:51 | |
| fungi | harlowja: we'll get it posted as a subsite of vishfacts | 18:51 |
| anteaya | it is fun to compose | 18:52 |
| anteaya | I uncertain if the people that need to consume them will consume them | 18:52 |
| anteaya | but at least we would have another thing to point to | 18:52 |
| anteaya | the confidential footer stuns me everytime | 18:53 |
| sigmavirus | anteaya: agreed | 18:54 |
| *** rkukura has joined #openstack-infra | 18:55 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add Craton's channels to accessbot and gerritbot https://review.openstack.org/375677 | 18:55 |
| *** devananda|lunch is now known as devananda | 18:56 | |
| *** rbrndt has quit IRC | 18:58 | |
| pabelanger | clarkb: 374366 works for xenial, doing debian-jessie now | 18:59 |
| pabelanger | ssh-keygen systemd service | 18:59 |
| irtermite | cloudnull: let's bump infra's instance quota to 1024 and see what happens. clear for launch | 18:59 |
| *** Na3iL has quit IRC | 19:00 | |
| clarkb | ooh | 19:00 |
| anteaya | since our node usage peaked at 900 nodes today would an increase in osic to 1024 mean that all of our nodes would come from osic and none of our nodes would come from any other cloud? | 19:01 |
| anteaya | that was my understanding from the noisy neighbour algothrim discussion from last week or the week prior | 19:01 |
| fungi | irtermite: clarkb: we're only using about 450 nodes at the moment and it will continue to fall into the weekend | 19:02 |
| fungi | er, s/clarkb/cloudnull/ (sorry clarkb) | 19:02 |
| clarkb | no nodepool scales based on percentage of total per provider | 19:02 |
| fungi | anteaya: it would mean even fewer nodes launched in other providers though, yes | 19:02 |
| anteaya | clarkb: okay so nodepool would continue to use all hosts | 19:02 |
| clarkb | 1024would be like 75℅ or so | 19:02 |
| clarkb | so about that much would come from osic | 19:03 |
| anteaya | might cause races to get in that are only discovered on non osic nodes | 19:03 |
| fungi | basically adds up all the quotas, then divide a given provider's quota by that and you'll have the proportion of nodes which will be launched there (roughly) | 19:03 |
| anteaya | okay well if I'm the only one with this concern that's fine | 19:03 |
| anteaya | just airing it | 19:03 |
| pabelanger | we are also down 2 regions today, rax-iad and ovh-bhs1 | 19:04 |
| pabelanger | rax-iad is fixed, we need a new glean release, ovh-bhs1 we need to follow up about image uploads | 19:04 |
| fungi | anteaya: it's a concern insofar as it's a dynamic we need to stay aware of, but i don't think it's something we need to avoid | 19:04 |
| *** ihrachys has joined #openstack-infra | 19:05 | |
| irtermite | Understood. Seeing as how the utilization of OSIC-Cloud-1 is low and the performance seemed to help you all, we figured it would be a benefit. Up to you if you feel it a hinder. But, sounds like the proportioning will handle it? | 19:05 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/system-config: Add craton to the status- and meet-bot lists https://review.openstack.org/375680 | 19:05 |
| irtermite | Is this a matter that there is not a need for any more instances to speed up testing, fungi | 19:05 |
| irtermite | ? | 19:05 |
| fungi | the vast majority of failures identified by our ci will continue to be failures which are consistent across all providers, so fine-tuning the scheduling algorithm to try to make (much rarer) provider-specific failures more obvious is in my opinion unwarranted additional complexity | 19:06 |
| *** armax has joined #openstack-infra | 19:07 | |
| fungi | irtermite: it's a few things... first, we're normally low on resource demand on fridays and across weekends as many (probably a significant majority) of our contributors are working business days, and most of those also western hemisphere business hours. but also this is traditionally the quietest time during the release cycle. but also yes there's the fact that all the additional resources we have | 19:08 |
| fungi | are causing us not to end up with lengthy backlogs to test | 19:08 |
| anteaya | fungi: fair enough | 19:08 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Added Gem Mirror to Infra https://review.openstack.org/253616 | 19:08 |
| *** sdague has joined #openstack-infra | 19:08 | |
| *** artmr has joined #openstack-infra | 19:08 | |
| anteaya | irtermite: the additional quota is most welcome, thank you | 19:09 |
| anteaya | irtermite: adding 500 additional nodes to the pool though is large enough to merit a discussion of the impact though | 19:09 |
| anteaya | to ensure any concerns are voiced | 19:09 |
| anteaya | as I have voiced mine and have been reassured they are worth evaluating and not worth doing anything about at the moment | 19:10 |
| irtermite | curious to see if it is a benefit or not. can always trim it back to current if it is | 19:10 |
| anteaya | we won't know really until after summit at this point | 19:10 |
| *** david-lyle_ has joined #openstack-infra | 19:10 | |
| anteaya | as fungi says this is the slowest time of the cycle | 19:11 |
| pabelanger | Has anybody seen 1k used nodes online? Max I've seen is about 800 used | 19:11 |
| clarkb | pabelanger: you can ask graphtie pretty easily | 19:12 |
| anteaya | we did have 1.2k in the olden days that I recall | 19:12 |
| pabelanger | clarkb: Ya, that's what I'm looking up now | 19:12 |
| anteaya | I don't think it lasted more than a few months | 19:12 |
| anteaya | but I remember it | 19:12 |
| *** david-lyle has quit IRC | 19:13 | |
| fungi | anteaya: i think that may have been during the hpcloud v1.1 to v2 transition, when we had them both running in parallel | 19:14 |
| fungi | but yeah, it was short-lived | 19:14 |
| *** e0ne_ has joined #openstack-infra | 19:16 | |
| *** e0ne has quit IRC | 19:17 | |
| *** r-mibu has quit IRC | 19:17 | |
| *** e0ne_ has quit IRC | 19:17 | |
| *** tonytan4ever has quit IRC | 19:17 | |
| *** r-mibu has joined #openstack-infra | 19:17 | |
| *** esikachev has joined #openstack-infra | 19:17 | |
| *** abregman is now known as abregman|afk | 19:18 | |
| anteaya | that sounds right | 19:18 |
| *** e0ne has joined #openstack-infra | 19:18 | |
| *** ihrachys has quit IRC | 19:19 | |
| *** e0ne has quit IRC | 19:19 | |
| *** e0ne_ has joined #openstack-infra | 19:19 | |
| clarkb | right I don't think we should reject donations beacuse we are worried that more resources in one location may mask problems with another | 19:19 |
| *** e0ne_ has quit IRC | 19:19 | |
| clarkb | openstack is our abstraction and when that breaks down we are lucky enough to be able to lean back on the devs >_> | 19:19 |
| *** e0ne has joined #openstack-infra | 19:20 | |
| anteaya | did what I say sound like a rejection? | 19:21 |
| *** e0ne has quit IRC | 19:21 | |
| clarkb | anteaya: I parsed it as "we should consider not accepting these resources to prevent potential regressions in testing" | 19:21 |
| anteaya | ah | 19:21 |
| anteaya | sorry | 19:21 |
| anteaya | I meant lets take a moment to discuss impact | 19:22 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add Craton's channels to accessbot https://review.openstack.org/375677 | 19:22 |
| anteaya | which we did, thank you | 19:22 |
| *** annegentle has quit IRC | 19:27 | |
| *** burgerk has joined #openstack-infra | 19:27 | |
| *** armax has quit IRC | 19:29 | |
| *** tonytan4ever has joined #openstack-infra | 19:29 | |
| clarkb | looks like grenade on xenial is live now | 19:32 |
| clarkb | will keep an eye on that but I tested lcoally and it was happy so don't expect anything | 19:32 |
| *** tuannguyen has quit IRC | 19:33 | |
| *** spzala has quit IRC | 19:33 | |
| anteaya | yay | 19:33 |
| *** itsuugo has quit IRC | 19:34 | |
| *** itsuugo has joined #openstack-infra | 19:35 | |
| *** dimtruck is now known as zz_dimtruck | 19:35 | |
| *** zz_dimtruck is now known as dimtruck | 19:35 | |
| *** spzala has joined #openstack-infra | 19:36 | |
| pabelanger | clarkb: debian works too, on to upstart now | 19:37 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Set logrotate config on mosquitto https://review.openstack.org/375688 | 19:38 |
| mtreinish | clarkb, fungi, pleia2, nibalizer: ^^^ | 19:38 |
| *** nexusz99 has joined #openstack-infra | 19:38 | |
| fungi | mtreinish: though before we change that, i'd like to see the circumstances around the current issue when it recurs rather than just hoping this will help | 19:39 |
| *** itsuugo has quit IRC | 19:39 | |
| fungi | but having that coded up and ready to go is a huge help | 19:39 |
| *** idoine is now known as hashar | 19:40 | |
| mtreinish | fungi: sure, that makes sense | 19:40 |
| fungi | if the next outage also sees no post-rotated log, i think we can be reasonably sure it's more than coincidence | 19:40 |
| *** spzala has quit IRC | 19:40 | |
| *** nherciu_ has joined #openstack-infra | 19:41 | |
| *** salv-orlando has joined #openstack-infra | 19:41 | |
| fungi | though getting that fixed in the daemon is probably nontrivial if the reason it bombs is related to the load it's under at the time its signal handler is invoked | 19:41 |
| *** itsuugo has joined #openstack-infra | 19:42 | |
| mtreinish | fungi: well if we can confirm it to be the issue I can file another mosquitto bug about it | 19:42 |
| mtreinish | they fixed the other one :) | 19:42 |
| *** notnownikki has quit IRC | 19:42 | |
| *** nexusz99 has quit IRC | 19:43 | |
| *** devananda is now known as devananda|bbl | 19:44 | |
| *** dimtruck is now known as zz_dimtruck | 19:44 | |
| *** salv-orlando has quit IRC | 19:46 | |
| *** andymaier has joined #openstack-infra | 19:46 | |
| *** thiagolib has joined #openstack-infra | 19:46 | |
| fungi | ooh! | 19:47 |
| nibalizer | mtreinish: looks like the noop test doesn't like the puppet you wrote | 19:47 |
| mtreinish | nibalizer: heh, sounds like most of my puppet :) | 19:47 |
| mtreinish | I'll take a look | 19:47 |
| nibalizer | yah probably don't need the service bit anyways | 19:47 |
| mtreinish | nibalizer: ok, I'll drop that line | 19:48 |
| *** tuannguyen has joined #openstack-infra | 19:48 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Set logrotate config on mosquitto https://review.openstack.org/375688 | 19:48 |
| *** itsuugo has quit IRC | 19:49 | |
| *** itsuugo has joined #openstack-infra | 19:50 | |
| *** pvaneck_ has joined #openstack-infra | 19:51 | |
| *** amitgandhinz has quit IRC | 19:53 | |
| *** burgerk has quit IRC | 19:53 | |
| *** amitgandhinz has joined #openstack-infra | 19:54 | |
| *** pvaneck has quit IRC | 19:54 | |
| *** itsuugo has quit IRC | 19:55 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 19:55 |
| *** itsuugo has joined #openstack-infra | 19:55 | |
| pabelanger | clarkb: doing one final test with upstart, but ^ should be all we need for ssh-keygen | 19:55 |
| *** tuannguyen has quit IRC | 19:55 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 19:57 |
| notmyname | does the CI system respect the bindep "test" profile? | 19:58 |
| clarkb | pabelanger: cool. reading up on what that upstart does now | 20:00 |
| clarkb | notmyname: yes it should | 20:00 |
| *** krtaylor has joined #openstack-infra | 20:00 | |
| *** sdake has quit IRC | 20:00 | |
| pabelanger | clarkb: Ya, that was the best I found to depend on a service. But, happy to change if something is better | 20:00 |
| notmyname | clarkb: ok, cool! I didn't know about it until just now, and our entries aren't decorated like that. I'd like to update it | 20:00 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Update the max-servers for cloud1 to 1024 https://review.openstack.org/375696 | 20:01 |
| clarkb | jroll: cool that ironic devstack plugin change gets the traceback to change http://logs.openstack.org/74/372374/16/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ssh-tinyipa-nv/0f1d731/logs/screen-n-cpu.txt.gz?level=TRACE | 20:01 |
| cloudnull | clarkb: fungi: irtermite: ^ sorry was AFK for a while | 20:01 |
| clarkb | jroll: looks like its breaking on ssl cert verification now. Slowly but surely | 20:02 |
| *** itsuugo has quit IRC | 20:02 | |
| cloudnull | I know there was some talk about it's effect but I figured I'd throw over the PR just incase. | 20:02 |
| *** ijw has quit IRC | 20:02 | |
| anteaya | cloudnull: yeah, we talked about it and agreed it is fine to increase | 20:03 |
| anteaya | thanks for submitting the patch | 20:03 |
| anteaya | I just didn't know what it would do to our allocation algorithm | 20:03 |
| *** csomerville has quit IRC | 20:03 | |
| anteaya | and have since learned all clouds will continue to be used | 20:04 |
| *** itsuugo has joined #openstack-infra | 20:04 | |
| *** nickola has joined #openstack-infra | 20:04 | |
| fungi | notmyname: if it makes you feel any better, swift reviewers are why AJaeger added support for that "test" profile to our jobs and our package caching automation ;) | 20:05 |
| cloudnull | that's good. we're not looking to cause issue. Just trying to help where we can. | 20:05 |
| notmyname | fungi: cool :-) | 20:05 |
| anteaya | cloudnull: oh yeah, totally get that | 20:05 |
| fungi | notmyname: or at least i think that was the case. AJaeger can correct me if i'm wrong there | 20:05 |
| anteaya | cloudnull: this was me not knowing how our algorithm works | 20:05 |
| notmyname | fungi: now I'm trying to figure out if just adding "test" is ok, or if I also need to update tox.ini | 20:05 |
| anteaya | cloudnull: and thank you | 20:06 |
| *** david-lyle_ is now known as david-lyle | 20:06 | |
| cloudnull | anteaya: i have no idea either :) | 20:06 |
| fungi | notmyname: yeah, the environment in tox.ini is just intended for the convenience of your developers in setting up their dev environments, but i would suggest having it run `bindep test` instead of just `bindep` | 20:06 |
| * cloudnull is now curious | 20:06 | |
| anteaya | cloudnull: I still have no idea, but clarkb and fungi reassure me all the clouds will get some nodes used on them, proportionally based on quota | 20:06 |
| notmyname | fungi: ok. and what if the tox.ini file doesn't have a bindep env at all? | 20:06 |
| notmyname | fungi: (but i'll be adding it. just wondering if it's broken as-is and we didn't realize) | 20:07 |
| *** annegentle has joined #openstack-infra | 20:08 | |
| fungi | notmyname: our jobs don't rely on it being in tox.ini, again the only reason to add it is so that devs can run `tox -e bindep` and have it tell them what distro packages they're missing to be able to use your other tox targets | 20:08 |
| notmyname | fungi: perfect. thanks :-) | 20:08 |
| fungi | glad to help! | 20:09 |
| notmyname | fwiw https://review.openstack.org/#/c/375697/ | 20:09 |
| *** amitgandhinz has quit IRC | 20:09 | |
| *** lucasagomes is now known as lucas-afk | 20:09 | |
| *** ijw has joined #openstack-infra | 20:10 | |
| *** amitgandhinz has joined #openstack-infra | 20:10 | |
| *** kaisers_ has quit IRC | 20:12 | |
| *** tonytan4ever has quit IRC | 20:12 | |
| pabelanger | clarkb: fungi: 374366 is in working order for ubuntu-trusty, ubuntu-xenial and debian-jessie | 20:12 |
| *** ijw has quit IRC | 20:14 | |
| fungi | notmyname: looks like it still ends up trying to install swiftclient into the tox venv. i think adding usedevelop=false to that environment will fix it. leaving a comment on the review now | 20:14 |
| *** esberglu has quit IRC | 20:15 | |
| *** rajinir has quit IRC | 20:15 | |
| openstackgerrit | Oleksii Butenko proposed openstack-infra/project-config: Revert "Disable test_volume_from_snapshot tests for Cinder BDD job" https://review.openstack.org/375700 | 20:15 |
| *** zz_dimtruck is now known as dimtruck | 20:15 | |
| *** akshai has quit IRC | 20:15 | |
| notmyname | fungi: heh, I think tim just beat you to it :-). and thanks | 20:17 |
| fungi | indeed! reviewer race condition | 20:18 |
| notmyname | TBH I just copy/pasted from an infra repo ;-) | 20:18 |
| notmyname | so this is really your fault ;-) | 20:18 |
| fungi | totally | 20:18 |
| fungi | which one? i'll check to see if it's warranted there as well | 20:18 |
| fungi | tox configuration is fiddly, so there are interactions between the global skipdist and per-environment usedevelop | 20:19 |
| clarkb | pabelanger: for the unit file does the | in ConditionPathExist=|! mean logical OR? | 20:19 |
| clarkb | pabelanger: eg run the command if any one of the files does not exist? | 20:19 |
| pabelanger | clarkb: exactly | 20:19 |
| *** akshai has joined #openstack-infra | 20:20 | |
| pabelanger | clarkb: I don't think we can do that in upstart, unless we add a fail bit of code | 20:20 |
| clarkb | ah yup finally found it in the docs. ^F found the | but then it was hard to find it visually | 20:20 |
| notmyname | fungi: openstack-infra/project-config | 20:20 |
| pabelanger | but ssh-keygen is smart enough not to regen the keys | 20:20 |
| *** inc0 has quit IRC | 20:20 | |
| pabelanger | if they exist | 20:20 |
| clarkb | ya man page confirms that | 20:20 |
| *** armax has joined #openstack-infra | 20:21 | |
| clarkb | pabelanger: reading upstart docs start on starting ssh means run this as soon as that event is emitted which doesn't mean it will necessariyl run before ssh? | 20:22 |
| *** csomerville has joined #openstack-infra | 20:23 | |
| clarkb | I am trying to figure out if there is a better way to do that | 20:24 |
| pabelanger | clarkb: http://upstart.ubuntu.com/cookbook/#start-must-precede-another-service is what I found | 20:24 |
| pabelanger | clarkb: but, you might be correct | 20:24 |
| *** itsuugo has quit IRC | 20:24 | |
| fungi | notmyname: thanks! as it turns out, project-config is not a python package so doesn't get "installed" or have declared python dependencies, making it a bit special in that regard | 20:24 |
| clarkb | pabelanger: oh hrm. I just read it as start on will start sometime after the event is emitted but ssh could continue to start at the same time. But that section seems to say I am wrong reading the other parts of the docs | 20:25 |
| fungi | thus running tox -e bindep on it only installs bindep | 20:25 |
| clarkb | seems like "starting" is magical | 20:25 |
| *** salv-orlando has joined #openstack-infra | 20:26 | |
| *** xarses has quit IRC | 20:26 | |
| fungi | gonna go grab early dinner. back in a bit | 20:26 |
| anteaya | fungi: enjoy dinner | 20:26 |
| *** spzala has joined #openstack-infra | 20:26 | |
| *** clenimar has quit IRC | 20:26 | |
| *** itsuugo has joined #openstack-infra | 20:26 | |
| clarkb | pabelanger: so this will install the systemd unit on centos7 and fedora but not enable it right? thats probably good enough | 20:27 |
| pabelanger | clarkb: that is a good question, let me do a build real quick to see | 20:27 |
| pabelanger | I didn't test that | 20:27 |
| clarkb | pabelanger: ok I -1' | 20:27 |
| clarkb | d so that we can include the link to the debian bug too | 20:28 |
| pabelanger | ack | 20:28 |
| *** xarses has joined #openstack-infra | 20:29 | |
| *** xarses has quit IRC | 20:30 | |
| *** xarses has joined #openstack-infra | 20:30 | |
| *** spzala has quit IRC | 20:31 | |
| *** kjorgensen has quit IRC | 20:31 | |
| *** dimtruck is now known as zz_dimtruck | 20:31 | |
| *** kjorgensen has joined #openstack-infra | 20:31 | |
| *** vhosakot has joined #openstack-infra | 20:32 | |
| *** Jeffrey4l_ has joined #openstack-infra | 20:35 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** xarses has quit IRC | 20:36 | |
| pabelanger | clarkb: Yup, installed not active for centos | 20:37 |
| *** tongli has quit IRC | 20:37 | |
| pabelanger | I don't know how to not install it | 20:37 |
| pabelanger | checking docs for the element | 20:37 |
| *** amitgandhinz has quit IRC | 20:38 | |
| clarkb | pabelanger: I don't think it matters all that much | 20:38 |
| pabelanger | k | 20:38 |
| clarkb | but I am also not dib core | 20:38 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 20:39 |
| pabelanger | lets see what they say | 20:39 |
| *** lascii is now known as alaski | 20:41 | |
| *** jkilpatr has quit IRC | 20:41 | |
| *** jheroux has quit IRC | 20:45 | |
| *** amitgandhinz has joined #openstack-infra | 20:46 | |
| pabelanger | clarkb: if you have bandwidth, could use some help on 375448, don't upload DIB if provider is disabled. Having some troubles getting a test for that | 20:46 |
| clarkb | pabelanger: sure | 20:46 |
| clarkb | I am mostly just trying to keep pushing on ssl things and waiting on jobs right now | 20:47 |
| pabelanger | cool, otherwise I don't mind waiting until next week | 20:47 |
| *** krtaylor has quit IRC | 20:47 | |
| clarkb | pabelanger: for testing it I would run the things to set up your provider as normal | 20:48 |
| clarkb | then update the config and call the update images function directly and check that no new image is uploaded | 20:48 |
| *** mriedem has quit IRC | 20:50 | |
| *** sdague has quit IRC | 20:50 | |
| *** matt-borland has quit IRC | 20:51 | |
| *** askb has joined #openstack-infra | 20:51 | |
| pabelanger | clarkb: okay, let me give that a go | 20:51 |
| clarkb | pabelanger: basically you want to get nodepool to a state where you can assert everything is as expected. Then update config and check that your state doesn't change with a new image upload | 20:52 |
| pabelanger | ya | 20:52 |
| pabelanger | that should work | 20:52 |
| *** mriedem has joined #openstack-infra | 20:53 | |
| *** vhosakot has quit IRC | 20:54 | |
| *** artmr has quit IRC | 20:54 | |
| *** aviau has quit IRC | 20:54 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: dstat: improve output to track high cpu process https://review.openstack.org/374982 | 20:55 |
| *** vaidy has joined #openstack-infra | 20:55 | |
| *** aviau has joined #openstack-infra | 20:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Copy ssl certs for multinode https://review.openstack.org/373219 | 20:55 |
| *** thorst has joined #openstack-infra | 20:56 | |
| openstackgerrit | Sridhar Ramaswamy proposed openstack-infra/project-config: Make tacker python3 jobs voting https://review.openstack.org/370388 | 20:56 |
| *** isviridov_away has joined #openstack-infra | 20:56 | |
| *** thorst has quit IRC | 20:56 | |
| *** thorst has joined #openstack-infra | 20:57 | |
| *** rkukura has quit IRC | 20:59 | |
| *** thorst has quit IRC | 21:01 | |
| *** edtubill has quit IRC | 21:01 | |
| *** thorst has joined #openstack-infra | 21:02 | |
| *** rkukura has joined #openstack-infra | 21:02 | |
| *** thorst has quit IRC | 21:04 | |
| *** kgiusti has left #openstack-infra | 21:04 | |
| clarkb | jroll: does http://logs.openstack.org/74/372374/16/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ssh-tinyipa-nv/5dac5e8/logs/screen-ir-cond.txt.gz?#_2016-09-23_20_45_13_261 mean that the agent on the instance was unable to verify ssl? | 21:04 |
| clarkb | I am guessing that we build an image that doesn't "inherit" the ssl config from devstack | 21:05 |
| *** raildo has quit IRC | 21:06 | |
| clarkb | jroll: ok I think I tracked it down to http://logs.openstack.org/74/372374/16/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ssh-tinyipa-nv/5dac5e8/logs/ironic-bm-logs/node-0_console_23-09-2016-20:45:16.txt.gz#_2016-09-23_20_45_11_115 | 21:07 |
| *** thorst has joined #openstack-infra | 21:07 | |
| *** trown is now known as trown|outtypewww | 21:09 | |
| *** rbrndt has joined #openstack-infra | 21:10 | |
| clarkb | jroll: I think we can probably go ahead and review https://review.openstack.org/375649 then figure out how we want IPA to be talking to glance later | 21:10 |
| *** pvaneck__ has joined #openstack-infra | 21:11 | |
| *** thorst has quit IRC | 21:12 | |
| *** pradk has quit IRC | 21:14 | |
| *** pvaneck_ has quit IRC | 21:14 | |
| *** csomerville has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** jamesdenton has quit IRC | 21:17 | |
| *** pvaneck_ has joined #openstack-infra | 21:18 | |
| *** itsuugo has quit IRC | 21:20 | |
| *** itsuugo has joined #openstack-infra | 21:22 | |
| *** pvaneck__ has quit IRC | 21:22 | |
| *** coolsvap has quit IRC | 21:22 | |
| *** cody-somerville has joined #openstack-infra | 21:22 | |
| *** cody-somerville has quit IRC | 21:22 | |
| *** cody-somerville has joined #openstack-infra | 21:22 | |
| anteaya | clarkb: do you know if anyone did any follow up on logstash after jhesketh worked on it? | 21:23 |
| *** fguillot has quit IRC | 21:23 | |
| clarkb | anteaya: no | 21:24 |
| anteaya | okay thanks | 21:24 |
| *** amitgandhinz has quit IRC | 21:26 | |
| *** nexusz99 has joined #openstack-infra | 21:26 | |
| *** jkilpatr has joined #openstack-infra | 21:27 | |
| *** senk has joined #openstack-infra | 21:27 | |
| *** gouthamr has quit IRC | 21:28 | |
| *** esikachev has quit IRC | 21:28 | |
| *** itsuugo has quit IRC | 21:30 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas functional job not experimental https://review.openstack.org/359320 | 21:31 |
| *** abregman|afk has quit IRC | 21:31 | |
| *** itsuugo has joined #openstack-infra | 21:32 | |
| *** thorst has joined #openstack-infra | 21:32 | |
| *** thorst has quit IRC | 21:36 | |
| *** annegent_ has joined #openstack-infra | 21:37 | |
| *** thorst has joined #openstack-infra | 21:37 | |
| *** thorst has quit IRC | 21:37 | |
| *** rtheis has quit IRC | 21:38 | |
| *** GordonTX has quit IRC | 21:39 | |
| *** GordonTX has joined #openstack-infra | 21:39 | |
| *** esberglu has joined #openstack-infra | 21:40 | |
| *** gordc has quit IRC | 21:40 | |
| *** annegentle has quit IRC | 21:40 | |
| *** hashar has quit IRC | 21:41 | |
| *** mriedem has quit IRC | 21:41 | |
| *** thorst has joined #openstack-infra | 21:41 | |
| *** tonytan4ever has joined #openstack-infra | 21:43 | |
| *** kzaitsev_mb has quit IRC | 21:45 | |
| *** gouthamr has joined #openstack-infra | 21:45 | |
| *** thorst has quit IRC | 21:46 | |
| *** tonytan4ever has quit IRC | 21:47 | |
| *** rlandy has quit IRC | 21:49 | |
| *** nexusz99 has quit IRC | 21:50 | |
| anteaya | what is going on on the cacti server that there are gaps in the graph? http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=7&select_first=true | 21:52 |
| *** senk has quit IRC | 21:52 | |
| clarkb | anteaya: usually means the host was too busy to process the snmp requests | 21:53 |
| anteaya | okay thanks | 21:53 |
| *** senk has joined #openstack-infra | 21:53 | |
| anteaya | graphite is showing the same: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=25 | 21:55 |
| anteaya | and zuul: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=23 | 21:56 |
| anteaya | I'm guessing something is going on with the cacti server | 21:56 |
| *** yamahata has quit IRC | 21:57 | |
| anteaya | however since it only offers 3 graphs for localhost I don't know what | 21:57 |
| *** yamahata has joined #openstack-infra | 21:59 | |
| *** kaisers_ has joined #openstack-infra | 22:00 | |
| *** mdrabe has quit IRC | 22:01 | |
| *** mdrabe has joined #openstack-infra | 22:01 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Copy ssl certs for multinode https://review.openstack.org/373219 | 22:01 |
| anteaya | 5 deer grazing in my yard | 22:02 |
| clarkb | its possible that cacti is experiencing packet loss for the snmp requests | 22:02 |
| clarkb | or similar | 22:02 |
| anteaya | okay, fair enough | 22:02 |
| anteaya | the graphs for all the services appear to be affected | 22:02 |
| anteaya | any services I have checked at the very least | 22:02 |
| anteaya | looks like a decent chunk of yesterday is missing | 22:05 |
| anteaya | and intermittent skips starting about 4 hours ago | 22:05 |
| clarkb | were there network issues in rax dfw? | 22:05 |
| *** kaisers_ has quit IRC | 22:05 | |
| *** psilvad has quit IRC | 22:06 | |
| anteaya | pabelanger: said there was a rax region down or having issues, earlier did he not? | 22:06 |
| anteaya | rax-iad according to backscroll | 22:06 |
| *** xyang1 has quit IRC | 22:06 | |
| anteaya | status page says a system disruption in london: https://status.rackspace.com/ | 22:07 |
| anteaya | so not north america | 22:07 |
| clarkb | its beginnign to feel a lot like friday | 22:09 |
| clarkb | also jetlag | 22:09 |
| anteaya | I hear ya | 22:10 |
| anteaya | 2 8 point bucks, 2 fawns and a doe | 22:10 |
| *** mdrabe has quit IRC | 22:11 | |
| *** GordonTX has quit IRC | 22:11 | |
| anteaya | I'll see if I can find fungi after he returns from dinner to help me with cacti | 22:12 |
| *** senk__ has joined #openstack-infra | 22:13 | |
| *** senk has quit IRC | 22:13 | |
| *** yamamoto has joined #openstack-infra | 22:15 | |
| *** salv-orlando has quit IRC | 22:21 | |
| fungi | i don't know how helpful i am, but i can try | 22:21 |
| fungi | lots of gaps in the localhost graphs, that's fun | 22:22 |
| fungi | similar gaps starting to appear in graphs for remote systems too | 22:23 |
| *** senk__ has quit IRC | 22:23 | |
| *** spzala has joined #openstack-infra | 22:23 | |
| fungi | could indicate a network problem in rax-dfw, or a local issue with the cacti.o.o server | 22:23 |
| anteaya | I can buy that | 22:24 |
| *** senk has joined #openstack-infra | 22:24 | |
| fungi | load average on the cacti server is higher than i'd normally expect | 22:24 |
| *** itsuugo has quit IRC | 22:24 | |
| anteaya | hmmmm | 22:24 |
| fungi | around 1.5 for the 15-minute average | 22:24 |
| anteaya | has the addition of the infra cloud nodes upset cacti? | 22:24 |
| fungi | php processes eating cpu | 22:24 |
| fungi | anteaya: any estimate when it started collecting those? | 22:25 |
| *** akshai has quit IRC | 22:25 | |
| anteaya | I have none let me see if cacti shows anything useful | 22:25 |
| *** itsuugo has joined #openstack-infra | 22:25 | |
| fungi | baremetal00.vanilla.ic.o.o seems to be in cacti for almost a week | 22:26 |
| anteaya | it started tracking compute000.vanilla 4 weeks ago: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=5056&rra_id=all | 22:26 |
| fungi | sorry, almost a month | 22:26 |
| fungi | yeah, same for baremetal00 | 22:27 |
| *** amitgandhinz has joined #openstack-infra | 22:27 | |
| anteaya | compute045 doesn't show anything: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=350 | 22:27 |
| fungi | found the same. working backward to find one that does | 22:27 |
| anteaya | compute027 also nothing: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=328 | 22:27 |
| fungi | compute015 had very spotty data | 22:28 |
| anteaya | compute008 doesn't show anything at all: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=332 | 22:28 |
| anteaya | not even blank graphs | 22:28 |
| anteaya | which is an outlier | 22:28 |
| fungi | i suspect a bunch of them are not unreachable | 22:28 |
| *** armax has quit IRC | 22:28 | |
| anteaya | 009 has no data | 22:29 |
| clarkb | yes infra cloud is or was unreachable earlier | 22:29 |
| fungi | er, are unreachable or are not reachable. take your pick | 22:29 |
| anteaya | 010 has a month, but very large gaps | 22:29 |
| clarkb | rcarrillocruz said networking issues affecting the houston site not jsut infracloud | 22:29 |
| anteaya | when did cacti issues begin? | 22:29 |
| anteaya | yesterday: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1&rra_id=all | 22:30 |
| *** aspiers has quit IRC | 22:30 | |
| *** senk__ has joined #openstack-infra | 22:30 | |
| fungi | looks like load average on cacti.o.o has been creeping up | 22:30 |
| fungi | and then has a week-long gap about a week ago | 22:30 |
| *** senk has quit IRC | 22:30 | |
| fungi | and has been both high and spotty ever since | 22:30 |
| *** shadowfku has joined #openstack-infra | 22:30 | |
| fungi | no, sorry, day-long gap a couple days ago | 22:31 |
| anteaya | a week long gap a week ago? | 22:31 |
| anteaya | yesterday is what i see | 22:31 |
| fungi | i was looking at the wrong graph scale | 22:31 |
| anteaya | end of 21 beginning of 22 | 22:31 |
| anteaya | okay then we are seeing the same | 22:31 |
| fungi | yep, so about 1.5 days ago it started reporting again | 22:31 |
| fungi | the memory usage graph is strange-looking, but if you pay attention to the fact that it's inverted (graphing free memory) it indicates we're running at the edge of available ram there | 22:33 |
| anteaya | oh my | 22:34 |
| *** spzala has quit IRC | 22:35 | |
| fungi | teh `free` command locally suggests about 85% of ram is in active use, with another ~10% used by cache and buffers | 22:35 |
| fungi | looking at rackspace incident tickets, it appears mirror.iad.rax lost connectivity to its cinder volume and has been spewing kernel panics | 22:39 |
| fungi | i'm going to reboot it now in hopes it regains sanity | 22:39 |
| *** shadowfku has quit IRC | 22:39 | |
| *** itsuugo has quit IRC | 22:40 | |
| fungi | #status log mirror.iad.rax.openstack.org has been rebooted to restore sanity following connectivity issues to its cinder volume | 22:40 |
| openstackstatus | fungi: finished logging | 22:40 |
| fungi | it's back online now | 22:40 |
| fungi | also a ticket about the hypervisor host logstash-worker20 is on having trouble | 22:41 |
| *** amitgandhinz has quit IRC | 22:42 | |
| *** itsuugo has joined #openstack-infra | 22:42 | |
| fungi | looks like they rebooted it after they restored the host to working order | 22:42 |
| anteaya | where do you see the incident tickets? | 22:43 |
| *** tonytan4ever has joined #openstack-infra | 22:44 | |
| fungi | logging into the rackspace management dashboard for our tenant | 22:44 |
| anteaya | ah | 22:44 |
| fungi | was hoping to spot something there to explain the cacti.o.o strangeness, but no such luck | 22:44 |
| anteaya | thanks for fixing the mirror | 22:45 |
| anteaya | and sharing the logstash worker20 status update | 22:45 |
| fungi | we had swap memory allocated on cacti.o.o up until sometime in may according to the graphs | 22:45 |
| *** EricGonczer_ has joined #openstack-infra | 22:46 | |
| anteaya | or, rebooting the mirror (hoping that fixes it) | 22:46 |
| fungi | which i think corresponds with us rebuilding it on trusty | 22:46 |
| anteaya | yeah I saw that | 22:46 |
| anteaya | I think so | 22:46 |
| anteaya | the sever rebuilds included a number of server changes | 22:46 |
| fungi | i've confirmed from a shell that it definitely has no swap configured now at any rate | 22:46 |
| anteaya | size adjustments mostly is what I saw | 22:46 |
| *** EricGonc_ has joined #openstack-infra | 22:47 | |
| *** senk__ has quit IRC | 22:47 | |
| *** markvoelker has quit IRC | 22:47 | |
| *** senk has joined #openstack-infra | 22:48 | |
| fungi | but yeah, load average on cacti.o.o really seems to have shot up in the past week or two | 22:48 |
| *** tonytan4ever has quit IRC | 22:48 | |
| *** annegent_ has quit IRC | 22:48 | |
| *** AnarchyAo has quit IRC | 22:48 | |
| anteaya | hmmmmm | 22:48 |
| fungi | if that coincides with adding the bulk of infra-cloud, them maybe we want to move cacti to a larger flavor | 22:48 |
| anteaya | that is the only thing springing to my mind | 22:49 |
| fungi | rcarrillocruz: ^ when you're awake/around | 22:49 |
| fungi | i don't think it's dire, but it's probably worth hacking on early next week | 22:49 |
| anteaya | but with 40+ nodes which may or may not have data I'm not interested in clicking each individual one | 22:49 |
| anteaya | I know jim likes consistent graphite data | 22:49 |
| fungi | oh, as do i | 22:50 |
| fungi | and cacti data too | 22:50 |
| *** EricGonczer_ has quit IRC | 22:50 | |
| anteaya | okay | 22:50 |
| jroll | clarkb: oh my, that'll be fun | 22:50 |
| fungi | i just don't think anybody needs to lose their weekend of well-deserved downtime over it | 22:50 |
| mtreinish | pleia2, nibalizer, fungi: so I just dug through the ruby mqtt code and the logstash mqtt plugin and nothing stuck out as obvious as to why http://paste.openstack.org/show/582792/ is happening | 22:50 |
| *** Swami has quit IRC | 22:51 | |
| anteaya | fungi: no argument | 22:51 |
| mtreinish | although I didn't check the ruby ssl code | 22:51 |
| fungi | mtreinish: is there any additional debugging we could turn on client side? | 22:52 |
| mtreinish | but I am able to confirm that using the mosquitto cli subscriber with ssl and a ca cert downloaded from instant ssl works fine on firehose from my desktop | 22:52 |
| anteaya | fungi: since the bulk of infra cloud is offline anyway how hard is it to remove infra cloud from cacti for the weekend? | 22:53 |
| fungi | i honestly don't know. i mean i know how to remove it from our configuration in git but it seems like there are additional manual steps involved in purging it from cacti | 22:54 |
| anteaya | oh | 22:54 |
| mtreinish | fungi: hmm, not that I saw in the plugin, the code was pretty minimal: https://github.com/kompa3/logstash-output-mqtt/blob/master/lib/logstash/outputs/mqtt.rb | 22:54 |
| mtreinish | fungi: it's probably worth checking the logs of a logstash worker though to see if there was anything in there | 22:55 |
| *** aspiers has joined #openstack-infra | 22:55 | |
| *** nherciu_ has quit IRC | 22:57 | |
| *** erlon has quit IRC | 22:57 | |
| *** thiagolib has quit IRC | 22:58 | |
| *** annegentle has joined #openstack-infra | 22:58 | |
| *** hongbin has quit IRC | 23:00 | |
| *** itsuugo has quit IRC | 23:01 | |
| *** itsuugo has joined #openstack-infra | 23:02 | |
| *** akshai has joined #openstack-infra | 23:03 | |
| fungi | mtreinish: looking at worker01, i see some crm114 timeouts | 23:04 |
| fungi | i assume i should be looking in /var/log/logprocessor/log-worker-A-debug.log anyway | 23:04 |
| fungi | 404 errors pulling the occasional log, i think | 23:05 |
| fungi | i'm not seeing any references to mqtt at least | 23:05 |
| fungi | mtreinish: is support directly implemented in the log processors? | 23:06 |
| mtreinish | fungi: we install that output plugin on the log processors and add it to the output config | 23:07 |
| fungi | i'm not sure what that means. may need a logstash to human filter to translate for me | 23:07 |
| mtreinish | fungi: and that paste nibalizer shared from mosquitto shows at least some workers are trying to connect via mqtt at least | 23:07 |
| mtreinish | fungi: on the worker nodes we run /opt/logstash/bin/plugin install logstash-output-mqtt | 23:08 |
| mtreinish | and then update the config to use the plugin | 23:08 |
| mtreinish | it's not a built into logstash itself is what I was trying to say | 23:08 |
| fungi | i'm just trying to figure out where log output from that would surface | 23:09 |
| openstackgerrit | Abhishek Raut proposed openstack-infra/project-config: Use python-db-jobs for vmware-nsx https://review.openstack.org/375766 | 23:09 |
| fungi | whether it's the log processors or the logstash process | 23:09 |
| fungi | logstash process, looks like | 23:10 |
| fungi | {:timestamp=>"2016-09-23T23:08:04.265000+0000", :message=>"Error certificate verify failed while publishing to MQTT server. Will retry in 10 seconds.", :level=>:error} | 23:10 |
| fungi | so, er, i guess that's the 'splanatorization | 23:10 |
| mtreinish | heh | 23:10 |
| mtreinish | it's about all the info we get from the mosquitto side logs too | 23:11 |
| fungi | where do you configure cert trusts in logstash? no clue. but i'm sure it will be exciting to figure out | 23:11 |
| fungi | if the cert itself verifies against the ca cert, then i'm guessing we're missing a trust on the ca cert | 23:12 |
| fungi | logstash is java-based? | 23:12 |
| fungi | if so, the jvm maintains its own trust chain for ca certs that needs updating directly | 23:12 |
| mtreinish | fungi: we should pass it in via the output.conf for logstash | 23:12 |
| pleia2 | ruby running in a jvm | 23:12 |
| fungi | assuming the service is relying on that rather than bringing its own trust implementation anyway | 23:13 |
| mtreinish | fungi: yeah what pleia2 said. The ruby code is what should be talking to mosquitto | 23:13 |
| anteaya | this sounds like too much fun for me | 23:14 |
| anteaya | have a nice weekend all | 23:14 |
| anteaya | g'night | 23:14 |
| fungi | night anteaya! | 23:14 |
| mtreinish | fungi: heh, that's why I looked at the plugin code and traced through it | 23:15 |
| fungi | mtreinish: yep, i'm mostly just catching up to your already on-the-ball analysis | 23:16 |
| *** nexusz99 has joined #openstack-infra | 23:16 | |
| mtreinish | fungi: the only ssl handling bits were in ruby-mqtt: https://github.com/njh/ruby-mqtt/blob/master/lib/mqtt/client.rb#L214-L220 | 23:16 |
| fungi | server says handshake didn't complete. client says server cert couldn't be verified | 23:16 |
| *** sdake has joined #openstack-infra | 23:17 | |
| mtreinish | fungi: heh, actually rereading that method if we don't set a ca_file it doesn't set ssl_verify | 23:17 |
| mtreinish | maybe we should just do that :) | 23:17 |
| fungi | a disappointing but perhaps pragmatic observation | 23:18 |
| fungi | though i'm betting the ca_file being pointed to is simply not in a format that it is able to make use of or something | 23:18 |
| fungi | maybe missing a chain from the ca to the server cert? | 23:18 |
| fungi | or wrong encoding'? | 23:19 |
| mtreinish | fungi: oh, perhaps. We just reused the existing thing in heira for the mosquitto ca cert file and dumped it to a file: http://git.openstack.org/cgit/openstack-infra/puppet-logstash/tree/manifests/indexer.pp#n96 | 23:19 |
| *** rbrndt has quit IRC | 23:20 | |
| *** nexusz99 has quit IRC | 23:21 | |
| fungi | ahh, what it probably expects is not a copy of the ca-signed server cert, but the cert for the ca itself | 23:22 |
| fungi | unless i'm misunderstanding what got passed in there | 23:22 |
| *** mlekkas has quit IRC | 23:24 | |
| *** itsuugo has quit IRC | 23:24 | |
| mtreinish | fungi: oh perhaps. I assumed it was looking for the cert for the ca itself. That's what I was using for the cafile param on the mosquitto cli client | 23:24 |
| *** kjorgensen has quit IRC | 23:24 | |
| mtreinish | like I just downloaded the cabundle from instantssl and passed that to mosquitto_sub | 23:24 |
| fungi | er, yeah, that's what i was saying too, so i'm again just missing context i think | 23:25 |
| mtreinish | but maybe that's not what the ruby code is expecting | 23:25 |
| * mtreinish double checks what we're passing to the workers | 23:25 | |
| *** itsuugo has joined #openstack-infra | 23:25 | |
| fungi | well, the ruby code might expect it in some format other than pem for example, or may only work with the penultimate cert in teh chain rather than the root cert | 23:25 |
| mtreinish | fungi: http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n465 | 23:26 |
| mtreinish | fungi: ah, ok | 23:26 |
| fungi | just trying to think up various ways this could go surprisingly wrong | 23:26 |
| mtreinish | fungi: hmm, the docs seem to imply it's the root ca in pem: https://github.com/njh/ruby-mqtt#connecting | 23:27 |
| fungi | "root" could be confusing here | 23:27 |
| mtreinish | heh, yeah | 23:27 |
| fungi | root cert plus chain certs that together validate the server cert? | 23:27 |
| fungi | or _just_ the cert that directly signed the server cert? | 23:27 |
| fungi | the latter is pretty guaranteed to work if what is being set up is a direct trust | 23:28 |
| mtreinish | fungi: the only thing I see in the docs on that is: http://ruby-doc.org/stdlib-2.0.0/libdoc/openssl/rdoc/OpenSSL.html#module-OpenSSL-label-Peer+Verification | 23:28 |
| fungi | the former is what, e.g., browsers need | 23:28 |
| fungi | because browsers have an established trust and simply need any intermediate signatories between their existing trust and the server cert | 23:29 |
| *** yamamoto has quit IRC | 23:29 | |
| *** senk has quit IRC | 23:30 | |
| fungi | in hiera, mosquitto_tls_ca_file seems to contain multiple certificates, so likely a chain | 23:31 |
| fungi | depending on how that is getting processed by the client, it could just be picking the first cert it comes to in there, which likely hasn't directly signed the server cert | 23:31 |
| *** senk has joined #openstack-infra | 23:32 | |
| *** itsuugo has quit IRC | 23:32 | |
| *** itsuugo has joined #openstack-infra | 23:33 | |
| *** askb has quit IRC | 23:34 | |
| mtreinish | fungi: well according to the ruby docs it should just be wrapping openssl. Although, I'm not sure what the behavior is there off the top of my head | 23:36 |
| mtreinish | I assume it would be smarter than just reading the first cert | 23:37 |
| *** yamamoto has joined #openstack-infra | 23:37 | |
| *** amitgandhinz has joined #openstack-infra | 23:38 | |
| *** ijw has joined #openstack-infra | 23:39 | |
| *** itsuugo has quit IRC | 23:39 | |
| *** itsuugo has joined #openstack-infra | 23:40 | |
| *** akshai has quit IRC | 23:41 | |
| *** ddieterly has joined #openstack-infra | 23:42 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 23:43 |
| *** armax has joined #openstack-infra | 23:43 | |
| *** yamamoto has quit IRC | 23:47 | |
| *** markvoelker has joined #openstack-infra | 23:48 | |
| *** ijw has quit IRC | 23:49 | |
| *** kaisers_ has joined #openstack-infra | 23:49 | |
| *** itsuugo has quit IRC | 23:50 | |
| *** psilvad has joined #openstack-infra | 23:51 | |
| *** salv-orlando has joined #openstack-infra | 23:52 | |
| *** itsuugo has joined #openstack-infra | 23:52 | |
| *** markvoelker has quit IRC | 23:52 | |
| *** kaisers_ has quit IRC | 23:54 | |
| mtreinish | fungi: so I wrote a little ruby test script (which I can add to the firehose docs for fun if you think it's worth it) http://paste.openstack.org/show/582834/ | 23:55 |
| mtreinish | and that works fine | 23:55 |
| mtreinish | so I bet it's definitely something with the content of the hiera key | 23:56 |
| *** itsuugo has quit IRC | 23:56 | |
| *** salv-orlando has quit IRC | 23:57 | |
| mtreinish | fungi: we probably could just download the same bundle I did from instantssl and use that | 23:57 |
| *** itsuugo has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
