| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Serve keys from canonical project name https://review.openstack.org/504807 | 00:03 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add support for result data in child jobs https://review.openstack.org/504808 | 00:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: add abstract job attribute https://review.openstack.org/504809 | 00:03 |
| *** mat128 has joined #openstack-infra | 00:05 | |
| dmsimard | Even a recheck doesn't trigger anything, I suspect there's probably some errors somewhere in logs.. maybe it's the depends-on(s) ? | 00:06 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 00:06 |
| dmsimard | Yup, as soon as I removed the depends-on, it queued | 00:06 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add multinode integration jobs and integration tests for known_hosts https://review.openstack.org/504787 | 00:07 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 00:07 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge https://review.openstack.org/504789 | 00:08 |
| *** mat128 has quit IRC | 00:08 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge https://review.openstack.org/504789 | 00:08 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 00:09 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge https://review.openstack.org/504789 | 00:09 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add multinode integration jobs and integration tests for known_hosts https://review.openstack.org/504787 | 00:09 |
| *** thorst has joined #openstack-infra | 00:13 | |
| *** thorst has quit IRC | 00:13 | |
| dmsimard | I basically had a tree of 3 patches with a depends-on on the same patch http://paste.openstack.org/raw/621297/ | 00:16 |
| dmsimard | To make it work I had to remove the depends-on on the two 'child' patches | 00:16 |
| dmsimard | Sounds like Zuul could at least tell me how horribly wrong I was | 00:16 |
| *** sshnaidm has joined #openstack-infra | 00:43 | |
| *** LindaWang has joined #openstack-infra | 00:55 | |
| fungi | dmsimard: any chance there was a circular dep? if so, that's basically feature parity with zuul v2 for now | 00:59 |
| *** yamahata has joined #openstack-infra | 01:05 | |
| *** thorst has joined #openstack-infra | 01:08 | |
| *** thorst has quit IRC | 01:08 | |
| *** cody-somerville has quit IRC | 01:16 | |
| *** cshastri has joined #openstack-infra | 01:23 | |
| *** cuongnv has joined #openstack-infra | 01:33 | |
| *** liujiong has joined #openstack-infra | 01:36 | |
| *** bobh has joined #openstack-infra | 01:51 | |
| *** mwarad has joined #openstack-infra | 01:54 | |
| *** mat128 has joined #openstack-infra | 01:54 | |
| *** dave-mccowan has joined #openstack-infra | 01:55 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Actually sort mount-point list https://review.openstack.org/504819 | 01:55 |
| *** dave-mcc_ has joined #openstack-infra | 01:58 | |
| *** mat128 has quit IRC | 02:00 | |
| *** dave-mccowan has quit IRC | 02:00 | |
| *** Daisy_ has joined #openstack-infra | 02:08 | |
| *** Daisy_ has quit IRC | 02:08 | |
| *** thorst has joined #openstack-infra | 02:09 | |
| *** gouthamr has quit IRC | 02:10 | |
| *** jmorgan1_ has quit IRC | 02:11 | |
| *** jmorgan1 has joined #openstack-infra | 02:11 | |
| *** thorst has quit IRC | 02:14 | |
| *** hongbin has joined #openstack-infra | 02:14 | |
| *** cody-somerville has joined #openstack-infra | 02:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Use olso to parse size strings https://review.openstack.org/503574 | 02:22 |
| *** cody-somerville has quit IRC | 02:27 | |
| *** gongysh has joined #openstack-infra | 02:35 | |
| *** yamahata has quit IRC | 02:37 | |
| *** larainema has quit IRC | 02:38 | |
| *** dave-mcc_ has quit IRC | 02:39 | |
| *** cody-somerville has joined #openstack-infra | 02:51 | |
| *** ramishra has joined #openstack-infra | 02:52 | |
| *** cody-somerville has quit IRC | 02:57 | |
| *** yamahata has joined #openstack-infra | 02:59 | |
| *** gouthamr has joined #openstack-infra | 03:01 | |
| *** tiswanso has joined #openstack-infra | 03:01 | |
| *** tiswanso has quit IRC | 03:01 | |
| *** tiswanso has joined #openstack-infra | 03:02 | |
| *** jesusaur has quit IRC | 03:08 | |
| *** baoli has quit IRC | 03:08 | |
| *** thorst has joined #openstack-infra | 03:10 | |
| *** thorst has quit IRC | 03:14 | |
| *** gouthamr has quit IRC | 03:17 | |
| *** cody-somerville has joined #openstack-infra | 03:17 | |
| *** cody-somerville has joined #openstack-infra | 03:17 | |
| *** bobh has quit IRC | 03:19 | |
| *** bobh has joined #openstack-infra | 03:22 | |
| openstackgerrit | zhangyangyang proposed openstack-infra/log_processor master: Remove py26 support https://review.openstack.org/504831 | 03:22 |
| *** hongbin has quit IRC | 03:25 | |
| *** hongbin has joined #openstack-infra | 03:25 | |
| *** bobh has quit IRC | 03:26 | |
| *** jesusaur has joined #openstack-infra | 03:27 | |
| *** tiswanso has quit IRC | 03:33 | |
| openstackgerrit | zhangyangyang proposed openstack-infra/nodepool master: Remove py26 support https://review.openstack.org/504836 | 03:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Upgrade review.openstack.org to Gerrit 2.13 https://review.openstack.org/504838 | 03:41 |
| clarkb | fungi: I think getting ^ proposed was the last thing I needed to do before running through the upgrade steps tomorrow. Note that the new db schema uses the existing trove user and passwd for access if you end up wanting to create that before I wake | 03:44 |
| clarkb | I will see everyone in the morning | 03:44 |
| *** hongbin has quit IRC | 03:45 | |
| *** ihrachys has joined #openstack-infra | 03:45 | |
| *** yamahata has quit IRC | 03:53 | |
| *** yamahata has joined #openstack-infra | 03:55 | |
| *** gildub_ has joined #openstack-infra | 03:55 | |
| *** ihrachys has quit IRC | 03:56 | |
| openstackgerrit | zhangyangyang proposed openstack-infra/os-loganalyze master: Remove py26 support https://review.openstack.org/504842 | 03:57 |
| *** mwarad has quit IRC | 03:58 | |
| openstackgerrit | zhangyangyang proposed openstack-dev/pbr master: Remove py26 support https://review.openstack.org/504843 | 04:01 |
| SpamapS | anybody around who can tell me what infra's zuulv3 has for [webapp]status_url ? | 04:10 |
| * SpamapS has sort of given up trying to find how @status_url gets set :-P | 04:10 | |
| *** thorst has joined #openstack-infra | 04:11 | |
| *** psachin has joined #openstack-infra | 04:15 | |
| *** thorst has quit IRC | 04:15 | |
| *** gongysh has quit IRC | 04:16 | |
| *** yamahata has quit IRC | 04:17 | |
| *** ykarel has joined #openstack-infra | 04:18 | |
| *** ykarel has quit IRC | 04:23 | |
| *** Sree has joined #openstack-infra | 04:29 | |
| *** gongysh has joined #openstack-infra | 04:34 | |
| *** yamahata has joined #openstack-infra | 04:37 | |
| *** Sree has quit IRC | 04:39 | |
| *** Sree has joined #openstack-infra | 04:40 | |
| *** Sree has quit IRC | 04:44 | |
| *** links has joined #openstack-infra | 04:46 | |
| openstackgerrit | Ankit Kumar proposed openstack-infra/project-config master: Add experimental py35 gate for ironic-inspector https://review.openstack.org/462695 | 04:53 |
| *** gildub_ has quit IRC | 04:59 | |
| *** dhajare has joined #openstack-infra | 05:03 | |
| *** larainema has joined #openstack-infra | 05:06 | |
| *** thorst has joined #openstack-infra | 05:11 | |
| *** cody-somerville has quit IRC | 05:13 | |
| *** thorst has quit IRC | 05:16 | |
| *** jaosorior has joined #openstack-infra | 05:22 | |
| *** jbadiapa has joined #openstack-infra | 05:23 | |
| *** bobh has joined #openstack-infra | 05:23 | |
| *** Sree has joined #openstack-infra | 05:25 | |
| *** bobh has quit IRC | 05:27 | |
| *** david-lyle has quit IRC | 05:30 | |
| *** Dinesh_Bhor has joined #openstack-infra | 05:33 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Only inject cloud-init in CentOS 7.3 https://review.openstack.org/504850 | 05:36 |
| *** gongysh has quit IRC | 05:39 | |
| *** udesale has joined #openstack-infra | 05:39 | |
| *** jtomasek has joined #openstack-infra | 05:48 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Update Gentoo element for element changes https://review.openstack.org/503844 | 05:50 |
| *** khappone has joined #openstack-infra | 05:53 | |
| *** dhajare has quit IRC | 05:57 | |
| *** udesale has quit IRC | 06:01 | |
| *** AJaeger has quit IRC | 06:03 | |
| *** martinkopec has joined #openstack-infra | 06:06 | |
| *** AJaeger has joined #openstack-infra | 06:07 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Only inject cloud-init in CentOS 7.3 https://review.openstack.org/504850 | 06:09 |
| *** yolanda has joined #openstack-infra | 06:11 | |
| *** thorst has joined #openstack-infra | 06:12 | |
| *** rwsu has joined #openstack-infra | 06:14 | |
| *** sshnaidm has quit IRC | 06:17 | |
| *** thorst has quit IRC | 06:17 | |
| *** bhavik1 has joined #openstack-infra | 06:19 | |
| *** dhajare has joined #openstack-infra | 06:23 | |
| *** bobh has joined #openstack-infra | 06:24 | |
| *** florianf has joined #openstack-infra | 06:25 | |
| *** sshnaidm has joined #openstack-infra | 06:25 | |
| *** jtomasek has quit IRC | 06:26 | |
| *** makowals has joined #openstack-infra | 06:27 | |
| *** bobh has quit IRC | 06:29 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Update Gentoo element for element changes https://review.openstack.org/503844 | 06:30 |
| *** jtomasek has joined #openstack-infra | 06:37 | |
| *** Sree has quit IRC | 06:44 | |
| *** aeng has quit IRC | 06:45 | |
| *** hashar has joined #openstack-infra | 06:46 | |
| *** shardy_afk is now known as shardy | 06:52 | |
| *** Sree has joined #openstack-infra | 06:53 | |
| *** Sree has quit IRC | 06:58 | |
| *** threestrands has quit IRC | 06:58 | |
| *** ykarel has joined #openstack-infra | 06:59 | |
| *** eumel8 has joined #openstack-infra | 07:02 | |
| *** ccamacho has joined #openstack-infra | 07:03 | |
| *** pgadiya has joined #openstack-infra | 07:04 | |
| *** jpena|off is now known as jpena | 07:05 | |
| *** thorst has joined #openstack-infra | 07:13 | |
| *** rcernin has joined #openstack-infra | 07:13 | |
| *** sshnaidm has quit IRC | 07:15 | |
| *** Sree has joined #openstack-infra | 07:16 | |
| *** Sree has quit IRC | 07:16 | |
| *** Sree has joined #openstack-infra | 07:17 | |
| *** bhavik1 has quit IRC | 07:17 | |
| *** thorst has quit IRC | 07:18 | |
| *** alexchadin has joined #openstack-infra | 07:19 | |
| *** pcaruana has joined #openstack-infra | 07:21 | |
| *** Sree has quit IRC | 07:21 | |
| *** tesseract has joined #openstack-infra | 07:22 | |
| apuimedo | yolanda: could you please review https://review.openstack.org/#/c/504374/ ? | 07:22 |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/irc-meetings master: Switch chair for RPM packaging https://review.openstack.org/504865 | 07:26 |
| *** danpawlik has joined #openstack-infra | 07:27 | |
| *** sshnaidm has joined #openstack-infra | 07:27 | |
| *** shardy is now known as shardy_afk | 07:28 | |
| *** szaher has joined #openstack-infra | 07:28 | |
| frickler | infra-root: is apps.o.o still a thing? no A record, only AAAA, and seems not to respond there either | 07:34 |
| *** sshnaidm has quit IRC | 07:36 | |
| AJaeger | frickler: I think that was taken out of commission, let's check... | 07:39 |
| frickler | AJaeger: someone forgetting to remove the AAAA would seem rather plausible | 07:41 |
| AJaeger | frickler: https://wiki.openstack.org/wiki/Infrastructure_Status - search for apps | 07:42 |
| AJaeger | frickler: let's tell fungi about it ^ | 07:43 |
| frickler | AJaeger: agree, interesting page, don't think I've seen that before *bookmark* | 07:44 |
| AJaeger | frickler: http://lists.openstack.org/pipermail/openstack-operators/2017-July/013965.html | 07:44 |
| AJaeger | frickler: that page records all the #status log/alert etc messages we send | 07:45 |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| frickler | AJaeger: cool, I've always wondered where those might get logged :) | 07:46 |
| AJaeger | frickler: and tweeted - see also the box at https://docs.openstack.org/infra/manual/ | 07:47 |
| *** gongysh has joined #openstack-infra | 07:48 | |
| frickler | AJaeger: a box titled "Important" on top of a page, who would ever read such a thing? ;) | 07:48 |
| *** ralonsoh has joined #openstack-infra | 07:52 | |
| *** shardy_afk is now known as shardy | 07:53 | |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin https://review.openstack.org/504870 | 07:54 |
| *** dizquierdo has joined #openstack-infra | 07:54 | |
| AJaeger | frickler: should we use a smaller font? White on white? ;) | 07:55 |
| openstackgerrit | Aparna proposed openstack/diskimage-builder master: Change to install a package in 'proliant-tools' https://review.openstack.org/501568 | 08:01 |
| *** LindaWang has quit IRC | 08:05 | |
| *** shardy is now known as shardy_afk | 08:05 | |
| *** LindaWang has joined #openstack-infra | 08:06 | |
| *** dizquierdo has quit IRC | 08:07 | |
| *** LindaWang has quit IRC | 08:07 | |
| *** LindaWang has joined #openstack-infra | 08:09 | |
| *** LindaWang has quit IRC | 08:09 | |
| *** LindaWang has joined #openstack-infra | 08:11 | |
| *** egonzalez has joined #openstack-infra | 08:12 | |
| *** thorst has joined #openstack-infra | 08:14 | |
| *** akscram1 has quit IRC | 08:15 | |
| *** tinwood has quit IRC | 08:15 | |
| *** electrofelix has joined #openstack-infra | 08:15 | |
| *** akscram1 has joined #openstack-infra | 08:16 | |
| *** tinwood has joined #openstack-infra | 08:18 | |
| *** thorst has quit IRC | 08:19 | |
| *** jpich has joined #openstack-infra | 08:19 | |
| *** bobh has joined #openstack-infra | 08:26 | |
| *** slaweq has joined #openstack-infra | 08:27 | |
| *** bobh has quit IRC | 08:30 | |
| *** ykarel|lunch is now known as ykarel | 08:35 | |
| *** pgadiya has quit IRC | 08:35 | |
| *** shardy_afk is now known as shardy | 08:40 | |
| *** mwarad has joined #openstack-infra | 08:42 | |
| *** efoley has joined #openstack-infra | 08:42 | |
| odyssey4me | We have a patch that seems to not want to enter into the gate queue for some reason - I wonder if someone could take a peek to figure out why: https://review.openstack.org/504170 | 08:43 |
| AJaeger | odyssey4me: does it use depends-on? Or is on top of one that uses depends-on? | 08:44 |
| odyssey4me | AJaeger nope - just a single patch, ported back. | 08:44 |
| AJaeger | odyssey4me: toggle the +A - or add another +A. WE had some downtime over the weekend and the even might not have been recorded properly. | 08:45 |
| odyssey4me | AJaeger I rechecked it this morning. | 08:45 |
| AJaeger | odyssey4me: recheck will not help in those cases, Zuul reacts on *Changes* | 08:45 |
| AJaeger | odyssey4me: Just add another +A | 08:45 |
| odyssey4me | AJaeger you also rechecked it after the CI failure ;) | 08:45 |
| odyssey4me | ok, added another workflow | 08:46 |
| AJaeger | odyssey4me: I did - in the hope that this would work out ;) | 08:46 |
| odyssey4me | ok, that did it | 08:46 |
| odyssey4me | thanks! | 08:46 |
| AJaeger | next time: As a core, a +A will be quicker than a recheck ;) | 08:46 |
| odyssey4me | understood | 08:47 |
| AJaeger | glad to see the change in the gate now... | 08:48 |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** _ari_ has quit IRC | 09:07 | |
| *** weshay has quit IRC | 09:07 | |
| *** _ari_ has joined #openstack-infra | 09:07 | |
| *** ssbarnea has quit IRC | 09:08 | |
| *** weshay has joined #openstack-infra | 09:10 | |
| *** slaweq has quit IRC | 09:10 | |
| *** ssbarnea has joined #openstack-infra | 09:10 | |
| *** slaweq has joined #openstack-infra | 09:11 | |
| *** Douhet has quit IRC | 09:11 | |
| *** thorst has joined #openstack-infra | 09:15 | |
| *** thorst has quit IRC | 09:19 | |
| *** Sree has joined #openstack-infra | 09:24 | |
| *** bobh has joined #openstack-infra | 09:26 | |
| *** Sree has quit IRC | 09:29 | |
| *** bobh has quit IRC | 09:31 | |
| *** slaweq has quit IRC | 09:44 | |
| *** tosky has joined #openstack-infra | 09:46 | |
| *** slaweq has joined #openstack-infra | 09:47 | |
| *** otherwiseguy has quit IRC | 09:48 | |
| *** pabelanger has quit IRC | 09:48 | |
| *** dtantsur is now known as dtantsur|afk | 09:49 | |
| *** dgonzalez has left #openstack-infra | 09:49 | |
| *** LindaWang has quit IRC | 09:49 | |
| *** _ari_ has quit IRC | 09:49 | |
| *** ssbarnea has quit IRC | 09:50 | |
| *** ssbarnea has joined #openstack-infra | 09:50 | |
| *** _ari_ has joined #openstack-infra | 09:50 | |
| *** otherwiseguy has joined #openstack-infra | 09:51 | |
| *** pabelanger has joined #openstack-infra | 09:51 | |
| *** ociuhandu has quit IRC | 09:52 | |
| *** ociuhandu has joined #openstack-infra | 09:54 | |
| *** mwarad has quit IRC | 09:57 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Use zaqar-tempest-plugin for zaqar jobs https://review.openstack.org/504901 | 09:59 |
| *** cuongnv has quit IRC | 10:03 | |
| *** rcarrillocruz has quit IRC | 10:04 | |
| *** cshastri has quit IRC | 10:05 | |
| *** Hal has joined #openstack-infra | 10:09 | |
| *** Hal is now known as Guest82210 | 10:09 | |
| *** rcarrillocruz has joined #openstack-infra | 10:10 | |
| *** efoley is now known as elfoley | 10:13 | |
| *** elfoley is now known as efoley | 10:14 | |
| *** thorst has joined #openstack-infra | 10:16 | |
| *** alexchadin has quit IRC | 10:18 | |
| *** alexchadin has joined #openstack-infra | 10:19 | |
| *** alexchadin has quit IRC | 10:19 | |
| *** alexchadin has joined #openstack-infra | 10:19 | |
| *** alexchadin has quit IRC | 10:20 | |
| *** alexchadin has joined #openstack-infra | 10:20 | |
| *** thorst has quit IRC | 10:20 | |
| *** alexchadin has quit IRC | 10:20 | |
| *** alexchadin has joined #openstack-infra | 10:21 | |
| *** nicolasbock has joined #openstack-infra | 10:22 | |
| *** tosky is now known as tosky_ | 10:23 | |
| *** tosky_ is now known as tosky | 10:24 | |
| *** alexchadin has quit IRC | 10:25 | |
| *** bobh has joined #openstack-infra | 10:27 | |
| *** egonzalez has quit IRC | 10:29 | |
| *** bobh has quit IRC | 10:32 | |
| *** wolverineav has joined #openstack-infra | 10:33 | |
| *** wolverineav has quit IRC | 10:33 | |
| *** wolverineav has joined #openstack-infra | 10:34 | |
| *** alex_xu has quit IRC | 10:35 | |
| *** alex_xu has joined #openstack-infra | 10:36 | |
| *** Qiming has quit IRC | 10:40 | |
| *** amotoki has quit IRC | 10:41 | |
| *** Qiming has joined #openstack-infra | 10:41 | |
| *** amotoki has joined #openstack-infra | 10:43 | |
| *** dizquierdo has joined #openstack-infra | 10:43 | |
| *** rhallisey has joined #openstack-infra | 10:43 | |
| *** ociuhandu has quit IRC | 10:44 | |
| *** egonzalez has joined #openstack-infra | 10:45 | |
| *** e0ne has joined #openstack-infra | 10:48 | |
| *** Douhet has joined #openstack-infra | 10:49 | |
| *** jkilpatr has quit IRC | 10:49 | |
| *** mat128 has joined #openstack-infra | 10:53 | |
| *** ffledgling has joined #openstack-infra | 10:54 | |
| *** psachin has quit IRC | 10:54 | |
| ffledgling | Hello, is this right place for questions about jenkins-job-builder? | 10:54 |
| *** links has quit IRC | 10:55 | |
| *** e0ne has quit IRC | 11:04 | |
| *** liujiong has quit IRC | 11:06 | |
| *** e0ne has joined #openstack-infra | 11:08 | |
| *** LindaWang has joined #openstack-infra | 11:08 | |
| *** ociuhandu has joined #openstack-infra | 11:08 | |
| *** jkilpatr has joined #openstack-infra | 11:09 | |
| *** sdague_ has joined #openstack-infra | 11:11 | |
| *** gongysh has quit IRC | 11:15 | |
| *** gongysh has joined #openstack-infra | 11:17 | |
| *** alexchadin has joined #openstack-infra | 11:22 | |
| *** alexchadin has quit IRC | 11:24 | |
| *** egonzalez has quit IRC | 11:26 | |
| *** bobh has joined #openstack-infra | 11:28 | |
| *** sdague has joined #openstack-infra | 11:31 | |
| *** mat128 has quit IRC | 11:32 | |
| *** alexchadin has joined #openstack-infra | 11:32 | |
| *** mat128 has joined #openstack-infra | 11:33 | |
| *** bobh has quit IRC | 11:33 | |
| AJaeger | ffledgling: yes, in general - but might be wrong time. Just ask and if nobody answers, ask again at US morning time. Please stay in the channel, people might read your question and answer later | 11:36 |
| ffledgling | AJaeger: got it | 11:37 |
| ffledgling | I was wondering if there's a standard way to convert handwritten XML Jenkins configs to jjb job configs (or at least get a rough cut of some sort to get started with). I see a few OSS projects, that I'm going to try, but I was wondering if there's a standard/accepting way of doing it | 11:38 |
| AJaeger | ffledgling: no idea - hope others can help... | 11:39 |
| *** ldnunes has joined #openstack-infra | 11:42 | |
| *** jamesdenton has joined #openstack-infra | 11:46 | |
| *** mat128 has quit IRC | 11:46 | |
| *** dminca has joined #openstack-infra | 11:48 | |
| openstackgerrit | Justin Kilpatrick proposed openstack-infra/project-config master: Add devstack job for Cyborg https://review.openstack.org/504141 | 11:48 |
| dminca | hi guys | 11:48 |
| *** sambetts_ is now known as sambetts | 11:53 | |
| *** arxcruz has quit IRC | 12:01 | |
| fungi | AJaeger: frickler: yes, i thought i had deleted all the dns records but i must have missed the aaaa. the service was decommissioned months ago after the community agreed it was better to help support other application registries emergent in the ecosystem rather than create our own to compete with them | 12:02 |
| fungi | i'll make a note to clean up that rr in a bit | 12:02 |
| *** thorst has joined #openstack-infra | 12:03 | |
| fungi | #status notice Gerrit will be offline for the upgrade to 2.13 starting at 15:00 UTC (in roughly 3 hours) and is expected to probably be down/unusable for 8+ hours while an offline reindex is performed: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 12:04 |
| openstackstatus | fungi: sending notice | 12:04 |
| *** mattmceuen has quit IRC | 12:05 | |
| -openstackstatus- NOTICE: Gerrit will be offline for the upgrade to 2.13 starting at 15:00 UTC (in roughly 3 hours) and is expected to probably be down/unusable for 8+ hours while an offline reindex is performed: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 12:05 | |
| *** sdague has quit IRC | 12:06 | |
| *** dave-mccowan has joined #openstack-infra | 12:06 | |
| *** trown|ptg is now known as trown | 12:07 | |
| *** egonzalez has joined #openstack-infra | 12:07 | |
| *** jpena is now known as jpena|lunch | 12:07 | |
| *** dave-mcc_ has joined #openstack-infra | 12:08 | |
| fungi | ffledgling: AJaeger: there's also an #openstack-jjb channel specifically for jenkins job builder discussions, which seems to be relatively active (i lurk in there but don't tend to get involved unless someone asks me infra-specific questions) | 12:11 |
| *** dave-mccowan has quit IRC | 12:11 | |
| AJaeger | fungi: thanks, wasn't aware of that one... | 12:14 |
| *** jcoufal has joined #openstack-infra | 12:16 | |
| *** rlandy has joined #openstack-infra | 12:17 | |
| *** tiswanso has joined #openstack-infra | 12:17 | |
| *** sdague_ is now known as sdague | 12:18 | |
| *** sshnaidm has joined #openstack-infra | 12:20 | |
| smcginnis | So the upgrade outage is expected to be pretty much the whole US work day? | 12:21 |
| fungi | smcginnis: yep | 12:21 |
| smcginnis | fungi: Guess this will be a good day to get my basement cloud up to pike then. ;) | 12:22 |
| fungi | expectation was monday after the ptg would be slow, but not so slow that people won't be able to spot subtle breakage we miss when it comes back online | 12:22 |
| smcginnis | Yeah, I would bet there are a few folks still travelling. At least a lot of the non-US folks. | 12:23 |
| fungi | we may be lucky and finish faster, but previous reindexes have taken quite a long time to complete | 12:23 |
| smcginnis | Best not to be too optimistic for upgrades. | 12:24 |
| fungi | also, one way to look at it is that the infra team is encouraging geographic diversity by not taking the service offline during EMEA/APAC workdays ;) | 12:26 |
| smcginnis | Hah | 12:27 |
| AJaeger | fungi, that's an interesting twist! ;) | 12:28 |
| *** tpsilva has joined #openstack-infra | 12:29 | |
| *** arxcruz has joined #openstack-infra | 12:35 | |
| *** markmcd has quit IRC | 12:37 | |
| openstackgerrit | Justin Kilpatrick proposed openstack-infra/project-config master: Add devstack job for Cyborg https://review.openstack.org/504141 | 12:41 |
| *** markmcd has joined #openstack-infra | 12:46 | |
| *** dminca has quit IRC | 12:48 | |
| ffledgling | fungi: thanks, let me re-ask there... | 12:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove #openstack-neutron-ovn. https://review.openstack.org/501722 | 12:54 |
| *** tiswanso has quit IRC | 12:54 | |
| *** markmcd has quit IRC | 12:54 | |
| *** tiswanso has joined #openstack-infra | 12:55 | |
| *** Goneri has joined #openstack-infra | 12:56 | |
| *** markmcd has joined #openstack-infra | 12:57 | |
| *** bobh has joined #openstack-infra | 12:59 | |
| *** dave-mcc_ has quit IRC | 12:59 | |
| *** dprince has joined #openstack-infra | 13:00 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Only uninstall rdo-release if it was installed in the first place https://review.openstack.org/504936 | 13:02 |
| *** LindaWang has quit IRC | 13:07 | |
| *** dminca has joined #openstack-infra | 13:09 | |
| *** Sree has joined #openstack-infra | 13:09 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set repo setup release in playbook https://review.openstack.org/504939 | 13:10 |
| *** hrybacki|ptg is now known as hrybacki | 13:10 | |
| *** dminca has quit IRC | 13:13 | |
| *** xyang1 has joined #openstack-infra | 13:18 | |
| *** kgiusti has joined #openstack-infra | 13:18 | |
| *** jpena|lunch is now known as jpena | 13:18 | |
| *** esberglu has joined #openstack-infra | 13:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add new repo for Neutron tempest plugin https://review.openstack.org/502224 | 13:19 |
| *** baoli has joined #openstack-infra | 13:19 | |
| *** sshnaidm has quit IRC | 13:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Publish Freezer docs https://review.openstack.org/504329 | 13:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Really fix kuryr-kubernetes Octavia job https://review.openstack.org/504374 | 13:25 |
| openstackgerrit | Merged openstack-infra/project-config master: networking-midonet: Move legacy job from experimental to check https://review.openstack.org/504240 | 13:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Update gerritbot branch alerts in #openstack-swift https://review.openstack.org/504127 | 13:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove pypy jobs from python-barbicanclient https://review.openstack.org/504230 | 13:26 |
| *** alexchadin has quit IRC | 13:27 | |
| *** alexchadin has joined #openstack-infra | 13:28 | |
| *** ihrachys has joined #openstack-infra | 13:29 | |
| *** dminca has joined #openstack-infra | 13:29 | |
| *** makowals has quit IRC | 13:30 | |
| *** martinkopec has quit IRC | 13:30 | |
| *** dave-mccowan has joined #openstack-infra | 13:31 | |
| fungi | huh, rackspace says elasticsearch02 was the target of a denial of service attack yesterday... i guess they're referring to this spike in packet rate between 16:50 and 17:10 utc yesterday: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1665&rra_id=all | 13:32 |
| *** makowals has joined #openstack-infra | 13:32 | |
| *** dminca has quit IRC | 13:33 | |
| fungi | 13.45kpps average at peak, though that's just one 5-minute sample so no clue how much higher it really went | 13:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove ironic nv job from tempest check pipeline https://review.openstack.org/504066 | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Add old python-barbicanclient gate logic back for the stable branches https://review.openstack.org/504118 | 13:34 |
| fungi | not seeing any similar spikes (inbound nor outbound) on any of the other es cluster members not the kibana server, so looks likely to be a random target | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Update gate job to stop cloning old gnocchi location. https://review.openstack.org/504083 | 13:34 |
| fungi | and not a very useful target at that | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Add devstack and tempest to the real section https://review.openstack.org/504158 | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Propose nova-lxd job using OpenStack-Ansible https://review.openstack.org/482185 | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Add test job for zun with split tempest plugin https://review.openstack.org/503825 | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-vpnaas: Move the coverage job from post to check https://review.openstack.org/503176 | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Barbican: Make dogtag job voting and add it to gate https://review.openstack.org/503846 | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new project networking-lagopus https://review.openstack.org/501730 | 13:34 |
| *** dminca has joined #openstack-infra | 13:34 | |
| fungi | #status notice Gerrit will be offline for the upgrade to 2.13 starting at 15:00 UTC (in roughly 1.5 hours) and is expected to probably be down/unusable for 8+ hours while an offline reindex is performed: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 13:35 |
| openstackstatus | fungi: sending notice | 13:35 |
| -openstackstatus- NOTICE: Gerrit will be offline for the upgrade to 2.13 starting at 15:00 UTC (in roughly 1.5 hours) and is expected to probably be down/unusable for 8+ hours while an offline reindex is performed: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 13:36 | |
| AJaeger | fungi, do you want to send an updated email with new subject like the IRC messages? | 13:37 |
| fungi | AJaeger: clarkb sent a reminder on friday. do you think we need more? | 13:38 |
| dmsimard | fungi: gl with the upgrade | 13:38 |
| fungi | thanks dmsimard! | 13:38 |
| fungi | #status log updatepuppetmaster cron job on puppetmaster.openstack.org has been disabled in preparation for the upcoming gerrit upgrade maintenance | 13:38 |
| openstackstatus | fungi: finished logging | 13:38 |
| *** alexchadin has quit IRC | 13:39 | |
| dmsimard | fungi: re: dmsimard: any chance there was a circular dep? if so, that's basically feature parity with zuul v2 for now | 13:39 |
| AJaeger | fungi: we should sent one today IMHO. Shall I? | 13:39 |
| *** mriedem has joined #openstack-infra | 13:39 | |
| fungi | AJaeger: sure, that would be a huge help--thanks! | 13:40 |
| *** mat128 has joined #openstack-infra | 13:40 | |
| dmsimard | fungi: I don't know, it looked like this: http://paste.openstack.org/raw/621297/ | 13:40 |
| fungi | i'm currently working on getting the additional accountPatchReviewDb created for 2.13 now | 13:40 |
| dmsimard | fungi: so you had 3 patches from openstack-zuul-jobs (based on top of each other) depending on a patch from zuul-jobs | 13:40 |
| *** dminca has quit IRC | 13:40 | |
| fungi | dmsimard: yep, that constitutes a loop | 13:40 |
| *** dhajare has quit IRC | 13:41 | |
| fungi | dmsimard: so known/expected behavior | 13:41 |
| dmsimard | fungi: fair, could zuul bubble up an exception like it currently does on things like syntax errors ? :) | 13:41 |
| *** mat128 has quit IRC | 13:42 | |
| fungi | dmsimard: i agree, that would be nice. i'm not sure what safety considerations there might be (i know the current behavior was chosen in order to avoid zuul doing anything dangerous when it identified a sequencing cycle) | 13:42 |
| *** makowals has quit IRC | 13:43 | |
| AJaeger | fungi: sent | 13:43 |
| fungi | dmsimard: the normal way to sequence them, as a patch author, would have been to make just the bottom git parent in the series depends-on the one from the other repo | 13:43 |
| fungi | then zuul would have been able to clearly identify a serial order | 13:44 |
| *** srobert has joined #openstack-infra | 13:44 | |
| dmsimard | fungi: right, I wasn't sure if the parent patch's depends-on would have pulled in the dep for the two child commits so I did it that way first and then removed them when I saw it wasn't working | 13:45 |
| *** mat128 has joined #openstack-infra | 13:46 | |
| fungi | yeah, zuul is smart enough to work out that cross-repo deps and git deps are transitive with one another | 13:46 |
| fungi | #status log accountPatchReviewDb database created and gerrit2 account granted access in Review-MySQL trove instance, in preparation for upcoming gerrit upgrade maintenance | 13:48 |
| openstackstatus | fungi: finished logging | 13:48 |
| *** felipemonteiro__ has joined #openstack-infra | 13:51 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:52 | |
| clarkb | good morning. Looks like prep is going well. I am about 15-20 minutes away from proper keyboard | 13:54 |
| *** dminca has joined #openstack-infra | 13:55 | |
| fungi | i think we're all set. i'm planning to do two more notices around 14:00 and 14:30 | 13:55 |
| *** felipemonteiro__ has quit IRC | 13:56 | |
| *** erlon has joined #openstack-infra | 13:57 | |
| *** makowals has joined #openstack-infra | 13:59 | |
| *** mat128 has quit IRC | 14:02 | |
| *** dminca has quit IRC | 14:04 | |
| *** Swami has joined #openstack-infra | 14:04 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: WIP: Honor cloud quotas before launching nodes https://review.openstack.org/503838 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional https://review.openstack.org/504282 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool https://review.openstack.org/504283 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded https://review.openstack.org/503051 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool https://review.openstack.org/504284 | 14:06 |
| *** armax has joined #openstack-infra | 14:07 | |
| *** trown is now known as trown|brb | 14:09 | |
| *** trown|brb is now known as trown | 14:12 | |
| clarkb | ok at proper keyboard now with tea | 14:16 |
| *** gongysh has joined #openstack-infra | 14:17 | |
| *** armax has quit IRC | 14:18 | |
| * mordred waves at clarkb and fungi | 14:18 | |
| *** marst_ has joined #openstack-infra | 14:19 | |
| *** marst_ has quit IRC | 14:19 | |
| clarkb | it is raining here. When I went to denver it was proper summer weather, when I got back it was to the worst air quality in the country due to forest fires and today it is fall | 14:20 |
| jeblair | o/ | 14:20 |
| fungi | in case anyone needs the pad url: https://etherpad.openstack.org/p/gerrit-2.13.-upgrade-steps | 14:21 |
| mnaser | good luck :D | 14:21 |
| fungi | we should be ready up to "stop zuul" now | 14:21 |
| fungi | thanks mnaser! | 14:21 |
| *** jtomasek has quit IRC | 14:22 | |
| jeblair | my gertty is synced | 14:23 |
| AJaeger | clarkb: raining? That's great for the fires, isn't it? | 14:24 |
| *** mat128 has joined #openstack-infra | 14:25 | |
| clarkb | AJaeger: ya, though there is now concern of flooding and landslides in the fire area | 14:25 |
| clarkb | I am ready for the rain its been hot and dry for too long | 14:26 |
| jeblair | also if the ash runs into reservoirs it's bad too. | 14:26 |
| AJaeger | nothing perfect ;( | 14:26 |
| clarkb | fungi: looks like you set the charactern encoding and collation on the databaase as well. Thanks for getting everything prepped | 14:27 |
| mnaser | (except openstack) | 14:27 |
| *** chlong has joined #openstack-infra | 14:27 | |
| *** LindaWang has joined #openstack-infra | 14:28 | |
| *** LindaWang has quit IRC | 14:29 | |
| *** Guest82210 has quit IRC | 14:29 | |
| fungi | clarkb: yep, no sweat | 14:29 |
| fungi | #status notice Gerrit will be offline for the upgrade to 2.13 starting at 15:00 UTC (in roughly 30 minutes) and is expected to probably be down/unusable for 8+ hours while an offline reindex is performed: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 14:30 |
| openstackstatus | fungi: sending notice | 14:30 |
| -openstackstatus- NOTICE: Gerrit will be offline for the upgrade to 2.13 starting at 15:00 UTC (in roughly 30 minutes) and is expected to probably be down/unusable for 8+ hours while an offline reindex is performed: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 14:31 | |
| *** hongbin has joined #openstack-infra | 14:32 | |
| *** rbrndt has joined #openstack-infra | 14:33 | |
| *** david-lyle has joined #openstack-infra | 14:34 | |
| *** Apoorva has joined #openstack-infra | 14:36 | |
| *** e0ne has quit IRC | 14:37 | |
| *** krtaylor has joined #openstack-infra | 14:37 | |
| *** chlong has quit IRC | 14:37 | |
| *** makowals has quit IRC | 14:37 | |
| *** makowals has joined #openstack-infra | 14:38 | |
| *** dminca has joined #openstack-infra | 14:38 | |
| *** ykarel has quit IRC | 14:38 | |
| *** chlong has joined #openstack-infra | 14:40 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Modify the check-uc jobs to only trigger on upper-constratints changes https://review.openstack.org/504501 | 14:41 |
| mordred | clarkb, fungi, jeblair: should I follow up to the last zuul rollout email with an update on status and the next-monday rollout intent? | 14:44 |
| fungi | mordred: maybe wait until after we're into the gerrit upgrade window so that people don't get confused? not sure if anyone would | 14:45 |
| jeblair | mordred, fungi: ++ | 14:45 |
| *** Apoorva has quit IRC | 14:45 | |
| mordred | ++ | 14:45 |
| clarkb | "I thought you said next monday!" | 14:46 |
| clarkb | I've gone ahead and wgetted the war into /tmp on review.o.o | 14:46 |
| clarkb | being the same DC that probably could've happened as scheduled on the etherpad though. Was quite fast | 14:47 |
| fungi | also, reminder that we wanted to announce https://review.openstack.org/482856 some time today (though after discovering that zuul master isn't enqueuing changes from check to gate for us without that, i wonder if there's any point to waiting to merge it) | 14:47 |
| *** makowals has quit IRC | 14:47 | |
| clarkb | fungi: for most third party ci (maybe all?) that isn' ta problem so probably best to announce it and then merge and/or tag later this week as originally planned | 14:48 |
| prometheanfire | time for some luck :D | 14:49 |
| clarkb | I'm expecting the reindex step to be the "go and make lunch/dinner, send emails, etc" portion of the day | 14:49 |
| fungi | ahh, yeah i guess merging it does cause them to need config adjustments if they're following master rather than pinning to a version like puppet-openstackci defaults to | 14:49 |
| clarkb | if anyone is wondering where they might squeeze that stuff in | 14:49 |
| jeblair | file expense reports. review changes in gertty. | 14:49 |
| *** makowals has joined #openstack-infra | 14:50 | |
| prometheanfire | not all charges may be on the cards yet :| | 14:50 |
| prometheanfire | review, but don't +W :P | 14:50 |
| *** jtomasek has joined #openstack-infra | 14:51 | |
| *** dminca has quit IRC | 14:53 | |
| *** dminca has joined #openstack-infra | 14:54 | |
| clarkb | jeblair: did you want to be in charge of stopping/starting zuul? Will likely have to reinstall with the tobias change because puppet probably undid my install over the weekend | 14:54 |
| fungi | yeah, expense report catch-up is my #2 task once the reindex is underway | 14:54 |
| jeblair | clarkb: will do | 14:54 |
| clarkb | jeblair: I have a branch in /opt/zuul on that node with tobias change rebased on master locally | 14:54 |
| clarkb | jeblair: since I wasn't sure if there were any other changes to master that we wanted too | 14:54 |
| jeblair | clarkb: got it | 14:54 |
| jeblair | puppet is stopped globally, right? | 14:55 |
| clarkb | fungi: ^ | 14:55 |
| *** Apoorva has joined #openstack-infra | 14:55 | |
| pabelanger | morning | 14:55 |
| fungi | correct | 14:56 |
| fungi | has been for a few hours now | 14:56 |
| fungi | i've triple-checked the cronjob and ps on puppetmaster | 14:56 |
| *** xarses_ has joined #openstack-infra | 14:56 | |
| clarkb | we will puppet infracloud but that is it | 14:56 |
| fungi | the cronjob for infracloud is still intact but shouldn't pose a problem | 14:56 |
| clarkb | ++ | 14:56 |
| fungi | yeah | 14:56 |
| fungi | i'll prep the status alert | 14:56 |
| jeblair | patched zuul installed; ready to stop. | 14:57 |
| *** gongysh has quit IRC | 14:57 | |
| clarkb | fungi: you may want to create the screen on review.o.o so that it is appropriately sized :) | 14:57 |
| *** dminca has quit IRC | 14:58 | |
| fungi | status notice The Gerrit service at https://review.openstack.org/ is offline for the upgrade to 2.13, for an indeterminate period of time hopefully not to exceed 23:59 UTC today: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 14:59 |
| fungi | ^ that look good? | 14:59 |
| clarkb | lgtm | 14:59 |
| fungi | clarkb: root screen created on review.o.o | 14:59 |
| fungi | remember to `su - gerrit2` from there as needed | 14:59 |
| clarkb | ya I tried to capture that in the etherpad | 14:59 |
| clarkb | fungi: I think you can go ahead and send that notice then we can get started | 15:00 |
| *** dminca has joined #openstack-infra | 15:00 | |
| fungi | #status notice The Gerrit service at https://review.openstack.org/ is offline, upgrading to 2.13, for an indeterminate period of time hopefully not to exceed 23:59 UTC today: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 15:01 |
| openstackstatus | fungi: sending notice | 15:01 |
| clarkb | I will stop gerrit and run the database backup once jeblair reports zuul is stopped | 15:02 |
| jeblair | i will stop zuul now | 15:02 |
| -openstackstatus- NOTICE: The Gerrit service at https://review.openstack.org/ is offline, upgrading to 2.13, for an indeterminate period of time hopefully not to exceed 23:59 UTC today: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 15:02 | |
| jeblair | zuul is stopped | 15:02 |
| jeblair | clarkb: ^ | 15:02 |
| clarkb | ok stopping gerrit now | 15:02 |
| jeblair | (i'm going to work on a maintenance webpage for gerrit) | 15:03 |
| clarkb | ok | 15:03 |
| fungi | thanks! i remember someone mentioning that but kept forgetting what the status was | 15:03 |
| clarkb | ok gerrit is stopped. Starting the db backup now | 15:03 |
| openstackstatus | fungi: finished sending notice | 15:04 |
| fungi | for those following along, reminder that the plan pad is: https://etherpad.openstack.org/p/gerrit-2.13.-upgrade-steps | 15:04 |
| clarkb | db backup is in progress | 15:05 |
| clarkb | thinking about disabling gerrit should we move the init script aside too? | 15:05 |
| clarkb | realizing the reindex will take time and ^ may be good belts and suspenders? | 15:05 |
| mordred | clarkb: ++ | 15:05 |
| clarkb | fungi: maybe you can do that? | 15:05 |
| fungi | i suppose that's not a terrible notion in case of unanticipated reboot | 15:05 |
| *** gouthamr has joined #openstack-infra | 15:05 | |
| fungi | shall i just `exit 0` after the shebang? | 15:06 |
| fungi | i have done that ^ for now | 15:06 |
| clarkb | I think its a symlink to the thing the war init installs? in which case that might be undone when we drop the new war in place (can just double check that afterwards) | 15:06 |
| clarkb | we are about half done on the db backup based on file sizes | 15:07 |
| fungi | ahh, yep, it's /home/gerrit2/review_site/bin/gerrit.sh so i guess i'll undo that and mv the symlink out of /etc/init.d/ to /root/ temporarily | 15:07 |
| *** gongysh has joined #openstack-infra | 15:08 | |
| jeblair | how does this look? https://review.openstack.org/ | 15:08 |
| clarkb | jeblair: ++ | 15:08 |
| jeblair | i edited 50-review.openstack.org.conf in place, but backed it up to ~root. i will update etherpad with revert steps | 15:09 |
| *** armax has joined #openstack-infra | 15:09 | |
| clarkb | ok | 15:09 |
| fungi | jeblair: maintenance page lgtm, thanks! | 15:10 |
| pabelanger | jeblair: thanks, I know I was on the hook for cowsay | 15:10 |
| clarkb | alright db backup is complete | 15:10 |
| *** Sree has quit IRC | 15:11 | |
| jeblair | pabelanger: we'll have plenty of time to update the content :) | 15:11 |
| clarkb | mordred: anythin gwe should do to check the validity of the db backup? | 15:11 |
| *** Sree has joined #openstack-infra | 15:11 | |
| clarkb | I'm moving ahead to updating the war | 15:12 |
| fungi | the cow says: maintenance | 15:12 |
| clarkb | that sha1sum lgtm. Anyone else want to double check it? | 15:12 |
| fungi | https://i.pinimg.com/736x/8e/c0/2c/8ec02c37ebafeed98cd47abd292e99af.jpg | 15:12 |
| *** gongysh has quit IRC | 15:12 | |
| pabelanger | fungi: hah | 15:13 |
| fungi | clarkb: checksum matches what i downloaded | 15:14 |
| jeblair | it matches the etherpad; that's immutable, right? | 15:14 |
| fungi | heh | 15:14 |
| *** Sree has quit IRC | 15:16 | |
| clarkb | ok moving on to cleaning up old libs | 15:16 |
| clarkb | now on to editing the secure.config | 15:17 |
| fungi | lookin' good | 15:17 |
| clarkb | fungi: same password for db connection right? | 15:18 |
| *** eumel8 has quit IRC | 15:18 | |
| *** annegentle has joined #openstack-infra | 15:18 | |
| *** annegentle has quit IRC | 15:18 | |
| *** annegentle has joined #openstack-infra | 15:18 | |
| fungi | clarkb: correct, same account | 15:19 |
| fungi | i simply added access to the new db to the existing account in that trove instance | 15:19 |
| fungi | also noticed we have a couple extra databases in that instance we could probably clean up (later) | 15:19 |
| *** martinkopec has joined #openstack-infra | 15:20 | |
| clarkb | ok, I have edited and written to that file (still open in vim in root screen_ if someone else can sanity check that | 15:20 |
| fungi | seems fine to me | 15:22 |
| clarkb | the next step is running gerrit init. Is there anything else we want to do before that? | 15:23 |
| clarkb | mordred: ^ maybe check the db backup validity somehow/ | 15:23 |
| clarkb | the size looked right at least | 15:24 |
| jeblair | it looks like a db backup and it looks appropriately large. | 15:25 |
| clarkb | ok not hearing any opposition I'm moving ahead and running the init step | 15:26 |
| clarkb | will run it under time too to collect data on production site so that we can record that and have it for the future | 15:27 |
| fungi | perfect | 15:27 |
| *** Apoorva has quit IRC | 15:28 | |
| clarkb | it is running, expected to take a few minutes | 15:28 |
| *** hashar is now known as hasharAway | 15:28 | |
| mordred | clarkb: yah - I'm comfortable with db backup validity - sorry for delay | 15:30 |
| clarkb | one of the underappreciated pluses to do this over irc is the built in timing info | 15:30 |
| mordred | right? | 15:30 |
| fungi | indeed, makes reconstructing timelines later a lot easyer | 15:31 |
| fungi | easier too | 15:31 |
| clarkb | and I just don't have to explicitly mark down a timestamp when I say " expected to take a few minutes " | 15:31 |
| *** dizquierdo has quit IRC | 15:31 | |
| clarkb | instead I can sip my tea | 15:31 |
| *** dminca has left #openstack-infra | 15:32 | |
| fungi | as a larger example, once the reindexing is underway i'm going to go grocery shopping ;) | 15:32 |
| *** vhosakot has joined #openstack-infra | 15:33 | |
| mordred | yah | 15:35 |
| *** martinkopec has quit IRC | 15:40 | |
| fungi | it's almost done i think | 15:41 |
| fungi | oh, maybe not | 15:41 |
| Diabelko | hello \o | 15:42 |
| Diabelko | I | 15:42 |
| Diabelko | I'm back with more questions | 15:42 |
| clarkb | this should go relatively quickly compared to the schema updates iirc | 15:42 |
| Diabelko | :) | 15:42 |
| Diabelko | Do you have any particular date in mind for Zuul v3 release? | 15:42 |
| clarkb | Diabelko: we are currently in the middle of a Gerrit upgrade. Discussing that in #zuul might be easiest today so that we can keep this channel focused on the gerrit upgrade | 15:44 |
| Diabelko | ok, thanks and sorry then | 15:44 |
| clarkb | no problem | 15:45 |
| fungi | i guess we didn't note the final schema revision number | 15:46 |
| clarkb | 129 iirc | 15:47 |
| fungi | oh, so nearly there after all | 15:47 |
| *** camunoz has joined #openstack-infra | 15:49 | |
| smcginnis | Wasn't the reindexing the part that was expected to take hours? | 15:51 |
| clarkb | smcginnis: yes, but this was also expected to take a while | 15:51 |
| *** lin_yang has joined #openstack-infra | 15:51 | |
| clarkb | it took almost 4 minutes on review-dev which hsa significantly less data on it | 15:51 |
| *** Sree has joined #openstack-infra | 15:53 | |
| clarkb | fungi: I feel like 127 heard you and is now taking its time | 15:56 |
| fungi | yeah, i shouldn't tempt fate | 15:56 |
| fungi | that must be one serious schema bump | 15:59 |
| *** sbezverk has quit IRC | 16:00 | |
| fungi | https://gerrit.googlesource.com/gerrit/+/master/gerrit-server/src/main/java/com/google/gerrit/server/schema/Schema_127.java | 16:02 |
| clarkb | this is moving all the data into that new database | 16:03 |
| clarkb | I think? | 16:04 |
| fungi | hard to say... maybe? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=34&rra_id=all | 16:05 |
| *** ijw has joined #openstack-infra | 16:05 | |
| clarkb | account_patch_reviews is the new table in accountPatchReviewDb I think | 16:05 |
| *** trown is now known as trown|lunch | 16:06 | |
| *** dtantsur|afk is now known as dtantsur | 16:08 | |
| clarkb | fungi: if you still have a db connection handy you should be able to easily check | 16:08 |
| fungi | 974459 rows so far | 16:10 |
| fungi | 985076 noq | 16:10 |
| fungi | now | 16:10 |
| fungi | so growing steadily, yes | 16:10 |
| mordred | clarkb, fungi: yes - that migration should be transferring data | 16:10 |
| fungi | select count(*) from account_patch_reviews; | 16:10 |
| fungi | if you want the quick and dirty rowcount | 16:11 |
| mordred | oh holy jesus - that's TERRIBLE SQL | 16:11 |
| clarkb | just imagine if we had run it against an h2 db | 16:11 |
| mordred | it could have been done in a single line of sql, at least for mysql, rather than a select into java followed by a billion inserts | 16:11 |
| *** supertakumi86 has joined #openstack-infra | 16:12 | |
| fungi | mordred: i expect that's attempting to emulate a h2 db in sql ;) | 16:12 |
| fungi | written by someone who mostly just knows h2? | 16:12 |
| clarkb | well mysql only works on our branch so ya... | 16:12 |
| mordred | for the record: "INSERT INTO account_patch_reviews (account_id, change_id, patch_set_id, file_name) SELECT account_id, change_id, patch_set_id, file_name from account_patch_reviews;" | 16:12 |
| mordred | not that we can do anything about that now | 16:12 |
| clarkb | ya time to just ride it out | 16:13 |
| fungi | up to 1110435 rows now, but no idea what the end count should be | 16:13 |
| mordred | yup. oh - is this the thing moving things from the one schema to the other? | 16:13 |
| clarkb | mordred: ya I think so | 16:13 |
| clarkb | mordred: by default into an h2 db | 16:13 |
| clarkb | but for us into the mysql db you unbroke | 16:14 |
| *** rcernin has quit IRC | 16:14 | |
| mordred | fungi: if you do "select count(*) from account_patch_reviews" on the other db | 16:14 |
| mordred | fungi: that should tell you target row count | 16:14 |
| fungi | ahh, i missed this is just migrating a table from one to the other | 16:14 |
| *** ramishra has quit IRC | 16:14 | |
| mordred | fungi: I'm guessing this is copying data from reviewdb to the new schedma right? | 16:14 |
| mordred | so it's select count(*) from reviewdb.account_patch_reviews ; select count(*) from accountPatchReviewDb.account_patch_reviews; | 16:15 |
| fungi | 11566783 in reviewdb.account_patch_reviews so we're ~10% complete now :( | 16:15 |
| mordred | boo | 16:15 |
| clarkb | wow | 16:15 |
| mordred | yah- it's a hideously inefficient way to do that | 16:15 |
| *** egonzalez has quit IRC | 16:16 | |
| *** pcaruana has quit IRC | 16:16 | |
| clarkb | good thing we allocated all the time for this upgrade :/ | 16:16 |
| fungi | have a feeling the outage is going to end up well past midnight utc given we're not even on to the reindex yet | 16:16 |
| fungi | maybe 2.13 offline reindex is faster (fingers crossed) | 16:17 |
| mordred | http://paste.openstack.org/show/621345/ <-- in case somehting goes wrong and we have to abort and try again later, we should update that code to do that | 16:17 |
| clarkb | its unfortunate that all of this code is basically unused by gerrit's maintainers | 16:17 |
| fungi | well, 11% now so it's going fast-ish i suppose | 16:17 |
| *** Sree has quit IRC | 16:20 | |
| fungi | 12%... maybe i should go get my grocery shopping out of the way now | 16:20 |
| *** Sree has joined #openstack-infra | 16:20 | |
| *** sbezverk has joined #openstack-infra | 16:22 | |
| *** dave-mccowan has quit IRC | 16:22 | |
| clarkb | ya this is on the order of a couple hours? I think I did that math right | 16:23 |
| fungi | we seem to be inching along at ~0.34% per minute, so at this rate we're looking at completion of the schema 127 migration at least 4 hours from now | 16:23 |
| *** gongysh has joined #openstack-infra | 16:23 | |
| clarkb | hrm fungis math is probably better than my math | 16:23 |
| fungi | (rough numbers based on timestamps in irc scrollback) | 16:23 |
| clarkb | that would almost certainly push the upgrade past 0000 UTC | 16:24 |
| fungi | indeed | 16:24 |
| jeblair | is it worth considering aborting the current process, applying mordred's change, and restarting? | 16:24 |
| fungi | i doubt we'll even start the reindex until 20:30-21:00 utc at the earliest | 16:24 |
| fungi | we'd need to rebuild a new gerrit war with the new schema migration, correct? | 16:25 |
| clarkb | jeblair: possibly? We'd have to get a new war built with that fix compiled in which may be painful (have to undo current upgade and get zuul running again) | 16:25 |
| clarkb | mordred: ^ how much faster would that be? | 16:25 |
| *** Sree has quit IRC | 16:25 | |
| fungi | we could try a manual gerrit build, but... | 16:25 |
| fungi | if things go wrong, we could end up losing more time on the workaround than the tested (if slow) path | 16:26 |
| clarkb | or we could do that migration step by hand but figuring out how to properly record that may also be tricky | 16:26 |
| jeblair | true... i guess all we'd really need to do is abort, perform 127 by hand, then resume? | 16:26 |
| mordred | jeblair: yah. I was thinking that earlier | 16:26 |
| fungi | assuming it can figure out 127 is actually applied, right | 16:27 |
| mordred | since the db schema states record which schema version they are | 16:27 |
| *** e0ne has joined #openstack-infra | 16:27 | |
| mordred | all the migration does is check the schema version in the db | 16:27 |
| fungi | okay, so that can be updated by hand i suppose | 16:27 |
| mordred | yup | 16:27 |
| clarkb | I'd defer to mordred/jeblair on deciding if that is a route we want to take as they are more familiar both with mysql and hte gerrit db | 16:27 |
| fungi | yeah, i'm certainly open to it if it shaves another 4+ hours off the maintenance | 16:28 |
| mordred | I think aborting the current java process, which should abort the sql | 16:28 |
| *** dizquierdo has joined #openstack-infra | 16:28 | |
| *** gongysh has quit IRC | 16:28 | |
| mordred | then doing the sql by hand - should be fine | 16:29 |
| clarkb | has it committed all of the previous migrations? | 16:29 |
| fungi | we have a solid db backup, so worst case we just end up with gerrit down for longer when we have to backtrack and retry... which is basically what we're already faced with | 16:29 |
| mordred | the jdbc is configured to do 1000 row batches | 16:29 |
| dmsimard | I would tend to err more on the conservative/safe side of things and say that around midnight UTC most people are no longer actively working and bleeding beyond midnight is probably not a huge impact | 16:29 |
| clarkb | dmsimard: ya and we intentionally chose today as it was expected to be quieter | 16:29 |
| *** jpich has quit IRC | 16:29 | |
| jeblair | db says we are at version 126 | 16:29 |
| fungi | dmsimard: yeah, but i'm not sure i personally want to be awake too much past that either ;) | 16:29 |
| fungi | (if it can be helped, that is) | 16:30 |
| mordred | so if we abort in the middle of one of the inserts (which we certaily will) the rollback size will only be 1000 rows, which will not be a long rollback | 16:30 |
| jeblair | clarkb: that suggests that we have committed previous updates | 16:30 |
| clarkb | jeblair: ya | 16:30 |
| dmsimard | fungi: of course :D | 16:30 |
| clarkb | mordred: but also we'd probably have to drop the entire table and start from scratch? or is ther ea good way to only insert what we havne't already insterted? | 16:30 |
| clarkb | mordred: remember that we have the auto incrementing file_id on that table in our setup | 16:31 |
| jeblair | i'd drop and start from scratch since this will be so much faster anyway | 16:31 |
| mordred | yah. lemme try something real quick | 16:31 |
| *** baoli has quit IRC | 16:32 | |
| jeblair | clarkb: and that should clear out the file_id autoincrement | 16:32 |
| clarkb | jeblair: ya | 16:32 |
| clarkb | fungi: where are we at now percentage wise? | 16:32 |
| *** baoli has joined #openstack-infra | 16:33 | |
| fungi | ~16.5% | 16:33 |
| * fungi also just realized he did a #status notice rather than alert | 16:33 | |
| clarkb | so ya ~0.4%/minute | 16:34 |
| * tosky points out that everyone probably noticed it at this point | 16:34 | |
| fungi | #status alert The Gerrit service at https://review.openstack.org/ is offline, upgrading to 2.13, for an indeterminate period of time hopefully not to exceed 23:59 UTC today: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 16:34 |
| openstackstatus | fungi: sending alert | 16:34 |
| clarkb | ~3.5 hours away | 16:34 |
| fungi | tosky: well, point being this will update channel topics too | 16:34 |
| tosky | oh, sure | 16:34 |
| fungi | so newcomers also notice, hopefully | 16:35 |
| -openstackstatus- NOTICE: The Gerrit service at https://review.openstack.org/ is offline, upgrading to 2.13, for an indeterminate period of time hopefully not to exceed 23:59 UTC today: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html | 16:36 | |
| *** ChanServ changes topic to "The Gerrit service at https://review.openstack.org/ is offline, upgrading to 2.13, for an indeterminate period of time hopefully not to exceed 23:59 UTC today: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120533.html" | 16:36 | |
| mordred | clarkb, fungi, jeblair: ok - I'm not sure the insert into / select from will be quicker - just did some poking at that | 16:37 |
| *** Sree has joined #openstack-infra | 16:37 | |
| *** nicolasbock has quit IRC | 16:37 | |
| clarkb | given that uncertaintly and the need to drop and restart maybe we just ride it out and complete at ~2000 ? | 16:38 |
| *** panda is now known as panda|bbl | 16:38 | |
| mordred | clarkb: yah- I think that's best here | 16:41 |
| clarkb | ok | 16:41 |
| clarkb | that also means that we won't get to secondary index reindenxing until much later in the process... which likely means a late night :/ | 16:41 |
| mordred | clarkb: remind me again why we didn't make file_id the primary key of the table since it's an auto-inc? | 16:41 |
| mordred | clarkb: I should just not ask that question | 16:42 |
| clarkb | mordred: I think becuase that other key already was the primary key? | 16:42 |
| mordred | clarkb: I think our intent was "modify this as little as possible from upstream but still work" | 16:42 |
| clarkb | ya | 16:42 |
| fungi | echo $((`sudo -H mysql -e 'select count(*) from account_patch_reviews' accountPatchReviewDb|tail -n1`*100/11566783)) | 16:43 |
| fungi | for anybody who wants a quick percent completion check | 16:43 |
| fungi | just hit 19% | 16:44 |
| fungi | if the decision is to just ride this out, i'm going grocery shopping nowish | 16:44 |
| clarkb | proposal: run as is, check in at 1930UTC. Then we either continue because it is close or have a sad and consider rollback. If we continue then do a sanity check during reindexing? | 16:45 |
| clarkb | Absolute worst case we could consider picking it up in the morning again and basically double our outage time :/ | 16:45 |
| fungi | sanity check of the accountPatchReviewDb.account_patch_reviews table you mean? | 16:45 |
| clarkb | fungi: no, I mean sanity check the runtime of reindexing once we get it going | 16:46 |
| fungi | ahh | 16:46 |
| clarkb | to give us an indication of when it will end. It does give you data about completion rates. Biggest problem is going to be large projects like nova because you can only single thread reindex those | 16:46 |
| clarkb | (so basically once we get to the last project (like nova) I think we can roughly guestimate completion time) | 16:47 |
| fungi | yeah, we've done plenty in recent history and know what sort of runtime we're looking at there | 16:47 |
| clarkb | jeblair: mordred ^ does that sound reasonable? | 16:47 |
| fungi | hopefully with 2.13 it's relatively consistent (if not faster) | 16:47 |
| mordred | clarkb: yes. I agree with the above | 16:48 |
| fungi | i think zaro also got a patch in a while back to optimize reindexing by sorting to reindex the projects with the most changes first | 16:48 |
| *** camunoz has quit IRC | 16:48 | |
| mordred | at this rate I do expect this db copy to take an additional 4 hours | 16:48 |
| jeblair | oh i hope that made it in :) | 16:48 |
| fungi | so that it ends up with less time at the end where threads are sitting around idle | 16:49 |
| clarkb | ok lets keep an eye on it but not worry about a status check until 1930UTC that should be long enough to do a grocery run :) | 16:49 |
| mordred | as it seems to be doing roughly 3 minutes per percent complete, and we're at 20% | 16:49 |
| jeblair | clarkb: sounds good | 16:49 |
| fungi | mordred: that jives with my earlier estimates too, yes | 16:49 |
| fungi | so seems to be steady at that rate at least | 16:49 |
| fungi | 21% | 16:49 |
| mordred | ++ | 16:49 |
| fungi | headed to the store, bbs | 16:50 |
| *** jpena is now known as jpena|away | 16:50 | |
| mordred | it's honestly going to be a fairly constant number based on disk speed since this is working on 1000 row chunks | 16:50 |
| * clarkb wonders how large shops like sony/sap/etc manage these upgrades sanely | 16:51 | |
| jeblair | disk+network | 16:51 |
| *** lewo` has joined #openstack-infra | 16:51 | |
| mordred | that said it IS going to slow down slightly as we continue, because of updating the two secondary indexes - but since those are each fairly small I don't expect those index updates to make things slower aggressively | 16:51 |
| clarkb | I'm going to put dinner in the crock pot. Will check back in after | 16:52 |
| mordred | (the initial insert should remain linear since innodb does inserts in primary key order into a b+tree) | 16:52 |
| jeblair | https://review.openstack.org/status.html | 16:52 |
| jeblair | that's % complete updates once a minute | 16:52 |
| mordred | jeblair: ++ | 16:53 |
| clarkb | cool thank you for that | 16:53 |
| jeblair | (it's fungi's one-liner in a loop) | 16:53 |
| *** lewo has quit IRC | 16:54 | |
| smcginnis | Nice jeblair. ;) | 16:56 |
| jeblair | i, erm, made a small update. | 16:57 |
| *** dave-mccowan has joined #openstack-infra | 16:58 | |
| *** claudiub has joined #openstack-infra | 16:58 | |
| smcginnis | Hah, love it! | 16:58 |
| mordred | jeblair: haha. nice | 16:58 |
| *** ralonsoh has quit IRC | 16:59 | |
| clarkb | I refreshed before reading irc :) | 17:00 |
| *** shardy has quit IRC | 17:00 | |
| zaro | Ahh nice, you guys are upgrading :) | 17:01 |
| zaro | fungi: i think you were referring to this? https://gerrit-review.googlesource.com/c/gerrit/+/78481 | 17:01 |
| jeblair | zaro: yes! and thank you so much for that. :) | 17:02 |
| *** efoley has quit IRC | 17:02 | |
| clarkb | zaro attempting to at least. thus db migration is not fast | 17:02 |
| jeblair | 25% | 17:04 |
| *** gouthamr has quit IRC | 17:04 | |
| kmalloc | I just wanted to say the "hey gerrit is down" page is nice. | 17:05 |
| kmalloc | thanks :) | 17:05 |
| jeblair | and zaro's change is in 2.13, so we should benefit from that | 17:06 |
| jeblair | kmalloc: thanks :) | 17:06 |
| kmalloc | also, hi everyone, welcome back from the PTG (to IRC, if not physically at your normal-ish location) | 17:06 |
| zaro | clarkb: yep, large companies have multi-master (ish) setup so they can do this without downtime. | 17:08 |
| *** gongysh has joined #openstack-infra | 17:09 | |
| *** Apoorva has joined #openstack-infra | 17:11 | |
| *** srobert_ has joined #openstack-infra | 17:15 | |
| mordred | kmalloc: ++ | 17:17 |
| *** srobert has quit IRC | 17:17 | |
| * mordred thinks the gerrit is down page will likely cut down on the number of people who come in to let us know gerrit is down this time | 17:17 | |
| *** Swami has quit IRC | 17:17 | |
| *** srobert_ has quit IRC | 17:17 | |
| *** srobert has joined #openstack-infra | 17:18 | |
| kmalloc | mordred: so... did you know... gerrit is down | 17:18 |
| * kmalloc ducks. | 17:18 | |
| pabelanger | http://paste.openstack.org/raw/621349/ | 17:19 |
| pabelanger | :) | 17:19 |
| tosky | but the page that gerrit is down is only visible from the homepage of review.gerrit.org; if you try to access a review, you see only "Not Found" | 17:19 |
| clarkb | zaro I thought multi master was still only a google thing | 17:22 |
| *** trown|lunch is now known as trown | 17:22 | |
| mordred | tosky: hrm. | 17:23 |
| *** bobh has quit IRC | 17:25 | |
| *** sdague has quit IRC | 17:25 | |
| zaro | clarkb: your right, not fully multi-master more like active/passive | 17:26 |
| zaro | with the help of a bunch of plugins | 17:26 |
| *** camunoz has joined #openstack-infra | 17:26 | |
| *** rhallisey has quit IRC | 17:27 | |
| mordred | zaro: like upgrade passive, failover promoting passive to active and old-active to passive, upgrade old-active? | 17:28 |
| *** rhallisey has joined #openstack-infra | 17:28 | |
| *** sdague has joined #openstack-infra | 17:29 | |
| *** dtantsur is now known as dtantsur|afk | 17:30 | |
| *** ociuhandu has quit IRC | 17:31 | |
| *** chlong has quit IRC | 17:33 | |
| *** tosky has quit IRC | 17:33 | |
| *** nicolasbock has joined #openstack-infra | 17:33 | |
| *** rbrndt has quit IRC | 17:33 | |
| *** electrofelix has quit IRC | 17:35 | |
| *** ijw has quit IRC | 17:35 | |
| *** nicolasbock has quit IRC | 17:38 | |
| *** dave-mcc_ has joined #openstack-infra | 17:39 | |
| *** dave-mccowan has quit IRC | 17:39 | |
| zaro | yes, i know that’s what erricson does for sure. | 17:44 |
| *** Sree has quit IRC | 17:45 | |
| *** gongysh has quit IRC | 17:46 | |
| *** sambetts is now known as sambetts|afk | 17:47 | |
| *** baoli_ has joined #openstack-infra | 17:47 | |
| *** chlong has joined #openstack-infra | 17:48 | |
| *** Swami has joined #openstack-infra | 17:50 | |
| *** baoli has quit IRC | 17:50 | |
| *** nicolasbock has joined #openstack-infra | 17:50 | |
| *** baoli_ has quit IRC | 17:51 | |
| *** baoli has joined #openstack-infra | 17:51 | |
| fungi | groceries achieved | 17:52 |
| fungi | jeblair: the cowsay addition to the migration status is a nice touch! | 17:52 |
| fungi | the cow says: now it's time for lunch | 17:53 |
| *** gongysh has joined #openstack-infra | 17:53 | |
| *** srobert has quit IRC | 17:53 | |
| *** jtomasek has quit IRC | 17:57 | |
| mordred | fungi, clarkb, jeblair: how's https://etherpad.openstack.org/p/kpYi29mKM9 for a followup email on v3 status? | 18:02 |
| *** liusheng has quit IRC | 18:02 | |
| *** liusheng has joined #openstack-infra | 18:03 | |
| clarkb | mordred: lgtm | 18:05 |
| clarkb | zaro: gotcha | 18:06 |
| *** ijw has joined #openstack-infra | 18:07 | |
| fungi | mordred: lgtm^2 thanks! | 18:07 |
| *** ijw has joined #openstack-infra | 18:07 | |
| clarkb | not to get ahead of ourselves, but how comfortable are people with a potential later evening to get gerrit up? I can be around but would prefer to not be the only root checking gerrit is happyness this evening | 18:07 |
| fungi | i will stick around | 18:07 |
| fungi | though later gets later earlier where i am | 18:07 |
| clarkb | ya | 18:08 |
| jeblair | clarkb: i'm on board | 18:08 |
| clarkb | ok cool, thanks everyone | 18:08 |
| jeblair | mordred: making some minor changes to etherpad | 18:08 |
| *** tosky has joined #openstack-infra | 18:09 | |
| jeblair | mordred: s/some/one/. lgtm. | 18:09 |
| *** jdandrea has joined #openstack-infra | 18:14 | |
| mordred | clarkb: I'm on board | 18:15 |
| *** tesseract has quit IRC | 18:15 | |
| mordred | jeblair: I have incorporated your change - does my version of your change look ok? | 18:15 |
| jeblair | mordred: that was fungi's, but it looks good to me. mine is "we're still" in line 20. | 18:17 |
| *** dizquierdo is now known as dizquierdo_afk | 18:17 | |
| jeblair | (previously; sentence no verb) | 18:18 |
| pabelanger | clarkb: I too can be around this evening | 18:18 |
| jdandrea | @clarkb At the PTG jaypipes suggested I reach out to you about project ownership changes on launchpad et. al - should I email w/Q's or PM here? | 18:18 |
| jeblair | jdandrea: what needs to happen? | 18:18 |
| jdandrea | jeblair Previous owner is no longer involved and has also left place of business. Now re-reviewing Project Owner's Guide to see where changes may be needed. (I'm also PTL.) | 18:20 |
| jeblair | jdandrea: which project(s)? | 18:20 |
| jdandrea | From a launchpad POV: https://launchpad.net/valet | 18:20 |
| jdandrea | May be others (e.g., IRC, etc.) | 18:20 |
| *** hasharAway is now known as hashar | 18:21 | |
| fungi | it doesn't look like the openstack admins group owns the maintainer group for that project (unsurprising for something which isn't an official openstack deliverable) | 18:22 |
| jeblair | agreed | 18:22 |
| jdandrea | fungi *nod* ... and I can't speak to the completeness of the setup, ofc ... but I'll dot i's and cross t's if need be. Whatever the appopriate steps to remediate are, I'm happy to do 'em. | 18:23 |
| fungi | jdandrea: more that we can't really do anything to help other than point you to the launchpad admins, i expect | 18:23 |
| jdandrea | Aye, not official. | 18:23 |
| jdandrea | fungi That's fine. I'll check with them. As for other areas (IRC, Wiki?, gerrit [down for maint atm] f'rinstance), not sure what to do there. | 18:24 |
| clarkb | zaro: do you know why this table is moved into its own db? | 18:24 |
| *** supertakumi86 has quit IRC | 18:24 | |
| jeblair | jdandrea: we don't really get involved until the project applies to join as an official project (the guide should have info about that). but even from a technical point of view, we don't have that access. so to remedy this, you'll need to ask one of the "administrator" members of this group: https://launchpad.net/~valet-pathfinders/+members to elevate you to an administrator. if they aren't responsive, you can ask the launchpad ... | 18:24 |
| jeblair | ... admins. | 18:24 |
| clarkb | seems like a lot of effort and upgrade pain for data that is essentially in the same format just in a different location | 18:24 |
| fungi | jdandrea: you have an irc channel not listed at https://wiki.openstack.org/wiki/IRC i guess | 18:24 |
| jdandrea | @jeblair Already on it, mhm. Yes, IRC appears unlisted (makes sense). P'raps I contact freenode folks in that case. | 18:25 |
| fungi | ahh, yeah looks like we at least have an accessbot entry for a #openstack-valet channel | 18:25 |
| *** bobh has joined #openstack-infra | 18:26 | |
| *** mixos has joined #openstack-infra | 18:26 | |
| jeblair | jdandrea: if accessbot is okay with it, the irc channel should be fine; you'll just want to add it to the openstack wiki to help others find it | 18:26 |
| *** srobert has joined #openstack-infra | 18:26 | |
| fungi | jdandrea: as for gerrit, yeah there are likely review groups you may need to be added into for those repos... again similar to launchpad it's better if you can get existing members of those review groups to add you (easier to check that once the gerrit maintenance is over, so maybe revisit tomorrow?) | 18:27 |
| jdandrea | @jeblair OK. But do I have wiki editing access, I wonder (unsure if it's tied to the project or if I just need an AuthN'ed ubuntu one login). | 18:28 |
| jdandrea | @fungi Aye, Gerrit we'll handle later. | 18:28 |
| fungi | jdandrea: the wiki doesn't have any project-specific acls so shouldn't require any adjustment | 18:28 |
| jdandrea | Appreciate the advisement. | 18:28 |
| jdandrea | *nod* ok | 18:28 |
| fungi | as long as you can log into the wiki, you can edit the same pages as everyone else (except those locked to admin-only edits) | 18:29 |
| *** bobh has quit IRC | 18:31 | |
| *** bobh has joined #openstack-infra | 18:31 | |
| jeblair | clarkb, fungi: is it the case that changing the label case in the config only works if this change is present? http://git.openstack.org/cgit/openstack-infra/zuul/commit/?id=00ae6c9b0959c9850180f1dfe684149f4d8e268e | 18:33 |
| jeblair | clarkb, fungi: i'm trying to catch up with the reverts, etc, over the weekend | 18:33 |
| mordred | jeblair: gotcha - sending now | 18:34 |
| *** jtomasek has joined #openstack-infra | 18:34 | |
| clarkb | jeblair: yes as well as tobiash's change for checking the NEED results properly | 18:34 |
| clarkb | --Verified is not a valid arg to gerrit review re the change you linked | 18:35 |
| clarkb | but --label Verified=1 is | 18:35 |
| fungi | yup, caught us by surprise | 18:36 |
| jeblair | clarkb: so, if you're gating, *both* of those changes must be running *before* the case can be changed in the config? | 18:36 |
| *** e0ne has quit IRC | 18:36 | |
| clarkb | corrwct | 18:37 |
| *** e0ne has joined #openstack-infra | 18:37 | |
| clarkb | but if not gating only the one you linked | 18:37 |
| clarkb | also the one you linked got lost in the connections changes iirc so a change got pushed in july to add it back in | 18:38 |
| jeblair | so there's no sequence that folks can do to avoid force merging a change...? | 18:38 |
| clarkb | the fix for --label can go in before anything else | 18:38 |
| clarkb | but then you need both layout.yaml updatr and tobiash change roughly together | 18:39 |
| clarkb | so ya I think no proper sequemce | 18:39 |
| tobiash | I agree | 18:39 |
| *** e0ne has quit IRC | 18:39 | |
| jeblair | 1) run with label; 2) merge case config change; 3) restart with case-sensitive change | 18:40 |
| jeblair | i guess ^ that would work | 18:40 |
| jeblair | as long as 3 happens right after 2 | 18:40 |
| tobiash | Just wanted to suggest that... | 18:40 |
| tobiash | But was too slow ;) | 18:41 |
| jeblair | so we should probably issue 2 point releases; first with (1), then wait a week and then (3).... | 18:41 |
| jeblair | that way people consuming from releases have a path forward | 18:41 |
| clarkb | jeblair: ya | 18:41 |
| jeblair | okay, i will try to write that up in an etherpad | 18:41 |
| jeblair | well, the change for (1) doesn't really get you anything if you're gating, so maybe we should just stick with one relase in a week. but still tell folks that they should merge the config change right before upgrading/restarting. | 18:51 |
| clarkb | that sould work as well | 18:52 |
| mordred | jeblair: ++ | 18:52 |
| *** rbrndt has joined #openstack-infra | 18:52 | |
| *** jtomasek has quit IRC | 18:53 | |
| clarkb | I have to say cowsay status updates has made me eel significantly better about this whole situation | 18:54 |
| jeblair | clarkb: maybe we can pipe the reindex to a file and have another cow read that out to us? | 18:54 |
| clarkb | hahaha | 18:55 |
| *** caphrim007 has joined #openstack-infra | 18:55 | |
| jeblair | (it's funny, but i'm actually serious since i'm always in favor of tools that let everyone participate in sysadminning :) | 18:56 |
| clarkb | I think the reindexing output is ncurses? I want ot say it has a spinning bar thing. Might make it interesting to pipe to cowsay | 18:56 |
| fungi | if memory serves, it's using ansi to back up the cursor and overwrite | 18:57 |
| jeblair | maybe we can split it to a file and i can write a quick parser | 18:58 |
| clarkb | jeblair: you can test on review-dev. It is a 16 minute or so reindex there (though maybe we don't want to take that server offline if we need to cross check anything?) | 18:59 |
| *** mixos has quit IRC | 19:00 | |
| jeblair | clarkb, fungi: maybe we can just "C-a H" and use the screen log? | 19:00 |
| jeblair | maybe that will work with no substantial change to the process | 19:01 |
| * jeblair lunches | 19:02 | |
| *** spzala has joined #openstack-infra | 19:02 | |
| clarkb | that is a good idea. I'll be back for 1930 check in. | 19:02 |
| fungi | yeah, that's probably fine if we don't accidentally cat any files containing passwords in that session | 19:02 |
| *** chlong has quit IRC | 19:12 | |
| *** mixos has joined #openstack-infra | 19:15 | |
| *** tiswanso has quit IRC | 19:24 | |
| *** tinwood has quit IRC | 19:24 | |
| *** bobh has quit IRC | 19:24 | |
| *** chlong has joined #openstack-infra | 19:24 | |
| *** tiswanso has joined #openstack-infra | 19:25 | |
| *** ociuhandu has joined #openstack-infra | 19:25 | |
| *** tinwood has joined #openstack-infra | 19:26 | |
| clarkb | zaro: are you around? is the reindex thread count half the cpu count? | 19:27 |
| *** tiswanso has quit IRC | 19:28 | |
| *** annegentle has quit IRC | 19:28 | |
| clarkb | looks like I ran threads = 4 on review-dev with 8 vcpu and we have 16 vcpu on review.o.o so thinking we should bump that number up to 8 | 19:28 |
| * clarkb edits etherpad and makes it 8 | 19:28 | |
| *** tiswanso has joined #openstack-infra | 19:29 | |
| clarkb | we also have twice the memory on review.o.o so that should scale up mostly cleanly | 19:30 |
| clarkb | ok it is 1930ish now | 19:30 |
| dmsimard | how are things looking ? | 19:31 |
| clarkb | cowsays says we are 76% complete on that migration | 19:31 |
| clarkb | doing rough math I think that means it should complete in about an hour | 19:31 |
| *** slaweq_ has joined #openstack-infra | 19:31 | |
| clarkb | 3 minutes per percent, 3 * 24 = 72 minutes | 19:31 |
| jeblair | clarkb: ack. will be back in 1h. :) | 19:32 |
| clarkb | ya I think lets sync back up again at 2030 as I expect we will be close to actually doing thinsg then | 19:33 |
| *** srobert has quit IRC | 19:33 | |
| prometheanfire | wom 24 | 19:33 |
| *** dave-mcc_ has quit IRC | 19:35 | |
| *** yamahata has quit IRC | 19:39 | |
| fungi | there's a strong chance the schema migration will finish at an inconvenient time for dinner with inlaws, and i may disappear for a couple hours right around that timeframe :( | 19:41 |
| fungi | but i'll be back on after to hopefully watch the exciting reindex | 19:41 |
| clarkb | thats ok ya for that reason :) | 19:42 |
| clarkb | we go from watching glue set to paint dry | 19:42 |
| fungi | spackle, primer and paint | 19:42 |
| *** ijw has quit IRC | 19:42 | |
| *** florianf has quit IRC | 19:45 | |
| *** tiswanso has quit IRC | 19:45 | |
| *** Sree has joined #openstack-infra | 19:46 | |
| *** dave-mccowan has joined #openstack-infra | 19:50 | |
| *** Sree has quit IRC | 19:50 | |
| *** spzala_ has joined #openstack-infra | 19:51 | |
| *** spzala has quit IRC | 19:52 | |
| mordred | clarkb, fungi: I bought both primer and paint this weekend | 19:57 |
| jeblair | last year i watched plaster dry; it's actually kind of awesome. changes color and sheen over the course of about a week. it's really pretty. | 19:57 |
| fungi | i'm still trying to use drywall mud and wet sanding to smooth out the irregularities in our master bathroom following a very frustrating wallpaper removal episode | 19:59 |
| fungi | primer and paint are waiting patiently for me | 19:59 |
| mordred | we've got some cabinets we're going to repaint and have selected a couple of doors to use to test the options of "just paint", "prime and paint", "sand, prime and paint" and "sand, prime, paint, sand" and "sand, prime, paint, sand and polycrylic finish" | 20:00 |
| mordred | obviously all on a scale of "is this option better enough to warrant the effort" | 20:00 |
| fungi | spam, spam, spam, eggs, beans and spam | 20:00 |
| *** e0ne has joined #openstack-infra | 20:00 | |
| sdake | is cdent on pto - or does he have a new nickname now? :) | 20:00 |
| *** ccamacho has quit IRC | 20:01 | |
| sdake | fungi - for some reason i think chris's nick was not cdent, could your verify? | 20:02 |
| *** hashar has quit IRC | 20:03 | |
| *** tiswanso has joined #openstack-infra | 20:03 | |
| fungi | sdake: according to my client log, he seems to have disconnected around utc midnight friday and not rejoined | 20:03 |
| *** tiswanso has quit IRC | 20:03 | |
| fungi | if his client used the same ident but a different nick, i would see that in my logs too, but he doesn't appear to have done | 20:03 |
| *** tiswanso has joined #openstack-infra | 20:04 | |
| fungi | odds are he's still travelling | 20:04 |
| fungi | or taking a few days off | 20:04 |
| sdake | fungi thanks man | 20:04 |
| fungi | you're welcome | 20:05 |
| * fungi gets back to the excitement which is filing expense reports | 20:05 | |
| clarkb | I should do that | 20:07 |
| clarkb | byt last I checked not everything had posted yet :( | 20:07 |
| fungi | i use receipts, the old fashioned way | 20:07 |
| fungi | well, coupled with a semi-ancient usb flatbed scanner | 20:08 |
| jeblair | mordred: you may want to add "trisodium phosphate" to the beginning of that list (ie, "tsp, prime, paint"). sanding should obviate the need for tsp though. | 20:09 |
| *** yamahata has joined #openstack-infra | 20:12 | |
| mordred | jeblair: indeed! | 20:13 |
| fungi | or a nice long soak in aqua regia | 20:14 |
| fungi | after which you shouldn't have anything at all which needs painting | 20:14 |
| *** esberglu has quit IRC | 20:14 | |
| *** annegentle has joined #openstack-infra | 20:14 | |
| *** ijw has joined #openstack-infra | 20:15 | |
| mordred | jeblair: the question there would be which thing, sanding or tsp, has better impact when divided by effort (with tsp we might need to mask/protect more things, but then we'd be sponging and not actually sanding which could certainly be a win) | 20:15 |
| mordred | jeblair: so - yah - good thing to test | 20:15 |
| mordred | since we're doing kitchen cabinets with fluting, sanding will be extra suck | 20:17 |
| jeblair | https://etherpad.openstack.org/p/ppQOrvMAz8 | 20:21 |
| jeblair | how's that look for the case sensitive zuul email? | 20:21 |
| * clarkb tastes chili then will look | 20:22 | |
| *** armax has quit IRC | 20:23 | |
| clarkb | jeblair: I think that is a very clear description of the steps to take +1 | 20:25 |
| mordred | jeblair: I like it - I think it could use a note that the driver for this is that in newer gerrit labels are case sensitive | 20:28 |
| mordred | jeblair: (we're not just doing it for fun) | 20:28 |
| *** cody-somerville has joined #openstack-infra | 20:29 | |
| *** cody-somerville has quit IRC | 20:29 | |
| *** cody-somerville has joined #openstack-infra | 20:29 | |
| clarkb | 2030 is here and sure enough 127 appears to be near done | 20:29 |
| fungi | 97% complete and i have to disappear for dinner :/ | 20:29 |
| clarkb | fungi: enjoy | 20:29 |
| *** esberglu has joined #openstack-infra | 20:29 | |
| fungi | i'll be back on as soon as possible. sorry all! | 20:30 |
| *** esberglu has quit IRC | 20:30 | |
| *** esberglu has joined #openstack-infra | 20:30 | |
| clarkb | I'll ping when it is complete then track the remainder of the db migration | 20:30 |
| jeblair | mordred: to the bsest of my knowledge, it's actually that certain database configurations make it case sensitive; it's not necessarily driven by gerrit upgrades itself... | 20:30 |
| *** srobert has joined #openstack-infra | 20:30 | |
| clarkb | jeblair: ya thinking about that more we've used the binary colation in mysql for a long time so notsure if that is the case (I thought it may have been a switch) | 20:31 |
| jeblair | clarkb: oh, i thought we tested it both ways | 20:31 |
| jeblair | so perhaps the shared understanding is more like "dude a bunch of stuff is changing and it's way too complicated for us to untangle". | 20:32 |
| clarkb | ++ | 20:32 |
| clarkb | mordred: when the migration is over can you do the "make sure accountPatchReviewDb" has data in mysql and not h2 step (though maybe thats trivial since that is where cowsay gets its data from | 20:33 |
| clarkb | ya I think that is redundant iwth cowsay doing maths | 20:33 |
| jeblair | clarkb, mordred: how's the new sentence in there? | 20:34 |
| clarkb | wfm | 20:34 |
| jeblair | i will strike the second unfortunately so things aren't double unfortunate. | 20:35 |
| clarkb | 99% | 20:36 |
| clarkb | its done | 20:38 |
| pabelanger | yay | 20:38 |
| smcginnis | But the cow still says 99%. I need closure! | 20:39 |
| jeblair | smcginnis: refresh? | 20:39 |
| smcginnis | Oh, whew. | 20:39 |
| smcginnis | :) | 20:39 |
| clarkb | if someone can just double check accountPatchReviewDb has contnet, I am moving forward with the jar installations | 20:40 |
| jeblair | clarkb: on it | 20:40 |
| jeblair | mordred: can i 'drop table testmt' from accountPatchReviewDb ? | 20:41 |
| *** annegentle has quit IRC | 20:41 | |
| mordred | jeblair: yes | 20:41 |
| jeblair | clarkb: i agree with the cow; the table has data, and some rows i spot checked look sane. | 20:42 |
| jeblair | mordred: done | 20:43 |
| clarkb | jeblair: thanks | 20:43 |
| clarkb | I have sorted the jar situation too | 20:43 |
| clarkb | backing up the 2.11 secondary index now | 20:44 |
| jeblair | pabelanger: are we really building nodes? (nodepool graph looks like it) | 20:46 |
| clarkb | ok infra-root, ready for reindexing? | 20:46 |
| jeblair | i have started a screen log | 20:46 |
| clarkb | I bumped the threads from 4 on review-dev to 8 on review.o.o as there are twice as many vcpu on review.o.o | 20:46 |
| pabelanger | jeblair: ya, looks like nodepool is trying, but failing to connect to gearman | 20:47 |
| clarkb | here goes | 20:47 |
| jeblair | pabelanger: oh okay. thanks. | 20:47 |
| jeblair | clarkb: ++ | 20:47 |
| clarkb | jeblair: there is a percentage number there so just have to grep that out if possible | 20:49 |
| *** e0ne has quit IRC | 20:49 | |
| *** e0ne has joined #openstack-infra | 20:50 | |
| jeblair | ya, i'll start working on that now | 20:50 |
| *** e0ne has quit IRC | 20:50 | |
| *** e0ne has joined #openstack-infra | 20:51 | |
| jeblair | i thought it had a thing telling us project names too? | 20:51 |
| clarkb | jeblair: I think that is the second part of the reindex | 20:51 |
| *** e0ne has quit IRC | 20:51 | |
| clarkb | I want to say this going as quickly as it appears to be is just a trick to make usfeel better | 20:51 |
| clarkb | but once changes are done it goes thorugh and does project by project | 20:51 |
| *** e0ne has joined #openstack-infra | 20:51 | |
| *** kgiusti has left #openstack-infra | 20:52 | |
| *** e0ne has quit IRC | 20:52 | |
| *** bobh has joined #openstack-infra | 20:52 | |
| *** e0ne has joined #openstack-infra | 20:52 | |
| *** e0ne has quit IRC | 20:52 | |
| clarkb | gerrit needs a gc command | 20:53 |
| *** e0ne has joined #openstack-infra | 20:53 | |
| *** e0ne has quit IRC | 20:53 | |
| clarkb | jeblair: ya it just added a second percentage related to projects | 20:53 |
| *** jpena|away is now known as jpena|off | 20:54 | |
| *** dprince has quit IRC | 20:56 | |
| *** dizquierdo_afk is now known as dizquierdo | 20:56 | |
| *** jcoufal has quit IRC | 20:57 | |
| *** tiswanso has quit IRC | 21:00 | |
| *** trown is now known as trown|outtypewww | 21:01 | |
| *** yamahata has quit IRC | 21:01 | |
| *** yamahata has joined #openstack-infra | 21:01 | |
| *** srobert has quit IRC | 21:02 | |
| jeblair | there are exceptions in the log | 21:02 |
| jeblair | cannot merge xxx ; missing blob yyy | 21:03 |
| clarkb | ya I just read them those are mergeability checks and appears to have tracebacks when they fail | 21:03 |
| clarkb | in the general case I am guessing that is ok? | 21:04 |
| *** tiswanso has joined #openstack-infra | 21:04 | |
| *** srobert has joined #openstack-infra | 21:04 | |
| clarkb | jeblair: are you able to pull the full traceback out of your screen log and paste it somewhere? | 21:04 |
| jeblair | clarkb: yep | 21:04 |
| clarkb | I wonder if that is just the normal way that unmergeable changes show up when reindexed | 21:05 |
| jeblair | http://paste.openstack.org/show/621357/ | 21:05 |
| clarkb | also when doing this on review-dev when there were fatal reindex problems reindexing stopped iirc | 21:05 |
| jeblair | clarkb: i think the missing blob suggests that there's an error in a git repo | 21:05 |
| clarkb | jeblair: ya | 21:06 |
| clarkb | maybe related to the nova specs problem(s)? | 21:06 |
| jeblair | clarkb: this may be related to the errors we saw in ... i can't remember the repo name.... ya that one | 21:06 |
| clarkb | ianw: ^ if you are around does that traceback look familar | 21:06 |
| jeblair | clarkb: there's at least one other traceback, but it looks similar. just different shas | 21:06 |
| clarkb | ya it has gone by a few times definitely more than just two | 21:07 |
| jeblair | cow is updated: https://review.openstack.org/status.html | 21:10 |
| clarkb | note it requires a hard refresh, or at least did for me | 21:11 |
| clarkb | also that look sawesome | 21:11 |
| jeblair | i regexed it a bit since the native output is confusing | 21:11 |
| ianw | clarkb: http://paste.openstack.org/show/621357/? | 21:11 |
| clarkb | ya native output didn't seem to rewrite the line properly when it added the projects number | 21:11 |
| clarkb | ianw: ya | 21:11 |
| ianw | oh right, like the corrupt infra-specs repo ... sorry just catching up ... looking | 21:12 |
| clarkb | tyty | 21:12 |
| clarkb | I bet those last 3 projects are nova, neutron and openstack-manuals nad they have probably been running since the start :) | 21:13 |
| clarkb | I expect we'll be sitting at 99% for a while | 21:14 |
| *** thorst has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:19 | |
| ianw | clarkb: ahh, yeah, no sorry ... that manifested only as the remote end (github) rejecting the push (actually still happens in the replication log ... i think agreed with fungi there was probably no better ideas than http://lists.openstack.org/pipermail/openstack-dev/2017-June/119166.html atm) | 21:20 |
| clarkb | ok thanks for looking. I imagine that those things are likely related. Haven't seen any recent tracebacks fly by | 21:20 |
| *** srobert_ has joined #openstack-infra | 21:21 | |
| clarkb | we appear to be indexing about a couple hunrdred changes per minute and have about 10k change sto go | 21:23 |
| clarkb | so ~50 minutes away from being done? | 21:23 |
| *** srobert has quit IRC | 21:24 | |
| clarkb | another thing to look into is if we can do the database migration and the secondary reindex at the same time | 21:25 |
| clarkb | zaro: ^ any idea if that is a possibility? Seems potentially dangerous if the secondary index has strong deps on the mysql db though | 21:25 |
| *** srobert_ has quit IRC | 21:25 | |
| dmsimard | Couple hundred per minute, 10k changes to go at 50 minutes ? Math seems wrong :) | 21:27 |
| *** Goneri has quit IRC | 21:27 | |
| *** cody-somerville has quit IRC | 21:27 | |
| clarkb | did I do that wrong? | 21:27 |
| jeblair | seems about right to me | 21:29 |
| jeblair | i just clocked 99 in 30 seconds | 21:29 |
| *** gouthamr has joined #openstack-infra | 21:30 | |
| *** tiswanso has quit IRC | 21:30 | |
| *** thorst has joined #openstack-infra | 21:33 | |
| dmsimard | Ah ok, thought it was in the upper hundreds :p | 21:35 |
| *** baoli has quit IRC | 21:37 | |
| *** thorst has quit IRC | 21:37 | |
| *** wolverineav has quit IRC | 21:38 | |
| *** jkilpatr has quit IRC | 21:39 | |
| jeblair | heh, more like almost exactly 2.0 hundreds. :) | 21:39 |
| *** aeng has joined #openstack-infra | 21:40 | |
| *** mat128 has quit IRC | 21:40 | |
| *** xyang1 has quit IRC | 21:40 | |
| *** lbragstad has joined #openstack-infra | 21:43 | |
| *** Sukhdev has joined #openstack-infra | 21:45 | |
| *** slaweq_ has quit IRC | 21:50 | |
| clarkb | I'm drinking mate now | 21:50 |
| clarkb | I think that means the upgrade is serious now | 21:50 |
| bkero | mate++ | 21:51 |
| *** dizquierdo has quit IRC | 21:51 | |
| ianw | i think that means something different in .au | 21:52 |
| clarkb | ianw: haha | 21:52 |
| *** esberglu has quit IRC | 21:52 | |
| *** tiswanso has joined #openstack-infra | 21:52 | |
| zaro | clarkb: those errors do seem familiar and benign | 21:53 |
| clarkb | https://en.wikipedia.org/wiki/Mate_(beverage) that though not really since its actually a tea and mate blend | 21:53 |
| clarkb | zaro: thanks for looking and good to know | 21:53 |
| zaro | i don’t remember if mysql and index can be done at same time. i thought the sequence was always db migration then reindex? | 21:55 |
| clarkb | zaro: ya that is how we've always done it | 21:56 |
| zaro | db migration is usually pretty quick so probably no benefit to do them in parallel anyways | 21:57 |
| *** bobh has quit IRC | 21:57 | |
| *** tiswanso has quit IRC | 21:57 | |
| *** dave-mccowan has quit IRC | 21:58 | |
| clarkb | zaro: usually except for this time it was 5 hours :/ | 21:58 |
| clarkb | ok changes is up to 99% | 21:59 |
| *** mriedem is now known as mriedem_afk | 21:59 | |
| *** andreww has joined #openstack-infra | 22:03 | |
| *** armax has joined #openstack-infra | 22:03 | |
| *** xarses_ has quit IRC | 22:03 | |
| *** rbrndt has quit IRC | 22:04 | |
| mordred | \o/ | 22:04 |
| ianw | anyone interested in claiming "openstack" for https://keybase.io/blog/introducing-keybase-teams ? | 22:04 |
| ianw | looks like it's reserved, sensibly | 22:05 |
| clarkb | fungi maybe? I also know someone that works there if we need help | 22:05 |
| *** xarses_ has joined #openstack-infra | 22:05 | |
| clarkb | mordred does too | 22:05 |
| *** gouthamr has quit IRC | 22:05 | |
| mordred | clarkb: I do? | 22:05 |
| *** dave-mccowan has joined #openstack-infra | 22:05 | |
| clarkb | thta is where chromakode is last I checked | 22:06 |
| mordred | oh! neat | 22:06 |
| mnaser | mordred while gerrit is down, im trying to look around in shade's code for the potential of this bug.. seen this mentioned in the canadian openstack users slack .. "I have found one problem with my clean keystone urls with the single host, the openstack shade library strips the /image/ path of the url, but just for the glance endpoint, everything else works" | 22:06 |
| dmsimard | someone mentioned on hackernews that there are some team names that are reserved, yes | 22:06 |
| mordred | mnaser: yah - I got a bug on that topic | 22:06 |
| mordred | mnaser: https://storyboard.openstack.org/#!/story/2001203 | 22:07 |
| mnaser | haha | 22:07 |
| mnaser | its the same person :p | 22:07 |
| *** andreww has quit IRC | 22:07 | |
| mordred | mnaser: it's actually almost certainly actually in ksa - we just recently stopped doing logic related to that in shade directly | 22:07 |
| mordred | mnaser: I havne't gotten a chance to poke and try to reproduce | 22:07 |
| mnaser | mordred i'm bored and no gerrit all day => i find out at how much i do openstack daily | 22:08 |
| mnaser | so ill try to see if i can reproduce with a test case | 22:08 |
| mordred | mnaser: sweet! that would definitely be helpful | 22:08 |
| clarkb | mnaser: sorry for the forced boredom :) | 22:08 |
| clarkb | I definitely do not find these upgrades boring myself... quite the opposite | 22:08 |
| mordred | btw - openstack/publiccloud-wg runs gate-api-wg-python27-ubuntu-xenial in check and gate and not gate-publiccloud-wg-python27-ubuntu-xenial | 22:09 |
| mordred | anybody know if that's a bug or on purpose? | 22:09 |
| mnaser | clarkb no worries, i didnt know how a) useful gerrit was to going back on pending changes and b) how much time i spend working on openstack :p | 22:09 |
| mordred | (saw it while spot-checking shared queues) | 22:09 |
| jeblair | i'm totally going to get my expense report done today, which is the most on-time i will have ever filed one. | 22:09 |
| clarkb | mordred: that seems like a bug at first thought | 22:09 |
| clarkb | due to how those jobs work | 22:09 |
| *** chlong has quit IRC | 22:09 | |
| mordred | it really does | 22:09 |
| jeblair | probably copy pasta | 22:09 |
| *** rbrndt has joined #openstack-infra | 22:10 | |
| mordred | I've added it to my stack :) | 22:10 |
| *** sdague has quit IRC | 22:10 | |
| clarkb | just 3000 changes to go | 22:12 |
| dmsimard | that's less than 9000 | 22:12 |
| * dmsimard tired | 22:12 | |
| bkero | jeblair: kudos | 22:12 |
| *** dave-mccowan has quit IRC | 22:13 | |
| jeblair | i have the live view of the reindex in a terminal window, but i *still* use the web browser because cowsay is a *significant* ui improvement! it's really easy to underestimate the importance of information design. :) | 22:15 |
| clarkb | I've noticed that I put the new db config in a different location than it is in puppet in the secure.config file. I am going to add a step to the process to move it so that that file doesn't end up being updated unnecessarily by puppet | 22:16 |
| clarkb | this is my ocd | 22:16 |
| *** mixos has quit IRC | 22:16 | |
| *** stevebak` is now known as stevebaker | 22:17 | |
| *** claudiub has quit IRC | 22:17 | |
| *** jkilpatr has joined #openstack-infra | 22:18 | |
| clarkb | its done | 22:19 |
| jeblair | i stopped the cowsayer | 22:19 |
| clarkb | ok I'm moving on to next step of installing plugins | 22:19 |
| *** Apoorva_ has joined #openstack-infra | 22:20 | |
| *** slaweq_ has joined #openstack-infra | 22:20 | |
| jeblair | (screenlog is still enabled, but that seems useful so maybe let's leave it be for now :) | 22:20 |
| clarkb | ++ | 22:22 |
| clarkb | jeblair: the step to move the new db section in the config will grab the db password in the log | 22:22 |
| clarkb | (so good ot have cowsayer off) | 22:22 |
| clarkb | jeblair: you ready for me to do ^ (eg no expected leakages?) | 22:22 |
| *** Apoorva has quit IRC | 22:23 | |
| jeblair | clarkb: yep. i chmoded the log too. | 22:23 |
| clarkb | ok | 22:23 |
| *** rlandy is now known as rlandy|brb | 22:24 | |
| clarkb | how does that look? | 22:24 |
| *** spzala_ has quit IRC | 22:24 | |
| *** spzala has joined #openstack-infra | 22:24 | |
| * clarkb is reenabling the init script now | 22:25 | |
| jeblair | clarkb: erm, i'm not up to date on this. looks generally reasonable though. :) | 22:25 |
| clarkb | jeblair: the next two steps are to undo your apache changes and to start gerrit, do you want to go ahead and clean up the apache stuff now? | 22:26 |
| jeblair | clarkb: yep. i can do that if you want | 22:26 |
| clarkb | please | 22:26 |
| jeblair | clarkb: apache done | 22:26 |
| clarkb | ok, everyone ready to start gerrit? | 22:27 |
| jeblair | infra-root: ^ | 22:27 |
| jeblair | clarkb: ++ | 22:27 |
| mordred | clarkb: ++ | 22:27 |
| clarkb | ok here we go | 22:27 |
| clarkb | [2017-09-18 22:28:06,490] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.13.9-4-g2a605d5 ready says the log | 22:28 |
| clarkb | might be a little longer before it is properly ready though | 22:28 |
| jeblair | lemme restart apache to clear any negative cache | 22:28 |
| clarkb | ok | 22:28 |
| jeblair | done and it's accessible now | 22:29 |
| *** spzala has quit IRC | 22:29 | |
| clarkb | I can get it | 22:30 |
| jeblair | my gertty is happy | 22:30 |
| clarkb | I think we may need more css fixes for the logo stuff | 22:30 |
| *** felipemonteiro_ has quit IRC | 22:30 | |
| *** Apoorva_ has quit IRC | 22:30 | |
| clarkb | but that is not urgent | 22:30 |
| jeblair | clarkb: what's wrong with the logo? | 22:30 |
| clarkb | it pushes everything to the right so we hvae horizonal scroll bar | 22:30 |
| *** Apoorva has joined #openstack-infra | 22:30 | |
| clarkb | this was problem before too, guessing the newer css just needs to be accomodated | 22:31 |
| clarkb | I'm going to log out then in | 22:31 |
| clarkb | that works | 22:31 |
| jeblair | ah, i'm anonymous, that explains the discrepancy | 22:31 |
| clarkb | someone should push a new change before we get zuulrunning | 22:32 |
| jeblair | mordred: ^ | 22:32 |
| mordred | jeblair: I can push changes | 22:32 |
| clarkb | search seems to work | 22:32 |
| *** baoli has joined #openstack-infra | 22:33 | |
| mordred | actually - new gerrit seems to have fixed my horizontal scroll issues | 22:33 |
| clarkb | weird maybe its my local browser cache? | 22:34 |
| mordred | maybe so | 22:34 |
| mordred | clarkb: you wanted me to push up a change real quick? | 22:34 |
| clarkb | yes please | 22:34 |
| clarkb | lets make sure that process works and if so I think we can start zuul up again | 22:34 |
| jeblair | heh, there's a bunch of updates right at 22:39, likely from gertty users. :) | 22:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Rework openstack doc publishing to not be overridable https://review.openstack.org/504233 | 22:34 |
| mordred | I'll keep that at one change for now :) | 22:35 |
| clarkb | I just dropped a test +1 on that change | 22:35 |
| clarkb | seemed to work | 22:35 |
| jeblair | do we want a new-from-scratch change? | 22:36 |
| *** baoli_ has joined #openstack-infra | 22:36 | |
| mordred | I can push some of those too if we want them | 22:36 |
| clarkb | sure | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add tag-releases to the mapping https://review.openstack.org/504505 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add specs publication jobs https://review.openstack.org/504506 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping entries for proposal jobs https://review.openstack.org/504608 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add publish-service-types-authority job and mapping https://review.openstack.org/504609 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add xstatic-check-version and openstack-tox-pypy https://review.openstack.org/504610 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove liberty/mitaka job regexes https://review.openstack.org/504964 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove unmatched single quotes from jenkins jobs https://review.openstack.org/504965 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping file setting to skip jobs from share queues https://review.openstack.org/504966 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Have publiccloud-wg gate on itself, not api-wg https://review.openstack.org/504967 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make yaml2ical publication job https://review.openstack.org/504968 | 22:36 |
| mordred | patches | 22:36 |
| mordred | yah - my web ui is better re: scrollbars | 22:37 |
| jeblair | gertty is not happy fetching that change | 22:37 |
| jeblair | git fetch https://git.openstack.org/openstack-infra/project-config +refs/changes/65/504965/1:refs/changes/65/504965/1 | 22:37 |
| jeblair | that command doesn't seem to work | 22:37 |
| *** baoli has quit IRC | 22:37 | |
| jeblair | oh | 22:37 |
| jeblair | git.o.o | 22:37 |
| jeblair | we're probabbly mirror lagged | 22:37 |
| clarkb | oh ya | 22:37 |
| * clarkb looks | 22:37 | |
| jeblair | git fetch https://review.openstack.org/openstack-infra/project-config +refs/changes/65/504965/1:refs/changes/65/504965/1 | 22:38 |
| jeblair | that works fine | 22:38 |
| jeblair | so no worries | 22:38 |
| clarkb | 12k tasks, will check that it falls over time | 22:38 |
| mordred | there's a bunch of replication stuff happening in th elog | 22:39 |
| clarkb | was 12897 now 12801 | 22:39 |
| jeblair | yeah, and the top tasks are changing | 22:39 |
| clarkb | small sample shows it trending the right direction | 22:39 |
| clarkb | now 12339 | 22:40 |
| clarkb | do we want that to settle to zero before starting zuul? | 22:40 |
| jeblair | should we proceed? | 22:40 |
| clarkb | I think we can proceed if you are ready, this is normal on every gerrit restart not just upgrades | 22:40 |
| jeblair | okay, i'll start zuul now | 22:41 |
| mordred | I'm fine either way | 22:41 |
| mtreinish | ooh, you can see all the replecation events on firehose | 22:42 |
| clarkb | also we'll want to restart gerrit again at some point to pikc up the commentlink fixes | 22:42 |
| clarkb | (probably soon after puppet applies it today | 22:42 |
| clarkb | oh except you know what | 22:42 |
| clarkb | gah | 22:42 |
| clarkb | git0X.o.o are not puppeting which means review.o.o won't puppet | 22:43 |
| clarkb | we may want to use kick.sh before turning puppet back on globally | 22:43 |
| jeblair | clarkb: why aren't git0x puppeting? | 22:43 |
| clarkb | infra-root please review https://review.openstack.org/#/c/504838/1 as that is the change we need to get review.o.o puppeting again | 22:43 |
| clarkb | jeblair: we think epel and centos 7.4 are out of sync and causing package install fails | 22:44 |
| clarkb | jeblair: for git0x in particular the package failing is python-paramiko | 22:44 |
| jeblair | zuul is started and changes re-enqueued | 22:44 |
| clarkb | it is also possible its been fixed upstream of us | 22:44 |
| clarkb | I think we want to use kick.sh on review.o.o either way just to see it all happen in the foreground | 22:44 |
| jeblair | clarkb: if we're happy, i think we can clear the alert now | 22:46 |
| jeblair | clarkb: i +2d 838 | 22:46 |
| clarkb | can we get that change applied to gerrit first? | 22:46 |
| *** dave-mccowan has joined #openstack-infra | 22:46 | |
| clarkb | since I think we want to restart gerrit again afterwards | 22:46 |
| jeblair | oh, yeah, if you want to do that then yes | 22:46 |
| clarkb | mordred: pabelanger do you want to look at https://review.openstack.org/#/c/504838/1 before I approve it? | 22:46 |
| mordred | clarkb: +2 | 22:47 |
| clarkb | jeblair: ya it will fix some small problems with commentlinks to do so | 22:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_upper_constraints_file to tox_constraints_file https://review.openstack.org/502348 | 22:47 |
| clarkb | ok puppet change is approved | 22:47 |
| clarkb | 9482 gerrit tasks in queue so definitely falling | 22:48 |
| *** baoli_ has quit IRC | 22:49 | |
| *** baoli has joined #openstack-infra | 22:49 | |
| *** inc0 has quit IRC | 22:53 | |
| clarkb | still waiting on a trusty node to get 838 merged | 22:53 |
| *** inc0 has joined #openstack-infra | 22:54 | |
| *** spzala has joined #openstack-infra | 22:55 | |
| *** spzala has quit IRC | 22:55 | |
| clarkb | mordred: is there an easy way to run kick.sh puppet as noop first before running properly? | 22:59 |
| clarkb | mordred: thinking that will be a good sanity check | 22:59 |
| mordred | clarkb: uhm. ... | 22:59 |
| mordred | clarkb: I dunno? I'd have to go read code | 23:00 |
| * mordred reads | 23:00 | |
| *** aeng has quit IRC | 23:00 | |
| mordred | clarkb: I believe if you add -epuppet_noop=true to the ansible-playbook invocation it will pass noop to puppet | 23:01 |
| mordred | but we don't expose that in kick ... so you'd just need to run ansible-playbook I think | 23:01 |
| fungi | okay, back from dinner and catching up | 23:01 |
| *** nicolasbock has quit IRC | 23:02 | |
| clarkb | mordred: ya I can edit kick.sh or just run the command it runs directly | 23:02 |
| clarkb | thanks for looking | 23:02 |
| mordred | clarkb: ansible-playbook -f1 --limit review.openstack.org:localhost /opt/system-config/production/playbooks/remote_puppet_adhoc.yaml -epuppet_noop=true | 23:03 |
| mordred | clarkb: COMPLETELY UNTESTED | 23:03 |
| *** baoli has quit IRC | 23:03 | |
| fungi | wow, reindex only took an hour-ish?!? | 23:04 |
| clarkb | fungi: 90 minutes | 23:04 |
| clarkb | fungi: we appear to be mostly in business, just need to get puppet change merged, applied, and gerrit restarted for picking up commentlink changes and sanity check that it can be restarted | 23:04 |
| *** xarses_ has quit IRC | 23:05 | |
| clarkb | merge should happen shortly | 23:05 |
| openstackgerrit | Merged openstack-infra/system-config master: Upgrade review.openstack.org to Gerrit 2.13 https://review.openstack.org/504838 | 23:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make yaml2ical publication job https://review.openstack.org/504968 | 23:05 |
| clarkb | mordred: maybe you can test that against review-dev really quickly? Also please do not run puppet against review.o.o manually untilyou have confirmed that ^ is on disk on the puppetmaster | 23:05 |
| * clarkb is going to work on getting the git repo on puppetmaster updated now | 23:06 | |
| mordred | clarkb: testing against review-dev | 23:06 |
| clarkb | ok puppetmaster uses git.o.o so have to wait for replication to catch up (or use gerrit as remote, I will just wait as gerrit seems to be quickly catching up on replication requests) | 23:07 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Move github webhook from webapp to zuul-web https://review.openstack.org/504267 | 23:09 |
| jeblair | clarkb: git pull --ff-only https://review.openstack.org/openstack-infra/system-config ? | 23:09 |
| clarkb | jeblair: does that work? | 23:09 |
| jeblair | ya | 23:09 |
| clarkb | I didn't realize you could use raw url on a pull, huh | 23:09 |
| * clarkb runs that | 23:09 | |
| clarkb | that does indeed work, neat. TIL | 23:10 |
| *** cody-somerville has joined #openstack-infra | 23:10 | |
| *** cody-somerville has joined #openstack-infra | 23:10 | |
| mordred | clarkb: I'm not sure if noop did anything or not | 23:10 |
| caphrim007 | is it possible to tell shade to ignore its envvars cloud even if you have envvars? | 23:10 |
| *** threestrands has joined #openstack-infra | 23:10 | |
| *** threestrands has quit IRC | 23:10 | |
| *** threestrands has joined #openstack-infra | 23:10 | |
| mtreinish | hah, new gerrit allows you to remove votes... | 23:10 |
| clarkb | mordred: I think syslog on the node should tell you? | 23:10 |
| caphrim007 | for instance, because i use a clouds.yaml for ansible, but env vars for the openstack command itself | 23:10 |
| mordred | caphrim007: the openstack command supports clouds.yaml fwiw | 23:10 |
| clarkb | mtreinish: ya that was a regression in new change screen iirc | 23:10 |
| clarkb | mtreinish: should only let you remove your own votes? | 23:11 |
| caphrim007 | mordred: that might be the answer then | 23:11 |
| mordred | caphrim007: one sec though - checking athing for the gerrit upgade | 23:11 |
| *** csomerville has joined #openstack-infra | 23:11 | |
| mtreinish | clarkb: umm, I just removed a powerkvm ci vote on: https://review.openstack.org/#/c/490903/ | 23:11 |
| mordred | caphrim007: if you set OS_CLOUD you can avoid passing --os-cloud to openstack and it won't create an envvars cloud | 23:11 |
| jeblair | that's a pretty bad regression | 23:11 |
| * mtreinish will try to remove sdague's +2 now | 23:11 | |
| caphrim007 | mordred: thanks! | 23:12 |
| mtreinish | hah, that worked too | 23:12 |
| clarkb | in old change screen you could remove your own votes | 23:12 |
| *** cody-somerville has quit IRC | 23:12 | |
| clarkb | removing any votes seemed to go away in new change screen, I guess they added it back poorly? | 23:12 |
| clarkb | mtreinish: can you try removing a vote on a project you aren't core in? use an infra one say zuul if you want people to be around to add it back | 23:12 |
| * mtreinish tries a project without core | 23:12 | |
| clarkb | it does at least record that you did it | 23:12 |
| mordred | mtreinish: can you do thatanywhere? or only projects in which you have core? | 23:12 |
| mtreinish | clarkb: yeah that was my next test | 23:12 |
| * mordred has admin, so isn't sure what extra powers that might entail | 23:13 | |
| jeblair | mordred: admin has always had remove any vote | 23:13 |
| mtreinish | I don't have an x to remove on neutron patches | 23:13 |
| mtreinish | let me try to find a project with a patch I own without core | 23:13 |
| mtreinish | I think it'll be on any patch I am the owner for | 23:14 |
| mtreinish | I have the little x's on: https://review.openstack.org/#/c/442125/ | 23:14 |
| clarkb | mordred: reading syslog on review-dev I think it may have properly puppet applied | 23:14 |
| clarkb | which isn't the end of the world this has been tested on review-dev before review.o.o I just am paranoid | 23:15 |
| mordred | clarkb: properly as in without noop? | 23:15 |
| clarkb | mordred: ya | 23:15 |
| mordred | clarkb: ok. cool | 23:15 |
| mtreinish | yeah, I was able to remove andreaf's +1 there. I don't have d-g core | 23:15 |
| *** ijw has quit IRC | 23:15 | |
| mordred | mtreinish, clarkb, jeblair: there is a permission "remove reviewer" | 23:15 |
| mordred | perhaps we need to add a setting for that which restricts who can do it? | 23:16 |
| clarkb | mordred: ah ok, so probably we just need to update all-projects and let only admins do it | 23:16 |
| clarkb | (and maybe let cores do it on their own projects later | 23:16 |
| mtreinish | mordred: yeah, I think that'll be a good idea | 23:16 |
| mordred | well - I'm not sure it's the right permission | 23:16 |
| *** rbrndt has quit IRC | 23:16 | |
| mordred | mtreinish: lemme try setting it on a thing where you're core and see if I can remove your x's | 23:16 |
| mtreinish | mordred: ok | 23:16 |
| clarkb | oh remove reviewer is probably for the table above votes | 23:17 |
| mordred | yah - I'm guessing that's what it is | 23:17 |
| clarkb | but maybe it applies to both locations? in any case that is annoying | 23:17 |
| jeblair | https://review.openstack.org/Documentation/access-control.html#category_remove_reviewer | 23:17 |
| jeblair | "Change owners can always remove reviewers who have given a zero or positive score (even without having the Remove Reviewer access right assigned)." | 23:17 |
| mtreinish | clarkb: I'm not sure there are any situations where remove vote is the right thing for even a proejct core | 23:17 |
| *** aeng has joined #openstack-infra | 23:17 | |
| clarkb | mtreinish: removing procedural -2s | 23:17 |
| mtreinish | oh, yeah I guess that's true | 23:18 |
| jeblair | it looks like that means that even that permission won't change this new behavior | 23:18 |
| mtreinish | there have been times where someone is stuck because the original -2 is away | 23:18 |
| mordred | mtreinish: ok - I just set remove reviewer to exclusively administrators on tempest - anyhting change? | 23:18 |
| clarkb | my initial thought about it is we can live with it because there is a paper trail | 23:18 |
| clarkb | if it comes down to it we can track down offenders | 23:18 |
| jeblair | mordred: ^ (pls continue test, but see above from me) | 23:18 |
| clarkb | mtreinish: are you able to remove negative votes? | 23:18 |
| mtreinish | mordred: I don't have the little x anymore on: https://review.openstack.org/#/c/504728/ | 23:18 |
| mordred | jeblair: ah - yes - I see | 23:18 |
| mtreinish | clarkb: it doesn't seem that I can remove any -1s on: https://review.openstack.org/#/c/482759/ | 23:19 |
| clarkb | mtreinish: but can remove the +1s? | 23:19 |
| mordred | ok. cool. so removing negative votes at least isn't easily available | 23:19 |
| mtreinish | clarkb: yes, the +1s have little x's | 23:19 |
| mordred | clarkb: yah - because he's the change owner | 23:20 |
| *** tosky has quit IRC | 23:20 | |
| mordred | and they are positive or zero as per jeblair's text above | 23:20 |
| clarkb | ok makes sense, I think we can live with that even if it is annoying | 23:20 |
| jeblair | mordred, mtreinish: did mordred's change remove the ability to remove -1 ? | 23:20 |
| mordred | mtreinish: can you remove -1'd on tempest? | 23:20 |
| mordred | on changes you own? | 23:20 |
| jeblair | (i wasn't paying attention to -1 vs +1 earlier) | 23:20 |
| mordred | me either | 23:20 |
| mtreinish | mordred: I don't have the x to remove the votes on: https://review.openstack.org/#/c/457369/ (which is a tempest change) | 23:20 |
| mtreinish | jeblair: all the removals before were on +1 and +2s | 23:20 |
| mordred | ok. | 23:20 |
| fungi | it's possible we live with owners removing positive votes. while it seems suboptimal i can't immediately think of places where that actually gains anyone anything, even mischief-wise | 23:21 |
| mtreinish | jeblair: I don't seem to have the ability to remove a -1 on patches I own, or patches on proejcts I'm core on | 23:21 |
| mordred | mtreinish: what aobut a +1 on a tempest patch you own? | 23:22 |
| mordred | mtreinish: I just left one on that patch https://review.openstack.org/#/c/457369/ | 23:22 |
| clarkb | if we are satisfied with the vote removal behavior I think next step is running puppet and restarting gerrit one last time. There are less than 1k tasks in the gerrit queue now (probably a good idea to let those finish before puppeting and restarting too) | 23:22 |
| fungi | sounds reasonable | 23:22 |
| mtreinish | mordred: i just removed your +1 | 23:23 |
| jeblair | yep | 23:23 |
| mordred | mtreinish: htanks | 23:23 |
| jeblair | mordred: want to undo your change and verify no behavior change with mtreinish? | 23:23 |
| clarkb | hrm I guess one possible concern is that we'd start getting errors for new project replication if the git backends aren't up to date? | 23:23 |
| mordred | jeblair: done | 23:23 |
| mordred | mtreinish: I have removed the permissions setting from tempest | 23:24 |
| clarkb | 2017-09-18 12:32:23,123 p=12458 u=root | git01.openstack.org : ok=7 changed=2 unreachable=0 failed=1 means that could be the case I think | 23:24 |
| pabelanger | sorry, got pulled away for a bit. back now if needed | 23:24 |
| mordred | mtreinish: I expect you to be able to remove +1s and not be able to remove -1s | 23:24 |
| fungi | clarkb: note that restarting gerrit again will start the mirror sync back over too... not necessarily a problem, just something to keep in mind | 23:24 |
| clarkb | fungi: ya | 23:24 |
| mordred | mtreinish: (and I +1'd that patch again just to be sure) | 23:24 |
| mtreinish | mordred: I can still remove your +1 (and did) | 23:24 |
| mordred | ok. so you can still do that | 23:25 |
| *** ijw has joined #openstack-infra | 23:25 | |
| mtreinish | although the operation seems to be slow (but that's probably just gerrit being sluggish because of all the replications) | 23:25 |
| mordred | and you still can't remove masayukig's vote? | 23:25 |
| mtreinish | mordred: nope, there is no littl x next to masayukig's -1 there | 23:25 |
| mordred | cool | 23:25 |
| jeblair | 8cc59d46 waiting .... 23:24:07.035 (retry 1) [6aa59802] push git@github.com:openstack/nova-specs.git | 23:25 |
| mordred | ok - we do not need a global permissions change | 23:25 |
| jeblair | mordred: thanks | 23:26 |
| fungi | at least we know nova-specs is in need of repair still | 23:26 |
| clarkb | http://git.openstack.org/cgit/openstack/networking-lagopus is the only one we'd need to get in place on git backends that I can see | 23:26 |
| clarkb | I'm going to quickly see if I can somewhat forcefully do that | 23:27 |
| *** erlon has quit IRC | 23:27 | |
| clarkb | I think I can update /home/cgit/projects.yaml then run create-cgitrepos command | 23:28 |
| jeblair | clarkb: i vote for manually updating the commentlinks and leave puppet/restarting for when git.o.o is fixed. | 23:28 |
| clarkb | jeblair: ok that also works for me | 23:28 |
| fungi | sure, no objections | 23:29 |
| jeblair | (i'm not opposed to the other; i just mostly don't want to have to fix the world before finishing this maint) | 23:29 |
| clarkb | jeblair: ++ | 23:29 |
| clarkb | fungi: is review.o.o in the emergency file? | 23:29 |
| fungi | nope | 23:29 |
| *** hichihara has joined #openstack-infra | 23:29 | |
| clarkb | do we want to leave puppet off globally or add review.o.o to the emergency file and enable puppet again? | 23:29 |
| fungi | i only disabled puppet globally and then the gerrit initscript | 23:29 |
| jeblair | suggest adding review/git to emergency | 23:30 |
| fungi | yeah, sounds safe | 23:30 |
| jeblair | or i guess just review should be fine? | 23:30 |
| clarkb | jeblair: just review should be fine | 23:30 |
| clarkb | if someone can get that done I will work on the commentlinks now | 23:30 |
| jeblair | i'll do emergency | 23:30 |
| *** mixos has joined #openstack-infra | 23:31 | |
| jeblair | pabelanger: zuulv3-dev.o.o is dead, right? | 23:31 |
| pabelanger | jeblair: yes | 23:31 |
| jeblair | it still has dns records, we should clean those up | 23:31 |
| pabelanger | agree, I can take care of that | 23:32 |
| fungi | oh, right, just remembered it was pointed out earlier i left behind an aaaa for apps.o.o | 23:32 |
| fungi | i'll do that and zuulv3-dev.o.o record cleanup now | 23:32 |
| ianw | jeblair: if we want we could try the update-ref stuff from http://lists.openstack.org/pipermail/openstack-dev/2017-June/119166.html on nova-specs to see if we can get rid of this. although it's one more thing, it might not be bad to make sure it clears up while people are watching | 23:32 |
| clarkb | ok comment links config is updated on disk | 23:33 |
| * mordred needs to step away for a sec - back in a couple | 23:34 | |
| clarkb | jeblair: fungi let me know when you are ready for gerrit to be restarted again? | 23:34 |
| *** hongbin has quit IRC | 23:34 | |
| clarkb | er s/\?/\./ | 23:34 |
| pabelanger | zuulv3-dev.o.o DNS removed | 23:34 |
| jeblair | #status log review.openstack.org added to emergency file until git.o.o puppet is fixed and we can supervise a puppet run on review.o.o | 23:34 |
| openstackstatus | jeblair: finished logging | 23:34 |
| Sukhdev | clarkb : can I have you relook at this please - https://review.openstack.org/#/c/503829/ | 23:34 |
| jeblair | clarkb: ready | 23:34 |
| clarkb | ok going to stop, check, start now | 23:35 |
| fungi | fire at will | 23:35 |
| *** yamamoto has joined #openstack-infra | 23:36 | |
| clarkb | should be up again | 23:37 |
| fungi | pabelanger: you beat me to the zuulv3-dev dns cleanup, but i went ahead and removed the stray apps.o.o aaaa | 23:37 |
| clarkb | apache may need to learn to agree which should happen shortly | 23:37 |
| pabelanger | review.o.o online for me | 23:37 |
| clarkb | change link links seem to work now | 23:38 |
| *** s-shiono has joined #openstack-infra | 23:38 | |
| jeblair | managed to restart without bumping any gating changes out either :) | 23:38 |
| clarkb | I've updated the etherpad with what needs to be done once puppet is happy again | 23:39 |
| jeblair | status ok now? :) | 23:39 |
| clarkb | I think at this point we are done | 23:39 |
| clarkb | ya | 23:39 |
| jeblair | clarkb: you want to issue the command? | 23:40 |
| *** aeng has quit IRC | 23:40 | |
| *** Sukhdev has quit IRC | 23:40 | |
| clarkb | sure | 23:40 |
| clarkb | its just #status ok right? | 23:40 |
| jeblair | clarkb: say something afterwords | 23:41 |
| jeblair | like "#status ok gerrit is happy, maintenance completed on time, recheck if needed, etc." | 23:41 |
| clarkb | "review.openstack.org Gerrit 2.13 upgrade is functionally complete. The Infra team will be cleaning up bookkeeping items over the next couple days. If you have any questions please let us know" | 23:41 |
| clarkb | how is ^ | 23:41 |
| jeblair | wfm | 23:42 |
| clarkb | #status ok review.openstack.org Gerrit 2.13 upgrade is functionally complete. The Infra team will be cleaning up bookkeeping items over the next couple days. If you have any questions please let us know | 23:42 |
| openstackstatus | clarkb: sending ok | 23:42 |
| *** david-lyle has quit IRC | 23:43 | |
| clarkb | also super thank you to everyone that helped. I was really worried we'd all be running on caffeine late tonight. We got it done in the allocated time in the end. | 23:43 |
| clarkb | I will make sorting out git backends my priority for tomorrow | 23:44 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 23:44 | |
| -openstackstatus- NOTICE: review.openstack.org Gerrit 2.13 upgrade is functionally complete. The Infra team will be cleaning up bookkeeping items over the next couple days. If you have any questions please let us know | 23:45 | |
| clarkb | jeblair: I didn't notice the git backend thing until this last weekend :/ didn't think at the time there would be any conflict with the gerrit upgrade | 23:45 |
| fungi | i'm shocked this completed on time | 23:45 |
| fungi | excellent planning! | 23:46 |
| clarkb | I'll send a follow up to the mailing list thread shortly as well | 23:46 |
| jeblair | fungi: i think zaro saved us :) | 23:46 |
| clarkb | ya reindexing was much quicker than expected | 23:46 |
| clarkb | as for turning on puppet for everything else do we want to do that now or wait for morning? | 23:47 |
| clarkb | (I feel like I've got enough disctractions for finishing up today that that can wait for tomorrow) | 23:47 |
| jeblair | i'd like to afk for now; if other infra-roots will be around and want to babysit that, that's cool. otherwis, i think it can wait until tomorrow. | 23:48 |
| clarkb | lets do it tomorrow then. We managed to finish at a reasonable time lets not find excuses to work late anyways :) | 23:48 |
| *** jmorgan1 has left #openstack-infra | 23:49 | |
| clarkb | jeblair: does statusbot notify when the ok is finished sending? | 23:50 |
| clarkb | I want to put a status log item in about not approving new project changes until git backends are working and I think overlapping notices hsa caused problems in the psat | 23:51 |
| jeblair | clarkb: it should i think... | 23:51 |
| jeblair | apparently not? | 23:52 |
| jeblair | it looks like it's done | 23:52 |
| clarkb | ok safe to send #status log then? | 23:52 |
| pabelanger | puppet for morning works for me | 23:52 |
| jeblair | clarkb: gimme a sec | 23:52 |
| fungi | yup, better the few people who stick around focus on keeping an eye out for reports of broken gerrit behaviors instead | 23:52 |
| pabelanger | can get started earily if people would like | 23:52 |
| *** Sukhdev has joined #openstack-infra | 23:52 | |
| *** aeng has joined #openstack-infra | 23:53 | |
| *** tiswanso has joined #openstack-infra | 23:53 | |
| *** mriedem_afk is now known as mriedem | 23:55 | |
| fungi | gertty seems to be working well for me | 23:55 |
| *** mixos has quit IRC | 23:58 | |
| clarkb | ok mail sent to the dev list | 23:58 |
| *** tiswanso has quit IRC | 23:58 | |
| jeblair | clarkb: as best as i can tell, statusbot is somehow stuck or broken at the twitter stage. i do not know if anything further will work. perhaps the folks who merged the twitter change can help? | 23:58 |
| fungi | clarkb: thanks! i was about to ask if you needed someone to send that | 23:58 |
| clarkb | jeblair: fun :/ | 23:58 |
| * clarkb will just make a note here | 23:58 | |
| jeblair | well, i think *something* needs to be done. | 23:59 |
| clarkb | project-config reviewers infra-root and AJaeger please avoid merging new project creation changes until after we have the git backends puppeting properly | 23:59 |
| fungi | i wonder if twitter integration can be replaced with an rss/atom feed | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
