| *** aeng has quit IRC | 00:03 | |
| *** thorst has joined #openstack-infra | 00:20 | |
| *** aeng has joined #openstack-infra | 00:22 | |
| *** esberglu has joined #openstack-infra | 00:33 | |
| *** aeng has quit IRC | 00:34 | |
| *** aeng has joined #openstack-infra | 00:34 | |
| *** esberglu has quit IRC | 00:37 | |
| *** Apoorva has joined #openstack-infra | 00:41 | |
| *** markvoelker has joined #openstack-infra | 00:42 | |
| *** LindaWang has joined #openstack-infra | 00:45 | |
| *** gildub_ has joined #openstack-infra | 00:50 | |
| *** zhurong has joined #openstack-infra | 01:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test undercloud containers https://review.openstack.org/494151 | 01:07 |
|---|---|---|
| *** shu-mutou-AWAY is now known as shu-mutou | 01:09 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: send graphite stats from ARA https://review.openstack.org/480121 | 01:10 |
| *** ykarel_ has joined #openstack-infra | 01:14 | |
| *** liusheng has quit IRC | 01:14 | |
| *** cshastri has joined #openstack-infra | 01:15 | |
| *** markvoelker has quit IRC | 01:15 | |
| *** Apoorva has quit IRC | 01:15 | |
| *** cuongnv has joined #openstack-infra | 01:19 | |
| *** ykarel_ has quit IRC | 01:20 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Compress logs to tar.gz archive https://review.openstack.org/434143 | 01:22 |
| *** liusheng has joined #openstack-infra | 01:22 | |
| *** thorst has quit IRC | 01:22 | |
| *** thorst has joined #openstack-infra | 01:23 | |
| openstackgerrit | Daniel Speichert proposed openstack-infra/shade master: Support filtering servers in list_servers using arbitrary parameters https://review.openstack.org/506969 | 01:26 |
| *** sree has joined #openstack-infra | 01:27 | |
| *** sree has quit IRC | 01:31 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set repo setup release in playbook https://review.openstack.org/504939 | 01:32 |
| *** liujiong has joined #openstack-infra | 01:33 | |
| *** thorst has quit IRC | 01:35 | |
| *** yamamoto has joined #openstack-infra | 01:54 | |
| *** hongbin has joined #openstack-infra | 02:00 | |
| *** slaweq has joined #openstack-infra | 02:13 | |
| *** slaweq has quit IRC | 02:18 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set repo setup release in playbook https://review.openstack.org/504939 | 02:18 |
| *** esberglu has joined #openstack-infra | 02:21 | |
| *** esberglu has quit IRC | 02:25 | |
| *** thorst has joined #openstack-infra | 02:36 | |
| *** david-lyle has joined #openstack-infra | 02:38 | |
| *** namnh has joined #openstack-infra | 02:41 | |
| *** gildub_ has quit IRC | 02:51 | |
| *** slaweq has joined #openstack-infra | 02:55 | |
| *** slaweq has quit IRC | 02:59 | |
| *** baoli has quit IRC | 03:03 | |
| *** baoli has joined #openstack-infra | 03:04 | |
| *** kuromagi has quit IRC | 03:07 | |
| *** bobh has quit IRC | 03:08 | |
| *** zhurong has quit IRC | 03:08 | |
| *** zhurong has joined #openstack-infra | 03:10 | |
| *** kuromagi has joined #openstack-infra | 03:12 | |
| *** markvoelker has joined #openstack-infra | 03:12 | |
| *** esberglu has joined #openstack-infra | 03:15 | |
| *** thorst has quit IRC | 03:17 | |
| *** thorst has joined #openstack-infra | 03:17 | |
| *** esberglu has quit IRC | 03:20 | |
| *** hamzy has quit IRC | 03:20 | |
| *** thorst has quit IRC | 03:24 | |
| *** baoli has quit IRC | 03:25 | |
| *** slaweq has joined #openstack-infra | 03:36 | |
| *** slaweq has quit IRC | 03:40 | |
| *** markvoelker has quit IRC | 03:47 | |
| *** kjackal_ has quit IRC | 04:02 | |
| *** aeng has quit IRC | 04:03 | |
| *** hongbin has quit IRC | 04:04 | |
| *** esberglu has joined #openstack-infra | 04:09 | |
| *** dmsimard is now known as dmsimard|off | 04:10 | |
| *** esberglu has quit IRC | 04:13 | |
| *** udesale has joined #openstack-infra | 04:13 | |
| *** Dinesh_Bhor has joined #openstack-infra | 04:14 | |
| openstackgerrit | Tuan Luong-Anh proposed openstack/diskimage-builder master: Replace new location of isolinux.bin for Ubuntu 16.04 https://review.openstack.org/477437 | 04:14 |
| *** ykarel has joined #openstack-infra | 04:14 | |
| *** aeng has joined #openstack-infra | 04:16 | |
| *** gildub has joined #openstack-infra | 04:16 | |
| *** claudiub has joined #openstack-infra | 04:22 | |
| *** thorst has joined #openstack-infra | 04:25 | |
| *** links has joined #openstack-infra | 04:25 | |
| *** coolsvap has joined #openstack-infra | 04:26 | |
| *** sree has joined #openstack-infra | 04:27 | |
| *** thorst has quit IRC | 04:29 | |
| *** psachin has joined #openstack-infra | 04:31 | |
| *** sree has quit IRC | 04:31 | |
| *** aeng has quit IRC | 04:32 | |
| *** aeng has joined #openstack-infra | 04:32 | |
| *** stakeda has joined #openstack-infra | 04:33 | |
| *** slaweq has joined #openstack-infra | 04:37 | |
| *** slaweq has quit IRC | 04:42 | |
| *** sree has joined #openstack-infra | 04:44 | |
| *** markvoelker has joined #openstack-infra | 04:44 | |
| *** udesale has quit IRC | 04:48 | |
| *** udesale has joined #openstack-infra | 04:49 | |
| *** dhajare has joined #openstack-infra | 04:50 | |
| *** eranrom has quit IRC | 04:51 | |
| *** esberglu has joined #openstack-infra | 05:03 | |
| *** esberglu has quit IRC | 05:08 | |
| *** markvoelker has quit IRC | 05:17 | |
| *** zhurong has quit IRC | 05:21 | |
| *** thorst has joined #openstack-infra | 05:25 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set repo setup release in playbook https://review.openstack.org/504939 | 05:29 |
| *** florianf has joined #openstack-infra | 05:56 | |
| *** florianf has quit IRC | 05:57 | |
| *** esberglu has joined #openstack-infra | 05:58 | |
| *** slaweq has joined #openstack-infra | 06:00 | |
| *** esberglu has quit IRC | 06:02 | |
| *** florianf has joined #openstack-infra | 06:03 | |
| *** slaweq has quit IRC | 06:05 | |
| *** kuromagi has quit IRC | 06:06 | |
| *** kuromagi has joined #openstack-infra | 06:09 | |
| *** zhurong has joined #openstack-infra | 06:14 | |
| *** markvoelker has joined #openstack-infra | 06:14 | |
| *** aviau has quit IRC | 06:20 | |
| *** hashar has joined #openstack-infra | 06:20 | |
| *** aviau has joined #openstack-infra | 06:20 | |
| *** yamamoto_ has joined #openstack-infra | 06:36 | |
| *** yamamoto has quit IRC | 06:36 | |
| *** rcernin has joined #openstack-infra | 06:41 | |
| *** slaweq has joined #openstack-infra | 06:41 | |
| *** slaweq has quit IRC | 06:45 | |
| *** slaweq has joined #openstack-infra | 06:48 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set repo setup release in playbook https://review.openstack.org/504939 | 06:48 |
| *** markvoelker has quit IRC | 06:48 | |
| *** esberglu has joined #openstack-infra | 06:52 | |
| *** thorst has quit IRC | 06:52 | |
| *** andreas_s has joined #openstack-infra | 06:54 | |
| *** rossella_s has quit IRC | 06:56 | |
| *** jascott1 has quit IRC | 06:56 | |
| *** esberglu has quit IRC | 06:56 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Create rescue user on ironic agent https://review.openstack.org/506579 | 07:06 |
| *** pcaruana has joined #openstack-infra | 07:06 | |
| *** eroux has joined #openstack-infra | 07:13 | |
| *** yamamoto_ has quit IRC | 07:14 | |
| *** zhurong has quit IRC | 07:14 | |
| *** yamamoto has joined #openstack-infra | 07:16 | |
| *** tesseract has joined #openstack-infra | 07:23 | |
| *** jesusaur has quit IRC | 07:23 | |
| *** alexchadin has joined #openstack-infra | 07:24 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: DNM: Create rescue user on ironic agent https://review.openstack.org/506579 | 07:30 |
| *** zhurong has joined #openstack-infra | 07:32 | |
| *** yamamoto_ has joined #openstack-infra | 07:33 | |
| *** ykarel is now known as ykarel|lunch | 07:35 | |
| *** pgadiya has joined #openstack-infra | 07:36 | |
| *** yamamoto has quit IRC | 07:36 | |
| *** ccamacho has joined #openstack-infra | 07:37 | |
| *** alexchadin has quit IRC | 07:39 | |
| *** alexchadin has joined #openstack-infra | 07:40 | |
| *** jpena|off is now known as jpena | 07:40 | |
| *** yamamoto_ has quit IRC | 07:41 | |
| *** markvoelker has joined #openstack-infra | 07:45 | |
| *** ralonsoh has joined #openstack-infra | 07:46 | |
| *** esberglu has joined #openstack-infra | 07:46 | |
| *** egonzalez has joined #openstack-infra | 07:47 | |
| *** thorst has joined #openstack-infra | 07:49 | |
| *** esberglu has quit IRC | 07:50 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Unset UPPER_CONSTRAINTS_FILE on dib https://review.openstack.org/507009 | 07:52 |
| *** shardy has joined #openstack-infra | 07:55 | |
| *** jpich has joined #openstack-infra | 07:55 | |
| *** gildub has quit IRC | 08:01 | |
| *** pgadiya has quit IRC | 08:04 | |
| *** ykarel|lunch is now known as ykarel | 08:08 | |
| *** yamamoto has joined #openstack-infra | 08:10 | |
| *** zaneb has quit IRC | 08:17 | |
| *** markvoelker has quit IRC | 08:18 | |
| yolanda | AJaeger_, around ? | 08:21 |
| yolanda | can you look at https://review.openstack.org/507009 ? dib gates are broken | 08:21 |
| *** zaneb has joined #openstack-infra | 08:23 | |
| *** e0ne has joined #openstack-infra | 08:25 | |
| *** quite has quit IRC | 08:28 | |
| *** quite has joined #openstack-infra | 08:29 | |
| *** mikal_ has joined #openstack-infra | 08:29 | |
| *** jesusaur has joined #openstack-infra | 08:31 | |
| *** zhurong has quit IRC | 08:32 | |
| *** mikal has quit IRC | 08:32 | |
| *** yamamoto has quit IRC | 08:32 | |
| openstackgerrit | Merged openstack-infra/project-config master: Unset UPPER_CONSTRAINTS_FILE on dib https://review.openstack.org/507009 | 08:35 |
| *** tosky has joined #openstack-infra | 08:37 | |
| *** esberglu has joined #openstack-infra | 08:40 | |
| *** derekh has joined #openstack-infra | 08:42 | |
| *** rwsu has joined #openstack-infra | 08:43 | |
| *** udesale__ has joined #openstack-infra | 08:44 | |
| *** esberglu has quit IRC | 08:44 | |
| *** udesale has quit IRC | 08:46 | |
| *** alexchadin has quit IRC | 08:46 | |
| *** alexchadin has joined #openstack-infra | 08:47 | |
| *** dingyichen has quit IRC | 08:47 | |
| *** alexchadin has quit IRC | 08:47 | |
| *** finucannot is now known as stephenfin | 08:47 | |
| *** alexchadin has joined #openstack-infra | 08:48 | |
| *** alexchadin has quit IRC | 08:48 | |
| *** alexchadin has joined #openstack-infra | 08:49 | |
| *** zhurong has joined #openstack-infra | 08:51 | |
| *** alexchadin has quit IRC | 08:52 | |
| *** udesale has joined #openstack-infra | 08:52 | |
| *** udesale__ has quit IRC | 08:54 | |
| frickler | yolanda: IIRC AJaeger_ is travelling this week | 08:55 |
| yolanda | yep, i remember he told us... | 08:55 |
| *** armax has joined #openstack-infra | 08:56 | |
| *** jascott1 has joined #openstack-infra | 08:56 | |
| *** jascott1 has quit IRC | 09:01 | |
| *** thorst has quit IRC | 09:07 | |
| *** gongysh has joined #openstack-infra | 09:11 | |
| *** eranrom has joined #openstack-infra | 09:12 | |
| *** markvoelker has joined #openstack-infra | 09:15 | |
| *** electrofelix has joined #openstack-infra | 09:18 | |
| *** eranrom has quit IRC | 09:18 | |
| *** armax has quit IRC | 09:19 | |
| *** pbourke has quit IRC | 09:22 | |
| *** pbourke has joined #openstack-infra | 09:23 | |
| *** yamahata has joined #openstack-infra | 09:23 | |
| *** bauwser is now known as bauzas | 09:27 | |
| *** jaosorior has joined #openstack-infra | 09:28 | |
| *** ykarel is now known as ykarel|meeting | 09:30 | |
| *** armax has joined #openstack-infra | 09:32 | |
| *** ykarel_ has joined #openstack-infra | 09:32 | |
| *** esberglu has joined #openstack-infra | 09:34 | |
| *** ykarel|meeting has quit IRC | 09:35 | |
| *** dizquierdo has joined #openstack-infra | 09:35 | |
| *** ykarel has joined #openstack-infra | 09:35 | |
| *** ykarel_ has quit IRC | 09:37 | |
| *** esberglu has quit IRC | 09:38 | |
| *** yamamoto has joined #openstack-infra | 09:40 | |
| *** armax has quit IRC | 09:43 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 09:47 |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:47 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 09:48 |
| *** markvoelker has quit IRC | 09:48 | |
| *** gongysh has quit IRC | 09:48 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 09:53 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 09:57 |
| *** namnh has quit IRC | 09:59 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** nicolasbock has joined #openstack-infra | 10:02 | |
| *** cshastri has quit IRC | 10:03 | |
| *** yamahata has quit IRC | 10:04 | |
| *** thorst has joined #openstack-infra | 10:04 | |
| *** cuongnv has quit IRC | 10:10 | |
| *** cuongnv has joined #openstack-infra | 10:11 | |
| *** nicolasbock has quit IRC | 10:12 | |
| *** liujiong has quit IRC | 10:12 | |
| *** stakeda has quit IRC | 10:17 | |
| szaher | AJaeger_: Thanks for helping us in sorting out freezer docs :) | 10:18 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 10:19 |
| *** armax has joined #openstack-infra | 10:19 | |
| *** zhurong has quit IRC | 10:21 | |
| *** eroux has quit IRC | 10:22 | |
| *** nicolasbock has joined #openstack-infra | 10:24 | |
| *** cuongnv has quit IRC | 10:24 | |
| *** zhurong has joined #openstack-infra | 10:24 | |
| *** esberglu has joined #openstack-infra | 10:28 | |
| *** jaypipes has joined #openstack-infra | 10:30 | |
| *** esberglu has quit IRC | 10:33 | |
| *** gcb has joined #openstack-infra | 10:33 | |
| *** ykarel_ has joined #openstack-infra | 10:34 | |
| *** ykarel has quit IRC | 10:35 | |
| *** udesale has quit IRC | 10:37 | |
| *** alexchadin has joined #openstack-infra | 10:41 | |
| *** markvoelker has joined #openstack-infra | 10:45 | |
| *** jkilpatr has joined #openstack-infra | 10:48 | |
| *** pblaho has joined #openstack-infra | 10:51 | |
| *** gcb has quit IRC | 10:53 | |
| *** LindaWang has quit IRC | 10:56 | |
| *** armax has quit IRC | 10:57 | |
| *** sdague has joined #openstack-infra | 10:59 | |
| *** armax has joined #openstack-infra | 11:02 | |
| *** armax has quit IRC | 11:03 | |
| *** armax has joined #openstack-infra | 11:03 | |
| *** armax has quit IRC | 11:04 | |
| *** slaweq has quit IRC | 11:04 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 11:11 |
| *** LindaWang has joined #openstack-infra | 11:13 | |
| *** thorst has quit IRC | 11:14 | |
| *** wolverineav has joined #openstack-infra | 11:18 | |
| *** markvoelker has quit IRC | 11:18 | |
| *** sree has quit IRC | 11:20 | |
| *** sree has joined #openstack-infra | 11:20 | |
| *** slaweq has joined #openstack-infra | 11:21 | |
| *** esberglu has joined #openstack-infra | 11:22 | |
| *** sree has quit IRC | 11:24 | |
| *** sree has joined #openstack-infra | 11:24 | |
| *** jpena is now known as jpena|lunch | 11:24 | |
| *** clayton has quit IRC | 11:25 | |
| *** dtantsur|afk is now known as dtantsur | 11:26 | |
| *** esberglu has quit IRC | 11:27 | |
| *** clayton has joined #openstack-infra | 11:30 | |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 11:32 |
| *** bobh has joined #openstack-infra | 11:36 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Create rescue user on ironic agent https://review.openstack.org/506579 | 11:41 |
| *** sree has quit IRC | 11:41 | |
| *** sree has joined #openstack-infra | 11:42 | |
| *** rhallisey has joined #openstack-infra | 11:42 | |
| *** sree has quit IRC | 11:42 | |
| *** sree has joined #openstack-infra | 11:43 | |
| *** pblaho has quit IRC | 11:44 | |
| *** jascott1 has joined #openstack-infra | 11:55 | |
| *** eranrom has joined #openstack-infra | 11:56 | |
| *** iyamahat has joined #openstack-infra | 11:57 | |
| *** iyamahat_ has joined #openstack-infra | 11:58 | |
| *** iyamahat has quit IRC | 11:58 | |
| *** jascott1 has quit IRC | 11:59 | |
| *** yamamoto has quit IRC | 12:00 | |
| *** thorst has joined #openstack-infra | 12:01 | |
| *** dprince has joined #openstack-infra | 12:01 | |
| *** thorst_ has joined #openstack-infra | 12:04 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Create rescue user on ironic agent https://review.openstack.org/506579 | 12:05 |
| *** thorst has quit IRC | 12:05 | |
| *** alexchadin has quit IRC | 12:07 | |
| *** alexchadin has joined #openstack-infra | 12:07 | |
| *** pblaho has joined #openstack-infra | 12:08 | |
| openstackgerrit | Daniel Speichert proposed openstack-infra/shade master: Support filtering servers in list_servers using arbitrary parameters https://review.openstack.org/506969 | 12:10 |
| *** rlandy has joined #openstack-infra | 12:10 | |
| *** jcoufal has joined #openstack-infra | 12:10 | |
| *** bobh has quit IRC | 12:12 | |
| *** sambetts|afk is now known as sambetts | 12:13 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Remove promotion logic from the toci scripts https://review.openstack.org/507076 | 12:15 |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| *** esberglu has joined #openstack-infra | 12:16 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** MasterofJOKers_ has joined #openstack-infra | 12:20 | |
| *** esberglu has quit IRC | 12:21 | |
| *** sree has quit IRC | 12:21 | |
| *** sree has joined #openstack-infra | 12:21 | |
| *** MasterofJOKers has quit IRC | 12:22 | |
| *** MasterofJOKers_ is now known as MasterofJOKers | 12:22 | |
| *** alexchadin has quit IRC | 12:22 | |
| *** trown|outtypewww is now known as trown | 12:23 | |
| *** markvoelker has quit IRC | 12:23 | |
| *** markvoelker has joined #openstack-infra | 12:24 | |
| *** jcoufal has quit IRC | 12:24 | |
| *** jpena|lunch is now known as jpena | 12:25 | |
| *** sree has quit IRC | 12:26 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** erlon has joined #openstack-infra | 12:38 | |
| *** derekh has quit IRC | 12:40 | |
| *** derekh has joined #openstack-infra | 12:41 | |
| *** udesale has joined #openstack-infra | 12:48 | |
| *** zhurong has quit IRC | 12:50 | |
| frickler | what is the procedure for getting a new etcd tarball onto tarballs.o.o? trying to update devstack to deploy a more recent version fails because of this: http://logs.openstack.org/68/507068/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/da097b3/logs/devstacklog.txt.gz#_2017-09-25_11_52_10_757 | 12:51 |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 12:53 |
| *** amoralej is now known as amoralej|lunch | 12:56 | |
| *** kgiusti has joined #openstack-infra | 12:59 | |
| *** udesale has quit IRC | 13:00 | |
| *** mriedem has joined #openstack-infra | 13:00 | |
| *** jistr is now known as jistr|mtg | 13:03 | |
| *** sree has joined #openstack-infra | 13:05 | |
| *** iyamahat has joined #openstack-infra | 13:06 | |
| *** iyamahat_ has quit IRC | 13:07 | |
| *** iyamahat has quit IRC | 13:07 | |
| *** iyamahat has joined #openstack-infra | 13:07 | |
| fungi | frickler: i thought dims was working on that. there was supposed to be some automation to update builds of it, but that came about before we added caching reverse proxies in every provider so maybe at this point it would make more sense to set up a proxy for that instead of putting it on the tarballs.o.o site? | 13:09 |
| *** gcb has joined #openstack-infra | 13:10 | |
| *** esberglu has joined #openstack-infra | 13:11 | |
| *** baoli has joined #openstack-infra | 13:12 | |
| mnaser | around what time is the cutover expected to happen? | 13:16 |
| *** dhajare has quit IRC | 13:17 | |
| *** larsks has quit IRC | 13:20 | |
| *** kjackal_ has joined #openstack-infra | 13:21 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Create rescue user on ironic agent https://review.openstack.org/506579 | 13:21 |
| *** larsks has joined #openstack-infra | 13:23 | |
| *** links has quit IRC | 13:23 | |
| *** iyamahat has quit IRC | 13:23 | |
| dirk | fungi: any pointers to the reverse proxy thing? | 13:23 |
| dims | frickler : last i remember was tonyb taking over that. will check with him tomorrow morning his time | 13:26 |
| *** gouthamr has joined #openstack-infra | 13:27 | |
| *** lbragstad has joined #openstack-infra | 13:28 | |
| *** kiennt26 has joined #openstack-infra | 13:29 | |
| *** gouthamr has quit IRC | 13:29 | |
| *** jistr|mtg is now known as jistr | 13:29 | |
| fungi | dirk: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n149 and https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n97 | 13:29 |
| *** gouthamr has joined #openstack-infra | 13:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add the rest of the projects to Zuul v3 https://review.openstack.org/506878 | 13:33 |
| Shrews | \o/ | 13:33 |
| fungi | mnaser: ^ | 13:34 |
| fungi | that is looking close to cutoverish | 13:34 |
| mnaser | https://m.popkey.co/fe8997/9wGMD.gif | 13:34 |
| *** coolsvap has quit IRC | 13:35 | |
| *** dave-mccowan has joined #openstack-infra | 13:41 | |
| *** Goneri has joined #openstack-infra | 13:42 | |
| *** amoralej|lunch is now known as amoralej | 13:43 | |
| *** srobert has joined #openstack-infra | 13:45 | |
| *** tpsilva has joined #openstack-infra | 13:46 | |
| *** jheroux has joined #openstack-infra | 13:49 | |
| *** dpawlik has quit IRC | 13:49 | |
| *** danpawlik has joined #openstack-infra | 13:50 | |
| *** bswartz has quit IRC | 13:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Split zuul.yaml into separate files in zuul.d https://review.openstack.org/507107 | 13:50 |
| dirk | fungi: thanks! | 13:51 |
| dirk | looks like we don't want to mess with complicated downloading but just install the distro provided package | 13:52 |
| *** rkukura has quit IRC | 13:54 | |
| *** chlong has joined #openstack-infra | 13:57 | |
| *** MasterofJOKers has quit IRC | 14:01 | |
| *** beekneemech is now known as bnemec | 14:02 | |
| *** superdan is now known as dansmith | 14:02 | |
| *** MasterofJOKers has joined #openstack-infra | 14:03 | |
| *** sree has quit IRC | 14:03 | |
| *** sree has joined #openstack-infra | 14:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Sort output in generated legacy files https://review.openstack.org/507118 | 14:05 |
| jeblair | mordred: good morning! what's our situation? | 14:05 |
| *** sree has quit IRC | 14:08 | |
| openstackgerrit | Julie Pichon proposed openstack-infra/project-config master: Adjust branches for OSC jobs https://review.openstack.org/503500 | 14:13 |
| *** armax has joined #openstack-infra | 14:18 | |
| mwhahaha | hey if someone gets a chance can you add me to the ops list in #tripleo | 14:19 |
| *** hongbin has joined #openstack-infra | 14:20 | |
| *** BlartVersenwald is now known as NobodyCam | 14:22 | |
| *** rkukura has joined #openstack-infra | 14:23 | |
| *** dizquierdo has quit IRC | 14:25 | |
| *** srobert has quit IRC | 14:25 | |
| *** bobh has joined #openstack-infra | 14:25 | |
| *** rbrndt has joined #openstack-infra | 14:26 | |
| mordred | jeblair: pretty good - I'm writing a couple of quick patches to make things a little cleaner | 14:32 |
| mordred | jeblair: if you havea sec, could you look at https://review.openstack.org/#/c/507107/ ? | 14:32 |
| mordred | jeblair: we got a build timeout - I haven't looked yet - but I'm guessing it's zuul-sphinx zuul.d related | 14:32 |
| jeblair | mordred: i'll check it out | 14:33 |
| mordred | jeblair, fungi: also https://review.openstack.org/#/c/506877 and https://review.openstack.org/#/c/506878 are ready for review ... one is a tool to add things to main.yaml - the other is a patch adding them | 14:34 |
| jeblair | mordred: yeah, zuul-sphinx is in an infinite loop | 14:38 |
| jeblair | mordred: i'll fix | 14:38 |
| mordred | jeblair, fungi: also - in writing that tool, I have learned that ruayaml is nice for doing yaml-config-file transformations where we wantto keep comments - but WOW is it a billion times slower than normal yaml | 14:38 |
| *** ykarel_ has quit IRC | 14:38 | |
| fungi | mordred: yeah, i tried to use it myself for a few projects and eventually gave up (instead going with additional structured data to replace most of the need for comments) | 14:39 |
| mordred | fungi: it took some reading of the source to figure out how to do the things I wanted to do | 14:39 |
| mordred | fungi: so I shouldn't say "nice" | 14:39 |
| mordred | fungi: as much as "possible" | 14:39 |
| *** Goneri has quit IRC | 14:39 | |
| Shrews | fungi: i ran the tool. it runs quick enough for this :) | 14:40 |
| Shrews | err, mordred, rather ^^^ | 14:40 |
| mordred | Shrews: ++ | 14:40 |
| fungi | yeah, for a one-off thing it seems like a fine fit | 14:40 |
| *** iyamahat has joined #openstack-infra | 14:41 | |
| fungi | for general software design thuogh, i finally arrived at the conclusion that if you want to read and regurgitate serialized data while retaining contents and ordering (beyond explicitly ordered data types) you're missing the point of serialization | 14:41 |
| *** liujiong has joined #openstack-infra | 14:41 | |
| mordred | jeblair: also - idea based on the zuul.yaml -> zuul.d split - perhaps we should allow having a .zuul.yaml in a repo that also has a zuul.d - openstack-zuul-jobs in particular having the project pipeline config in the directory with the shared job config seems eversomildly off to me - especially once we allow omitting a project's name from its own project pipeline config | 14:42 |
| jeblair | mordred: i believe the problem is that we did not release zuul-sphinx after adding support for zuul.d | 14:42 |
| *** iyamahat has quit IRC | 14:42 | |
| *** iyamahat has joined #openstack-infra | 14:42 | |
| jeblair | mordred: if you like, i will release 0.1.6 now. | 14:42 |
| mordred | jeblair: ah! that would certainly do it | 14:42 |
| mordred | jeblair: ++ | 14:42 |
| fungi | and also that if you're (ab)using a serialization protocol as a configuration file format, it's not the worst thing in the world to add keys like "description" and "comment" and so on, as they can also be accessed from within the data structure if you should ever want them | 14:42 |
| jeblair | zuul-sphinx 0.1.6 pushed | 14:43 |
| *** Goneri has joined #openstack-infra | 14:43 | |
| *** armax has quit IRC | 14:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Merge project pipeline definitions https://review.openstack.org/507163 | 14:47 |
| mordred | ok . that ^^ and its parents are ready for review - and I'll push up patches generated from it next | 14:47 |
| *** ihrachys has joined #openstack-infra | 14:48 | |
| openstackgerrit | Merged openstack-dev/cookiecutter master: Switch to use stestr as the default test runner https://review.openstack.org/505772 | 14:51 |
| *** LindaWang has quit IRC | 14:52 | |
| jeblair | pabelanger: are you around today, or are you busy moving? | 14:53 |
| *** makowals_ has quit IRC | 14:53 | |
| pabelanger | jeblair: I'm here today | 14:54 |
| pabelanger | so far plan to be out Thursday, Friday of this week | 14:54 |
| *** mwarad has joined #openstack-infra | 14:55 | |
| *** gcb has quit IRC | 14:55 | |
| *** LindaWang has joined #openstack-infra | 14:55 | |
| *** xarses_ has joined #openstack-infra | 14:56 | |
| jeblair | pabelanger: cool; would you mind working up an etherpad with a list of hosts being used by zuulv2 and another list of hosts used by zuulv3 please? i've lost track. :) | 14:56 |
| pabelanger | sure | 14:56 |
| jeblair | i'm going to grab some breakfast, biab. | 14:56 |
| *** bobh has quit IRC | 14:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add main.yaml tool https://review.openstack.org/506877 | 14:58 |
| *** iyamahat has quit IRC | 14:58 | |
| *** felipemonteiro has joined #openstack-infra | 14:58 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't double quote rsync options https://review.openstack.org/506802 | 14:59 |
| *** srobert has joined #openstack-infra | 14:59 | |
| mordred | jeblair, pabelanger: before we go too crazy this morning, we may want to investigate pabelanger's xenial/trusty issue from the weekend | 14:59 |
| mordred | Shrews: ^^ you might be intersted in that and/or be able to help investigate | 14:59 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Sort output in generated legacy files https://review.openstack.org/507118 | 15:00 |
| mordred | Shrews, jeblair: tl;dr - job ran with a node declared as ubuntu-xenial but got an ubuntu-trusty node instead | 15:00 |
| dmellado | pabelanger: are you moving? good luck! I still have boxes at home from 6 months ago xD | 15:00 |
| Shrews | i missed that over the weekend. review? | 15:00 |
| pabelanger | mordred: I was thinking about it more, this might be the first job build-wheel-mirror, where we are mixing nodes. | 15:01 |
| clarkb | dhellmann notes on the dev list that zuul-sphinx release failed | 15:01 |
| clarkb | (not sure if already caught) | 15:01 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: Add zun-tempest-plugin into PROJECTS and local.conf https://review.openstack.org/506886 | 15:01 |
| dmellado | regarding the zuulv3 migration and move, are there any final dates yet? | 15:01 |
| dmellado | I'd like to refactor de kuryr gates and would like to start checking if a base zuulv3 job works ;)ç | 15:02 |
| *** felipemonteiro_ has joined #openstack-infra | 15:02 | |
| pabelanger | jeblair: I've updated https://etherpad.openstack.org/p/zuulv3-pre-ptg with current breakdown of server for zuulv2.5 and zuulv3 | 15:02 |
| pabelanger | dmellado: Yup, on the road Friday | 15:03 |
| *** mwarad has quit IRC | 15:03 | |
| dmellado | pabelanger: good luck! hopefully it won't be too far! | 15:03 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 15:04 |
| mordred | pabelanger, Shrews: well yah - BUT - the host info at http://logs.openstack.org/51/506951/1/periodic/build-wheel-mirror/f86d8de/zuul-info/host-info.wheel-mirror-ubuntu-xenial-python3.yaml is for one of those hosts and its name leads me to believe it was supposed to be a xenial node | 15:04 |
| mordred | Shrews: ^^ | 15:05 |
| jeblair | clarkb, dhellmann: thanks.... annoyingly, that release is a zuulv3 blocker and it failed due to a situation that zuulv3 fixes. :| i'll retrigger it | 15:05 |
| *** felipemonteiro has quit IRC | 15:05 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 15:06 |
| pabelanger | Shrews: mordred: http://logs.openstack.org/51/506951/1/periodic/build-wheel-mirror/f86d8de/zuul-info/host-info.wheel-mirror-ubuntu-trusty-python2.yaml is the node which it seemed to switch with. labeled trusty, actually xenial | 15:07 |
| *** links has joined #openstack-infra | 15:09 | |
| *** dizquierdo has joined #openstack-infra | 15:09 | |
| Shrews | where are these links coming from? i don't see any zuul voting on that review | 15:09 |
| pabelanger | Shrews: they were manually enqueued into periodic pipeline | 15:10 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.yaml#n547 is the job | 15:10 |
| Shrews | pabelanger: the nodeset defined for that job includes both xenial and trusty. i'm confused as to what the problem is | 15:11 |
| Shrews | "job ran with a node declared as ubuntu-xenial but got an ubuntu-trusty node instead" <--- where is that happening? | 15:12 |
| pabelanger | Shrews: http://logs.openstack.org/51/506951/1/periodic/build-wheel-mirror/f86d8de/zuul-info/host-info.wheel-mirror-ubuntu-trusty-python2.yaml has a hostname of ubuntu-trusty, but look at the codename, it is actually a xenial node. | 15:12 |
| *** ccamacho has quit IRC | 15:13 | |
| pabelanger | you'll have to match it up with http://logs.openstack.org/51/506951/1/periodic/build-wheel-mirror/f86d8de/zuul-info/inventory.yaml | 15:13 |
| pabelanger | you can see the labels in inventory are correct, but the actually nodes are not | 15:14 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 15:16 |
| *** links has quit IRC | 15:17 | |
| *** Julien-z_ has joined #openstack-infra | 15:17 | |
| Shrews | pabelanger: ok. so looking at all of those hosts, looks like we got 2 xenials and 2 trustys (as requestd), but they were not assigned correctly to the declared nodes. this seems to suggest we should look at that section of zuul code | 15:19 |
| Shrews | (just trying to eliminate if nodepool behaved incorrectly. seems like it returned what was requested) | 15:19 |
| jeblair | Shrews: did it return them in the correct order? | 15:20 |
| Shrews | jeblair: is there an order? | 15:20 |
| Shrews | or an expected order? i honestly forget | 15:20 |
| Shrews | i think nodepool just adds the node ids to the noderequest, yeah? | 15:20 |
| jeblair | Shrews: yeah, so as long as it doesn't change the order, should be fine | 15:21 |
| jeblair | (but of course, in zk we're rewriting the znode all the time so anything could happen :) | 15:21 |
| *** kiennt26 has quit IRC | 15:21 | |
| *** e0ne has quit IRC | 15:23 | |
| jeblair | Shrews: node_types is an ordered list, and nodes is expected to match it | 15:23 |
| Shrews | i don't recall any order requirements, so it's possible it's not added them as zuul would expect. looking | 15:23 |
| openstackgerrit | Reedip proposed openstack-infra/irc-meetings master: Proposal for change of FWaaS meeting https://review.openstack.org/507172 | 15:25 |
| *** LindaWang has quit IRC | 15:27 | |
| jeblair | Shrews: yeah, it looks like _waitForNodeSet has a couple opportunities to change the order | 15:28 |
| *** links has joined #openstack-infra | 15:30 | |
| Shrews | jeblair: definitely not considering order: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/__init__.py?h=feature/zuulv3#n194 | 15:30 |
| *** apuimedo has quit IRC | 15:32 | |
| Shrews | i'll put a fix together, and a test | 15:32 |
| jeblair | Shrews: cool, thx | 15:32 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Create rescue user on ironic agent https://review.openstack.org/506579 | 15:33 |
| clarkb | https://review.openstack.org/#/c/506889/ fixes one of the last remaining gerrit upgrade issues if we can get that reviewed this morning would be great (but lots going on too so understand if focus is elsewhere) | 15:33 |
| *** pcaruana has quit IRC | 15:33 | |
| *** GregHous- has quit IRC | 15:33 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Use '[sphinx_build] builders' with Sphinx < 1.6 https://review.openstack.org/481620 | 15:33 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate support for Sphinx < 1.6 https://review.openstack.org/481621 | 15:33 |
| *** pcaruana has joined #openstack-infra | 15:34 | |
| jeblair | mordred: was there any information you wanted in the infra-manual zuulv3 migration page before we switch? | 15:35 |
| openstackgerrit | Daniel Speichert proposed openstack-infra/shade master: Support filtering servers in list_servers using arbitrary parameters https://review.openstack.org/506969 | 15:35 |
| *** ociuhandu has joined #openstack-infra | 15:36 | |
| openstackgerrit | Daniel Speichert proposed openstack-infra/shade master: Support filtering servers in list_servers using arbitrary parameters https://review.openstack.org/506969 | 15:36 |
| *** GregHouse has joined #openstack-infra | 15:36 | |
| jeblair | pabelanger: thanks, i copied that info into a new etherpad (and updated slightly): https://etherpad.openstack.org/p/zuulv3-cutover | 15:37 |
| pabelanger | ack | 15:38 |
| clarkb | pabelanger: jeblair does that imply we need to move zk at some point? | 15:38 |
| openstackgerrit | Merged openstack-infra/jeepyb master: Handle newer gerrit in update blueprint script https://review.openstack.org/506889 | 15:38 |
| jeblair | pabelanger: it looks like we're all set on executors, but we're under-supplied on mergers. | 15:38 |
| clarkb | (maybe during cutover?) | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Move single-use wheel-builder nodeset into job https://review.openstack.org/507177 | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Split zuul.yaml into files in zuul.d https://review.openstack.org/507178 | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove zuul v2 and jjb content https://review.openstack.org/507180 | 15:38 |
| jeblair | clarkb: i don't think so -- we can just keep using nodepool.o.o as our zk server for the time being. | 15:38 |
| clarkb | ok | 15:38 |
| jeblair | pabelanger: the zuul-sphinx job failed because it tried to use zm06 | 15:39 |
| pabelanger | jeblair: ya, we still need 4 more mergers for zuulv3 to bring online | 15:39 |
| mordred | jeblair: sounds like we'll have a good opportunity to learn about setting up a zk install into multiple nodes to migrate it off of nodepool.o.o :) | 15:39 |
| *** dtantsur is now known as dtantsur|bbl | 15:39 | |
| mnaser | +185256, -0 | 15:39 |
| pabelanger | jeblair: Hmm, I am not sure why that would happen | 15:39 |
| jeblair | pabelanger: is it possible that zm06 is registered with the wrong zuul? | 15:39 |
| pabelanger | checking | 15:39 |
| *** srobert_ has joined #openstack-infra | 15:40 | |
| jeblair | pabelanger: http://logs.openstack.org/fc/fca979f1eaaed60af404e74c51c0ad67710e18cf/release/zuul-sphinx-pypi-both-upload/2826256/console.html is the log | 15:40 |
| *** ociuhandu has quit IRC | 15:40 | |
| pabelanger | jeblair: zm06 has zuulv3.o.o as gearman server | 15:40 |
| mordred | odyssey4me: I mentioned you on friday but then never followed up - the tl;dr is that we'll likely need a base job for openstack-ansible jobs that declares the base list of openstack projects that zuul should clone for you | 15:40 |
| odyssey4me | mordred this is for the zuul v3 change in implementation for z-c right? | 15:41 |
| mordred | odyssey4me: yes. in v3 z-c is just a shim that knows how to get the git repos from where they've been put on the test node | 15:42 |
| *** rhallisey has quit IRC | 15:43 | |
| mordred | odyssey4me: and it does not have the ability to actually clone things from git.o.o if they are missing | 15:43 |
| jeblair | pabelanger: the only merger:update functions registered with zuulv2 gearman are zm01 and zm04 | 15:43 |
| *** liujiong has quit IRC | 15:43 | |
| *** srobert has quit IRC | 15:43 | |
| odyssey4me | mordred hmm, so even if I give it a project to clone it would not do it? | 15:43 |
| mordred | odyssey4me: the migration script trolls through job definitions for zuul-cloner use and sets up required-projects as necessary - but for some more complex folks (like you) that will not be sufficient | 15:43 |
| mordred | odyssey4me: the key will be making something like this: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.yaml#n327 | 15:44 |
| pabelanger | jeblair: k, I'm not sure how that could happen. During work at PTG for zuulv3 mergers, I never actually touched zm01 - zm04 servers. | 15:44 |
| mordred | odyssey4me: which tells zuul to prepare and put onto the node all the projects you need - so that zuul-cloner will properly deal with them for you | 15:44 |
| jeblair | pabelanger: right; the question is how zuulv2 interacted with zm06 | 15:45 |
| *** andreas_s has quit IRC | 15:46 | |
| mordred | inc0: ^^ please see scrollback between odyssey4me and I - I'm fairly sure we'll need something similar for kolla jobs | 15:46 |
| *** srobert_ has quit IRC | 15:46 | |
| pabelanger | jeblair: it shouldn't, in fact there is no more firewalls rules to allow zm05-zm08 to talk with zuul.o.o | 15:46 |
| *** srobert has joined #openstack-infra | 15:47 | |
| jeblair | hrm, the zuul_url for that job was zm01 | 15:47 |
| jeblair | oh! | 15:47 |
| jeblair | pabelanger: zlstatic01 keeps repos around (because it's a static host) | 15:47 |
| pabelanger | Ah | 15:47 |
| jeblair | it originally cloned that project from zm06 | 15:47 |
| jeblair | so when it was checking for updates, it tried to contact zm06 | 15:47 |
| jeblair | pabelanger: i think we need to delete all the project workspaces on zlstatic01 | 15:47 |
| pabelanger | okay | 15:48 |
| jeblair | pabelanger: i'll do that | 15:48 |
| pabelanger | +1 | 15:48 |
| mordred | inc0, mnaser, odyssey4me, EmilienM: it's also highly likely that we will have missed something due to the complexity of your jobs and will need to fast-fail-forward - so we should keep our eyes peeled on your jobs and be prepared to troubleshoot and fix rapidly | 15:48 |
| *** rcernin has quit IRC | 15:48 | |
| mordred | pabelanger: oh - can you update test_pypi secret and replace it with the real pypi secret? | 15:48 |
| mnaser | mordred yes, i'm grabbing food in a little bit and im pretty much free-ing up the rest of my day to clean up p-o-i jobs :) | 15:48 |
| jeblair | pabelanger: er, not zlstatic01, but release.slave, and other static slaves | 15:49 |
| EmilienM | mordred: ready to get hands dirty :D | 15:49 |
| jeblair | #status log deleted all workspaces from release.slave.openstack.org to deal with changes to zuul v2 mergers | 15:50 |
| openstackstatus | jeblair: finished logging | 15:50 |
| mordred | jeblair: we could also switch zuul-sphinx to use v3 for releasing real quick | 15:50 |
| mordred | jeblair: as a final smoke test that the v3 release jobs work once we swich the secret to real pypi from test_pypi | 15:51 |
| EmilienM | mwhahaha: ^ jfyi | 15:51 |
| jeblair | mordred: the zuul-sphinx release is blocking further v3 progress; it will slow us down considerably. sure you want to do that? | 15:52 |
| pabelanger | mordred: yes, let me do that now | 15:52 |
| odyssey4me | mordred when do we expect the changeover to happen? | 15:53 |
| mordred | jeblair: no - I'm not - was mostly just a thought since pabelanger is about to push up a patch to update the pypi release secret - but I'm pretty confident in that code | 15:53 |
| mordred | odyssey4me: today is the plan | 15:53 |
| jeblair | mordred: okay, i'll use v2 then | 15:54 |
| mordred | jeblair: cool | 15:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix node request assignment order https://review.openstack.org/507182 | 15:56 |
| Shrews | pabelanger: jeblair: mordred: ^^^^ | 15:56 |
| *** jascott1 has joined #openstack-infra | 15:57 | |
| jeblair | mordred, pabelanger, Shrews: should we consider the node ordering issue a blocker? it should only affect nodesets of mixed types, none of which exist in v2. so we could say that since it only affects new functionality, we should fix it asap but not block migration on it... ...and i typed all of this before Shrews pushed up the patch. :) | 15:57 |
| *** bobh has joined #openstack-infra | 15:57 | |
| *** baoli_ has joined #openstack-infra | 15:58 | |
| pabelanger | jeblair: I'm okay with it not being a blocker, so far we only have 1 job that is mixing nodes. And doesn't actually break anything | 15:58 |
| *** ociuhandu has joined #openstack-infra | 15:58 | |
| Shrews | jeblair: U 2 SLOW | 15:58 |
| jeblair | it's all those letters! | 15:59 |
| *** baoli has quit IRC | 15:59 | |
| mordred | jeblair: I agree with all of the words, including "before Shrews pushed up the patch" | 16:00 |
| *** sbezverk has quit IRC | 16:00 | |
| clarkb | Shrews: reading that it doesn't seem to enforce order as much as label type, which is maybe equivalent in this case? | 16:00 |
| jeblair | i'm keeping https://etherpad.openstack.org/p/zuulv3-cutover up to date | 16:00 |
| Shrews | clarkb: correct | 16:00 |
| jeblair | clarkb: the requested labels are an ordered list | 16:01 |
| *** egonzalez has quit IRC | 16:01 | |
| jeblair | fungi, pabelanger: i'm replying to dhellmann on list with root cause | 16:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add pypi_secret for publishing to pypi https://review.openstack.org/507186 | 16:01 |
| jeblair | Shrews: ohnoes pep8 | 16:01 |
| Shrews | gah | 16:01 |
| clarkb | jeblair: ya but it could assign node 2 with label foo before assigning node 1 with label bar | 16:01 |
| *** jascott1 has quit IRC | 16:01 | |
| clarkb | which is fine for this case | 16:02 |
| Shrews | i did verify the new test though | 16:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix node request assignment order https://review.openstack.org/507182 | 16:03 |
| *** psachin has quit IRC | 16:03 | |
| *** bobh has quit IRC | 16:03 | |
| Shrews | clarkb: request looks like "trusty, xenial, xenial, centos", so we have to match to that label order | 16:03 |
| clarkb | right and assignment order could be xenial, centos, trusty, xenial but we'll map them back to the original requiest properly now | 16:04 |
| clarkb | +2 | 16:04 |
| fungi | thanks jeblair! | 16:06 |
| *** Dinesh_Bhor has quit IRC | 16:07 | |
| * fungi needs to step out for a lunch errand and then can review stuff | 16:09 | |
| *** Apoorva has joined #openstack-infra | 16:09 | |
| *** Apoorva has quit IRC | 16:09 | |
| *** jascott1 has joined #openstack-infra | 16:09 | |
| *** Apoorva has joined #openstack-infra | 16:10 | |
| *** jascott1 has quit IRC | 16:10 | |
| *** links has quit IRC | 16:10 | |
| mordred | jeblair: I'm writinga quick infra-manual patch to include a little more info about job migration | 16:12 |
| jeblair | mordred: cool | 16:14 |
| *** pcaruana has quit IRC | 16:15 | |
| *** ccamacho has joined #openstack-infra | 16:15 | |
| openstackgerrit | Trevor McCasland proposed openstack-infra/subunit2sql master: Add subunit2sql CLI option to use non_subunit_name https://review.openstack.org/507192 | 16:19 |
| *** ociuhandu has quit IRC | 16:20 | |
| *** jpich has quit IRC | 16:21 | |
| *** SumitNaiksatam has quit IRC | 16:23 | |
| inc0 | mordred: well, we pull our sources from tarballs.o.o so I don't think it's going to be affected | 16:27 |
| inc0 | also, zuul v3 is ready? Can we use secrets yet? | 16:30 |
| openstackgerrit | Reedip proposed openstack-infra/irc-meetings master: Proposal for change of FWaaS meeting https://review.openstack.org/507172 | 16:30 |
| *** jpena is now known as jpena|brb | 16:31 | |
| *** ralonsoh has quit IRC | 16:33 | |
| *** ccamacho has quit IRC | 16:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Add section with specifics about migrated job naming https://review.openstack.org/507200 | 16:38 |
| mordred | jeblair: ^^ | 16:38 |
| mordred | inc0: great! and we're working on getting migrated today - so very very soon | 16:38 |
| *** dizquierdo has quit IRC | 16:39 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix node request assignment order https://review.openstack.org/507182 | 16:39 |
| *** pblaho has quit IRC | 16:42 | |
| *** bhavik1 has joined #openstack-infra | 16:44 | |
| *** Swami has joined #openstack-infra | 16:47 | |
| *** bobh has joined #openstack-infra | 16:47 | |
| *** kjackal_ has quit IRC | 16:48 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:50 | |
| *** sree has joined #openstack-infra | 16:52 | |
| *** rhallisey has joined #openstack-infra | 16:53 | |
| jeblair | mordred: a mix of -1 and 0 comments on https://review.openstack.org/507200 . if you are busy and agree with those, i'll be happy to rework the change for you. | 16:53 |
| mordred | jeblair: I can fix | 16:54 |
| *** electrofelix has quit IRC | 16:56 | |
| jeblair | Shrews: do you want to restart the nodepool launchers, or shall i? | 16:57 |
| *** dtantsur|bbl is now known as dtantsur | 16:58 | |
| *** srobert has quit IRC | 16:58 | |
| *** derekh has quit IRC | 16:59 | |
| Shrews | jeblair: stepped out for lunch. You should | 17:01 |
| *** tosky has quit IRC | 17:01 | |
| mgagne | have the metric names changed for create server and friends? I can't find them in graphite for our provider http://grafana.openstack.org/dashboard/db/nodepool-inap | 17:02 |
| mgagne | is it now ComputePostServers ? (instead of ServerCreate) | 17:02 |
| fungi | back and catching up | 17:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Add section with specifics about migrated job naming https://review.openstack.org/507200 | 17:03 |
| fungi | also i initially saw ComputePostServers as CompuServ | 17:03 |
| mordred | mgagne: yes - it is not ComputePostServers (the name is now generated from service-type, http-method, url-path) | 17:04 |
| mgagne | now* | 17:04 |
| mordred | mgagne: s/not/now/ | 17:04 |
| mgagne | should I propose a change? | 17:04 |
| *** slaweq_ has joined #openstack-infra | 17:05 | |
| jeblair | Shrews, pabelanger: nodepool launchers restarted with ordering fix | 17:06 |
| mordred | mgagne: to the grafana dashboard! yes please - that would be great | 17:07 |
| mgagne | working on it | 17:07 |
| *** slaweq_ has quit IRC | 17:07 | |
| *** sbezverk has joined #openstack-infra | 17:09 | |
| *** masber has quit IRC | 17:10 | |
| *** chlong has quit IRC | 17:10 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config master: grafana: Update metric names for API operations timers https://review.openstack.org/507217 | 17:13 |
| clarkb | mordred: in the v3 migration doc change in the "zuul v3 manual i" is the 'i' an errant vim insert mode command? | 17:14 |
| *** caphrim007 has joined #openstack-infra | 17:15 | |
| mordred | clarkb: yup | 17:19 |
| clarkb | oh jeblair -1'd over it as I +2'd and said could cleanup in a followup :) in that case lets just get a new patchset up and merge it | 17:19 |
| *** hashar is now known as hasharDinner | 17:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Add section with specifics about migrated job naming https://review.openstack.org/507200 | 17:20 |
| *** slaweq_ has joined #openstack-infra | 17:20 | |
| *** camunoz has joined #openstack-infra | 17:20 | |
| jeblair | +3 | 17:20 |
| *** trown is now known as trown|lunch | 17:20 | |
| *** harlowja has joined #openstack-infra | 17:20 | |
| *** bhavik1 has quit IRC | 17:20 | |
| jeblair | mordred: i checked the output locally, and the addition of 'v3' to the first quick link is good; it doesn't force a line break or anything. | 17:21 |
| fungi | should i continue reviewing zuulv3 changes oldest-first, or do we have a formal priority list of changes for the cutover i should focus on reviewing? | 17:22 |
| jeblair | if we want to make a list, please put it on https://etherpad.openstack.org/p/zuulv3-cutover | 17:22 |
| *** iyamahat has joined #openstack-infra | 17:22 | |
| jeblair | (the only one there right now is the infra-manual change i just approved) | 17:23 |
| jeblair | i need to afk for ~30m | 17:23 |
| *** yamahata has joined #openstack-infra | 17:24 | |
| fungi | thanks! | 17:25 |
| *** tosky has joined #openstack-infra | 17:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 17:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove zuul v2 and jjb content https://review.openstack.org/507180 | 17:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 17:26 |
| *** kgiusti has quit IRC | 17:27 | |
| *** chlong has joined #openstack-infra | 17:29 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add section with specifics about migrated job naming https://review.openstack.org/507200 | 17:29 |
| *** jpena|brb is now known as jpena | 17:29 | |
| *** kgiusti has joined #openstack-infra | 17:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: grafana: Update metric names for API operations timers https://review.openstack.org/507217 | 17:31 |
| *** ijw has joined #openstack-infra | 17:35 | |
| *** baoli_ has quit IRC | 17:35 | |
| clarkb | mordred: is 507107 a noop? just refactoring? | 17:35 |
| *** kjackal_ has joined #openstack-infra | 17:36 | |
| *** baoli has joined #openstack-infra | 17:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Move single-use wheel-builder nodeset into job https://review.openstack.org/507177 | 17:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Split zuul.yaml into files in zuul.d https://review.openstack.org/507178 | 17:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add pypi_secret for publishing to pypi https://review.openstack.org/507186 | 17:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add the rest of the projects to Zuul v3 https://review.openstack.org/506878 | 17:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 17:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 17:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove zuul v2 and jjb content https://review.openstack.org/507180 | 17:38 |
| mordred | clarkb: yes - just a refactoring - should be a total no-op | 17:39 |
| pabelanger | +1, -59093 :D | 17:39 |
| fungi | clarkb: it seems to add one line in total, based on the summary. i'm trying to figure out where the extra line comes in (whitespace maybe?) | 17:39 |
| mordred | I re-organized that patch series so that we can land a few more of the patches before we start in to throwing trigger actions | 17:39 |
| fungi | ahh, it's adding comments at the start of each section, and dropping the blank lines in between the sections since they're now in separate files, so that explains why it adds one more line than it removes | 17:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 17:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add the rest of the projects to Zuul v3 https://review.openstack.org/506878 | 17:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 17:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove zuul v2 and jjb content https://review.openstack.org/507180 | 17:43 |
| mordred | (sorry - one more rebase - noticed two patches in swapped order) | 17:43 |
| mordred | fungi: aha! good | 17:43 |
| mordred | nevermind. I was wrong about that last rebase - the original order was correct | 17:44 |
| * mordred apologizes for patch spam | 17:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add the rest of the projects to Zuul v3 https://review.openstack.org/506878 | 17:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 17:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 17:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove zuul v2 and jjb content https://review.openstack.org/507180 | 17:45 |
| *** srobert has joined #openstack-infra | 17:49 | |
| *** amoralej is now known as amoralej|off | 17:50 | |
| *** Goneri has quit IRC | 17:50 | |
| *** slaweq_ has quit IRC | 17:52 | |
| clarkb | mordred: https://review.openstack.org/#/c/507178/ has a syntax error | 17:53 |
| *** tesseract has quit IRC | 17:53 | |
| clarkb | site_logs secret apparently isn't found | 17:54 |
| mordred | ah! excellent. | 17:54 |
| mordred | jeblair: ^^ potential issue with config loading and split files | 17:54 |
| mnaser | should "point status.openstack.org/zuul to zuulv3" be on the rollout steps too | 17:55 |
| mordred | mnaser: yah- although we should also put "make a themed version of v3 status page" onto the followup tasks | 17:55 |
| * mordred is making nodepool quota shift patch | 17:55 | |
| jeblair | mordred: back and catching up | 17:56 |
| mnaser | mordred: i'll leave it for someone to add, i can help with that sort of stuff after making sure puppet jobs are healthy | 17:56 |
| pabelanger | mnaser: ya, zuulv3.o.o will work for now, until we fix up status.o.o/zuul | 17:56 |
| jeblair | mordred: tbh, i'd rather drop the themed status page. folks just aren't interested in making that work. | 17:56 |
| jeblair | i'd rather we just have status.o.o point to zuul(v3).o.o | 17:57 |
| *** ihrachys has quit IRC | 17:57 | |
| jeblair | (and, you know, maybe add a provision for a logo or something) | 17:57 |
| *** ihrachys has joined #openstack-infra | 17:57 | |
| mnaser | why dont we keep the header and setup an iframe with pointing to zuulv3.o.o | 17:57 |
| * mnaser giggles | 17:57 | |
| jeblair | mnaser: you're in charge of iframes now. ;) | 17:57 |
| mnaser | terrific | 17:58 |
| *** felipemonteiro_ has quit IRC | 17:58 | |
| jeblair | mordred, clarkb: perhaps the secret issue is solved by https://review.openstack.org/505293 ? | 17:59 |
| mordred | jeblair: I betcha it is - perhaps we just need a scheduler restart to pick thatup? | 17:59 |
| jeblair | mordred: yeah, that's what i'm thinking. i'll do that. | 17:59 |
| mordred | cool | 17:59 |
| *** Goneri has joined #openstack-infra | 17:59 | |
| *** baoli has quit IRC | 18:00 | |
| *** sree has quit IRC | 18:00 | |
| *** sree has joined #openstack-infra | 18:00 | |
| *** jamesmcarthur has joined #openstack-infra | 18:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Shift nodepool quota from v2 to v3 https://review.openstack.org/507222 | 18:00 |
| jeblair | andreaf: ^ sorry need to restart zuulv3 | 18:00 |
| *** sree has quit IRC | 18:00 | |
| *** srobert_ has joined #openstack-infra | 18:01 | |
| *** baoli has joined #openstack-infra | 18:01 | |
| jeblair | pabelanger: can you make the filesystem layouts on all the executors match what's on ze01 please? | 18:02 |
| mnaser | hey um | 18:04 |
| *** srobert has quit IRC | 18:04 | |
| jeblair | pabelanger: specifically, i think the second partition needs to be mounted on /var/lib/zuul | 18:04 |
| mnaser | merge-check pipeline has 770 jobs in it | 18:04 |
| mnaser | is..that normal? | 18:04 |
| jeblair | pabelanger: also, it looks like ze01 doesn't have swap (but ze08) does; can you check the other executors and fix the ones that don't as well? | 18:04 |
| jeblair | pabelanger: it's okay if we have to reformat /dev/xvde | 18:05 |
| *** ijw has quit IRC | 18:05 | |
| *** ijw has joined #openstack-infra | 18:05 | |
| jeblair | mnaser: we stole 4 mergers from v2 to give to v3, so both are a little slow at merging right now | 18:05 |
| mnaser | ah okay no worries, just thought i'd point it out if it's something that shouldn't be happening | 18:06 |
| jeblair | mnaser: merge-check is low priority though, so shouldn't slow check or gate down. | 18:06 |
| *** Sukhdev has joined #openstack-infra | 18:07 | |
| *** sambetts is now known as sambetts|afk | 18:07 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Move fstrim to block device layer https://review.openstack.org/504579 | 18:08 |
| *** shamail has joined #openstack-infra | 18:09 | |
| shamail | Hi infra team! A previous meeting in openstack-meeting was not ended and prevents us from starting our meeting. Can anyone please assist us? | 18:09 |
| fungi | mnaser: also, every time a change merges to, say, nova we generate a new queue item for every single open change in nova. which i would not be surprised if it exceeded that number | 18:09 |
| shamail | nevermind, fixed ! | 18:10 |
| mnaser | ah gotcha | 18:10 |
| fungi | shamail: anyone can #endmeeting an hour after a meeting has been started | 18:10 |
| shamail | TIL | 18:10 |
| shamail | thanks fungi! | 18:10 |
| fungi | shamail: that's what the "due to finish in 60 minutes" output from the meetbot means | 18:10 |
| fungi | (the time after which anyone can end it without needing teh chair to do so) | 18:11 |
| *** rcernin has joined #openstack-infra | 18:11 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Unify apt-sources handling https://review.openstack.org/454456 | 18:11 |
| mordred | pabelanger: are you working on the filesystem thing? if not, I'm free to work on it | 18:11 |
| *** jpena is now known as jpena|off | 18:11 | |
| *** Sukhdev has quit IRC | 18:12 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Fix race condition when caching in debootstap https://review.openstack.org/454523 | 18:13 |
| *** rwsu has quit IRC | 18:13 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Docker matrix build https://review.openstack.org/414347 | 18:14 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Remove base element functionality https://review.openstack.org/444590 | 18:14 |
| jeblair | mordred: yah, let's assume pabelanger isn't here and take that | 18:14 |
| *** kgiusti has quit IRC | 18:14 | |
| jeblair | mordred: can you do ze08 first? | 18:14 |
| mordred | jeblair: yup. on it | 18:14 |
| *** wolverineav has quit IRC | 18:15 | |
| mordred | jeblair: 5-8 have swap, 1-4 do not | 18:15 |
| jeblair | mordred: when you're done with ze08, can you start on ze07? meanwhile, i'll run the script to warm the repos on ze08. once that's done, we can bring ze08 online, then take ze01 and ze02 offline and do all the rest at once. | 18:15 |
| fungi | we fixed/added swap setup between when those were built, if memory serves | 18:16 |
| jeblair | mordred: (to be clear, i'll start the cache warming when you finish ze08) | 18:16 |
| mordred | jeblair: yah. so - my thinking in general is "stop zuul-executor ; rm -rf /var/lib/zuul/* ; umount /opt/ ; mount /dev/blah /var/lib/zuul " | 18:17 |
| mnaser | mordred is it possible that your 'recheck' wasn't caught by zuul for the site_logs syntax error job? | 18:17 |
| mordred | jeblair: does that seems right-ish? | 18:17 |
| jeblair | mordred: yeah, though no executors should be running on anything except ze01 right now | 18:17 |
| mordred | zuul 1795 1 0 Sep12 ? 00:37:31 /usr/bin/python3 /usr/local/bin/zuul-executor | 18:17 |
| mordred | is on 08 | 18:18 |
| pabelanger | okay back, and catching up | 18:18 |
| mordred | it seems defucnt though | 18:18 |
| jeblair | mordred: weird | 18:18 |
| jeblair | mordred: actually can we modify your procedure? | 18:18 |
| mordred | so mostly just stopping / killing on general principle | 18:18 |
| mordred | jeblair: totally | 18:18 |
| mnaser | i dont see zuul responding to the recheck on this https://review.openstack.org/#/c/507178 | 18:18 |
| jeblair | mordred: can you move things instead of rming? | 18:18 |
| mnaser | neither do i see it in the status | 18:18 |
| mordred | jeblair: sure | 18:18 |
| jeblair | mordred: i think there's stuff in /var/lib/zuul that's installed by cfgmgt so would take time to repair | 18:18 |
| jeblair | mordred: might be good to mv /opt as well | 18:19 |
| mordred | jeblair: so move aside, remount - move back? | 18:19 |
| jeblair | mordred: ya | 18:19 |
| jeblair | mnaser: looking | 18:19 |
| mnaser | jeblair i just hit recheck again | 18:19 |
| mnaser | and it appeared | 18:19 |
| mnaser | so maybe the recheck was right in the middle of the restart.. or something. | 18:19 |
| mnaser | actually, i think i saw it pop up in status, disappear, and then no update in gerrit | 18:20 |
| jeblair | 2017-09-25 18:12:47,345 DEBUG zuul.IndependentPipelineManager: No jobs for change <Change 0x7f439b74d978 507178,2> | 18:20 |
| jeblair | 2017-09-25 18:12:47,345 DEBUG zuul.IndependentPipelineManager: Removing change <Change 0x7f439b74d978 507178,2> from queue | 18:20 |
| jeblair | mnaser: ^ | 18:20 |
| pabelanger | jeblair: re: swap: ze01-ze04 are missing, which make sense as they were the servers that were already online at PTG. We'll have to manually run make_swap.sh from system-config to fix them up. New servers no longer have this issue | 18:20 |
| jeblair | mnaser: oh that's project-config. we don't run any zuulv3 jobs on it yet | 18:20 |
| jeblair | mnaser: but we have zuul leaving -1's on syntax errors | 18:21 |
| mnaser | aaaah okay | 18:21 |
| *** slaweq_ has joined #openstack-infra | 18:21 | |
| jeblair | mnaser: it's an unusual pre-transition state | 18:21 |
| mnaser | sorry, didnt know that, thought it should be leaving a +1 at some point once it accepts the syntax, got it | 18:21 |
| jeblair | mnaser: jenkins will ignore that -1 though | 18:21 |
| *** ijw has quit IRC | 18:21 | |
| jeblair | mnaser: knowing whether it needs to do that is...complicated :) | 18:21 |
| *** dtantsur is now known as dtantsur|afk | 18:21 | |
| mnaser | yeah so complain if its broken but be quiet if everything is okay. seems fair | 18:21 |
| mnaser | i don't have much on hand right now, so be more than happy to help if there's anything :> | 18:23 |
| mordred | jeblair: 08 is done | 18:23 |
| jeblair | mordred: thanks, will start warming cache now | 18:23 |
| mordred | jeblair: 07 done | 18:24 |
| pabelanger | make_swap.sh ran on ze04.o.o, swap and /opt recreated | 18:25 |
| *** ijw has joined #openstack-infra | 18:25 | |
| pabelanger | looks like mordred is doing /var/lib/zuul change out | 18:25 |
| mordred | pabelanger: cool. can you do 03 too? | 18:25 |
| pabelanger | yup, 1 sec | 18:25 |
| mordred | pabelanger: then once 07 and 08 are up and we take 01 and 02 down we can do them too | 18:25 |
| pabelanger | swap and /opt created on ze03.o.o | 18:26 |
| jeblair | i went ahead and stopped 02 already; 01 should be only one running | 18:26 |
| mordred | jeblair, pabelanger: 06 and 05 done | 18:27 |
| clarkb | I don't think this stuff is on the cutover etherpad? curious why those nodes weren't booted iwth a swap in the first place | 18:27 |
| jeblair | clarkb: i added these items to blockers section at top | 18:28 |
| fungi | clarkb: they predate when swap setup was fixed | 18:28 |
| mordred | clarkb: it was a timing thing I believe - there was a bug in launch node that has since been fixed | 18:28 |
| pabelanger | yes | 18:28 |
| clarkb | ah ok | 18:28 |
| fungi | we can as a separate (non-migration-related) takeaway, audit existing systems in our ansible inventory for any missing swap setup | 18:28 |
| mnaser | https://review.openstack.org/#/c/507176/ -- looks like its not finding 'legacy-base' job? even though it exists in zuul.yaml ? | 18:29 |
| jeblair | cloning on ze08 is about 10% done | 18:29 |
| *** wolverineav has joined #openstack-infra | 18:30 | |
| *** jpena|off has quit IRC | 18:30 | |
| jeblair | gertty is not the best way to review that change. | 18:31 |
| *** ijw has quit IRC | 18:31 | |
| *** ijw has joined #openstack-infra | 18:31 | |
| *** amoralej|off has quit IRC | 18:32 | |
| *** pblaho has joined #openstack-infra | 18:32 | |
| clarkb | what is "filesystem layouts on executors"? is that the /opt move? | 18:33 |
| *** kgiusti has joined #openstack-infra | 18:33 | |
| mordred | yah | 18:33 |
| jeblair | mordred, mnaser: i guess 'jobs.yaml' sorts before '99legacy-jobs' | 18:35 |
| jeblair | mordred, mnaser: er, reverse that ^ | 18:35 |
| jeblair | mordred, mnaser: 99legacy-jobs sorts before jobs | 18:35 |
| mordred | jeblair: oh - piddle. wanna just acall it legacy-jobs then? | 18:35 |
| *** ltomasbo has quit IRC | 18:35 | |
| jeblair | mordred: that would work :) | 18:35 |
| mnaser | 00jobs.yaml? :> | 18:35 |
| jeblair | as long as it's not deprecated-jobs.yaml :) | 18:36 |
| clarkb | zuul-legacy-jobs for end of alphaybet sorting | 18:36 |
| *** ltomasbo has joined #openstack-infra | 18:37 | |
| *** jpena|off has joined #openstack-infra | 18:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 18:37 |
| pabelanger | mordred: ze02 swap and /opt setup again, if you want to move back to /var/lib/zuul | 18:37 |
| mordred | pabelanger: cool - thanks! | 18:37 |
| fungi | trying to wrap my head around the ordering issue there... these are parsed one-by-one rather than concatenated brfore parsing? | 18:38 |
| mordred | pabelanger, jeblair: 03 and 04 done | 18:38 |
| *** amoralej has joined #openstack-infra | 18:38 | |
| pabelanger | only ze01 left, but holding until we stop | 18:38 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Split zuul.yaml into separate files in zuul.d https://review.openstack.org/507107 | 18:38 |
| mordred | fungi: jobs parsing is order-dependent | 18:38 |
| jeblair | fungi: yes. we're going to change that, but after the migration, because it's not trivial. | 18:38 |
| fungi | got it | 18:38 |
| *** shamail has left #openstack-infra | 18:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Update links to legacy jobs https://review.openstack.org/507231 | 18:39 |
| mordred | that ^^ updates the docs to be consistnet with the rename patch | 18:39 |
| jeblair | ze08 cloning at 50% | 18:40 |
| *** slaweq_ has quit IRC | 18:41 | |
| mordred | jeblair: thereis nothing in /var/lib/zuul on ze02 | 18:41 |
| jeblair | mordred: that's... unfortunate | 18:41 |
| jeblair | mordred: ze02 had the volume mounted there | 18:42 |
| mnaser | https://review.openstack.org/#/c/507176/ now says "Unknown project openstack/ansible-hardening" but i assume that will get fixed once https://review.openstack.org/#/c/507179/ merges? | 18:42 |
| pabelanger | ze02 was reformated when I did make_swap, | 18:42 |
| mordred | yah. that ^^ | 18:42 |
| jeblair | welp, that host is going to need significant repair. | 18:42 |
| jeblair | i guess you can see if ansipuppet can fix it. | 18:43 |
| mordred | yah. lemme do that real quick | 18:43 |
| mordred | pabelanger: before we do swap on 01 - let's do the move-things-out-of-the-way step | 18:43 |
| pabelanger | sure, and appologies | 18:44 |
| pabelanger | apologies* | 18:44 |
| jeblair | no worries; it's a disposable host :) | 18:44 |
| mordred | yay cloud! | 18:44 |
| * mordred running puppet on ze01 | 18:44 | |
| mordred | gah | 18:44 |
| * mordred running puppet on ze02 | 18:44 | |
| fungi | tasty, tasty cattle | 18:44 |
| mordred | jeblair, pabelanger: puppet ran just fine - I believe ze02 is ready to go | 18:45 |
| *** caphrim007 has quit IRC | 18:46 | |
| pabelanger | great | 18:46 |
| *** caphrim007 has joined #openstack-infra | 18:46 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update links to legacy jobs https://review.openstack.org/507231 | 18:47 |
| mordred | pabelanger, clarkb, fungi, Shrews: https://review.openstack.org/#/c/507186 https://review.openstack.org/#/c/507178 and https://review.openstack.org/#/c/507177 should be good to land - the -1 on https://review.openstack.org/#/c/507178 is ok - there would be a new error message if it was still broken | 18:47 |
| *** shamail_ has joined #openstack-infra | 18:47 | |
| *** trown|lunch is now known as trown | 18:47 | |
| clarkb | mordred: will it not report back and +1 to clera the -1 on 178? | 18:48 |
| mordred | clarkb: normally yes - but we don't run any zuulv3 jobs on project-config - and the syntax checker only reports on failure | 18:48 |
| clarkb | ah | 18:48 |
| fungi | yeah, so it will only clear on a new patchset | 18:48 |
| mordred | it's not a situation that's expected in most circumstances :) | 18:49 |
| *** shamail_ is now known as shamail | 18:50 | |
| pabelanger | mordred: +2 | 18:50 |
| clarkb | I like how git decides that secrets.yaml is the new file content | 18:50 |
| jeblair | mordred: executor on ze08 is running, you can take down ze01 at will | 18:51 |
| jeblair | mordred: i'm clear to clone on ze02-ze07, rightL | 18:51 |
| jeblair | ? | 18:51 |
| clarkb | mordred: we are good to merge those ya? | 18:51 |
| mordred | clarkb: yah | 18:51 |
| *** shamail has quit IRC | 18:51 | |
| mordred | jeblair: taking down 01 | 18:51 |
| *** dhill_ has quit IRC | 18:53 | |
| *** dave-mccowan has quit IRC | 18:56 | |
| *** dhill_ has joined #openstack-infra | 18:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move single-use wheel-builder nodeset into job https://review.openstack.org/507177 | 18:57 |
| jeblair | ze02-ze07 and zm05-zm08 are all cloning all of the repos. | 19:00 |
| jeblair | just so folks know what to expect, the v3 startup time estimate is 2.5 minutes if we have 8 mergers and 8 executors; we only have 4 mergers (and 8 executors) at the moment, which pushes it out to a bit more than 3 minutes. | 19:01 |
| *** dhill_ has quit IRC | 19:02 | |
| jeblair | i promise it will be an exciting 3 minutes if you're watching the log file though. :) | 19:02 |
| *** dhill_ has joined #openstack-infra | 19:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Split zuul.yaml into files in zuul.d https://review.openstack.org/507178 | 19:02 |
| jeblair | (also, i'd like to emphasize the word 'estimate') | 19:03 |
| clarkb | just as long sa it doesn't try to copy all the data from one git repo to another we'll be fine ;) | 19:03 |
| mnaser | zuul first startup post cutover - https://i.giphy.com/media/cZHNk21INIlKo/giphy.webp | 19:04 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 19:04 |
| jeblair | neat, apparently i can't see webp | 19:05 |
| clarkb | I had to open in chrome | 19:05 |
| fungi | same here, no clue what format that is | 19:05 |
| clarkb | guessing it uses some non free video codec | 19:05 |
| jeblair | i guess gifs are not in gif format anymore :| | 19:05 |
| fungi | oh, it's a video format then? | 19:05 |
| openstackgerrit | Daniel Speichert proposed openstack-infra/shade master: Support filtering servers in list_servers using arbitrary parameters https://review.openstack.org/506969 | 19:05 |
| clarkb | jeblair: ya they use real video now because its smaller than gif for large animations | 19:05 |
| *** chlong has quit IRC | 19:06 | |
| fungi | (specifically niche video formats that need special browser support, i guess) | 19:06 |
| mordred | pabelanger: content on ze01 is moved out of the way if you wanna do swap | 19:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Add pypi_secret for publishing to pypi https://review.openstack.org/507186 | 19:06 |
| clarkb | h264 likely | 19:06 |
| pabelanger | mordred: sure | 19:07 |
| mordred | jeblair: while we're waiting on cache warming - you wanna make sure my steps listed at the bottom of the etherpad are ok - if you havne't already? | 19:07 |
| *** felipemonteiro has joined #openstack-infra | 19:08 | |
| jeblair | yes! and i'm also cooking grits | 19:08 |
| mordred | jeblair: awesome | 19:09 |
| *** srobert_ has quit IRC | 19:10 | |
| jlk | grits++ | 19:10 |
| fungi | are we okay with the fact that 507176 is getting a syntax error from zuul v3 until 507179 lands, even though the rollout steps have us merging them in the opposite order? | 19:10 |
| *** srobert has joined #openstack-infra | 19:10 | |
| mordred | fungi: looking | 19:11 |
| fungi | seems like that's not going to work out unless we bypass gating | 19:11 |
| mordred | fungi: it's not 507179 thats' bothering it - it's 506878 | 19:12 |
| *** e0ne has joined #openstack-infra | 19:12 | |
| mordred | fungi: it needs the projects to be defined in main.yaml before they can be referneced in config | 19:12 |
| *** dizquierdo has joined #openstack-infra | 19:13 | |
| *** srobert has quit IRC | 19:13 | |
| *** baoli has quit IRC | 19:13 | |
| mordred | fungi, clarkb, jeblair: where is http://status.openstack.org/zuul/ defined in apacheconfig? | 19:13 |
| *** srobert has joined #openstack-infra | 19:13 | |
| *** jascott1 has joined #openstack-infra | 19:13 | |
| mordred | it's not in 50-status.openstack.org.conf or the puppet module for status.o.o apache vhost | 19:14 |
| mnaser | afaik its in puppet-zuul when i wanted to clean that up once mordred | 19:14 |
| *** lifeless has quit IRC | 19:14 | |
| clarkb | mordred: that is puzzling | 19:15 |
| mnaser | https://github.com/openstack-infra/puppet-zuul/blob/master/templates/zuul.vhost.erb | 19:15 |
| mordred | oh! duh. we're not redirecting it | 19:15 |
| *** lifeless has joined #openstack-infra | 19:15 | |
| mnaser | thats the vhost though | 19:15 |
| mordred | we deploy the status html in v2 to /srv/static/status/zuul/ | 19:15 |
| clarkb | mordred: ah | 19:15 |
| pabelanger | mordred: Hmm, I think I need to reboot ze01, the kernel didn't like the repartition | 19:15 |
| clarkb | and then it does its own js callout to the status.json in browser | 19:15 |
| pabelanger | Error: Partition(s) 1 on /dev/xvde have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes. | 19:15 |
| jeblair | yeah, it's a bunch of static files in /system-config | 19:15 |
| mordred | pabelanger: fine by me | 19:16 |
| *** vhosakot has joined #openstack-infra | 19:16 | |
| pabelanger | k, rebooting ze01 | 19:16 |
| fungi | mordred: thanks, i was looking at the wrong project-config change | 19:16 |
| mordred | jeblair: so for now do we want to just put in a temp direct or have status.o.o/zuul do a proxy-pass to zuulv3.o.o ? | 19:17 |
| jeblair | cache warming is complete on all hosts except ze01 | 19:17 |
| jeblair | mordred: temp redirect? | 19:17 |
| *** dizquierdo has quit IRC | 19:18 | |
| rm_work | did devstack just drop keystone v2 by chance? O_o | 19:18 |
| clarkb | rm_work: I think that happened months ago? | 19:18 |
| mordred | rm_work: the keystone team is in the process of dropping v2 from master | 19:18 |
| rm_work | had a few gate runs where it can't find keystone, but seems like only v2 | 19:18 |
| rm_work | ah ok | 19:18 |
| clarkb | you had to opt in to v2 iirc | 19:18 |
| rm_work | just started today or over the weekend | 19:18 |
| clarkb | and ya now keystone itself is dropping it | 19:18 |
| rm_work | lol k | 19:19 |
| rm_work | so yeah that's it | 19:19 |
| rm_work | guess they finally did it | 19:19 |
| rm_work | thanks for the info :P | 19:19 |
| mordred | rm_work: it's actually not yet landed: https://review.openstack.org/#/c/504465 - but as clarkb said having v2 routes present in devstack is an opt-in currently - so it might be related to that | 19:20 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Enable Debug: True for all OVB jobs https://review.openstack.org/507241 | 19:20 |
| pabelanger | Hmm, ssh not coming back on ze01.o.o | 19:20 |
| pabelanger | seeing if I can get console log from webui | 19:20 |
| mordred | pabelanger: I agree - I am also not getting ssh | 19:20 |
| mordred | although it's pinging fine | 19:20 |
| pabelanger | ya, I too get ping | 19:20 |
| mordred | might be doing fsck or something | 19:20 |
| pabelanger | maybe | 19:20 |
| *** e0ne has quit IRC | 19:22 | |
| mordred | pabelanger: I'm getting nothingin web console, and of course it's rackspace so there's no nova console log | 19:24 |
| *** dave-mccowan has joined #openstack-infra | 19:24 | |
| pabelanger | I have console up now, I rebooted to watch | 19:24 |
| mordred | oh - although onw ssh isn't immediately failing | 19:24 |
| mordred | cool | 19:25 |
| Shrews | we have 1 held node, comment "Tracking down d-g issue" with no name. do we still need that? | 19:25 |
| pabelanger | A start job is running for vde1 / vde2 1min 30s, what it is currently on | 19:25 |
| mordred | Shrews: nope! | 19:25 |
| Shrews | mordred: marked for deletion | 19:25 |
| mordred | pabelanger: wow. it sure did boot into emergency mode | 19:26 |
| pabelanger | ya | 19:26 |
| fungi | couldn't find/fix the rootfs huh? | 19:26 |
| mordred | pabelanger: I vote we delete and make a new one | 19:26 |
| mordred | pabelanger: because cloud | 19:26 |
| fungi | yeah | 19:26 |
| pabelanger | I think it failed to find anything on xvde1 | 19:26 |
| clarkb | partition table went away? | 19:26 |
| pabelanger | wfm | 19:26 |
| fungi | i no longer trust that server | 19:26 |
| clarkb | ya rebuilding ++ | 19:26 |
| pabelanger | okay, let me delete and rebuild | 19:27 |
| Shrews | awww, i had grown attached to ze1 and named it and everything | 19:27 |
| mordred | pabelanger: ++ thanks! | 19:27 |
| fungi | even after repair i'm not sure i'd trust it as well as a fresh build | 19:27 |
| pabelanger | will this be a blocker for roll out? Cause we have 60minTTL on dns right now | 19:27 |
| jeblair | we could just call the replacement ze09 :) | 19:27 |
| mordred | I don't think so - we have 7 other executors | 19:27 |
| mordred | and yah - what jeblair said | 19:27 |
| jeblair | i also agree we can work with 7 for now if we want to fill in the hole and call it ze01 | 19:28 |
| jeblair | either wfm | 19:28 |
| clarkb | maybe defer on it just to simplify rollout? | 19:28 |
| clarkb | one less thing to owrry about | 19:29 |
| pabelanger | sure, if not a blocker | 19:29 |
| fungi | also deleting the dns records for it now allows the replacement to start working immediately later | 19:31 |
| jeblair | good news! i'm now done with the cache warmup! :) | 19:31 |
| fungi | heh ;) | 19:31 |
| pabelanger | fungi: +1 | 19:31 |
| SamYaple | ive made some syntax additions to bindep that I am basing alot of additional work on, I would love to get some reviews on it. if it doesnt merge ill need to go a completely different direction https://review.openstack.org/#/c/506502/ | 19:31 |
| mordred | jeblair: so - the status thing - I know of at least one person (me) that has a bookmark for status.openstack.org/zuul - but also I thought we'd talked about renaming zuulv3.o.o to just zuul.o.o once we've finished the transition ... | 19:31 |
| *** chlong has joined #openstack-infra | 19:31 | |
| jeblair | mordred: yes, we should rename it | 19:32 |
| clarkb | we just need to update the js at status.o.o to pull status.json from zuulv3 for now right? then switch it back when we rename? | 19:32 |
| *** dave-mccowan has quit IRC | 19:32 | |
| *** rwsu has joined #openstack-infra | 19:32 | |
| *** dave-mccowan has joined #openstack-infra | 19:32 | |
| pabelanger | clarkb: we also need to scope tennat into apache config for zuulv3 | 19:32 |
| jeblair | clarkb: i don't think the customized status page is viable in the long term | 19:33 |
| mordred | jeblair: so I was thinking a 302 rather than a 301 for now | 19:33 |
| jeblair | mordred: yes, temp redirect | 19:33 |
| clarkb | ah | 19:33 |
| clarkb | so run it more like the other pages on status.o.o rather than what we currently have | 19:33 |
| mordred | yah | 19:33 |
| *** baoli has joined #openstack-infra | 19:34 | |
| pabelanger | fungi: dns records for ze01 removed | 19:34 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: switch whereto to publish docs on docs.o.o https://review.openstack.org/507243 | 19:34 |
| *** slaweq_ has joined #openstack-infra | 19:36 | |
| fungi | also, the future state potentially allows us to drop status.openstack.org as a separate server instance and just make it a vhost on static.o.o or something | 19:36 |
| fungi | since there will be less and less smarts on it (i guess the elastic-recheck bits are the majority of the remaining complexity at that point?) | 19:36 |
| mnaser | swift staticweb anyone? :> | 19:37 |
| mtreinish | fungi: aren't there a bunch of vhosts on status.o.o? | 19:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add redirect from status.o.o/zuul to zuulv3.openstack.org https://review.openstack.org/507244 | 19:37 |
| mtreinish | fungi: the e-r stuff is static, it just updates on a cron | 19:37 |
| *** felipemonteiro_ has joined #openstack-infra | 19:37 | |
| fungi | mtreinish: no additional vhosts there, just status.o.o (same as teh server name itself) | 19:37 |
| mordred | clarkb, fungi, jeblair, pabelanger: ^^ there's redirect | 19:38 |
| *** slaweq_ has quit IRC | 19:38 | |
| fungi | mtreinish: i was thinking more that the irc bot is currently running from there too | 19:38 |
| mtreinish | fungi: I thought I added o-h as a separate vhost on the apache there | 19:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Zuul v3: honor previous votes from jenkins https://review.openstack.org/507247 | 19:38 |
| mtreinish | but it's been a while | 19:38 |
| *** slaweq_ has joined #openstack-infra | 19:38 | |
| jeblair | mordred, fungi, clarkb: the discussion on line 50 of ethermad made me think of that ^ | 19:39 |
| mordred | I added it to the rollout list - probably shoudn't do it until we've turned on jobs - or turned off v2 | 19:39 |
| fungi | mtreinish: health.o.o is a separate server | 19:39 |
| mordred | jeblair: ++ | 19:39 |
| *** baoli has quit IRC | 19:39 | |
| pabelanger | okay, AFK for a minute while I get new coffee | 19:39 |
| jeblair | mordred, fungi, clarkb: also, i'd like us to agree not to remove jenkins from the ci group so we don't lose any votes | 19:39 |
| mtreinish | no the frontend, but it wasn't a separate vhost | 19:39 |
| mtreinish | that's what I was remembering the original puppet module only worked as deploying a separate vhost | 19:40 |
| mtreinish | and I had to fix that to just deploy to a dir | 19:40 |
| fungi | jeblair: we wouldn't "lose" the votes, they just wouldn't be displayed or returned in api queries until it was readded to the group | 19:40 |
| *** baoli has joined #openstack-infra | 19:40 | |
| mtreinish | fungi: just ignore me :) | 19:40 |
| fungi | jeblair: but i agree on that point removing the account from that group is not something we want | 19:40 |
| jeblair | fungi: more like how i lose my glasses then | 19:40 |
| fungi | heh | 19:41 |
| jeblair | there's still there; i just can't see them | 19:41 |
| *** slaweq_ has quit IRC | 19:41 | |
| *** felipemonteiro has quit IRC | 19:41 | |
| *** baoli has quit IRC | 19:41 | |
| mordred | jeblair: ++ to not removing it | 19:42 |
| clarkb | oh wait gerrit doesn't display votes if made by a no longer member of a group? | 19:42 |
| *** baoli has joined #openstack-infra | 19:42 | |
| clarkb | ++ to not removing it then | 19:42 |
| clarkb | maybe we remove the ssh key from the account then or some other method of making it not work to fight zuul user | 19:42 |
| jeblair | to be fair, i have no idea what happens in 2.13. :) | 19:43 |
| fungi | clarkb: yeah, up to the point where the change is merged if memory serves | 19:43 |
| jeblair | but it certainly did that in some previous version | 19:43 |
| fungi | clarkb: at least behavior i've observed in the past is that your vote will be hidden on open changes if the acl no longer allows you to provide it, but once the change is in a closed state all votes are displayed instead | 19:43 |
| clarkb | I've approved 507247 | 19:43 |
| clarkb | fungi: gotcha | 19:43 |
| clarkb | seems like a good reason to not reove the account from the group then | 19:43 |
| fungi | so we _could_ probably remove jenkins from that group eventually, just not while we're still wanting its votes displayed on open changes | 19:44 |
| fungi | months/years from now | 19:44 |
| clarkb | I think updating the ssh key would effectively do the same thing as far as preventing jenkin suser from fighting zuul innadverdently though | 19:44 |
| clarkb | (we can sort that out later) | 19:45 |
| fungi | though we're also stopping zuul v2... we could just as well take extra steps to make sure it doesn't inadvertently get started again | 19:45 |
| clarkb | the reason I bring it up wsa when I was testing zuul against review-dev I had to really force the jenins account there to go away | 19:46 |
| jeblair | i'd be okay removing it after a couple weeks | 19:46 |
| *** slaweq_ has joined #openstack-infra | 19:46 | |
| fungi | until such time as we feel safe deleting the server (probably when we're ready to move/rebuild zuulv3.o.o to the old name) | 19:46 |
| clarkb | puppet kept restarting the service iirc | 19:46 |
| clarkb | something like that | 19:46 |
| pabelanger | coffvee obtained | 19:46 |
| mordred | yah - although probably the period where a rollback rather than a roll-forward-and-fix-problems remains viable is not going to be a super long period | 19:47 |
| mnaser | (maybe just physically shutdown and lock the vm?) | 19:48 |
| mnaser | so if someone was to start it, he'd have to consciously unlock the vm (and most automation won't handle that probably) | 19:48 |
| mordred | mostly just saying that we'll almost certainly want to keep the jenkins accont for longer than the safe-rollback period | 19:48 |
| *** hasharDinner is now known as hashar | 19:49 | |
| *** jdandrea_ has joined #openstack-infra | 19:49 | |
| mordred | jeblair, pabelanger, Shrews: nb01 and nb02 are not listed in the servers lists - are they are v3 builders? do we have v3 builders or just the one set of builders (nb03 and nb04) | 19:50 |
| Shrews | mordred: they're still running 2.5 afaik | 19:50 |
| mordred | Shrews: 03 and 04 are, yeah? | 19:51 |
| pabelanger | right, we need to rebuild them as xenial / python3 | 19:51 |
| mordred | and 01 and 02 don't actually exist currently | 19:51 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add timestamps to bootstrapping log https://review.openstack.org/503749 | 19:51 |
| pabelanger | right, nb01, 02 don't exist yet, and nb03 / nb04 are xenial python2 2.5 | 19:51 |
| mordred | so we'll spin up v3 versions on 01 and 02 after this and tear down 03 and 04? | 19:52 |
| pabelanger | Ya, on my list | 19:52 |
| mordred | cool | 19:52 |
| jeblair | mordred: yeah, it's not urgent -- v2 and v3 builders are the same | 19:52 |
| pabelanger | was hoping we'd merge master -> feature/zuulv3 and back then stand up nb01 / nb02 properly | 19:52 |
| mordred | pabelanger: if you still have dns stuff handy - nb01 and nb02 still have dns entries (that currently point to something that is not ours) | 19:53 |
| Shrews | 506969 seems to be stuck according to zuulv3.o.o. is that expected? | 19:53 |
| pabelanger | mordred: ya, I can delete quickly | 19:53 |
| mordred | Shrews: it's not something I expect | 19:54 |
| jeblair | that'll be because no executors are running | 19:54 |
| jeblair | i'll look at what happened to ze08 | 19:54 |
| fungi | certainly makes it tough to execute... things | 19:54 |
| *** e0ne has joined #openstack-infra | 19:55 | |
| jeblair | for one, i think zuulv3 firewall is out of date; checking | 19:55 |
| jeblair | what's the iptables persistent service called now? | 19:56 |
| pabelanger | netfilter | 19:57 |
| pabelanger | netfilter-persistent* | 19:57 |
| jeblair | -- Unit netfilter-persistent.service has failed. | 19:57 |
| jeblair | -- The result is failed. | 19:57 |
| jeblair | Sep 25 19:57:35 zuulv3 systemd[1]: netfilter-persistent.service: Unit entered failed state. | 19:58 |
| jeblair | Sep 25 19:57:35 zuulv3 systemd[1]: netfilter-persistent.service: Failed with result 'exit-code'. | 19:58 |
| jeblair | i think systemd may be trying to tell us it failed. | 19:58 |
| mordred | jeblair: that's awesome | 19:58 |
| jeblair | well, for one, it's going to be hard for it to add ze01's ip address now | 19:58 |
| *** hashar has quit IRC | 19:59 | |
| *** hashar has joined #openstack-infra | 19:59 | |
| jeblair | so ze01 needs to be removed from the server config, or we need to launch ze01 and add it to dns | 19:59 |
| jeblair | i estimate either to take about the same amount of time | 19:59 |
| pabelanger | okay, I'll start work on ze01 server | 19:59 |
| fungi | i can launch the replacement unless anyone's already on it | 19:59 |
| clarkb | oh because dns record was deleted | 19:59 |
| fungi | pabelanger: wins! | 19:59 |
| pabelanger | fungi: go for it :) | 20:00 |
| fungi | on it in that case | 20:00 |
| mordred | dns records were removed at 19:34 - so it'll be another 26 minutes for the old ttl to go away | 20:00 |
| jeblair | mordred: yeah, but that ttl has already expired for zuulv3.o.o | 20:01 |
| mordred | ah - excellent | 20:01 |
| jeblair | (which is the source of the problem) | 20:01 |
| * mordred understands all the things now | 20:01 | |
| fungi | 8gb performance flavor, xenial base image | 20:01 |
| fungi | judging from ze02 | 20:01 |
| pabelanger | fungi: yes | 20:01 |
| jeblair | fungi: ya | 20:01 |
| mordred | so as soon as rax allocates an IP we can go ahead and put it in dns | 20:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add ze05-ze08 to zuulv3 firewall https://review.openstack.org/507263 | 20:02 |
| jeblair | as it turns out, we need to modify the manifest anyway. we could remove ze01 in that patchset, but i say we just stay on course. | 20:02 |
| mordred | jeblair: may want to jst land https://review.openstack.org/#/c/502858/ | 20:03 |
| openstackgerrit | Merged openstack-infra/project-config master: Zuul v3: honor previous votes from jenkins https://review.openstack.org/507247 | 20:03 |
| mordred | jeblair: as it adds 05-08 to other things too | 20:03 |
| pabelanger | https://review.openstack.org/502858/ | 20:03 |
| pabelanger | ya, that | 20:03 |
| jeblair | mordred: yes, let's land that one 2 weeks ago. | 20:03 |
| mordred | jeblair: totally | 20:03 |
| pabelanger | Oh | 20:03 |
| pabelanger | needs https://review.openstack.org/502856 | 20:04 |
| mordred | oh - that one seems to depend on othe rhtings | 20:04 |
| mordred | le'ts land jeblair's | 20:04 |
| jeblair | i abandoned | 20:04 |
| pabelanger | ++ | 20:04 |
| mordred | oh - or, we can rebase the other one | 20:04 |
| fungi | instance is booting now | 20:04 |
| *** baoli has quit IRC | 20:04 | |
| jeblair | mordred: that works, i'm also okay with landing 856 too | 20:05 |
| mordred | awesome. +3 to the pabelanger stack | 20:05 |
| *** chlong has quit IRC | 20:05 | |
| mordred | pabelanger, fungi, jeblair: ze01 has ips now - want me to put them into the DNS? | 20:06 |
| jeblair | mordred: fungi will have handy scripts | 20:07 |
| mordred | nod. I'll stop being impatient :) | 20:07 |
| jeblair | mordred: (and we're waiting on the ze05-ze08 add to proceed anyway) | 20:07 |
| mordred | indeed | 20:07 |
| fungi | yeah, like give it ~3 more minutes and i'll just cut-n-paste api commands | 20:07 |
| fungi | maybe not even that long | 20:08 |
| jeblair | the other changes will merge as soon as nodepool v0 rustles up some trusty nodes | 20:08 |
| jeblair | i'll kick puppet as soon as they do | 20:10 |
| mnaser | does zuul v3 check zuul.yaml of all required-projects? ex: if puppet-nova has puppet-openstack-integration as required-project, could i have the base job in puppet-openstack-integration? | 20:10 |
| *** rcernin has quit IRC | 20:11 | |
| mnaser | im just trying to imagine how i can define a base job in puppet-openstack-integration for all puppet modules | 20:12 |
| clarkb | mnaser: I think jobs are global so yes. | 20:13 |
| *** esberglu has quit IRC | 20:13 | |
| *** baoli has joined #openstack-infra | 20:13 | |
| fungi | Sep 25 20:10:51 ze01 puppet-agent[16032]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known) | 20:14 |
| fungi | not sure if that's fatal | 20:14 |
| pabelanger | fungi: it is not | 20:14 |
| jeblair | mnaser: yes, a job defined in puppet-nova can inherit from p-o-i as long as puppet-openstack-integration is listed earlier in the config. if it isn't, and you need it to be, we can re-order. (this is a temporary situation until we rework some things post-migration.) | 20:14 |
| fungi | pabelanger: it does seem to delay the build by a couple minutes | 20:14 |
| *** jpena|off has quit IRC | 20:14 | |
| *** ltomasbo has quit IRC | 20:14 | |
| fungi | so i guess worth figuring out a fix at some point | 20:14 |
| jeblair | mnaser: (note that you also said 'required-projects' which isn't related to inheritance and doesn't require any special ordering; let me know if that's not clear) | 20:14 |
| pabelanger | fungi: agree | 20:15 |
| *** amoralej has quit IRC | 20:15 | |
| fungi | pabelanger: appears to raise that error message after a two-minute timeouty | 20:15 |
| mnaser | jeblair but that inheritance is assuming they both sit in the same repo.. i was imagining moving job definitions to puppet-openstack-integration (out of openstack-zuul-jobs) | 20:15 |
| jeblair | fungi: timeouty is your new hipster timeout-as-a-service startup? | 20:15 |
| fungi | i sure hope so | 20:15 |
| fungi | oof, happens more than once in fact | 20:15 |
| Shrews | how much timeoutiness will timeouty allow? | 20:15 |
| fungi | also we have this: Sep 25 20:13:50 ze01 openafs-client-precheck[5379]: modprobe: FATAL: Module openafs not found in directory /lib/modules/4.4.0-93-generic | 20:16 |
| jeblair | mnaser: no, inheritance doesn't require jobs be in the same repo | 20:16 |
| *** jamesmcarthur has quit IRC | 20:16 | |
| fungi | have a feeling we're missing a dependency somewhere in the manifest on the module build | 20:16 |
| *** esberglu_ has joined #openstack-infra | 20:16 | |
| *** jamesmcarthur has joined #openstack-infra | 20:17 | |
| jeblair | fungi: that may be a dkms-related projeblm | 20:17 |
| fungi | oh, or that may just be stderr from openafs-client-precheck | 20:17 |
| mnaser | jeblair so how does that logic come about (aka, do i have to tell zuul somewhere to check for project-templates etc in a specific repo?) or is it all ~magic~ | 20:17 |
| fungi | the closer i look at the log | 20:17 |
| pabelanger | fungi: possible https://review.openstack.org/506949/ could be the fix | 20:17 |
| *** ltomasbo has joined #openstack-infra | 20:17 | |
| fungi | counted at least three places the "Could not request certificate" error happens after about 2 minutes of silence in the build log | 20:17 |
| openstackgerrit | Merged openstack-infra/system-config master: Bump default image to xenial to launch-node.py https://review.openstack.org/502856 | 20:18 |
| *** amoralej has joined #openstack-infra | 20:18 | |
| *** jpena|off has joined #openstack-infra | 20:18 | |
| pabelanger | fungi: did ansible fail to provision server on openafs failure? | 20:18 |
| *** e0ne has quit IRC | 20:18 | |
| mordred | fungi: I thought at some point that put in something to prevent puppet-agent from starting | 20:18 |
| clarkb | mordred: fungi ya I think that happens after it installs the package | 20:18 |
| clarkb | since on xenial we install the puppet package from the distro | 20:18 |
| fungi | pabelanger: not sure yet, it's still starting services, but it does seem to have subsequently successfully started OpenAFS client. | 20:18 |
| jeblair | mnaser: project-templates also don't relate to inheritance -- that's a group of jobs that all get applied to a project's pipeline together | 20:18 |
| jeblair | mnaser: i may not be following what you're asking | 20:18 |
| openstackgerrit | Merged openstack-infra/system-config master: Support ze05.o.o to ze08.o.o https://review.openstack.org/502858 | 20:19 |
| *** dprince has quit IRC | 20:19 | |
| mnaser | jeblair: my end goal is: i want the playbooks that tests puppet-{nova,swift,etc} modules to sit in puppet-openstack-integration | 20:19 |
| fungi | there's at least the fourth lookup timeout on the certificate request now. i wonder if we're invoking puppet-agent a bunch of times and it does this each time? | 20:19 |
| mnaser | (puppet-openstack-integration repo that is) | 20:19 |
| mnaser | that way we can manage job playbooks in a single location and not need infra to approve changes to the job logic | 20:20 |
| mordred | mnaser, jeblair: the answer to "do i have to tell zuul somewhere to check for project-templates" is "no, it's just magic" | 20:20 |
| jeblair | mnaser: okay, you can define all those jobs there (a job's definition and playbook must be in the same repo) and you can use those jobs from any repo. | 20:21 |
| *** esberglu_ has quit IRC | 20:21 | |
| mnaser | okay gotcha :) that makes sense now, and i guess for jobs that are "useful" for the entire openstack ecosystem those will live in openstack-zuul-jobs | 20:21 |
| mnaser | for example, puppet-lint, etc | 20:21 |
| mordred | mnaser: yes. that's right | 20:21 |
| jeblair | yeah, it's a kind of gradient | 20:21 |
| mnaser | cool | 20:21 |
| mnaser | <insert magic gif> | 20:22 |
| fungi | okay, the build got past installing zuul so i expect we're nearly done | 20:22 |
| jeblair | okay, just waiting on the ze01 ip now, everything else is in place | 20:22 |
| mnaser | alright, more questions while we have some time.. if i want to use a role defined in zuul-jobs in the openstack-zuul-jobs repo (for example, revoke-sudo).. anyone want to point towards an example of this? :> | 20:23 |
| jeblair | mnaser: yes! https://docs.openstack.org/infra/manual/zuulv3.html#ansible-roles | 20:23 |
| mnaser | ok, i didnt see that doc, i was reading feature/zuulv3 branch docs | 20:24 |
| fungi | jeblair: 2001:4800:7818:104:be76:4eff:fe04:8d26 and 104.239.136.252 are now added into dns for the new server | 20:24 |
| fungi | i'm updating the host key on the puppetmaster | 20:24 |
| jeblair | mnaser: it's covered there too, but the infra-manual one is rather specifically targeted to us :) | 20:24 |
| mordred | mnaser: also: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/python-tarball/run.yaml uses the tox role which is defined in zuul-jobs - if you wanted to see an example in one of the jobs | 20:24 |
| mordred | fungi: woot! | 20:25 |
| fungi | i've also cleared the ansible inventory cache | 20:25 |
| jeblair | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.roles is the manual page | 20:25 |
| fungi | if someone else could log into the new ze01 and double-check it looks sane, that would be terrific | 20:25 |
| jeblair | mordred: ^ can you set up var/lib/zuul? | 20:26 |
| mordred | jeblair: looking | 20:26 |
| jeblair | 2017-09-25 19:55:19,880 DEBUG zuul.ExecutorServer: Waiting for server | 20:27 |
| jeblair | 2017-09-25 20:27:01,985 DEBUG zuul.ExecutorServer: Registering | 20:27 |
| *** jamesmcarthur has quit IRC | 20:27 | |
| jeblair | ze08 should be online now | 20:27 |
| pabelanger | cool | 20:27 |
| tonyb | frickler, dims, fungi: I'm workign on it. I have a tool that will do the mirroing but I was waitign for the zuulv3 transation before I looked at properly | 20:28 |
| mordred | jeblair: zuul-executor is running on 01 ... | 20:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Comment out server in puppet.conf https://review.openstack.org/507266 | 20:29 |
| jeblair | mordred: that explains that :) | 20:29 |
| *** kgiusti has left #openstack-infra | 20:29 | |
| *** florianf has quit IRC | 20:29 | |
| mordred | jeblair: is it ok for me to stop it to swap the disk things? | 20:29 |
| pabelanger | clarkb: mordred: fungi: ^ untested, but should fix dns lookup to puppet:8140 | 20:29 |
| tonyb | frickler, dims, fungi: I'm also working on gettign etcd 3.2.x into debian/ubuntu/Fedora so we don't need to do this dance for longer and required | 20:29 |
| *** Goneri has quit IRC | 20:29 | |
| pabelanger | tonyb: ++ | 20:29 |
| jeblair | mordred: yep. it's running a shade job but we can recheck. | 20:29 |
| *** esberglu has joined #openstack-infra | 20:29 | |
| pabelanger | mordred: fungi: clarkb: infact, puppet.conf is no longer managed by puppet, so most of the settings are now the defaults | 20:30 |
| tonyb | pabelanger: I figure once it's in those other distros have prior art to follow ;P | 20:30 |
| mordred | jeblair: disk is fixed on ze01 | 20:31 |
| clarkb | pabelanger: I think the problem is the distro package though | 20:31 |
| clarkb | we systemctl disable puppet but don't stop it | 20:31 |
| clarkb | I think what we really want to do is stop and disable the unit | 20:31 |
| jeblair | mordred: why does "ansible zl* -m shell -a 'hostname' -vvv" hang? | 20:31 |
| jeblair | mordred: on puppetmaster | 20:31 |
| mordred | jeblair: I don't know - looking (I was using that with ze* earlier and it worked fine) | 20:33 |
| jeblair | (i just cleared and accepted ze01's host key btw) | 20:33 |
| mordred | jeblair: thatdid not hang for me | 20:34 |
| jeblair | mordred: i agree it's fast now | 20:34 |
| jeblair | oh | 20:34 |
| mordred | jeblair: maybe you hit inventory cache expiration | 20:34 |
| jeblair | was it because of the inventory cache because we just launched a server? | 20:34 |
| mordred | yah- probably so | 20:34 |
| jeblair | i think we delete it | 20:34 |
| mordred | yah - so that it gets recreated next time | 20:34 |
| jeblair | okay, all good now, sorry | 20:35 |
| *** nicolasbock has quit IRC | 20:35 | |
| jeblair | i'm cloning repos on ze01 | 20:36 |
| jeblair | i think as long as we keep it offline until that's complete, we don't need to wait for it | 20:36 |
| mordred | ++ | 20:36 |
| jeblair | i know of no transition blockers now. | 20:36 |
| * jeblair sets console light to green | 20:37 | |
| * mordred either | 20:37 | |
| mordred | y'all ready to do this thing? | 20:37 |
| fungi | mordred: jeblair: yes, i manually deleted the inventory cache; we need to do that after any replacement of a server with another of the same name, so it doesn't get confused and whacked out | 20:37 |
| fungi | so that was probably it | 20:37 |
| fungi | it's just habit that i do that after building replacement servers | 20:37 |
| jeblair | fungi: oh, i thought that was automatic :) | 20:38 |
| jeblair | mordred: yes | 20:38 |
| fungi | it didn't used to be, and it's only necessary when building a replacement server afaik | 20:38 |
| fungi | since new servers will get added to the cache automagically | 20:38 |
| fungi | (i think?) | 20:39 |
| *** trown is now known as trown|outtypewww | 20:39 | |
| fungi | maybe not | 20:39 |
| jeblair | mordred: maybe we should check with the release team | 20:40 |
| jeblair | and then maybe start sending status notices | 20:40 |
| mordred | we should look in to the whole system now that 2.4 is out- between inventory plugins and the hiera lookup plugin we have new tools available to us | 20:40 |
| mordred | jeblair: ++ | 20:40 |
| dims | thanks tonyb | 20:40 |
| pabelanger | ready here! | 20:41 |
| mordred | jeblair, clarkb: unfortunately, I will have to AFK in 1 hour and 20 minutes - but I'm ready to go for the next 80 minutes | 20:42 |
| jeblair | mordred: maybe we should do this tomorrow? | 20:42 |
| mordred | jeblair: we could also do it first thing tomorrow since we're prepped | 20:42 |
| mordred | (by first thing, I mean when jeblair and clarkb wake up of course) | 20:43 |
| *** felipemonteiro_ has quit IRC | 20:43 | |
| fungi | worth noting, the odds of me losing electricity go up steeply around 16:00z tomorrow | 20:43 |
| *** felipemonteiro_ has joined #openstack-infra | 20:43 | |
| fungi | so i may or may not be around to help, storms permitting | 20:44 |
| clarkb | I'm lunching now but around after | 20:44 |
| fungi | i am however around for the rest of this evening if needed | 20:44 |
| pabelanger | same, I'm around for a while this evening | 20:44 |
| prometheanfire | is the pip mirror out of sync or something? I'm getting inconsistant results in our tox-py35-check-uc job https://review.openstack.org/507128 | 20:45 |
| jeblair | okay, maybe we have enough folks it's worth a shot, but with the rollback on a hair trigger? | 20:45 |
| fungi | wfm! | 20:45 |
| fungi | also we have mordred for another 75 minutes! | 20:45 |
| fungi | prometheanfire: bandersnatch is updating at its usual 5-minute cadence according to our logs | 20:46 |
| pabelanger | wfm | 20:46 |
| jeblair | mordred: you want to start approving things? | 20:46 |
| mordred | sure - I will now approve adding all the projects in the world to main.yaml | 20:46 |
| *** srobert has quit IRC | 20:46 | |
| mordred | +a has been hit - when that lands we might want to just pull it on zuulv3 and trigger a reconfigure rather than waiting on a puppet cycle | 20:47 |
| *** dprince has joined #openstack-infra | 20:47 | |
| * Shrews watches teh nodepoolz | 20:47 | |
| jeblair | status notice The infra team is bringing Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 20:47 |
| jeblair | how's that look ^? | 20:47 |
| mordred | jeblair: wfm | 20:47 |
| prometheanfire | fungi: ok, I'll keep watching | 20:47 |
| jeblair | mordred: that hasn't been updated with your changes :( | 20:48 |
| jeblair | mordred: they're sitting in post | 20:48 |
| mordred | boo | 20:48 |
| fungi | prometheanfire: oh, right, we update less frequently now (every 4 hours?) unless something gets released | 20:48 |
| *** chenying has joined #openstack-infra | 20:48 | |
| jeblair | mordred: oh wait! they just landed :) | 20:48 |
| mordred | \o/ | 20:48 |
| fungi | prometheanfire: er, rather, unless upper-constraints.txt changes | 20:49 |
| jeblair | #status notice The infra team is bringing Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 20:49 |
| openstackstatus | jeblair: sending notice | 20:49 |
| Shrews | are server launch errors in citycloud-sto2 a known thing? | 20:49 |
| prometheanfire | fungi: well, today was oslo release day it seems :P | 20:49 |
| fungi | prometheanfire: so last actual update timestamp can be found like http://mirror.dfw.rax.openstack.org/pypi/last-modified | 20:49 |
| fungi | ~1.5 hours ago | 20:50 |
| pabelanger | Shrews: possible, we've had issues from that cloud from time to time | 20:50 |
| prometheanfire | ok | 20:50 |
| -openstackstatus- NOTICE: The infra team is bringing Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 20:50 | |
| smcginnis | Happening in a kitchen somewhere right now: https://www.youtube.com/watch?v=y3ns6H5dHrU | 20:51 |
| *** chenying_ has quit IRC | 20:51 | |
| jeblair | that is what my fridge looks like | 20:51 |
| openstackstatus | jeblair: finished sending notice | 20:52 |
| *** liusheng has quit IRC | 20:52 | |
| pabelanger | Stay-Puft | 20:52 |
| fungi | prometheanfire: looking at the script more closely, we pull files from pypi every 5 minutes, and if any of them happen to appear in the constraints list then we update the published volume, otherwise we refrain from publishing so as not to wreak havoc on afs caches | 20:52 |
| *** liusheng has joined #openstack-infra | 20:53 | |
| mordred | waiting for project-config patch to land is not fun :) | 20:53 |
| *** srobert has joined #openstack-infra | 20:53 | |
| * fungi has some paint you can watch drying if you like | 20:54 | |
| jeblair | eta 3min | 20:54 |
| prometheanfire | fungi: the way the test is failing feels like there's a node out of sync or something, but like I said, I'll watch it | 20:55 |
| clarkb | prometheanfire: fungi you should be able to just check what is in afs either through the mirror or mounting afs directly | 20:56 |
| fungi | prometheanfire: got it. one of the reasons we use afs for this purpose is that all mirrors should see the exact same content at the same time. updates to the volume are atomic | 20:57 |
| *** jtomasek has quit IRC | 20:57 | |
| pabelanger | mordred: we also need to add tripleo-test-cloud-rh1 to nodepool-launchers. I can get a patch going now | 20:58 |
| mordred | pabelanger: ++ | 20:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Add the rest of the projects to Zuul v3 https://review.openstack.org/506878 | 20:59 |
| mnaser | woo | 20:59 |
| mordred | infra-root: "/opt/system-config/production/tools/kick.sh zuulv3.openstack.org" on puppetmaster - yeah? | 20:59 |
| *** mwarad has joined #openstack-infra | 20:59 | |
| pabelanger | ya, think so | 20:59 |
| clarkb | that looks right to me | 20:59 |
| fungi | yup | 20:59 |
| mordred | running kick | 20:59 |
| jeblair | now we find out if someone added a .zuul.yaml file ahead of schedule. :) | 20:59 |
| mordred | \o/ | 21:00 |
| mordred | won't THAT be fun :) | 21:00 |
| fungi | *edge of seat* | 21:00 |
| jeblair | 2017-09-25 21:00:23,088 DEBUG zuul.Scheduler: Performing reconfiguration | 21:00 |
| jeblair | oh, right, keypair generation is going to have us taking a bit longer this time too | 21:00 |
| mordred | and now all of the mergers go crazy | 21:00 |
| clarkb | jeblair: are the key pairs just on disk (so if we need to in the future we could pregenerate them?) | 21:01 |
| jeblair | clarkb: yes, and actually, we have a bunch from last time still | 21:01 |
| jeblair | (from the test) | 21:01 |
| jeblair | so probably not a big deal after all | 21:02 |
| mordred | jeblair: oh - we should have most of them from last time I'd guess | 21:02 |
| *** contra-bot has joined #openstack-infra | 21:02 | |
| *** mwarad has quit IRC | 21:03 | |
| *** contra-bot has quit IRC | 21:03 | |
| *** contra-bot has joined #openstack-infra | 21:03 | |
| *** contra-bot has quit IRC | 21:03 | |
| *** contra-bot has joined #openstack-infra | 21:04 | |
| jeblair | submitted 4932 cat jobs | 21:04 |
| *** srobert has quit IRC | 21:04 | |
| clarkb | thats projects * branches right? | 21:04 |
| mordred | aren't distributed systems neat? | 21:04 |
| mnaser | i see an openstack/nova job in zuulv3 :D | 21:04 |
| mnaser | but its probably just gonna noop out | 21:04 |
| jeblair | mnaser: it's not a real one yet :) | 21:04 |
| *** gouthamr has quit IRC | 21:04 | |
| clarkb | ya understanding that behavior was a real light bulb moment the other day | 21:05 |
| mordred | infra-root: https://review.openstack.org/#/c/507176 is the next change | 21:05 |
| mordred | as soon as the reconfigure is done I think we should recheck it (which should be the largest speculative job config change any zuul v3 will ever see) | 21:05 |
| *** contra-bot has quit IRC | 21:06 | |
| SpamapS | 13:59:10 openstackgerrit | Merged openstack-infra/project-config master: Add the rest of the projects to Zuul v3 https://review.openstack.org/506878 -- "At 13:59:10 Pacific time Zuulv3 became fully self aware... " | 21:06 |
| jeblair | http://paste.openstack.org/show/621875/ | 21:06 |
| fungi | mordred: here's hoping anyway! | 21:06 |
| fungi | i put nothing past our users | 21:06 |
| jeblair | reconfiguration failed | 21:06 |
| mordred | BOOOO | 21:06 |
| clarkb | did gerrit not like that? | 21:06 |
| jeblair | that happened on zm06.openstack.org | 21:07 |
| *** hashar has quit IRC | 21:07 | |
| fungi | now we're herding cat jobs ;) | 21:07 |
| SpamapS | We decided with zuulv3 to just admit that this is all for showing cat pics anyway. Cutting out the middleware. | 21:07 |
| jeblair | http://paste.openstack.org/show/621876/ | 21:08 |
| SpamapS | Is that maybe a mirror problem? | 21:08 |
| *** jascott1 has quit IRC | 21:08 | |
| mordred | jeblair: so we have a repo with no HEAD ? | 21:08 |
| clarkb | no thats a deb repo problem | 21:08 |
| mordred | SpamapS: there are some weird git repo states in those deb repos | 21:08 |
| clarkb | many of them didn't use master and gerrit/git don't understand repo inits without master HEADs | 21:08 |
| SpamapS | ah | 21:08 |
| mordred | clarkb, fungi: where did we get to with retiring them? | 21:08 |
| SpamapS | so you need to set the branch to something other than master | 21:09 |
| fungi | wendar got the initial retirement changes in i think | 21:09 |
| jeblair | zuul@zm06:/var/lib/zuul/git/git.openstack.org/openstack/deb-python-hplefthandclient$ git branch -a | 21:09 |
| jeblair | fatal: HEAD not found below refs/heads! | 21:09 |
| clarkb | mordred: I think tonyb and wendar were going to set things up to point at wherever packaging is now | 21:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Move tripleo-test-cloud-rh1 into nl01 https://review.openstack.org/507275 | 21:09 |
| fungi | SpamapS: except that gerrit doesn't like having non-master default branches, we've observed in the past | 21:09 |
| pabelanger | mordred: ^tripleo-test-cloud-rh1 | 21:09 |
| clarkb | the trivial fix here is to make a master branch iirc | 21:09 |
| jeblair | is it okay to remove them from zuulv3? or do we still need to merge changes to them? | 21:09 |
| mordred | fungi: she did? if so landing them and doing the migration minus all those repos ... | 21:10 |
| mordred | yah - what jeblair said | 21:10 |
| clarkb | jeblair: I think you can likely remove them for now | 21:10 |
| clarkb | but let me check if I can just create a master branch real quick | 21:10 |
| mordred | anybody know where the retire change is? | 21:10 |
| tonyb | clarkb: we did it for master not for stable/{newton,ocata,pike} | 21:10 |
| clarkb | https://review.openstack.org/#/admin/projects/openstack/deb-python-hplefthandclient,branches | 21:10 |
| clarkb | well that is neat | 21:10 |
| clarkb | it has two HEAD refs? | 21:10 |
| mnaser | i think it has a HEAD branch and a "HEAD" | 21:11 |
| mnaser | github mirror shows a HEAD branch | 21:11 |
| jeblair | oh dear | 21:11 |
| mordred | WOW | 21:11 |
| tonyb | :( | 21:11 |
| mnaser | it looks like its limited to that repo only though | 21:11 |
| mordred | so maybe just deleting the HEAD branch? | 21:11 |
| *** ijw has quit IRC | 21:11 | |
| pabelanger | darn, I thought we retired all the deb projects | 21:11 |
| clarkb | mordred: ya I think that may be the fix here | 21:11 |
| fungi | with prejudice | 21:12 |
| mnaser | https://review.openstack.org/#/admin/projects/openstack/deb-nova,branches seems ok | 21:12 |
| mnaser | i think deleting the HEAD branch might clear it out | 21:12 |
| *** ijw has joined #openstack-infra | 21:12 | |
| mordred | for the record, 48996b77c29f704262822d828c79d10e85b51a20 is the ref the HEAD branch is pointing to | 21:12 |
| clarkb | mordred: you going to delete it? | 21:12 |
| clarkb | (I'll stay out of the way if so) | 21:13 |
| mordred | yah. unless someone screams in the next 5 seconds | 21:13 |
| mordred | done | 21:13 |
| tonyb | NOOOOO! | 21:13 |
| tonyb | just kidding ;P | 21:13 |
| mordred | tonyb: too late. totally impossile to recreate :) | 21:13 |
| tonyb | :) | 21:13 |
| jeblair | okay i'll remove that from all the mergers/executors | 21:13 |
| mordred | jeblair: k. lemme know when done and I'll re-trigger the reconfigure | 21:14 |
| jeblair | mordred: done | 21:15 |
| mordred | I will now run: /etc/init.d/zuul-scheduler reload | 21:15 |
| mordred | done | 21:15 |
| jeblair | 2017-09-25 21:15:44,394 DEBUG zuul.Scheduler: Performing reconfiguration | 21:15 |
| mordred | woot | 21:15 |
| *** vhosakot has quit IRC | 21:16 | |
| SpamapS | how's memory usage btw? | 21:16 |
| * SpamapS is just curious | 21:16 | |
| *** jascott1 has joined #openstack-infra | 21:17 | |
| jeblair | SpamapS: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all | 21:17 |
| SpamapS | In theory with python3 we should get more effective GC cycles. | 21:17 |
| *** vhosakot has joined #openstack-infra | 21:17 | |
| mordred | I'm interested to see how well the console log streaming scales :) | 21:18 |
| mordred | Imean - I'm interested to see how ALL of it scales of course | 21:18 |
| pabelanger | indeed | 21:18 |
| SpamapS | mordred: yeah, the one I setup on this single VM is still having issues. | 21:18 |
| SpamapS | The threads aren't terminating. | 21:18 |
| SpamapS | I have to restart it once a day. | 21:18 |
| andreaf | mordred, jeblair: I think https://review.openstack.org/#/c/504246 should be ready - I hope I addressed your comments | 21:18 |
| SpamapS | have not had time to debug it. | 21:19 |
| Shrews | SpamapS: they won't if you have keep enabled (iirc) | 21:19 |
| jeblair | andreaf: awesome! i'll look as soon as things settle down :) | 21:19 |
| *** thorst_ has quit IRC | 21:19 | |
| SpamapS | Shrews: good to know. It hink I turned it off. I think. ;) | 21:19 |
| *** chlong has joined #openstack-infra | 21:20 | |
| mordred | jeblair: some day when we're bored, we should make a thing like the status page except for showing zuul reconfiguration status | 21:20 |
| mordred | jeblair: because I imagine watching the current event would be amazing | 21:21 |
| jeblair | 2017-09-25 21:21:02,997 DEBUG zuul.Scheduler: Reconfiguration complete | 21:21 |
| mordred | woot! | 21:21 |
| clarkb | woo | 21:21 |
| pabelanger | YAY | 21:21 |
| andreaf | jeblair: ok - cool | 21:21 |
| mordred | I have rechecked https://review.openstack.org/#/c/507176/2 | 21:21 |
| fungi | so just that one problem repo | 21:21 |
| fungi | excellent | 21:21 |
| mnaser | 🔥 | 21:21 |
| Shrews | SpamapS: reason being that the ansible log file never gets removed, so the finger streamer doesn't know it is done and should terminate | 21:21 |
| mordred | Shrews, SpamapS: I have some thoughts on ways to improve that for after rollout | 21:22 |
| jeblair | 6 minutes; my estimate was off by a factor of 2, so correctible with a scotty factor. | 21:22 |
| mnaser | The job "legacy-ec2-api-functional-neutron-dsvm-full" exceeds tenant max-job-timeout 10800. | 21:22 |
| mnaser | bump up build timeout for tenant to 11400 ? | 21:22 |
| mordred | NEAT! | 21:22 |
| andreaf | mordred: that's a nice one to review https://review.openstack.org/#/c/507176 :P | 21:22 |
| mordred | I've never seen this error before | 21:23 |
| mordred | The job "legacy-ec2-api-functional-neutron-dsvm-full" exceeds tenant | 21:23 |
| mordred | max-job-timeout 10800. | 21:23 |
| pabelanger | wow, that is a large build_timeout | 21:23 |
| mordred | we have several things with 11400 | 21:23 |
| mordred | I think just bump them back down to 10800 yeah? | 21:24 |
| jeblair | that's 190 mins -- so 3 hours with a 10 minute grace | 21:24 |
| jeblair | mordred: i think so | 21:24 |
| jeblair | or we can bump to 11400 | 21:24 |
| jeblair | <shrug> | 21:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:25 |
| mordred | let's shave the 10 minutes | 21:25 |
| jeblair | jeepers that's a long job though | 21:25 |
| mordred | yah | 21:25 |
| mnaser | lmao | 21:25 |
| mnaser | now there's another one | 21:25 |
| mnaser | with 14400 timeout | 21:25 |
| mnaser | :> | 21:25 |
| jeblair | this is just going to get worse and worse | 21:26 |
| mordred | another long one. yah | 21:26 |
| jeblair | 4 hours | 21:26 |
| mordred | lemme see if I can find anything with a timeout over 10800 real quck | 21:26 |
| pabelanger | Wow... | 21:26 |
| mordred | it'll be nice having that limit in place :) | 21:26 |
| mnaser | mordred 18000 | 21:26 |
| mnaser | just did a little shell-fu | 21:26 |
| mnaser | grep timeout zuul.d/zuul-legacy-jobs.yaml | cut -d':' -f2 | cut -d' ' -f2 | sort -n | 21:26 |
| pabelanger | 5 hours? | 21:27 |
| mordred | mnaser: neat! so 18000 and 14400 are the only two bad ones? | 21:27 |
| *** chlong has quit IRC | 21:27 | |
| *** jtomasek has joined #openstack-infra | 21:27 | |
| jeblair | the 14400 is a periodic job... we may want to consider pipeline based timout limits... | 21:27 |
| mordred | jeblair: agree | 21:27 |
| jeblair | if we wanted to say "2h is okay in gate, 6h is okay in periodic" | 21:27 |
| clarkb | ++ | 21:27 |
| mordred | altough that still eats quota | 21:27 |
| jeblair | ya, tradeoffs :| | 21:27 |
| mnaser | tripleo periodic jobs (four) seem to have the 18000 | 21:27 |
| clarkb | mordred: it does but for periodic jobs thats one set that starts rather than an infinite number | 21:28 |
| mnaser | legacy-periodic-tripleo-ci-centos-7-ovb-nonha-tempest-oooq-{master,ocata,pike} and legacy-periodic-tripleo-ci-centos-7-ovb-nonha-containers | 21:28 |
| clarkb | mordred: so still mostly manageable | 21:28 |
| *** chlong has joined #openstack-infra | 21:28 | |
| *** jdandrea_ has quit IRC | 21:28 | |
| pabelanger | mnaser: those are tripleo-test-cloud-rh1 only, and should be getting moved to 3rd party soon | 21:28 |
| jeblair | (also, low precedence) | 21:28 |
| *** jtomasek has quit IRC | 21:28 | |
| mnaser | pabelanger yep but likely zuul won't accept them because of tenant limits i assume | 21:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:29 |
| *** Goneri has joined #openstack-infra | 21:30 | |
| jeblair | 12600 | 21:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:30 |
| jeblair | zuul is a numbers station | 21:30 |
| pabelanger | :) | 21:31 |
| clarkb | so now we wait for 507176 to get cat'ed and run? | 21:31 |
| clarkb | oh looks like its done? | 21:31 |
| mnaser | Job legacy-tempest-dsvm-neutron-full in openstack-infra/devstack-gate is not permitted to shadow job legacy-tempest-dsvm-neutron-full in openstack-infra/openstack-zuul-jobs | 21:31 |
| mnaser | anoooother one | 21:31 |
| clarkb | zuul says syntax error, nice | 21:31 |
| mordred | oh. HAHAHAHA | 21:32 |
| mordred | that's my bad | 21:32 |
| jeblair | i guess we're done with that and can clean it up now, huh? :) | 21:32 |
| clarkb | indeed | 21:32 |
| jeblair | it was a test job we were using | 21:32 |
| pabelanger | ya | 21:32 |
| jeblair | this one's tricky | 21:32 |
| jeblair | it'll take several hours to merge that change to devstack-gate | 21:33 |
| jeblair | i think we should force-merge it after zuulv3 verifies it doesn't break the config | 21:33 |
| mordred | ++ | 21:33 |
| clarkb | wfm | 21:33 |
| * mordred make patch | 21:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Remove manually added legacy-tempest-dsvm-neutron-full https://review.openstack.org/507281 | 21:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:34 |
| pabelanger | -1 on 507281, missed job deleting | 21:35 |
| mordred | pabelanger: gah | 21:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Remove manually added legacy-tempest-dsvm-neutron-full https://review.openstack.org/507281 | 21:35 |
| jeblair | okay, zuulv3 is happy with it | 21:36 |
| *** jheroux has quit IRC | 21:36 | |
| mordred | jeblair: it's "branches" in v3 isn't it? | 21:37 |
| clarkb | looking at that the playbooks were in playbooks/legacy/tempest-dsvm-neutron-full but the job name is legacy-tempest-dsvm-neutron-full. Are -'s potentially /'s in the file pathing? | 21:37 |
| jeblair | mordred: i think so and it's the same in both? | 21:37 |
| clarkb | re 507281, I think you can force merge it now? | 21:38 |
| jeblair | clarkb: it was a manually specified path | 21:38 |
| jeblair | and yeah, i'll force-merge it now | 21:38 |
| clarkb | aha I see that now, thanks | 21:38 |
| *** kjackal_ has quit IRC | 21:38 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Remove manually added legacy-tempest-dsvm-neutron-full https://review.openstack.org/507281 | 21:38 |
| clarkb | do we recheck 507176 now? | 21:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:39 |
| clarkb | oh that should do it too :) | 21:39 |
| *** thorst has joined #openstack-infra | 21:39 | |
| jeblair | mordred: yes, branches: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.branches | 21:39 |
| mnaser | Job build-openstack-docs not defined | 21:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: It's branches, not branch https://review.openstack.org/507285 | 21:40 |
| mordred | there's the migration script update, just for completeness ^^ | 21:40 |
| Shrews | mordred: gah. that's probably my fault on the matchers change | 21:40 |
| Shrews | ah, that. thx | 21:40 |
| pabelanger | mordred: build-openstack-sphinx-docs now | 21:40 |
| clarkb | I've +2'd 7285 but not approved as I think approvals will be lost right now? | 21:41 |
| pabelanger | mnaser: ^ | 21:41 |
| mordred | yah. I thought we updated the mapping filefor that | 21:41 |
| pabelanger | same | 21:41 |
| clarkb | oh wait that is on zuul itself so should be safe to approve it | 21:41 |
| pabelanger | looking too | 21:41 |
| * clarkb approves | 21:41 | |
| mnaser | yeah i see build-openstack-sphinx-docs job | 21:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:42 |
| mnaser | :D | 21:43 |
| mnaser | syntax passed | 21:43 |
| mnaser | its running integration jobs | 21:43 |
| mordred | \o/ | 21:43 |
| *** dave-mccowan has quit IRC | 21:43 | |
| mordred | that's not bad, considering how many jobs we migrated :) | 21:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update mapping file for build-openstack-sphinx-docs https://review.openstack.org/507287 | 21:43 |
| mordred | just for completeness in case we have to do this again | 21:44 |
| jeblair | ++ | 21:44 |
| clarkb | and that is on project-config so +Workflow will get lost on that right now /me leaves as is | 21:44 |
| *** thorst has quit IRC | 21:45 | |
| *** Goneri has quit IRC | 21:45 | |
| mordred | once that lands, https://review.openstack.org/#/c/507219 is the next one | 21:46 |
| *** dprince has quit IRC | 21:46 | |
| mnaser | integration tests are starting to pass | 21:46 |
| pabelanger | that will be a good one to test migrating scripts too | 21:47 |
| mnaser | ansible lint is throwing a bunch of indexerrors | 21:47 |
| mnaser | http://zuulv3.openstack.org/static/stream.html?uuid=67d13cc9437240a4a16e8ccabd90c031&logfile=console.log | 21:47 |
| *** shardy has quit IRC | 21:47 | |
| pabelanger | I'm still unsure why we run base-itegration jobs on openstack-zuul-jobs, I don't see anything related to those jobs in openstack-zuul-jobs | 21:47 |
| clarkb | mordred: slight nit, would be nice if the template came before check and gate | 21:47 |
| clarkb | mordred: but we can clean that up later | 21:48 |
| mordred | oh - we might want to add a "hey, don't ansible-lint the legacy playbooks dir" | 21:48 |
| pabelanger | ++ | 21:48 |
| mordred | clarkb: yah - agree | 21:49 |
| *** armax has joined #openstack-infra | 21:49 | |
| clarkb | is 219 another force merge or do we turn zuulv2 off after that one merges? | 21:50 |
| clarkb | ah ya thats in the etherpad /me shuts up | 21:50 |
| mordred | I think we're gonna nee to re-submit 507176 with an exclusion added for the linters | 21:50 |
| clarkb | oh ya ansible-lint is unhappy | 21:51 |
| mordred | - bash -c "cd playbooks; find . -type f -regex '.*.y[a]?ml' -print0 | xargs -t -n1 -0 \ | 21:51 |
| mordred | + bash -c "cd playbooks; find . -type f -regex '.*.y[a]?ml' -print0 | grep -v legacy | xargs -t -n1 -0 \ | 21:51 |
| SpamapS | cool to see zuulv3 light up a bit more than usual! :) | 21:51 |
| mordred | that look ok? | 21:52 |
| clarkb | mordred: maybe legacy/ just to make sure it doesn't conflict with something legacy- ? | 21:52 |
| pabelanger | mordred: no, fails to run for me | 21:52 |
| mnaser | mordred -- id do grep -v '^\.\/legacy' just in case | 21:53 |
| mnaser | (which seems to generate a reasonable list for me) | 21:53 |
| clarkb | pabelanger: I think mordred left off the actual command bits of xargs is that why it failed for you? | 21:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:53 |
| mordred | ok. there's that ^^ | 21:53 |
| mordred | unfortunately I need to stand up and start moving away | 21:54 |
| fungi | thanks mordred!] | 21:54 |
| pabelanger | echo Binary file (standard input) matches | 21:54 |
| pabelanger | clarkb: ^what I get when I run it | 21:54 |
| mnaser | did you run that in the playbooks folder? | 21:54 |
| mnaser | sounds like it ran in /bin | 21:54 |
| SpamapS | grep -v in a find? how gauche | 21:54 |
| mordred | oh - right - you need to do it after xargs | 21:54 |
| mordred | anyway - reverting the one project-config patch adding to main.yaml is as far as we are at if people want to roll back and start again in the morning | 21:55 |
| clarkb | pabelanger: its because of hte print0 I think | 21:56 |
| clarkb | hrm grep --null doesn't help | 21:56 |
| SpamapS | mordred: find . -name '*.yaml' -a -not -path './legacy/*' | 21:56 |
| SpamapS | or whatever your regex is :) | 21:57 |
| mnaser | SpamapS nice. that works for me | 21:57 |
| clarkb | spamaps with a proper solution | 21:57 |
| *** rwsu has quit IRC | 21:57 | |
| clarkb | bash -c "cd playbooks; find . -type f -regex '.*.y[a]?ml' -not -path 'legacy/' -print0" | 21:58 |
| clarkb | that seems to work for me | 21:58 |
| SpamapS | yeah | 21:58 |
| mnaser | ill submit a change assuming mordred is out | 21:59 |
| clarkb | mnaser: go for it, you'll have to push an update to 176 | 21:59 |
| clarkb | I think | 21:59 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 21:59 |
| * SpamapS had to install GNU find on his mac to make it actually work right with regex. Silly BSD. | 22:00 | |
| pabelanger | find: warning: -path legacy/ will not match anything because it ends with /. | 22:00 |
| *** srobert has joined #openstack-infra | 22:00 | |
| pabelanger | clarkb: what I get | 22:00 |
| *** claudiub has quit IRC | 22:00 | |
| clarkb | pabelanger: weird, I totally get a listing that works locally | 22:00 |
| clarkb | pabelanger: maybe you need to quite it? | 22:00 |
| * mnaser will watch the tox-linters job | 22:00 | |
| clarkb | 'legacy/; | 22:00 |
| clarkb | er 'legacy/' I can type | 22:01 |
| mnaser | http://zuulv3.openstack.org/static/stream.html?uuid=eca61ac74d9e4877abb87d8e950262cc&logfile=console.log | 22:01 |
| mnaser | if anyone else enjoys watcvhing | 22:01 |
| pabelanger | ya, ^ fails for me on fedora | 22:02 |
| SpamapS | clarkb: I get what pabelanger gets with latest GNU find | 22:02 |
| SpamapS | clarkb: path wants globs | 22:02 |
| pabelanger | + bash -c "cd playbooks; find . -type f -regex '.*.y[a]?ml' -a -not -path './legacy/*' -print0 | xargs -t -n1 -0 \ | 22:02 |
| pabelanger | that works | 22:02 |
| SpamapS | what find are you running I wonder? | 22:02 |
| pabelanger | based on SpamapS original | 22:02 |
| mnaser | welp | 22:03 |
| mnaser | find: warning: -path legacy/ will not match anything because it ends with /. | 22:03 |
| pabelanger | mnaser: yup, that is the error I get | 22:03 |
| SpamapS | http://paste.openstack.org/show/621882 | 22:03 |
| clarkb | find (GNU findutils) 4.6.0 here | 22:03 |
| SpamapS | weird | 22:04 |
| SpamapS | same | 22:04 |
| clarkb | I didn't use the -a fwiw since its implied I think | 22:04 |
| *** slaweq_ has quit IRC | 22:04 | |
| clarkb | maybe that modifies the rule somehow? | 22:04 |
| mnaser | pabelanger can you confirm you -not -path './legacy/*' works for you pabelanger ? | 22:04 |
| pabelanger | http://paste.openstack.org/show/621883/ is my working locally | 22:04 |
| SpamapS | no | 22:04 |
| SpamapS | the -a is implied | 22:04 |
| *** SumitNaiksatam has quit IRC | 22:05 | |
| pabelanger | find (GNU findutils) 4.6.0 too | 22:05 |
| clarkb | ya oh I see I wasn't testing on mordreds change | 22:05 |
| SpamapS | but I wonder if your find uses a different thing to back -path than ours | 22:05 |
| clarkb | I get the warning its just at the beginning then I only get the things already in project-config | 22:05 |
| *** SumitNaiksatam has joined #openstack-infra | 22:05 | |
| SpamapS | ah ok yeah | 22:05 |
| pabelanger | confirmed, -a is not needed in my patch | 22:05 |
| clarkb | -path './legacy/*' fixes here | 22:05 |
| *** thorst has joined #openstack-infra | 22:06 | |
| SpamapS | yeah I just like to put -a in because it makes it read a little more programmatically. | 22:06 |
| mnaser | do we all agree to "-not -path './legacy/*'" ? | 22:06 |
| mnaser | ok, the quotes are confusing | 22:06 |
| mnaser | agree on: -not -path './legacy/*' | 22:06 |
| clarkb | mnaser: yes | 22:06 |
| clarkb | or at least I do :) | 22:06 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 22:06 |
| *** bgmccollum has joined #openstack-infra | 22:07 | |
| pabelanger | mnaser: line 36 is wrong | 22:07 |
| pabelanger | needs to be playbooks/legacy/* | 22:07 |
| pabelanger | cd into playbooks first | 22:07 |
| mnaser | oh good catch | 22:07 |
| pabelanger | cd playbooks; find . blah | 22:08 |
| mnaser | ill add the -0 | 22:08 |
| mnaser | for consistency sake | 22:08 |
| clarkb | sounds good | 22:08 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 22:08 |
| pabelanger | blarg | 22:10 |
| pabelanger | now test/invetory is not right | 22:10 |
| mnaser | hahaha | 22:10 |
| mnaser | damn | 22:10 |
| mnaser | should we drop cd playbooks from both | 22:10 |
| pabelanger | {toxinidir}/tests/inventory | 22:11 |
| clarkb | also, the things that ansible-lint was complaining about, they aren't fatal are they? | 22:11 |
| * Shrews stepping out for a bit to grab something for dinner. will check back in when i return | 22:11 | |
| clarkb | er fatal to ansible that is (and not the job) | 22:11 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 22:11 |
| mnaser | clarkb i dont think its a problem but it was taking forever to run | 22:12 |
| clarkb | ok, just wondering if all the zuul jobs will fail based on something that the linter would've caught | 22:12 |
| pabelanger | clarkb: ya, that is possible | 22:12 |
| fungi | takes gertty a solid minute just to load 507176 for me, for that matter | 22:12 |
| *** gouthamr has joined #openstack-infra | 22:13 | |
| mnaser | fungi for the first time i saw a "loading patchset" progress bar in gerrit for the first time :p | 22:13 |
| mnaser | http://zuulv3.openstack.org/static/stream.html?uuid=640afb9f6fd34bcdba9a165d04b03a8e&logfile=console.log if anyone wants to watch | 22:13 |
| clarkb | fungi: its is almost 200k lines changed | 22:13 |
| pabelanger | clarkb: infact, --syntax-check should work for legacy jobs | 22:14 |
| pabelanger | might be good to keep that | 22:14 |
| mnaser | sweet, it ran fine :) | 22:14 |
| pabelanger | well | 22:15 |
| clarkb | ya before we merge if we can get ^ running against legacy too that would probably be good | 22:15 |
| pabelanger | ansible-playbook --syntax-check -i /home/pabelanger/git/openstack-infra/openstack-zuul-jobs/tests/inventory ./legacy/python-saharaclient-pypy/run.yaml | 22:15 |
| pabelanger | that is a valid failure | 22:15 |
| clarkb | just to be sure we aren't going to merge a set of broken things | 22:15 |
| SpamapS | At the very least... valid yaml? ;) | 22:15 |
| pabelanger | ya, I am starting to think we keep syntax-check for legacy | 22:15 |
| pabelanger | cause I see 2 failures right now | 22:15 |
| mnaser | i think syntax check for legacy is a good idea | 22:16 |
| mnaser | should i bring syntax check back and leave linter out? | 22:16 |
| clarkb | mnaser: please | 22:16 |
| pabelanger | Ya, +1 to bring it back | 22:16 |
| clarkb | it might be slow but probably worthwhile for now | 22:16 |
| clarkb | especially if there are valid fails | 22:16 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 22:16 |
| pabelanger | Ya, /home/pabelanger/git/openstack-infra/openstack-zuul-jobs/playbooks/legacy/pypy/run.yaml has a single quote (') | 22:16 |
| pabelanger | which is breaking ansible | 22:17 |
| *** jpena|off has quit IRC | 22:17 | |
| clarkb | do we fix those individually in 176 or update the migration tool (or maybe both?) | 22:17 |
| openstackgerrit | Alex Schultz proposed openstack-infra/elastic-recheck master: Query for 1719469 https://review.openstack.org/507294 | 22:18 |
| *** amoralej has quit IRC | 22:18 | |
| pabelanger | clarkb: ya, fixing in project-config | 22:18 |
| jeblair | clarkb: i believe the tool expects code to be shell-embeddable, so morged previously fixed all the single quotes | 22:20 |
| pabelanger | clarkb: okay, project-config is correct. I think we need to run migrating tool again using latest version of project-config | 22:20 |
| mnaser | i have to run :( i'll drop back to see if i can help out later | 22:20 |
| clarkb | pabelanger: meaning the bug was in the source side of the migration and fixed since we last migrated? | 22:21 |
| jeblair | i just checked to see if mordred's fixup of the migration script have landed since we're talking about running it again | 22:22 |
| jeblair | and 507285 ended up with a whole mess of post-failures | 22:22 |
| pabelanger | clarkb: https://review.openstack.org/504965/ was the fix in project-config, but doesn't seem we ran the migration with that change | 22:22 |
| *** jpena|off has joined #openstack-infra | 22:23 | |
| clarkb | oh right this bug again | 22:23 |
| clarkb | ugh | 22:23 |
| clarkb | ok | 22:23 |
| *** amoralej has joined #openstack-infra | 22:23 | |
| jeblair | i'm looking into post-fails | 22:23 |
| clarkb | jeblair: ok | 22:23 |
| *** ssbarnea has quit IRC | 22:24 | |
| jeblair | the post playbook timed out | 22:25 |
| *** sai has quit IRC | 22:25 | |
| jeblair | i'm guessing it's because of host verification | 22:25 |
| jeblair | oh hrm | 22:25 |
| *** sai has joined #openstack-infra | 22:25 | |
| jeblair | that doesn't make sense because results were uploaded | 22:25 |
| jeblair | oh! | 22:25 |
| jeblair | it's probably logstash job submission | 22:26 |
| clarkb | firewall rules? | 22:26 |
| jeblair | ya | 22:26 |
| jeblair | i'll restart there | 22:26 |
| clarkb | ok | 22:26 |
| *** wolverineav has quit IRC | 22:28 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Initial skeleton for devstack-tempest base job https://review.openstack.org/504246 | 22:28 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 22:29 |
| *** iyamahat has quit IRC | 22:29 | |
| *** yamahata has quit IRC | 22:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Remove logstash-worker15 -- 20 https://review.openstack.org/507296 | 22:30 |
| jeblair | clarkb, pabelanger: ^ | 22:30 |
| pabelanger | +2 | 22:30 |
| pabelanger | running zuul-migrate locally to see if results in single quote fixes | 22:31 |
| jeblair | er, commit message is wrong, sorry, should be 16-20 | 22:31 |
| *** ssbarnea has joined #openstack-infra | 22:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Remove logstash-worker16 -- 20 https://review.openstack.org/507296 | 22:31 |
| jeblair | pabelanger: ^ | 22:31 |
| clarkb | +3'd | 22:32 |
| fungi | i should just run gertty somewhere with more ram | 22:32 |
| clarkb | er I guess just +2'd | 22:32 |
| clarkb | I'll go ahead and approve too since its just a commit message change | 22:32 |
| fungi | 507176 is slowing it down so much i'm still waiting for 507296 to load | 22:32 |
| jeblair | fungi: i don't think ram will help | 22:33 |
| fungi | oh :( | 22:33 |
| jeblair | fungi: i did initial review using the web ui. but now i'm just diffing patchsets. that's almost workable. (just keep the diff view up and hit p to change patchsets when they're uploaded) | 22:33 |
| clarkb | ']' to move to next file too | 22:34 |
| pabelanger | Hmm, I don't see playbooks/legacy/pypy/run.yaml in http://logs.openstack.org/85/507285/1/check/zuul-migrate/55cfb09/playbooks/legacy/ | 22:36 |
| pabelanger | why would that be? | 22:36 |
| clarkb | might not hvae ended up in the find listing for some reason? | 22:37 |
| pabelanger | if so, they how did https://review.openstack.org/507176/ add it? | 22:38 |
| pabelanger | unless I am running the incorrect commands | 22:38 |
| *** lin_yang has joined #openstack-infra | 22:43 | |
| *** baoli has quit IRC | 22:43 | |
| *** dave-mccowan has joined #openstack-infra | 22:44 | |
| pabelanger | okay, ansible 2.4 is getting pulled into tox -elinters, so we need to cap that too | 22:44 |
| pabelanger | I cannot figure out the syntax mordred used to generate legacy playbooks for 507285, but going to manually fix locally and see if that is the only issue (single quote) | 22:45 |
| mwhahaha | did logs.o.o die? | 22:47 |
| *** armax has quit IRC | 22:47 | |
| jeblair | oh, if it did that would explain the post failures | 22:48 |
| clarkb | mwhahaha: it certainly doesn't ping | 22:48 |
| fungi | eek | 22:48 |
| mwhahaha | RIP | 22:48 |
| fungi | wow, yeah static.o.o is unresponsive | 22:49 |
| fungi | going to the oob console now | 22:49 |
| fungi | no open tickets for that tenant | 22:50 |
| fungi | server status claims to be active | 22:52 |
| clarkb | firewall update maybe? | 22:52 |
| pabelanger | I can SSH to static.o.o | 22:52 |
| pabelanger | and ping | 22:52 |
| fungi | a ton of xen_netfront xennet skb->len 65538 too big for wire format errors in the console | 22:52 |
| fungi | but login prompt is responsive | 22:53 |
| pabelanger | fungi: ssh should be working again | 22:53 |
| fungi | that's disconcerting | 22:53 |
| ianw|pto | fungi: fyi those hvae been there for quite a while, i noticed them last week i think | 22:54 |
| fungi | k, thanks | 22:54 |
| clarkb | upstream networking issues maybe? | 22:54 |
| fungi | so far nothing out of the ordinary on cacti traffic graphs | 22:54 |
| fungi | i caught the 5-minute load average over 5 and it's dropping now | 22:55 |
| fungi | eclouds? | 22:55 |
| mwhahaha | looks like it's back | 22:55 |
| clarkb | or maybe the live mirated us or something | 22:56 |
| pabelanger | still running syntax-check locally | 22:56 |
| pabelanger | might have to parallel it to speed things up | 22:56 |
| fungi | yeah, looks like cacti has a current gap in polling that host and hasn't caught it back online again yet, but i don't see anything in the samples leading up to that to suggest anything unusual was going on before it became unreachable | 22:57 |
| pabelanger | so far, failures are single quote related | 22:57 |
| *** rbrndt has quit IRC | 22:58 | |
| *** aeng has quit IRC | 23:00 | |
| clarkb | jeblair: where are we on getting the migration script fix(es) in? just waiting for jobs to complete? | 23:03 |
| *** xarses_ has quit IRC | 23:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Cap ansible<2.4.0 for linters job https://review.openstack.org/507300 | 23:04 |
| pabelanger | clarkb: jeblair:^ will be needed | 23:05 |
| *** dave-mcc_ has joined #openstack-infra | 23:05 | |
| pabelanger | right now, I am stuck on how legacy playbooks in 507285 were generated | 23:05 |
| clarkb | pabelanger: using the migration script output I thought | 23:06 |
| pabelanger | ya, I ran it locally, but didn't get same output | 23:06 |
| pabelanger | http://logs.openstack.org/85/507285/1/check/zuul-migrate/55cfb09/playbooks/legacy/ is the output I got | 23:06 |
| pabelanger | but playbooks in 507285 are missing | 23:07 |
| pabelanger | eg: legacy/pypy/run.yaml | 23:07 |
| *** dave-mccowan has quit IRC | 23:07 | |
| clarkb | ah gotcha | 23:07 |
| jeblair | clarkb: logstash fix, then restart, then recheck the migration fixes | 23:08 |
| clarkb | pabelanger: for pypy specifically there was a thread about removing it entirely | 23:08 |
| jeblair | clarkb: logstash fix is gaing now | 23:08 |
| clarkb | pabelanger: its possible that all got in and now we don't have any legacy pypy jobs to manage? | 23:08 |
| pabelanger | clarkb: Oh, so maybe the failures we are seeing are on deleted playbooks? | 23:08 |
| clarkb | pabelanger: at least for that particular one | 23:08 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove logstash-worker16 -- 20 https://review.openstack.org/507296 | 23:09 |
| jeblair | i'd like to eod in ~ hour; it may be worth thinking about whether our goal is still to cutover today, or just to prep for tomorrow. | 23:09 |
| clarkb | ya | 23:09 |
| clarkb | I too will be cooking dinner and doing family stuffsoon | 23:09 |
| clarkb | as mordred says its just a single change to revert at this point I think | 23:10 |
| pabelanger | +1 | 23:10 |
| pabelanger | tomorrow seems likely now | 23:10 |
| fungi | i'm cool for pushing a little further with the expectation of reverting and starting fresh when people are around | 23:10 |
| jeblair | yep. i suggest we keep working on getting 507176 ready to merge, and then revert 506878 | 23:10 |
| clarkb | ++ | 23:10 |
| jeblair | i'm going to kick logstash.o.o now | 23:11 |
| fungi | mildly concerned that if we go into production and all eod, then i'll be the one the emea peeps are waiting to ambush first thing tomorrow ;) | 23:11 |
| fungi | but this way we have a great head start on resuming tomorrow | 23:11 |
| *** felipemonteiro_ has quit IRC | 23:12 | |
| * persia is only observing, but also believes that restarting tomorrow will result in a happier transition | 23:13 | |
| pabelanger | jeblair: clarkb: okay, so I am thinking that propose-openstack-manuals-updates/run.yaml, pypy/run.yaml, python-saharaclient-pypy/run.yaml are actually stale playbooks. Since they only contain the single quote issue | 23:14 |
| pabelanger | while other playbooks that use the same macro, are actually correct | 23:14 |
| pabelanger | zaqar-tox-integration/run.yaml for exmaple | 23:14 |
| pabelanger | example* | 23:14 |
| jeblair | pabelanger: oh, so you think mordred didn't clear out the directory when re-running the migration script between patchsets? | 23:15 |
| clarkb | ah | 23:15 |
| pabelanger | jeblair: ya, that is my best guess right now | 23:15 |
| jeblair | i've added the 3 changes we need to land in order to re-run the migration script to the "Changes" section of the etherpad | 23:16 |
| jeblair | lines 18-20 | 23:16 |
| jeblair | logstash firewall restarted; i'll recheck those now | 23:16 |
| pabelanger | http://paste.openstack.org/show/621887/ is the diff I get now if I first delete legacy folder then re-run migrate locally | 23:17 |
| jeblair | okay, all 3 of those are set to merge now | 23:17 |
| *** aeng has joined #openstack-infra | 23:18 | |
| jeblair | pabelanger: wow that's a big diff? | 23:18 |
| jeblair | pabelanger: are you running with all 3 of those changes applied locally? | 23:18 |
| pabelanger | jeblair: Oh, maybe not | 23:19 |
| pabelanger | let me do that | 23:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Revert "Add the rest of the projects to Zuul v3" https://review.openstack.org/507301 | 23:22 |
| pabelanger | a little less, but not by much: http://paste.openstack.org/show/621888/ | 23:25 |
| pabelanger | that is the latest master of project-config, so possible we merged some changes today | 23:25 |
| pabelanger | I need to step away here for a bit, but should return shortly | 23:26 |
| *** jascott1_ has joined #openstack-infra | 23:27 | |
| jeblair | okay that's pretty concerning. we may need to see what mordred says. | 23:27 |
| *** bobh has quit IRC | 23:27 | |
| *** jascott1 has quit IRC | 23:28 | |
| jeblair | unless you or anyone else wants to dig into that discrepancy, i think the only other outstanding things are those 3 changes. i'll continue to shepherd them in. | 23:29 |
| *** hongbin has quit IRC | 23:29 | |
| *** tosky has quit IRC | 23:30 | |
| clarkb | looks like it did delete the pypi file at least? | 23:30 |
| clarkb | er pypy | 23:30 |
| jeblair | meanwhile, i think we should merge the revert now, let release know we're suspending activity, and then maybe send an email letting folks know we're going to try again tomorrow and project-config is frozen. | 23:30 |
| clarkb | wfm reviewing the revert now | 23:31 |
| jeblair | i'll let release know and draft an email | 23:31 |
| clarkb | I'll go ahead and approve the revert as well since it already has fungi's +2 | 23:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Update mapping file for build-openstack-sphinx-docs https://review.openstack.org/507287 | 23:32 |
| *** vhosakot has quit IRC | 23:34 | |
| fungi | i saw AJaeger_ mention something earlier today on the pypi job removal thread for the -dev ml | 23:35 |
| jeblair | #status log project-config is frozen until further notice for the zuul v3 transition; please don't approve any changes without discussion with folks familiar with the migration in #openstack-infra | 23:37 |
| openstackstatus | jeblair: finished logging | 23:37 |
| jeblair | infra-root, infra-core, project-config-core, AJaeger_: ^ | 23:37 |
| SpamapS | so close :) | 23:38 |
| *** sdague has quit IRC | 23:39 | |
| fungi | http://lists.openstack.org/pipermail/openstack-dev/2017-September/122575.html | 23:39 |
| fungi | doesn't look like 504748 has merged yet anyway | 23:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Add the rest of the projects to Zuul v3" https://review.openstack.org/507301 | 23:43 |
| clarkb | there it is | 23:44 |
| clarkb | do we need to run kick.sh against zuulv3.o.o now? | 23:45 |
| jeblair | clarkb: we could, or just leave it to ansipuppet | 23:46 |
| jeblair | how does http://paste.openstack.org/show/621889/ look ? | 23:46 |
| clarkb | 0000UTC run of ansipuppet will pick that up so will happen soonish | 23:47 |
| clarkb | jeblair: lgtm | 23:47 |
| fungi | jeblair: perfect. fire at will | 23:48 |
| *** masber has joined #openstack-infra | 23:48 | |
| jeblair | sent | 23:48 |
| jeblair | ugh, i'm afraid 163 is going to fail tests due to flakiness/slow node | 23:49 |
| *** aeng has quit IRC | 23:51 | |
| *** aeng has joined #openstack-infra | 23:51 | |
| *** stakeda has joined #openstack-infra | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
