| *** esikachev has quit IRC | 00:01 | |
| *** ijw has quit IRC | 00:01 | |
| *** gouthamr has quit IRC | 00:03 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Automatically read both voting and non-voting forms of job https://review.openstack.org/453806 | 00:08 |
|---|---|---|
| *** Apoorva_ has joined #openstack-infra | 00:12 | |
| *** thorst has quit IRC | 00:12 | |
| *** dkehn has quit IRC | 00:12 | |
| *** tosky has quit IRC | 00:13 | |
| *** adriant has joined #openstack-infra | 00:14 | |
| *** wolverineav has quit IRC | 00:15 | |
| *** Apoorva has quit IRC | 00:16 | |
| *** Apoorva_ has quit IRC | 00:16 | |
| *** esikachev has joined #openstack-infra | 00:17 | |
| *** john-davidge has joined #openstack-infra | 00:17 | |
| *** rlandy has quit IRC | 00:17 | |
| *** sflanigan has joined #openstack-infra | 00:17 | |
| *** sflanigan has joined #openstack-infra | 00:17 | |
| *** jerryz has joined #openstack-infra | 00:18 | |
| jerryz | clarkb: ping | 00:18 |
| clarkb | jerryz: hi | 00:18 |
| jerryz | clarkb: i just reinstalled zuul 2.5.1 version but found layout.yaml can't be validated any more | 00:19 |
| jerryz | voluptuous.error.MultipleInvalid: expected a list for dictionary value @ data['pipelines'][0]['success']['gerrit'] | 00:19 |
| *** jaypipes has quit IRC | 00:19 | |
| jerryz | clarkb: voluptuous got upgraded to 0.10.5 | 00:19 |
| jerryz | when i downgraded to 0.7, the validation passed | 00:20 |
| clarkb | yes you need 2.5.2 | 00:20 |
| fungi | jerryz: you need 2.5.2 instead (or olver voluptuous) | 00:20 |
| fungi | s/olver/older/ | 00:20 |
| clarkb | ya that | 00:20 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'pr-comment' github webhook event https://review.openstack.org/443959 | 00:21 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support GitHub PR webhooks https://review.openstack.org/439834 | 00:21 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add basic Github Zuul Reporter. https://review.openstack.org/443323 | 00:21 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'push' and 'tag' github webhook events. https://review.openstack.org/443947 | 00:21 |
| *** john-davidge has quit IRC | 00:21 | |
| jerryz | clarkb, fungi: thanks | 00:22 |
| *** salv-orlando has quit IRC | 00:22 | |
| *** esikachev has quit IRC | 00:22 | |
| *** esikachev has joined #openstack-infra | 00:23 | |
| *** markvoelker has joined #openstack-infra | 00:25 | |
| *** tphummel has joined #openstack-infra | 00:26 | |
| *** esikachev has quit IRC | 00:27 | |
| dmsimard | clarkb, mordred: fyi, I can no longer reproduce that weird shade installation error I got after untangling the mess of mixed distro/pip packages | 00:30 |
| *** esikachev has joined #openstack-infra | 00:36 | |
| *** MasterOfBugs has quit IRC | 00:37 | |
| *** esikachev has quit IRC | 00:40 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add multinode-upgrades scenario jobs to cistatus https://review.openstack.org/453807 | 00:40 |
| *** mriedem has quit IRC | 00:43 | |
| *** zhurong has joined #openstack-infra | 00:50 | |
| *** esikachev has joined #openstack-infra | 00:52 | |
| *** cuongnv has joined #openstack-infra | 00:53 | |
| *** dimtruck is now known as zz_dimtruck | 00:56 | |
| *** dkehn has joined #openstack-infra | 00:57 | |
| *** ijw has joined #openstack-infra | 00:58 | |
| *** esikachev has quit IRC | 00:59 | |
| *** Sukhdev has joined #openstack-infra | 01:00 | |
| *** amrith has quit IRC | 01:04 | |
| *** xingchao has joined #openstack-infra | 01:04 | |
| *** yee37916 has quit IRC | 01:04 | |
| *** xingchao has joined #openstack-infra | 01:05 | |
| *** esikachev has joined #openstack-infra | 01:06 | |
| *** Julien-zte has joined #openstack-infra | 01:06 | |
| *** yee379 has joined #openstack-infra | 01:07 | |
| *** ijw has quit IRC | 01:07 | |
| *** Julien-zte has quit IRC | 01:10 | |
| *** Julien-zte has joined #openstack-infra | 01:11 | |
| *** esikachev has quit IRC | 01:12 | |
| *** zz_dimtruck is now known as dimtruck | 01:14 | |
| openstackgerrit | Li Ma proposed openstack-infra/project-config master: Add zmq fullstack job for Dragonflow project https://review.openstack.org/458699 | 01:16 |
| *** rodrigopaiva has joined #openstack-infra | 01:16 | |
| *** john-davidge has joined #openstack-infra | 01:18 | |
| *** rkukura has quit IRC | 01:18 | |
| *** salv-orlando has joined #openstack-infra | 01:19 | |
| *** john-davidge has quit IRC | 01:22 | |
| *** dimtruck is now known as zz_dimtruck | 01:22 | |
| *** aeng has quit IRC | 01:22 | |
| *** gildub has joined #openstack-infra | 01:24 | |
| *** rkukura has joined #openstack-infra | 01:25 | |
| *** Sukhdev has quit IRC | 01:25 | |
| *** thorst has joined #openstack-infra | 01:25 | |
| *** aeng has joined #openstack-infra | 01:25 | |
| *** aeng is now known as aeng_afk | 01:25 | |
| *** Julien-zte has quit IRC | 01:26 | |
| *** liujiong has joined #openstack-infra | 01:26 | |
| *** Julien-zte has joined #openstack-infra | 01:27 | |
| openstackgerrit | Li Ma proposed openstack-infra/project-config master: Add zmq fullstack job for Dragonflow project https://review.openstack.org/458699 | 01:28 |
| *** Sukhdev has joined #openstack-infra | 01:29 | |
| *** Sukhdev has quit IRC | 01:30 | |
| *** gcb has joined #openstack-infra | 01:31 | |
| *** Julien-zte has quit IRC | 01:31 | |
| *** thorst has quit IRC | 01:32 | |
| *** Julien-zte has joined #openstack-infra | 01:32 | |
| *** sree has joined #openstack-infra | 01:35 | |
| *** yanyanhu has joined #openstack-infra | 01:35 | |
| *** sree has quit IRC | 01:39 | |
| EmilienM | pabelanger: back online | 01:44 |
| *** j_king has joined #openstack-infra | 01:45 | |
| *** baoli has joined #openstack-infra | 01:46 | |
| *** salv-orlando has quit IRC | 01:48 | |
| *** hichihar_ has joined #openstack-infra | 01:48 | |
| *** sekelso has joined #openstack-infra | 01:49 | |
| *** sekelso is now known as skelso | 01:49 | |
| *** zz_dimtruck is now known as dimtruck | 01:50 | |
| *** baoli has quit IRC | 01:50 | |
| *** baoli has joined #openstack-infra | 01:50 | |
| *** skelso has quit IRC | 01:50 | |
| *** rkukura has quit IRC | 01:50 | |
| *** skelso has joined #openstack-infra | 01:51 | |
| *** amrith has joined #openstack-infra | 01:58 | |
| *** amrith has quit IRC | 02:00 | |
| *** Julien-zte has quit IRC | 02:02 | |
| *** yamahata has quit IRC | 02:03 | |
| *** Julien-zte has joined #openstack-infra | 02:03 | |
| *** amrith has joined #openstack-infra | 02:04 | |
| EmilienM | pabelanger: I don't think we fixed it for the upgrade jobs... | 02:04 |
| *** ijw has joined #openstack-infra | 02:04 | |
| *** thorst has joined #openstack-infra | 02:05 | |
| *** thorst has quit IRC | 02:05 | |
| *** thorst has joined #openstack-infra | 02:05 | |
| *** ijw has quit IRC | 02:09 | |
| *** thorst has quit IRC | 02:10 | |
| *** knangia has quit IRC | 02:11 | |
| EmilienM | pabelanger: i've reached the folks who work on sensu, so maybe they can help | 02:11 |
| *** gouthamr has joined #openstack-infra | 02:13 | |
| *** tphummel has quit IRC | 02:14 | |
| *** sdague has quit IRC | 02:16 | |
| *** john-davidge has joined #openstack-infra | 02:19 | |
| *** esikachev has joined #openstack-infra | 02:21 | |
| *** EricGonc_ has quit IRC | 02:21 | |
| *** EricGonczer_ has joined #openstack-infra | 02:22 | |
| *** gongysh has joined #openstack-infra | 02:22 | |
| *** john-davidge has quit IRC | 02:23 | |
| *** Julien-zte has quit IRC | 02:24 | |
| *** Julien-zte has joined #openstack-infra | 02:25 | |
| *** hongbin_ has joined #openstack-infra | 02:26 | |
| *** esikachev has quit IRC | 02:29 | |
| *** gcb has quit IRC | 02:31 | |
| *** esikachev has joined #openstack-infra | 02:32 | |
| *** thorst has joined #openstack-infra | 02:36 | |
| *** Julien-zte has quit IRC | 02:37 | |
| *** esikachev has quit IRC | 02:37 | |
| *** jerryz has quit IRC | 02:38 | |
| *** Julien-zte has joined #openstack-infra | 02:39 | |
| *** salv-orlando has joined #openstack-infra | 02:45 | |
| *** Douhet has joined #openstack-infra | 02:47 | |
| *** esikachev has joined #openstack-infra | 02:47 | |
| *** EricGonczer_ has quit IRC | 02:49 | |
| *** thorst has quit IRC | 02:50 | |
| *** thorst has joined #openstack-infra | 02:50 | |
| *** thorst has quit IRC | 02:50 | |
| *** esikachev has quit IRC | 02:52 | |
| *** ijw has joined #openstack-infra | 02:52 | |
| *** Julien-zte has quit IRC | 02:54 | |
| *** Julien-zte has joined #openstack-infra | 02:55 | |
| *** psachin has joined #openstack-infra | 02:57 | |
| *** ijw has quit IRC | 02:58 | |
| *** amotoki has joined #openstack-infra | 02:58 | |
| *** felipemonteiro__ has joined #openstack-infra | 03:02 | |
| *** salv-orl_ has joined #openstack-infra | 03:03 | |
| *** felipemonteiro_ has quit IRC | 03:03 | |
| *** salv-orlando has quit IRC | 03:03 | |
| *** ihrachys has quit IRC | 03:03 | |
| *** amotoki has quit IRC | 03:03 | |
| *** chem_gone has quit IRC | 03:03 | |
| *** chem_gone has joined #openstack-infra | 03:04 | |
| *** ihrachys has joined #openstack-infra | 03:04 | |
| *** david-lyle has joined #openstack-infra | 03:11 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 03:15 | |
| *** portdirect has quit IRC | 03:16 | |
| *** salv-orl_ has quit IRC | 03:16 | |
| *** tphummel has joined #openstack-infra | 03:17 | |
| *** david-lyle has quit IRC | 03:19 | |
| *** john-davidge has joined #openstack-infra | 03:19 | |
| *** esikachev has joined #openstack-infra | 03:20 | |
| *** portdirect has joined #openstack-infra | 03:23 | |
| *** john-davidge has quit IRC | 03:24 | |
| *** esikachev has quit IRC | 03:25 | |
| *** hongbin_ has quit IRC | 03:25 | |
| *** nicolasbock has quit IRC | 03:28 | |
| *** Sukhdev has joined #openstack-infra | 03:30 | |
| *** zhurong has quit IRC | 03:43 | |
| *** ijw has joined #openstack-infra | 03:46 | |
| *** thorst has joined #openstack-infra | 03:51 | |
| *** thorst has quit IRC | 03:56 | |
| *** erikmwilson has quit IRC | 03:57 | |
| *** erikmwilson has joined #openstack-infra | 03:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Stop using install-static for jenkins sudoers file https://review.openstack.org/458142 | 03:58 |
| *** ykarel has joined #openstack-infra | 04:00 | |
| *** dave-mcc_ has quit IRC | 04:02 | |
| *** esikachev has joined #openstack-infra | 04:08 | |
| *** amotoki has joined #openstack-infra | 04:09 | |
| *** baoli has quit IRC | 04:09 | |
| *** esikachev has quit IRC | 04:12 | |
| *** salv-orlando has joined #openstack-infra | 04:13 | |
| *** gongysh has quit IRC | 04:14 | |
| *** john-davidge has joined #openstack-infra | 04:20 | |
| *** esikachev has joined #openstack-infra | 04:20 | |
| *** yamahata has joined #openstack-infra | 04:24 | |
| *** john-davidge has quit IRC | 04:24 | |
| *** esikachev has quit IRC | 04:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove trusty jobs from nova experimental queue https://review.openstack.org/458675 | 04:26 |
| *** xyang1_ has joined #openstack-infra | 04:29 | |
| *** Julien-zte has quit IRC | 04:31 | |
| *** xyang1 has quit IRC | 04:32 | |
| *** xyang1_ is now known as xyang1 | 04:32 | |
| *** Julien-zte has joined #openstack-infra | 04:32 | |
| *** MasterOfBugs has joined #openstack-infra | 04:33 | |
| *** pgadiya has joined #openstack-infra | 04:34 | |
| *** salv-orlando has quit IRC | 04:41 | |
| *** gongysh has joined #openstack-infra | 04:42 | |
| *** esikachev has joined #openstack-infra | 04:43 | |
| *** salv-orlando has joined #openstack-infra | 04:46 | |
| *** esikachev has quit IRC | 04:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove trusty jobs from nova experimental queue https://review.openstack.org/458675 | 04:49 |
| *** armax has quit IRC | 04:50 | |
| *** tphummel has quit IRC | 04:51 | |
| *** thorst has joined #openstack-infra | 04:52 | |
| *** thorst has quit IRC | 04:57 | |
| *** Rockyg has joined #openstack-infra | 05:04 | |
| *** aeng_afk has quit IRC | 05:05 | |
| *** charz has quit IRC | 05:07 | |
| *** timburke has quit IRC | 05:08 | |
| *** dimtruck is now known as zz_dimtruck | 05:09 | |
| *** gongysh has quit IRC | 05:09 | |
| *** baoli has joined #openstack-infra | 05:10 | |
| *** apetrich has joined #openstack-infra | 05:12 | |
| *** baoli has quit IRC | 05:15 | |
| *** gcb has joined #openstack-infra | 05:18 | |
| *** john-davidge has joined #openstack-infra | 05:21 | |
| *** skelso has quit IRC | 05:21 | |
| *** Julien-zte has quit IRC | 05:23 | |
| *** dhajare has joined #openstack-infra | 05:24 | |
| *** Julien-zte has joined #openstack-infra | 05:25 | |
| *** HeOS has joined #openstack-infra | 05:25 | |
| *** john-davidge has quit IRC | 05:25 | |
| *** apetrich has quit IRC | 05:30 | |
| *** apetrich has joined #openstack-infra | 05:32 | |
| *** esikachev has joined #openstack-infra | 05:32 | |
| *** prateek has joined #openstack-infra | 05:35 | |
| *** HeOS has quit IRC | 05:35 | |
| *** esikachev has quit IRC | 05:37 | |
| *** apetrich has quit IRC | 05:38 | |
| *** Shrews has quit IRC | 05:43 | |
| *** Hal1 has quit IRC | 05:45 | |
| *** cshastri has joined #openstack-infra | 05:47 | |
| *** Rockyg has quit IRC | 05:48 | |
| *** Sukhdev has quit IRC | 05:48 | |
| *** apetrich has joined #openstack-infra | 05:50 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: BlockDevice - pass command line arguments directly https://review.openstack.org/458727 | 05:52 |
| *** ralonsoh has joined #openstack-infra | 05:52 | |
| *** thorst has joined #openstack-infra | 05:52 | |
| *** Julien-zte has quit IRC | 05:54 | |
| *** Julien-zte has joined #openstack-infra | 05:54 | |
| *** hichihar_ has quit IRC | 05:55 | |
| *** thorst has quit IRC | 05:57 | |
| *** yolanda has joined #openstack-infra | 05:58 | |
| *** gongysh has joined #openstack-infra | 05:59 | |
| *** jbadiapa has joined #openstack-infra | 05:59 | |
| *** Rockyg has joined #openstack-infra | 06:00 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 06:01 |
| *** xingchao has quit IRC | 06:03 | |
| *** xingchao has joined #openstack-infra | 06:03 | |
| *** matrohon_ has joined #openstack-infra | 06:04 | |
| *** Julien-zte has quit IRC | 06:06 | |
| *** jaosorior has joined #openstack-infra | 06:06 | |
| openstackgerrit | Li Ma proposed openstack-infra/project-config master: Add zmq fullstack job for Dragonflow project https://review.openstack.org/458699 | 06:07 |
| *** Julien-zte has joined #openstack-infra | 06:08 | |
| *** esikachev has joined #openstack-infra | 06:09 | |
| *** sree has joined #openstack-infra | 06:10 | |
| *** tie has quit IRC | 06:11 | |
| *** salv-orlando has quit IRC | 06:11 | |
| *** salv-orlando has joined #openstack-infra | 06:11 | |
| *** rcernin has joined #openstack-infra | 06:15 | |
| *** salv-orlando has quit IRC | 06:16 | |
| *** pcaruana has joined #openstack-infra | 06:18 | |
| *** zhurong has joined #openstack-infra | 06:21 | |
| *** john-davidge has joined #openstack-infra | 06:22 | |
| *** john-davidge has quit IRC | 06:26 | |
| AJaeger | morning, yolanda. Could you put infra-manuals on your review queue, please? | 06:28 |
| yolanda | hi AJaeger , good morning | 06:28 |
| yolanda | ok i will do | 06:28 |
| AJaeger | thanks | 06:29 |
| *** kaisers1 has quit IRC | 06:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Load nova configuration for devstack-plugin-ceph job" https://review.openstack.org/458349 | 06:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove trusty jobs from nova experimental queue https://review.openstack.org/458675 | 06:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Add iptables to DIB elements https://review.openstack.org/458623 | 06:35 |
| *** chem_gone is now known as chem | 06:36 | |
| *** chem has quit IRC | 06:36 | |
| *** chem has joined #openstack-infra | 06:36 | |
| *** HeOS has joined #openstack-infra | 06:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Make octavia scenario jobs voting https://review.openstack.org/458311 | 06:38 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Improve rebase section https://review.openstack.org/456044 | 06:38 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Review List for New Projects https://review.openstack.org/400288 | 06:39 |
| *** adriant has quit IRC | 06:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: gnocchi: fix functional tests on master https://review.openstack.org/458368 | 06:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Transit OVB OOOQ job to new mode https://review.openstack.org/454202 | 06:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Add mosquitto grafana dashboard https://review.openstack.org/458627 | 06:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove root user .cache folder https://review.openstack.org/458628 | 06:39 |
| *** gouthamr has quit IRC | 06:39 | |
| *** zhurong has quit IRC | 06:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: jenkins: scripts: run-tox.sh: Increase verbosity https://review.openstack.org/457709 | 06:40 |
| *** kaisers1 has joined #openstack-infra | 06:41 | |
| *** zhurong has joined #openstack-infra | 06:42 | |
| *** gcb has quit IRC | 06:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create heat-specific patrole gate https://review.openstack.org/456300 | 06:43 |
| openstackgerrit | Merged openstack-infra/project-config master: Enable nova in kuryr fullstack and rally test https://review.openstack.org/457070 | 06:43 |
| openstackgerrit | Merged openstack-infra/project-config master: Re-order plugins in collectd-ceiloemter-plugin job https://review.openstack.org/456193 | 06:44 |
| *** Julien-zte has quit IRC | 06:45 | |
| *** camunoz has quit IRC | 06:46 | |
| *** Julien-zte has joined #openstack-infra | 06:47 | |
| *** andreas_s has joined #openstack-infra | 06:47 | |
| *** e0ne has joined #openstack-infra | 06:47 | |
| *** ralonsoh_ has joined #openstack-infra | 06:49 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Add ovb-fakeha-caserver (TLS everywhere) job to cistatus https://review.openstack.org/458468 | 06:50 |
| *** tosky has joined #openstack-infra | 06:50 | |
| *** ralonsoh_ has quit IRC | 06:51 | |
| *** Julien-zte has quit IRC | 06:51 | |
| *** ralonsoh has quit IRC | 06:51 | |
| *** Julien-zte has joined #openstack-infra | 06:51 | |
| *** HeOS has quit IRC | 06:52 | |
| *** thorst has joined #openstack-infra | 06:53 | |
| *** zhurong has quit IRC | 06:54 | |
| *** e0ne has quit IRC | 06:54 | |
| *** Hal has joined #openstack-infra | 06:55 | |
| *** chlong has quit IRC | 06:56 | |
| *** gcb has joined #openstack-infra | 06:56 | |
| *** vsaienko has joined #openstack-infra | 06:56 | |
| *** thorst has quit IRC | 06:58 | |
| *** ralonsoh has joined #openstack-infra | 06:59 | |
| *** tosky has quit IRC | 06:59 | |
| *** salv-orlando has joined #openstack-infra | 07:01 | |
| *** markus_z has joined #openstack-infra | 07:01 | |
| *** jascott1 has quit IRC | 07:02 | |
| *** Julien-zte has quit IRC | 07:02 | |
| *** Julien-zte has joined #openstack-infra | 07:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Enable non-apache devstack job for Mistral https://review.openstack.org/457106 | 07:08 |
| *** Julien-zte has quit IRC | 07:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add py35 jobs for Zaqar UI https://review.openstack.org/456490 | 07:08 |
| *** MasterOfBugs has quit IRC | 07:09 | |
| *** Julien-zte has joined #openstack-infra | 07:09 | |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** baoli has joined #openstack-infra | 07:11 | |
| *** florianf has joined #openstack-infra | 07:13 | |
| *** Rockyg has quit IRC | 07:13 | |
| *** ggnel_t has joined #openstack-infra | 07:14 | |
| *** saibarspeis has joined #openstack-infra | 07:15 | |
| *** baoli has quit IRC | 07:16 | |
| *** gongysh has quit IRC | 07:18 | |
| *** aviau has quit IRC | 07:24 | |
| *** aviau has joined #openstack-infra | 07:24 | |
| *** shardy has joined #openstack-infra | 07:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove most python 34 jobs https://review.openstack.org/456340 | 07:25 |
| *** hashar has joined #openstack-infra | 07:26 | |
| *** ssbarnea has joined #openstack-infra | 07:31 | |
| *** gildub has quit IRC | 07:31 | |
| *** zhurong has joined #openstack-infra | 07:31 | |
| *** sree has quit IRC | 07:33 | |
| *** sree has joined #openstack-infra | 07:34 | |
| *** sree has quit IRC | 07:36 | |
| *** e0ne has joined #openstack-infra | 07:37 | |
| openstackgerrit | dongwenjuan proposed openstack-infra/system-config master: use "service puppet stop", not "service puppet disable" https://review.openstack.org/458749 | 07:39 |
| *** Julien-zte has quit IRC | 07:42 | |
| *** Julien-zte has joined #openstack-infra | 07:42 | |
| *** jpich has joined #openstack-infra | 07:43 | |
| openstackgerrit | Merged openstack-dev/pbr master: trivial: Add note about multiple builders support https://review.openstack.org/439545 | 07:46 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Update fakeha net-iso configuration https://review.openstack.org/458596 | 07:49 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: Add pbr to job periodic-{name}-{pyhton}-with-oslo-master https://review.openstack.org/458753 | 07:49 |
| *** alexchadin has joined #openstack-infra | 07:49 | |
| AJaeger | gcb: could you review and answer http://lists.openstack.org/pipermail/openstack-dev/2017-April/115682.html , please? | 07:52 |
| *** thorst has joined #openstack-infra | 07:54 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** apetrich has quit IRC | 08:01 | |
| *** apetrich has joined #openstack-infra | 08:02 | |
| *** lucas-afk is now known as lucasagomes | 08:07 | |
| *** gongysh has joined #openstack-infra | 08:09 | |
| *** HeOS has joined #openstack-infra | 08:09 | |
| *** john-davidge has joined #openstack-infra | 08:11 | |
| *** derekh has joined #openstack-infra | 08:12 | |
| *** gouthamr has joined #openstack-infra | 08:13 | |
| *** rcernin has quit IRC | 08:13 | |
| *** Julien-zte has quit IRC | 08:14 | |
| *** thorst has quit IRC | 08:14 | |
| *** Julien-zte has joined #openstack-infra | 08:15 | |
| *** zhurong has quit IRC | 08:15 | |
| openstackgerrit | Merged openstack-infra/groups master: Update to new OpenStack logos https://review.openstack.org/441589 | 08:15 |
| *** john-davidge has quit IRC | 08:16 | |
| *** efoley_ has joined #openstack-infra | 08:16 | |
| *** rossella_s has joined #openstack-infra | 08:16 | |
| *** bhavik1 has joined #openstack-infra | 08:18 | |
| *** Julien-zte has quit IRC | 08:18 | |
| *** efoley__ has joined #openstack-infra | 08:18 | |
| *** Julien-zte has joined #openstack-infra | 08:19 | |
| *** efoley_ has quit IRC | 08:21 | |
| *** gouthamr has quit IRC | 08:22 | |
| *** zhurong has joined #openstack-infra | 08:22 | |
| *** mhickey has joined #openstack-infra | 08:22 | |
| *** sflanigan has quit IRC | 08:23 | |
| openstackgerrit | dongwenjuan proposed openstack-infra/system-config master: use "service puppet stop", not "service puppet disable" https://review.openstack.org/458749 | 08:25 |
| *** mhickey has quit IRC | 08:26 | |
| *** mhickey has joined #openstack-infra | 08:26 | |
| *** Julien-zte has quit IRC | 08:26 | |
| *** rcernin has joined #openstack-infra | 08:27 | |
| *** Julien-zte has joined #openstack-infra | 08:28 | |
| *** jpich has quit IRC | 08:29 | |
| *** jpich has joined #openstack-infra | 08:30 | |
| *** ralonsoh has quit IRC | 08:30 | |
| *** ralonsoh has joined #openstack-infra | 08:32 | |
| *** shardy has quit IRC | 08:34 | |
| *** Julien-zte has quit IRC | 08:34 | |
| *** shardy has joined #openstack-infra | 08:35 | |
| *** Julien-zte has joined #openstack-infra | 08:35 | |
| *** Julien-zte has quit IRC | 08:43 | |
| *** Julien-zte has joined #openstack-infra | 08:44 | |
| openstackgerrit | Li Ma proposed openstack-infra/project-config master: Add zmq fullstack job for Dragonflow project https://review.openstack.org/458699 | 08:46 |
| *** phschwartz has quit IRC | 08:50 | |
| *** martinkopec has joined #openstack-infra | 08:50 | |
| *** Julien-zte has quit IRC | 08:50 | |
| *** afazekas has quit IRC | 08:50 | |
| *** gongysh has quit IRC | 08:51 | |
| *** electrofelix has joined #openstack-infra | 08:51 | |
| *** Julien-zte has joined #openstack-infra | 08:51 | |
| *** e0ne has quit IRC | 08:54 | |
| *** yanyanhu has quit IRC | 08:55 | |
| *** amotoki has quit IRC | 08:57 | |
| *** jrist has quit IRC | 08:57 | |
| *** pgadiya has quit IRC | 08:59 | |
| *** vgridnev has joined #openstack-infra | 09:00 | |
| *** tie has joined #openstack-infra | 09:01 | |
| *** andreas-f has quit IRC | 09:01 | |
| *** aarefiev_afk is now known as aarefiev | 09:02 | |
| vgridnev | who can help me to get my change reviewed? https://review.openstack.org/#/c/447158/ | 09:02 |
| *** pgadiya has joined #openstack-infra | 09:04 | |
| *** e0ne has joined #openstack-infra | 09:07 | |
| *** amotoki has joined #openstack-infra | 09:08 | |
| *** sambetts|afk is now known as sambetts | 09:08 | |
| *** Guest41303 has joined #openstack-infra | 09:09 | |
| *** tosky has joined #openstack-infra | 09:12 | |
| *** john-davidge has joined #openstack-infra | 09:12 | |
| *** nadya has joined #openstack-infra | 09:13 | |
| *** baoli has joined #openstack-infra | 09:14 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add ovsdbapp to gerritbot https://review.openstack.org/458782 | 09:15 |
| *** john-davidge has quit IRC | 09:16 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:18 | |
| *** baoli has quit IRC | 09:18 | |
| *** claudiub|2 has joined #openstack-infra | 09:19 | |
| *** Guest41303 has quit IRC | 09:20 | |
| *** Julien-zte has quit IRC | 09:21 | |
| *** flepied has quit IRC | 09:22 | |
| *** Julien-zte has joined #openstack-infra | 09:22 | |
| *** e0ne has quit IRC | 09:23 | |
| *** flepied has joined #openstack-infra | 09:24 | |
| *** Julien-zte has quit IRC | 09:26 | |
| *** Julien-zte has joined #openstack-infra | 09:27 | |
| *** pgadiya has quit IRC | 09:28 | |
| *** amotoki has quit IRC | 09:30 | |
| *** zhurong has quit IRC | 09:33 | |
| *** amotoki has joined #openstack-infra | 09:33 | |
| openstackgerrit | Lee Yarwood proposed openstack-infra/project-config master: Run neutron scenario multinode nv in nova experimental queue https://review.openstack.org/458676 | 09:33 |
| *** Julien-zte has quit IRC | 09:33 | |
| *** flepied has quit IRC | 09:34 | |
| *** gouthamr has joined #openstack-infra | 09:35 | |
| *** zhurong has joined #openstack-infra | 09:36 | |
| *** ralonsoh has quit IRC | 09:36 | |
| *** ramishra has quit IRC | 09:37 | |
| *** zhurong has quit IRC | 09:38 | |
| *** ralonsoh has joined #openstack-infra | 09:38 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Support container upgrades in multinode OOOQ CI https://review.openstack.org/450784 | 09:40 |
| *** pgadiya has joined #openstack-infra | 09:42 | |
| *** john-davidge has joined #openstack-infra | 09:42 | |
| *** gouthamr has quit IRC | 09:44 | |
| *** ramishra has joined #openstack-infra | 09:45 | |
| *** salv-orl_ has joined #openstack-infra | 09:48 | |
| *** afazekas has joined #openstack-infra | 09:50 | |
| *** salv-orlando has quit IRC | 09:51 | |
| *** psachin has quit IRC | 09:54 | |
| *** winggundamth has joined #openstack-infra | 09:54 | |
| *** alexchadin has quit IRC | 09:56 | |
| lyarwood | anyone able to +W https://review.openstack.org/458676 now the conflicts have been removed thanks to the rebase? | 09:58 |
| *** jordanP has quit IRC | 10:01 | |
| *** nadya has quit IRC | 10:01 | |
| *** pgadiya has quit IRC | 10:04 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/devstack-gate master: Do not set ephemeral size based on driver https://review.openstack.org/326061 | 10:04 |
| *** amotoki has quit IRC | 10:05 | |
| *** jordanP has joined #openstack-infra | 10:05 | |
| *** e0ne has joined #openstack-infra | 10:07 | |
| *** ccamacho has joined #openstack-infra | 10:07 | |
| *** alexchadin has joined #openstack-infra | 10:07 | |
| *** thorst has joined #openstack-infra | 10:11 | |
| *** baoli has joined #openstack-infra | 10:15 | |
| *** nicolasbock has joined #openstack-infra | 10:15 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Add undercloud-containers job to cistatus https://review.openstack.org/446957 | 10:15 |
| *** thorst has quit IRC | 10:16 | |
| *** Julien-zte has joined #openstack-infra | 10:18 | |
| *** baoli has quit IRC | 10:19 | |
| *** pgadiya has joined #openstack-infra | 10:19 | |
| *** psachin has joined #openstack-infra | 10:21 | |
| *** mat128 has joined #openstack-infra | 10:21 | |
| *** ralonsoh has quit IRC | 10:23 | |
| *** yamamoto has quit IRC | 10:25 | |
| arxcruz | mtreinish: hey, what's the name again of the service that provides the zuul queue ? has something to do with horse iirc | 10:25 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: [WIP] Replace neutronclient list_networks call with call to REST Api https://review.openstack.org/458803 | 10:27 |
| *** winggundamth has quit IRC | 10:27 | |
| *** alexchadin has quit IRC | 10:28 | |
| arxcruz | mtreinish: nevermind, firehose, not sure where I came with horse :) | 10:28 |
| *** winggundamth has joined #openstack-infra | 10:28 | |
| *** Julien-zte has quit IRC | 10:28 | |
| *** liujiong has quit IRC | 10:29 | |
| *** Julien-zte has joined #openstack-infra | 10:30 | |
| *** nadya has joined #openstack-infra | 10:32 | |
| vgridnev | who can help me to get my change reviewed? https://review.openstack.org/#/c/447158/ | 10:34 |
| *** cuongnv has quit IRC | 10:37 | |
| *** nadya has quit IRC | 10:38 | |
| *** nadya has joined #openstack-infra | 10:38 | |
| *** vsaienko has quit IRC | 10:41 | |
| *** ralonsoh has joined #openstack-infra | 10:42 | |
| *** bhavik1 has quit IRC | 10:43 | |
| *** jtomasek_ has joined #openstack-infra | 10:43 | |
| *** Julien-zte has quit IRC | 10:48 | |
| *** Julien-zte has joined #openstack-infra | 10:51 | |
| rcarrillocruz | +2 | 10:51 |
| vgridnev | thanks, \o/ | 10:51 |
| hwoarang | fungi: sorry to be a pain but is there a chance to make a new bindep release please? :) | 10:52 |
| *** pgadiya has quit IRC | 10:54 | |
| *** skelso has joined #openstack-infra | 10:57 | |
| *** skelso has quit IRC | 10:58 | |
| asettle | Hola infra! I have a request. The openstack-doc list is currently moderated by me and anyone else that has that password. We get an astonishing amount of spam. How can we change this? I'd like to have the ML to be self-serving if possible. | 10:59 |
| *** yamahata has quit IRC | 11:00 | |
| *** rossella_s has quit IRC | 11:00 | |
| *** vgridnev has quit IRC | 11:02 | |
| *** rossella_s has joined #openstack-infra | 11:06 | |
| *** pgadiya has joined #openstack-infra | 11:07 | |
| *** markvoelker has quit IRC | 11:14 | |
| *** xingchao has quit IRC | 11:14 | |
| *** baoli has joined #openstack-infra | 11:15 | |
| *** baoli has quit IRC | 11:20 | |
| *** mat128 has quit IRC | 11:21 | |
| *** thorst has joined #openstack-infra | 11:25 | |
| *** lucasagomes is now known as lucas-afk | 11:25 | |
| *** mat128 has joined #openstack-infra | 11:27 | |
| *** shardy is now known as shardy_lunch | 11:32 | |
| *** yamamoto has joined #openstack-infra | 11:33 | |
| *** yamamoto has quit IRC | 11:35 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 11:36 |
| *** Julien-zte has quit IRC | 11:37 | |
| *** Julien-zte has joined #openstack-infra | 11:38 | |
| *** yamamoto has joined #openstack-infra | 11:39 | |
| *** salv-orl_ has quit IRC | 11:40 | |
| *** mat128 has quit IRC | 11:45 | |
| *** sree has joined #openstack-infra | 11:45 | |
| *** makowals has quit IRC | 11:48 | |
| *** vsaienko has joined #openstack-infra | 11:49 | |
| EmilienM | pabelanger (and other infra folks): cschwede and I think some cloud provider are slower than some others (we haven't determined if it's network or cpu or both) but it affects the runtime of tripleo multinode jobs. For example, we think Internal / RAX is much slower than OSIC. Can you confirm all of that? | 11:50 |
| *** vsaienko has quit IRC | 11:51 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** prateek has quit IRC | 11:52 | |
| *** makowals has joined #openstack-infra | 11:53 | |
| *** hashar has quit IRC | 11:53 | |
| *** sdague has joined #openstack-infra | 11:54 | |
| *** Julien-zte has quit IRC | 11:55 | |
| openstackgerrit | Merged openstack-infra/bindep master: Update 'pbr' version requirements https://review.openstack.org/451333 | 11:56 |
| *** alexchadin has joined #openstack-infra | 11:56 | |
| openstackgerrit | Merged openstack-infra/bindep master: Exit the loop when we don't have a package https://review.openstack.org/452007 | 11:57 |
| *** Julien-zte has joined #openstack-infra | 11:57 | |
| *** mat128 has joined #openstack-infra | 11:59 | |
| *** baoli has joined #openstack-infra | 11:59 | |
| fungi | hwoarang: thanks for the reminder. i just finished going through open changes and approved a couple of additional small ones. i'm on the fence about releases on a friday though. i guess if i do it early enough (now?) then maybe it's not so terrible | 12:01 |
| *** mat128 has quit IRC | 12:01 | |
| openstackgerrit | Merged openstack-infra/bindep master: Add Amazon Linux support to bindep https://review.openstack.org/454240 | 12:02 |
| *** baoli has quit IRC | 12:03 | |
| *** trown|outtypewww is now known as trown | 12:03 | |
| hwoarang | fungi: no problem. i am fine if you want to release next week. i can wait. Thank you! | 12:03 |
| *** jaypipes has joined #openstack-infra | 12:04 | |
| *** esikachev has quit IRC | 12:04 | |
| *** jaypipes is now known as leakypipes | 12:04 | |
| AJaeger | fungi, could you put infra-manual review queue also on your list, please? - nothing urgent for today, I just see that it needs some love | 12:07 |
| *** hashar has joined #openstack-infra | 12:07 | |
| fungi | EmilienM: different providers have different numbers of vcpus, cpu speeds, i/o bandwidth, backing storage technologies, network bandwidth and latency to central resources, et cetera. and then factor in overcommit ratios and noisy neighbor problem and you get all manner of variability depending on what your particular job uses most of | 12:08 |
| EmilienM | fungi: yeah | 12:08 |
| EmilienM | fungi: jobs on OSIC that take 1h30 take 2h40 on RAX or Internap. | 12:08 |
| EmilienM | it's pretty rough | 12:08 |
| fungi | we target sufficient performance to be able to complete a devstack+tempest full job within 1 hour | 12:08 |
| EmilienM | I'm happy it works for devstack | 12:09 |
| fungi | is (soon was) highly performant hardware | 12:09 |
| fungi | er, osic | 12:09 |
| fungi | so i'm not surprised jobs ran faster there | 12:09 |
| EmilienM | oh dear it's Friday, I can troll today | 12:09 |
| fungi | especially for disk (all ssd there) | 12:09 |
| fungi | also effectively no overcommit as we were the only people using it much of the time | 12:10 |
| *** baoli has joined #openstack-infra | 12:11 | |
| *** baoli has quit IRC | 12:12 | |
| *** jaosorior is now known as jaosorior_away | 12:12 | |
| fungi | AJaeger: thanks for the suggestion. i've starred all the infra-manual changes passing ci (and i have one on my to do list to write as well) | 12:13 |
| fungi | need to step out to get some more yardwork done before the temperature gets oppressive, back in time for the release team meeting | 12:13 |
| fungi | then need to whip up a renames maintenance plan and will hopefully get to reviews after that | 12:14 |
| AJaeger | thanks, fungi. Happy yard working ;) | 12:14 |
| fungi | thanks | 12:14 |
| *** markvoelker has joined #openstack-infra | 12:15 | |
| *** mat128 has joined #openstack-infra | 12:16 | |
| *** salv-orlando has joined #openstack-infra | 12:17 | |
| *** EricGonczer_ has joined #openstack-infra | 12:17 | |
| *** markvoelker has quit IRC | 12:20 | |
| arxcruz | fungi: mtreinish do you know if periodic jobs also appears in firehose.openstack.org ? | 12:21 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: sahara: explicitly set the list of repos in PROJECTS https://review.openstack.org/458815 | 12:24 |
| *** shardy_lunch is now known as shardy | 12:25 | |
| *** dalvarez has quit IRC | 12:25 | |
| *** wznoinsk has quit IRC | 12:25 | |
| *** dmellado has quit IRC | 12:26 | |
| *** dmanchad has quit IRC | 12:26 | |
| *** yanyanhu has joined #openstack-infra | 12:26 | |
| *** lucas-afk is now known as lucasagomes | 12:26 | |
| *** mat128 has quit IRC | 12:27 | |
| *** yanyanhu has quit IRC | 12:27 | |
| *** dalvarez has joined #openstack-infra | 12:28 | |
| *** mat128 has joined #openstack-infra | 12:29 | |
| *** wznoinsk has joined #openstack-infra | 12:31 | |
| *** jtomasek_ has quit IRC | 12:34 | |
| *** sree has quit IRC | 12:35 | |
| *** gordc has joined #openstack-infra | 12:36 | |
| *** dmanchad has joined #openstack-infra | 12:38 | |
| *** markvoelker has joined #openstack-infra | 12:40 | |
| *** gcb has quit IRC | 12:45 | |
| *** dmellado has joined #openstack-infra | 12:47 | |
| *** Goneri has joined #openstack-infra | 12:49 | |
| *** rlandy has joined #openstack-infra | 12:50 | |
| *** winggundamth_ has joined #openstack-infra | 12:52 | |
| *** xarses has quit IRC | 12:54 | |
| mtreinish | arxcruz: zuul jobs aren't reporting to firehose.o.o yet, right now it just has events from gerrit and launchpad | 12:54 |
| mtreinish | arxcruz: https://docs.openstack.org/infra/system-config/firehose.html#services-publishing-to-firehose (I should remove the logstash one from that table, we had to disable that) | 12:55 |
| *** winggundamth has quit IRC | 12:55 | |
| *** yamamoto has joined #openstack-infra | 12:57 | |
| *** srobert has joined #openstack-infra | 12:57 | |
| *** alexchadin has quit IRC | 12:58 | |
| *** yamamoto has quit IRC | 12:58 | |
| *** Julien-zte has quit IRC | 12:58 | |
| *** yamamoto has joined #openstack-infra | 12:58 | |
| arxcruz | mtreinish: thanks | 12:59 |
| arxcruz | mtreinish: hey, if you don't mind, can you take a look at https://review.openstack.org/#/c/458054/ ? | 12:59 |
| *** jrist has joined #openstack-infra | 12:59 | |
| *** jrist has quit IRC | 12:59 | |
| *** jrist has joined #openstack-infra | 12:59 | |
| *** Julien-zte has joined #openstack-infra | 12:59 | |
| *** saibarspeis has quit IRC | 12:59 | |
| mtreinish | arxcruz: I'm working at adding a bunch of other stuff right now too: https://review.openstack.org/#/q/topic:more-firehose (I think I need another couple of docs pages) | 12:59 |
| *** pgadiya has quit IRC | 13:00 | |
| *** dprince has joined #openstack-infra | 13:02 | |
| *** mdrabe has joined #openstack-infra | 13:04 | |
| *** Julien-zte has quit IRC | 13:06 | |
| *** psachin has quit IRC | 13:07 | |
| *** Julien-zte has joined #openstack-infra | 13:08 | |
| *** vsaienko has joined #openstack-infra | 13:09 | |
| *** alexchadin has joined #openstack-infra | 13:10 | |
| *** askb has quit IRC | 13:13 | |
| *** martinkopec has quit IRC | 13:14 | |
| *** yamamoto has quit IRC | 13:15 | |
| *** baoli has joined #openstack-infra | 13:16 | |
| *** baoli has quit IRC | 13:16 | |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| *** mriedem has joined #openstack-infra | 13:16 | |
| *** yamamoto has quit IRC | 13:16 | |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| *** john-dav_ has joined #openstack-infra | 13:17 | |
| *** john-davidge has quit IRC | 13:17 | |
| *** jamesdenton has joined #openstack-infra | 13:17 | |
| *** Goneri has quit IRC | 13:17 | |
| *** esberglu has joined #openstack-infra | 13:19 | |
| *** Julien-zte has quit IRC | 13:20 | |
| *** Julien-zte has joined #openstack-infra | 13:21 | |
| *** esikachev has joined #openstack-infra | 13:21 | |
| *** yamamoto has quit IRC | 13:25 | |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| *** Julien-zte has quit IRC | 13:26 | |
| *** baoli has joined #openstack-infra | 13:26 | |
| *** Julien-zte has joined #openstack-infra | 13:27 | |
| *** dave-mccowan has joined #openstack-infra | 13:28 | |
| *** yamamoto has quit IRC | 13:30 | |
| *** Julien-zte has quit IRC | 13:31 | |
| *** baoli has quit IRC | 13:31 | |
| *** Julien-zte has joined #openstack-infra | 13:32 | |
| *** winggundamth_ has quit IRC | 13:33 | |
| *** flepied has joined #openstack-infra | 13:34 | |
| *** baoli has joined #openstack-infra | 13:34 | |
| *** Dinesh_Bhor has quit IRC | 13:37 | |
| *** igormarnat has quit IRC | 13:42 | |
| *** dhajare has quit IRC | 13:42 | |
| *** igormarnat has joined #openstack-infra | 13:42 | |
| *** Julien-zte has quit IRC | 13:42 | |
| *** xarses has joined #openstack-infra | 13:42 | |
| *** Julien-zte has joined #openstack-infra | 13:43 | |
| *** chlong has joined #openstack-infra | 13:44 | |
| mtreinish | clarkb, fungi: if you get time today do you want to poke at the subunit workers again, so we can try and get the mqtt working | 13:44 |
| *** shardy is now known as shardy_afk | 13:44 | |
| *** eharney has joined #openstack-infra | 13:49 | |
| arxcruz | mtreinish: thanks for the infos so, in the future zuul will also appear in firehose right? any idea when in the future? | 13:52 |
| mtreinish | arxcruz: right now we're waiting for zuul v3 to finish. It'll be high on my priority list after that | 13:56 |
| fungi | mtreinish: what poking do they need? probably just service restarts? | 13:56 |
| arxcruz | mtreinish: cool, i would like to help you on that since it's my interest as well :) | 13:56 |
| *** Julien-zte has quit IRC | 13:56 | |
| mtreinish | fungi: yeah, I think clarkb rebooted subunit-worker01.o.o so the daemon is already down there | 13:56 |
| arxcruz | also a good way to learn more about it | 13:56 |
| sdague | vsaienk0: would this work for you https://review.openstack.org/#/c/458625/1 ? | 13:56 |
| *** andreas_s has quit IRC | 13:57 | |
| fungi | arxcruz: you can probably infer some of it from the subunit2sql and logstash worker events in the firehose in the meantime | 13:57 |
| mtreinish | arxcruz: ok, cool. Yeah I'll keep you in the loop | 13:57 |
| mtreinish | fungi: that's true, heh but we got to get those running first :) | 13:57 |
| *** prateek has joined #openstack-infra | 13:57 | |
| fungi | right | 13:57 |
| arxcruz | mtreinish: fungi another question, is it okay to use health.o.o api ? | 13:57 |
| fungi | arxcruz: definitely. it's a public api server for a reason ;) | 13:58 |
| vsaienk0 | sdague: I think it will, but we already moved source openrc under source lib/keystone in our upgrade script | 13:58 |
| arxcruz | I got some interesting data that I need for a project | 13:58 |
| sdague | vsaienk0: ok | 13:58 |
| arxcruz | and since it's already implemented, up and running... | 13:58 |
| mtreinish | arxcruz: yeah you can use the api directly. But, we don't really write it with any stability guarantees | 13:58 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Adding subunit files to artcl_collect_list https://review.openstack.org/455219 | 13:58 |
| *** prateek has quit IRC | 13:58 | |
| *** Julien-zte has joined #openstack-infra | 13:58 | |
| mtreinish | so there is a chance things will just change on you if we need to change the format for a graph in the frontend or something | 13:58 |
| sdague | vsaienk0: I think you brought up an interesting transition case regardless, I'm going to get that openrc change in as well as copy the .stackenv over | 13:58 |
| arxcruz | mtreinish: I see, so maybe I can clone and put only the api in some server.. | 13:58 |
| fungi | arxcruz: we also provide a public mysql socket proxy with a read-only account (query/query for username/password i believe) if you want to directly query the backend | 13:59 |
| arxcruz | fungi: the logstash queue provides info about when the periodic job finishes ? | 14:00 |
| arxcruz | would be nice to have a list of the topics in firehose documented | 14:00 |
| *** Julien-zte has quit IRC | 14:01 | |
| mtreinish | arxcruz: yeah you can do that very easily. I do that locally when I'm testing changes | 14:01 |
| fungi | arxcruz: once we get the logstash events into the firehose (hopefully today or early next week) there will be events corresponding to when the logstash workers figure out that jobs have completed and start indexing their logs | 14:01 |
| mtreinish | the openstack-health docs should tell you how to run the server | 14:01 |
| *** ykarel has quit IRC | 14:01 | |
| mtreinish | for credentials fungi is right, query/query at logstash.openstack.org | 14:01 |
| arxcruz | fungi: cool, I'll keep an eye on that :) | 14:01 |
| *** Julien-zte has joined #openstack-infra | 14:02 | |
| mtreinish | arxcruz: I have a patch adding detailed firehose schema docs up for review now | 14:02 |
| fungi | arxcruz: and similarly, when the subunit2sql workers find out that they should analyze the logs, there are similar events injected into the firehose | 14:02 |
| mtreinish | arxcruz: https://review.openstack.org/#/c/454862/ and https://review.openstack.org/#/c/454911/2 | 14:02 |
| arxcruz | yeah, I know the query/query, but the health.o.o have what I want already in json format, just waiting to be consumed :D | 14:03 |
| *** Julien-zte has quit IRC | 14:03 | |
| fungi | mtreinish: so do you want the daemon started on subunit-worker01 now? i can do that if so | 14:03 |
| *** claudiub|3 has joined #openstack-infra | 14:03 | |
| mtreinish | fungi: yeah lets give that a try | 14:03 |
| *** Julien-zte has joined #openstack-infra | 14:03 | |
| *** claudiub|4 has joined #openstack-infra | 14:04 | |
| fungi | mtreinish: just jenkins-subunit-worker-A or all four of them? | 14:04 |
| mtreinish | just A to start is fine | 14:04 |
| fungi | looks like B through D are disabled at boot anyway | 14:04 |
| fungi | AttributeError: 'dict' object has no attribute 'mqtt-host' | 14:05 |
| fungi | missing configuration? | 14:05 |
| mtreinish | hmm, did it not get the gettattr fix? | 14:05 |
| *** jaosorior_away is now known as jaosorior_mtg | 14:05 | |
| mtreinish | https://review.openstack.org/#/c/456811/ should have fixed that | 14:06 |
| fungi | subunit2sql==1.8.0 # git sha e6b5cab | 14:06 |
| *** claudiub has quit IRC | 14:06 | |
| *** rbrndt has joined #openstack-infra | 14:06 | |
| mtreinish | it's in the puppet repo | 14:06 |
| *** felipemonteiro has joined #openstack-infra | 14:07 | |
| *** claudiub|2 has quit IRC | 14:07 | |
| *** claudiub has joined #openstack-infra | 14:07 | |
| *** claudiub|2 has joined #openstack-infra | 14:07 | |
| *** EricGonczer_ has quit IRC | 14:07 | |
| fungi | ahh, digging | 14:08 |
| fungi | interesting... | 14:08 |
| fungi | the calls in /etc/puppet/modules/subunit2sql/files/subunit-gearman-worker.py are the fixed .get() | 14:09 |
| *** EricGonczer_ has joined #openstack-infra | 14:09 | |
| fungi | but in :q | 14:09 |
| fungi | grrr | 14:09 |
| fungi | but in /usr/local/bin/subunit-gearman-worker.py they're still getattr() | 14:09 |
| *** Julien-zte has quit IRC | 14:09 | |
| pabelanger | morning | 14:09 |
| fungi | i wonder if we dropped or otherwise broke the file installation | 14:10 |
| *** claudiub|3 has quit IRC | 14:10 | |
| fungi | Could not find data item mqtt_service_user_password in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:490 on node subunit-worker01.openstack.org | 14:10 |
| fungi | seeing that in syslog | 14:10 |
| *** claudiub|4 has quit IRC | 14:10 | |
| *** Julien-zte has joined #openstack-infra | 14:11 | |
| fungi | i bet we need to copy that into the right fqdn | 14:11 |
| *** smarcet has joined #openstack-infra | 14:11 | |
| fungi | or group | 14:11 |
| mtreinish | ah ok, yeah I don't know how the hiera setup was done. That's mostly a black box to me since I only ever pull out of it in site.pp | 14:12 |
| fungi | yeah, we have it in group/logstash-worker.yaml right now but not group/subunit-worker.yaml | 14:12 |
| fungi | i'll copy it over | 14:12 |
| fungi | okay, added. tailing syslog to see if it manages to get through the manifest on that server this time around | 14:15 |
| mtreinish | fungi: we might want to make that heira key global. (if there is a group for that) Because we're going to have to add the password to all the remote servers that want to push mqtt | 14:16 |
| *** jcoufal has joined #openstack-infra | 14:16 | |
| mtreinish | which will basically be everything when https://review.openstack.org/#/c/458248/ lands | 14:16 |
| mtreinish | although I guess that's just puppet master, since the remote ansible isn't pushing the mqtt | 14:17 |
| mtreinish | so not everything, but another server :) | 14:17 |
| fungi | someone with their head deeper in our puppet data model would need to chime in... i know we have a global public hiera but not sure if we have a global private hiera (it's all broken up by fqdn and group afaik, and i'm not sure if a given server can be in multiple groups) | 14:17 |
| *** shardy_afk is now known as shardy | 14:17 | |
| mtreinish | yeah, I dunno either. I guess as long as we're aware of it it doesn't really matter too much | 14:20 |
| *** bnemec is now known as beekneemech | 14:20 | |
| pabelanger | clarkb: https://review.openstack.org/#/q/topic:unpuppet-zuul-workers+status:open is ready for another pass. You have already +2 most before, but lost vote on new patchsets | 14:20 |
| *** bauzas is now known as bauwser | 14:21 | |
| *** yamahata has joined #openstack-infra | 14:21 | |
| *** Julien-zte has quit IRC | 14:24 | |
| *** rcernin has quit IRC | 14:24 | |
| *** Julien-zte has joined #openstack-infra | 14:25 | |
| *** rfolco has joined #openstack-infra | 14:26 | |
| dmsimard | pabelanger: mtreinish pointed out https://github.com/ansible/ansible/pull/23769 to me | 14:26 |
| dmsimard | and how it could possibly interact with ara | 14:26 |
| dmsimard | I believe we discussed the usage of a bus for ara before | 14:26 |
| mtreinish | dmsimard: fwiw, there is an infra patch doing it to: https://review.openstack.org/458248 | 14:26 |
| *** yamamoto has joined #openstack-infra | 14:27 | |
| *** baoli has quit IRC | 14:28 | |
| dmsimard | mtreinish: so, what it boils down to is that ara (currently) only works as a callback and writes data in a way it can comprehend and parse (sqlalchemy and the model) and then display | 14:28 |
| dmsimard | mtreinish: having ara listening on a bus for incoming messages would require, perhaps, a driver to translate things | 14:28 |
| mtreinish | dmsimard: ah, ok. Well the simple way to work with that, is just write a little script to listen to mqtt and write that to something ara can understand | 14:28 |
| *** salv-orlando has quit IRC | 14:28 | |
| mtreinish | heh probably just calling the sqlalchemy directly | 14:29 |
| *** baoli has joined #openstack-infra | 14:29 | |
| dmsimard | mtreinish: yeah, I mean, it's not super complicated https://github.com/openstack/ara/blob/master/ara/plugins/callbacks/log_ara.py | 14:29 |
| *** zz_dimtruck is now known as dimtruck | 14:29 | |
| dmsimard | but there /is/ some stuff that ARA provides itself that's not provided by ansible out of the box | 14:29 |
| pabelanger | dmsimard: ya, I was going to test with fedmsg locally, since it already supported a sqlalchemy collector. See what data ara needed, and see if it could just read from that | 14:29 |
| *** cshastri has quit IRC | 14:30 | |
| dmsimard | so what I'll do is review that mqtt patch and see if there's anything missing that ara would need | 14:30 |
| *** rfolco has quit IRC | 14:30 | |
| pabelanger | dmsimard: http://www.fedmsg.com/en/latest/deployment/#store-all-messages was the approach I was going to test | 14:31 |
| dmsimard | pabelanger: the schema is not very complicated, I guess the biggest hurdle would probably be serialization and compression/decompression | 14:31 |
| dmsimard | i.e, zipping and unzipping task results and files (to save on database space) | 14:32 |
| mtreinish | dmsimard: yeah I was just looking at that. The things that stood out to me were just some metadata on tasks and I don't think I had anything emitted on task start either | 14:32 |
| dmsimard | that's provided by the model, though, so it should be possible to use it - I don't know | 14:32 |
| *** Julien-zte has quit IRC | 14:32 | |
| *** amotoki has joined #openstack-infra | 14:33 | |
| dmsimard | mtreinish: yeah, we store the task on start and just mark the completion and status when it's done, basically | 14:33 |
| dmsimard | there's a built-in method to mark start and end times | 14:33 |
| *** Julien-zte has joined #openstack-infra | 14:33 | |
| *** baoli has quit IRC | 14:34 | |
| dmsimard | there's also host facts and file storage, I guess | 14:34 |
| *** baoli has joined #openstack-infra | 14:34 | |
| mtreinish | dmsimard: ah ok, thats good to know. I was kinda grasping at straws when I wrote it. I was having a hard time finding examples of everything availble inside the callback methods | 14:34 |
| *** baoli has quit IRC | 14:35 | |
| *** mriedem has left #openstack-infra | 14:35 | |
| *** mriedem has joined #openstack-infra | 14:35 | |
| dmsimard | mtreinish: I like to think that ARA exploits almost all of the features out of the callback stuff | 14:35 |
| dmsimard | but I did need to reverse engineer some obscure things | 14:36 |
| *** amotoki has quit IRC | 14:36 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: move test_disable_at to simple_layout https://review.openstack.org/458252 | 14:38 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move check_ignore_dependencies to simple_layout https://review.openstack.org/458255 | 14:39 |
| *** amotoki has joined #openstack-infra | 14:39 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move test_repo_deleted to simple_layout https://review.openstack.org/458269 | 14:39 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove org_unknown git repo from single-client test config https://review.openstack.org/458270 | 14:39 |
| dmsimard | mtreinish: are mqtt messages timestamped by default ? | 14:40 |
| fungi | mtreinish: jenkins-subunit-worker-A is running on subunit-worker01 now | 14:40 |
| mtreinish | dmsimard: no | 14:40 |
| pabelanger | hmm | 14:40 |
| mtreinish | fungi: ok cool, I'll listen on firehose and see if the message pop up | 14:41 |
| pabelanger | 2017-04-21 14:18:38,575 INFO nodepool.image.build.ubuntu-trusty: Usage: /etc/init.d/puppet {start|stop|status|restart|force-reload|reload} | 14:41 |
| pabelanger | seeing that in nodepool-builder this morning | 14:41 |
| *** gongysh has joined #openstack-infra | 14:41 | |
| *** vsaienko has quit IRC | 14:42 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move semaphore tests to their own class https://review.openstack.org/458584 | 14:42 |
| *** Julien-zte has quit IRC | 14:42 | |
| *** adarazs has quit IRC | 14:42 | |
| *** slagle has quit IRC | 14:42 | |
| *** cschwede has quit IRC | 14:43 | |
| *** stevebaker has quit IRC | 14:44 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move timer tests to commitConfigUpdate https://review.openstack.org/458599 | 14:44 |
| *** Julien-zte has joined #openstack-infra | 14:44 | |
| dmsimard | mtreinish: I'm not sure if you'd be open to that but for example, ARA uses the same configuration scheme as ansible: https://github.com/openstack/ara/blob/master/ara/config.py ( from https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py ). This gets you "for free" env variables and, for example, config through a [mqtt] block in any ansible.cfg location | 14:46 |
| *** armax has joined #openstack-infra | 14:46 | |
| *** vsaienko has joined #openstack-infra | 14:47 | |
| mtreinish | dmsimard: yeah I could do that. It ends up being the same thing more or less | 14:47 |
| *** marst has joined #openstack-infra | 14:47 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Use requests-mock for all the attach/detach/delete tests https://review.openstack.org/458100 | 14:48 |
| *** dizquierdo has joined #openstack-infra | 14:48 | |
| *** vsaienko has quit IRC | 14:49 | |
| dmsimard | mtreinish: yeah, just probably more user friendly | 14:49 |
| dmsimard | i.e, they can keep everything in their ansible.cfg file | 14:49 |
| mtreinish | dmsimard: yeah I agree | 14:49 |
| mtreinish | heh, it comes back to how I was looking at the other callback plugins to figure out how to write one. I think most just called getenv directly | 14:50 |
| *** Julien-zte has quit IRC | 14:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove commitLayoutUpdate https://review.openstack.org/458648 | 14:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move two more tests to commitConfigUpdate https://review.openstack.org/458647 | 14:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix race in test_queue_rate_limiting https://review.openstack.org/458863 | 14:51 |
| mtreinish | fungi: hmm, I haven't seen any subunit-worker mqtt message yet | 14:51 |
| *** Julien-zte has joined #openstack-infra | 14:51 | |
| pabelanger | cmurphy: just looking at https://review.openstack.org/#/c/455016 I don't think we stop puppet on fedora / centos now. We only added logic for ubuntu | 14:51 |
| fungi | mtreinish: /var/log/subunit2sql/subunit-worker-A-debug.log is silent after this from >10 minutes ago: | 14:52 |
| fungi | 2017-04-21 14:41:28,941 Sending packet to <gear.Connection 0x2a4da10 host: logstash.openstack.org port: 4730>: <gear.Packet 0x30c7950 type: WORK_COMPLETE handle: H:127.0.0.1:36902486> | 14:52 |
| *** john-dav_ is now known as john-davidge | 14:52 | |
| mtreinish | fungi: hmm, did the config for mqtt get updated? | 14:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Switch to update-rc.d for ubuntu-trusty https://review.openstack.org/458749 | 14:53 |
| *** yamamoto has quit IRC | 14:53 | |
| mtreinish | I might have missed it in the noise (I forgot to filter my client to gearman-worker/#) | 14:53 |
| pabelanger | remote: https://review.openstack.org/458749 Switch to update-rc.d for ubuntu-trusty | 14:53 |
| pabelanger | cmurphy: also^ | 14:53 |
| pabelanger | can confirm on nb01 / nb02 this morning | 14:53 |
| *** chlong_ has joined #openstack-infra | 14:54 | |
| mtreinish | and it might just not have gotten any other work from gearman since that last message | 14:54 |
| *** chlong_ has quit IRC | 14:54 | |
| fungi | mtreinish: this is what we have in /etc/subunit2sql/jenkins-subunit-worker-A.yaml right now (with the mqtt-pass value redacted of course): http://paste.openstack.org/show/607474/ | 14:55 |
| *** markus_z has quit IRC | 14:56 | |
| mordred | fwiw - we made a release of shade yesterday which improved an end-user's experience creating a server - the process went from 3 minutes and 35 seconds in 1.19.0 to 35 seconds in 1.20.0 | 14:56 |
| fungi | mordred: wow! what was most of that impact from? | 14:58 |
| pabelanger | cmurphy: actually, the more I think about it. We can just use the dib-init-script element for diskimages to disable puppet for now | 14:58 |
| *** jaosorior_mtg is now known as jaosorior | 14:58 | |
| pabelanger | clarkb: then we can remove that logic from install_puppet.sh | 14:58 |
| *** xarses_ has joined #openstack-infra | 14:58 | |
| *** jaosorior has quit IRC | 14:59 | |
| *** florianf has quit IRC | 14:59 | |
| mtreinish | fungi: that looks correct to me | 14:59 |
| *** Julien-zte has quit IRC | 14:59 | |
| mtreinish | oh except maybe the quotes on the path for the ca certs path | 15:00 |
| *** andreww has joined #openstack-infra | 15:00 | |
| fungi | mtreinish: if it's being parsed as yaml, should be fine either way | 15:00 |
| fungi | though we should probably drop them for consistency | 15:00 |
| mtreinish | yeah it goes through yaml.load() | 15:00 |
| *** Julien-zte has joined #openstack-infra | 15:01 | |
| mtreinish | then I guess I'll keep listening on firehose.o.o and hope something pops up eventually | 15:01 |
| mtreinish | I probably should have added logging for the mqtt messages | 15:01 |
| fungi | i'm still tailing the debug log | 15:01 |
| *** mdrabe has quit IRC | 15:02 | |
| mtreinish | fungi: heh, we just need more gate jobs to finish :) | 15:02 |
| fungi | mtreinish: should i stop it on subunit-worker02 for now? | 15:02 |
| *** flepied has quit IRC | 15:02 | |
| fungi | maybe that way we up our chances | 15:02 |
| fungi | though the last event in its debug log was 2017-04-21 14:42:11 so really not any better | 15:03 |
| mtreinish | fungi: yeah that's a good idea. That'll force all new events to go to subunit-worker01 | 15:03 |
| *** yamahata has quit IRC | 15:03 | |
| fungi | looks like there's a neutron change about to merge | 15:03 |
| *** xarses_ has quit IRC | 15:03 | |
| *** yamahata_ has joined #openstack-infra | 15:03 | |
| mtreinish | ooh | 15:03 |
| * mtreinish sits on the edge of his seat | 15:04 | |
| fungi | okay, stopped jenkins-subunit-worker-A on subunit-worker02 for now | 15:04 |
| *** ykarel has joined #openstack-infra | 15:04 | |
| fungi | last job for that change just passed, so zuul should merge it momentarily | 15:05 |
| *** ijw has joined #openstack-infra | 15:07 | |
| *** annegentle has joined #openstack-infra | 15:07 | |
| *** slagle has joined #openstack-infra | 15:07 | |
| *** mat128 has quit IRC | 15:07 | |
| mtreinish | fungi: anything on the worker log? I haven't seen any mqtt go through | 15:08 |
| fungi | mtreinish: that neutron change merged several minutes ago but the debug log on subunit-worker01 still hasn't awoken | 15:08 |
| mtreinish | hmm, zmq -> gearman delay? | 15:09 |
| *** MasterOfBugs has joined #openstack-infra | 15:09 | |
| fungi | maybe... i've still got it tailing in the bg while i catch up on other stuff | 15:09 |
| mtreinish | ok | 15:09 |
| *** amotoki has quit IRC | 15:09 | |
| *** Julien-zte has quit IRC | 15:10 | |
| *** mdrabe has joined #openstack-infra | 15:10 | |
| *** MasterOfBugs has quit IRC | 15:10 | |
| *** MasterOfBugs has joined #openstack-infra | 15:11 | |
| *** aarefiev is now known as aarefiev_afk | 15:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Include two transitive dependencies to work around conflicts https://review.openstack.org/458875 | 15:12 |
| *** Julien-zte has joined #openstack-infra | 15:12 | |
| *** MasterOfBugs has quit IRC | 15:12 | |
| mordred | fungi: when return a Server to a user, we make some other API calls to fill in the info for that server for the user - for intsance, we replace the nova provided addresses dict with port information from neutron | 15:13 |
| *** migi has quit IRC | 15:13 | |
| *** chlong has quit IRC | 15:13 | |
| mordred | fungi: however, we also have operations internal to shade which list and get servers for things like polling to see if the server is ready yet | 15:13 |
| *** migi has joined #openstack-infra | 15:14 | |
| mordred | fungi: _those_ operations were also causing all of the additional "fill in data" calls to be made, even though we can know absolutely in shade that we are not using _any_ of the data they produce | 15:14 |
| mordred | fungi: so we plumbed through a "hey, seriously, don't fill in any extra info" call and then had all of our internal calls that don't need the data pass that flag | 15:14 |
| *** Julien-zte has quit IRC | 15:15 | |
| fungi | heh, nice! | 15:15 |
| mordred | fungi: it was especially bad for that user because they have a large cloud with many servers in it, so the nested loop of calls was ... yuck | 15:15 |
| mordred | I _also_ think I need to write up some better docs and make them obvious for "how to configure caching" | 15:15 |
| openstackgerrit | Sorin Sbârnea proposed openstack-infra/jenkins-job-builder master: Documented default choice https://review.openstack.org/458878 | 15:15 |
| mordred | but this was an easy fix - and honestly much more correct | 15:15 |
| *** mat128 has joined #openstack-infra | 15:16 | |
| *** Julien-zte has joined #openstack-infra | 15:16 | |
| mordred | fungi: while we're on the topic ... https://review.openstack.org/458875 puts in a workaround to shade's requirements.txt to prevent the problems that were reported with conflicting transitive depends | 15:16 |
| *** florianf has joined #openstack-infra | 15:16 | |
| *** Julien-zte has quit IRC | 15:17 | |
| *** Julien-zte has joined #openstack-infra | 15:17 | |
| *** MasterOfBugs has joined #openstack-infra | 15:18 | |
| mordred | fungi: AAAANNNNDDDD - https://review.openstack.org/456394 is a neutron spec to add hostkey fetching - you may find it interesting | 15:19 |
| fungi | ooh! | 15:19 |
| pabelanger | fungi: clarkb: mind a +3 on https://review.openstack.org/#/c/439248/ and https://review.openstack.org/#/c/453325/ | 15:20 |
| pabelanger | removes our dependency from rdoproject to centos CI | 15:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Run neutron scenario multinode nv in nova experimental queue https://review.openstack.org/458676 | 15:21 |
| pabelanger | fungi: clarkb will be same packages, just hosted on different infrastructure | 15:21 |
| pabelanger | fungi: clarkb: which is actually good, since rdoproject.org seems to be having an outage atm :( | 15:21 |
| dmsimard | Just a heads up, there's a major outage at a Red Hat facility right now -- you might hit RDO related failures until it's resolved. Good places to follow updates are http://status.redhat.com/ and http://status.fedoraproject.org/ | 15:21 |
| fungi | yikes | 15:22 |
| pabelanger | wow, didn't know it was that bad | 15:22 |
| pabelanger | status.fedoraproject.org is angry | 15:22 |
| *** mkebich has joined #openstack-infra | 15:22 | |
| mtreinish | that's a lot of red x's | 15:23 |
| *** nadya has quit IRC | 15:24 | |
| *** efoley__ has quit IRC | 15:24 | |
| *** amotoki has joined #openstack-infra | 15:26 | |
| *** esikachev has quit IRC | 15:29 | |
| mordred | we're mirroring RDO currently right? or pointing to it with our proxy-cache layer? | 15:30 |
| *** amotoki has quit IRC | 15:30 | |
| pabelanger | mordred: just reverse proxy cache now | 15:30 |
| pabelanger | should offer some protection | 15:31 |
| mordred | cool | 15:31 |
| openstackgerrit | Merged openstack-infra/system-config master: centos7: don't install EPEL https://review.openstack.org/439248 | 15:31 |
| *** amotoki has joined #openstack-infra | 15:32 | |
| *** e0ne has quit IRC | 15:32 | |
| *** esikachev has joined #openstack-infra | 15:32 | |
| *** gongysh has quit IRC | 15:33 | |
| *** migi has quit IRC | 15:33 | |
| *** migi has joined #openstack-infra | 15:33 | |
| pabelanger | mordred: stable branches should be fine, they would centos repos for RDO things. master will hit our proxy cache | 15:33 |
| pabelanger | we'll find out quickly which jobs aren't using it :) | 15:34 |
| *** Julien-zte has quit IRC | 15:34 | |
| openstackgerrit | Merged openstack-infra/system-config master: centos: use Centos openstack repos, not RDO https://review.openstack.org/453325 | 15:34 |
| *** amotoki has quit IRC | 15:35 | |
| *** Julien-zte has joined #openstack-infra | 15:35 | |
| *** MasterOfBugs has quit IRC | 15:36 | |
| *** baoli has joined #openstack-infra | 15:37 | |
| *** MasterOfBugs has joined #openstack-infra | 15:37 | |
| mordred | pabelanger: :) | 15:37 |
| mordred | clarkb: you may also want to look at the https://review.openstack.org/456394 neutron spec since you're opinionated if you haven't already (but I just updated it) | 15:38 |
| cmurphy | pabelanger: I don't see a dib-init-script element, do you mean https://docs.openstack.org/developer/diskimage-builder/elements/dib-init-system/README.html ? i'm not sure how that's much different since we want to remove it soon anyway | 15:38 |
| pabelanger | cmurphy: ya, dib-init-system. I'm actually testing a patch now | 15:39 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: OSA: Enable cross-repo testing for integration testing https://review.openstack.org/458890 | 15:39 |
| pabelanger | but agree, hopefully we'll remove this all next week | 15:39 |
| *** Julien-zte has quit IRC | 15:40 | |
| *** dprince has quit IRC | 15:41 | |
| *** baoli has quit IRC | 15:41 | |
| *** Julien-zte has joined #openstack-infra | 15:41 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: cookiecutter need '-f' option https://review.openstack.org/334372 | 15:42 |
| *** rbrndt_ has joined #openstack-infra | 15:42 | |
| *** mhickey has quit IRC | 15:42 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: explain how to install git-review on Arch Linux https://review.openstack.org/388036 | 15:43 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_queue_rate_limiting https://review.openstack.org/458863 | 15:44 |
| *** esikachev has quit IRC | 15:44 | |
| *** ildikov is now known as coffee_cat | 15:45 | |
| *** rbrndt has quit IRC | 15:45 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Fix creators doc about launchpad https://review.openstack.org/393651 | 15:45 |
| *** Julien-zte has quit IRC | 15:45 | |
| mtreinish | fungi: hmm, anything in the worker log yet? I still haven't seen any mqtt from it | 15:45 |
| *** MasterOfBugs has quit IRC | 15:45 | |
| fungi | nada | 15:46 |
| fungi | now getting worried | 15:46 |
| *** tphummel has joined #openstack-infra | 15:46 | |
| *** Julien-zte has joined #openstack-infra | 15:47 | |
| *** amotoki has joined #openstack-infra | 15:48 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: logs: chmod 755 all workspace directories https://review.openstack.org/458895 | 15:49 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move two more tests to commitConfigUpdate https://review.openstack.org/458647 | 15:49 |
| *** Julien-zte has quit IRC | 15:49 | |
| mtreinish | fungi: :( my bet is one of my changes broke something somwhere (that always has the best odds) | 15:49 |
| clarkb | pabelanger: is centos rebuilds of rdo packages something we want long term or just for this outage? | 15:50 |
| *** MasterOfBugs has joined #openstack-infra | 15:50 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/diskimage-builder master: diskimage-builder: Use correct path for element-info https://review.openstack.org/458896 | 15:50 |
| fungi | i assumed longer-term, but great question | 15:50 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Include information for translating ReactJS projects https://review.openstack.org/426211 | 15:50 |
| clarkb | where would someone normally consuming rdo source their pacakges from might be another way to ask that question :) | 15:51 |
| pabelanger | clarkb: I think long term, that's where stable packages are hosted today | 15:51 |
| *** yamamoto has joined #openstack-infra | 15:51 | |
| *** jcoufal_ has joined #openstack-infra | 15:51 | |
| *** Julien-zte has joined #openstack-infra | 15:52 | |
| clarkb | pabelanger: and beneift to us is we mirror centos rebuilds? | 15:52 |
| *** matrohon_ has quit IRC | 15:52 | |
| pabelanger | clarkb: right, we get them for free with our centos-7 mirror too | 15:52 |
| pabelanger | http://mirror.dfw.rax.openstack.org/centos/7/cloud/x86_64/openstack-ocata/ | 15:52 |
| clarkb | cool | 15:53 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add a note that submodules are not supported https://review.openstack.org/440724 | 15:54 |
| openstackgerrit | Merged openstack-infra/infra-manual master: No longer ask to add openstack-release https://review.openstack.org/441823 | 15:54 |
| *** dansmith is now known as superdan | 15:55 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add UI Cookiecutter into template selections https://review.openstack.org/444254 | 15:55 |
| *** jcoufal has quit IRC | 15:55 | |
| *** Julien-zte has quit IRC | 15:55 | |
| johnsom | Not sure if you folks own the "openstack-dev" mailing list, but e-mails to "openstack-owner@lists.openstack.org" are bouncing. This is the "if you have a problem with openstack-dev" email address. | 15:56 |
| *** Julien-zte has joined #openstack-infra | 15:56 | |
| *** lyarwood is now known as me | 15:56 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Split out section about finding post-queue logs https://review.openstack.org/452824 | 15:56 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 15:56 |
| *** me is now known as Guest41383 | 15:57 | |
| pabelanger | clarkb: fungi: care to review, +3 https://review.openstack.org/#/c/458749/ fixes ubuntu-trusty builds for us | 15:57 |
| *** MasterOfBugs has quit IRC | 15:57 | |
| *** Guest41383 is now known as lyarwood | 15:57 | |
| clarkb | johnsom: its administrators are listed at the bottom of http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev | 15:58 |
| clarkb | johnsom: we run the server though | 15:58 |
| *** david-lyle has joined #openstack-infra | 15:58 | |
| clarkb | johnsom: also I don't see where openstack-owner@ is listed? | 15:58 |
| fungi | johnsom: where are you seeing that address? | 15:58 |
| johnsom | https://www.irccloud.com/pastebin/ptyJAAxF/ | 15:59 |
| fungi | i can't find it in any message from the openstack-dev ml (headers or footer) | 15:59 |
| johnsom | I got this strange "you aren't subscribed" email | 15:59 |
| johnsom | Though I am and it looks like the e-mail went through | 15:59 |
| fungi | oh, i bet that's a typo | 15:59 |
| fungi | we'll need one of the admins for that list to correct the rejection e-mail | 16:00 |
| *** salv-orlando has joined #openstack-infra | 16:00 | |
| fungi | it's a freeform field | 16:00 |
| *** Julien-zte has quit IRC | 16:00 | |
| fungi | wait, that's not for openstack-dev | 16:00 |
| *** Julien-zte has joined #openstack-infra | 16:00 | |
| fungi | johnsom: did you try sending to openstack@lists.openstack.org instead of openstack-dev@lists.openstack.org? that looks like a rejection for the general ml, not the dev ml | 16:01 |
| clarkb | pabelanger: done | 16:01 |
| johnsom | Ah, I see now, that email was cc'd to another list. | 16:01 |
| johnsom | Yeah, it was the cc to the openstack list | 16:01 |
| pabelanger | clarkb: danke | 16:01 |
| *** beslemon has joined #openstack-infra | 16:02 | |
| fungi | johnsom: okay. and when you tried to subsequently contact openstack-owner@ that also bounced back? | 16:02 |
| fungi | normally it should have gone to the admins for the general ml | 16:02 |
| johnsom | https://www.irccloud.com/pastebin/6nUHzALa/ | 16:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable puppet service on boot https://review.openstack.org/458906 | 16:03 |
| *** tie has quit IRC | 16:03 | |
| pabelanger | cmurphy: I was thinking something like ^. I still need to test on ubuntu-trusty and fedora-25 | 16:03 |
| *** salv-orlando has quit IRC | 16:04 | |
| *** salv-orlando has joined #openstack-infra | 16:05 | |
| cmurphy | pabelanger: that's way more LOC :P | 16:05 |
| pabelanger | cmurphy: stackalytics :D | 16:06 |
| cmurphy | ha | 16:06 |
| pabelanger | I only thought it if, because I don't think we now disable puppet for fedora / centos | 16:06 |
| pabelanger | so, we can add that to install_puppet.sh or DIB element | 16:07 |
| *** felipemonteiro__ has quit IRC | 16:07 | |
| cmurphy | systemctl disable puppet doesn't work on fedora/centos? | 16:07 |
| *** alexchadin has quit IRC | 16:07 | |
| clarkb | pabelanger: we don't need to there because rpm/yum/redhat rules are to not enable services aiui | 16:07 |
| clarkb | we could however just sytemctl and be safe | 16:07 |
| cmurphy | oh that too | 16:07 |
| *** esikachev has joined #openstack-infra | 16:07 | |
| pabelanger | ah, right | 16:07 |
| pabelanger | okay, so maybe this isn't an issue | 16:08 |
| *** EricGonczer_ has quit IRC | 16:09 | |
| *** smarcet has quit IRC | 16:09 | |
| *** EricGonczer_ has joined #openstack-infra | 16:10 | |
| *** jpich has quit IRC | 16:10 | |
| *** dizquierdo has quit IRC | 16:11 | |
| pabelanger | clarkb: think we can approve the last round of unpuppet? | 16:12 |
| fungi | johnsom: though maybe we'd broken -owner@ addresses for the lists with the recent mailman upgrade, but they're definitely still being matched in our mailman_router in exim... i'll need to do a little more testing to track this down but thanks for bringing it to our attention! | 16:16 |
| fungi | s/though/thought/ | 16:16 |
| *** jascott1 has joined #openstack-infra | 16:16 | |
| johnsom | fungi Sure, NP. Let me know if I can help (test emails or whatever) | 16:17 |
| mordred | fungi: armax was asking questions about the important of hostkeys if auth is done via ssh keypair auth - and linked to this: http://www.gremwell.com/ssh-mitm-public-key-authentication | 16:22 |
| mordred | fungi: which indicates there is a session id that a part of the secure channel handshake which means a MITM couldn't relay the server's key challenge to the client in a way the client would accept | 16:23 |
| *** katyafervent_awa has quit IRC | 16:23 | |
| *** HeOS has quit IRC | 16:23 | |
| *** HeOS has joined #openstack-infra | 16:23 | |
| mordred | fungi: if that is true it changes my understanding of hostkeys and their importance - but I'd like someone who is not me to also look/verify? | 16:24 |
| mordred | (I mean, it's basically saying that if you use keypairs and not passwords that hostkeys are essentially meaningless) | 16:24 |
| fungi | mordred: so that makes mitm hard for key-only auth, but doesn't solve the possibility of masquerading as the server | 16:24 |
| *** ramishra has quit IRC | 16:24 | |
| *** andreww is now known as xarses_ | 16:25 | |
| *** ramishra has joined #openstack-infra | 16:26 | |
| *** annegentle has quit IRC | 16:26 | |
| mordred | fungi: yah - and then grabbing your password when you try to sudo do something or something | 16:26 |
| *** katyafervent_awa has joined #openstack-infra | 16:26 | |
| openstackgerrit | Chi Lo proposed openstack-infra/project-config master: Create neutron lbaas-specific patrole gate https://review.openstack.org/456390 | 16:27 |
| *** ihrachys has quit IRC | 16:27 | |
| fungi | so i agree the use case is slightly less critical if you're relying on key-only auth and disabling password login for your accounts or in your sshd_config but there is still some risk | 16:27 |
| *** annegentle has joined #openstack-infra | 16:27 | |
| *** ihrachys has joined #openstack-infra | 16:27 | |
| mordred | yah. and also only doing password-less sudo too | 16:27 |
| mordred | and not allowing agent forwarding | 16:28 |
| fungi | yep | 16:28 |
| fungi | defense in depth | 16:28 |
| *** jkilpatr has quit IRC | 16:28 | |
| *** jkilpatr_ has joined #openstack-infra | 16:28 | |
| *** hashar has quit IRC | 16:29 | |
| *** dprince has joined #openstack-infra | 16:29 | |
| *** trown is now known as trown|lunch | 16:30 | |
| *** beekneemech has quit IRC | 16:30 | |
| *** esberglu has quit IRC | 16:30 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode https://review.openstack.org/454152 | 16:31 |
| *** beslemon_ has joined #openstack-infra | 16:33 | |
| *** beslemon has quit IRC | 16:33 | |
| *** gyee has joined #openstack-infra | 16:36 | |
| *** lucasagomes is now known as lucas-afk | 16:36 | |
| pabelanger | fungi: mordred: do you mind a +3 to fix ubuntu-trusty builds, we are currently looping: https://review.openstack.org/#/c/458749/ | 16:38 |
| mordred | pabelanger: looking | 16:38 |
| *** baoli has joined #openstack-infra | 16:38 | |
| *** amotoki has quit IRC | 16:39 | |
| openstackgerrit | Nicolas Helgeson proposed openstack-infra/project-config master: Add bandit-baseline job for Murano https://review.openstack.org/457744 | 16:39 |
| pabelanger | http://nb01.openstack.org/dib.ubuntu-trusty.log for the log files of failure | 16:41 |
| fungi | heh, i was just in the process of reviewing that one | 16:42 |
| fungi | double-checking that if you remove the service it won't be automagically replaced on the next package upgrade (as opposed to setting it to not start at any runlevel) | 16:42 |
| pabelanger | ya, I was thinking about that too | 16:42 |
| pabelanger | maybe we should set it to disable | 16:42 |
| pabelanger | over remove | 16:42 |
| *** baoli has quit IRC | 16:43 | |
| fungi | remove is typically used in package post-remove scripts | 16:43 |
| *** adarazs has joined #openstack-infra | 16:43 | |
| fungi | (also update-rc.d, like invoke-rc.d, isn't _really_ seen as a user-facing cli tool, it's generally just used by package maintainer scripts) | 16:44 |
| *** rkukura has joined #openstack-infra | 16:44 | |
| *** bnemec has joined #openstack-infra | 16:44 | |
| fungi | but in dib elements we're sort of operating like a maintscript so that seems like an okayish tool | 16:45 |
| pabelanger | I think update-rc.d -f puppet disable is safer, reading google | 16:45 |
| *** bnemec is now known as beekneemech | 16:45 | |
| pabelanger | I think puppet will use update-rc.d too | 16:46 |
| *** shrircha has joined #openstack-infra | 16:46 | |
| *** adarazs has quit IRC | 16:47 | |
| fungi | wanna propose that alternative (i guess just as a new patchset on 458749)? | 16:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Switch to update-rc.d for ubuntu-trusty https://review.openstack.org/458749 | 16:50 |
| *** shardy has quit IRC | 16:50 | |
| fungi | i'm not finding anything to contradict my recollection that remove is stateless, so the next time the package is upgraded the maintscripts have no idea they're not supposed to install the initscript and default start symlinks anew | 16:50 |
| fungi | testing locally | 16:50 |
| *** shrircha has quit IRC | 16:50 | |
| clarkb | ah so we hae to use the default file? | 16:51 |
| *** dmellado has quit IRC | 16:52 | |
| *** dalvarez has quit IRC | 16:53 | |
| *** _ari_ has quit IRC | 16:53 | |
| *** pabelanger has quit IRC | 16:54 | |
| *** dmanchad has quit IRC | 16:54 | |
| *** weshay has quit IRC | 16:54 | |
| *** migi has quit IRC | 16:54 | |
| *** wznoinsk has quit IRC | 16:54 | |
| *** ijw has quit IRC | 16:54 | |
| *** rkukura has quit IRC | 16:54 | |
| *** ijw has joined #openstack-infra | 16:55 | |
| *** yamamoto has quit IRC | 16:55 | |
| *** Apoorva has joined #openstack-infra | 16:55 | |
| clarkb | we lost pabelanger, more red hat hosting issues? anyways I need caffeiene and food then I will rereview the puppet/dib changes | 16:56 |
| *** florianf has quit IRC | 16:56 | |
| *** rkukura has joined #openstack-infra | 16:59 | |
| *** ijw has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| *** dalvarez has joined #openstack-infra | 17:00 | |
| *** salv-orlando has quit IRC | 17:00 | |
| fungi | confirmed... (on a debian system) i did update-rc.d -f someservice remove and confirmed the symlinks were gone from the /etc/rc?.d directories, then apt-get upgraded since there was a new version of the package providing that service, and the symlinks all magically reappeared | 17:00 |
| *** migi has joined #openstack-infra | 17:00 | |
| *** pabelanger has joined #openstack-infra | 17:00 | |
| *** dmanchad has joined #openstack-infra | 17:01 | |
| *** weshay has joined #openstack-infra | 17:01 | |
| fungi | so "remove" is definitely not what you want for this case. disable is | 17:01 |
| tosky | clarkb: still the same issue (RH), details still unclear | 17:02 |
| *** wznoinsk has joined #openstack-infra | 17:02 | |
| *** _ari_ has joined #openstack-infra | 17:02 | |
| EmilienM | fungi, pabelanger: i'm sure you're aware about https://twitter.com/RedHatSupport/status/855445861211549696 but just in case: tripleo RH1 is unreachable | 17:04 |
| fungi | EmilienM: i figured it was offline, yeah | 17:04 |
| fungi | though i didn't explicitly check | 17:04 |
| EmilienM | ok | 17:05 |
| EmilienM | so yeah, things are bad for us right now | 17:06 |
| pabelanger | Ya, was looking at http://grafana.openstack.org/dashboard/db/nodepool-tripleo-test-cloud | 17:06 |
| fungi | it's always darkest before the dawn... or something | 17:06 |
| mnaser | on a friday too :( | 17:06 |
| pabelanger | also, my IRC proxy is back :) | 17:06 |
| *** dmellado has joined #openstack-infra | 17:06 | |
| fungi | pabelanger: if you missed it while offline, i manually confirmed with local testing that update-rc.d remove does indeed get undone on package upgrade so disable is defintely the way to go | 17:06 |
| mnaser | we had a few hosts kernel panic with the same reason overnight yesterday so not much of a better friday here, heh | 17:07 |
| pabelanger | fungi: great | 17:07 |
| clarkb | mnaser: all the counters rolled over at the same time? | 17:08 |
| mnaser | clarkb http://paste.openstack.org/show/607491/ -- across many different machines, same call stack | 17:09 |
| *** marst has quit IRC | 17:09 | |
| *** marst_ has joined #openstack-infra | 17:09 | |
| openstackgerrit | Merged openstack-infra/shade master: Use requests-mock for all the attach/detach/delete tests https://review.openstack.org/458100 | 17:09 |
| *** tosky has quit IRC | 17:10 | |
| AJaeger | mnaser: any idea what causes this? | 17:11 |
| mnaser | AJaeger https://github.com/torvalds/linux/commit/54d27365cae88fbcc853b391dcd561e71acb81fa | 17:11 |
| mnaser | i remember having to deal with this on ubuntu, but we switched to centos a while back (ok maybe over a year) | 17:11 |
| mnaser | i think it has to do with giving cgroup quotas for VMs during runtime (rather than nova setting it in the libvirt xml) | 17:11 |
| AJaeger | interesting. thanks, mnaser | 17:12 |
| mnaser | i havent had a chance to do some more investigating but i *think* newer kernels might not be affected (older ones seem to be the ones which crashed).. but fun kernel troubleshooting time for a not-easily-reproducable problem ;[ | 17:12 |
| *** Rockyg has joined #openstack-infra | 17:13 | |
| *** trown|lunch is now known as trown | 17:14 | |
| *** timburke has joined #openstack-infra | 17:17 | |
| *** charz has joined #openstack-infra | 17:17 | |
| fungi | pabelanger: heh, this bug report is for exactly the same mistake in update-rc.d usage: https://bugs.debian.org/636851 | 17:18 |
| openstack | Debian bug 636851 in linux-container "make update-rc.d upgrade safe" [Important,Fixed] | 17:18 |
| pabelanger | fungi: neat! | 17:19 |
| *** tesseract has quit IRC | 17:19 | |
| fungi | "Using update-rc.d disable instead of update-rc.d remove (Closes: #636851)." | 17:20 |
| clarkb | do we have a patchset with disable instead of remove? | 17:21 |
| pabelanger | ya, I updated https://review.openstack.org/#/c/458749/ to use disable | 17:21 |
| *** ig0r_ has quit IRC | 17:24 | |
| clarkb | I'm about to compile python2.7 from source using gcc6 and gcc7 then benchmakr them. Not how I expected this day to go :) | 17:24 |
| *** jcoufal_ has quit IRC | 17:25 | |
| fungi | what has driven you to such madness? | 17:25 |
| clarkb | fungi: zuul test slowness | 17:25 |
| fungi | ahh | 17:25 |
| *** ralonsoh has quit IRC | 17:25 | |
| *** jtomasek has quit IRC | 17:26 | |
| *** jcoufal has joined #openstack-infra | 17:26 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch to update-rc.d for ubuntu-trusty https://review.openstack.org/458749 | 17:27 |
| *** jcoufal has quit IRC | 17:28 | |
| pabelanger | yay | 17:29 |
| *** ig0r_ has joined #openstack-infra | 17:30 | |
| *** annegentle has quit IRC | 17:30 | |
| *** ykarel has quit IRC | 17:31 | |
| *** salv-orlando has joined #openstack-infra | 17:34 | |
| clarkb | mordred: remember how drizzle required clean compiles? | 17:35 |
| openstackgerrit | Nicolas Helgeson proposed openstack-infra/project-config master: Add bandit-baseline job for Murano https://review.openstack.org/457744 | 17:35 |
| *** hashar has joined #openstack-infra | 17:37 | |
| *** hashar is now known as hasharAway | 17:38 | |
| *** baoli has joined #openstack-infra | 17:39 | |
| *** esberglu has joined #openstack-infra | 17:40 | |
| *** dprince has quit IRC | 17:41 | |
| *** esberglu_ has joined #openstack-infra | 17:41 | |
| *** rkukura has quit IRC | 17:41 | |
| *** esberglu has quit IRC | 17:41 | |
| *** ssbarnea has quit IRC | 17:41 | |
| *** ramishra has quit IRC | 17:42 | |
| *** baoli has quit IRC | 17:43 | |
| *** ramishra has joined #openstack-infra | 17:44 | |
| *** rbrndt_ has quit IRC | 17:48 | |
| pabelanger | looks like problems getting to https://download.openvz.org/ from nb02.openstack.org | 17:49 |
| pabelanger | timed out | 17:49 |
| pabelanger | checking nb01.o.o | 17:49 |
| *** ramishra has quit IRC | 17:49 | |
| *** rwsu has quit IRC | 17:51 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove commitLayoutUpdate https://review.openstack.org/458648 | 17:51 |
| *** ramishra has joined #openstack-infra | 17:51 | |
| mordred | clarkb: I do indeed | 17:52 |
| clarkb | mordred: ya I miss that | 17:52 |
| pabelanger | ianw: any objections is I start work on nodepool-builder xenial server in vexxhost? | 17:52 |
| mordred | clarkb: yah. it's one of the things I like about rust ... | 17:53 |
| *** dprince has joined #openstack-infra | 17:53 | |
| *** dprince has quit IRC | 17:53 | |
| *** dprince has joined #openstack-infra | 17:53 | |
| *** tosky has joined #openstack-infra | 17:59 | |
| pabelanger | clarkb: how can I find out what is using http://download.openvz.org/template/precreated/ubuntu-12.04-x86_64.tar.gz | 17:59 |
| pabelanger | or does anybody know | 17:59 |
| clarkb | pabelanger: it was for trove testing way way back | 18:01 |
| clarkb | I don't think it ever went anywhere | 18:01 |
| clarkb | you can likely remove it from devstack so that our image builds stop using it. Note that you ahve to do it on stable branches too iirc | 18:01 |
| pabelanger | I don't see it on master for devstack, maybe dropped | 18:02 |
| pabelanger | let me search history | 18:02 |
| *** EricGonc_ has joined #openstack-infra | 18:03 | |
| *** yamahata_ has quit IRC | 18:04 | |
| *** EricGonczer_ has quit IRC | 18:04 | |
| pabelanger | clarkb: https://review.openstack.org/#/c/435126/ | 18:04 |
| pabelanger | sdague: ^any issues with backporting that to stable branches you can think off? | 18:05 |
| sdague | pabelanger: let me look | 18:05 |
| pabelanger | at lest the dropping of openvz image | 18:05 |
| sdague | so... I don't know if we need to care about ironic on older stable | 18:06 |
| pabelanger | I can skip Ironic, looks like we already download that image with libvirt | 18:07 |
| sdague | actually, if it doesn't merge conflict, it's probably ok | 18:07 |
| sdague | pabelanger: I'd say push the backports | 18:07 |
| pabelanger | sdague: cool, let me do that now | 18:07 |
| *** ykarel has joined #openstack-infra | 18:09 | |
| *** dhinesh has joined #openstack-infra | 18:12 | |
| *** ccamacho has left #openstack-infra | 18:12 | |
| *** ccamacho has quit IRC | 18:12 | |
| *** ykarel has quit IRC | 18:14 | |
| *** sambetts is now known as sambetts|afk | 18:15 | |
| *** rkukura has joined #openstack-infra | 18:15 | |
| *** diltram has quit IRC | 18:17 | |
| *** Hal has quit IRC | 18:19 | |
| pabelanger | clarkb: caffeinated enough to review unpuppet patches? | 18:20 |
| *** electrofelix has quit IRC | 18:21 | |
| clarkb | pabelanger: ya and grumpy about python builds enough too | 18:21 |
| *** yamahata_ has joined #openstack-infra | 18:22 | |
| *** beslemon_ has quit IRC | 18:23 | |
| *** cschwede has joined #openstack-infra | 18:24 | |
| jeblair | grum.py | 18:24 |
| pabelanger | ha | 18:24 |
| clarkb | pabelanger: for https://review.openstack.org/#/c/458542/ looks like ianw wanted to hold off approving until the keypair stuff was done | 18:24 |
| clarkb | jeblair: it looks like if you make && make install, you get a working install for about 30 seconds until the make install decides to completely recompile everything again and toast your .so files | 18:25 |
| clarkb | I don't get it | 18:25 |
| pabelanger | clarkb: right, keypairs are live now | 18:25 |
| clarkb | pabelanger: you want to +W that one? | 18:25 |
| pabelanger | sure | 18:25 |
| clarkb | jeblair: so about to try just `make install` and see if thats maybe how they intend the makefile to be used | 18:25 |
| jeblair | clarkb: is that because of the PGO stuff? | 18:26 |
| clarkb | jeblair: no, the pgo stuff happens regardless | 18:26 |
| clarkb | jeblair: you actually see it run each of the test suite tests | 18:26 |
| clarkb | I'm just going to clear out the install dir, make clean, and try a bare `make install` | 18:27 |
| *** stevebaker has joined #openstack-infra | 18:27 | |
| *** rbrndt_ has joined #openstack-infra | 18:29 | |
| *** slagle_ has joined #openstack-infra | 18:29 | |
| *** salv-orlando has quit IRC | 18:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Include two transitive dependencies to work around conflicts https://review.openstack.org/458875 | 18:30 |
| mordred | jeblair: wow. how has that joke (grum.py) never occured to me before now? | 18:31 |
| *** rwsu has joined #openstack-infra | 18:36 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove puppet-sudoers from nodepool diskimages https://review.openstack.org/458542 | 18:36 |
| *** baoli has joined #openstack-infra | 18:39 | |
| *** amotoki has joined #openstack-infra | 18:39 | |
| *** amotoki has quit IRC | 18:44 | |
| *** rkukura has quit IRC | 18:45 | |
| *** rwsu has quit IRC | 18:45 | |
| pabelanger | tripleo-test-cloud-rh1 looks online agian | 18:46 |
| *** Sukhdev has joined #openstack-infra | 18:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Confirm haveged has been enabled on boot https://review.openstack.org/458933 | 18:48 |
| *** ijw has joined #openstack-infra | 18:48 | |
| clarkb | ./python -m test.regrtest --pgo -x test_asyncore test_gdb test_multiprocessing test_subprocess || true <- I like the || true | 18:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Confirm haveged has been enabled on boot https://review.openstack.org/458933 | 18:49 |
| clarkb | pabelanger: is ^ necessary? | 18:50 |
| *** rwsu has joined #openstack-infra | 18:50 | |
| pabelanger | clarkb: not sure, WIP for now | 18:50 |
| pabelanger | need to confirm when we remove puppet-haveged | 18:50 |
| pabelanger | clarkb: your RHEL comment maybe me think about it for services | 18:50 |
| *** esikachev has quit IRC | 18:51 | |
| fungi | for those playing along on the project rename maintenance at 20:00 (in a little over an hour), the lists i'm building the yaml input from are now on https://etherpad.openstack.org/p/project-renames-2017-04-21 | 18:51 |
| fungi | i'll put the yaml together shortly | 18:51 |
| clarkb | I don't get this make file it rebuilds from scratch every time... | 18:51 |
| *** ijw has quit IRC | 18:52 | |
| *** med_ has quit IRC | 18:54 | |
| *** med_ has joined #openstack-infra | 18:54 | |
| mordred | clarkb: that seems to defeat the entire purpose of Make | 18:54 |
| pabelanger | clarkb: are you okay with https://review.openstack.org/#/c/458550 ? | 18:55 |
| *** med_ is now known as Guest76746 | 18:55 | |
| pabelanger | had a previous +2 from you | 18:55 |
| clarkb | mordred: yes thats why I am confused | 18:55 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Configurable SSH access to GitHub https://review.openstack.org/444034 | 18:55 |
| mordred | clarkb: maybe they wanted make to work more like setuptools? | 18:56 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Remove references to the simple_slave class https://review.openstack.org/458936 | 18:59 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Remove certname parameter https://review.openstack.org/458937 | 18:59 |
| cmurphy | more spring cleaning | 18:59 |
| *** rwsu has quit IRC | 19:00 | |
| *** erikmwil_ has joined #openstack-infra | 19:01 | |
| pabelanger | getting close | 19:01 |
| *** dotplus has left #openstack-infra | 19:01 | |
| *** pcaruana has quit IRC | 19:02 | |
| *** esikachev has joined #openstack-infra | 19:03 | |
| *** erikmwilson has quit IRC | 19:04 | |
| *** rossella_s has quit IRC | 19:04 | |
| clarkb | for the number of files thing, that was beacuse making an npm mirror needed that? | 19:04 |
| clarkb | and by moving to server.pp we are still good? | 19:04 |
| *** rossella_s has joined #openstack-infra | 19:05 | |
| fungi | infra-root: i've disabled the puppet run all cronjob on the puppetmaster per step #3 | 19:05 |
| *** e0ne has joined #openstack-infra | 19:06 | |
| pabelanger | clarkb: yes. I plan to keep an eye on it and update DIB elements as needed | 19:06 |
| pabelanger | but, my gut is staying we should be okay | 19:07 |
| pabelanger | famous last words | 19:07 |
| *** andreas-f has joined #openstack-infra | 19:07 | |
| *** rwsu has joined #openstack-infra | 19:07 | |
| *** esikachev has quit IRC | 19:08 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Move afs client to openstack_project::server https://review.openstack.org/458944 | 19:11 |
| clarkb | https://bugs.python.org/issue12271 is affecting me | 19:12 |
| clarkb | which I think leads to failed builds which in turn derps make | 19:12 |
| *** ssbarnea has joined #openstack-infra | 19:13 | |
| mordred | clarkb: maybe we should make a drython? :) | 19:15 |
| clarkb | meh I have applied hack patch for now | 19:16 |
| clarkb | you'd think that would eventually get merged, I guess suse packaging just hacks it in too | 19:16 |
| *** john-davidge has quit IRC | 19:17 | |
| mtreinish | fungi: btw, did you see: http://grafana.openstack.org/dashboard/db/mosquitto-status | 19:18 |
| fungi | ooh! | 19:18 |
| mtreinish | although I realized I need to change which metrics we collect (since the message and data are totals from start instead of samples) | 19:19 |
| fungi | hrm, messages and bytes sent look like we're charting totals rather than averages | 19:19 |
| fungi | heh, yep | 19:19 |
| *** john-davidge has joined #openstack-infra | 19:19 | |
| mtreinish | and I think I made a typo on heap size | 19:19 |
| *** esikachev has joined #openstack-infra | 19:19 | |
| rcarrillocruz | o-k, looks like i'm going to the summit | 19:19 |
| rcarrillocruz | weeeeeeeeeeeeeeeeeeee | 19:19 |
| rcarrillocruz | so happy to meet my openstack infra folks *again* | 19:19 |
| pabelanger | rcarrillocruz: nice | 19:20 |
| *** john-davidge has quit IRC | 19:22 | |
| pabelanger | clarkb: did you want me to add files for 458550 first to DIB? | 19:22 |
| fungi | rcarrillocruz: awesome! glad you can make it | 19:23 |
| rcarrillocruz | ++++ | 19:23 |
| *** esikachev has quit IRC | 19:23 | |
| * rcarrillocruz goes check infra sessions on schedule | 19:23 | |
| fungi | there aren't much in the way of infra-specific sessions. my feeling is we're there more in a capacity to provide feedback on openstack | 19:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Remove install_resolv_conf from dib puppet script https://review.openstack.org/458945 | 19:24 |
| cmurphy | rcarrillocruz: yay \o/ | 19:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Remove install_resolv_conf param from s_u_s https://review.openstack.org/458946 | 19:24 |
| fungi | rcarrillocruz: we have a who/what is infra semi-lightning summary in the shared onboarding session with release/requirements/stable/qa/et cetera if you want to help with that | 19:25 |
| fungi | rcarrillocruz: and there's a talk mtreinish and i are giving in the upstream dev track about firehose | 19:25 |
| fungi | i haven't looked yet to see what other infra-relevant talks are going on, but i'm sure there are some | 19:25 |
| *** esikachev has joined #openstack-infra | 19:26 | |
| rcarrillocruz | sweet, let me find the links, i assume there's an etherpad for that? | 19:26 |
| fungi | rcarrillocruz: for the general conference stuff, the schedule's already posted | 19:26 |
| fungi | rcarrillocruz: https://www.openstack.org/summit/boston-2017/summit-schedule/ | 19:26 |
| fungi | rcarrillocruz: if you want to filter down to just the forum, there's an option under the summit categories filter | 19:27 |
| *** cody-somerville has joined #openstack-infra | 19:28 | |
| rcarrillocruz | ah | 19:29 |
| rcarrillocruz | so forum == old design sessions | 19:29 |
| rcarrillocruz | makes sense | 19:29 |
| *** csomerville has quit IRC | 19:30 | |
| rcarrillocruz | haha | 19:30 |
| rcarrillocruz | 'should we kill stackalytics' | 19:30 |
| rcarrillocruz | that's awesome :D | 19:30 |
| *** esikachev has quit IRC | 19:31 | |
| rcarrillocruz | so yeah, fungi, happy to help on that shared obnoarding session | 19:31 |
| *** john-davidge has joined #openstack-infra | 19:32 | |
| AJaeger | clarkb: https://build.opensuse.org/package/view_file/openSUSE:Leap:42.2/python-base/python-2.6b3-curses-panel.patch?expand=1 ? | 19:33 |
| *** dtroyer_zz has quit IRC | 19:33 | |
| *** dtroyer has joined #openstack-infra | 19:33 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-mqtt_statsd master: Update subscribed mosquitto metrics https://review.openstack.org/458953 | 19:35 |
| clarkb | AJaeger: thats it | 19:35 |
| mtreinish | fungi: ^^^ that should fix the stats | 19:35 |
| pabelanger | fungi: maybe we can plan to stop zuul-launchers for step 4 too | 19:36 |
| *** john-davidge has quit IRC | 19:37 | |
| *** pbourke has quit IRC | 19:37 | |
| *** pbourke has joined #openstack-infra | 19:37 | |
| pabelanger | not sure if that is implied | 19:38 |
| fungi | pabelanger: it's not implied, but you can do that while we have zuul offline i guess | 19:38 |
| fungi | though launchers can also be rotated out live, right? | 19:38 |
| pabelanger | ya, we can do it live. Figured if zuul was stopped it would be a good time too | 19:39 |
| fungi | fine with me | 19:39 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config master: Fix typo in heap_size field for mosquitto grafana https://review.openstack.org/458956 | 19:40 |
| mtreinish | fungi: ^^ that should fix heap_size | 19:40 |
| mtreinish | pabelanger: ^^^ in that file I'm doing something wrong for graphing 2 lines on the same graph. If you have pointers to what I screwed up | 19:40 |
| fungi | #status notice Gerrit will be offline briefly starting at 20:00 for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 19:41 |
| openstackstatus | fungi: sending notice | 19:41 |
| openstackgerrit | Merged openstack-infra/shade master: Remove stray line https://review.openstack.org/458569 | 19:42 |
| -openstackstatus- NOTICE: Gerrit will be offline briefly starting at 20:00 for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 19:42 | |
| AJaeger | mtreinish: +3 for the obvious typo | 19:43 |
| mtreinish | AJaeger: cool, thanks | 19:43 |
| *** john-davidge has joined #openstack-infra | 19:43 | |
| openstackstatus | fungi: finished sending notice | 19:44 |
| pabelanger | fungi: clarkb: think we can sneak https://review.openstack.org/#/c/458550/ in before gerrit downtime? That gives us the time to rebuild images when it comes back online | 19:46 |
| clarkb | pabelanger: I'm just worried on that one because it could be making things pass that we don't expect | 19:46 |
| clarkb | since its fairly global | 19:46 |
| *** esikachev has joined #openstack-infra | 19:47 | |
| *** HeOS has quit IRC | 19:48 | |
| *** stevebaker has quit IRC | 19:48 | |
| *** vsaienko has joined #openstack-infra | 19:48 | |
| *** cschwede has quit IRC | 19:48 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove install_resolv_conf from dib puppet script https://review.openstack.org/458945 | 19:48 |
| pabelanger | clarkb: make things pass? | 19:49 |
| *** Apoorva has quit IRC | 19:49 | |
| clarkb | pabelanger: basically if anything needs more than 2k files or whatever the default limit is | 19:49 |
| *** Hal has joined #openstack-infra | 19:49 | |
| pabelanger | Ya right. I was planning on making the changes in JJB for that. Or is that not okay? | 19:50 |
| pabelanger | figured we'd make it job level, then global | 19:50 |
| clarkb | pabelanger: the problem is it may affect more than what you expect is what I am saying | 19:50 |
| clarkb | like maybe devstack jobs need it nwo for some reason (because stack user opens a lot of files) | 19:50 |
| pabelanger | right, history was related to npm for that change. And pretty recent | 19:51 |
| pabelanger | I would assume devstack users would see the issue local, where they have defaults? | 19:51 |
| *** esikachev has quit IRC | 19:52 | |
| clarkb | ya I am not saying its likely, just that would be the concern | 19:53 |
| *** esikachev has joined #openstack-infra | 19:53 | |
| clarkb | and with gerrit downtime then weekend, maybe we wait for monday? | 19:54 |
| *** baoli has quit IRC | 19:54 | |
| pabelanger | sure | 19:54 |
| fungi | anybody happen to know if /opt/system-config/production/playbooks/rename_repos.yaml needs to run as root on puppetmaster or whether a regular user will do? | 19:54 |
| fungi | oh, nevermind. instructions have sudo | 19:55 |
| *** ig0r_ has quit IRC | 19:55 | |
| fungi | i just had the box scrolled too far to the right to see it | 19:55 |
| clarkb | it will need root in order to ssh | 19:55 |
| clarkb | unless you are agent forwarding | 19:55 |
| clarkb | and have a key that will work | 19:55 |
| fungi | yeah, seems like it should need root | 19:55 |
| fungi | er, seemed | 19:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo in heap_size field for mosquitto grafana https://review.openstack.org/458956 | 19:56 |
| *** e0ne has quit IRC | 19:56 | |
| fungi | any volunteers wanna grab one or more of steps 4-10 on https://etherpad.openstack.org/p/project-renames-2017-04-21 ? | 19:57 |
| fungi | i should probably do step #5 since i already have a copy of the yaml on puppetmaster | 19:57 |
| fungi | thanks jeblair! | 19:57 |
| *** esikachev has quit IRC | 19:58 | |
| fungi | when the clock hits 20 (in another minute) i'll do a status alert about the maintenance | 19:58 |
| *** felipemonteiro has quit IRC | 19:58 | |
| fungi | and then we should be all clear to start | 19:58 |
| jeblair | 19:58 < fungi> when the clock hits 20 (in another minute) | 19:59 |
| jeblair | i guess our clocks are out of sync | 19:59 |
| clarkb | 19:58:54 fungi | when the clock hits 20 | 19:59 |
| * fungi rounded down | 19:59 | |
| clarkb | minute and 6 seconds! | 19:59 |
| fungi | #status alert Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 20:00 |
| openstackstatus | fungi: sending alert | 20:00 |
| *** rkukura has joined #openstack-infra | 20:00 | |
| clarkb | let me know when gerrit is down and I will start renaming git repos in github | 20:00 |
| fungi | will do | 20:00 |
| jeblair | zuul stopped | 20:01 |
| fungi | cool, running playbook now | 20:01 |
| *** vsaienko has quit IRC | 20:02 | |
| jeblair | pabelanger: did you restart the launchers? | 20:02 |
| pabelanger | jeblair: not yet, was about to start, unless you get it first | 20:02 |
| jeblair | pabelanger: go for it | 20:02 |
| pabelanger | ack | 20:02 |
| fungi | clarkb: gerrit is down | 20:02 |
| -openstackstatus- NOTICE: Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 20:02 | |
| *** ChanServ changes topic to "Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html" | 20:02 | |
| fungi | gerrit is far enough back up to process ssh api calls, so proceeding with the rest of the playbook now | 20:04 |
| * mordred is back - can helpif needed | 20:04 | |
| fungi | playbook completed without errors | 20:04 |
| *** vsaienko has joined #openstack-infra | 20:04 | |
| clarkb | github is done | 20:05 |
| jeblair | i think we want to pause here for a moment to let pabelanger finish restarting the zuul launchers | 20:05 |
| fungi | that's fine. gerrit's webui is still spinning on me | 20:05 |
| pabelanger | ya, running into hiccup with hard_restart_playbook | 20:05 |
| pabelanger | https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user | 20:05 |
| *** pvaneck has joined #openstack-infra | 20:06 | |
| fungi | there we go. gerrit webui is responsive again | 20:06 |
| *** cschwede has joined #openstack-infra | 20:06 | |
| openstackstatus | fungi: finished sending alert | 20:06 |
| pabelanger | guess we haven't switched to pipelining = True yet | 20:06 |
| jeblair | that doesn't need to run as zuul | 20:07 |
| jeblair | you can run 'zuul-launcher stop' as root | 20:07 |
| pabelanger | okay, will setup a patch | 20:07 |
| *** Sukhdev has quit IRC | 20:07 | |
| pabelanger | restarting things now | 20:07 |
| *** xyang1 has quit IRC | 20:10 | |
| fungi | once 6 and 7 are complete, i'll status ok with a note that git replicas will be stale for a few hours until gerrit catches up | 20:10 |
| *** erikmwil_ has quit IRC | 20:11 | |
| *** erikmwilson has joined #openstack-infra | 20:11 | |
| pabelanger | still restarting | 20:12 |
| *** Apoorva has joined #openstack-infra | 20:12 | |
| *** slagle_ has quit IRC | 20:12 | |
| pabelanger | hard restart has serial set to 1, we can likely remove that | 20:12 |
| fungi | huh... gerrit show-queue lists a bunch of git push tasks (~10k) but no reindex tasks | 20:12 |
| fungi | no mention of the OnlineReindexer in ~gerrit2/review_site/logs/error_log either | 20:13 |
| *** cschwede has quit IRC | 20:13 | |
| pabelanger | jeblair: fungi: okay, zuul-launchers restarted | 20:13 |
| fungi | heh... http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/rename_repos.yaml#n53 | 20:14 |
| fungi | guess i'll do that now | 20:14 |
| jeblair | fungi: http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/rename_repos.yaml#n54 | 20:14 |
| jeblair | fungi: looks like we're seeing the same thing :) | 20:14 |
| clarkb | at least there is a todo | 20:15 |
| jeblair | okay, i'll start zuul and re-enqueue now. | 20:15 |
| jeblair | clarkb: ++ | 20:15 |
| fungi | found... it's gerrit index start --force changes | 20:17 |
| jeblair | fungi: oh, you have wip-1 on the changes. can you go ahead and approve them? | 20:17 |
| fungi | jeblair: wip removed now | 20:18 |
| fungi | [2017-04-21 20:18:23,811] INFO com.google.gerrit.lucene.OnlineReindexer : Starting online reindex from schema version 14 to 14 | 20:18 |
| fungi | (from ~gerrit2/review_site/logs/error_log) | 20:18 |
| jeblair | changes approved | 20:19 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for github commit status https://review.openstack.org/444060 | 20:19 |
| fungi | 1585 repos to go on the reindex | 20:19 |
| jeblair | re-enquing saved queue | 20:19 |
| fungi | once the reenqueue completes i'll lift the alert | 20:19 |
| *** ramishra has quit IRC | 20:20 | |
| pabelanger | Hmm, ansible reported restarts but zuul launchers stoped | 20:21 |
| *** mat128 has quit IRC | 20:21 | |
| pabelanger | checking why | 20:21 |
| pabelanger | maybe pbr issue | 20:21 |
| pabelanger | checking that | 20:21 |
| fungi | oh, yeah, may need to manually upgrade... python-jenkins? | 20:22 |
| pabelanger | ya | 20:22 |
| pabelanger | python-jenkins | 20:22 |
| pabelanger | okay for manual upgrades? | 20:22 |
| fungi | i guess we need to add a pin for that in puppet-zuul as well, but for now yeah i guess | 20:22 |
| fungi | or somehow get it to do a pip install -U when it updates jjb | 20:23 |
| pabelanger | doing zl01 first | 20:23 |
| pabelanger | okay, back online | 20:23 |
| pabelanger | will start on others | 20:23 |
| fungi | already under 1k repos left to reindex | 20:24 |
| *** alexchadin has joined #openstack-infra | 20:25 | |
| mordred | fungi: what's the story with python-jenkins? | 20:25 |
| fungi | mordred: same old song and dance | 20:25 |
| mordred | I love singing and dancing | 20:26 |
| fungi | we removed the pbr<2.0 pin from zul and tagged a new 2.5.x release, then needed to do the same in jjb and up our pin for it, but also needed the same in python-jenkins | 20:26 |
| fungi | s/zul/zuul/ | 20:26 |
| *** felipemonteiro has joined #openstack-infra | 20:26 | |
| *** felipemonteiro has quit IRC | 20:26 | |
| fungi | (sorry zul!) | 20:26 |
| pabelanger | and since we didn't bump requirements.txt for python-jenkins, it was never updated to uncapped pbr version | 20:27 |
| fungi | so basically zuul imports jjb which chokes on the python-jenkins entrypoint claiming it needs pbr<2.0 because when jjb got updated python-jenkins was left at the same version it had been running | 20:27 |
| pabelanger | okay, all launchers back online | 20:27 |
| *** felipemonteiro has joined #openstack-infra | 20:28 | |
| *** dprince has quit IRC | 20:29 | |
| fungi | jeblair: looks like reenqueuing has completed? | 20:31 |
| jeblair | fungi: yep | 20:31 |
| fungi | any objections to me lifting the status alert? | 20:31 |
| jeblair | nope, looks like jobs are running. | 20:31 |
| jeblair | the changes should merge rsn. | 20:31 |
| *** ssbarnea has quit IRC | 20:31 | |
| *** srobert has quit IRC | 20:32 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Remove neutronclient mocks from network list tests https://review.openstack.org/458803 | 20:32 |
| fungi | #status ok Gerrit is back in service and generally usable, though remote Git replicas (git.openstack.org and github.com) will be stale for the next few hours until online reindexing completes | 20:32 |
| openstackstatus | fungi: sending ok | 20:32 |
| pabelanger | Did somebody do step 9. Re-enable Puppet? | 20:32 |
| pabelanger | cannot see it in emergency file | 20:32 |
| *** rossella_s has quit IRC | 20:32 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rename DefCore Committee -> Interop WG https://review.openstack.org/433414 | 20:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Rename user stories related repos https://review.openstack.org/458605 | 20:34 |
| clarkb | I didn't do it | 20:34 |
| fungi | pabelanger: the cronjob is disabled | 20:34 |
| fungi | pabelanger: details at https://docs.openstack.org/infra/system-config/gerrit.html#renaming-a-project | 20:35 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 20:35 | |
| -openstackstatus- NOTICE: Gerrit is back in service and generally usable, though remote Git replicas (git.openstack.org and github.com) will be stale for the next few hours until online reindexing completes | 20:35 | |
| pabelanger | fungi: ah | 20:35 |
| fungi | and now that the changes have merged, it's safe to proceed with #9, yes | 20:35 |
| jeblair | changes merged | 20:35 |
| jeblair | ya | 20:35 |
| pabelanger | cool | 20:36 |
| pabelanger | doing so now | 20:36 |
| fungi | if puppet gets reenabled _before_ the changes merge, that's bad | 20:36 |
| fungi | like crossing the streams | 20:36 |
| jeblair | don't cross the streams | 20:36 |
| pabelanger | and done | 20:36 |
| fungi | (well, okay, more like setting yourself up for some extra cleanup work the next time we take gerrit offline, but still annoying and preventable) | 20:37 |
| pabelanger | will confirm crontab is triggered | 20:37 |
| fungi | yeah, start grepping the process list for run_all after 20:45 | 20:38 |
| jeblair | fungi: stay puft caused "extra cleanup work" too | 20:38 |
| * fungi did not envy the nyc sanitation workers that year | 20:38 | |
| *** ramishra has joined #openstack-infra | 20:38 | |
| * fungi bets they carved out an entire county in nj to dump it all in | 20:38 | |
| openstackstatus | fungi: finished sending ok | 20:38 |
| mordred | fungi: s/an entire county in// | 20:39 |
| fungi | heh | 20:40 |
| *** amotoki has joined #openstack-infra | 20:40 | |
| pabelanger | I'll get started on .gitreview changes here in a minute | 20:41 |
| fungi | pabelanger: looks like it's running now | 20:45 |
| pabelanger | fungi: yup! | 20:45 |
| *** amotoki has quit IRC | 20:46 | |
| *** xarses_ has quit IRC | 20:46 | |
| *** john-davidge has quit IRC | 20:46 | |
| fungi | and we're under 500 repos left to reindex now | 20:46 |
| *** john-davidge has joined #openstack-infra | 20:47 | |
| *** ociuhandu has joined #openstack-infra | 20:47 | |
| pabelanger | okay, 4 reviews up for .gitreview changes | 20:48 |
| *** openstackgerrit has quit IRC | 20:48 | |
| clarkb | have banged my head against python compiles for a bit then find that haypo has a friendly tool called performance to do all tis automagically. It fails the exact same way I have been failing :/ | 20:48 |
| jeblair | pabelanger: link to the puppet one? | 20:48 |
| *** Julien-zte has quit IRC | 20:48 | |
| pabelanger | https://review.openstack.org/#/c/458963/ and https://review.openstack.org/#/c/458964/ | 20:49 |
| *** ramishra has quit IRC | 20:49 | |
| *** Julien-zte has joined #openstack-infra | 20:49 | |
| fungi | clarkb: why does that not actually surprise me? | 20:49 |
| *** baoli has joined #openstack-infra | 20:49 | |
| clarkb | something is off with make install in python make | 20:50 |
| *** trown is now known as trown|outtypewww | 20:50 | |
| clarkb | (I fixed the curses issues by uninstalling readline-devel and ncurses-devel)_ | 20:50 |
| pabelanger | okay, stepping away to cook up some steak! | 20:52 |
| *** alexchadin has quit IRC | 20:52 | |
| *** yamamoto has joined #openstack-infra | 20:54 | |
| *** esikachev has joined #openstack-infra | 20:54 | |
| EmilienM | mordred: not sure you know already but I saw that https://twitter.com/openstackinfra/status/855507486606012418 | 20:54 |
| *** Julien-zte has quit IRC | 20:55 | |
| * fungi was grilling hotdogs and hamburgers while prepping for maintenance | 20:55 | |
| fungi | sshhh | 20:55 |
| fungi | jeblair: clarkb: pabelanger: thanks and congrats on another successful maintenance! | 20:55 |
| *** baoli has quit IRC | 20:57 | |
| *** john-davidge has quit IRC | 20:57 | |
| fungi | heh, if i look at https://twitter.com/openstackinfra i get a "promoted tweet" from ameritrade about home financing right between part 0 and part 1 of the maintenance start announcement | 20:58 |
| jeblair | i see "GitHub" in 108 point type | 20:58 |
| * fungi goes back to not caring about the twitterverse | 20:58 | |
| clarkb | ok figured it out sys.path doesn't have the compiled modules in it for some reason | 20:58 |
| *** esikachev has quit IRC | 20:58 | |
| *** felipemonteiro has quit IRC | 20:58 | |
| *** Julien-zte has joined #openstack-infra | 20:58 | |
| fungi | oh, yikes, not i see the crazy huge github too | 20:58 |
| fungi | s/not/now/ | 20:58 |
| *** yamamoto has quit IRC | 20:58 | |
| *** openstackgerrit has joined #openstack-infra | 20:59 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Merge pull requests from github reporter https://review.openstack.org/444463 | 21:00 |
| *** jascott1 has quit IRC | 21:02 | |
| *** thorst has quit IRC | 21:03 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Move unit tests for list networks to test_network.py file https://review.openstack.org/458970 | 21:05 |
| *** xyang1 has joined #openstack-infra | 21:10 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for github commit status https://review.openstack.org/444060 | 21:12 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Merge pull requests from github reporter https://review.openstack.org/444463 | 21:13 |
| *** gordc has quit IRC | 21:13 | |
| *** Julien-zte has quit IRC | 21:14 | |
| *** esikachev has joined #openstack-infra | 21:16 | |
| *** jascott1 has joined #openstack-infra | 21:16 | |
| *** Julien-zte has joined #openstack-infra | 21:18 | |
| *** EricGonc_ has quit IRC | 21:19 | |
| *** esikachev has quit IRC | 21:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Apply caching to network resources https://review.openstack.org/458979 | 21:22 |
| *** gouthamr has joined #openstack-infra | 21:24 | |
| *** tjones has joined #openstack-infra | 21:27 | |
| *** thorst has joined #openstack-infra | 21:27 | |
| *** gouthamr has quit IRC | 21:28 | |
| clarkb | so libdir is a site local autoconf setting being set to lib64 | 21:28 |
| clarkb | but then python's build doesn't add that relative path to sys.path | 21:28 |
| *** zxiiro is now known as zxiiro-away | 21:29 | |
| *** Sukhdev has joined #openstack-infra | 21:29 | |
| *** thorst has quit IRC | 21:32 | |
| *** ijw has joined #openstack-infra | 21:33 | |
| *** ramishra has joined #openstack-infra | 21:34 | |
| *** jamesdenton has quit IRC | 21:34 | |
| *** bauwser is now known as bauzas | 21:34 | |
| *** askb has joined #openstack-infra | 21:36 | |
| *** HeOS has joined #openstack-infra | 21:37 | |
| *** salv-orlando has joined #openstack-infra | 21:38 | |
| *** ramishra has quit IRC | 21:40 | |
| *** ramishra has joined #openstack-infra | 21:42 | |
| *** amotoki has joined #openstack-infra | 21:42 | |
| *** amotoki has quit IRC | 21:47 | |
| *** dimtruck is now known as zz_dimtruck | 21:47 | |
| *** ramishra has quit IRC | 21:48 | |
| *** jascott1_ has joined #openstack-infra | 21:49 | |
| *** salv-orl_ has joined #openstack-infra | 21:49 | |
| *** jcoufal has joined #openstack-infra | 21:52 | |
| *** jascott1 has quit IRC | 21:52 | |
| *** salv-orlando has quit IRC | 21:52 | |
| *** Julien-zte has quit IRC | 21:52 | |
| *** vsaienko has quit IRC | 21:55 | |
| *** esikachev has joined #openstack-infra | 21:56 | |
| *** marst_ has quit IRC | 21:56 | |
| *** Julien-zte has joined #openstack-infra | 21:57 | |
| *** baoli has joined #openstack-infra | 21:57 | |
| *** rlandy has quit IRC | 21:57 | |
| *** mdrabe has quit IRC | 22:00 | |
| *** esikachev has quit IRC | 22:01 | |
| *** baoli has quit IRC | 22:02 | |
| *** Apoorva_ has joined #openstack-infra | 22:03 | |
| openstackgerrit | Merged openstack-infra/shade master: Include two transitive dependencies to work around conflicts https://review.openstack.org/458875 | 22:04 |
| *** sbezverk_ has joined #openstack-infra | 22:04 | |
| *** esikachev has joined #openstack-infra | 22:04 | |
| *** john-davidge has joined #openstack-infra | 22:04 | |
| *** ramishra has joined #openstack-infra | 22:05 | |
| *** Apoorva has quit IRC | 22:06 | |
| *** Julien-zte has quit IRC | 22:08 | |
| *** sbezverk_ has quit IRC | 22:08 | |
| *** Julien-zte has joined #openstack-infra | 22:09 | |
| *** esikachev has quit IRC | 22:12 | |
| *** ramishra has quit IRC | 22:13 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Add iptables rules to nodepool-base element https://review.openstack.org/458985 | 22:16 |
| *** Julien-zte has quit IRC | 22:16 | |
| *** Julien-zte has joined #openstack-infra | 22:19 | |
| *** sbezverk_ has joined #openstack-infra | 22:19 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Move iptables management to o_p::server https://review.openstack.org/458986 | 22:19 |
| *** csomerville has joined #openstack-infra | 22:20 | |
| *** cody-somerville has quit IRC | 22:20 | |
| *** thorst has joined #openstack-infra | 22:22 | |
| *** Julien-zte has quit IRC | 22:23 | |
| *** Julien-zte has joined #openstack-infra | 22:25 | |
| *** csomerville has quit IRC | 22:26 | |
| *** sbezverk_ has quit IRC | 22:26 | |
| *** rbrndt_ has quit IRC | 22:32 | |
| *** rbrndt_ has joined #openstack-infra | 22:32 | |
| *** rbrndt_ has quit IRC | 22:32 | |
| *** Julien-zte has quit IRC | 22:35 | |
| *** Julien-zte has joined #openstack-infra | 22:36 | |
| *** thorst has quit IRC | 22:37 | |
| *** rkukura has quit IRC | 22:37 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: Add a dsvm job for devstack-plugin-container https://review.openstack.org/458631 | 22:37 |
| *** Julien-zte has quit IRC | 22:39 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: Add a dsvm job for devstack-plugin-container https://review.openstack.org/458631 | 22:40 |
| *** Julien-zte has joined #openstack-infra | 22:40 | |
| *** ramishra has joined #openstack-infra | 22:41 | |
| *** Sukhdev has quit IRC | 22:45 | |
| *** Julien-zte has quit IRC | 22:45 | |
| *** Julien-zte has joined #openstack-infra | 22:46 | |
| *** hasharAway has quit IRC | 22:46 | |
| *** ramishra has quit IRC | 22:46 | |
| *** Julien-zte has quit IRC | 22:46 | |
| *** zz_dimtruck is now known as dimtruck | 22:47 | |
| *** Julien-zte has joined #openstack-infra | 22:47 | |
| *** baoli has joined #openstack-infra | 22:47 | |
| *** salv-orl_ has quit IRC | 22:48 | |
| *** ramishra has joined #openstack-infra | 22:48 | |
| *** ramishra has quit IRC | 22:53 | |
| *** jcoufal has quit IRC | 22:55 | |
| *** ramishra has joined #openstack-infra | 23:00 | |
| *** rkukura has joined #openstack-infra | 23:01 | |
| *** masber has quit IRC | 23:02 | |
| *** Julien-zte has quit IRC | 23:03 | |
| *** Julien-zte has joined #openstack-infra | 23:04 | |
| *** masber has joined #openstack-infra | 23:04 | |
| *** ramishra has quit IRC | 23:05 | |
| *** HeOS has quit IRC | 23:05 | |
| *** ramishra has joined #openstack-infra | 23:06 | |
| *** Apoorva_ has quit IRC | 23:07 | |
| *** Apoorva has joined #openstack-infra | 23:07 | |
| *** thorst has joined #openstack-infra | 23:08 | |
| *** esikachev has joined #openstack-infra | 23:08 | |
| *** thorst has quit IRC | 23:12 | |
| *** esikachev has quit IRC | 23:12 | |
| *** rkukura has quit IRC | 23:12 | |
| *** Julien-zte has quit IRC | 23:13 | |
| *** rkukura has joined #openstack-infra | 23:13 | |
| *** Julien-zte has joined #openstack-infra | 23:16 | |
| *** ramishra has quit IRC | 23:17 | |
| *** esberglu_ has quit IRC | 23:17 | |
| *** baoli has quit IRC | 23:20 | |
| *** Julien-zte has quit IRC | 23:23 | |
| *** Julien-zte has joined #openstack-infra | 23:23 | |
| *** rkukura has quit IRC | 23:24 | |
| *** ramishra has joined #openstack-infra | 23:24 | |
| *** sdague has quit IRC | 23:24 | |
| *** baoli has joined #openstack-infra | 23:25 | |
| *** baoli has quit IRC | 23:26 | |
| *** Julien-zte has quit IRC | 23:27 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Use requests-mock for the list/add/remove volume types tests https://review.openstack.org/459022 | 23:27 |
| *** Julien-zte has joined #openstack-infra | 23:28 | |
| *** ramishra has quit IRC | 23:29 | |
| otherwiseguy | The gate-tripleo-ci-centos-7-nonha-multinode-oooq-nv job seems to just stay queued forever. | 23:33 |
| *** xarses_ has joined #openstack-infra | 23:34 | |
| *** Julien-zte has quit IRC | 23:35 | |
| *** Julien-zte has joined #openstack-infra | 23:36 | |
| fungi | otherwiseguy: massive network outage in red hat took the environment where the tripleo team hosts those test nodes offline all day | 23:36 |
| fungi | no idea whether they've solved their issues yet | 23:36 |
| fungi | survey says... no http://status.fedoraproject.org/ | 23:36 |
| otherwiseguy | is there a way to disable the non-voting test until back up? | 23:36 |
| otherwiseguy | otherwise things just sit there for hours. | 23:36 |
| *** ramishra has joined #openstack-infra | 23:37 | |
| fungi | yeah, the tripleo team really shouldn't have pushed to have other projects run those jobs on their changes until they can come up with a redundant provider setup | 23:37 |
| fungi | should probably just remove them where relevant in openstack-infra/project-config:zuul/layout.yaml | 23:38 |
| *** thorst has joined #openstack-infra | 23:38 | |
| fungi | EmilienM: any objection there? | 23:39 |
| mwhahaha | do we not have any centos nodes on any other providers? | 23:39 |
| mwhahaha | (not related to tripleo jobs) | 23:39 |
| *** notmyname has quit IRC | 23:39 | |
| fungi | regular centos-7 nodes, sure | 23:39 |
| *** notmyname has joined #openstack-infra | 23:40 | |
| mwhahaha | are they all in use at the moment? | 23:40 |
| fungi | i see a bunch in use and running jobs currently | 23:40 |
| *** yamahata_ has quit IRC | 23:40 | |
| *** esikachev has joined #openstack-infra | 23:41 | |
| EmilienM | What do we want to remove? | 23:41 |
| otherwiseguy | maybe non-voting jobs could have a queueing ttimeout | 23:41 |
| otherwiseguy | :) | 23:41 |
| fungi | otherwiseguy: mwhahaha: can you give me an example job name or two you're talking about, just so i'm on the same page? | 23:42 |
| mwhahaha | i'm asking abotu the gate-puppet-aodh-puppet-module-build | 23:43 |
| mwhahaha | which is a centos node job | 23:43 |
| EmilienM | We're not blocking anything if our cloud is down | 23:43 |
| EmilienM | Ovb jobs aren't in gate | 23:43 |
| mwhahaha | EmilienM: yea they are, the tripleo-ci jobs are | 23:43 |
| fungi | EmilienM: example, a neutron change with nonvoting gate-tripleo-ci-centos-7-nonha-multinode-oooq-nv job is waiting forever in the check pipeline to get a node | 23:43 |
| fungi | 458982,1 | 23:44 |
| mwhahaha | but that runs on infra doesn't it? | 23:44 |
| mwhahaha | not rh cloud? | 23:44 |
| EmilienM | It's multinode, | 23:44 |
| EmilienM | I'm confused | 23:44 |
| fungi | i thought the tripleo-ci jobs all ran in tripleo cloud | 23:44 |
| EmilienM | It's not ovb | 23:44 |
| fungi | but i should look at the job definition, yes | 23:44 |
| otherwiseguy | gate-tripleo-ci-centos-7-nonha-multinode-oooq-nv: is what I'm talking about | 23:44 |
| EmilienM | No fungi | 23:44 |
| *** ramishra has quit IRC | 23:44 | |
| mwhahaha | yea that's kinda my question about centos nodes | 23:44 |
| mwhahaha | because i don't see any active centos jobs on the status page | 23:44 |
| EmilienM | Multinode jobs run anywhere | 23:45 |
| *** Julien-zte has quit IRC | 23:45 | |
| mwhahaha | but trunk.rdoproject.org has been sketchy so maybe that's impacting things? | 23:45 |
| EmilienM | So again nothing is blocked except ovb jobs but not in gate so... | 23:45 |
| EmilienM | No it's stable | 23:45 |
| *** esikachev has quit IRC | 23:45 | |
| EmilienM | It's not in Phoenix | 23:45 |
| mwhahaha | eh it's not loading well for me | 23:45 |
| mwhahaha | maybe dns | 23:45 |
| mwhahaha | oh it's the stylesheets, rdopject.org isn't working | 23:46 |
| mwhahaha | but the repos seem find | 23:46 |
| *** rkukura has joined #openstack-infra | 23:46 | |
| mwhahaha | fine | 23:46 |
| fungi | i do see some stale centos-7 nodes which are probably hanging around due to missed zeromq messages during the zuul restart. i'll clean those up real fast and see if it helps | 23:46 |
| EmilienM | Anyway, RH1 might​ be down it should not affect anything in OpenStack | 23:47 |
| EmilienM | Can I leave IRC? | 23:47 |
| fungi | you can leave irc | 23:47 |
| EmilienM | I'm in my phone sorry | 23:47 |
| *** Julien-zte has joined #openstack-infra | 23:47 | |
| fungi | no problem | 23:47 |
| EmilienM | On even | 23:48 |
| EmilienM | Happy weekend folks | 23:48 |
| fungi | thanks, you too | 23:48 |
| otherwiseguy | zuul shows lots of neutron checks all waiting indifinitely on the ooo job | 23:48 |
| mwhahaha | the only jobs that should be held up from the rh outage are the triploci-*-ovb-* jobs | 23:49 |
| mwhahaha | the rest run anywhere where there are centos nodes which is why i asked about centos node availablity since i noticed the puppet-module-build jobs queued up | 23:50 |
| fungi | i see a couple which just picked up nodes on the python-tripleoclient change at the top of the gate and several behind it | 23:52 |
| mwhahaha | yea there we go | 23:53 |
| mwhahaha | so maybe that'll work through the backlog now | 23:53 |
| *** tosky has quit IRC | 23:54 | |
| otherwiseguy | mwhahaha, thanks. for example, https://review.openstack.org/458947 was stuck for around 3 hours, i respun the patch and it didn't help. several others are around 3.5 hours, etc. | 23:55 |
| otherwiseguy | The RH outage has been fun today (I work there). | 23:55 |
| mwhahaha | (ditto) | 23:56 |
| otherwiseguy | a day without bugzilla :D | 23:56 |
| mwhahaha | and nothing of value was lost | 23:56 |
| * mwhahaha shows himself out | 23:56 | |
| fungi | yeah, 3.5 hours was roughly when we brought zuul back online from the gerrit maintenance and reenqueued active changes, so entirely possible those had been queued much longer but their queue times were reset at that point | 23:56 |
| *** yamamoto has joined #openstack-infra | 23:56 | |
| *** gouthamr has joined #openstack-infra | 23:57 | |
| otherwiseguy | Fun. Yeah, I count 15 neutron patches just sitting around waiting for the tripleo job to eventually do something. it does seem like some kind of queueing timeout would be a good idea. | 23:58 |
| *** thorst has quit IRC | 23:58 | |
| otherwiseguy | (at least for nonvoting jobs) | 23:58 |
| otherwiseguy | Of course, I don't really know how anything works so I don't really know what a good idea is. :p | 23:58 |
| fungi | i don't think i did anything which *should* have changed that situation. the four leaked used nodes i deleted were just plain centos-7 not multi-node types | 23:59 |
| mwhahaha | well the problem is that we would just fill the queue with voting j obs | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!