| *** claudiub has quit IRC | 00:00 | |
| *** ijw has joined #openstack-infra | 00:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:01 | |
| *** ijw has quit IRC | 00:01 | |
| *** ijw has joined #openstack-infra | 00:02 | |
| *** sbezverk has joined #openstack-infra | 00:06 | |
| *** adam_g has quit IRC | 00:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/bindep master: Update tox.ini with py35 https://review.openstack.org/467803 | 00:07 |
|---|---|---|
| *** esikachev has joined #openstack-infra | 00:08 | |
| pabelanger | and our first opensuse-422 job: http://logs.openstack.org/03/467803/1/check/gate-bindep-bindep-fallback-opensuse-422-nv/4ae142a/console.html | 00:08 |
| *** adam_g has joined #openstack-infra | 00:09 | |
| pabelanger | we'll need to update install-distro-packages.sh | 00:09 |
| pabelanger | also, looks like traceroute and traceroute6 are missing packages | 00:09 |
| pabelanger | I'll dive into that tomorrow | 00:10 |
| *** hongbin has quit IRC | 00:10 | |
| *** yamahata has quit IRC | 00:11 | |
| *** bobh has joined #openstack-infra | 00:12 | |
| *** SumitNaiksatam has quit IRC | 00:14 | |
| *** sflanigan has joined #openstack-infra | 00:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 00:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 00:16 |
| *** Sukhdev_ has quit IRC | 00:16 | |
| *** rlandy has quit IRC | 00:16 | |
| *** esikachev has quit IRC | 00:17 | |
| *** gardlt has joined #openstack-infra | 00:20 | |
| *** r-daneel has quit IRC | 00:20 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Display command/shell results in a banner https://review.openstack.org/467603 | 00:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Strip unneeded cmd, changed, stdout and stderr from results https://review.openstack.org/467310 | 00:21 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Disable leaked repo check https://review.openstack.org/466810 | 00:23 |
| *** jamesmca_ has joined #openstack-infra | 00:25 | |
| *** aeng has quit IRC | 00:26 | |
| *** wolverineav has quit IRC | 00:28 | |
| *** wolverineav has joined #openstack-infra | 00:28 | |
| mordred | pabelanger: well done! | 00:29 |
| *** sflanigan has quit IRC | 00:31 | |
| *** gomarivera has joined #openstack-infra | 00:34 | |
| *** tuanluong has joined #openstack-infra | 00:36 | |
| *** dimtruck is now known as zz_dimtruck | 00:37 | |
| *** bobh has quit IRC | 00:39 | |
| *** ijw has quit IRC | 00:39 | |
| *** aeng has joined #openstack-infra | 00:43 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Wait for merger to complete before item is ready https://review.openstack.org/463054 | 00:44 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for requiring github pr head status https://review.openstack.org/449390 | 00:44 |
| *** gomarivera has quit IRC | 00:45 | |
| *** gomarivera has joined #openstack-infra | 00:45 | |
| mnaser | mordred yaml ansible inventories are awesome | 00:46 |
| *** kgiusti has quit IRC | 00:47 | |
| bkero | clarkb: I think puppet nerfed my user. | 00:47 |
| *** yamahata has joined #openstack-infra | 00:47 | |
| *** sflanigan has joined #openstack-infra | 00:48 | |
| *** gomarivera has quit IRC | 00:50 | |
| bkero | clarkb: nevermind, agent problem | 00:50 |
| openstackgerrit | Merged openstack-infra/shade master: Updated from global requirements https://review.openstack.org/467428 | 00:50 |
| *** Julien-zte has joined #openstack-infra | 00:51 | |
| *** gardlt has quit IRC | 00:53 | |
| *** gomarivera has joined #openstack-infra | 00:55 | |
| *** yamahata has quit IRC | 00:55 | |
| *** gomarivera has quit IRC | 00:59 | |
| *** Julien-zte has quit IRC | 01:00 | |
| *** bobh has joined #openstack-infra | 01:00 | |
| *** Julien-zte has joined #openstack-infra | 01:01 | |
| clarkb | bkero we shouldnt have puppet running | 01:01 |
| bkero | clarkb: yeah, I did ssh-add -c without an agent capable of asking for confirmation | 01:02 |
| *** cuongnv has joined #openstack-infra | 01:02 | |
| *** esikachev has joined #openstack-infra | 01:02 | |
| *** zz_dimtruck is now known as dimtruck | 01:03 | |
| *** srobert has joined #openstack-infra | 01:05 | |
| *** pvaneck has quit IRC | 01:05 | |
| *** pvaneck has joined #openstack-infra | 01:06 | |
| *** esikachev has quit IRC | 01:07 | |
| *** srobert has quit IRC | 01:08 | |
| *** srobert has joined #openstack-infra | 01:08 | |
| *** pvaneck has quit IRC | 01:10 | |
| *** wolverineav has quit IRC | 01:11 | |
| *** wolverin_ has joined #openstack-infra | 01:14 | |
| fungi | i can't remember... we did say to put the uuid for the snapshotted instance in the emergency disable list, right? | 01:16 |
| clarkb | fungi: I didnt, but wouldnt the ssh host key be rejected? | 01:16 |
| clarkb | fungi: its probably worth doingthough | 01:16 |
| fungi | oh, hrm, maybe yeah since it's not connecting by the old hostname anyway | 01:18 |
| *** wolverin_ has quit IRC | 01:18 | |
| *** jamesmca_ has quit IRC | 01:19 | |
| *** jamesmca_ has joined #openstack-infra | 01:20 | |
| *** yolkfull has joined #openstack-infra | 01:21 | |
| *** esikachev has joined #openstack-infra | 01:22 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Don't break root ssh with authorized_keys copy https://review.openstack.org/467773 | 01:24 |
| *** jamesmca_ has quit IRC | 01:24 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Add etcd3 to the list of base services https://review.openstack.org/467687 | 01:25 |
| *** esikachev has quit IRC | 01:26 | |
| *** ihrachys has joined #openstack-infra | 01:28 | |
| *** links has joined #openstack-infra | 01:30 | |
| ihrachys | folks, we have some suckers in #openstack-neutron spamming. would love to see them banned/dropped, but I don't have op status. so... | 01:31 |
| ihrachys | namely Semoner, malcom682, remet94... | 01:31 |
| clarkb | ihrachys: I can op you really quick | 01:34 |
| clarkb | ihrachys: that work? I think you should have op now | 01:34 |
| *** aeng has quit IRC | 01:35 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement trigger require/reject-review https://review.openstack.org/463387 | 01:35 |
| clarkb | (sorry I am not more help, dealing with evening tasks like dinner) | 01:35 |
| ihrachys | yeah, thanks. | 01:36 |
| ihrachys | it worked | 01:36 |
| ihrachys | enjoy the dinner | 01:36 |
| *** sree has joined #openstack-infra | 01:36 | |
| *** bobh has quit IRC | 01:37 | |
| *** dhinesh has quit IRC | 01:39 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Include tenant name in github context https://review.openstack.org/463871 | 01:39 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Save installation ids to a cache and fetch them per project https://review.openstack.org/463424 | 01:39 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Catch integration key file not found in zuul https://review.openstack.org/463425 | 01:39 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Create github client each time it's called. https://review.openstack.org/463421 | 01:39 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Use integration_id with github https://review.openstack.org/463420 | 01:39 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Refactor integration key loading to always load https://review.openstack.org/463423 | 01:40 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Decode JSON body once for requests https://review.openstack.org/463422 | 01:40 |
| *** sree has quit IRC | 01:41 | |
| *** sbezverk has quit IRC | 01:42 | |
| *** thorst has joined #openstack-infra | 01:46 | |
| *** aeng has joined #openstack-infra | 01:47 | |
| *** ihrachys has quit IRC | 01:48 | |
| *** esikachev has joined #openstack-infra | 01:49 | |
| *** esikachev has quit IRC | 01:53 | |
| *** esikachev has joined #openstack-infra | 01:56 | |
| *** Julien-zte has quit IRC | 01:58 | |
| *** Julien-zte has joined #openstack-infra | 01:59 | |
| *** esikachev has quit IRC | 02:01 | |
| *** thorst has quit IRC | 02:02 | |
| *** ram5391 has joined #openstack-infra | 02:04 | |
| *** zhurong has joined #openstack-infra | 02:05 | |
| *** sbezverk has joined #openstack-infra | 02:07 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: tooz: gate on etcd3gw https://review.openstack.org/467814 | 02:08 |
| *** notmyname has quit IRC | 02:08 | |
| *** hongbin has joined #openstack-infra | 02:09 | |
| *** notmyname has joined #openstack-infra | 02:13 | |
| *** thorst has joined #openstack-infra | 02:14 | |
| *** ramishra has joined #openstack-infra | 02:14 | |
| *** sbezverk has quit IRC | 02:15 | |
| *** ijw has joined #openstack-infra | 02:15 | |
| *** masayukig has joined #openstack-infra | 02:18 | |
| *** ijw has quit IRC | 02:18 | |
| *** ijw_ has joined #openstack-infra | 02:19 | |
| *** ijw__ has joined #openstack-infra | 02:20 | |
| *** ijw_ has quit IRC | 02:24 | |
| *** esikachev has joined #openstack-infra | 02:26 | |
| *** adam_g has quit IRC | 02:28 | |
| *** adam_g has joined #openstack-infra | 02:30 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:30 | |
| *** esikachev has quit IRC | 02:30 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create and use plugin/node abstract classes https://review.openstack.org/467826 | 02:32 |
| *** esikachev has joined #openstack-infra | 02:32 | |
| *** xinliang has quit IRC | 02:34 | |
| *** thorst has quit IRC | 02:37 | |
| *** esikachev has quit IRC | 02:37 | |
| *** stakeda has joined #openstack-infra | 02:39 | |
| *** xinliang has joined #openstack-infra | 02:47 | |
| *** xinliang has quit IRC | 02:47 | |
| *** xinliang has joined #openstack-infra | 02:47 | |
| *** ihrachys has joined #openstack-infra | 02:49 | |
| *** ihrachys has quit IRC | 02:50 | |
| *** tuanluong has quit IRC | 02:50 | |
| *** rfolco has quit IRC | 02:52 | |
| openstackgerrit | dongwenjuan proposed openstack-infra/puppet-nodepool master: Add 'bind_address' for override_options https://review.openstack.org/467830 | 02:59 |
| *** armax has quit IRC | 03:06 | |
| *** bobh has joined #openstack-infra | 03:10 | |
| *** namnh has joined #openstack-infra | 03:13 | |
| *** nunchuck has quit IRC | 03:14 | |
| *** nunchuck has joined #openstack-infra | 03:15 | |
| *** sree has joined #openstack-infra | 03:22 | |
| *** yuanying has joined #openstack-infra | 03:24 | |
| *** esikachev has joined #openstack-infra | 03:27 | |
| *** mriedem has quit IRC | 03:27 | |
| *** esikachev has quit IRC | 03:31 | |
| *** hamzy has joined #openstack-infra | 03:31 | |
| *** Sukhdev has joined #openstack-infra | 03:32 | |
| *** hongbin has quit IRC | 03:32 | |
| *** bobh has quit IRC | 03:33 | |
| *** jamesmca_ has joined #openstack-infra | 03:41 | |
| *** jamesmca_ has quit IRC | 03:42 | |
| *** vikrant has joined #openstack-infra | 03:42 | |
| *** jamesmca_ has joined #openstack-infra | 03:43 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Disable rally jobs for stable/newton https://review.openstack.org/467515 | 03:44 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: remove -nv from job regex https://review.openstack.org/467866 | 03:44 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Barbican grenade gate" append "old" to DEST variable https://review.openstack.org/466174 | 03:46 |
| *** tuanluong has joined #openstack-infra | 03:47 | |
| *** gouthamr has quit IRC | 03:57 | |
| *** jamesmca_ has quit IRC | 03:57 | |
| *** jamesmca_ has joined #openstack-infra | 03:58 | |
| *** tuanluong has quit IRC | 03:59 | |
| *** vikrant is now known as vikrant|brb | 04:00 | |
| *** benj_ has quit IRC | 04:03 | |
| *** benj_ has joined #openstack-infra | 04:04 | |
| *** psachin has joined #openstack-infra | 04:04 | |
| *** zhurong has quit IRC | 04:08 | |
| *** ykarel has joined #openstack-infra | 04:09 | |
| *** Sukhdev has quit IRC | 04:11 | |
| *** markvoelker has joined #openstack-infra | 04:13 | |
| *** yamamoto has quit IRC | 04:13 | |
| *** yamamoto has joined #openstack-infra | 04:13 | |
| *** vikrant|brb is now known as vikrant | 04:14 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Produce API documentation https://review.openstack.org/467873 | 04:15 |
| *** thorst has joined #openstack-infra | 04:16 | |
| *** zhurong has joined #openstack-infra | 04:16 | |
| *** yamamoto has quit IRC | 04:17 | |
| *** thorst has quit IRC | 04:18 | |
| *** aeng has quit IRC | 04:19 | |
| *** thorst has joined #openstack-infra | 04:26 | |
| *** winggundamth has joined #openstack-infra | 04:27 | |
| *** winggundamth has quit IRC | 04:27 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Fix $DEVSTACK_PROJECT_FROM_GIT for amqp1 job https://review.openstack.org/467876 | 04:31 |
| *** aeng has joined #openstack-infra | 04:36 | |
| *** thorst has quit IRC | 04:38 | |
| *** kzaitsev_mb has quit IRC | 04:40 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Set a "creator" rule for tempest https://review.openstack.org/466174 | 04:40 |
| *** dhinesh has joined #openstack-infra | 04:43 | |
| *** esikachev has joined #openstack-infra | 04:44 | |
| *** xyang1 has quit IRC | 04:45 | |
| *** jtomasek has quit IRC | 04:46 | |
| *** esikachev has quit IRC | 04:48 | |
| *** jamesmca_ has quit IRC | 04:51 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Add neutron periodic job with ryu master https://review.openstack.org/467885 | 04:56 |
| *** ym200 has joined #openstack-infra | 04:56 | |
| *** ltomasbo|away is now known as ltomasbo | 04:57 | |
| *** ym200_ has joined #openstack-infra | 04:59 | |
| openstackgerrit | Nisha Agarwal proposed openstack-infra/project-config master: disable swift for py3 enablement in ironic https://review.openstack.org/467538 | 05:00 |
| *** ym200 has quit IRC | 05:01 | |
| *** yamamoto has joined #openstack-infra | 05:01 | |
| *** gyee has quit IRC | 05:03 | |
| *** ym200_ has quit IRC | 05:03 | |
| *** mikal_ is now known as mikal | 05:06 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Add neutron periodic job with ryu master https://review.openstack.org/467885 | 05:11 |
| *** markvoelker_ has joined #openstack-infra | 05:11 | |
| *** esikachev has joined #openstack-infra | 05:11 | |
| *** markvoelker has quit IRC | 05:14 | |
| *** esikachev has quit IRC | 05:16 | |
| *** claudiub has joined #openstack-infra | 05:17 | |
| *** markvoelker has joined #openstack-infra | 05:17 | |
| *** markvoel_ has joined #openstack-infra | 05:19 | |
| *** markvoelker has quit IRC | 05:19 | |
| *** markvoelker_ has quit IRC | 05:21 | |
| *** pvaneck has joined #openstack-infra | 05:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use networkx for digraph https://review.openstack.org/467419 | 05:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create and use plugin/node abstract classes https://review.openstack.org/467826 | 05:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Produce API documentation https://review.openstack.org/467873 | 05:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Adding unit testing for configuration https://review.openstack.org/466959 | 05:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a more generic tree->graph parser https://review.openstack.org/465417 | 05:22 |
| *** aeng has quit IRC | 05:25 | |
| *** bhavik1 has joined #openstack-infra | 05:25 | |
| *** pvaneck has quit IRC | 05:25 | |
| *** watersoul_ has quit IRC | 05:25 | |
| *** rwsu has joined #openstack-infra | 05:27 | |
| *** sree has quit IRC | 05:28 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 05:29 | |
| *** SumitNaiksatam has quit IRC | 05:29 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 05:29 | |
| *** watersoul has joined #openstack-infra | 05:32 | |
| *** watersoul has quit IRC | 05:34 | |
| *** ram5391 has quit IRC | 05:35 | |
| *** cshastri has joined #openstack-infra | 05:36 | |
| *** watersoul has joined #openstack-infra | 05:37 | |
| *** thorst has joined #openstack-infra | 05:38 | |
| *** aeng has joined #openstack-infra | 05:38 | |
| *** prateek has joined #openstack-infra | 05:40 | |
| *** thorst has quit IRC | 05:42 | |
| *** sree has joined #openstack-infra | 05:45 | |
| *** SumitNaiksatam has quit IRC | 05:51 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 05:51 | |
| *** markvoel_ has quit IRC | 05:53 | |
| *** markvoelker has joined #openstack-infra | 05:53 | |
| *** markvoelker has quit IRC | 05:54 | |
| *** Dinesh_Bhor has quit IRC | 06:06 | |
| *** thorst has joined #openstack-infra | 06:09 | |
| *** dizquierdo has joined #openstack-infra | 06:10 | |
| *** ccamacho|out has left #openstack-infra | 06:10 | |
| *** Dinesh_Bhor has joined #openstack-infra | 06:11 | |
| *** yolkfull_ has joined #openstack-infra | 06:15 | |
| *** yolkfull has quit IRC | 06:17 | |
| *** rcernin has joined #openstack-infra | 06:18 | |
| *** sree has quit IRC | 06:20 | |
| *** pcaruana has joined #openstack-infra | 06:20 | |
| *** sree has joined #openstack-infra | 06:22 | |
| *** sree has quit IRC | 06:23 | |
| *** dhajare has joined #openstack-infra | 06:26 | |
| *** markvoelker has joined #openstack-infra | 06:27 | |
| *** thorst has quit IRC | 06:27 | |
| *** sree has joined #openstack-infra | 06:28 | |
| *** markvoelker has quit IRC | 06:32 | |
| *** esikachev has joined #openstack-infra | 06:38 | |
| *** pgadiya has joined #openstack-infra | 06:40 | |
| *** pgadiya has quit IRC | 06:41 | |
| *** yolkfull_ has quit IRC | 06:42 | |
| *** dhinesh has quit IRC | 06:42 | |
| *** dizquierdo has quit IRC | 06:45 | |
| *** aeng has quit IRC | 06:47 | |
| *** sree_ has joined #openstack-infra | 06:51 | |
| *** sree_ is now known as Guest52217 | 06:52 | |
| *** sree has quit IRC | 06:52 | |
| *** ralonsoh has joined #openstack-infra | 06:55 | |
| *** markvoelker has joined #openstack-infra | 06:59 | |
| *** bhavik1 has quit IRC | 07:01 | |
| *** markvoelker has quit IRC | 07:03 | |
| *** bulat has joined #openstack-infra | 07:04 | |
| *** markvoelker has joined #openstack-infra | 07:09 | |
| *** dizquierdo has joined #openstack-infra | 07:09 | |
| *** shardy has joined #openstack-infra | 07:10 | |
| *** gcb has quit IRC | 07:10 | |
| *** gcb has joined #openstack-infra | 07:14 | |
| *** markvoelker has quit IRC | 07:14 | |
| *** ym200 has joined #openstack-infra | 07:14 | |
| *** ym200 has quit IRC | 07:18 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** Guest52217 has quit IRC | 07:21 | |
| *** sree has joined #openstack-infra | 07:21 | |
| *** sree has quit IRC | 07:22 | |
| *** thorst has joined #openstack-infra | 07:24 | |
| *** sree has joined #openstack-infra | 07:24 | |
| *** markvoelker has joined #openstack-infra | 07:29 | |
| *** thorst has quit IRC | 07:29 | |
| *** zhurong has quit IRC | 07:30 | |
| *** jpich has joined #openstack-infra | 07:35 | |
| *** markvoelker has quit IRC | 07:37 | |
| *** dtantsur|afk is now known as dtantsur | 07:38 | |
| *** DangerousDaren has joined #openstack-infra | 07:39 | |
| *** markvoelker has joined #openstack-infra | 07:42 | |
| *** zhurong has joined #openstack-infra | 07:45 | |
| openstackgerrit | shangxdy proposed openstack-infra/project-config master: Fix ZUUL_USER_SSH_PUBLIC_KEY to support ssh key content https://review.openstack.org/467919 | 07:47 |
| *** markvoelker has quit IRC | 07:49 | |
| *** jpena|off is now known as jpena | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** Hal has quit IRC | 08:03 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Produce API documentation https://review.openstack.org/467873 | 08:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add openSUSE support for install-distro-packages.sh https://review.openstack.org/467922 | 08:04 |
| AJaeger | dirk, pabelanger, cmurphy ^ | 08:04 |
| *** lin_yang has quit IRC | 08:07 | |
| *** lucas-pto is now known as lucasagomes | 08:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Handle traceroute in /usr/sbin https://review.openstack.org/467924 | 08:10 |
| AJaeger | dirk, pabelanger, cmurphy: and fix for second problem ^ | 08:10 |
| AJaeger | pabelanger, dirk: Congratulations to get openSUSE images building! | 08:11 |
| *** gildub has quit IRC | 08:14 | |
| *** yolkfull_ has joined #openstack-infra | 08:14 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Add an experimental job for centos-7 https://review.openstack.org/467925 | 08:14 |
| *** yolkfull_ has quit IRC | 08:16 | |
| *** yolkfull_ has joined #openstack-infra | 08:17 | |
| *** markvoelker has joined #openstack-infra | 08:24 | |
| *** thorst has joined #openstack-infra | 08:25 | |
| *** panda|bbl is now known as panda | 08:25 | |
| *** efoley__ has joined #openstack-infra | 08:26 | |
| *** efoley_ has joined #openstack-infra | 08:28 | |
| *** thorst has quit IRC | 08:29 | |
| *** pbourke has quit IRC | 08:30 | |
| *** markvoelker has quit IRC | 08:30 | |
| *** pbourke has joined #openstack-infra | 08:30 | |
| *** efoley__ has quit IRC | 08:31 | |
| dirk | AJaeger: how do you see that they finished building? there is an error in http://nb03.openstack.org/image.log | 08:32 |
| *** markvoelker has joined #openstack-infra | 08:32 | |
| *** electrofelix has joined #openstack-infra | 08:33 | |
| dirk | ah, nb04.openstack.log/image.log looks better | 08:33 |
| *** markvoelker has quit IRC | 08:40 | |
| *** sree has quit IRC | 08:42 | |
| *** sree has joined #openstack-infra | 08:42 | |
| *** sree has quit IRC | 08:44 | |
| *** sree has joined #openstack-infra | 08:45 | |
| dirk | AJaeger: ah, I see, http://logs.openstack.org/03/467803/1/check/gate-bindep-bindep-fallback-opensuse-422-nv/4ae142a/console.html -- pretty sexy to see it working :) (somewhat) | 08:46 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Add experimental jobs for centos-7 https://review.openstack.org/467925 | 08:52 |
| *** zhurong has quit IRC | 08:54 | |
| *** e0ne has joined #openstack-infra | 08:56 | |
| *** dhajare has quit IRC | 08:58 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 08:59 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 09:01 |
| dirk | AJaeger: ok ^^ should get us going somewhat, I hope | 09:01 |
| * dirk afk | 09:01 | |
| *** baoli has joined #openstack-infra | 09:02 | |
| *** sree_ has joined #openstack-infra | 09:02 | |
| *** sree_ is now known as Guest88807 | 09:03 | |
| *** esikachev has quit IRC | 09:04 | |
| *** zhurong has joined #openstack-infra | 09:04 | |
| *** sree has quit IRC | 09:04 | |
| *** baoli has quit IRC | 09:06 | |
| *** Guest88807 has quit IRC | 09:07 | |
| *** sflanigan has quit IRC | 09:09 | |
| *** derekh has joined #openstack-infra | 09:09 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add openSUSE support for install-distro-packages.sh https://review.openstack.org/467922 | 09:14 |
| *** markvoelker has joined #openstack-infra | 09:16 | |
| *** priteau has joined #openstack-infra | 09:17 | |
| *** markvoelker has quit IRC | 09:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix $DEVSTACK_PROJECT_FROM_GIT for amqp1 job https://review.openstack.org/467876 | 09:22 |
| *** thorst has joined #openstack-infra | 09:26 | |
| *** thorst has quit IRC | 09:30 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:35 | |
| *** zhurong has quit IRC | 09:35 | |
| *** esikachev has joined #openstack-infra | 09:38 | |
| *** bhavik1 has joined #openstack-infra | 09:41 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change meeting and channel for Monasca https://review.openstack.org/467673 | 09:42 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: update meeting time for requirements team https://review.openstack.org/467711 | 09:43 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add openSUSE support for install-distro-packages.sh https://review.openstack.org/467922 | 09:43 |
| *** esikachev has quit IRC | 09:43 | |
| *** sambetts|afk is now known as sambetts | 09:46 | |
| *** sdague has joined #openstack-infra | 09:46 | |
| *** gomarivera has joined #openstack-infra | 09:47 | |
| *** esikachev has joined #openstack-infra | 09:47 | |
| *** esikachev has quit IRC | 09:47 | |
| *** gomarivera has quit IRC | 09:51 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add openSUSE support for install-distro-packages.sh https://review.openstack.org/467922 | 09:51 |
| *** markvoelker has joined #openstack-infra | 09:54 | |
| *** adriant has quit IRC | 09:58 | |
| *** markvoelker has quit IRC | 09:58 | |
| electrofelix | @here any infra cores around to review some patches for git-review? would like to see if https://review.openstack.org/389701, https://review.openstack.org/244083, https://review.openstack.org/398353, & https://review.openstack.org/271705 are all acceptable | 10:01 |
| *** stakeda has quit IRC | 10:05 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci master: Don't maintain stable scenario env in upgrade CI https://review.openstack.org/467973 | 10:07 |
| *** apetrich_ has quit IRC | 10:07 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: update meeting time for requirements team https://review.openstack.org/467711 | 10:08 |
| *** yamamoto has quit IRC | 10:11 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: tooz: gate on etcd3gw https://review.openstack.org/467814 | 10:13 |
| *** Julien-zte has quit IRC | 10:15 | |
| *** cuongnv has quit IRC | 10:17 | |
| *** sfinucan has quit IRC | 10:19 | |
| *** sfinucan has joined #openstack-infra | 10:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use networkx for digraph https://review.openstack.org/467419 | 10:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create and use plugin/node abstract classes https://review.openstack.org/467826 | 10:19 |
| *** thorst has joined #openstack-infra | 10:27 | |
| *** thorst has quit IRC | 10:31 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Add neutron periodic job with ryu master https://review.openstack.org/467885 | 10:33 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: Revert "Temporary make stable compat ocata jobs non-voting" https://review.openstack.org/467990 | 10:38 |
| openstackgerrit | shangxdy proposed openstack-infra/project-config master: Fix ZUUL_USER_SSH_PUBLIC_KEY to support ssh key content https://review.openstack.org/467919 | 10:38 |
| *** markvoelker has joined #openstack-infra | 10:47 | |
| *** lucasagomes is now known as lucas-hungry | 10:48 | |
| *** markvoelker has quit IRC | 10:52 | |
| *** markvoelker has joined #openstack-infra | 10:55 | |
| *** jkilpatr has quit IRC | 10:57 | |
| *** yamamoto has joined #openstack-infra | 10:58 | |
| *** markvoelker has quit IRC | 10:59 | |
| *** rhallisey has joined #openstack-infra | 11:12 | |
| *** jkilpatr has joined #openstack-infra | 11:14 | |
| *** markvoelker has joined #openstack-infra | 11:14 | |
| *** ym200 has joined #openstack-infra | 11:15 | |
| *** markvoelker has quit IRC | 11:19 | |
| *** ym200 has quit IRC | 11:19 | |
| *** yolkfull_ has quit IRC | 11:21 | |
| *** dimtruck is now known as zz_dimtruck | 11:24 | |
| *** namnh has quit IRC | 11:25 | |
| *** isaacb has joined #openstack-infra | 11:26 | |
| *** jpena is now known as jpena|lunch | 11:26 | |
| *** prateek has quit IRC | 11:28 | |
| *** markvoelker has joined #openstack-infra | 11:29 | |
| *** isaacb has quit IRC | 11:30 | |
| *** markvoelker has quit IRC | 11:34 | |
| *** pvaneck has joined #openstack-infra | 11:37 | |
| *** pvaneck has quit IRC | 11:42 | |
| *** dizquierdo has quit IRC | 11:42 | |
| *** bhavik1 has quit IRC | 11:43 | |
| *** jtomasek has joined #openstack-infra | 11:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 11:48 |
| *** pradk has joined #openstack-infra | 11:48 | |
| *** Julien-zte has joined #openstack-infra | 11:49 | |
| *** rfolco has joined #openstack-infra | 11:50 | |
| *** coreyob has joined #openstack-infra | 11:50 | |
| andreaf | clarkb, fungi: I believe used to be able to hit the gerrit API directly at review.openstack.org, but I get 404 now - is the API still publicly accessible? endpoint? | 11:51 |
| *** lucas-hungry is now known as lucasagomes | 11:55 | |
| *** thorst has joined #openstack-infra | 11:58 | |
| *** rook is now known as rook-pto | 11:59 | |
| *** rook-pto is now known as rook | 11:59 | |
| *** apetrich_ has joined #openstack-infra | 12:00 | |
| *** markvoelker has joined #openstack-infra | 12:00 | |
| *** jkilpatr has quit IRC | 12:00 | |
| *** jkilpatr has joined #openstack-infra | 12:01 | |
| fungi | andreaf: it should still be there... for example gertty relies on it for all gerrit interactions and i'm using that just fine | 12:04 |
| *** markvoelker has quit IRC | 12:04 | |
| *** mat128 has joined #openstack-infra | 12:05 | |
| *** vhosakot has joined #openstack-infra | 12:05 | |
| fungi | andreaf: https://review.openstack.org/projects/ is working just fine for me in a browser too | 12:09 |
| *** markvoelker has joined #openstack-infra | 12:11 | |
| *** zz_dimtruck is now known as dimtruck | 12:11 | |
| *** markvoelker has quit IRC | 12:15 | |
| *** jkilpatr has quit IRC | 12:16 | |
| *** markvoelker has joined #openstack-infra | 12:17 | |
| *** prateek has joined #openstack-infra | 12:21 | |
| *** markvoelker has quit IRC | 12:22 | |
| sshnaidm | hi all | 12:25 |
| sshnaidm | for merging my patch https://review.openstack.org/#/c/467620 zuul collected 16(!) changes: http://logs.openstack.org/20/467620/2/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/93cb247/console.html#_2017-05-25_10_12_46_954250 | 12:25 |
| sshnaidm | is it expected behavior? | 12:26 |
| sshnaidm | http://logs.openstack.org/20/467620/2/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/93cb247/logs/reproduce.sh | 12:26 |
| *** gomarivera has joined #openstack-infra | 12:26 | |
| openstackgerrit | Matthew Booth proposed openstack/os-client-config master: Don't pop from os.environ https://review.openstack.org/468020 | 12:27 |
| mdbooth | Who here does os-client-config? mordred? | 12:28 |
| fungi | sshnaidm: if all those changes merged ahead of it, then yes that's normal. they presumably weren't merged yet when those jobs started | 12:28 |
| mdbooth | ^^^ is pretty severe | 12:28 |
| mdbooth | I've been tearing my hair out all morning trying to work out why the hell OVB deploy can't find my environment variables | 12:29 |
| mdbooth | Turns out we were throwing them away | 12:29 |
| *** ivyyy has quit IRC | 12:29 | |
| *** ivyyy has joined #openstack-infra | 12:29 | |
| andreaf | fungi: heh yeah sorry I must be doing the auth part wrong, it works without auth | 12:30 |
| fungi | mdbooth: yeah, that repo seems to be mostly mordred and dtroyer (under the guise of the sdks team) | 12:30 |
| *** alexchadin has joined #openstack-infra | 12:30 | |
| mdbooth | fungi: Thanks | 12:31 |
| fungi | andreaf: remember that authenticated methods are all prefixed with a/ | 12:31 |
| *** dprince has joined #openstack-infra | 12:31 | |
| andreaf | fungi: ah right, thanks | 12:31 |
| fungi | andreaf: and you need to use digest auth with the api key you generate in the gerrit webui | 12:32 |
| *** srobert has quit IRC | 12:34 | |
| *** gomarivera has quit IRC | 12:35 | |
| *** efoley__ has joined #openstack-infra | 12:35 | |
| *** gomarivera has joined #openstack-infra | 12:35 | |
| *** efoley_ has quit IRC | 12:39 | |
| *** rlandy has joined #openstack-infra | 12:40 | |
| *** gomarivera has quit IRC | 12:40 | |
| andreaf | fungi: missing trailing '/' - it always gets me | 12:40 |
| *** jkilpatr has joined #openstack-infra | 12:43 | |
| *** vikrant has quit IRC | 12:44 | |
| *** rcarrillocruz has quit IRC | 12:46 | |
| *** mat128 has quit IRC | 12:46 | |
| *** kgiusti has joined #openstack-infra | 12:48 | |
| *** bobh has joined #openstack-infra | 12:48 | |
| *** jpena|lunch is now known as jpena | 12:51 | |
| *** sbezverk has joined #openstack-infra | 12:52 | |
| *** links has quit IRC | 12:54 | |
| *** fguillot has joined #openstack-infra | 12:55 | |
| *** markvoelker has joined #openstack-infra | 12:56 | |
| *** yamamoto has quit IRC | 12:56 | |
| *** trown|outtypewww is now known as trown | 12:57 | |
| *** Goneri has joined #openstack-infra | 12:59 | |
| *** bh526r has joined #openstack-infra | 13:00 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: jenkins: scripts: Sync the .gitignore file in the OpenStack Ansible projects https://review.openstack.org/468023 | 13:01 |
| *** gouthamr has joined #openstack-infra | 13:01 | |
| *** yamamoto has joined #openstack-infra | 13:02 | |
| *** mat128 has joined #openstack-infra | 13:02 | |
| *** Julien-zte has quit IRC | 13:03 | |
| *** baoli has joined #openstack-infra | 13:05 | |
| dirk | clarkb: pabelanger : https://review.openstack.org/#/c/467922/ -- do the images need to be rebuild for that? | 13:06 |
| *** chlong has quit IRC | 13:06 | |
| *** psachin has quit IRC | 13:13 | |
| *** baoli has quit IRC | 13:13 | |
| *** baoli has joined #openstack-infra | 13:13 | |
| *** mdrabe has joined #openstack-infra | 13:16 | |
| *** wolverineav has joined #openstack-infra | 13:18 | |
| *** efoley_ has joined #openstack-infra | 13:19 | |
| *** vhosakot has quit IRC | 13:19 | |
| mordred | mdbooth: ah! I see you were asking in here and then I responded over in #openstack-dev. whoops | 13:21 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable E305 pep8 errors https://review.openstack.org/466286 | 13:21 |
| *** efoley__ has quit IRC | 13:23 | |
| *** mriedem has joined #openstack-infra | 13:23 | |
| mordred | mdbooth: actually - don't worry about it - I can take care of it | 13:29 |
| *** ihrachys has joined #openstack-infra | 13:30 | |
| pabelanger | dirk: yes | 13:31 |
| pabelanger | just catching up now | 13:31 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Don't pop from os.environ https://review.openstack.org/468020 | 13:31 |
| *** esberglu has joined #openstack-infra | 13:32 | |
| *** markvoelker has quit IRC | 13:33 | |
| mdbooth | mordred: Ah, thanks. I was just squinting at the code trying to work out what those lines are doing. | 13:33 |
| *** prateek has quit IRC | 13:33 | |
| *** sree has joined #openstack-infra | 13:33 | |
| mdbooth | mordred: Not important as long as OVB deploy runs :) | 13:33 |
| *** gordc has joined #openstack-infra | 13:35 | |
| pabelanger | mordred: fungi mind a +3 on https://review.openstack.org/#/c/467922 so we can kick off an image build things morning | 13:35 |
| mordred | mdbooth: yah - I realized it was a super easy follow up patch for me so just doing it was friendly :) | 13:37 |
| mordred | mdbooth: thanks again for the fix | 13:37 |
| fungi | pabelanger: i just noticed, i think our gentoo stanza there is missing the test profile keyword | 13:37 |
| *** spzala has joined #openstack-infra | 13:37 | |
| mordred | mdbooth: I shall try to get that (and a couple others) landed asap and a new release cut | 13:37 |
| mdbooth | mordred: Thanks | 13:38 |
| fungi | pabelanger: i'll get a follow-on change in there to fix | 13:38 |
| *** yamahata has joined #openstack-infra | 13:39 | |
| dirk | pabelanger: fungi : next one needed before image rebuild: https://review.openstack.org/#/c/467924/ | 13:39 |
| *** dizquierdo has joined #openstack-infra | 13:40 | |
| *** baoli_ has joined #openstack-infra | 13:40 | |
| *** sree has quit IRC | 13:42 | |
| *** sree has joined #openstack-infra | 13:43 | |
| *** baoli has quit IRC | 13:43 | |
| *** gomarivera has joined #openstack-infra | 13:44 | |
| pabelanger | dirk: we don't need an image rebuild for JJB jobs, only jenkins/scripts and jenkins/data are image build. | 13:44 |
| pabelanger | fungi: thanks | 13:44 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Install the test profile on Gentoo too https://review.openstack.org/468037 | 13:44 |
| fungi | pabelanger: ^ quick fix to slip in | 13:44 |
| fungi | prometheanfire isn't in here at the moment i guess | 13:45 |
| fungi | or i'd give him a heads up on that one | 13:45 |
| pabelanger | +2 | 13:45 |
| pabelanger | mordred: mind +3 on https://review.openstack.org/#/c/468037 | 13:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Add openSUSE support for install-distro-packages.sh https://review.openstack.org/467922 | 13:46 |
| *** eharney has joined #openstack-infra | 13:46 | |
| pabelanger | actually | 13:47 |
| pabelanger | we might as well also fix bindep_fallback.txt | 13:47 |
| pabelanger | that will need to be in image build to | 13:47 |
| pabelanger | too* | 13:47 |
| *** sree has quit IRC | 13:49 | |
| andreaf | fungi: is it normal that the reply of the gerrit API starts with )]}' before the actual json? | 13:52 |
| cloudnull | pabelanger: mordred: sorry was out yesterday, in today. you still seeing oddness in the osic? | 13:53 |
| pabelanger | cloudnull: ya, we still haven't been successful at an image upload | 13:54 |
| pabelanger | actually | 13:54 |
| pabelanger | fedora-25 was able to upload to osic-cloud1-s3700 | 13:55 |
| pabelanger | that was about 6 hours ago | 13:55 |
| dirk | pabelanger: well, I have a review for that | 13:57 |
| dirk | pabelanger: https://review.openstack.org/#/c/467934/ | 13:58 |
| dirk | pabelanger: but I can't get it tested because the install-system-pacakges are missing | 13:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: WIP: update bindep_fallback.txt for opensuse https://review.openstack.org/468042 | 13:58 |
| pabelanger | dirk: ah, cool ^ is some other things I noticed | 13:58 |
| cloudnull | pabelanger: are you seeing a timeout ? | 13:59 |
| cloudnull | maybe something DNS related? | 13:59 |
| * cloudnull testing an upload atm | 13:59 | |
| pabelanger | cloudnull: yes, we timeout on upload, I thinkwe try again then get 409 back | 13:59 |
| pabelanger | mordred: ^ | 13:59 |
| pabelanger | cloudnull: I agree it might be DNS related, since this started after the domain switch 2 days ago | 13:59 |
| mordred | cloudnull: did you see the trace I posted? | 13:59 |
| cloudnull | no. | 14:00 |
| * cloudnull reading back more | 14:00 | |
| mordred | cloudnull: http://paste.openstack.org/show/610606/ | 14:00 |
| mordred | cloudnull: that's client-side http tracing | 14:00 |
| * cloudnull reding http://paste.openstack.org/show/610606/ | 14:00 | |
| mordred | cloudnull: complete with request-ids just for you | 14:00 |
| *** isaacb has joined #openstack-infra | 14:00 | |
| pabelanger | dirk: what is your SSH public key, I'll add you to the held opensuse node I am working on | 14:00 |
| cloudnull | " Conflict for url:" interesting | 14:01 |
| mordred | yah | 14:01 |
| dirk | pabelanger: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6JHF/jzetTlV4YgjlNrx0BRh/aAuSW13dxYJpLakBx dmueller@oldboy | 14:01 |
| *** vhosakot has joined #openstack-infra | 14:01 | |
| mordred | it's like the first connection timeout was a load-balancer or something | 14:01 |
| mordred | and the request actually succeeded on the openstack side eventually | 14:01 |
| *** dimtruck is now known as zz_dimtruck | 14:02 | |
| pabelanger | dirk: actually, the patchset is self testing | 14:02 |
| mordred | but we got the retriable error from the first timeout - so we tried again - and then when that got to the finalize point it discovered that there was an object already | 14:02 |
| pabelanger | dirk: so when you upload changes, a bindep job will run to validate | 14:02 |
| *** yamahata has quit IRC | 14:02 | |
| mordred | which hit the conflict error | 14:02 |
| pabelanger | sorry, should have said that before, but need coffee | 14:02 |
| mordred | at which point we did the delete | 14:02 |
| dirk | pabelanger: yes, but its not running against the depends-on code state | 14:02 |
| dirk | pabelanger: so its failing on the fix that AJ did this morning (which just merged) | 14:03 |
| pabelanger | dirk: Ah | 14:03 |
| pabelanger | dirk: k, 1 sec | 14:03 |
| fungi | andreaf: yes, while annoying that's an intentional mitigation against xss/csrf attacks where javascript might get used to call api methods... that leading garbage prevents the browser from interpreting it as valid json | 14:03 |
| cloudnull | pabelanger: mordred: ok will look into this and let you know what i find out. | 14:03 |
| mordred | cloudnull: cool! I'm mostly super curious because that sequence of http calls and responses makes no sense to me | 14:04 |
| andreaf | fungi: heh it makes sense, thank | 14:04 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 14:04 |
| mordred | cloudnull: I'd put it in the category of "that should never happen" :) | 14:04 |
| cloudnull | yea, its a little odd. I suspect I've foobar'd something on the F5 | 14:05 |
| *** rbrndt has joined #openstack-infra | 14:05 | |
| mordred | cloudnull: integration with load-balancing tier has been at the root of a high percentage of issues we've seen with openstack clouds | 14:05 |
| pabelanger | dirk: ssh root@185.62.207.173 | 14:06 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 14:06 |
| *** apetrich_ has quit IRC | 14:06 | |
| mordred | cloudnull: because those edge devices affect the http transport in ways not defined by the API itself | 14:06 |
| *** apetrich has joined #openstack-infra | 14:06 | |
| pabelanger | dirk: work? | 14:07 |
| mordred | but since the openstack api is deployed by multiple people, we don't actually define what the lb/edge tier behavior _should_ be | 14:07 |
| *** esikachev has joined #openstack-infra | 14:07 | |
| pabelanger | dirk: yup :) | 14:07 |
| cloudnull | yea and we add quite a few acls to the mix so there | 14:07 |
| cloudnull | there's quite a bit that I could have messed up | 14:07 |
| *** ccamacho has joined #openstack-infra | 14:07 | |
| pabelanger | dirk: okay, going to get some coffee, that node is held so you can work on updating bindep_fallback.txt. | 14:07 |
| *** jaypipes has joined #openstack-infra | 14:09 | |
| *** spzala has quit IRC | 14:14 | |
| *** markvoelker has joined #openstack-infra | 14:15 | |
| *** ijw__ has quit IRC | 14:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add mailing list for API Users and SDK authors https://review.openstack.org/468046 | 14:16 |
| *** jtomasek has quit IRC | 14:16 | |
| *** sree has joined #openstack-infra | 14:17 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 14:17 |
| *** alexchadin has quit IRC | 14:18 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: preparing infra servers in RDO cloud https://review.openstack.org/468047 | 14:20 |
| mordred | fungi: ^^ mailing list request patch is follow on to the sdk-wrangling work flanders and I have been doing | 14:20 |
| *** sree has quit IRC | 14:22 | |
| *** flepied has quit IRC | 14:22 | |
| *** markvoelker has quit IRC | 14:23 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 14:23 |
| *** yamamoto has quit IRC | 14:24 | |
| *** aspiers has quit IRC | 14:24 | |
| *** r-daneel has joined #openstack-infra | 14:25 | |
| *** jamesdenton has joined #openstack-infra | 14:27 | |
| *** armax has joined #openstack-infra | 14:27 | |
| *** chlong has joined #openstack-infra | 14:27 | |
| pabelanger | dirk: ^ready for review? | 14:28 |
| dirk | pabelanger: I think yes. I identified two problems though | 14:28 |
| dirk | the base distro of 42.2 doesn't have liberasurecode-devel, so that's going to be a problem for swift | 14:28 |
| dirk | currently we have a extra repository with the openstack packages in the versions that openstack needs them | 14:29 |
| dirk | what would be a good way to loop that into the image? | 14:29 |
| *** zz_dimtruck is now known as dimtruck | 14:29 | |
| dirk | so if we want to fix that first then we need to add an extra repo first and then undo some of the removals | 14:29 |
| *** felipemonteiro has joined #openstack-infra | 14:30 | |
| pabelanger | dirk: in that case, we've been having jobs opt into extra repos for missing packages. If swift is going to run on opensuse, we can do the same for job run time | 14:30 |
| *** Swami has joined #openstack-infra | 14:30 | |
| pabelanger | but, we'll need to likely setup a mirror | 14:30 |
| pabelanger | where is the repo today? | 14:30 |
| pabelanger | when is 42.3 released? | 14:31 |
| dirk | I guess http://download.opensuse.org/repositories/Cloud:/OpenStack:/Master/openSUSE_Leap_42.2/ | 14:31 |
| dirk | but we can make a smaller one as well with just the dependencies | 14:31 |
| pabelanger | will 42.3 fix this? | 14:31 |
| *** Swami has quit IRC | 14:31 | |
| dirk | yes | 14:32 |
| dirk | 42.3 is in feature freeze right now | 14:32 |
| dirk | so we could start iwth the work on 42.3 immediately instead | 14:32 |
| dirk | it is fairly usable already, we have all the ci stuff running against 42.3 already | 14:32 |
| pabelanger | right, that is what I am thinking | 14:32 |
| dirk | end release is going to be mid july | 14:32 |
| pabelanger | lets get 42.2 online now, but hold off on things that 42.3 will fix | 14:33 |
| pabelanger | then, it should be straightforward to migrate to 42.3 in july | 14:33 |
| *** claudiub has quit IRC | 14:33 | |
| *** isaacb has quit IRC | 14:34 | |
| dirk | pabelanger: ok, let me change the blacklistings to 42.2 specifically then, so we'll have that fixed for 42.3 then | 14:35 |
| *** dimtruck is now known as zz_dimtruck | 14:35 | |
| *** dtantsur is now known as dtantsur|brb | 14:36 | |
| *** ram5391 has joined #openstack-infra | 14:37 | |
| *** zz_dimtruck is now known as dimtruck | 14:37 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 14:38 |
| *** ykarel has quit IRC | 14:38 | |
| *** ykarel has joined #openstack-infra | 14:38 | |
| *** yamamoto has joined #openstack-infra | 14:39 | |
| *** sree has joined #openstack-infra | 14:41 | |
| TheJulia | Greetings Infra! Bifrost's stable/ocata gate is presently broken because we expect /opt/git/openstack/requirements to match the requested branch for the job. Previously we were under the impression that the /opt/git/openstack/* folders would be updated from cache to match the branch being requested. That does not appear to be the case so, I'm wondering if | 14:42 |
| TheJulia | we should be tooling off of $WORKSPACE/ and add requirements as a repo that is cloned as well. | 14:42 |
| *** aspiers has joined #openstack-infra | 14:43 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: [NO MERGE] test multinode containers https://review.openstack.org/467100 | 14:43 |
| *** yamamoto has quit IRC | 14:44 | |
| *** hongbin has joined #openstack-infra | 14:45 | |
| *** sree has quit IRC | 14:46 | |
| *** ram5391 has quit IRC | 14:46 | |
| AJaeger | https://review.openstack.org/467924 is also needed for openSUSE - could a project-config core review, please? | 14:46 |
| AJaeger | TheJulia: do you have an exacmple job and log file for me? | 14:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Install the test profile on Gentoo too https://review.openstack.org/468037 | 14:47 |
| TheJulia | AJaeger: http://logs.openstack.org/43/468043/2/check/gate-bifrost-integration-tinyipa-ubuntu-xenial/51f524d/console.html | 14:48 |
| TheJulia | AJaeger: I put a git log | head -n20 at the very beginning of the test script that is invoked | 14:48 |
| *** thorst_ has joined #openstack-infra | 14:49 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 14:49 |
| AJaeger | TheJulia: http://logs.openstack.org/43/468043/2/check/gate-bifrost-integration-tinyipa-ubuntu-xenial/51f524d/console.html#_2017-05-25_14_17_54_586418 has "Prepared openstack/requirements repo with branch stable/ocata at commit cbda84e5ef30d28e8c681642dae7cdb75ef16880" - that looks correct, doesn't it? | 14:49 |
| TheJulia | at a quick glance, the output looks like it is master branch | 14:49 |
| AJaeger | TheJulia: you look at /opt/git/openstack/requirements - that one is not updated. | 14:49 |
| AJaeger | What do you need from /opt/git/openstack/requirements ? | 14:50 |
| pabelanger | TheJulia: AJaeger: we need to add net-info macro to that job too, it will say how old the image is | 14:50 |
| TheJulia | path to requirements on disk | 14:50 |
| pabelanger | to see if the cache is stale | 14:50 |
| dirk | AJaeger: pabelanger : sorry guys, there is an issue with the platform:opensuseproject-42.2 specifier, bindep can not parse that (the dot seems hurting) | 14:50 |
| AJaeger | Argh ;( | 14:50 |
| *** gyee has joined #openstack-infra | 14:50 | |
| *** Julien-zte has joined #openstack-infra | 14:50 | |
| AJaeger | TheJulia: why do you need path to requirements on disk? If you need requirements, you need to clone it explicitely. | 14:51 |
| pabelanger | dirk: okay, lets just remove it for now | 14:51 |
| pabelanger | dirk: and we'll patch bindep | 14:51 |
| TheJulia | AJaeger: We need it to be the single source of truth on disk of the requirements. Is there such a location on disk as of zuul-cloner exiting? | 14:51 |
| fungi | TheJulia: contents of /opt/git are not guaranteed to even be super recent much less checked out to a specific branch for the job you're running. we recommend using zuul-cloner to make a copy of the repo to work from (which will use the cache in /opt/git, update it over the network and checkout the exact branch or ref your job needs according to envvars zuul provides in the worker's shell environment) | 14:51 |
| *** thorst has quit IRC | 14:51 | |
| TheJulia | Okay, so the explanation I got originally was wrong. yay! | 14:52 |
| fungi | TheJulia: we install zuul-cloner into a virtualenv (to avoid polluting the global environment with its dependencies), so calling /usr/zuul-env/bin/zuul-cloner | 14:52 |
| fungi | should get you what you want | 14:52 |
| TheJulia | Just interesting that it is doing something with requirements.... Where is it putting that on disk? | 14:53 |
| pabelanger | Right, our osic images about lagging about 3 days now. So the git cache on those images are going to be stale for sure | 14:53 |
| fungi | TheJulia: if you're running under devstack-gate, then the git repos in /opt/stack/new _are_ checked out to the right branches/refs, as an alternative solution for you | 14:53 |
| AJaeger | TheJulia: your integration job already uses zuul-cloner, add requirements to that list if you need it checked out | 14:53 |
| TheJulia | AJaeger: Yup, that is what I'm going to do first | 14:54 |
| pabelanger | dirk: should it be !platform:suse ? | 14:54 |
| *** mdrabe has quit IRC | 14:54 | |
| dirk | pabelanger: both work. I took the opensuseproject as a mental reminder that these are 42.2 specific workarounds | 14:54 |
| pabelanger | dirk: okay, can me make consistent with other changes in patch | 14:55 |
| *** spzala has joined #openstack-infra | 14:55 | |
| pabelanger | opensuseproject or suse either work for me | 14:55 |
| dirk | well, suse means "this is always going to be that way" and "opensuseproject" means "this is only for 42.2" (until the specifier opensuseproject-42.2 works properly) | 14:56 |
| dirk | so it is consistent in that meaning. I can add that to the commit message to make it understandable | 14:56 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 14:57 |
| fungi | TheJulia: the thing you see it doing with openstack/requirements already in that console log is from the zuul-git-prep-upper-constraints macro which uses zuul-cloner to properly checkout the requirements repo into a tempdir, copy just the upper-constraints.txt file out of it to a known location, and then deletes the temporary clone of that repo so there's nothing reusable from it besides the | 14:57 |
| dirk | pabelanger: commit message updated | 14:57 |
| fungi | constraints list | 14:58 |
| pabelanger | dirk: thanks | 14:58 |
| pabelanger | +2 | 14:58 |
| pabelanger | AJaeger: ^ | 14:58 |
| AJaeger | pabelanger: +3 | 14:59 |
| *** ijw has joined #openstack-infra | 15:00 | |
| pabelanger | great, once that lands I'll kick off new images | 15:00 |
| *** dave-mccowan has joined #openstack-infra | 15:01 | |
| *** claudiub has joined #openstack-infra | 15:01 | |
| *** markvoelker has joined #openstack-infra | 15:01 | |
| *** mdrabe has joined #openstack-infra | 15:01 | |
| *** rossella__ has quit IRC | 15:01 | |
| AJaeger | thanks, pabelanger | 15:02 |
| AJaeger | dirk, will you put the bindep problem on your todo list, please? | 15:02 |
| pabelanger | dirk: do you also might updating configure_mirror to use syntax for other repos? http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n302 That way we know the contents we are writing to zypper repos | 15:03 |
| *** ltomasbo is now known as ltomasbo|away | 15:04 | |
| *** marst_ has joined #openstack-infra | 15:05 | |
| *** ijw has quit IRC | 15:05 | |
| *** cshastri has quit IRC | 15:06 | |
| *** dizquierdo has quit IRC | 15:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove .npm cache from root user https://review.openstack.org/468059 | 15:08 |
| *** shardy has quit IRC | 15:09 | |
| *** apetrich has quit IRC | 15:09 | |
| pabelanger | AJaeger: maybe we should start adding a big cowsay warning to if jobs are using bindep-fallback.txt still. Would be interesting to see which process still depending o nit | 15:10 |
| *** derekh has quit IRC | 15:10 | |
| fungi | we'd need to add cowsay to the bindep-fallback.txt file ;) | 15:11 |
| *** dizquierdo has joined #openstack-infra | 15:11 | |
| clarkb | out of nova neutron keystone swift cinder glance, cinder and glance don't have bindep.txts | 15:11 |
| clarkb | so I imagine its still very prevalent | 15:11 |
| *** apetrich has joined #openstack-infra | 15:12 | |
| *** ram5391 has joined #openstack-infra | 15:12 | |
| pabelanger | ya, I can start work on them | 15:12 |
| fungi | i thought nova merged one (they were my earliest test case) | 15:12 |
| clarkb | yes nova has it. Cinder and glance don't | 15:13 |
| fungi | oh, right i misread you the first time | 15:13 |
| clarkb | sorry comma is maybe to subtle | 15:13 |
| jeblair | yeah, i think that's called the 'clarkb' comma | 15:13 |
| clarkb | need more ; | 15:13 |
| pabelanger | we could do a mass copypaste of bindep-fallback.txt into project that have no bindep.txt file | 15:13 |
| jeblair | or the "portland comma" | 15:14 |
| pabelanger | then iterate from there | 15:14 |
| *** felipemonteiro has quit IRC | 15:14 | |
| clarkb | dirk re liberasurecode it is in tumbleweed so potential backport option there? (not sure what the rules are around that) | 15:15 |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config master: Explicitly clone requirements for bifrost https://review.openstack.org/468061 | 15:16 |
| *** dizquierdo has quit IRC | 15:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove .npm cache from stackviz element https://review.openstack.org/468059 | 15:17 |
| *** isaacb has joined #openstack-infra | 15:17 | |
| *** rossella__ has joined #openstack-infra | 15:18 | |
| dirk | AJaeger: yes, I was just looking at that | 15:18 |
| *** kjackal_ has joined #openstack-infra | 15:19 | |
| *** yamamoto has joined #openstack-infra | 15:20 | |
| *** yamamoto has quit IRC | 15:20 | |
| pabelanger | cloudnull: we seem to be seeing SSH timeouts for osic too, is that something you are able to confirm? | 15:21 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: toci_gate_test-orig.sh: add missing OVERCLOUD_DEPLOY_ARGS on ocata jobs https://review.openstack.org/468062 | 15:21 |
| pabelanger | I'm going to manually boot an image now and confirm | 15:21 |
| cloudnull | pabelanger: in a meeting, but will be happy to confirm | 15:23 |
| cloudnull | in a couple min | 15:23 |
| AJaeger | pabelanger: not to forget old branches ;( | 15:24 |
| mordred | TheJulia, fungi, AJaeger: it's worth noting that the situation with git repo state on disk changes when we roll out zuul v3 - and the git repos on the node WILL be set up in the state the job expects. this is not essential information today - but just thought I'd toss it out there so that in the future you'll be like "oh, right" | 15:25 |
| *** sree has joined #openstack-infra | 15:25 | |
| dirk | clarkb: yeah, its in 42.3 already.. I'll see what I can do about getting that added to 42.2 | 15:25 |
| pabelanger | cloudnull: thanks, no rush. | 15:25 |
| mordred | pabelanger: I like the big cowsay idea | 15:25 |
| dirk | AJaeger: https://storyboard.openstack.org/#!/story/2001042 -- more will come after public holiday ;-) | 15:25 |
| TheJulia | mordred: Thanks, filed away in the back of my brain :) | 15:26 |
| dirk | pabelanger: yes.. I'll do so | 15:27 |
| pabelanger | cloudnull: it appears to be intermittent network issue | 15:27 |
| *** sree has quit IRC | 15:27 | |
| *** sree has joined #openstack-infra | 15:28 | |
| *** apetrich has quit IRC | 15:29 | |
| *** panda is now known as panda|bbl | 15:30 | |
| *** sree has quit IRC | 15:32 | |
| tushar | Hi all ...any idea of this failure while DIB image creation - https://pastebin.com/0niH4p1h | 15:32 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/bindep master: Handle dotted versions in platform selectors https://review.openstack.org/468071 | 15:33 |
| *** wolverineav has quit IRC | 15:34 | |
| *** jamesmca_ has joined #openstack-infra | 15:34 | |
| pabelanger | tushar: odd, it is trying to connect to logstash.o.o | 15:34 |
| dirk | AJaeger: ^^ | 15:34 |
| pabelanger | 2017-05-25 20:12:46,263 INFO nodepool.image.build.dpc: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'logstash.openstack.org' ([Errno 101] Network is unreachable)") | 15:34 |
| *** wolverineav has joined #openstack-infra | 15:34 | |
| mhayden | AJaeger: i think i may have my needed-by/depends-on backwards in https://review.openstack.org/#/c/467339/ -- is it worth changing it or should i simply leave it be? :) | 15:34 |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config master: Explicitly clone requirements for bifrost https://review.openstack.org/468061 | 15:35 |
| tushar | Pablenger : Can we skip that ? | 15:35 |
| pabelanger | mtreinish: ^why do we need to connect to logstash.o.o again? | 15:35 |
| clarkb | pabelanger: tushar its trying to preseed the subunit info for test run order optimization | 15:35 |
| pabelanger | ah | 15:36 |
| pabelanger | clarkb: so not a hard requirement today? | 15:37 |
| fungi | gonna go grab some lunch and run errands... bbiaw | 15:37 |
| AJaeger | mhayden: you can leave it as is - but normally project-config merges faster, so if you revert, you get my +2 once PTL has +1ed... | 15:37 |
| clarkb | ya its not a hard requirement. tests will still run, just likely in an order that is less than optimal | 15:37 |
| AJaeger | mhayden: in any case, ansible PTL needs to +1 both changes | 15:38 |
| tushar | Pabelanger : my nodepool.yaml contains element as https://pastebin.com/9ajA9ENL | 15:38 |
| *** pcaruana has quit IRC | 15:38 | |
| mhayden | AJaeger: oh okay... looks like i've gotta go watch andymccr's car again | 15:38 |
| mhayden | :) | 15:38 |
| mhayden | s/watch/wash/ | 15:38 |
| tushar | Pabelanger : which are the mandatory elements for third party CI ? | 15:38 |
| AJaeger | mhayden: will you wash my bike as well? | 15:38 |
| mhayden | only if we can put it in the shed | 15:38 |
| *** shardy has joined #openstack-infra | 15:39 | |
| *** pcaruana has joined #openstack-infra | 15:39 | |
| mtreinish | clarkb: I thought we stopped doing that? | 15:39 |
| *** ltomasbo|away is now known as ltomasbo | 15:39 | |
| pabelanger | tushar: they should all be, this is failing because it seems you can connect to logstash.oopenstack.org. Are you restricting network access? | 15:39 |
| clarkb | mtreinish: maybe? | 15:39 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add ansible-hardening repository https://review.openstack.org/467339 | 15:39 |
| *** thorst_ has quit IRC | 15:39 | |
| pabelanger | mtreinish: oh, if we can remove it, that works for me | 15:39 |
| AJaeger | mhayden: it's in my garage;) | 15:39 |
| tushar | Pabelange : firewall might be restricting | 15:40 |
| mhayden | ah, i was hoping for bikeshedding! | 15:40 |
| AJaeger | mhayden: if you want to paint the garage as well ... ;) | 15:40 |
| mtreinish | pabelanger: we can, the optimization doesn't do much because of: https://bugs.launchpad.net/testrepository/+bug/1416512 | 15:40 |
| openstack | Launchpad bug 1416512 in Testrepository "Timing database shouldn't use attrs for test_id uniquness" [Wishlist,Triaged] | 15:40 |
| mhayden | haha | 15:40 |
| mtreinish | which is why I thought we removed it | 15:40 |
| pabelanger | mtreinish: okay, let me get a patch now | 15:41 |
| mtreinish | stestr doesn't have that bug, so if/when we switch to that we could try it again | 15:41 |
| mtreinish | because it's a good idea in theory :) | 15:41 |
| pabelanger | Ya, I'm surprised we have a public mysql instances actually | 15:41 |
| pabelanger | instance* | 15:41 |
| *** wolverineav has quit IRC | 15:42 | |
| clarkb | fungi: I'm going to work on sampling logs contents for disk usage this morning. To make sure we finish up work around running out of logs space again | 15:42 |
| mtreinish | pabelanger: oh, that by itself is useful. Accessing the subunit2sql data provides a lot of value | 15:42 |
| andymccr | mhayden: done! | 15:42 |
| mtreinish | but yeah, I'm kinda surprised someone hasn't exploited that endpoint yet | 15:43 |
| andymccr | mhayden: lucky i dont have a car :D | 15:43 |
| pabelanger | mtreinish: I take it we still need the subunit2sql-env or can that also be dropped? | 15:43 |
| clarkb | fungi: is your journal file find/delete still going? | 15:43 |
| *** jasondotstar has quit IRC | 15:43 | |
| pabelanger | looking at code, it seems we can also stop doing that | 15:43 |
| mhayden | AJaeger: it's received andymccr's blessing -> https://review.openstack.org/467339 | 15:44 |
| mtreinish | pabelanger: check the devstack-gate code, we might leverage the venv for subunit operations in there | 15:44 |
| mtreinish | like to generate the subunit stream on disk | 15:44 |
| mtreinish | I don't remember which venv we pulled that out of | 15:44 |
| *** leifmadsen has quit IRC | 15:45 | |
| AJaeger | mhayden: you're fast with car washing ;) | 15:46 |
| * mhayden works quickly | 15:46 | |
| AJaeger | mhayden: +2 - with one inline comment | 15:46 |
| *** lucasagomes is now known as lucas-afk | 15:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove 99-cache-testrepository-db from cache-devstack element https://review.openstack.org/468074 | 15:47 |
| pabelanger | mtreinish: clarkb: ^ | 15:47 |
| clarkb | pabelanger: re hitting the db every time we build an image I think that should be fine. its what 7 times a day or so? | 15:48 |
| *** dizquierdo has joined #openstack-infra | 15:48 | |
| pabelanger | clarkb: and every 3rd party CI too | 15:48 |
| pabelanger | clarkb: see tushar pastebin ^ | 15:48 |
| clarkb | ya I still think thats a likely small number for a db | 15:49 |
| *** leifmadsen has joined #openstack-infra | 15:49 | |
| pabelanger | sure, maybe we need to support OFFLINE installs for that | 15:49 |
| *** dtantsur|brb is now known as dtantsur | 15:49 | |
| AJaeger | pabelanger: any idea why there's no opensuse log files on http://nb04.openstack.org/ ? http://nb04.openstack.org/image.log starts with opensuse information... | 15:51 |
| *** jasondotstar has joined #openstack-infra | 15:51 | |
| *** e0ne has quit IRC | 15:51 | |
| pabelanger | AJaeger: we might have out data logging config | 15:51 |
| pabelanger | checking now | 15:51 |
| *** efoley_ has quit IRC | 15:52 | |
| AJaeger | pabelanger: do we need to update system-config/modules/openstack_project/templates/nodepool/nodepool-builder.logging.conf.erb ? | 15:52 |
| pabelanger | AJaeger: it should be latest | 15:53 |
| pabelanger | poh | 15:53 |
| pabelanger | I think I know why | 15:53 |
| AJaeger | pabelanger: indeed | 15:53 |
| pabelanger | checking | 15:53 |
| AJaeger | "INFO nodepool.image.build.opensuse-42.2" | 15:53 |
| AJaeger | in image.log - not 422 | 15:53 |
| AJaeger | is that the problem? | 15:53 |
| pabelanger | no, it is 422 | 15:54 |
| pabelanger | refresh image.log | 15:54 |
| pabelanger | I don't think nodepool was reconfigured to use latest logging file | 15:54 |
| pabelanger | trying to confirm | 15:54 |
| AJaeger | ok | 15:54 |
| clarkb | ok, tracked down an explosion in neutron dvr job logs | 15:55 |
| AJaeger | explosion? How bad is it? | 15:55 |
| clarkb | it is likely related to running out of disk space on the logs fs. ihrachys and haleyb are working on a fix in neutron | 15:55 |
| clarkb | AJaeger: 700MB on disk per dvr job | 15:56 |
| AJaeger | oops ;( | 15:56 |
| clarkb | AJaeger: regular dsvm job is ~125MB | 15:56 |
| clarkb | sorry no | 15:56 |
| clarkb | regular one is ~31MB | 15:56 |
| * ihrachys feels embarrassed :-x (even though I haven't landed the culprit) | 15:57 | |
| *** prometheanfire has joined #openstack-infra | 15:57 | |
| prometheanfire | who do I need to bug to get our channel topic updated (new meeting time for the requirements team) | 15:57 |
| clarkb | ihrachys: don't worry about it, now we know the problem and can fix it :) until the next one :) | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Enable test_one_branch cloner test https://review.openstack.org/467779 | 15:58 |
| *** ram5391 has quit IRC | 15:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Check out the appropriate branch in executor https://review.openstack.org/467778 | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use the executor cached repos more often https://review.openstack.org/467777 | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Return resulting commits from merger https://review.openstack.org/467776 | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add default-branch property to projects https://review.openstack.org/467334 | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to jobs https://review.openstack.org/467775 | 15:58 |
| AJaeger | prometheanfire: /msg chanserv access #channel list will tell you who has access to #channel - and thus whom to bug | 15:58 |
| pabelanger | AJaeger: Ya, we'll need to stop / start nodepool-builder | 15:59 |
| AJaeger | prometheanfire: I see "Prometheanfire" in that list, can you bug him ? ;) | 15:59 |
| openstackgerrit | Merged openstack-infra/project-config master: bindep-fallbacks for openSUSE https://review.openstack.org/467934 | 15:59 |
| *** jascott1 has joined #openstack-infra | 15:59 | |
| *** thorst has joined #openstack-infra | 16:00 | |
| *** ykarel has quit IRC | 16:00 | |
| *** jascott1_ has joined #openstack-infra | 16:00 | |
| pabelanger | AJaeger: http://nb03.openstack.org/ | 16:01 |
| pabelanger | I'll restart nb04 shortly | 16:01 |
| AJaeger | thanks, pabelanger | 16:01 |
| AJaeger | bbl | 16:01 |
| *** thorst has quit IRC | 16:02 | |
| *** pvaneck has joined #openstack-infra | 16:02 | |
| *** esikache1 has joined #openstack-infra | 16:03 | |
| *** jascott1 has quit IRC | 16:04 | |
| *** thorst has joined #openstack-infra | 16:04 | |
| pabelanger | cloudnull: it seems we might not be able to boot the new images we uploaded | 16:05 |
| pabelanger | cloudnull: maybe that means the compute nodes are not able to download said images? | 16:06 |
| *** esikachev has quit IRC | 16:06 | |
| pabelanger | fungi: AJaeger: When you have time: https://review.openstack.org/#/c/468074/ this will help tushar build DIBs | 16:07 |
| *** Goneri has quit IRC | 16:07 | |
| *** Apoorva has joined #openstack-infra | 16:07 | |
| *** Goneri has joined #openstack-infra | 16:07 | |
| *** DangerousDaren has quit IRC | 16:08 | |
| cloudnull | pabelanger: I think I need to do a rolling restart of services since the domain changed. I also might need to update the swift storage URL (saved in the DB) to get things talking to the right places. | 16:08 |
| *** electrofelix has quit IRC | 16:08 | |
| cloudnull | s/glance/swift/ | 16:09 |
| cloudnull | which is using swift | 16:09 |
| cloudnull | or reverse that ... | 16:09 |
| * cloudnull needs more coffee | 16:09 | |
| pabelanger | cloudnull: okay, sounds good | 16:09 |
| cloudnull | mordred: I checked the LB (f5) all looks good there. so no noticable pebkac there. | 16:09 |
| cloudnull | I suspect its services that still thing that osic.org exists. | 16:10 |
| *** Apoorva_ has joined #openstack-infra | 16:11 | |
| *** flepied has joined #openstack-infra | 16:11 | |
| mordred | cloudnull: oh - yah. and also any config files that reference direct URLs for things (like nova's glance config that has the api server location not "please use catalog") | 16:12 |
| mordred | cloudnull: this is a really good usecase, tbh, for why "use logical refernces via catalog" is important | 16:13 |
| *** pvaneck_ has joined #openstack-infra | 16:13 | |
| mordred | sdague: ^^ in case anyone ever gives pushback | 16:13 |
| sdague | mordred: sure | 16:14 |
| cloudnull | ++ | 16:14 |
| sdague | mordred: I honestly do get the glance fast path use case though | 16:14 |
| *** Apoorva has quit IRC | 16:15 | |
| *** esikache1 has quit IRC | 16:15 | |
| mordred | sdague: oh - totally | 16:15 |
| mordred | sdague: just saying for the general case - doing trickier things is fine to be tricky :) | 16:15 |
| sdague | sure | 16:15 |
| *** pvaneck has quit IRC | 16:16 | |
| *** Apoorva_ has quit IRC | 16:17 | |
| pabelanger | clarkb: glance seems to still be using swift for nodepool-dsvm python3: http://logs.openstack.org/94/452494/8/experimental/gate-dsvm-nodepool-py35-src-nv/cd67392/logs/screen-g-api.txt.gz | 16:17 |
| pabelanger | clarkb: was there anything else I needed to do? | 16:17 |
| pabelanger | I am also not sure why s-proxy is still trying to start: http://logs.openstack.org/94/452494/8/experimental/gate-dsvm-nodepool-py35-src-nv/cd67392/logs/screen-s-proxy.txt.gz | 16:17 |
| *** Apoorva has joined #openstack-infra | 16:17 | |
| *** jpich has quit IRC | 16:17 | |
| clarkb | pabelanger: you should look at the normal tempest py35 job to see what it does, but I thought disabling swift was sufficient. Maybe whatever is keeping s-proxy around is the problem | 16:18 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Fix issues in OVB preparement servers scripts https://review.openstack.org/467620 | 16:19 |
| mtreinish | clarkb: s-proxy is what glance looks for to setup the swift store | 16:19 |
| pabelanger | clarkb: ya, that was copied from tempest-dsvm-py35. it looks like we are failing to stop servers from starting | 16:20 |
| mtreinish | clarkb: https://github.com/openstack-dev/devstack/blob/master/lib/glance#L153-L181 | 16:20 |
| pabelanger | clarkb: but I'll dig more into it | 16:20 |
| *** esikache1 has joined #openstack-infra | 16:21 | |
| *** yamamoto has joined #openstack-infra | 16:21 | |
| *** jamesmca_ has quit IRC | 16:21 | |
| *** Apoorva has quit IRC | 16:22 | |
| *** Apoorva has joined #openstack-infra | 16:24 | |
| *** jamesmca_ has joined #openstack-infra | 16:26 | |
| *** shardy has quit IRC | 16:28 | |
| pabelanger | clarkb: it is possible they should be going into our devstack/setting file, not JJB? | 16:29 |
| *** ramishra has quit IRC | 16:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Log specific error message from RetriableConnectionFailure https://review.openstack.org/468091 | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add logging of non-standard error message documents https://review.openstack.org/468092 | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set some logger names explicitly https://review.openstack.org/468093 | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add novaclient interactions to http_debug https://review.openstack.org/468094 | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add documentation about shade's use of logging https://review.openstack.org/468095 | 16:29 |
| *** bhavik1 has joined #openstack-infra | 16:29 | |
| clarkb | pabelanger: well localrc is sourced so that should be fine unless we override in the setting file and enable swift there (I don't think we do) | 16:29 |
| clarkb | pabelanger: your best bet is probably reading the devstack log to see why swift is enabled and work back from there | 16:30 |
| *** pcaruana has quit IRC | 16:30 | |
| pabelanger | clarkb: best I can tell it is because local.conf doesn't contain our disabled services: http://logs.openstack.org/94/452494/8/check/gate-dsvm-nodepool/72984b6/logs/local.conf.txt.gz | 16:30 |
| *** esikache1 has quit IRC | 16:30 | |
| *** pcaruana has joined #openstack-infra | 16:30 | |
| pabelanger | I am expecting to see: http://logs.openstack.org/87/467687/1/gate/gate-tempest-dsvm-py35-ubuntu-xenial/de8f1f3/logs/local.conf.txt.gz | 16:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add mailing list for API Users and SDK authors https://review.openstack.org/468046 | 16:31 |
| *** yamamoto has quit IRC | 16:32 | |
| clarkb | pabelanger: see line 58 of nodepool.yaml jjb config | 16:32 |
| pabelanger | blarg | 16:32 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: [NO MERGE] test multinode containers https://review.openstack.org/467100 | 16:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix typo with DEVSTACK_LOCAL_CONFIG for nodepool-dsvm job https://review.openstack.org/468098 | 16:35 |
| pabelanger | clarkb: it's been like that since the beginning of the job | 16:36 |
| clarkb | pabelanger: I think we used to select what we wanted via enabled services directly then that broke when nova made the placement api so we switched to this disabled services stuff | 16:38 |
| clarkb | pabelanger: thats when we "broke" | 16:38 |
| clarkb | we didn't notice because default services is superset of what we need/want | 16:38 |
| pabelanger | good times | 16:39 |
| *** rcernin has quit IRC | 16:40 | |
| *** jheroux has joined #openstack-infra | 16:44 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Adds github triggering from status updates https://review.openstack.org/453844 | 16:45 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Implement pipeline requirement on github reviews https://review.openstack.org/453845 | 16:46 |
| prometheanfire | AJaeger: lol | 16:46 |
| *** dhajare has joined #openstack-infra | 16:47 | |
| prometheanfire | AJaeger: I can't op myself there | 16:47 |
| prometheanfire | or change the topic as is | 16:47 |
| *** ltomasbo is now known as ltomasbo|away | 16:48 | |
| *** jascott1 has joined #openstack-infra | 16:49 | |
| *** prateek has joined #openstack-infra | 16:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo with DEVSTACK_LOCAL_CONFIG for nodepool-dsvm job https://review.openstack.org/468098 | 16:49 |
| prometheanfire | AJaeger: nvm, got it | 16:50 |
| *** jascott1_ has quit IRC | 16:51 | |
| *** jtomasek has joined #openstack-infra | 16:53 | |
| *** esikache1 has joined #openstack-infra | 16:55 | |
| *** felipemonteiro has joined #openstack-infra | 16:58 | |
| *** Swami has joined #openstack-infra | 16:58 | |
| *** baoli_ has quit IRC | 16:59 | |
| *** esikache1 has quit IRC | 16:59 | |
| *** vhosakot has quit IRC | 17:00 | |
| *** ralonsoh has quit IRC | 17:01 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming https://review.openstack.org/463353 | 17:01 |
| *** ijw has joined #openstack-infra | 17:01 | |
| *** isaacb has quit IRC | 17:02 | |
| *** esikache1 has joined #openstack-infra | 17:02 | |
| *** dtantsur is now known as dtantsur|afk | 17:03 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming https://review.openstack.org/463353 | 17:03 |
| *** baoli has joined #openstack-infra | 17:04 | |
| *** yamamoto has joined #openstack-infra | 17:06 | |
| *** trown is now known as trown|lunch | 17:08 | |
| *** esikache1 has quit IRC | 17:10 | |
| *** yamamoto has quit IRC | 17:10 | |
| *** jamesmca_ has quit IRC | 17:11 | |
| *** SumitNaiksatam_ has quit IRC | 17:11 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Ensure PRs arent rejected for stale negative reviews https://review.openstack.org/460700 | 17:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Represent github change ID in status page by PR number https://review.openstack.org/460716 | 17:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Comment on PRs if a remote call to merge a change failed https://review.openstack.org/460762 | 17:13 |
| *** kjackal_ has quit IRC | 17:16 | |
| *** dhinesh has joined #openstack-infra | 17:17 | |
| *** sambetts is now known as sambetts|afk | 17:18 | |
| *** spzala has quit IRC | 17:20 | |
| *** dhinesh has quit IRC | 17:23 | |
| AJaeger | pabelanger: http://nb04.openstack.org/image-list gives me an "Internal server error" - shouldn't this work? | 17:26 |
| *** jpena is now known as jpena|off | 17:26 | |
| pabelanger | AJaeger: only nodepool.o.o | 17:26 |
| AJaeger | ok, htanks | 17:27 |
| AJaeger | wow, that one has a long list of osic failures ;/ | 17:27 |
| pabelanger | yes, working on resolving them | 17:27 |
| *** vhosakot has joined #openstack-infra | 17:31 | |
| *** prometheanfire has left #openstack-infra | 17:32 | |
| clarkb | fungi: there isn't a change yet to make the journal file cleanup more permanent is there? in my digging around this morning it seems like that is quite a bit of the cost. The tempest full jobs for example have quite a bit of journal log data | 17:35 |
| clarkb | and the dvr neutron logging explosion was amplified by the journal as we essentially had a copy in the neutron log file and in the journal file | 17:35 |
| clarkb | so I think we want to have a daily cron pruning those down to the last week? | 17:36 |
| clarkb | sdague: ^ you may be interested | 17:36 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: process branch locations based on input data not name prefix https://review.openstack.org/468111 | 17:38 |
| *** vhosakot has quit IRC | 17:40 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:41 | |
| fungi | clarkb: daily is probably not going to work great since my manual run went for >48 hours | 17:41 |
| *** lbragstad has joined #openstack-infra | 17:42 | |
| openstackgerrit | David Ames proposed openstack-infra/project-config master: Add charm-glusterfs https://review.openstack.org/468112 | 17:42 |
| fungi | and no, not aware of anything encoding yet. the first pass completed this morning after i was online | 17:42 |
| clarkb | fungi: but that was also pruning ~6 weeks at once right? | 17:42 |
| clarkb | whereas a daily run would only have to do a days worth? I guess it still has to list all the dirs though | 17:42 |
| fungi | i don't think that matters since it has to run through the whole tree, right | 17:43 |
| clarkb | fungi: maybe rerun it today and see how long it takes? | 17:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: WIP: Use ansible for zuulv3.openstack.org https://review.openstack.org/468113 | 17:43 |
| * fungi finishes looking at scrollback now | 17:43 | |
| lbragstad | hey folks - we have a couple patches to stable/ocata that are failing the requirements jobs, curious if there is a way to address this | 17:43 |
| clarkb | and then throw a flock around it and be as daily as possible? | 17:43 |
| pabelanger | clarkb: fungi: jeblair: mordred: so, maybe something like ^ is what we could bikeshed at next infra meeting | 17:43 |
| lbragstad | https://review.openstack.org/#/c/443881/1 is one of the reviews | 17:43 |
| pabelanger | clarkb: fungi: jeblair: mordred: still a 2 stage process, but one example of a migration to ansible | 17:44 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/system-config master: ignore files created by test jobs https://review.openstack.org/466039 | 17:45 |
| clarkb | pabelanger: for simplicity I wouldn't have a zuul specific playbook, I'd have a mostly use ansible to manage playbook and then you can add whatever to that | 17:45 |
| pabelanger | clarkb: agree, we'd likely need to reorg our playbook folder | 17:45 |
| clarkb | pabelanger: also that role seems to not be defined anywhere? | 17:46 |
| pabelanger | clarkb: right, I didn't add logic to add it to puppetmaster, I am not sure how we do that today | 17:46 |
| clarkb | we hae a list of things to galaxy I think | 17:47 |
| clarkb | using the module installer | 17:47 |
| pabelanger | we must, because we use cloud-launcher role | 17:47 |
| clarkb | I'd have to grep for it I don't remember where it is, just that we do such athing | 17:47 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: port list_deliverable_changes.py to python 3 https://review.openstack.org/460637 | 17:47 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: port list_deliverable_branches.py to python 3 https://review.openstack.org/460635 | 17:47 |
| pabelanger | clarkb: Ah, roles.yaml | 17:47 |
| *** Goneri has quit IRC | 17:48 | |
| *** dhinesh has joined #openstack-infra | 17:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: WIP: Use ansible for zuulv3.openstack.org https://review.openstack.org/468113 | 17:48 |
| lbragstad | digging in the review a bit and https://review.openstack.org/#/c/443881/1/requirements.txt seems to be inconsistent with the latest in https://github.com/openstack/requirements/blob/stable/ocata/requirements.txt | 17:49 |
| *** esikachev has joined #openstack-infra | 17:50 | |
| clarkb | lbragstad: so I think what happened was that chang ewas proposed, never merged, then we stopped running proposals to the stable branches and now it can't merge because you are right it doesn't match | 17:51 |
| fungi | clarkb: good idea. i'm running again now under time(1) in a screen session | 17:51 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Test openSUSE 42.3 image builds https://review.openstack.org/468115 | 17:51 |
| lbragstad | clarkb: ah - that kinda makes sense | 17:51 |
| clarkb | lbragstad: we typically don't propose requirements updates to stable so that must've snuck through when we were first creating new branches | 17:51 |
| lbragstad | clarkb: so would it be safe to abandon that review then? | 17:52 |
| clarkb | lbragstad: I think if you want to update the pbr requirement to match by hand that would be fine, but the change there can likely be abandoned | 17:52 |
| clarkb | yup | 17:52 |
| AJaeger | lbragstad: always safe to abandon then - if there weould be proposals, you would get a new one ;) | 17:52 |
| lbragstad | clarkb: awesome - i'll go through and do that then | 17:52 |
| openstackgerrit | David Ames proposed openstack-infra/project-config master: Add charm-glusterfs and charm-manila-glusterfs https://review.openstack.org/468112 | 17:54 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Prepare nodepool-builder logging config for openSUSE 42.3 https://review.openstack.org/468117 | 17:56 |
| *** claudiub has quit IRC | 17:56 | |
| clarkb | fungi: I do think once neutron gets their revert in we will gnerally be happier too | 17:56 |
| pabelanger | dirk: did you do that by hand ^ | 17:56 |
| pabelanger | dirk: we have a script in tools/ we use to generate it | 17:56 |
| dirk | pabelanger: yes. ah, didn't know that | 17:57 |
| openstackgerrit | David Ames proposed openstack-infra/project-config master: Add charm-glusterfs and charm-manila-glusterfs https://review.openstack.org/468112 | 17:57 |
| *** eharney has quit IRC | 17:57 | |
| pabelanger | dirk: the script depends on project-config, which we run usually have changes to nodepool.yaml | 17:57 |
| dirk | ah, ok. | 17:57 |
| *** dizquierdo has quit IRC | 17:58 | |
| dirk | pabelanger: I'm marking it as wip for now then. I first need to verify that diskimage-builder works and probably we have to mirror 42.3 first | 17:58 |
| pabelanger | clarkb: I've also plumbed though the virtualenv setting for pip task in ansible, so we can start running services in their own virtuelenvs if we wanted | 17:58 |
| openstackgerrit | David Ames proposed openstack-infra/project-config master: Add charm-glusterfs and charm-manila-glusterfs https://review.openstack.org/468112 | 17:59 |
| pabelanger | Hmm | 17:59 |
| pabelanger | 25 May 17:57:16 ntpdate[1576]: no servers can be used, exiting | 17:59 |
| pabelanger | wonder if we don't have ntpdate configured propelry | 18:00 |
| *** dhajare has quit IRC | 18:01 | |
| *** kjackal_ has joined #openstack-infra | 18:04 | |
| dirk | pabelanger: is that from the opensuse image? | 18:05 |
| dirk | there is no ntpd server configured by default | 18:05 |
| dirk | ah, its not coming via DHCP? | 18:05 |
| *** rossella__ has quit IRC | 18:06 | |
| amrith | clarkb was working on the trove libvirt-lxc stuff and happened upon https://wiki.openstack.org/wiki/HypervisorSupportMatrix. Does this mean that libvirt lxc is no longer supported? | 18:06 |
| pabelanger | dirk: no, that is xenail | 18:06 |
| pabelanger | dirk: we used to setup a static entry I think in puppet | 18:06 |
| pabelanger | it is possible we don't any more with DIB | 18:07 |
| pabelanger | I'll dig into it | 18:07 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: explicitly configure the openSUSE mirror paths https://review.openstack.org/468118 | 18:07 |
| * dirk afk | 18:07 | |
| *** HeOS has quit IRC | 18:07 | |
| *** baoli has quit IRC | 18:08 | |
| *** kjackal_ has quit IRC | 18:10 | |
| *** cody-somerville has joined #openstack-infra | 18:11 | |
| *** cody-somerville has joined #openstack-infra | 18:11 | |
| bulat | JJB reviewers, please take a look https://review.openstack.org/#/c/463554/ | 18:11 |
| *** david-lyle has joined #openstack-infra | 18:12 | |
| amrith | clarkb never mind, doc/source/support-matrix.ini obsoletes that stuff on the wiki :) | 18:12 |
| *** baoli has joined #openstack-infra | 18:14 | |
| *** baoli has quit IRC | 18:15 | |
| *** eharney has joined #openstack-infra | 18:16 | |
| pabelanger | clarkb: wow, all sorts of fail: http://logs.openstack.org/94/452494/8/experimental/gate-dsvm-nodepool-py35-src-nv/7bbad0f/logs/devstacklog.txt.gz#_2017-05-25_18_13_55_141 | 18:16 |
| clarkb | permissions issues? | 18:17 |
| clarkb | weird | 18:17 |
| pabelanger | ya seems that way | 18:18 |
| clarkb | btw reproduce.sh should work there I think (not sure if it captures the use python3 flag properyl) | 18:18 |
| pabelanger | cool | 18:18 |
| pabelanger | will give that a try | 18:18 |
| *** david-lyle has quit IRC | 18:19 | |
| pabelanger | clarkb: Hmm, we use pip3.5 above, but pip install in our plugins.sh | 18:20 |
| pabelanger | ya, it is using sudo above | 18:20 |
| pabelanger | let me see why this isn't an issue with 2.7 | 18:21 |
| clarkb | wait | 18:21 |
| clarkb | if shade is isntalling outside of the venv then that means the shade <-> nodepool cross testing isn't working | 18:21 |
| pabelanger | this is on the -src job | 18:22 |
| *** dgonzalez has quit IRC | 18:22 | |
| clarkb | ya but the src job installs shade globally , we install nodepool in a venv, that means we won't use the src shade that is global | 18:22 |
| *** dgonzalez has joined #openstack-infra | 18:22 | |
| clarkb | pabelanger: is that devstack installing shade on its own? | 18:25 |
| pabelanger | clarkb: it looks like we pip_install shade in devstack first, then we do it with nodepool | 18:25 |
| pabelanger | http://logs.openstack.org/94/452494/8/check/gate-dsvm-nodepool-src-nv/91a0f48/logs/devstacklog.txt.gz#_2017-05-24_20_50_12_154 | 18:25 |
| pabelanger | that is from working job | 18:26 |
| clarkb | ya but the plugin doesn't pip_install | 18:26 |
| pabelanger | setup_dev_lib? | 18:26 |
| pabelanger | I see that | 18:26 |
| clarkb | oh and the comemnt there ok | 18:26 |
| clarkb | I know why it does that, devsetack checks that libs from git get installed from git and if they aren't found it fails | 18:27 |
| pabelanger | so with http://logs.openstack.org/94/452494/8/check/gate-dsvm-nodepool-src-nv/91a0f48/logs/devstacklog.txt.gz#_2017-05-24_20_50_21_157 | 18:27 |
| pabelanger | I think is should be using sudo | 18:27 |
| pabelanger | but never fails, because nothing is installed | 18:27 |
| *** baoli has joined #openstack-infra | 18:27 | |
| pabelanger | but py35 does | 18:27 |
| pabelanger | http://logs.openstack.org/94/452494/8/experimental/gate-dsvm-nodepool-py35-src-nv/7bbad0f/logs/devstacklog.txt.gz#_2017-05-25_18_13_55_141 | 18:28 |
| clarkb | yes, its trying to do a setup.py develop/pip install -e which needs to write to the repo | 18:28 |
| pabelanger | I can fix our plugin.sh to use sudo | 18:28 |
| pabelanger | 1 sec | 18:28 |
| clarkb | I think the repos are owned by stack:stack because d-g will chown them? and we must be running as jenkins there | 18:29 |
| clarkb | pabelanger: http://logs.openstack.org/94/452494/8/check/gate-dsvm-nodepool-src-nv/91a0f48/logs/devstacklog.txt.gz#_2017-05-24_20_50_12_937 it uses sudo in the 2.7 job | 18:30 |
| *** baoli has quit IRC | 18:30 | |
| clarkb | so this is actually a bug in devstack I think | 18:31 |
| clarkb | when they set up the command for python3 they must lose the sudo somehow | 18:31 |
| pabelanger | clarkb: that is in devstack however | 18:31 |
| *** trown|lunch is now known as trown | 18:31 | |
| pabelanger | a few lines down it has no sudo | 18:32 |
| pabelanger | http://logs.openstack.org/94/452494/8/check/gate-dsvm-nodepool-src-nv/91a0f48/logs/devstacklog.txt.gz#_2017-05-24_20_50_21_157 | 18:32 |
| pabelanger | which is our plugin.sh script | 18:32 |
| *** baoli has joined #openstack-infra | 18:32 | |
| pabelanger | I'd be cool with moving that to use pip_install but not sure how to handle constraints | 18:33 |
| pabelanger | or virtualenv | 18:33 |
| clarkb | OH | 18:33 |
| clarkb | we are failing to setup.py develop/pip install -e in our virtualenv | 18:34 |
| clarkb | got it | 18:34 |
| pabelanger | ya, setup_develop appears to use sudo | 18:35 |
| pabelanger | which we then don't | 18:35 |
| clarkb | ya | 18:35 |
| pabelanger | and some reason, this works okay with python2.7 | 18:35 |
| clarkb | possibly beause python3 does more install checks than python2? | 18:35 |
| pabelanger | maybe | 18:35 |
| clarkb | we can drop the -e on the pip installs or we can sudo | 18:35 |
| pabelanger | really? | 18:36 |
| clarkb | ya because dropping the -e means we write into the virtualenv which we own | 18:36 |
| clarkb | and shouldn't need to write anything to the git repo | 18:36 |
| pabelanger | k | 18:36 |
| clarkb | -e just writes meta data files to the venv that says find your content over here in the git repo and so we need to be able to write to the git repo apparently | 18:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Drop -e for pip install for devstack plugin.sh https://review.openstack.org/468127 | 18:37 |
| pabelanger | k, check experimental in play | 18:37 |
| clarkb | I bet how -e works in python2 is different than in python3 and python3 is more hands on in the destination | 18:39 |
| *** gomarivera has quit IRC | 18:39 | |
| *** bhavik1 has quit IRC | 18:39 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Test openSUSE 42.3 image builds https://review.openstack.org/468115 | 18:40 |
| sdague | very early infra cloud fails? http://logs.openstack.org/98/467998/5/check/gate-nova-pep8-ubuntu-xenial/7484fd6/console.html#_2017-05-25_16_40_16_876676 | 18:43 |
| *** Sukhdev has joined #openstack-infra | 18:43 | |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-infra-cloud | 18:44 |
| pabelanger | only 2 launch failures in 6 hours | 18:44 |
| *** Goneri has joined #openstack-infra | 18:44 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Include tenant name in github context https://review.openstack.org/463871 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add cachecontrol to requests to github https://review.openstack.org/461587 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Save installation ids to a cache and fetch them per project https://review.openstack.org/463424 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Catch integration key file not found in zuul https://review.openstack.org/463425 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Include exc_info in reporter failure https://review.openstack.org/460765 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement github trigger requirement status https://review.openstack.org/463386 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Create github client each time it's called. https://review.openstack.org/463421 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Use integration_id with github https://review.openstack.org/463420 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Refactor integration key loading to always load https://review.openstack.org/463423 | 18:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Decode JSON body once for requests https://review.openstack.org/463422 | 18:44 |
| clarkb | sdague: that looks like it failed to update nova repo against zm06 | 18:45 |
| *** rcarrillocruz has joined #openstack-infra | 18:46 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2168&rra_id=all does look like we are hitting network throttling | 18:46 |
| *** dhajare has joined #openstack-infra | 18:47 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Mirror openSUSE 42.3 repository https://review.openstack.org/468130 | 18:47 |
| clarkb | 15.184.67.165 - - [25/May/2017:16:40:15 +0000] "GET /p/openstack/nova/info/refs?service=git-upload-pack HTTP/1.1" 200 335 "-" "git/2.7.4" from apache logs | 18:48 |
| clarkb | network issue between rax dfw and hpe infralcoud? | 18:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming https://review.openstack.org/463353 | 18:51 |
| *** jamesmca_ has joined #openstack-infra | 18:53 | |
| *** dhajare has quit IRC | 18:53 | |
| *** watersoul has quit IRC | 18:57 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add experimental openSUSE 42.2 based job for devstack and dib https://review.openstack.org/468132 | 18:58 |
| *** watersoul has joined #openstack-infra | 18:58 | |
| *** claudiub has joined #openstack-infra | 18:59 | |
| *** bh526r has quit IRC | 19:00 | |
| *** e0ne has joined #openstack-infra | 19:01 | |
| *** gomarivera has joined #openstack-infra | 19:02 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: explicitly configure the openSUSE mirror paths https://review.openstack.org/468118 | 19:03 |
| cloudnull | mordred: pabelanger: after a rolloing restart of glance / nova image uploads seems to be working now. | 19:05 |
| pabelanger | cloudnull: Nice, thanks. | 19:06 |
| cloudnull | as for interemitent network issues, I rebalanced the routers and things seem to be working ok. Are you still seeing an intermitent issue? | 19:06 |
| pabelanger | I'll keep an eye on logs | 19:06 |
| *** gomarivera has quit IRC | 19:06 | |
| pabelanger | some: http://grafana.openstack.org/dashboard/db/nodepool-osic | 19:06 |
| pabelanger | but lets wait and see | 19:06 |
| pabelanger | look to be clearing up | 19:06 |
| *** gomarivera has joined #openstack-infra | 19:07 | |
| cloudnull | I just got through the rolling restart/rebalancing so that's likely the massive spike | 19:07 |
| cloudnull | I've not had a lot of time to focus on this today, but my meetings have mostly completed for the day so I'm working on things now. | 19:08 |
| cloudnull | meetings will continue until moral improves :) | 19:08 |
| *** yamamoto has joined #openstack-infra | 19:08 | |
| pabelanger | cloudnull: np, thanks again for the help | 19:08 |
| cloudnull | absolutly. | 19:09 |
| * cloudnull is just happy that the cloud is still up and getting used :) | 19:09 | |
| *** pcaruana has quit IRC | 19:09 | |
| clarkb | pabelanger: latest nodepool python3 job failed bceause c-api did not start | 19:10 |
| clarkb | pabelanger: you'll likely need to grab the journal file and dump logs to find out why | 19:11 |
| *** gomarivera has quit IRC | 19:11 | |
| pabelanger | clarkb: ya, looking. It should be disabled, atleast we added it to be | 19:11 |
| clarkb | oh | 19:11 |
| pabelanger | http://logs.openstack.org/27/468127/1/experimental/gate-dsvm-nodepool-py35-src-nv/5f09014/logs/local.conf.txt.gz | 19:11 |
| openstackgerrit | Merged openstack-infra/project-config master: remove stable compat jobs from oslo libs https://review.openstack.org/467760 | 19:13 |
| *** yamamoto has quit IRC | 19:13 | |
| clarkb | ya, is_service_enabled even returned 1 for false | 19:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Temporary make stable compat ocata jobs non-voting" https://review.openstack.org/467990 | 19:14 |
| clarkb | start_cinder doesn't actually seem to check if cinder is enabled? | 19:14 |
| pabelanger | that is what I am looking at now | 19:15 |
| clarkb | ya thats a bug in devstack, run_process checks so won't actually start the service, but the start_cinder function then checks if the service is alive regardless | 19:15 |
| pabelanger | k | 19:16 |
| *** e0ne has quit IRC | 19:17 | |
| *** jtomasek_ has joined #openstack-infra | 19:17 | |
| *** wolverineav has joined #openstack-infra | 19:17 | |
| *** wolverineav has quit IRC | 19:22 | |
| pabelanger | clarkb: is the patch to wrap that logic in is_service_enable check? | 19:22 |
| mordred | cloudnull: awesome! thankyou! | 19:23 |
| pabelanger | clarkb: or not to bother with wait_for_service | 19:23 |
| *** jamesmca_ has quit IRC | 19:23 | |
| *** baoli has quit IRC | 19:23 | |
| *** jamesmca_ has joined #openstack-infra | 19:23 | |
| mordred | clarkb: it seems that you found the shade-related issue in nodepool-dsvm and there is nothing for me to investigate atm? | 19:24 |
| *** baoli has joined #openstack-infra | 19:25 | |
| *** xyang1 has joined #openstack-infra | 19:25 | |
| *** e0ne has joined #openstack-infra | 19:27 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: drop deprecated map-services/packages from zypper element https://review.openstack.org/454786 | 19:32 |
| *** apetrich has joined #openstack-infra | 19:33 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add experimental openSUSE 42.2 based job for devstack and dib https://review.openstack.org/468132 | 19:37 |
| pabelanger | clarkb: maybe we also need disable_service cinder | 19:37 |
| clarkb | pabelanger: not to bother with wait_for_service | 19:38 |
| clarkb | pabelanger: ya that might also do it, let me see what calls start_cinder | 19:38 |
| *** trown is now known as trown|brb | 19:38 | |
| clarkb | ah yup start_cinder has a is_service_enabled cinder check so we can just do that | 19:38 |
| clarkb | mordred: its fine, I misunderstood what the install was doing | 19:39 |
| pabelanger | clarkb: k, incoming patch | 19:39 |
| mordred | clarkb: cool | 19:39 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Test openSUSE 42.3 image builds https://review.openstack.org/468115 | 19:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add novaclient interactions to http_debug https://review.openstack.org/468094 | 19:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add documentation about shade's use of logging https://review.openstack.org/468095 | 19:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add logging of non-standard error message documents https://review.openstack.org/468092 | 19:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set some logger names explicitly https://review.openstack.org/468093 | 19:40 |
| *** trown|brb is now known as trown | 19:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Also include disable_service cinder for nodepool dsvm https://review.openstack.org/468144 | 19:40 |
| pabelanger | clarkb: ^ | 19:41 |
| *** hashar has joined #openstack-infra | 19:42 | |
| *** apetrich has quit IRC | 19:42 | |
| *** jamesmca_ has quit IRC | 19:45 | |
| *** gomarivera has joined #openstack-infra | 19:47 | |
| *** funzo has quit IRC | 19:48 | |
| *** funzo has joined #openstack-infra | 19:49 | |
| *** bobh has quit IRC | 19:50 | |
| pabelanger | Oh wow | 19:50 |
| pabelanger | The following 2022 NEW packages are going to be installed | 19:50 |
| pabelanger | that is for opensuse bindep job | 19:50 |
| pabelanger | telnet://104.130.119.67:19885 | 19:51 |
| clarkb | nice, zypper is fast houldn't take too long :) | 19:51 |
| pabelanger | Ya, once we warm up the regional mirrors :) | 19:51 |
| pabelanger | AJaeger: dirk ^ | 19:52 |
| pabelanger | job running now | 19:52 |
| *** annegentle has joined #openstack-infra | 19:52 | |
| clarkb | looks like cinder jobs may also load up the journal files | 19:53 |
| clarkb | smcginnis: ^ any idea why? | 19:53 |
| mordred | pabelanger: why does it need to install so many packages? | 19:53 |
| pabelanger | mordred: not sure, they must be getting pulled in a dependencies | 19:53 |
| pabelanger | mordred: maybe suggested packages? | 19:54 |
| mordred | pabelanger: what is that job testing? | 19:54 |
| clarkb | oh maybe no more than the normal neutron job | 19:54 |
| pabelanger | mordred: that bindep-fallback.txt is valid | 19:54 |
| clarkb | so ya I think its largely the journal filse that are larger now. And made worse when we load up on logging | 19:54 |
| mordred | pabelanger: ah | 19:54 |
| clarkb | sdague: any thoughts on how we might make these smaller? | 19:54 |
| mordred | pabelanger: many of them seem to be tex related :) | 19:55 |
| smcginnis | clarkb: journal what? | 19:55 |
| *** priteau has quit IRC | 19:55 | |
| *** apetrich has joined #openstack-infra | 19:55 | |
| pabelanger | mordred: ya, hopefully AJaeger or dirk can comment | 19:55 |
| mtreinish | mordred: sphinx related? | 19:55 |
| clarkb | smcginnis: systemd journal files. cinder isn't at fault the "generic" jobs do it too | 19:55 |
| pabelanger | Overall download size: 943.0 MiB. Already cached: 0 B. After the operation, additional 2.1 GiB will be used. | 19:55 |
| clarkb | smcginnis: I just noticed it in cinder bceause you all seem to run a lot of jobs | 19:55 |
| pabelanger | 2.1 GB for bindep-fallback.txt will to large for jobs | 19:56 |
| mtreinish | mordred: all my upstream sphinx patches are to fix their broken latex support | 19:56 |
| smcginnis | clarkb: Way too many, unfortunately. | 19:56 |
| *** jamielennox is now known as jamielennox|away | 19:56 | |
| clarkb | smcginnis: so in aggregate you look bad :P | 19:56 |
| mordred | yah - it also came up with annegentle recently that for new contribs for docs, there's a whole bunch of tex that gets pulled in that isn't actually strictly needed for most people | 19:56 |
| smcginnis | clarkb: Glad it's not something specific to us, but if you need to track anything down, let me know. | 19:56 |
| smcginnis | clarkb: :) | 19:56 |
| *** Sukhdev has quit IRC | 19:57 | |
| mordred | pabelanger: it might be worthwhile investigating if there are things like tex that are getting pulled in via fallback that are actually opt-in things | 19:57 |
| mordred | pabelanger: I mean, at some point | 19:57 |
| clarkb | dmsimard: is there any way to make ara not host the fonts for patternfly itself? thats 15MB of data in every ara report we do | 19:58 |
| mordred | trolling the bindep-fallback list for unnecessary things is not goin to be superfun | 19:58 |
| pabelanger | mordred: Ya, I want to audit with projects are still missing bindep.txt, cause we could be trying to remove bindep-fallback.txt now | 19:58 |
| *** amotoki is now known as amotoki_away | 19:58 | |
| mordred | pabelanger: +100 | 19:58 |
| pabelanger | it's been about 3 cycles since we started using bindep.txt | 19:58 |
| *** amotoki_away is now known as amotoki | 19:59 | |
| smcginnis | mordred: Really? I don't think I heard about it until the end of O. | 19:59 |
| pabelanger | also makes bringing on new images easier | 19:59 |
| *** amotoki is now known as amotoki_away | 19:59 | |
| dmsimard | clarkb: I've asked patternfly devs about it more than once, I feel it's super clunky too and especially large. It's part of why I made sure ARA behaved well when gzipped on logs.o.o (believe you reviewed those patches). | 19:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Also include disable_service cinder for nodepool dsvm https://review.openstack.org/468144 | 19:59 |
| *** fguillot has quit IRC | 20:00 | |
| dmsimard | clarkb: last I know they were transitioning to a new set of fonts ( http://overpassfont.org/ .. open source derivative of the Red Hat font ) and that one is/will be available over CDN. | 20:01 |
| clarkb | dmsimard: could we have a "don't install fonts" option and just have it fallback on using broswer fonts? | 20:01 |
| *** Apoorva has quit IRC | 20:01 | |
| *** jamesmca_ has joined #openstack-infra | 20:02 | |
| *** esikachev has quit IRC | 20:02 | |
| *** Apoorva has joined #openstack-infra | 20:02 | |
| clarkb | sdague: and for the journal I'm thinking we can't keep grabbing the entire journal contents, ist killing the log server | 20:02 |
| mtreinish | clarkb: why are we uploading the whole journal? What is using it? | 20:03 |
| mtreinish | like is the expected use case to download the whole thing and work with journactl locally on it? | 20:03 |
| clarkb | mtreinish: yes | 20:03 |
| mtreinish | because that seems pretty unrealistic for most people, it's a big blob | 20:03 |
| clarkb | but its like 250MB per job | 20:03 |
| mtreinish | like I can see sdague liking that, but I can't ever see myself doing that :) | 20:04 |
| dmsimard | clarkb: I'll see what I can do -- in the mean time we should definitely make at least sure that ARA reports are gzipped with --best. It brings just the fonts down from 15MB to 5MB. | 20:05 |
| mordred | clarkb, mtreinish: it seems the like the cost/benefit balance may be out of whack on this one | 20:05 |
| mtreinish | mordred: yep | 20:05 |
| mordred | like- having the actual thing is powerful ... but it seems less likely to actually get used | 20:05 |
| dmsimard | clarkb: there were some conscious design decisions in ara to make sure it worked 100% gzipped so we should be taking advantage of it | 20:06 |
| mtreinish | mordred: especially when we have a hard enough time convincing people to look at logs that get rendered in their browsers | 20:06 |
| *** Apoorva has quit IRC | 20:06 | |
| mordred | mtreinish: yah | 20:06 |
| pabelanger | dmsimard: clarkb: can use just default to https://cdnjs.com/libraries/patternfly for css in gate? | 20:06 |
| pabelanger | some install flag | 20:06 |
| clarkb | dmsimard: is there a change for that? you mentioned one earlier but I don't recall it and I'm seeing 15MB installs on the logs server | 20:06 |
| dmsimard | pabelanger: I don't follow | 20:06 |
| clarkb | mordred: mtreinish ya working on a patch | 20:07 |
| pabelanger | dmsimard: rather then installing css files for jobs, just point them to the CND | 20:07 |
| pabelanger | then we don't need to bother copying css / js things | 20:07 |
| mordred | otoh - grabbing the journal, dumping it with all of the associated metadata and injecting into logstash somehow, then deleting the journal might have more of a chance of seeing the data get used | 20:07 |
| dirk | pabelanger: mordred : it seems to be triggered by "asciidoc" in the bindep file | 20:07 |
| dirk | I don't know what that one is good for | 20:07 |
| pabelanger | dirk: agree, I am trying to see what is using it | 20:07 |
| dmsimard | pabelanger: ara pulls xstatic-patternfly regardless, there needs to be something in ara to support some other way | 20:08 |
| mtreinish | mordred: or making it the input to os-loganalyze so it can use the metadata for viewing things in the web | 20:08 |
| mordred | pabelanger: gerrit | 20:08 |
| mordred | pabelanger: we install asciidoc so we can build gerrit | 20:08 |
| mordred | $::openstack_project::jenkins_params::asciidoc_package, # for building gerrit | 20:08 |
| mtreinish | mordred: but those are both things someone would have to write | 20:08 |
| mordred | from thick_slave.pp :) | 20:08 |
| mordred | mtreinish: totally | 20:08 |
| pabelanger | mordred: Oh, that is easy to fix then | 20:08 |
| dmsimard | clarkb: I had sent https://review.openstack.org/#/c/397773/ and https://review.openstack.org/#/c/441144/ | 20:09 |
| mordred | pabelanger: we should probably make sure gerrit has a bindep file :) | 20:09 |
| dmsimard | clarkb: I'm not sure if devstack-gate ends up gzipping the report, we should check. I know that openstack-ansible does a gzip --recursive --best on everything | 20:09 |
| pabelanger | mordred: yup, doing that now | 20:09 |
| *** felipemonteiro has quit IRC | 20:10 | |
| mordred | pabelanger: also - fwiw, "ant" seems to be in there so that we can build buck so that we can build gerrit | 20:10 |
| *** yamahata has joined #openstack-infra | 20:10 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Save less journal data https://review.openstack.org/468150 | 20:11 |
| clarkb | mtreinish: mordred ^ there is a stab at making journal less big | 20:11 |
| *** bulat has quit IRC | 20:11 | |
| *** david-lyle has joined #openstack-infra | 20:12 | |
| clarkb | dmsimard: thats only on the hosting side, we need to also compress on the build side | 20:12 |
| *** eharney has quit IRC | 20:12 | |
| mordred | clarkb: what about the service logs like nova/cinder etc - we already exporting those I assume? | 20:12 |
| dmsimard | clarkb: openstack-ansible compresses the files during the job | 20:12 |
| pabelanger | wow | 20:12 |
| dmsimard | clarkb: it doesn't rely on logs.o.o to compress some days later | 20:12 |
| pabelanger | Could not connect to gerrit at ssh://pabelanger@review.openstack.org:29418/openstack-ci/gerrit.git | 20:13 |
| *** jamesdenton has quit IRC | 20:13 | |
| clarkb | dmsimard: devstack-gate does not, can you push a patch https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n530 to that? | 20:13 |
| pabelanger | that is an old gitreview file | 20:13 |
| mordred | clarkb: oh- nevermind- I was reading the file wrongly | 20:13 |
| dmsimard | clarkb: https://github.com/openstack/openstack-ansible-tests/blob/401e02fd8c294ece49f55569f90adde300f6afc0/test-log-collect.sh#L61 | 20:13 |
| dmsimard | clarkb: yes, I can send a patch. | 20:13 |
| dirk | pabelanger: mordred : well, we can just not pull in asciidoc for opensuse then | 20:13 |
| *** bulat has joined #openstack-infra | 20:13 | |
| dmsimard | clarkb: but I mean, everything should be gzipped, not just ARA ? | 20:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/gerrit master: Add bindep.txt file https://review.openstack.org/468151 | 20:13 |
| clarkb | dmsimard: not sure I understand? | 20:13 |
| clarkb | dmsimard: its only gzipped if you gzip it | 20:14 |
| pabelanger | dmsimard: at this point, lets delete it from all operating system | 20:14 |
| pabelanger | dmsimard: sorry | 20:14 |
| pabelanger | dirk: ^ | 20:14 |
| dmsimard | clarkb: what I'm trying to say is that job artifacts that are meant to be uploaded to logs.o.o should be gzipped *before* they are collected | 20:14 |
| clarkb | dmsimard: yes, and that has to be explicitly done | 20:14 |
| clarkb | its not currently done for ara in the vast majority of our jobs | 20:15 |
| dmsimard | clarkb: that's what OSA does today and what every project should be doing | 20:15 |
| clarkb | osa is like a drop in the puddle compared to everything else | 20:15 |
| mordred | clarkb: it looks like that's going to export the service logs twice | 20:15 |
| clarkb | its great its done right there, but we have to do it right everywhere else | 20:15 |
| *** chlong has quit IRC | 20:15 | |
| clarkb | mordred: yes we already do that | 20:15 |
| mordred | kk | 20:15 |
| clarkb | mordred: except right now the first time is the entire journal | 20:15 |
| openstackgerrit | Jef Mallal proposed openstack-infra/jenkins-job-builder master: Builders: conditional-step fix https://review.openstack.org/456684 | 20:15 |
| clarkb | then individial services | 20:15 |
| mordred | (other order, but yes) | 20:16 |
| dmsimard | clarkb: correct -- so what I'm trying to say is that I can send a patch to gzip ARA explicitely but it'd be better if we could get something in the log collection perhaps. | 20:16 |
| dmsimard | but log collection is just a dumb scp right ? | 20:16 |
| clarkb | correct | 20:16 |
| clarkb | log collections job isn't to change formats | 20:16 |
| mordred | clarkb: for the new thing you did - is it possible with journalctl to get @devstack and kern and sudo all one? cause right now that's gonna make a devstack@ file and then a kern/sudo file - which seems to lose some o the benefits of combined but still be the same size | 20:16 |
| *** david-lyle has quit IRC | 20:17 | |
| *** bulat has quit IRC | 20:17 | |
| clarkb | mordred: they are different, in that the syslog file is serialized out to human readable format | 20:17 |
| clarkb | mordred: as are the individual service logs | 20:17 |
| mordred | ah - ok | 20:17 |
| clarkb | the devstack@* is a literal copy of the journal contents for things matching that pattern which you have to use journalctl to read | 20:17 |
| *** Apoorva has joined #openstack-infra | 20:17 | |
| mordred | I now understand | 20:17 |
| clarkb | its mostly a test to see how much smaller that makes it and then whether or not its worth keeping those files at all | 20:18 |
| mordred | dmsimard: perhaps a jjb macro that people can just add to the job that does the ara export | gzip ? | 20:18 |
| mordred | dmsimard: oh - it's actually just in devstack-gate - so doing the gzip there should be pretty easy no and capture it for most of the things? | 20:19 |
| dmsimard | mordred: I've been trying to think of a way for different jobs/builders to re-use ARA in general so that everything ends up in the same report at the end. Like for example some tripleo jobs use ARA but their report ended up being overwritten by devstack-gate (due to the same destination folder) | 20:20 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: gzip the ARA report after generating it https://review.openstack.org/468152 | 20:20 |
| dmsimard | clarkb: ^ | 20:20 |
| mordred | dmsimard: is it the same ara db getting exported in that case? | 20:20 |
| dmsimard | mordred: in the particular case of devstack gate, there's no database location specified so it ends up being at ~/.ara/ansible.sqlite | 20:21 |
| dmsimard | but the tripleo jobs specify a location for their database (maybe they shouldn't and that'd be the end of that) | 20:21 |
| mordred | dmsimard: cool. is our stuff configured to serve index.html.gz as an index do we know? | 20:21 |
| pabelanger | mordred: ha, we don't have any jobs on master branch for gerrit | 20:21 |
| mordred | pabelanger: yah - all of our stuff is always branch-oriented | 20:22 |
| pabelanger | mordred: guess we'll have to remember to add bindep.txt file then | 20:22 |
| dmsimard | mordred: yeah ara works perfectly gzipped on logs.o.o, this is gzipped: http://logs.openstack.org/01/468001/1/check/gate-openstack-ansible-tests-ansible-func-centos-7/30f5da0/logs/ara/ | 20:22 |
| mordred | ++ | 20:22 |
| mordred | dmsimard: sweet | 20:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove asciidoc from bindep-fallback.txt https://review.openstack.org/468154 | 20:24 |
| pabelanger | mordred: clarkb ^ dirk | 20:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove asciidoc / ant from bindep-fallback.txt https://review.openstack.org/468154 | 20:26 |
| pabelanger | dirk: AJaeger: still some bindep failures: http://logs.openstack.org/03/467803/1/check/gate-bindep-bindep-fallback-opensuse-422-nv/dd96fc4/console.html | 20:27 |
| dirk | pabelanger: yeah, just looking at it | 20:28 |
| dirk | I'm surprised to see that its checking for package names rather can provides | 20:28 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: gzip the ARA report after generating it https://review.openstack.org/468156 | 20:29 |
| *** jkilpatr has quit IRC | 20:29 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Further bindep-fallback fixes for openSUSE https://review.openstack.org/468157 | 20:31 |
| dmsimard | clarkb: I codesearch'd for ara generate commands and sent patches https://review.openstack.org/#/q/topic:ara-gzip | 20:32 |
| dirk | pabelanger: in theory.. ^^ | 20:33 |
| clarkb | dmsimard: thanks, will review | 20:33 |
| *** chlong has joined #openstack-infra | 20:33 | |
| *** esikachev has joined #openstack-infra | 20:33 | |
| *** chlong has quit IRC | 20:37 | |
| *** chlong has joined #openstack-infra | 20:38 | |
| pabelanger | dirk: should be able to rebase on 468154 | 20:39 |
| pabelanger | dirk: then we can see what gate job does | 20:39 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Further bindep-fallback fixes for openSUSE https://review.openstack.org/468157 | 20:40 |
| *** esikachev has quit IRC | 20:41 | |
| *** markvoelker has quit IRC | 20:42 | |
| pabelanger | just pkgconfig left | 20:42 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Further bindep-fallback fixes for openSUSE https://review.openstack.org/468157 | 20:43 |
| pabelanger | dirk: no rubyjems for 42.2? | 20:44 |
| pabelanger | gems* | 20:44 |
| dirk | pabelanger: well, it does exist, its called ruby2.X-rubygems or so | 20:45 |
| dirk | do we really need it for something? | 20:45 |
| mordred | clarkb, pabelanger: did y'all have a patch in flight dealing with gate-dsvm-nodepool-src ? | 20:45 |
| *** yamahata has quit IRC | 20:45 | |
| pabelanger | dirk: yes, we'll need to support it still | 20:45 |
| pabelanger | dirk: until we remove it from bindep-fallback.txt | 20:45 |
| pabelanger | mordred: yes, doing recheck now | 20:46 |
| mordred | pabelanger: cool. wasn't sure if I should be helping on that or not | 20:46 |
| pabelanger | mordred: TL;DR we didn't setup our localrc file for devstack properly | 20:46 |
| dirk | pabelanger: rubygems is installed by ruby I think | 20:46 |
| * dirk checks | 20:47 | |
| mtreinish | infra-root: if someone gets a sec can you restart the api server on health.o.o, it looks like it's behind by a few commits and my guess is the new code just isn't running in apache because it needs a restart | 20:47 |
| pabelanger | mordred: https://review.openstack.org/#/c/468098/ | 20:47 |
| pabelanger | mordred: which then broken on cinder, so I think we have it now | 20:47 |
| mordred | pabelanger: ah. cool | 20:47 |
| pabelanger | dirk: great | 20:47 |
| mordred | pabelanger: that reminds me- I have some patches up to make shade's jobs nicer - I should finish fixing them | 20:47 |
| clarkb | ok next on the list is stackviz | 20:48 |
| clarkb | it is largeish like ara but because it has 15MB json file | 20:48 |
| clarkb | which should be compressable too but i bet if we do that stackviz breaks? | 20:48 |
| pabelanger | mordred: ya, I am hoping nodepool-dsmv py35 job works now | 20:48 |
| clarkb | mtreinish: ^ any idae on stackviz? also I will poke health | 20:49 |
| mtreinish | clarkb: I think stackviz does break if the json is compressed | 20:49 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Further bindep-fallback fixes for openSUSE https://review.openstack.org/468157 | 20:49 |
| mtreinish | clarkb: I think timothyb89 said that was on his todo list for stackviz | 20:49 |
| dirk | pabelanger: rubygems situation fixed as well | 20:49 |
| *** yolanda has quit IRC | 20:49 | |
| *** jkilpatr has joined #openstack-infra | 20:49 | |
| clarkb | mtreinish: b1dddfd0e5691d56ff4041195818e47dc6177697 is commit in /opt/openstack-health | 20:50 |
| mtreinish | clarkb: that's current head, but I'm pretty sure that's not the version running | 20:50 |
| mtreinish | I think apache never restarted and it's running an old checkout sitll | 20:51 |
| clarkb | mtreinish: openstack-health==0.0.1.dev681 # git sha fa9c804 <- that is what pbr freeze says is installed | 20:51 |
| clarkb | so we haven't installed head | 20:51 |
| *** yamahata has joined #openstack-infra | 20:51 | |
| mtreinish | ok, that would do it | 20:51 |
| *** dhinesh has quit IRC | 20:52 | |
| dirk | pabelanger: it would be a nice feature to have a $rubygems in bindep.txt that is expanded "correctly" for all the distros. currently I"m fighting with various different bindeps everywhere | 20:52 |
| *** gomarivera has quit IRC | 20:52 | |
| mtreinish | clarkb: any indication in the logs as to why? Puppet should be installing it on each new commit | 20:52 |
| *** gomarivera has joined #openstack-infra | 20:53 | |
| *** gomarivera has quit IRC | 20:53 | |
| *** gomarivera has joined #openstack-infra | 20:53 | |
| pabelanger | dirk: ideally we want bindep.txt to be simple blob of data, so that means we need $rubygems packaging for each distro | 20:53 |
| pabelanger | guess everybody uses a different package name | 20:54 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Test openSUSE 42.3 image builds https://review.openstack.org/468115 | 20:54 |
| pabelanger | if only there was a universal packaging tool! | 20:54 |
| pabelanger | :D | 20:54 |
| clarkb | mtreinish: not in the last day or so | 20:54 |
| clarkb | mtreinish: when would the commit have been merged? | 20:54 |
| mtreinish | clarkb: heh, the last openstack-health commit was Apr 6th | 20:54 |
| pabelanger | dirk: you'll need to recheck, that is an old image | 20:55 |
| mtreinish | clarkb: the installed sha1 says oct 2016!? | 20:55 |
| pabelanger | dirk: we haven;t fully uploaded opensuse-422 to all regions today | 20:55 |
| clarkb | I can go back to may 18 | 20:55 |
| clarkb | log wise | 20:55 |
| mtreinish | http://git.openstack.org/cgit/openstack/openstack-health/commit/?id=fa9c804e4597b6189c3baad15d2d7a33754262ca | 20:56 |
| *** markvoelker has joined #openstack-infra | 20:56 | |
| mtreinish | yeah so it's probably been not working for a while and we just never noticed | 20:56 |
| *** ym200 has joined #openstack-infra | 20:58 | |
| clarkb | mtreinish: I think I see the bug | 20:59 |
| clarkb | mtreinish: the installation subscribes to the exec for requierments and is refreshonly I'm not sure exec resources do subscriptions/refresh | 20:59 |
| *** trown is now known as trown|outtypewww | 20:59 | |
| mtreinish | clarkb: they did in the past. I remember without the refresh only it was running on every puppet run | 20:59 |
| clarkb | I think you want both to subscribe and refershonly to the vcsrepo and have a require for the application on the requirements install to get the ordering right | 20:59 |
| mtreinish | I think | 21:00 |
| mtreinish | it's been a while | 21:00 |
| clarkb | mtreinish: ya that is what i would expect without refresh only | 21:00 |
| clarkb | but once yuo add refrseh only you need those internal pupept event which may nto fire in this case | 21:00 |
| *** markvoelker has quit IRC | 21:00 | |
| *** cody-somerville has quit IRC | 21:00 | |
| *** rfolco has quit IRC | 21:01 | |
| *** cody-somerville has joined #openstack-infra | 21:01 | |
| *** cody-somerville has quit IRC | 21:01 | |
| *** cody-somerville has joined #openstack-infra | 21:01 | |
| mtreinish | clarkb: wait don't we install it in editable mode? | 21:02 |
| clarkb | yes | 21:02 |
| mtreinish | that might explain why we didn't see it not reinstalling | 21:02 |
| mtreinish | but it still doesn't explain how it got stuck | 21:02 |
| mtreinish | because it should see the new code if it's committed right? | 21:03 |
| clarkb | I don't actually know howthat intersects with pip install -e | 21:03 |
| clarkb | it may just use the existing .pyc files and you lose | 21:03 |
| mtreinish | yeah, there is some weirdness there | 21:04 |
| *** jtomasek_ has quit IRC | 21:04 | |
| *** e0ne has quit IRC | 21:05 | |
| *** prateek has quit IRC | 21:05 | |
| fungi | i can't imagine any point to doing a system-wide install on an unattended server and using editable mode | 21:08 |
| *** jascott1 has quit IRC | 21:08 | |
| clarkb | its a venv too | 21:08 |
| clarkb | so its basically different than all our installas :) | 21:08 |
| *** jascott1 has joined #openstack-infra | 21:08 | |
| *** dprince has quit IRC | 21:10 | |
| *** chlong has quit IRC | 21:11 | |
| *** jascott1 has quit IRC | 21:13 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Save less journal data https://review.openstack.org/468150 | 21:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Don't record lvmdump, its not useful https://review.openstack.org/468164 | 21:15 |
| *** dimtruck is now known as zz_dimtruck | 21:16 | |
| jeblair | i thought we were going to change that, since the puppet machinery doesn't work with venvs? | 21:18 |
| *** yamahata has quit IRC | 21:18 | |
| jeblair | ie, i thought we were going to switch to our standard system-wide non-editiable install | 21:19 |
| *** wolverineav has joined #openstack-infra | 21:20 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming https://review.openstack.org/463353 | 21:20 |
| *** thorst has quit IRC | 21:22 | |
| *** ym200 has quit IRC | 21:23 | |
| *** sdague has quit IRC | 21:24 | |
| *** wolverineav has quit IRC | 21:24 | |
| *** jheroux has quit IRC | 21:25 | |
| *** esikachev has joined #openstack-infra | 21:27 | |
| *** kgiusti has quit IRC | 21:29 | |
| *** marst_ has quit IRC | 21:30 | |
| *** gardlt has joined #openstack-infra | 21:30 | |
| clarkb | ok neutron logging fix/revert is in | 21:30 |
| clarkb | we should start seeing that particular case of fun be less crazy now | 21:30 |
| clarkb | and dmsimard's change to make ara smaller appears to have worked, its just over 5MB total for ara now instead of 15MB | 21:32 |
| openstackgerrit | Omar Rivera proposed openstack-infra/project-config master: Add openstack/valet IRC channel https://review.openstack.org/468171 | 21:32 |
| *** dave-mccowan has quit IRC | 21:33 | |
| *** esikachev has quit IRC | 21:35 | |
| pvaneck_ | clarkb: hey, mind pushing this one through again: https://review.openstack.org/#/c/460787 ? It failed the gate the first time as the Gemfile needed adjusting. | 21:35 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/468173 | 21:35 |
| *** askb has quit IRC | 21:35 | |
| clarkb | pvaneck_: done | 21:36 |
| pvaneck_ | clarkb: thanks again! | 21:36 |
| *** hashar has quit IRC | 21:38 | |
| clarkb | and confirmed that neutron dvr jobs haev log totals inline with regular neutron (which are also large due to journal collection which my other change should hopefully address) | 21:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/468173 | 21:40 |
| *** cody-somerville has quit IRC | 21:42 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: drop deprecated map-services/packages from zypper element https://review.openstack.org/454786 | 21:43 |
| openstackgerrit | Merged openstack-infra/puppet-refstack master: Fix puppet lint errors https://review.openstack.org/460787 | 21:43 |
| pabelanger | clarkb: at a loss, cinder still enabled: http://logs.openstack.org/83/467783/1/check/gate-dsvm-nodepool/ec82716/logs/devstacklog.txt.gz | 21:43 |
| clarkb | is_service_enabled cinder certainly thinks so | 21:45 |
| clarkb | pabelanger: is_cinder_enabled is the problem :/ | 21:49 |
| clarkb | it doesn't check if cinder is enabled or disabled via the cinder name it checks if any of the c-* services are enabled, and c-bak is enabled :/ | 21:49 |
| *** jamesmca_ has quit IRC | 21:49 | |
| pabelanger | ah | 21:50 |
| pabelanger | so, should we also disable c-bak? | 21:50 |
| openstackgerrit | Jef Mallal proposed openstack-infra/jenkins-job-builder master: Add XUnit processing plugin to builders https://review.openstack.org/468175 | 21:50 |
| clarkb | so need to also disable c-bak | 21:50 |
| clarkb | ya | 21:50 |
| pabelanger | k | 21:50 |
| *** bobh has joined #openstack-infra | 21:50 | |
| clarkb | this has been a fun excursion into the insides of devstack | 21:51 |
| clarkb | I do think that if cinder is in disabled_services then is_cinder_enabled should return false. I will push that patch | 21:51 |
| pabelanger | clarkb: ya | 21:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: And disable c-bak for nodepool dsvm https://review.openstack.org/468176 | 21:52 |
| pabelanger | would be nice | 21:52 |
| pabelanger | clarkb: ^lets try | 21:52 |
| fungi | https://puppet.com/security/cve/cve-2017-2295 makes me glad we went masterless! | 21:52 |
| pabelanger | fungi: fun | 21:53 |
| fungi | debian only just updated their puppet packages for that today, ubuntu still hasn't that i can see | 21:53 |
| fungi | two weeks after it was announced | 21:53 |
| clarkb | thats at least the second time that has happened too | 21:54 |
| *** zz_dimtruck is now known as dimtruck | 21:54 | |
| *** cody-somerville has joined #openstack-infra | 21:55 | |
| *** bobh has quit IRC | 21:55 | |
| *** askb has joined #openstack-infra | 21:57 | |
| clarkb | pabelanger: https://review.openstack.org/468177 | 21:59 |
| pabelanger | clarkb: thanks | 22:00 |
| pabelanger | https://review.openstack.org/#/c/468176/ | 22:01 |
| pabelanger | should keep us moving forward, until 468177 lands | 22:01 |
| clarkb | ya approved | 22:01 |
| clarkb | fungi: https://arstechnica.com/information-technology/2017/05/in-a-throwback-to-the-90s-ntfs-bug-lets-anyone-hang-or-crash-windows-7-8-1/ is another fun bug | 22:03 |
| *** mdrabe has quit IRC | 22:03 | |
| fungi | hah! | 22:03 |
| timothyb89 | clarkb, mtreinish: stackviz mainly breaks because the logs server sends the wrong content type for *.json.gz, it would be safe to turn on compression if that was fixed | 22:03 |
| fungi | that should be fixable in the vhost config, yeah? | 22:04 |
| clarkb | timothyb89: ok so something like dmsimard's changes | 22:04 |
| clarkb | fungi: ya that | 22:04 |
| *** Apoorva_ has joined #openstack-infra | 22:04 | |
| *** jamesmca_ has joined #openstack-infra | 22:05 | |
| pabelanger | okay https://review.openstack.org/#/c/468157 and https://review.openstack.org/#/c/468154/ are our fixes for bindep for opensuse | 22:05 |
| pabelanger | if people don't mind reviewing | 22:05 |
| pabelanger | then we can make our opensuse bindep job voting tomorrow | 22:05 |
| *** markvoelker has joined #openstack-infra | 22:05 | |
| *** annegentle has quit IRC | 22:06 | |
| clarkb | eg https://review.openstack.org/#/c/441144/1/templates/logs.vhost.erb but for .json.gz | 22:06 |
| clarkb | timothyb89: is correct type text/json? | 22:06 |
| timothyb89 | clarkb: application/json | 22:06 |
| clarkb | ok let me push a patch | 22:07 |
| timothyb89 | clarkb: with content-encoding: gzip | 22:07 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Test openSUSE 42.2/42.3 image builds https://review.openstack.org/468115 | 22:08 |
| *** Apoorva has quit IRC | 22:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: And disable c-bak for nodepool dsvm https://review.openstack.org/468176 | 22:08 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstackci master: Set json.gz mimetype properly https://review.openstack.org/468180 | 22:09 |
| clarkb | timothyb89: fungi pabelanger dmsimard ^ | 22:09 |
| *** jascott1 has joined #openstack-infra | 22:09 | |
| *** jamesmca_ has quit IRC | 22:09 | |
| clarkb | then once that is deployed we can push a self testing update to d-g to make sure it all works happily | 22:09 |
| timothyb89 | clarkb: we'll also need to add a flag to stackviz-export to ask it to compress | 22:10 |
| *** jamesmca_ has joined #openstack-infra | 22:10 | |
| clarkb | timothyb89: ya I think we can start by having d-g just compress it after stackviz runs but tht would be more ideal | 22:10 |
| pabelanger | +2 | 22:10 |
| timothyb89 | clarkb: unfortunately stackviz needs to know the filename in advance | 22:10 |
| clarkb | timothyb89: I mean we can compress the results after stackviz runs which should be fine right? | 22:11 |
| clarkb | timothyb89: or do you mean it also hardcodes the paths to the files so we'd have to edit that too? | 22:11 |
| timothyb89 | clarkb: stackviz will generate a manifest file pointing to xyz.json, if it gets renamed to xyz.json.gz things will break | 22:12 |
| clarkb | got it | 22:12 |
| timothyb89 | clarkb: stackviz-export --gzip will just gzip by default :) | 22:12 |
| clarkb | I think our server may be smart and rewrite .json to .json.gz /me reads the vhost more | 22:12 |
| timothyb89 | ah that would work perfectly if so | 22:12 |
| clarkb | ya RewriteRule ^/(.*)$ %{REQUEST_URI}.gz | 22:13 |
| timothyb89 | ah cool :) | 22:13 |
| timothyb89 | I think stackviz should be fine without any other changes, then | 22:14 |
| *** esikachev has joined #openstack-infra | 22:14 | |
| *** jascott1 has quit IRC | 22:14 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/irc-meetings master: Update murano meeting time https://review.openstack.org/468182 | 22:14 |
| *** markvoelker has quit IRC | 22:15 | |
| *** markvoelker has joined #openstack-infra | 22:16 | |
| clarkb | timothyb89: easy enough to test once the little vhost edit lands | 22:16 |
| *** esikachev has quit IRC | 22:18 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: DIB: stop testing precise builds in non-voting https://review.openstack.org/468183 | 22:20 |
| *** thorst has joined #openstack-infra | 22:20 | |
| *** camunoz has joined #openstack-infra | 22:21 | |
| fungi | dirk: i spotted a concern with the bindep fallback change... it looks like you'll end up not installing mariadb-server nor mysql-server... is there an entry missing or did i overlook it? | 22:21 |
| pabelanger | fungi: I think mariadb brings it in | 22:22 |
| *** esberglu has quit IRC | 22:22 | |
| fungi | looking again | 22:23 |
| *** thorst has quit IRC | 22:23 | |
| *** esberglu has joined #openstack-infra | 22:23 | |
| fungi | ahh, i missed that was there... do we actually need to specify mariadb-server for the redhat family either then? | 22:24 |
| pabelanger | ya, think so | 22:24 |
| pabelanger | I have mariadb-server for fedora | 22:24 |
| *** camunoz has quit IRC | 22:26 | |
| *** Goneri has quit IRC | 22:26 | |
| fungi | clarkb: looks like the syntax job on puppet-openstackci has bitrotted on us | 22:26 |
| mtreinish | clarkb, jeblair: yeah, it's been in my backlog to fix the openstack-health puppet module, I've just had other priorities and haven't gotten around to it | 22:27 |
| *** esikachev has joined #openstack-infra | 22:27 | |
| fungi | pabelanger: just strange that on e.g. fedora and centos we're installing both mariadb and mariadb-server... what does the former provide that the latter doesn't? | 22:27 |
| *** camunoz has joined #openstack-infra | 22:28 | |
| *** esberglu has quit IRC | 22:28 | |
| jeblair | mtreinish: okay; just wanted to check i wasn't crazy or we were going in circles :) | 22:28 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Drop support for Ubuntu precise https://review.openstack.org/468186 | 22:28 |
| pabelanger | fungi: I think mariadb is clients and mariadb-server is server for redhat | 22:28 |
| jeblair | to be fair, i guess both of those questions are still up for debate | 22:28 |
| *** baoli has quit IRC | 22:29 | |
| *** gouthamr has quit IRC | 22:29 | |
| clarkb | fungi: grr ok, I'm trying to finish up a thought on using xz for the journal files. Makes them almost half the size they currently are | 22:30 |
| fungi | that's clarkb i can try to iron out the bitrot, no worries | 22:32 |
| fungi | no idea where the stray "that's" came from | 22:32 |
| fungi | i must have had an unfinished thought i forgot about | 22:32 |
| pabelanger | here is an interesting review: https://review.openstack.org/#/c/466321/7 now that gnocchi is on github.com, that patch updates JJB jobs to directly depend on github. I am curious how we handle network outages in that case, just lets the jobs deal with it themself? | 22:33 |
| fungi | pabelanger: dirk: okay, so given there's no separate mariadb client package being installed on suse, the mariadb package must provide both client and server there? | 22:33 |
| *** esikachev has quit IRC | 22:33 | |
| pabelanger | fungi: I am not sure about opensuse, lets see what dirk says | 22:33 |
| pabelanger | mariadb-tools maybe? | 22:34 |
| pabelanger | https://en.opensuse.org/SDB:LAMP_setup#Installing_MariaDB | 22:34 |
| pabelanger | The package mariadb-tools is necessary for administration | 22:34 |
| pabelanger | guess that is clients | 22:34 |
| *** Apoorva_ has quit IRC | 22:35 | |
| *** Apoorva has joined #openstack-infra | 22:35 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Use xz to compress the journal files https://review.openstack.org/468187 | 22:35 |
| *** jamielennox|away is now known as jamielennox | 22:36 | |
| fungi | pabelanger: any idea if their mariadb package declares a depedency on mariadb-tools? i'm going to preemptively -1 this until we have confirmation | 22:37 |
| fungi | dirk: ^ | 22:37 |
| jeblair | pabelanger: it should not be fetched from github, they should use releases of gnocchi from pypi | 22:37 |
| clarkb | pabelanger: that won't work becaused-g assumes it can cloen from the git farm | 22:38 |
| clarkb | fungi: whats the context? I can search locally really quick | 22:38 |
| fungi | clarkb: context is https://review.openstack.org/468157 | 22:39 |
| clarkb | fungi: zypper info mariadb says "This package only contains the server-side programs." | 22:40 |
| clarkb | so I don't think its got client pacakges | 22:40 |
| clarkb | mariadb-client is the client package | 22:40 |
| fungi | clarkb: thanks for checking. -1 stands then | 22:40 |
| clarkb | and tools says A set of scripts for administering a MariaDB or developing applications with MariaDB. so may need that too | 22:41 |
| jeblair | fungi: fyi the gnocchi change pabelanger was asking about: https://review.openstack.org/466321 | 22:41 |
| openstackgerrit | David Ames proposed openstack-infra/project-config master: Add charm-glusterfs and charm-manila-glusterfs https://review.openstack.org/468112 | 22:41 |
| fungi | jeblair: oh, awesome | 22:42 |
| clarkb | https://review.openstack.org/468150 gets us down to ~165MB per neutron full job | 22:42 |
| clarkb | from ~300 or so | 22:42 |
| clarkb | and then I expect the xz change to shave a bit more off too | 22:42 |
| *** gomarivera has quit IRC | 22:42 | |
| clarkb | so maybe we start with that and see if that gets log sizes under control? | 22:43 |
| clarkb | along with stackviz and ara changes we should be in decent shape I think | 22:43 |
| jeblair | do we have a target? | 22:43 |
| jeblair | like a budget? | 22:43 |
| clarkb | fitting 6 weeks of logs in 12TB ? | 22:44 |
| clarkb | I don't know that we have anything better than that right now. Just trying to prune where it makes sense to start | 22:44 |
| jeblair | clarkb: at some point we're going to need to give zuulv3 a max per-build value | 22:46 |
| pabelanger | jeblair: that make sense | 22:46 |
| fungi | clarkb: puppet syntax is choking on the \% in the elasticsearch crontab. right now we seem to be embedding literal \% into the crontab (looking at what's on the server). is that intentional? | 22:46 |
| clarkb | based on just digging through the logs today I would say that ~100MB is probably a good upper bound with the vast majority of "simple" jobs needing only a few megabytes | 22:47 |
| dirk | fungi: clarkb : mariadb-client is a requirement of mariadb. See comment in the review | 22:47 |
| clarkb | fungi: yes crontabs treat % special, now I have to remember why | 22:47 |
| fungi | thanks dirk! | 22:47 |
| jeblair | clarkb: with 468150 are we still doing two copies of the devstack logs: both binary and text log data? | 22:47 |
| pabelanger | fungi: ya. lets -1 until dirk comments | 22:47 |
| *** jascott1 has joined #openstack-infra | 22:47 | |
| fungi | clarkb: just trying to figure out if we need to have \\% in the puppet manifest now or what | 22:48 |
| clarkb | % is treated as a newline unless escaped so ya we need to escape it | 22:48 |
| fungi | pabelanger: looks like he just did | 22:48 |
| clarkb | fungi: possibly \\ is what we need | 22:48 |
| *** claudiub has quit IRC | 22:49 | |
| clarkb | jeblair: yes | 22:49 |
| clarkb | jeblair: but we are making two copies now with the binary data being much larger (everything from system) | 22:49 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/468173 | 22:49 |
| clarkb | dirk: oh sorry confirmed | 22:51 |
| * clarkb adds --requires to zypper info | 22:51 | |
| kong | morning/evening, infra guys, how can i apply an new irc channel? | 22:51 |
| kong | for my new project | 22:51 |
| fungi | clarkb: it's just strange that puppet only started caring in the last month about the \% there, and then only in the xenial syntax job... since we're using trusty for those servers i wonder if satisfying the xenial-only syntax freakout is going to break on trusty | 22:51 |
| jeblair | kong: https://docs.openstack.org/infra/system-config/irc.html#channel-requirements | 22:51 |
| *** gardlt has quit IRC | 22:52 | |
| kong | jeblair: thanks! | 22:52 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 22:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 22:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 22:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 22:53 |
| *** thorst has joined #openstack-infra | 22:53 | |
| clarkb | fungi: you know what, that cron is ensured absent, lets just delete the code entirely | 22:53 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackci master: Escape literal backslash in elasticsearch manifest https://review.openstack.org/468192 | 22:53 |
| clarkb | fungi: we use a more robust tool/system to clean out the old indexes now (and have for a while) | 22:54 |
| fungi | clarkb: d'oh! | 22:54 |
| fungi | will do (though it's _not_ absent on the server?!?) | 22:54 |
| clarkb | it should be commented out on the server I think | 22:54 |
| *** gtmanfred has quit IRC | 22:54 | |
| clarkb | since thats how puppet implements it so it knows how to enable it again or something | 22:54 |
| *** gtmanfred has joined #openstack-infra | 22:54 | |
| openstackgerrit | Merged openstack/os-client-config master: Add helper method to fetch service catalog https://review.openstack.org/464238 | 22:55 |
| openstackgerrit | Merged openstack/os-client-config master: Add ironicclient to constructors list https://review.openstack.org/464239 | 22:55 |
| openstackgerrit | Merged openstack/os-client-config master: Keep a singleton to support multiple get_config calls https://review.openstack.org/465195 | 22:55 |
| openstackgerrit | Merged openstack/os-client-config master: Don't pop from os.environ https://review.openstack.org/468020 | 22:55 |
| fungi | clarkb: doesn't look commented out at all on elasticsearch02.o.o at least | 22:56 |
| *** gordc has quit IRC | 22:57 | |
| clarkb | fungi: thats a different cron | 22:57 |
| clarkb | thats what it got replaced with | 22:57 |
| fungi | clarkb: hah, yes different name | 22:57 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackci master: Clean up elasticsearch cleanup cronjob https://review.openstack.org/468192 | 23:00 |
| fungi | clarkb: ^ | 23:00 |
| clarkb | fungi: +2, not sure if you want to self approve or maybe pabelanger will review? | 23:00 |
| pabelanger | looking | 23:01 |
| fungi | assuming that lands we can recheck your apache addition for the stackviz compression | 23:01 |
| pabelanger | clarkb: mind +3 on https://review.openstack.org/#/c/468154/ | 23:02 |
| pabelanger | or mordred ^ | 23:02 |
| pabelanger | removes ant / asciidoc from bindep-fallback | 23:02 |
| clarkb | pabelanger: done | 23:03 |
| pabelanger | danke | 23:03 |
| pabelanger | recheck for nodepool dsvm on going | 23:04 |
| *** gouthamr has joined #openstack-infra | 23:05 | |
| *** rbrndt has quit IRC | 23:06 | |
| *** cody-somerville has quit IRC | 23:06 | |
| fungi | pabelanger: heed my note on the ant/asciidoc one, trying to maintain bindep.txt files for that on our gerrit fork's branches is likely to be more of a pain than just writing them into the jjb config for those jobs | 23:06 |
| pabelanger | fungi: no, that is a good idea. I'll update JJB in the morning | 23:07 |
| *** thorst has quit IRC | 23:08 | |
| fungi | otherwise we have to forward port a patch adding bindep.txt every time we open a new fork branch from upstream | 23:09 |
| *** ashishb_ has quit IRC | 23:10 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove asciidoc / ant from bindep-fallback.txt https://review.openstack.org/468154 | 23:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Further bindep-fallback fixes for openSUSE https://review.openstack.org/468157 | 23:10 |
| openstackgerrit | Merged openstack/os-client-config master: Revert "Revert "Use interface not endpoint_type for keystoneclient"" https://review.openstack.org/464299 | 23:10 |
| fungi | and i think those jobs are only used for building our gerrit fork anyway | 23:10 |
| clarkb | ya | 23:10 |
| *** markvoelker has quit IRC | 23:11 | |
| fungi | so it's not like it's adding any inefficiency by just encoding the package additions directly in the job (the job could even write out a bindep.txt after the checkout and before the install-distro-packages builder i guess, if we want) | 23:11 |
| fungi | that might make for a simple compromise | 23:12 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Clean up elasticsearch cleanup cronjob https://review.openstack.org/468192 | 23:13 |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config master: Config gerrit bot to Qinling https://review.openstack.org/468202 | 23:15 |
| *** rhallisey has quit IRC | 23:15 | |
| pabelanger | blarg | 23:15 |
| pabelanger | I think devstack no longer does zookeeper things | 23:15 |
| pabelanger | which nodepool dsvm depended on | 23:16 |
| pabelanger | depends* | 23:16 |
| fungi | pabelanger: right, etc3d obsoleted it | 23:16 |
| fungi | we'll need to handle that in the plugin now | 23:16 |
| pabelanger | ya, I'll do that tomorrow | 23:16 |
| pabelanger | until then, nodepool-dsvm is not working | 23:16 |
| pabelanger | but nodepool did start until python35 | 23:17 |
| pabelanger | so yay | 23:17 |
| fungi | thanks for the heads up! | 23:17 |
| clarkb | we shouldn't need much, just add zk to the debs/rpms? | 23:18 |
| fungi | #status log The nodepool-dsvm jobs are failing for now, until we reimplement zookeeper handling in our devstack plugin | 23:18 |
| clarkb | oh have to start it too because it doesn't start by default | 23:18 |
| openstackstatus | fungi: finished logging | 23:18 |
| *** pvaneck_ has quit IRC | 23:19 | |
| *** dimtruck is now known as zz_dimtruck | 23:19 | |
| fungi | i feel a bit out of the loop on the dlm discussion these days... isn't etcd a tool for centralizing configuration? distributed lock management sounds to me like a much different scope of functionality | 23:20 |
| clarkb | fungi: its both | 23:21 |
| fungi | did people start (ab)using etcd as a dlm because it was convenient, and then it later grew the necessary robustness for that use case? | 23:21 |
| clarkb | yes basically | 23:21 |
| fungi | and just kept the original name i guess | 23:22 |
| clarkb | we already install zk in our plugin | 23:22 |
| fungi | so we just need to start it? i guess that's what devstack was doing for us up to now? | 23:22 |
| clarkb | ya I am thinking thats about it | 23:23 |
| clarkb | about to push a patch | 23:23 |
| *** ihrachys has quit IRC | 23:24 | |
| fungi | we've crept back up on logs disk utilization today. even with a repeat of the journal one-week expiry underway we're back to a scant 147gb free | 23:26 |
| *** rhallisey has joined #openstack-infra | 23:28 | |
| clarkb | fungi: https://review.openstack.org/#/c/468187/1 and its parent should help | 23:28 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 23:29 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/468173 | 23:29 |
| clarkb | uh | 23:30 |
| fungi | clarkb: looks like we got check results back on the latest journal filtering change... prognosis? | 23:30 |
| *** gildub has joined #openstack-infra | 23:30 | |
| clarkb | why is zookeeper not in the deps list on nodepool feature/zuulv3? | 23:30 |
| clarkb | thats the problem... I think master will work fine... | 23:30 |
| clarkb | fungi: it reduces job size to about ~165MB from ~270-300MB I think | 23:30 |
| fungi | clarkb: oh, we just need to merge from master again? | 23:31 |
| clarkb | no | 23:31 |
| clarkb | we need to revert mordred's change to feature/zuulv3 to remove zookeeper | 23:31 |
| openstackgerrit | Merged openstack-infra/infra-specs master: Block sphinx 1.6.1 https://review.openstack.org/467291 | 23:31 |
| clarkb | I will push that patch | 23:31 |
| *** jamesdenton has joined #openstack-infra | 23:32 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Revert "Use devstack's zookeeper support" https://review.openstack.org/468205 | 23:32 |
| clarkb | there | 23:32 |
| fungi | heh | 23:32 |
| fungi | that was a short-lived improvement | 23:33 |
| clarkb | and the xz change should save another 20 or 30MB I think | 23:34 |
| clarkb | of course there is still the question of whether or not its worthwhile to have those files at all in that state | 23:34 |
| clarkb | I was hoping that sdague would weigh in | 23:34 |
| fungi | was just looking at the neutron functional job failure to make sure it wasn't related to the journal filtering change. that job looks insanely fragile | 23:34 |
| clarkb | no comment :) | 23:35 |
| openstackgerrit | Tin Lam proposed openstack-infra/project-config master: Add new repo openstack-helm-addons under OSH https://review.openstack.org/468206 | 23:35 |
| mordred | clarkb: oh. bleh | 23:35 |
| *** jamesdenton has quit IRC | 23:36 | |
| *** Apoorva_ has joined #openstack-infra | 23:36 | |
| *** rhallisey has quit IRC | 23:36 | |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config master: Config gerrit bot to Qinling https://review.openstack.org/468202 | 23:37 |
| *** Apoorva has quit IRC | 23:40 | |
| *** Apoorva_ has quit IRC | 23:41 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Compress stackviz data https://review.openstack.org/468207 | 23:42 |
| clarkb | fungi: timothyb89 mtreinish ^ and that should test it all works once the openstackci change lands | 23:42 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Set json.gz mimetype properly https://review.openstack.org/468180 | 23:42 |
| fungi | like that ;) | 23:43 |
| clarkb | fungi: the xz change gets us down to ~113MB from ~165MB | 23:43 |
| fungi | nice! | 23:43 |
| clarkb | and then the stackviz change should get us down real near 100MB | 23:44 |
| *** markvoelker has joined #openstack-infra | 23:44 | |
| *** markvoelker has quit IRC | 23:45 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Make zuul ansible dir self-contained https://review.openstack.org/468208 | 23:45 |
| *** stakeda has joined #openstack-infra | 23:48 | |
| *** Swami has quit IRC | 23:49 | |
| openstackgerrit | Merged openstack-infra/infra-specs master: Add nodepool drivers spec https://review.openstack.org/461509 | 23:54 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement github pipepline req of open https://review.openstack.org/468213 | 23:56 |
| clarkb | infra-root review on https://review.openstack.org/#/q/status:open+topic:less-logs much appreciated as it should reduce logging overhead quite a bit in any jobs using d-g | 23:56 |
| *** dingyichen has joined #openstack-infra | 23:56 | |
| *** SumitNaiksatam has quit IRC | 23:57 | |
| *** zz_dimtruck is now known as dimtruck | 23:58 | |
| fungi | waiting to see the results of the stackviz compression works before voting on it | 23:58 |
| *** hongbin has quit IRC | 23:58 | |
| fungi | the other changes there all lgtm | 23:58 |
| clarkb | cool thanks | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!