| *** akshai has quit IRC | 00:02 | |
| *** achanda has quit IRC | 00:03 | |
| *** darvon has quit IRC | 00:04 | |
| *** cfarquhar has joined #openstack-infra | 00:07 | |
| *** cfarquhar has quit IRC | 00:07 | |
| *** cfarquhar has joined #openstack-infra | 00:07 | |
| *** darvon has joined #openstack-infra | 00:12 | |
| *** asettle has quit IRC | 00:27 | |
| *** IlyaG has joined #openstack-infra | 00:29 | |
| *** baoli has joined #openstack-infra | 00:31 | |
| *** IlyaG has quit IRC | 00:33 | |
| *** baoli has quit IRC | 00:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:37 | |
| *** hichihara has joined #openstack-infra | 00:39 | |
| *** kzaitsev_mb has quit IRC | 00:42 | |
| *** zz_dimtruck is now known as dimtruck | 00:43 | |
| *** jerryz has quit IRC | 00:46 | |
| *** thorst has joined #openstack-infra | 00:53 | |
| *** amitgandhinz has joined #openstack-infra | 00:55 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration to move attrs and tags into columns https://review.openstack.org/298069 | 00:59 |
|---|---|---|
| *** thorst has quit IRC | 01:01 | |
| *** achanda has joined #openstack-infra | 01:01 | |
| *** amitgandhinz has quit IRC | 01:09 | |
| *** piet has joined #openstack-infra | 01:09 | |
| *** zhurong has joined #openstack-infra | 01:10 | |
| openstackgerrit | watanabe isao proposed openstack-infra/devstack-gate: Let BUILD_TIMEOUT in Ironic local.conf configurable https://review.openstack.org/297531 | 01:11 |
| *** yamamoto has joined #openstack-infra | 01:16 | |
| *** mriedem has quit IRC | 01:19 | |
| *** zhurong has quit IRC | 01:23 | |
| *** baoli has joined #openstack-infra | 01:24 | |
| *** zhurong has joined #openstack-infra | 01:24 | |
| *** sabari has quit IRC | 01:24 | |
| *** sc68cal has quit IRC | 01:27 | |
| *** yamamoto has quit IRC | 01:28 | |
| *** hichihara has quit IRC | 01:29 | |
| *** hichihara has joined #openstack-infra | 01:29 | |
| *** Sukhdev has joined #openstack-infra | 01:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:38 | |
| *** sabari- has joined #openstack-infra | 01:38 | |
| *** achanda has quit IRC | 01:39 | |
| *** piet has quit IRC | 01:40 | |
| *** piet has joined #openstack-infra | 01:41 | |
| *** mylu has quit IRC | 01:42 | |
| *** mylu has joined #openstack-infra | 01:43 | |
| *** kzaitsev_mb has quit IRC | 01:43 | |
| *** IlyaG has joined #openstack-infra | 01:45 | |
| *** tphummel has joined #openstack-infra | 01:46 | |
| *** IlyaG has quit IRC | 01:49 | |
| *** chuck_ has joined #openstack-infra | 01:50 | |
| *** chuck_ has quit IRC | 01:50 | |
| *** cfarquhar has quit IRC | 01:52 | |
| *** Qiming has joined #openstack-infra | 01:52 | |
| *** ZZelle has quit IRC | 01:53 | |
| *** piet has quit IRC | 01:54 | |
| *** zhurong has quit IRC | 01:55 | |
| *** tphummel has quit IRC | 01:56 | |
| *** thorst has joined #openstack-infra | 01:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:03 | |
| *** Daisy has joined #openstack-infra | 02:03 | |
| *** amitgandhinz has joined #openstack-infra | 02:05 | |
| *** thorst has quit IRC | 02:06 | |
| *** baoli has quit IRC | 02:06 | |
| *** moravec has joined #openstack-infra | 02:10 | |
| openstackgerrit | Hieu LE proposed openstack-infra/groups-static-pages: Update Vietnam User Group info https://review.openstack.org/298076 | 02:11 |
| *** sdake has joined #openstack-infra | 02:11 | |
| *** baoli has joined #openstack-infra | 02:13 | |
| *** jamielennox is now known as jamielennox|away | 02:13 | |
| openstackgerrit | Hieu LE proposed openstack-infra/groups-static-pages: Update Vietnam User Group info https://review.openstack.org/298076 | 02:14 |
| *** sdake has quit IRC | 02:18 | |
| *** amitgandhinz has quit IRC | 02:19 | |
| *** kzaitsev_mb has quit IRC | 02:21 | |
| *** achanda has joined #openstack-infra | 02:26 | |
| *** yamamoto has joined #openstack-infra | 02:29 | |
| *** baoli has quit IRC | 02:32 | |
| *** gmann has quit IRC | 02:39 | |
| *** baoli has joined #openstack-infra | 02:41 | |
| *** Sukhdev has quit IRC | 02:42 | |
| *** mhayden has quit IRC | 02:43 | |
| *** mhayden has joined #openstack-infra | 02:43 | |
| *** gmann has joined #openstack-infra | 02:44 | |
| *** baoli has quit IRC | 02:44 | |
| *** Sukhdev has joined #openstack-infra | 02:50 | |
| *** Sukhdev has quit IRC | 02:51 | |
| *** mhayden has quit IRC | 02:51 | |
| *** baoli has joined #openstack-infra | 02:52 | |
| *** baoli has quit IRC | 02:54 | |
| *** cfarquhar has joined #openstack-infra | 02:55 | |
| *** cfarquhar has quit IRC | 02:55 | |
| *** cfarquhar has joined #openstack-infra | 02:55 | |
| *** baoli has joined #openstack-infra | 02:59 | |
| *** dimtruck is now known as zz_dimtruck | 03:00 | |
| *** IlyaG has joined #openstack-infra | 03:00 | |
| *** baoli has quit IRC | 03:02 | |
| *** thorst has joined #openstack-infra | 03:03 | |
| *** IlyaG has quit IRC | 03:05 | |
| *** thorst_ has joined #openstack-infra | 03:06 | |
| *** thorst has quit IRC | 03:09 | |
| *** thorst_ has quit IRC | 03:11 | |
| *** baoli has joined #openstack-infra | 03:11 | |
| *** Daisy has quit IRC | 03:13 | |
| *** Daisy has joined #openstack-infra | 03:14 | |
| *** amitgandhinz has joined #openstack-infra | 03:15 | |
| *** mhayden has joined #openstack-infra | 03:16 | |
| *** Daisy has quit IRC | 03:18 | |
| *** zz_dimtruck is now known as dimtruck | 03:25 | |
| *** IlyaG has joined #openstack-infra | 03:25 | |
| *** dimtruck is now known as zz_dimtruck | 03:28 | |
| *** zz_dimtruck is now known as dimtruck | 03:28 | |
| *** amitgandhinz has quit IRC | 03:28 | |
| *** links has joined #openstack-infra | 03:33 | |
| *** dimtruck is now known as zz_dimtruck | 03:38 | |
| *** achanda has quit IRC | 03:44 | |
| *** ramishra has quit IRC | 03:47 | |
| *** IlyaG has quit IRC | 03:48 | |
| *** piet has joined #openstack-infra | 03:51 | |
| *** ramishra has joined #openstack-infra | 03:51 | |
| *** yuanying has quit IRC | 03:53 | |
| *** baoli has quit IRC | 03:58 | |
| *** zz_dimtruck is now known as dimtruck | 03:59 | |
| *** Daisy has joined #openstack-infra | 04:00 | |
| *** asettle has joined #openstack-infra | 04:02 | |
| *** Jeffrey4l has joined #openstack-infra | 04:02 | |
| *** Daisy has quit IRC | 04:04 | |
| *** Nikolay_St has joined #openstack-infra | 04:04 | |
| *** salv-orlando has joined #openstack-infra | 04:06 | |
| *** salv-orlando has quit IRC | 04:06 | |
| *** piet has quit IRC | 04:07 | |
| *** salv-orlando has joined #openstack-infra | 04:07 | |
| *** salv-orl_ has quit IRC | 04:09 | |
| *** thorst has joined #openstack-infra | 04:10 | |
| *** salv-orlando has quit IRC | 04:12 | |
| *** thorst has quit IRC | 04:16 | |
| openstackgerrit | watanabe isao proposed openstack-infra/devstack-gate: Add a fork for Ironic vender driver https://review.openstack.org/298097 | 04:17 |
| *** piet has joined #openstack-infra | 04:21 | |
| *** amitgandhinz has joined #openstack-infra | 04:25 | |
| *** vikram has joined #openstack-infra | 04:31 | |
| vikram | Hi | 04:31 |
| vikram | I want to add 'onos' driver in marketplace | 04:31 |
| vikram | https://www.openstack.org/marketplace/drivers/ | 04:31 |
| vikram | What I am suppose to do for this? | 04:31 |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Enable Murano service as part of congress job https://review.openstack.org/296380 | 04:33 |
| vikram | fungi: ping | 04:34 |
| vikram | fungi: We want to add 'ONOS' driver support in "https://www.openstack.org/marketplace/drivers/".. What I am suppose to do for that? | 04:35 |
| *** amitgandhinz has quit IRC | 04:38 | |
| *** Sukhdev has joined #openstack-infra | 04:43 | |
| *** Daisy has joined #openstack-infra | 04:43 | |
| *** Daisy has quit IRC | 04:44 | |
| *** Daisy has joined #openstack-infra | 04:45 | |
| *** Daisy has quit IRC | 04:46 | |
| *** Daisy has joined #openstack-infra | 04:47 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr: Add pypi download + version badges into README.rst https://review.openstack.org/298102 | 04:52 |
| *** yuanying has joined #openstack-infra | 04:56 | |
| *** vikram has quit IRC | 05:02 | |
| *** vikram_ has joined #openstack-infra | 05:04 | |
| *** Goneri has quit IRC | 05:06 | |
| *** Nikolay_St has quit IRC | 05:10 | |
| *** Daisy has quit IRC | 05:12 | |
| *** thorst has joined #openstack-infra | 05:12 | |
| *** Daisy has joined #openstack-infra | 05:13 | |
| *** sputnik13 has joined #openstack-infra | 05:17 | |
| *** sputnik13 has quit IRC | 05:17 | |
| *** piet has quit IRC | 05:20 | |
| *** thorst has quit IRC | 05:21 | |
| *** asettle has quit IRC | 05:23 | |
| openstackgerrit | vikram.choudhary proposed openstack-infra/project-config: Adding neutron-bgp as a part of neutron project. https://review.openstack.org/268726 | 05:23 |
| *** dimtruck is now known as zz_dimtruck | 05:26 | |
| *** sridhar_ram has joined #openstack-infra | 05:29 | |
| *** achanda has joined #openstack-infra | 05:31 | |
| *** amitgandhinz has joined #openstack-infra | 05:35 | |
| *** asettle has joined #openstack-infra | 05:35 | |
| *** yuanying has quit IRC | 05:40 | |
| *** asettle has quit IRC | 05:40 | |
| *** fedexo has joined #openstack-infra | 05:41 | |
| *** roxanaghe has joined #openstack-infra | 05:45 | |
| *** roxanaghe has quit IRC | 05:47 | |
| *** abregman has joined #openstack-infra | 05:47 | |
| *** yuanying has joined #openstack-infra | 05:48 | |
| *** HeOS has quit IRC | 05:48 | |
| *** amitgandhinz has quit IRC | 05:49 | |
| openstackgerrit | fumihiko kakuma proposed openstack/requirements: Delete networking-ofagent from projects.txt https://review.openstack.org/298107 | 05:51 |
| *** vikram_ has quit IRC | 05:55 | |
| *** HeOS has joined #openstack-infra | 05:56 | |
| *** vikram_ has joined #openstack-infra | 05:59 | |
| *** [HeOS] has joined #openstack-infra | 06:00 | |
| *** HeOS has quit IRC | 06:02 | |
| *** baoli has joined #openstack-infra | 06:10 | |
| *** harlowja_at_home has quit IRC | 06:12 | |
| *** sridhar_ram has quit IRC | 06:12 | |
| *** Nikolay_St has joined #openstack-infra | 06:13 | |
| *** baoli has quit IRC | 06:15 | |
| *** zz_dimtruck is now known as dimtruck | 06:18 | |
| *** thorst has joined #openstack-infra | 06:19 | |
| *** [HeOS] has quit IRC | 06:19 | |
| *** salv-orlando has joined #openstack-infra | 06:21 | |
| *** harlowja_at_home has joined #openstack-infra | 06:23 | |
| *** Daisy_ has joined #openstack-infra | 06:25 | |
| *** thorst has quit IRC | 06:26 | |
| *** mylu has quit IRC | 06:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/298114 | 06:28 |
| *** Daisy has quit IRC | 06:28 | |
| *** Daisy_ has quit IRC | 06:28 | |
| *** HeOS has joined #openstack-infra | 06:28 | |
| *** Daisy has joined #openstack-infra | 06:28 | |
| *** maishsk has joined #openstack-infra | 06:30 | |
| *** salv-orlando has quit IRC | 06:34 | |
| *** [HeOS] has joined #openstack-infra | 06:39 | |
| *** HeOS has quit IRC | 06:41 | |
| *** fedexo has quit IRC | 06:41 | |
| *** amitgandhinz has joined #openstack-infra | 06:45 | |
| *** oanson has joined #openstack-infra | 06:47 | |
| *** roxanaghe has joined #openstack-infra | 06:47 | |
| *** [HeOS] has quit IRC | 06:48 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:48 |
| *** pcaruana has joined #openstack-infra | 06:49 | |
| *** nadya has joined #openstack-infra | 06:51 | |
| *** roxanaghe has quit IRC | 06:52 | |
| *** HeOS has joined #openstack-infra | 06:56 | |
| *** nadya has quit IRC | 06:57 | |
| *** amitgandhinz has quit IRC | 06:58 | |
| *** Liuqing has joined #openstack-infra | 06:59 | |
| *** Daisy_ has joined #openstack-infra | 07:02 | |
| *** Daisy has quit IRC | 07:02 | |
| *** HeOS has quit IRC | 07:06 | |
| *** Daisy has joined #openstack-infra | 07:07 | |
| *** Daisy_ has quit IRC | 07:07 | |
| *** HeOS has joined #openstack-infra | 07:08 | |
| *** vikram_ has quit IRC | 07:09 | |
| *** vikram_ has joined #openstack-infra | 07:09 | |
| *** e0ne has joined #openstack-infra | 07:15 | |
| *** e0ne has quit IRC | 07:18 | |
| *** asettle has joined #openstack-infra | 07:24 | |
| *** thorst has joined #openstack-infra | 07:24 | |
| *** asettle has quit IRC | 07:29 | |
| *** thorst has quit IRC | 07:31 | |
| *** nmagnezi has joined #openstack-infra | 07:34 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/298114 | 07:46 |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/project-config: 'fuel-abandon' group added to Fuel-related projects https://review.openstack.org/297252 | 07:47 |
| *** binbincong has quit IRC | 07:47 | |
| *** amitgandhinz has joined #openstack-infra | 07:55 | |
| *** |-paul-| has quit IRC | 07:56 | |
| *** tqtran has joined #openstack-infra | 07:58 | |
| *** sabari- is now known as sabari | 08:01 | |
| *** salv-orlando has joined #openstack-infra | 08:01 | |
| *** binbincong has joined #openstack-infra | 08:04 | |
| *** salv-orlando has quit IRC | 08:07 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: midonet: Make tempest jobs voting https://review.openstack.org/298142 | 08:07 |
| *** amitgandhinz has quit IRC | 08:08 | |
| *** nadya has joined #openstack-infra | 08:13 | |
| *** e0ne has joined #openstack-infra | 08:14 | |
| *** nihilifer has quit IRC | 08:16 | |
| *** nihilifer has joined #openstack-infra | 08:17 | |
| *** hichihara has quit IRC | 08:19 | |
| *** thorst has joined #openstack-infra | 08:30 | |
| *** yuanying has quit IRC | 08:31 | |
| *** yuanying has joined #openstack-infra | 08:31 | |
| *** GheRiver1 has joined #openstack-infra | 08:32 | |
| *** yuanying has quit IRC | 08:32 | |
| *** salv-orlando has joined #openstack-infra | 08:33 | |
| *** roxanaghe has joined #openstack-infra | 08:36 | |
| *** thorst has quit IRC | 08:36 | |
| *** ociuhandu has quit IRC | 08:36 | |
| *** dizquierdo has joined #openstack-infra | 08:36 | |
| *** salv-orlando has quit IRC | 08:37 | |
| *** yuanying has joined #openstack-infra | 08:39 | |
| *** roxanaghe has quit IRC | 08:40 | |
| *** armax has joined #openstack-infra | 08:44 | |
| *** armax has quit IRC | 08:49 | |
| *** mikelk has joined #openstack-infra | 08:53 | |
| *** mikelk has quit IRC | 08:55 | |
| *** jcooley_ has quit IRC | 08:56 | |
| *** GheRivero has quit IRC | 08:57 | |
| *** Sukhdev has quit IRC | 08:59 | |
| *** GheRivero has joined #openstack-infra | 09:01 | |
| *** GheRiver1 has quit IRC | 09:03 | |
| *** achanda has quit IRC | 09:04 | |
| *** amitgandhinz has joined #openstack-infra | 09:05 | |
| *** thorst has joined #openstack-infra | 09:08 | |
| *** lezbar has joined #openstack-infra | 09:09 | |
| *** ociuhandu has joined #openstack-infra | 09:10 | |
| *** achanda has joined #openstack-infra | 09:11 | |
| *** thorst has quit IRC | 09:15 | |
| *** yuanying has quit IRC | 09:16 | |
| *** Qiming has quit IRC | 09:17 | |
| *** Qiming has joined #openstack-infra | 09:17 | |
| *** amitgandhinz has quit IRC | 09:18 | |
| *** skolekonov|away is now known as skolekonov | 09:19 | |
| *** achanda has quit IRC | 09:28 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:31 | |
| *** thorst has joined #openstack-infra | 09:32 | |
| *** salv-orlando has joined #openstack-infra | 09:36 | |
| *** thorst has quit IRC | 09:37 | |
| *** yamahata has quit IRC | 09:37 | |
| *** pcaruana has quit IRC | 09:42 | |
| *** GheRiver1 has joined #openstack-infra | 09:43 | |
| *** zeih has joined #openstack-infra | 09:43 | |
| *** Daisy has quit IRC | 09:44 | |
| *** Daisy has joined #openstack-infra | 09:44 | |
| *** salv-orlando has quit IRC | 09:46 | |
| *** Daisy has quit IRC | 09:46 | |
| *** Daisy has joined #openstack-infra | 09:47 | |
| *** lezbar has quit IRC | 09:48 | |
| *** Daisy has quit IRC | 09:48 | |
| *** vgridnev has joined #openstack-infra | 09:53 | |
| *** pcaruana has joined #openstack-infra | 09:55 | |
| *** dizquierdo has quit IRC | 10:01 | |
| *** maishsk has quit IRC | 10:08 | |
| *** maishsk has joined #openstack-infra | 10:10 | |
| *** amitgandhinz has joined #openstack-infra | 10:15 | |
| openstackgerrit | Merged openstack-infra/project-config: 'fuel-abandon' group added to Fuel-related projects https://review.openstack.org/297252 | 10:16 |
| *** aloga_ is now known as aloga | 10:18 | |
| *** akaszuba has joined #openstack-infra | 10:18 | |
| *** Qiming has quit IRC | 10:24 | |
| *** roxanaghe has joined #openstack-infra | 10:24 | |
| *** akaszuba has quit IRC | 10:24 | |
| *** Liuqing has quit IRC | 10:25 | |
| *** sdague has joined #openstack-infra | 10:27 | |
| *** roxanaghe has quit IRC | 10:28 | |
| *** amitgandhinz has quit IRC | 10:28 | |
| *** achanda has joined #openstack-infra | 10:28 | |
| *** tqtran has quit IRC | 10:30 | |
| *** achanda has quit IRC | 10:33 | |
| *** thorst has joined #openstack-infra | 10:36 | |
| *** achanda has joined #openstack-infra | 10:36 | |
| *** vikram_ has quit IRC | 10:39 | |
| *** thorst has quit IRC | 10:42 | |
| *** GheRiver1 has quit IRC | 10:43 | |
| *** |-paul-| has joined #openstack-infra | 10:46 | |
| *** Liuqing has joined #openstack-infra | 10:51 | |
| *** asettle has joined #openstack-infra | 10:56 | |
| *** nadya has quit IRC | 10:56 | |
| *** maishsk has quit IRC | 10:56 | |
| *** yamamoto has quit IRC | 10:59 | |
| *** dizquierdo has joined #openstack-infra | 11:04 | |
| *** pcaruana has quit IRC | 11:09 | |
| *** zeih has quit IRC | 11:10 | |
| *** salv-orlando has joined #openstack-infra | 11:11 | |
| *** achanda has quit IRC | 11:11 | |
| *** salv-orlando has quit IRC | 11:13 | |
| *** Liuqing_ has joined #openstack-infra | 11:15 | |
| *** Liuqing has quit IRC | 11:16 | |
| *** maishsk has joined #openstack-infra | 11:17 | |
| *** tpsilva has joined #openstack-infra | 11:18 | |
| *** ldnunes has joined #openstack-infra | 11:18 | |
| *** pcaruana has joined #openstack-infra | 11:23 | |
| *** nadya has joined #openstack-infra | 11:24 | |
| *** amitgandhinz has joined #openstack-infra | 11:25 | |
| *** thorst has joined #openstack-infra | 11:26 | |
| *** tqtran has joined #openstack-infra | 11:26 | |
| *** doug-fish has joined #openstack-infra | 11:27 | |
| *** jpr has joined #openstack-infra | 11:29 | |
| *** nadya has quit IRC | 11:29 | |
| *** Qiming has joined #openstack-infra | 11:30 | |
| *** tqtran has quit IRC | 11:31 | |
| *** GheRiver1 has joined #openstack-infra | 11:33 | |
| *** gordc has joined #openstack-infra | 11:33 | |
| *** yfried has joined #openstack-infra | 11:35 | |
| *** vgridnev has quit IRC | 11:38 | |
| *** amitgandhinz has quit IRC | 11:38 | |
| *** maishsk has quit IRC | 11:41 | |
| *** nadya has joined #openstack-infra | 11:41 | |
| *** |-paul-| has quit IRC | 11:43 | |
| *** flaper87 has quit IRC | 11:48 | |
| *** wfoster has quit IRC | 11:49 | |
| HeOS | Hello, team! I have merged my request: https://review.openstack.org/#/c/297252/. So, could you add me to the group, to have ability to add a bot-user into the group? | 11:50 |
| *** wfoster has joined #openstack-infra | 11:50 | |
| *** flaper87 has joined #openstack-infra | 11:51 | |
| *** maishsk has joined #openstack-infra | 11:55 | |
| *** weshay has joined #openstack-infra | 11:56 | |
| *** baoli has joined #openstack-infra | 11:57 | |
| *** maishsk has quit IRC | 11:59 | |
| *** Liuqing_ has quit IRC | 12:02 | |
| *** asettle has quit IRC | 12:05 | |
| *** zeih has joined #openstack-infra | 12:09 | |
| *** roxanaghe has joined #openstack-infra | 12:12 | |
| *** maishsk has joined #openstack-infra | 12:12 | |
| *** jaypipes has joined #openstack-infra | 12:12 | |
| *** achanda has joined #openstack-infra | 12:12 | |
| *** aysyd has joined #openstack-infra | 12:13 | |
| *** zeih has quit IRC | 12:16 | |
| *** roxanaghe has quit IRC | 12:17 | |
| *** achanda has quit IRC | 12:17 | |
| *** vgridnev has joined #openstack-infra | 12:23 | |
| *** rlandy has joined #openstack-infra | 12:23 | |
| *** dprince has joined #openstack-infra | 12:24 | |
| *** ZZelle has joined #openstack-infra | 12:24 | |
| *** ilyashakhat has joined #openstack-infra | 12:26 | |
| *** erlon has joined #openstack-infra | 12:27 | |
| AJaeger | project-config cores, there's a patch to add a --verbose option to zuul_swift_upload.py. It has seen little review. Is that something we want? https://review.openstack.org/269288 | 12:27 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 12:27 |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| *** Daisy has joined #openstack-infra | 12:29 | |
| *** edmondsw has joined #openstack-infra | 12:30 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs https://review.openstack.org/297038 | 12:30 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: midonet: Make tempest jobs voting https://review.openstack.org/298142 | 12:32 |
| *** _degorenko is now known as degorenko | 12:32 | |
| AJaeger | infra-root, oslo.service is in the release since 9 hours now - waiting for wheel-release-cent7x64 to build . Are we missing a node again? | 12:32 |
| *** amrith is now known as _amrith_ | 12:33 | |
| *** Daisy has quit IRC | 12:33 | |
| *** amitgandhinz has joined #openstack-infra | 12:34 | |
| ilyashakhat | SergeyLukjanov: please do me a favor and add me to performa-core and performa-release groups :) | 12:35 |
| *** julim has joined #openstack-infra | 12:36 | |
| *** zhurong has joined #openstack-infra | 12:37 | |
| *** pcaruana has quit IRC | 12:39 | |
| *** kgiusti has joined #openstack-infra | 12:40 | |
| *** ilyashakhat has quit IRC | 12:42 | |
| *** yamamoto has quit IRC | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:47 | |
| *** piet has joined #openstack-infra | 12:47 | |
| *** amitgandhinz has quit IRC | 12:48 | |
| *** amitgandhinz has joined #openstack-infra | 12:49 | |
| *** salv-orlando has joined #openstack-infra | 12:50 | |
| sdague | http://logs.openstack.org/11/297311/1/check/gate-grenade-dsvm-multinode/be9e943/logs/grenade.sh.txt.gz#_2016-03-28_12_08_53_469 is quite surprising | 12:51 |
| *** amitgandhinz has quit IRC | 12:51 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again https://review.openstack.org/288915 | 12:51 |
| *** yamamoto has quit IRC | 12:52 | |
| *** pcaruana has joined #openstack-infra | 12:52 | |
| *** salv-orlando has quit IRC | 12:53 | |
| *** amitgandhinz has joined #openstack-infra | 12:54 | |
| *** Nikolay_St has quit IRC | 12:55 | |
| *** yamamoto has joined #openstack-infra | 12:55 | |
| *** amitgandhinz has quit IRC | 12:56 | |
| *** kushal has quit IRC | 12:56 | |
| *** annegentle has joined #openstack-infra | 12:56 | |
| *** amitgandhinz has joined #openstack-infra | 12:56 | |
| *** dimtruck is now known as zz_dimtruck | 13:01 | |
| *** huzhj_ has joined #openstack-infra | 13:01 | |
| *** Nikolay_St has joined #openstack-infra | 13:02 | |
| *** abregman has quit IRC | 13:02 | |
| pabelanger | AJaeger: Not missing, but it is about ~8mins slower then our ubuntu-trusty wheel mirrors. So it's been getting backed up in the pipeline | 13:04 |
| pabelanger | AJaeger: https://jenkins.openstack.org/label/wheel-mirror-centos-7-amd64/ is the node | 13:05 |
| *** jcoufal has joined #openstack-infra | 13:05 | |
| *** yfried has quit IRC | 13:06 | |
| pabelanger | Ubuntu trusty wheel is ~40mins where Centos 7 wheel is 48mins | 13:06 |
| pabelanger | https://jenkins.openstack.org/job/wheel-build-centos-7-amd64/buildTimeTrend vs https://jenkins.openstack.org/job/wheel-build-ubuntu-trusty-amd64/buildTimeTrend | 13:06 |
| *** yamamoto has quit IRC | 13:12 | |
| *** mtanino has quit IRC | 13:13 | |
| *** achanda has joined #openstack-infra | 13:15 | |
| *** rcrit_ has quit IRC | 13:16 | |
| *** julim has quit IRC | 13:17 | |
| *** yamamoto has joined #openstack-infra | 13:17 | |
| *** abregman has joined #openstack-infra | 13:17 | |
| *** _ari_ has joined #openstack-infra | 13:18 | |
| annegentle | AJaeger: you know about checksum failures? Just saw this it on this patch https://review.openstack.org/#/c/288315 | 13:18 |
| *** kushal has joined #openstack-infra | 13:19 | |
| *** achanda has quit IRC | 13:19 | |
| *** maishsk has quit IRC | 13:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove sbcl from bindep-fallback.txt https://review.openstack.org/298229 | 13:20 |
| fungi | AJaeger: the wheel release jobs run from the wheel-build workers, which are only able to run one job at a time | 13:20 |
| *** mtanino has joined #openstack-infra | 13:21 | |
| *** yamamoto has quit IRC | 13:21 | |
| *** maishsk has joined #openstack-infra | 13:21 | |
| *** ayoung has joined #openstack-infra | 13:22 | |
| *** annegentle has quit IRC | 13:23 | |
| *** julim has joined #openstack-infra | 13:23 | |
| *** zz_dimtruck is now known as dimtruck | 13:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add distro package install for run-tests https://review.openstack.org/298230 | 13:24 |
| AJaeger | fungi, it runs very slow. We shouldn't have to wait 9hours to release 10 packages or so... | 13:25 |
| *** annegentle has joined #openstack-infra | 13:25 | |
| *** annegentle has quit IRC | 13:25 | |
| fungi | AJaeger: well, all the important release jobs completed. those are just wheel mirror updates (and very redundant ones at this point) | 13:25 |
| *** jsavak has joined #openstack-infra | 13:25 | |
| AJaeger | pabelanger: the centos7 wheel looks far slower than those 8 mins. To me it sounds more like 80 mins instead of 48 ;( | 13:25 |
| fungi | the pypi upload completed and announcement was sent for oslo.utils >10 hours ago | 13:26 |
| AJaeger | fungi: yes, indeed. Just still seeing now three jobs there - plus the one in the periodic queue, is odd. | 13:26 |
| fungi | the one in post looks like a stuck job. it's been running for over 5 hours | 13:27 |
| *** mtanino has quit IRC | 13:27 | |
| *** rfolco_ has joined #openstack-infra | 13:27 | |
| AJaeger | annegentle: That's again a network problem on rax-ord ;( | 13:27 |
| *** tqtran has joined #openstack-infra | 13:28 | |
| *** piet has quit IRC | 13:28 | |
| fungi | looks like it stuck trying to upload logs to swift almost 5 hours ago | 13:28 |
| AJaeger | fungi, we still have network problems on rax-ord. Not sure whether there's anything we can do ;( | 13:28 |
| fungi | especially annoying since rackspace moved most of our quota to ord | 13:28 |
| AJaeger | fungi: ;( | 13:28 |
| * AJaeger will recheck annegentle's job... | 13:29 | |
| fungi | the increase in jobs running there may explain the increase in network related issues | 13:29 |
| fungi | oh, annegentle also had a rax-ord network issue on a job? i thought you were talking about the stuck swift upload for that heat coverage job (which is also in rax-ord) | 13:30 |
| AJaeger | fungi, I'm currently trying to move sbcl out of bindep-fallback and run into 298230, could you check that one later, please? | 13:30 |
| AJaeger | fungi: I was looking at backscroll - didn't look at post ;) | 13:30 |
| *** tqtran has quit IRC | 13:32 | |
| AJaeger | thanks, fungi | 13:32 |
| fungi | you're welcome! | 13:33 |
| AJaeger | pabelanger, fungi: Trying to get packages out of bindep-fallback is larger untertaking for some of them. firefox is used by 16+ repositories... | 13:34 |
| *** mriedem has joined #openstack-infra | 13:35 | |
| fungi | and may be hard to find them all | 13:35 |
| fungi | without some collateral damage anyway | 13:35 |
| AJaeger | and will need backports of other-requirements ;( | 13:35 |
| *** abregman has quit IRC | 13:35 | |
| *** links has quit IRC | 13:36 | |
| *** piet has joined #openstack-infra | 13:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding Mellanox Extension for the OpenStack Dashboard (Horizon) https://review.openstack.org/295260 | 13:38 |
| pabelanger | AJaeger: Ya, I imagine it will take some trial and error | 13:38 |
| openstackgerrit | Merged openstack-infra/project-config: Adding searchlight-ui repo and jobs https://review.openstack.org/293848 | 13:38 |
| * AJaeger will be back later | 13:39 | |
| *** akshai has joined #openstack-infra | 13:45 | |
| *** nmagnezi has quit IRC | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Add distro package install for run-tests https://review.openstack.org/298230 | 13:47 |
| *** armax has joined #openstack-infra | 13:48 | |
| *** openstackgerrit has quit IRC | 13:48 | |
| *** abregman has joined #openstack-infra | 13:48 | |
| *** openstackgerrit has joined #openstack-infra | 13:48 | |
| *** regXboi has joined #openstack-infra | 13:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Run the v3 only job in neutron https://review.openstack.org/293200 | 13:50 |
| *** dimtruck is now known as zz_dimtruck | 13:50 | |
| AJaeger | fungi, pabelanger: We have now bindep-fallback both as list of packages to install - and as list of packages to cache. If we remove entries from it and move them to other-requirements, should we create a cache file in addition? | 13:51 |
| *** armax has quit IRC | 13:52 | |
| *** yamamoto has quit IRC | 13:52 | |
| AJaeger | that cache file would include packages that some projects install via other-requirements but are not installed by default. | 13:52 |
| openstackgerrit | Merged openstack-infra/project-config: Make monasca persister tempest tests voting https://review.openstack.org/297278 | 13:52 |
| *** annegent_ has joined #openstack-infra | 13:52 | |
| *** annegent_ has quit IRC | 13:52 | |
| *** annegent_ has joined #openstack-infra | 13:53 | |
| fungi | AJaeger: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-bindep/install.d/50-cache-pkgs#n34 | 13:55 |
| AJaeger | fungi: wow! Didn't know that we do it already. thanks | 13:55 |
| *** _amrith_ is now known as amrith | 13:56 | |
| *** nadya has quit IRC | 13:56 | |
| *** eharney has joined #openstack-infra | 13:56 | |
| pabelanger | Ya, we do use other-requirements.txt as a cache file for DIBs. But the issue is, we only validate other-requirements.txt ubuntu-trusty. I've been wrapping my brain around how to test it over other OSes | 13:56 |
| openstackgerrit | Tom Cocozzello proposed openstack-infra/project-config: Experimental patch_cover gate job https://review.openstack.org/296808 | 13:56 |
| AJaeger | pabelanger: thanks for implementing that (if the NOTE is correct) | 13:56 |
| *** nadya has joined #openstack-infra | 13:56 | |
| AJaeger | fungi, did we have a bindep release? See http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-bindep/install.d/50-cache-pkgs#n48 | 13:56 |
| *** nadya has quit IRC | 13:56 | |
| fungi | pabelanger: well... we validate it for whatever jobs a given project runs. but yes it's probably quite easy for a project not running any install-distro-packages jobs on fedora to include an entry which breaks package caching on fedora. i think it was suggested there may be an option to tell the package manager to ignore missing packages? | 13:58 |
| AJaeger | pabelanger: you could run a periodic test that goes through all files and installs on fedora etc. | 13:58 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:59 | |
| pabelanger | Right, I was thinking of a periodic job for other-requirements for each other OS then ubuntu-trusty | 13:59 |
| fungi | AJaeger: https://review.openstack.org/195201 merged as 4db1a7a and `git tag --contains 4db1a7a` says 0.1.0 so we can update that element safely now, i think | 13:59 |
| fungi | thanks for spotting | 13:59 |
| *** roxanaghe has joined #openstack-infra | 14:00 | |
| fungi | yep, just checked out 0.1.0 specifically and the support for using "-f -" to read stdin is included | 14:01 |
| AJaeger | fungi: great. Will you update? | 14:01 |
| fungi | working on it now | 14:01 |
| AJaeger | thanks! | 14:02 |
| *** bryan_att has joined #openstack-infra | 14:02 | |
| *** knikolla has joined #openstack-infra | 14:04 | |
| *** roxanaghe has quit IRC | 14:04 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Drop tempfile workaround for aggregate bindep list https://review.openstack.org/298242 | 14:05 |
| fungi | AJaeger: ^ | 14:05 |
| HeOS | Infra-team, I want to repeat my question: what should I do to add myself and Fuel DevOps Robot to the following group https://review.openstack.org/#/admin/groups/1336,members? :) | 14:06 |
| pabelanger | AJaeger: we could do something like 297809 but for other-requirements.txt, and allow projects to add it as a template | 14:06 |
| *** rbrndt_ has joined #openstack-infra | 14:08 | |
| AJaeger | Neat idea - Just install all packages when other-requirements is updated. | 14:08 |
| fungi | that means making sure every project which might create an other-requirements.txt file (which could be _any_ project) has those jobs configured | 14:08 |
| AJaeger | we could add this to python-jobs, so that every project has it ;) | 14:08 |
| AJaeger | And then add a template similar to check-requirements | 14:09 |
| fungi | there are tons of projects which don't use the python-jobs group | 14:09 |
| pabelanger | well, to start, we could just have people add the template themselves if they are interested | 14:10 |
| AJaeger | fungi: you're right, many use it. But adding a new job-group would be nice | 14:10 |
| pabelanger | until we weed out the bugs | 14:10 |
| fungi | i think instead we should make the caching robust so that an untested but broken other-requirements.txt file doesn't stop image builds/package caching | 14:10 |
| AJaeger | pabelanger: I'm working on ha-guide setup right now - as docs core - so, I volunteer to guinea pig that repo ;) | 14:10 |
| pabelanger | AJaeger: Ya, odyssey4me has all the openstack ansible projects setup too | 14:11 |
| fungi | also the nova example is up for review now at https://review.openstack.org/200672 | 14:11 |
| fungi | mriedem has already provided awesome feedback ;) | 14:11 |
| pabelanger | AJaeger: let me see about setting up a template | 14:11 |
| pabelanger | fungi: Ooohh, nice | 14:11 |
| mriedem | \o/ | 14:12 |
| *** yamamoto has joined #openstack-infra | 14:12 | |
| AJaeger | fungi, I think we need both - the repo job will help projects spot problems but the robust caching is essential... | 14:13 |
| *** zeih has joined #openstack-infra | 14:13 | |
| Shrews | mtreinish: if I +A 285615, can you add a follow up with a new reno release note? | 14:14 |
| Shrews | mtreinish: or you can add it to the current review | 14:15 |
| mtreinish | Shrews: sure I can do that | 14:15 |
| mtreinish | I'm all for stat padding | 14:15 |
| fungi | AJaeger: well, the ideal way to validate the package list is to run useful jobs for those platforms, more than just testing installation of the package list (and i can see projects which explicitly don't care about fedora 23 support not wanting a voting job which tests that their package list installation doesn't break on fedora 23) | 14:15 |
| Shrews | mtreinish: approved | 14:16 |
| AJaeger | fungi: yeah, we have many projects only caring about one OS ;( | 14:16 |
| *** annegent_ has quit IRC | 14:18 | |
| *** zeih has quit IRC | 14:18 | |
| AJaeger | fungi, btw. do you want to kill the heat post job? Or what should be done with it? | 14:18 |
| dmsimard | It looks like Jenkins 2 release candidate is meant so ship sometime in April. Have you checked it out ? Any incentive to upgrade ? | 14:18 |
| fungi | AJaeger: i'll kill it | 14:19 |
| pabelanger | dmsimard: likely not. I suspect we want to land zuulv3 first | 14:19 |
| *** doug-fis_ has joined #openstack-infra | 14:19 | |
| Shrews | mtreinish: oh, actually, i'm going to go ahead and approve mordred's follow up, so if your note could reference the new wait_for_image() too, that would be awesome | 14:19 |
| pabelanger | which will remove the need for jenkins | 14:19 |
| dmsimard | pabelanger: fair enough, are you getting closer to v3 now ? | 14:19 |
| dhellmann | has anyone else reported a babel problem "AttributeError: 'datetime.date' object has no attribute 'hour'"? | 14:19 |
| fungi | dmsimard: yeah, we're planning to stop using jenkins soonish via http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 14:19 |
| mtreinish | Shrews: sure, I'll add that too | 14:20 |
| *** knikolla has quit IRC | 14:20 | |
| fungi | dmsimard: late in the newton release cycle or maybe early in ocata? hard to say for sure, depends on how many people pitch in as the momentum builds | 14:20 |
| dmsimard | fungi: thanks :D | 14:21 |
| pabelanger | dmsimard: we have a branch up: https://git.openstack.org/cgit/openstack-infra/zuul/log/?h=feature/zuulv3 and ansible is launching jobs. Like fungi says, more people helping out, the faster it will land. | 14:21 |
| *** knikolla has joined #openstack-infra | 14:21 | |
| *** doug-fish has quit IRC | 14:21 | |
| *** ajmiller has joined #openstack-infra | 14:22 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Add release notes for new create_image_snapshot() args https://review.openstack.org/298253 | 14:22 |
| mtreinish | Shrews: ^^^ | 14:22 |
| Shrews | mtreinish: thx | 14:23 |
| dims | network issues - "Download error on http://mirror.ord.rax.openstack.org/pypi/simple/pbr/: [Errno 101] Network is unreachable -- Some packages may not be found!" | 14:23 |
| dims | reading scrollback, sounds like known issues :) | 14:24 |
| *** tiswanso has joined #openstack-infra | 14:24 | |
| fungi | dims: yeah, i'm worried that we may need to stop using rax-ord for a while | 14:25 |
| fungi | is that still going on? | 14:25 |
| fungi | i know they postponed some of their disruptive network maintenance to last night or tonight | 14:25 |
| dhellmann | oh, fun, a new version of sphinx seems to be breaking the release repo doc build | 14:25 |
| *** zz_dimtruck is now known as dimtruck | 14:27 | |
| *** akshai has quit IRC | 14:29 | |
| *** smarcet has joined #openstack-infra | 14:29 | |
| *** akshai has joined #openstack-infra | 14:29 | |
| pabelanger | fungi: we never did restart zuul because of the memory leak | 14:30 |
| *** jamesmcarthur has joined #openstack-infra | 14:31 | |
| *** Hal has joined #openstack-infra | 14:31 | |
| *** Hal is now known as Guest21263 | 14:31 | |
| smarcet | fungi: morning, seems that openstackid-dev is down | 14:31 |
| *** annegentle has joined #openstack-infra | 14:31 | |
| fungi | smarcet: down and you don't know why, or down and you're flooding it with requests? | 14:32 |
| jamesmcarthur | down and we don’t know why | 14:32 |
| smarcet | fungi: down and dunno | 14:32 |
| smarcet | seems DNS problem | 14:32 |
| jamesmcarthur | ping openstackid-dev.org | 14:33 |
| jamesmcarthur | ping: cannot resolve openstackid-dev.org: Unknown host | 14:33 |
| *** Nikolay_St has quit IRC | 14:33 | |
| fungi | that has never existed | 14:34 |
| fungi | are you sure you're not thinking of openstackid-dev.openstack.org? | 14:34 |
| *** jpr has quit IRC | 14:34 | |
| fungi | looks like it's up for me(and that's the name it's always had) | 14:34 |
| jamesmcarthur | ok - web team going for coffee | 14:35 |
| fungi | heh. i could use some more myself ;) | 14:35 |
| fungi | mondays! | 14:35 |
| Clint | manic | 14:35 |
| * fungi imagines Clint walking like an egyptian | 14:36 | |
| * morgan is a coffee zombie still. | 14:36 | |
| Clint | all the better for you to walk like an egyptian in my stead | 14:37 |
| fungi | pabelanger: yep, looks like we edged into swap over the weekend http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=394&rra_id=all | 14:37 |
| fungi | pabelanger: we should probably coordinate a restart with the release team so that we don't disrupt any ongoing release activities by dropping jobs in ancillary pipelines | 14:38 |
| *** markvoelker has joined #openstack-infra | 14:38 | |
| *** jamesmcarthur has quit IRC | 14:38 | |
| jeblair | we might also experimentally try restarting it without the foreign pipeline patch, or without connections. | 14:38 |
| *** jamesmcarthur has joined #openstack-infra | 14:38 | |
| jeblair | to narrow down which of those is the cause of the leak | 14:39 |
| fungi | i'm in favor as long as one of they can still be easily untangled | 14:39 |
| fungi | er, as long as they can still be easily untangled | 14:40 |
| fungi | we're not relying on either of those features right? | 14:40 |
| jeblair | correct | 14:40 |
| fungi | the foreign pipelines change is really for third parties using a subset of our projects | 14:41 |
| fungi | iirc | 14:41 |
| fungi | and connections is a stepping stone to the github trigger and such? | 14:41 |
| jeblair | yep. though i should have called it 'foreign project'. | 14:41 |
| jeblair | yep. | 14:41 |
| fungi | either (or both) seem safe to me | 14:42 |
| jeblair | (though mostly originally written so we could use two accounts for the same gerrit) | 14:42 |
| *** sdake has joined #openstack-infra | 14:42 | |
| *** denisra has joined #openstack-infra | 14:42 | |
| oanson | Hi. | 14:42 |
| *** mylu has joined #openstack-infra | 14:42 | |
| dims | jeblair fungi : hmm, ec2-api went awol on gerrit - https://review.openstack.org/#/admin/projects/?filter=ec2 | 14:43 |
| *** yfried has joined #openstack-infra | 14:43 | |
| oanson | I got 'ValueError: '/usr/local/lib/python2.7/dist-packages' is not in list' in our CI tests using virtualenv 14.0.6. Forcing the gate to use 15.0.1 solved the issue. | 14:43 |
| *** nelsnelson has joined #openstack-infra | 14:43 | |
| fungi | removing both would allow us to disprove the theory that the leak is caused by one of them, i guess, but picking one to remove seems marginally more likely to yield a quicker answer since chances are it is one of them | 14:43 |
| oanson | Am I right in understanding that this sort of change would require upper-constraints to be raised for virtualenv? | 14:43 |
| openstackgerrit | Merged openstack-infra/project-config: Drop tempfile workaround for aggregate bindep list https://review.openstack.org/298242 | 14:43 |
| *** mylu has quit IRC | 14:44 | |
| *** asselin has joined #openstack-infra | 14:44 | |
| fungi | oanson: we don't constrain virtualenv, we constrain the things we install in a virtualenv. there are bootstrapping issues with managing pip, setuptools, virtualenv and tox in global requirements | 14:44 |
| *** mylu has joined #openstack-infra | 14:44 | |
| oanson | fungi: So the version of virtualenv on the gate is not decided by upper-constraints? | 14:45 |
| *** jamesmcarthur has quit IRC | 14:46 | |
| *** sdake has quit IRC | 14:46 | |
| fungi | oanson: depends on the job (i think if you're running under devstack, it may use that to decide what version of virtualenv to use when setting up tempest) | 14:46 |
| *** sdake has joined #openstack-infra | 14:47 | |
| fungi | oanson: most of our jobs don't use devstack, and run with whatever version of virtualenv we preinstall on our servers | 14:47 |
| oanson | fungi: I see. I think this job does run devstack, but I will verify. | 14:48 |
| fungi | oanson: since you don't say what job you're encountering this with, i don't really know | 14:48 |
| oanson | Thanks for your help. | 14:48 |
| oanson | It's on dragonflow fullstack tests | 14:48 |
| fungi | virtualenv in constraints with devstack is sort of a corner case for us | 14:48 |
| oanson | http://logs.openstack.org/97/296897/4/check/gate-dragonflow-dsvm-fullstack-nv/6f4999d/console.html | 14:48 |
| *** annegentle has quit IRC | 14:48 | |
| *** Daisy has joined #openstack-infra | 14:49 | |
| fungi | dims: oh, wow, that is all sorts of broken. the openstack/ec2-api repo is still present on our git servers but gerrit is pitching permission errors if i try to view any changes linked from it | 14:50 |
| oanson | fungi, thanks again for your help. | 14:50 |
| dims | fungi : guess we need to see who else is awol | 14:50 |
| openstackgerrit | Merged openstack-infra/shade: Add wait support to create_image_snapshot() https://review.openstack.org/285615 | 14:51 |
| fungi | dims: well, i'm looking for logs now. it's possible this is fallout from network disruption to cinder or trove | 14:51 |
| dims | fungi : thanks | 14:51 |
| fungi | doesn't seem to be filesystem related (nothing screaming recently in dmesg) | 14:52 |
| *** burgerk has joined #openstack-infra | 14:53 | |
| *** annegentle has joined #openstack-infra | 14:53 | |
| *** Daisy has quit IRC | 14:54 | |
| openstackgerrit | Merged openstack-infra/shade: Split waiting for images into its own method https://review.openstack.org/297836 | 14:54 |
| openstackgerrit | Merged openstack-infra/shade: Fix grant_role docstring https://review.openstack.org/297810 | 14:54 |
| fungi | infra-root: heads up ^ we may have gerrit corruption of some sort underway | 14:54 |
| fungi | [2016-03-28 14:54:44,073] WARN com.google.gerrit.server.project.ProjectCacheImpl : Cannot read project openstack/ec2-api | 14:55 |
| *** mylu has quit IRC | 14:56 | |
| *** asselin has quit IRC | 14:56 | |
| *** mylu has joined #openstack-infra | 14:56 | |
| fungi | here's the first thing i can find | 14:56 |
| fungi | [2016-03-27 05:01:15,884] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/openstack/ec2-driver.git/objects/pack/pack-193d82faea4408f0191a06cde2c253abb493fead.pack was deleted, removing it from pack list | 14:56 |
| fungi | lots of that for a variety of projects, so might just be normal git pack related messages | 14:57 |
| fungi | however these two are adjacent | 14:57 |
| fungi | [2016-03-27 05:41:19,448] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/openstack/ec2-api.git/objects/pack/pack-3a686787179a5e70d4c53e9be6897ec5ed0821b2.pack was deleted, removing it from pack list | 14:57 |
| fungi | [2016-03-27 05:41:19,450] WARN com.google.gerrit.server.project.ProjectCacheImpl : Cannot read project openstack/ec2-api | 14:57 |
| jeblair | the first message makes me wonder if we should flush the gerrit cache | 14:57 |
| *** fawadkhaliq has joined #openstack-infra | 14:58 | |
| *** maishsk has quit IRC | 14:58 | |
| *** asselin has joined #openstack-infra | 14:58 | |
| fungi | grepping for "Cannot read project" it seems to be only openstack/ec2-api affected luckily | 14:58 |
| *** annegentle has quit IRC | 14:59 | |
| zaro | morning | 14:59 |
| *** ilyashakhat has joined #openstack-infra | 14:59 | |
| fungi | i can call flush-caches | 15:00 |
| *** zhurong has quit IRC | 15:00 | |
| *** jsavak has quit IRC | 15:00 | |
| fungi | unless there's a good reason to worry it will make matters worse | 15:00 |
| *** mylu has quit IRC | 15:00 | |
| jeblair | fungi: note your first errer message was for ec2-driver, not ec2-api | 15:00 |
| fungi | aha, yep | 15:00 |
| fungi | but the adjacent packfile removal and read error were for ec2-api | 15:01 |
| *** jsavak has joined #openstack-infra | 15:01 | |
| jeblair | fungi: i copied the repo and ran git fsck --full; nothing alarming (only some dangling blobs which are expected due to our lack of gc) | 15:01 |
| fungi | zaro: any reason to worry about calling flush-caches in a situation like this, do you think? | 15:02 |
| *** sputnik13 has joined #openstack-infra | 15:03 | |
| *** mylu has joined #openstack-infra | 15:03 | |
| *** LaT0rtue_ has joined #openstack-infra | 15:04 | |
| fungi | dims: i take it you discovered the query issue with ec2-api while trying to do semi-urgent release work? | 15:06 |
| fungi | i can go ahead and give teh cache flush a try, and next on the list of possible terrible solutions is probably a gerrit service restart | 15:06 |
| dims | fungi : this was raised on mailing list | 15:06 |
| fungi | ahh, busy morning so haven't gotten through e-mail from overnight | 15:07 |
| fungi | thanks | 15:07 |
| dims | fungi : not urgent from release perspective, just concerned if we should tag releases or not | 15:07 |
| *** edmondsw has quit IRC | 15:07 | |
| fungi | cache flush is running now, will keep an eye out for new and exciting errors | 15:07 |
| dhellmann | dims : I'm going to make the call that we wait to tag anything until this is sorted. I don't want to have to retag and risk having irreproducible deliverables. | 15:08 |
| fungi | i don't like flush-caches because it makes gerrit pretty unresponsive for several minutes | 15:08 |
| dims | dhellmann : +1 | 15:08 |
| *** jamesmcarthur has joined #openstack-infra | 15:08 | |
| fungi | dhellmann: agreed, would like to make sure this is just a cosmetic issue with our git repack and gerrit's persistent cache fighting, rather than the start of something more significant | 15:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Document other-requirements https://review.openstack.org/298281 | 15:09 |
| AJaeger | fungi, pabelanger, could you review this later, please? ^ | 15:10 |
| jeblair | fungi: i don't necessarily think the repack is related. it could be, but i don't know we have a strong correlation yet. | 15:10 |
| dhellmann | fungi : yep. let me know if I can help, otherwise I'll monitor the discussion here | 15:11 |
| fungi | jeblair: right, we have a sample size of 1. i'm not confident we'll know what's caused it without seeing it happen more often | 15:11 |
| *** annegentle has joined #openstack-infra | 15:11 | |
| *** jcoufal has quit IRC | 15:12 | |
| fungi | cache flush doesn't seem to have solved this | 15:12 |
| *** esker has quit IRC | 15:12 | |
| *** esker has joined #openstack-infra | 15:13 | |
| jeblair | fungi: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 72803eb8013ff25d7751b4a7c6cd6571d6631435 | 15:14 |
| *** jamesmcarthur has quit IRC | 15:14 | |
| jeblair | fungi: i can see that object | 15:14 |
| *** oanson has quit IRC | 15:14 | |
| *** jamesmcarthur has joined #openstack-infra | 15:14 | |
| fungi | yep. this seems like the git operation timeout issue we saw back when we tried upgrading to... 2.9? | 15:14 |
| jeblair | http://git.openstack.org/cgit/openstack/ec2-api/commit/?id=72803eb8013ff25d7751b4a7c6cd6571d6631435 | 15:15 |
| *** akshai has quit IRC | 15:15 | |
| *** akshai has joined #openstack-infra | 15:15 | |
| jeblair | (it's no wonder it's not displaying at all if it thinks it can't read its config history) | 15:16 |
| fungi | i can see it in ~gerrit2/review_site/git/openstack/ec2-api.git too | 15:16 |
| *** yamamoto has quit IRC | 15:16 | |
| jeblair | fungi: agreed. i think we found that only restarting corrected that. | 15:16 |
| *** yamamoto has joined #openstack-infra | 15:17 | |
| fungi | so if there is a repack correlation, it might be that we pounded the fs with i/o or possibly hung for a bit updating that object (causing reads to block) right when gerrit tried to read it | 15:17 |
| *** vgridnev has quit IRC | 15:17 | |
| *** achanda has joined #openstack-infra | 15:17 | |
| fungi | okay, any objections to an emergency gerrit restart? | 15:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Document other-requirements https://review.openstack.org/298281 | 15:17 |
| jeblair | fungi: i thought the 'timeout -> failed read' bug was corrected | 15:17 |
| jeblair | fungi: no objections to restart. | 15:17 |
| *** dizquierdo has quit IRC | 15:17 | |
| *** cloudtrainme has joined #openstack-infra | 15:18 | |
| fungi | i'll get the full traceback into a paste once the restart is going | 15:18 |
| *** vgridnev has joined #openstack-infra | 15:18 | |
| *** sbalukoff has quit IRC | 15:18 | |
| anteaya | fungi: no objection to restart | 15:19 |
| *** woodster_ has joined #openstack-infra | 15:19 | |
| fungi | #status notice Gerrit is restarting on review.openstack.org in an attempt to address an issue reading an object from the ec2-api repository | 15:20 |
| openstackstatus | fungi: sending notice | 15:20 |
| fungi | looks like it should be back up now | 15:21 |
| -openstackstatus- NOTICE: Gerrit is restarting on review.openstack.org in an attempt to address an issue reading an object from the ec2-api repository | 15:21 | |
| *** yamamoto has quit IRC | 15:21 | |
| fungi | looks like that solved it | 15:22 |
| fungi | https://review.openstack.org/#/admin/projects/openstack/ec2-api | 15:22 |
| fungi | https://review.openstack.org/297305 | 15:22 |
| *** rraja has joined #openstack-infra | 15:22 | |
| fungi | et cetera | 15:22 |
| dansmith | fungi: have you seen some extra failures talking to git.openstack.org this morning in the tests? | 15:22 |
| openstackstatus | fungi: finished sending notice | 15:22 |
| dansmith | I thought that you were doing a restart for that reason, but I see not | 15:22 |
| fungi | dansmith: we've seen some, all of which seem to be for workers in rax-ord | 15:22 |
| *** achanda has quit IRC | 15:22 | |
| dansmith | https://jenkins07.openstack.org/job/gate-nova-python34-db/83/console | 15:23 |
| dansmith | okay | 15:23 |
| dansmith | yeah, in rax-ord | 15:23 |
| fungi | it seems like rackspace is having some sort of network issues not reported on https://status.rackspace.com/ | 15:24 |
| dansmith | they were last week too right? | 15:24 |
| fungi | yep | 15:24 |
| dansmith | awesome | 15:24 |
| *** dmellado is now known as dmellado|off | 15:24 | |
| fungi | they also shifted most of our quota to ord due to load problems in iad | 15:24 |
| dansmith | oof | 15:25 |
| fungi | and the uptick in network problems we're seeing for jobs in ord seems like it might roughly correspond | 15:25 |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config: Add stable branch jobs for tripleo-ci https://review.openstack.org/297390 | 15:25 |
| dansmith | fungi: seems like ever since alaski left, they've had nothing but problems.. coincidence? | 15:25 |
| *** bpokorny has joined #openstack-infra | 15:26 | |
| fungi | dims: dhellmann: openstack/ec2-api seems fine in gerrit now since the restart | 15:26 |
| alaski | dansmith: hah | 15:26 |
| dims | fungi : cool thanks! | 15:26 |
| fungi | dansmith: alaski: correlation implying causation there? ;) | 15:27 |
| dims | haha | 15:27 |
| *** armax has joined #openstack-infra | 15:27 | |
| sdague | wow - the git server failure rate is crazy - http://status.openstack.org//elastic-recheck/#1298006 | 15:27 |
| dhellmann | fungi : thanks! | 15:27 |
| dansmith | fungi: I dunno.. seems clear to me :D | 15:27 |
| fungi | sdague: yeah, i'm considering proposing turning off rax-ord in nodepool, though we need someone to write up a coherent trouble ticket | 15:28 |
| *** yamamoto has joined #openstack-infra | 15:28 | |
| *** yamamoto has quit IRC | 15:28 | |
| *** ddieterly has joined #openstack-infra | 15:28 | |
| *** annegent_ has joined #openstack-infra | 15:28 | |
| sdague | fungi: yeh, it looks almost entirely there | 15:28 |
| sdague | fungi: how many nodes do we lose with that going off? | 15:28 |
| anteaya | 19x | 15:28 |
| fungi | almost a couple hundred | 15:28 |
| *** fawadkhaliq has quit IRC | 15:28 | |
| fungi | i think | 15:28 |
| fungi | since rackspace asked us to move most of our load to ord | 15:29 |
| *** Qiming has quit IRC | 15:29 | |
| sdague | any idea if they realize ord is this bad? | 15:29 |
| fungi | 195 says nodepool.yaml | 15:29 |
| anteaya | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n219 | 15:29 |
| anteaya | yeah | 15:29 |
| *** ilyashakhat has quit IRC | 15:30 | |
| fungi | sdague: i'm thinking they don't have any clue since https://status.rackspace.com/ makes no mention of it | 15:30 |
| fungi | which is why someone should throw together a trouble ticket | 15:30 |
| fungi | infra-root have access to the rackspace dashboard | 15:30 |
| *** roxanaghe has joined #openstack-infra | 15:31 | |
| anteaya | fungi: do you want me to offer the project-config patch to turn off ord? | 15:31 |
| *** rraja has quit IRC | 15:31 | |
| fungi | we likely need to put together a list of nova instance uuids which exhibited the issue | 15:31 |
| *** annegentle has quit IRC | 15:31 | |
| *** salv-orlando has joined #openstack-infra | 15:32 | |
| fungi | i guess this is a holiday/hangoverday for enough people that we're not driving full quota so it's probably okay to take ord out for the moment | 15:32 |
| sdague | yeh, eu is basically entirely off today | 15:32 |
| *** tiswanso has quit IRC | 15:32 | |
| elmiko | AJaeger: ping | 15:32 |
| *** tiswanso has joined #openstack-infra | 15:32 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Turn off rax-ord https://review.openstack.org/298286 | 15:33 |
| anteaya | fungi: done ^^ | 15:33 |
| fungi | thanks | 15:34 |
| anteaya | welcome | 15:34 |
| jeblair | if someone can get me a list of some nodes that have exhibited the failure, i will put together the list of uuids and open the trouble ticket | 15:34 |
| fungi | thanks jeblair! | 15:34 |
| fungi | i'm looking to see how to easily extract that from kibana | 15:34 |
| *** roxanaghe has quit IRC | 15:35 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Turn off rax-ord https://review.openstack.org/298286 | 15:35 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 15:36 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job https://review.openstack.org/296765 | 15:36 |
| anteaya | I had an errant whitespace | 15:36 |
| *** Nikolay_St has joined #openstack-infra | 15:37 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 15:38 |
| fungi | jeblair: tab-delimited list of times and node names http://paste.openstack.org/show/492090/ | 15:38 |
| *** kzaitsev_mb has quit IRC | 15:39 | |
| fungi | will probably want to dedupe the node names | 15:39 |
| fungi | those were the 100 most recent hits | 15:39 |
| anteaya | fungi: sorry you had +2'd patchset 1 which had an extra whitespace, which I removed in patchset 2 | 15:39 |
| *** keedya has joined #openstack-infra | 15:39 | |
| jeblair | fungi: thx | 15:39 |
| *** mylu has quit IRC | 15:40 | |
| jeblair | fungi: those times are when the network error occured? | 15:40 |
| zxiiro | does git-review support signed pushes? | 15:40 |
| fungi | jeblair: yes, according to the logs where those timestamps were obtained | 15:41 |
| *** salv-orlando has quit IRC | 15:41 | |
| *** torandu has quit IRC | 15:41 | |
| *** torandu has joined #openstack-infra | 15:41 | |
| fungi | zxiiro: git-review supports git and doesn't break signed commits as far as i've noticed | 15:41 |
| anteaya | I'm going back to Easter Mondaying | 15:41 |
| AJaeger | elmiko: what's up? | 15:41 |
| *** tphummel has joined #openstack-infra | 15:42 | |
| fungi | zxiiro: it doesn't explicitly _support_ signed commits, it just treats them like any other commit | 15:42 |
| zxiiro | fungi: So latest Gerrit supports "signed pushes" which I think just means you do "git push --sign" or something. I guess git-review needs to be updated to support that flag since I can't find a way to enable ti. | 15:42 |
| *** roxanaghe has joined #openstack-infra | 15:42 | |
| fungi | zxiiro: i've never heard of "signed pushes" just signed objects (commits, tags, et cetera) | 15:43 |
| elmiko | AJaeger: had a question about the sahara-bandit review, gonna push a new version here shortly | 15:43 |
| zxiiro | fungi: I didn't either until 5 minutes ago. I guess it's a shiny new feature. | 15:43 |
| openstackgerrit | Michael McCune proposed openstack-infra/project-config: remove sahara-bandit job https://review.openstack.org/297271 | 15:43 |
| elmiko | AJaeger: i misunderstood one of your comments, but i think i figured it out | 15:43 |
| fungi | zxiiro: oh, wow, i just found the docs for push --signed | 15:43 |
| *** mtanino has joined #openstack-infra | 15:43 | |
| AJaeger | elmiko: ok, will review later | 15:44 |
| fungi | zxiiro: so git-review calls `git push` and would presumably need some mechanism to know that it needs to use a different command | 15:44 |
| elmiko | AJaeger: thanks! | 15:44 |
| zxiiro | fungi: yeah, in latest Gerrit. It tags your pushes with a "?" if you don't use a signed push. | 15:44 |
| zxiiro | fungi: Cool. I think this might be something I can contribute to. I'll poke around at the code. | 15:44 |
| *** vgridnev has quit IRC | 15:44 | |
| *** IlyaG has joined #openstack-infra | 15:45 | |
| fungi | zxiiro: also git-review could use some reviewing love. i started going through the oldest open reviews for it a week ago, but didn't get very far | 15:45 |
| *** eharney has quit IRC | 15:45 | |
| fungi | also hashar has asked on the ml if we would tag a new release for it, which i haven't followed up on yet | 15:46 |
| *** EricGonczer_ has joined #openstack-infra | 15:46 | |
| zxiiro | fungi: sure think I can help out. | 15:46 |
| fungi | would be good to see if there are any missing bug fixes of test coverage improvements we should land before making a new release | 15:46 |
| fungi | zxiiro: thanks! | 15:46 |
| fungi | er, bug fixes OR test coverage improvements | 15:47 |
| fungi | fingers gettign ahead of my brain this morning | 15:47 |
| *** mylu has joined #openstack-infra | 15:47 | |
| *** apoorvad has joined #openstack-infra | 15:48 | |
| *** vgridnev has joined #openstack-infra | 15:49 | |
| *** huzhj_ has quit IRC | 15:49 | |
| *** mylu has quit IRC | 15:49 | |
| *** mylu has joined #openstack-infra | 15:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:49 | |
| openstackgerrit | Michael McCune proposed openstack-infra/project-config: remove sahara-bandit job https://review.openstack.org/297271 | 15:50 |
| *** ihrachys has joined #openstack-infra | 15:50 | |
| *** annegent_ has quit IRC | 15:51 | |
| fungi | pleia2: StevenK: request on the i18n ml to look into a login problem for a user whose account may have been migrated: http://lists.openstack.org/pipermail/openstack-i18n/2016-March/002043.html | 15:52 |
| *** jsavak has quit IRC | 15:53 | |
| *** mylu has quit IRC | 15:53 | |
| mnaser | hi -infra, not sure if uploads have been still struggling lately, but we have a fix (which is unfortunately pending hardware shipment right now) | 15:53 |
| *** jsavak has joined #openstack-infra | 15:53 | |
| fungi | pleia2: StevenK: i'll reply letting them know to make sure they've got an account at www.openstack.org but you may have better suggestions | 15:53 |
| *** keedya has quit IRC | 15:53 | |
| *** annegentle has joined #openstack-infra | 15:54 | |
| *** hashar has joined #openstack-infra | 15:54 | |
| *** keedya has joined #openstack-infra | 15:56 | |
| fungi | i need to pop out to lunch, but will be back shortly | 15:56 |
| pc_m | Hi! Anyone know how I can specify a particular commit for a repo, when using devstack? Have a CI failure, and I want to try to bisect to find commit introducing the failure. | 15:56 |
| *** sputnik13 has quit IRC | 15:57 | |
| *** ArxCruz has joined #openstack-infra | 15:58 | |
| *** EricGonc_ has joined #openstack-infra | 15:58 | |
| jeblair | pc_m: http://docs.openstack.org/developer/devstack/stackrc.html | 15:59 |
| *** ilyashakhat has joined #openstack-infra | 15:59 | |
| *** akshai has quit IRC | 16:00 | |
| *** akshai has joined #openstack-infra | 16:01 | |
| *** eharney has joined #openstack-infra | 16:01 | |
| *** EricGonczer_ has quit IRC | 16:02 | |
| *** mylu has joined #openstack-infra | 16:02 | |
| *** hashar has quit IRC | 16:02 | |
| *** mylu has quit IRC | 16:03 | |
| *** tqtran has joined #openstack-infra | 16:04 | |
| *** maishsk has joined #openstack-infra | 16:04 | |
| *** vgridnev has quit IRC | 16:04 | |
| *** akshai has quit IRC | 16:05 | |
| *** nadya has joined #openstack-infra | 16:05 | |
| jeblair | fungi: these are errors talking to git.o.o? (so ord -> dfw ?) | 16:05 |
| *** fitoduarte has joined #openstack-infra | 16:05 | |
| *** akshai has joined #openstack-infra | 16:06 | |
| *** jamesmcarthur has quit IRC | 16:06 | |
| *** piet has quit IRC | 16:07 | |
| *** piet has joined #openstack-infra | 16:07 | |
| pleia2 | fungi: thanks | 16:07 |
| *** vtapia has joined #openstack-infra | 16:07 | |
| *** mylu has joined #openstack-infra | 16:08 | |
| jeblair | fungi: 160328-iad-0001210 submitted | 16:10 |
| pabelanger | and back | 16:10 |
| pabelanger | just catching up on backscroll | 16:10 |
| *** ilyashakhat has quit IRC | 16:12 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 16:12 | |
| *** kzaitsev_mb has quit IRC | 16:12 | |
| *** dimtruck is now known as zz_dimtruck | 16:13 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack-infra/project-config: Make gate-sahara-tests-python34 job voting https://review.openstack.org/298304 | 16:14 |
| *** zeih has joined #openstack-infra | 16:14 | |
| *** yamahata has joined #openstack-infra | 16:15 | |
| *** tiswanso has quit IRC | 16:16 | |
| *** e0ne has quit IRC | 16:16 | |
| *** tonytan4ever has joined #openstack-infra | 16:18 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: remove pypi publishing jobs for cloudkitty https://review.openstack.org/298307 | 16:19 |
| *** zeih has quit IRC | 16:19 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config: Add stable branch jobs for tripleo-ci https://review.openstack.org/297390 | 16:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: DNM: Revert "Dependencies from undefined projects" https://review.openstack.org/298308 | 16:19 |
| jeblair | fungi, pabelanger: assuming that passes tests, when we restart zuul, let's manually install that commit and run it ^ | 16:20 |
| *** piet has quit IRC | 16:21 | |
| *** jamesmcarthur has joined #openstack-infra | 16:22 | |
| *** esikachev has joined #openstack-infra | 16:23 | |
| nibalizer | good morning | 16:24 |
| nibalizer | I have returned from vacation | 16:24 |
| *** fawadkhaliq has joined #openstack-infra | 16:25 | |
| *** ilyashakhat_mobi has quit IRC | 16:25 | |
| jeblair | nibalizer: welcome back! i hope you enjoyed it! | 16:27 |
| nibalizer | i did | 16:27 |
| *** fedexo has joined #openstack-infra | 16:27 | |
| *** gyee has joined #openstack-infra | 16:28 | |
| *** tiswanso has joined #openstack-infra | 16:28 | |
| *** yamamoto has joined #openstack-infra | 16:28 | |
| pabelanger | nibalizer: welcome back | 16:29 |
| nibalizer | thanks ! | 16:29 |
| pabelanger | jeblair: WFM | 16:29 |
| nibalizer | how have things been here? | 16:29 |
| nibalizer | I've been intentionally not paying attention | 16:29 |
| pabelanger | were off devstack-trusty now | 16:29 |
| pabelanger | and centos7 wheel mirror online | 16:29 |
| pabelanger | that all I have | 16:30 |
| nibalizer | neat | 16:30 |
| *** Sukhdev has joined #openstack-infra | 16:30 | |
| pabelanger | and I started this over the weekend: https://review.openstack.org/#/q/topic:ubuntu-xenial | 16:30 |
| pabelanger | surprisingly ubuntu-xenial dibs are looking promising | 16:31 |
| *** cody-somerville has quit IRC | 16:31 | |
| *** LaT0rtue_ has quit IRC | 16:31 | |
| *** cody-somerville has joined #openstack-infra | 16:31 | |
| *** cody-somerville has quit IRC | 16:31 | |
| *** cody-somerville has joined #openstack-infra | 16:31 | |
| *** jsavak has quit IRC | 16:31 | |
| *** hieulq has joined #openstack-infra | 16:31 | |
| *** tphummel has quit IRC | 16:31 | |
| *** abregman is now known as abregman|afk | 16:32 | |
| greghaynes | why surprisingly? ;) | 16:32 |
| *** mylu has quit IRC | 16:32 | |
| *** jsavak has joined #openstack-infra | 16:32 | |
| pabelanger | jeblair: which account is rax does infra use for support issues? | 16:32 |
| pabelanger | greghaynes: not as much diskimage-builder, but our puppet manifests | 16:32 |
| greghaynes | ah | 16:32 |
| pabelanger | minimal changes so far | 16:32 |
| *** esikachev has quit IRC | 16:33 | |
| *** jsavak has quit IRC | 16:33 | |
| *** armax has quit IRC | 16:33 | |
| greghaynes | awesome | 16:33 |
| *** jsavak has joined #openstack-infra | 16:33 | |
| pabelanger | greghaynes: so, found an issue with the base element and dkms last night. Not sure the best way to fix it. Basically, dkms on centos7 is epel, which base does not depend on. I don't think I can add the epel element directly since it is yum syntax | 16:33 |
| jeblair | pabelanger: generally the account tied to whichever servers are having the issue. however, in this case, i spazzed out and filed the ticket under openstackci instead of openstackjenkins like i should have. | 16:34 |
| greghaynes | pabelanger: eek. If that can be moved out of the base element then that would be awesome | 16:34 |
| pabelanger | jeblair: Ah, make sense | 16:34 |
| jeblair | pabelanger: we'll see if that's an issue :) | 16:34 |
| pabelanger | greghaynes: Ya, was going to ping ianw about later today. And get his thoughts | 16:34 |
| greghaynes | pabelanger: as in, make centos/rhel/fedora all depend on a dkms element which does all the things it does need to | 16:34 |
| pabelanger | I've worked around it for now | 16:34 |
| greghaynes | ok | 16:34 |
| *** zz_dimtruck is now known as dimtruck | 16:35 | |
| pabelanger | greghaynes: right | 16:35 |
| *** mylu has joined #openstack-infra | 16:35 | |
| *** abregman|afk has quit IRC | 16:36 | |
| *** dizquierdo has joined #openstack-infra | 16:36 | |
| *** yamamoto has quit IRC | 16:36 | |
| *** annegentle has quit IRC | 16:40 | |
| *** earlephilhower has quit IRC | 16:42 | |
| *** david-lyle_ has joined #openstack-infra | 16:44 | |
| *** david-lyle has quit IRC | 16:44 | |
| jeblair | clarkb, mordred: the nodepool-shade change passes tests and has a +2 from me. https://review.openstack.org/226751 | 16:45 |
| *** claudiub|2 has quit IRC | 16:45 | |
| *** ArxCruz has quit IRC | 16:46 | |
| *** jerryz has joined #openstack-infra | 16:46 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 16:46 |
| *** maishsk has quit IRC | 16:47 | |
| *** Nikolay_St has quit IRC | 16:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Remove unused nspr dependency https://review.openstack.org/298319 | 16:47 |
| *** ihrachys has quit IRC | 16:47 | |
| *** Guest21263 has quit IRC | 16:48 | |
| *** david-lyle has joined #openstack-infra | 16:48 | |
| *** armax has joined #openstack-infra | 16:48 | |
| *** jamesmcarthur has quit IRC | 16:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Remove libnspr4-dev https://review.openstack.org/298320 | 16:48 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs https://review.openstack.org/297038 | 16:48 |
| *** ddieterly is now known as ddieterly[away] | 16:48 | |
| *** fedexo has quit IRC | 16:49 | |
| *** david-lyle_ has quit IRC | 16:49 | |
| *** LaT0rtue has joined #openstack-infra | 16:49 | |
| mordred | jeblair: woot! | 16:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove libnspr4-dev and nspr-devel https://review.openstack.org/298321 | 16:50 |
| *** tphummel has joined #openstack-infra | 16:50 | |
| AJaeger | pabelanger, fungi: Just removing libnspr4-dev is a large cleanup ;( https://review.openstack.org/#/q/topic:spidermonkey | 16:51 |
| *** jamesmcarthur has joined #openstack-infra | 16:52 | |
| *** annegentle has joined #openstack-infra | 16:52 | |
| *** skolekonov is now known as skolekonov|away | 16:53 | |
| *** ddieterly[away] is now known as ddieterly | 16:54 | |
| pabelanger | AJaeger: wow, that is a lot of places | 16:55 |
| *** sputnik13 has joined #openstack-infra | 16:55 | |
| AJaeger | pabelanger: yeah ;( | 16:55 |
| *** maishsk has joined #openstack-infra | 16:56 | |
| *** Swami has joined #openstack-infra | 16:57 | |
| *** jsavak has quit IRC | 16:58 | |
| jroll | fungi: would it be too difficult to be able to hold a node in rax-ord that exhibits network badness? | 16:58 |
| *** jamesmcarthur has quit IRC | 16:58 | |
| AJaeger | jroll: I just run into one like https://jenkins03.openstack.org/job/gate-swift-python27/2360/console - but I fear it's too late to notice these kind of errors | 16:59 |
| jroll | AJaeger: yeah, that was my thought, too fast between fail and delete | 16:59 |
| AJaeger | There's no long timeout, it's a direct refusal | 16:59 |
| AJaeger | jroll: exactly | 16:59 |
| *** fitoduarte has quit IRC | 16:59 | |
| *** asselin has quit IRC | 16:59 | |
| *** asselin has joined #openstack-infra | 17:00 | |
| *** tonytan4ever has quit IRC | 17:00 | |
| *** jamesmcarthur has joined #openstack-infra | 17:00 | |
| *** tonytan4ever has joined #openstack-infra | 17:01 | |
| *** piet has joined #openstack-infra | 17:01 | |
| *** salv-orlando has joined #openstack-infra | 17:02 | |
| *** ddieterly is now known as ddieterly[away] | 17:02 | |
| *** dizquierdo is now known as dizquierdo_afk | 17:04 | |
| *** arxcruz has joined #openstack-infra | 17:04 | |
| *** ociuhandu_ has joined #openstack-infra | 17:05 | |
| *** ociuhandu has quit IRC | 17:05 | |
| *** ociuhandu_ is now known as ociuhandu | 17:05 | |
| *** yamahata has quit IRC | 17:07 | |
| *** vgridnev has joined #openstack-infra | 17:07 | |
| *** yamahata has joined #openstack-infra | 17:07 | |
| openstackgerrit | Jay_Clark proposed openstack-infra/irc-meetings: As discussed with the team, our new timeslot is 1300UTC. This commit changes our meeting time, and sets the channel to openstack-meeting-4. https://review.openstack.org/298335 | 17:09 |
| *** jamesmcarthur has quit IRC | 17:11 | |
| *** cj has quit IRC | 17:13 | |
| *** EricGonc_ has quit IRC | 17:14 | |
| *** cj has joined #openstack-infra | 17:15 | |
| jeblair | fungi: do you know if clarkb is away today? | 17:15 |
| *** salv-orlando has quit IRC | 17:16 | |
| *** SumitNaiksatam has quit IRC | 17:16 | |
| *** mylu has quit IRC | 17:16 | |
| *** mylu has joined #openstack-infra | 17:18 | |
| fungi | i am back now, and do not know clarkb's whereabouts | 17:18 |
| *** mylu has quit IRC | 17:19 | |
| *** jsavak has joined #openstack-infra | 17:19 | |
| *** oanson has joined #openstack-infra | 17:19 | |
| fungi | jeblair: yes, those were nodes matching network errors reaching git.openstack.org for a variety of operations covered by the query at http://status.openstack.org//elastic-recheck/#1298006 | 17:20 |
| fungi | thanks for submitting 160328-iad-0001210 | 17:20 |
| *** piet has quit IRC | 17:21 | |
| *** achanda has joined #openstack-infra | 17:22 | |
| fungi | jeblair: jenkins reports py27 unit test failures on 298308 | 17:22 |
| *** edmondsw has joined #openstack-infra | 17:23 | |
| fungi | however, debug output for zuul unit test failures is so massive it's taking me a bit to retrieve and parse | 17:23 |
| jeblair | fungi: yeah, i'm fixing 308 up now | 17:23 |
| fungi | okay, great. i'll stop investigating ;) | 17:23 |
| fungi | wow, still downloading | 17:23 |
| jeblair | fungi: the connections change had a slight entanglement with foreign projects that i didn't catch in the revert | 17:24 |
| *** mylu has joined #openstack-infra | 17:24 | |
| *** david-lyle has quit IRC | 17:24 | |
| *** sridhar_ram has joined #openstack-infra | 17:24 | |
| fungi | quantum entanglement? (he asks, hopefully) | 17:25 |
| *** smarcet has quit IRC | 17:25 | |
| fungi | i think openstack is no longer quantum entangled. we renamed all that | 17:25 |
| *** esikachev has joined #openstack-infra | 17:25 | |
| *** david-lyle has joined #openstack-infra | 17:25 | |
| AJaeger | fungi, could you review my documentation of other-requirements when you have some time, please? https://review.openstack.org/298281 | 17:25 |
| *** Hal has joined #openstack-infra | 17:26 | |
| clarkb | jeblair: slow start today | 17:26 |
| *** Hal is now known as Guest79255 | 17:26 | |
| *** achanda has quit IRC | 17:27 | |
| fungi | clarkb: i recommend switching to a backoff algorithm if the day does not proceed according to base protocol | 17:27 |
| clarkb | ya... not sure how much I will end up being around today | 17:27 |
| *** cloudtrainme has quit IRC | 17:27 | |
| clarkb | it has been very meh so far | 17:28 |
| *** lakshmiS has joined #openstack-infra | 17:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: DNM: Revert "Dependencies from undefined projects" https://review.openstack.org/298308 | 17:28 |
| jeblair | clarkb: okay; i'm trying to determine the feasibility of landing the nodepool-shade change today | 17:29 |
| pabelanger | exciting! | 17:29 |
| *** IlyaG has quit IRC | 17:29 | |
| jeblair | clarkb: i'm here for the next 5.5 hours and could babysit it if we could land it soon. but i don't know what your review status is on it. | 17:30 |
| clarkb | jeblair: I had managed to review most of the change the other day, I still hadn't gotten through the entire set of provider manager changes | 17:30 |
| clarkb | I can try doing that this morning, but wouldn't count on me to babysit | 17:30 |
| pabelanger | I have some spare cycles to help today, if needed | 17:32 |
| AJaeger | pabelanger, sdague: jd__ confirmed that nspr is not used anymore, could you put this cleanup changes on your review list for the next days, please? https://review.openstack.org/#/q/topic:spidermonkey | 17:33 |
| *** Goneri has joined #openstack-infra | 17:34 | |
| *** BigWillie has joined #openstack-infra | 17:36 | |
| *** pvaneck has joined #openstack-infra | 17:37 | |
| jeblair | clarkb: cool, i think if you can manage to do that before 2000utc i can probably put it into production then with enough time to back it out if there are problems before i eod. | 17:37 |
| fungi | i can commit to reviewing it if more reviews are needed, and expect to be around at least until 2200 or so | 17:38 |
| fungi | but also have other things i can work on if it's in hand | 17:38 |
| jeblair | fungi: thanks; up to you -- i singled out clarkb because he had previous comments on it. | 17:39 |
| fungi | yep, preferred if he has time, but real life sometimes intercedes | 17:39 |
| *** dtantsur is now known as dtantsur|afk | 17:39 | |
| fungi | i'll try to give it a peek shortly | 17:40 |
| clarkb | I am catching up on the recent patches now then will review the prvider manager diff | 17:40 |
| *** esker has quit IRC | 17:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/gerrit: Fix .gitreview file https://review.openstack.org/298346 | 17:41 |
| *** jamesmcarthur has joined #openstack-infra | 17:41 | |
| *** hieulq has quit IRC | 17:41 | |
| *** piet has joined #openstack-infra | 17:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/gerrit: List system dependencies for running common tests https://review.openstack.org/298347 | 17:43 |
| *** doug-fis_ has quit IRC | 17:43 | |
| lakshmiS | AJaeger: https://review.openstack.org/#/c/294224 experimental worked. Thanks!!! Should i add another patch moving experimental to acutal? | 17:44 |
| *** krtaylor has quit IRC | 17:44 | |
| *** nelsnels_ has joined #openstack-infra | 17:46 | |
| *** achanda has joined #openstack-infra | 17:46 | |
| *** abe_music has joined #openstack-infra | 17:46 | |
| *** abe_music has quit IRC | 17:47 | |
| *** tongli has joined #openstack-infra | 17:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:47 | |
| *** nelsnelson has quit IRC | 17:48 | |
| AJaeger | lakshmiS: yes | 17:49 |
| lakshmiS | ok | 17:49 |
| *** arxcruz has quit IRC | 17:50 | |
| *** akshai has quit IRC | 17:51 | |
| *** akshai has joined #openstack-infra | 17:52 | |
| *** tqtran has quit IRC | 17:53 | |
| *** IlyaG has joined #openstack-infra | 17:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Document other-requirements https://review.openstack.org/298281 | 17:53 |
| AJaeger | fungi, thanks for review. I reworked a bit ^ | 17:53 |
| fungi | AJaeger: awesome, taking another look momentarily | 17:54 |
| *** krtaylor has joined #openstack-infra | 17:55 | |
| *** piet has quit IRC | 17:56 | |
| *** Sukhdev has quit IRC | 17:57 | |
| AJaeger | fungi, I'm experimenting a bit with other-requirements - or cleanup of bindep-fallback. I found one completely unused package - see https://review.openstack.org/#/q/topic:spidermonkey - and start using it for documentation projects. | 17:57 |
| *** e0ne has joined #openstack-infra | 17:57 | |
| * AJaeger will continue as time permits | 17:57 | |
| AJaeger | But enough of that for today | 17:57 |
| fungi | AJaeger: very cool find. i have a feeling some things fell out of use and were simply copied from the lists in our puppet manifests because i lacked time to research them all | 17:58 |
| AJaeger | fungi, yep. Similar with asciidoc: Removed from manuals long time ago. Gerrit is only user today... | 17:59 |
| AJaeger | I'll be off for now... | 17:59 |
| fungi | have a good evening AJaeger! | 17:59 |
| *** jamesmcarthur has quit IRC | 17:59 | |
| AJaeger | thanks, fungi | 18:00 |
| *** baoli has quit IRC | 18:00 | |
| *** tiswanso has quit IRC | 18:00 | |
| AJaeger | One quick question: Where do we install maven? I was surprised to see it on the nodes but couldn't find a reference? | 18:00 |
| clarkb | jeblair: +2'd but not approved, will let you do that when you are ready. The test logs looked good from what I could see too, though I didn't count api calls | 18:00 |
| *** baoli has joined #openstack-infra | 18:01 | |
| *** jsavak has quit IRC | 18:01 | |
| *** tiswanso has joined #openstack-infra | 18:01 | |
| *** jamesmcarthur has joined #openstack-infra | 18:02 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:02 | |
| *** jsavak has joined #openstack-infra | 18:02 | |
| *** ddieterly[away] has quit IRC | 18:03 | |
| AJaeger | off now, will read answer in backscroll later ;) | 18:03 |
| *** Jeffrey4l has quit IRC | 18:04 | |
| *** annegentle has quit IRC | 18:04 | |
| *** doug-fish has joined #openstack-infra | 18:04 | |
| *** piet has joined #openstack-infra | 18:05 | |
| *** doug-fis_ has joined #openstack-infra | 18:05 | |
| *** vgridnev has quit IRC | 18:06 | |
| jeblair | clarkb: thanks, aprvd, will restart after lunch | 18:07 |
| pabelanger | Going to give it a test run locally too | 18:08 |
| jeblair | infra-root: i will be periodically afk for a bit this week: traveling to/attending board meeting tuesday; conferencing wednesday; traveling home thursday. | 18:08 |
| *** oanson has quit IRC | 18:08 | |
| jeblair | (it's in my backyard so i figured i'd pop in) | 18:09 |
| *** doug-fish has quit IRC | 18:09 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: dhcp-all-interfaces depends on dhcp https://review.openstack.org/296626 | 18:11 |
| openstackgerrit | Lakshmi N Sampath proposed openstack-infra/project-config: Change experimental to actual for searchlight https://review.openstack.org/298360 | 18:11 |
| openstackgerrit | Merged openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 18:11 |
| pleia2 | jeblair: have fun | 18:11 |
| *** Rockyg has joined #openstack-infra | 18:12 | |
| *** smarcet has joined #openstack-infra | 18:12 | |
| *** david-lyle_ has joined #openstack-infra | 18:12 | |
| fungi | you're hosting a conference in your back yard? awesome! | 18:12 |
| *** david-lyle has quit IRC | 18:14 | |
| jeblair | well, someone else is hosting it, which is kind of weird i guess | 18:15 |
| *** zeih has joined #openstack-infra | 18:15 | |
| *** degorenko is now known as _degorenko|afk | 18:15 | |
| *** pcaruana has quit IRC | 18:16 | |
| *** sbalukoff has joined #openstack-infra | 18:16 | |
| fungi | don't forget to keep plenty of pickle relish on hand | 18:16 |
| *** salv-orlando has joined #openstack-infra | 18:18 | |
| *** amuller has joined #openstack-infra | 18:18 | |
| amuller | Good day everyone. Time for my weekly 'Is it just me or everyone else?'. We're seeing lots of git timeouts in various Neutron jobs. | 18:18 |
| amuller | And so I wanted to ask if ya'all are seeing it anywhere else, if it's a known issue, etc | 18:19 |
| *** ddieterly has joined #openstack-infra | 18:19 | |
| *** zeih has quit IRC | 18:19 | |
| jeblair | amuller: yeah, seems to be problems in one particular region, so we're in the process of shutting it off: https://review.openstack.org/298286 | 18:20 |
| *** ArxCruz has joined #openstack-infra | 18:20 | |
| *** ddieterly is now known as ddieterly[away] | 18:20 | |
| amuller | jeblair: You're a saint, thank you | 18:20 |
| amuller | kevinbenton: ^ | 18:20 |
| openstackgerrit | Merged openstack-infra/shade: Add release notes for new create_image_snapshot() args https://review.openstack.org/298253 | 18:21 |
| fungi | they're running openstack, so you know... all the usual sausage factory analogies applie | 18:21 |
| fungi | apply | 18:22 |
| fungi | what an odd typo | 18:22 |
| pabelanger | hopefully this is duped, lots connection to znc | 18:22 |
| pabelanger | amuller: Yup, ticket with rax already created | 18:22 |
| pabelanger | amuller: you can also check http://status.openstack.org/elastic-recheck/#1298006 too in the future | 18:22 |
| *** ddieterly[away] is now known as ddieterly | 18:22 | |
| amuller | checking http://status.openstack.org/elastic-recheck would be a good idea in general in these cases I suppose | 18:23 |
| *** jamesmcarthur has quit IRC | 18:23 | |
| mordred | jeblair: I'm around, although on a plane - so I can help look at any issues that arise if they do | 18:23 |
| amuller | sometimes I'm one of the first people to notice new failures so I often pop by in here to ask | 18:23 |
| mordred | jeblair: and I'm sure Shrews is excited to help too :) | 18:23 |
| *** IlyaG has quit IRC | 18:23 | |
| *** edmondsw has quit IRC | 18:23 | |
| mordred | Shrews: (btw, shade nodepool patch lnaded) | 18:23 |
| jeblair | mordred, Shrews: cool. i'm planning on restarting nodepool with it around 20:00utc. i'm prepared to babysit it myself, but figure if either/both of you are around that's ++. | 18:24 |
| Shrews | mordred: jeblair: so excited | 18:24 |
| *** ArxCruz has quit IRC | 18:24 | |
| fungi | early morning on summit wednesday! we have an infra fishbowl booked for 9am! | 18:24 |
| *** salv-orlando has quit IRC | 18:25 | |
| jeblair | fungi: i'd like to report your non-standard use of "!". | 18:25 |
| Shrews | jeblair: that's dinner time for me (if i did the conversion correctly), but will attempt to have my laptop close by | 18:25 |
| fungi | jeblair: stuck key, what can i say! | 18:25 |
| fungi | !!!1!eleven | 18:25 |
| openstack | fungi: Error: "!!1!eleven" is not a valid command. | 18:25 |
| jeblair | Shrews: it's ~1.5 hours from now | 18:25 |
| fungi | i know openstackbot, i know... | 18:26 |
| Shrews | jeblair: oh. i failed at conversion | 18:26 |
| jeblair | Shrews: it's more like your tea-time, which i know is important to you. :) | 18:26 |
| mordred | jeblair: I expect one of four outcomes, btw: | 18:26 |
| openstackgerrit | Eric Brown proposed openstack-infra/project-config: Remove oslo.messaging from bandit integration https://review.openstack.org/298366 | 18:26 |
| openstackgerrit | Lakshmi N Sampath proposed openstack-infra/project-config: Change experimental to actual for searchlight https://review.openstack.org/298360 | 18:26 |
| *** EricGonczer_ has joined #openstack-infra | 18:26 | |
| Shrews | jeblair: it IS important. but i will be here then | 18:27 |
| fungi | mordred: we're using a two-bit analysis now? | 18:27 |
| jeblair | mordred: fail; fail; success; success? | 18:27 |
| jeblair | fungi, mordred: big fail; little fail; little success; big success? | 18:27 |
| jeblair | "ba da boom" "big ba da boom" | 18:28 |
| mordred | jeblair: 1) it works perfectly fine and we notice essentially nothing 2) it works fine, but is making more API calls than we'd like, but we can fix by doing analysis and follow up patches 3) it breaks, but the breaks and easy/obvious and we can roll proper fixes in quickly 4) it breaks stupendously and we have to revert | 18:28 |
| mordred | so, yes | 18:28 |
| openstackgerrit | Merged openstack-infra/project-config: Turn off rax-ord https://review.openstack.org/298286 | 18:28 |
| jeblair | mordred: clarkb did mention he didn't count api calls; so we could be surprised there :) | 18:28 |
| fungi | or just switch to hard liquor | 18:29 |
| clarkb | part of that is firefox has really bad search counting now | 18:29 |
| mordred | jeblair: yah. I mean, I believe we've rectified the api call count imbalance from last time | 18:30 |
| clarkb | "more than 100 matches" | 18:30 |
| clarkb | why not give me a number... | 18:30 |
| mordred | but also, I'm hoping that if we missed something it's not so bad that we need to react harshly and can just work it | 18:30 |
| clarkb | http://logs.openstack.org/51/226751/92/check/gate-dsvm-nodepool/ead1b1a/logs/screen-nodepool.txt.gz#_2016-03-26_09_54_39_490 might get annoying but we dont shade at debug level in prod | 18:30 |
| *** pcaruana has joined #openstack-infra | 18:30 | |
| jeblair | clarkb: it's the same misguided thinking that gets us "about 10 minutes ago" | 18:30 |
| *** EricGonczer_ has quit IRC | 18:31 | |
| mordred | (1 extra call per server - annoying, but not crippling - 100 extra calls per server, obviously unworkable) | 18:31 |
| *** jaypipes has quit IRC | 18:31 | |
| fungi | i think gerrit is transitioning to lapine counting (one, two, three, hrair, hrair, hrair, hrair...) | 18:31 |
| *** EricGonczer_ has joined #openstack-infra | 18:32 | |
| *** vgridnev has joined #openstack-infra | 18:33 | |
| *** baoli has quit IRC | 18:33 | |
| *** kdas_ has joined #openstack-infra | 18:33 | |
| *** tqtran has joined #openstack-infra | 18:34 | |
| *** sdake_ has joined #openstack-infra | 18:34 | |
| *** sdake_ has quit IRC | 18:35 | |
| *** kushal has quit IRC | 18:35 | |
| clarkb | the thing to grep for is "running task" and compare that to teh run fro shadification and the runs prior | 18:35 |
| *** baoli has joined #openstack-infra | 18:35 | |
| clarkb | keep in mind the shadification change adds in some extra operations so it won't be a direct coparison | 18:35 |
| *** keedya has quit IRC | 18:36 | |
| *** sdake has quit IRC | 18:37 | |
| *** dizquierdo_afk is now known as dizquierdo | 18:37 | |
| zaro | fungi: sorry i wasn't around for the bad gerrit repo. did you ever get the log dump to see if we can narrow down that problem? | 18:39 |
| *** sdake has joined #openstack-infra | 18:39 | |
| *** baoli has quit IRC | 18:40 | |
| fungi | zaro: i have the stack trace, meant to put it into a paste... just a moment | 18:41 |
| *** knikolla_ has joined #openstack-infra | 18:41 | |
| *** baoli has joined #openstack-infra | 18:41 | |
| *** fitoduarte has joined #openstack-infra | 18:43 | |
| *** doug-fis_ has quit IRC | 18:43 | |
| zaro | jeblair: the change that wasn't searchable, https://review.openstack.org/#/q/change:13725 is actually due to a bug with cherry-pick in Gerrit 2.4, https://gerrit-review.googlesource.com/#/c/38220/ | 18:44 |
| *** cloudtrainme has joined #openstack-infra | 18:44 | |
| *** doug-fish has joined #openstack-infra | 18:44 | |
| openstackgerrit | James Slagle proposed openstack-infra/project-config: Add a periodic mitaka job for TripleO https://review.openstack.org/298369 | 18:44 |
| *** sdake has quit IRC | 18:44 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add support for mitaka jobs https://review.openstack.org/298370 | 18:44 |
| pabelanger | installing latest nodepool locally now | 18:44 |
| fungi | zaro: http://paste.openstack.org/show/492136/ | 18:44 |
| zaro | jeblair: might be able to fix that that by manually creating the ref but not sure if it's worth doing. | 18:45 |
| openstackgerrit | James Slagle proposed openstack-infra/project-config: Add a periodic mitaka job for TripleO https://review.openstack.org/298369 | 18:46 |
| *** baoli has quit IRC | 18:46 | |
| *** sdake has joined #openstack-infra | 18:48 | |
| *** baoli has joined #openstack-infra | 18:48 | |
| *** e0ne has quit IRC | 18:50 | |
| *** achanda has quit IRC | 18:51 | |
| *** jamesmcarthur has joined #openstack-infra | 18:51 | |
| *** esikachev has quit IRC | 18:52 | |
| *** achanda has joined #openstack-infra | 18:54 | |
| dims | AJaeger : can you please cross check my answer in https://review.openstack.org/#/c/270622/ (string translations!) | 18:54 |
| *** fawadkhaliq has quit IRC | 18:55 | |
| *** fawadkhaliq has joined #openstack-infra | 18:55 | |
| *** fitoduarte has quit IRC | 18:56 | |
| *** baoli has quit IRC | 18:58 | |
| *** baoli has joined #openstack-infra | 18:59 | |
| *** baoli has quit IRC | 19:00 | |
| *** brad_behle has joined #openstack-infra | 19:01 | |
| *** tiswanso has quit IRC | 19:01 | |
| *** edmondsw has joined #openstack-infra | 19:01 | |
| *** sridhar_ram1 has joined #openstack-infra | 19:01 | |
| *** sridhar_ram has quit IRC | 19:02 | |
| *** piet has quit IRC | 19:02 | |
| *** tiswanso has joined #openstack-infra | 19:02 | |
| *** jamesmcarthur has quit IRC | 19:08 | |
| pabelanger | jeblair: mordred: rebuild of local nodepool looks good. Everything came online as expect in my local cloud | 19:08 |
| *** earlephilhower has joined #openstack-infra | 19:10 | |
| openstackgerrit | Cedric Brandily proposed openstack-infra/shade: Support InsecureRequestWarning == None https://review.openstack.org/297791 | 19:10 |
| mordred | pabelanger: WOOT! | 19:10 |
| *** EricGonczer_ has quit IRC | 19:10 | |
| *** jamesmcarthur has joined #openstack-infra | 19:10 | |
| pabelanger | Yup, another data point! | 19:10 |
| *** annegentle has joined #openstack-infra | 19:13 | |
| *** IlyaG has joined #openstack-infra | 19:14 | |
| *** tphummel has quit IRC | 19:17 | |
| *** ZZelle has quit IRC | 19:18 | |
| *** e0ne has joined #openstack-infra | 19:19 | |
| *** nadya has quit IRC | 19:19 | |
| *** e0ne has quit IRC | 19:21 | |
| *** angdraug has joined #openstack-infra | 19:22 | |
| *** doug-fis_ has joined #openstack-infra | 19:22 | |
| *** brad_behle has left #openstack-infra | 19:23 | |
| *** IlyaG has quit IRC | 19:24 | |
| openstackgerrit | Merged openstack/requirements: Remove libnspr4-dev https://review.openstack.org/298320 | 19:25 |
| *** doug-fish has quit IRC | 19:25 | |
| *** roxanaghe has quit IRC | 19:25 | |
| *** e0ne has joined #openstack-infra | 19:26 | |
| *** Sukhdev has joined #openstack-infra | 19:27 | |
| *** annegentle has quit IRC | 19:27 | |
| *** doug-fis_ has quit IRC | 19:27 | |
| *** doug-fish has joined #openstack-infra | 19:28 | |
| *** e0ne has quit IRC | 19:28 | |
| *** brad_behle has joined #openstack-infra | 19:31 | |
| *** jamesmcarthur has quit IRC | 19:31 | |
| *** brad_behle has left #openstack-infra | 19:32 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: improve check_library_constraints.sh to edit the constraints file https://review.openstack.org/298398 | 19:33 |
| *** annegentle has joined #openstack-infra | 19:33 | |
| *** roxanaghe has joined #openstack-infra | 19:35 | |
| *** esker has joined #openstack-infra | 19:35 | |
| *** sdake_ has joined #openstack-infra | 19:36 | |
| *** mylu has quit IRC | 19:36 | |
| jeblair | zaro: wow, thanks | 19:36 |
| *** MarkAtwood has joined #openstack-infra | 19:37 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: improve check_library_constraints.sh to edit the constraints file https://review.openstack.org/298398 | 19:37 |
| *** ddieterly is now known as ddieterly[away] | 19:38 | |
| *** mylu has joined #openstack-infra | 19:38 | |
| *** fawadkhaliq has quit IRC | 19:38 | |
| *** sdake has quit IRC | 19:38 | |
| *** fawadkhaliq has joined #openstack-infra | 19:38 | |
| *** ddieterly[away] is now known as ddieterly | 19:42 | |
| *** salv-orlando has joined #openstack-infra | 19:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do not install php packages https://review.openstack.org/298407 | 19:46 |
| *** weshay_ has joined #openstack-infra | 19:46 | |
| *** Sukhdev has quit IRC | 19:47 | |
| *** baoli has joined #openstack-infra | 19:47 | |
| *** baoli has quit IRC | 19:48 | |
| *** baoli has joined #openstack-infra | 19:49 | |
| *** IlyaG has joined #openstack-infra | 19:49 | |
| *** yamahata has quit IRC | 19:50 | |
| *** weshay has quit IRC | 19:50 | |
| *** BigWillie has quit IRC | 19:50 | |
| *** ZZelle has joined #openstack-infra | 19:51 | |
| *** salv-orlando has quit IRC | 19:52 | |
| *** Rockyg has quit IRC | 19:53 | |
| *** tlbr has quit IRC | 19:53 | |
| *** ajmiller has quit IRC | 19:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Revert "Add script to facilitate project renaming in project-config repo" https://review.openstack.org/298410 | 19:55 |
| *** nadya has joined #openstack-infra | 19:55 | |
| pleia2 | fungi: thanks for the heads up with the login issue on the i18n list, though I haven't figured out what's going on yet | 19:55 |
| pleia2 | the user migration from translate-dev was nothing special | 19:56 |
| *** ajmiller has joined #openstack-infra | 19:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Revert "Add script to facilitate project renaming in project-config repo" https://review.openstack.org/298410 | 19:56 |
| *** ArxCruz has joined #openstack-infra | 19:56 | |
| pleia2 | wondering if it may just be worth deleting their "old" account that we migrated and just having them set something up again, since they haven't used it and there isn't history with that account in production | 19:57 |
| fungi | that seems reasonable | 19:57 |
| *** knikolla_ has quit IRC | 19:57 | |
| pleia2 | I'll go ahead and do that, if we run into this again with someone else I'll dive in deeper | 19:59 |
| *** EricGonczer_ has joined #openstack-infra | 19:59 | |
| *** esker has quit IRC | 19:59 | |
| *** esikachev has joined #openstack-infra | 19:59 | |
| pleia2 | "This user cannot be removed from the system. You can deactivate the user instead" | 20:00 |
| pleia2 | yay | 20:00 |
| pleia2 | what? | 20:00 |
| *** ArxCruz has quit IRC | 20:00 | |
| Shrews | jeblair: zero hour? | 20:00 |
| * mordred stares intently at jeblair | 20:02 | |
| * mordred hands Shrews a beer | 20:02 | |
| jeblair | o/ | 20:02 |
| * mordred screams with excitement | 20:03 | |
| *** sdake has joined #openstack-infra | 20:03 | |
| *** sputnik13 has quit IRC | 20:03 | |
| * mordred hands Shrews a box of baby goats and one mostly not unhappy baby rhino | 20:03 | |
| *** ddieterly is now known as ddieterly[away] | 20:03 | |
| jeblair | mordred: please don't get escorted off the plane. at 35k. | 20:03 |
| mordred | jeblair: it's literally the only thing I strive for on a consistent basis | 20:03 |
| jeblair | nodepool==0.2.1.dev72 # git sha e1f4a12 | 20:04 |
| jeblair | that looks good. | 20:04 |
| *** Sukhdev has joined #openstack-infra | 20:04 | |
| mordred | jeblair: should we tag nodepool the pre-shade commit? | 20:05 |
| mordred | jeblair: I think someone in packaging world asked us to do that at some point | 20:05 |
| mordred | jeblair: (we can do it after we deploy of course) | 20:05 |
| *** esikachev has quit IRC | 20:05 | |
| mordred | or, wait - that was the commit from before we started depending on shade | 20:05 |
| mordred | nevermind | 20:06 |
| *** sdake_ has quit IRC | 20:06 | |
| jeblair | mordred: well, if we wanted to tag another, no objections here :) | 20:06 |
| *** tphummel has joined #openstack-infra | 20:06 | |
| mordred | meh. let's just roll | 20:06 |
| *** maishsk has quit IRC | 20:06 | |
| jeblair | many images are building; this will probably be disruptive for them. | 20:06 |
| mordred | jeblair: you don't need to restart the builders | 20:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: remove pypi publishing jobs for cloudkitty https://review.openstack.org/298307 | 20:06 |
| openstackgerrit | Mark T. Voelker proposed openstack-infra/irc-meetings: Update DefCore Meeting description https://review.openstack.org/298416 | 20:07 |
| jeblair | mordred: yeah, but their gearman jobs will be canceled | 20:07 |
| mordred | ah. good point | 20:07 |
| mordred | yah. that will likely be disruptive | 20:07 |
| jeblair | it should be handled now though, so ahead we go | 20:07 |
| jeblair | restarting now | 20:08 |
| pabelanger | \o/ | 20:08 |
| *** nadya_ has joined #openstack-infra | 20:08 | |
| jeblair | 2016-03-28 20:08:41,615 DEBUG nodepool.ProviderManager: Manager internap-nyj01 ran task FlavorGetExtraSpecs in 0.202835083008s | 20:08 |
| jeblair | 2016-03-28 20:08:41,616 DEBUG nodepool.ProviderManager: Manager internap-nyj01 running task FlavorGetExtraSpecs (queue: 6) | 20:08 |
| jeblair | many of these tasks | 20:08 |
| pabelanger | I can confirm that locally | 20:09 |
| *** maishsk has joined #openstack-infra | 20:09 | |
| *** kgiusti has left #openstack-infra | 20:09 | |
| *** kzaitsev_mb has quit IRC | 20:09 | |
| *** baoli has quit IRC | 20:10 | |
| *** nadya has quit IRC | 20:10 | |
| *** baoli has joined #openstack-infra | 20:10 | |
| *** akscram has quit IRC | 20:11 | |
| *** baoli has quit IRC | 20:11 | |
| * Shrews sips tea... awaits more info | 20:11 | |
| jeblair | i wonder if that just happens on startup? | 20:11 |
| *** baoli has joined #openstack-infra | 20:11 | |
| *** akscram has joined #openstack-infra | 20:11 | |
| jeblair | or if that's each time we create a server | 20:12 |
| jeblair | 2016-03-28 20:13:01,956 DEBUG nodepool.ProviderManager: Manager osic-cloud1 ran task ServerList in 0.681684970856s | 20:13 |
| jeblair | 2016-03-28 20:13:02,801 DEBUG nodepool.ProviderManager: Manager osic-cloud1 ran task ServerList in 0.757560014725s | 20:13 |
| jeblair | those ran very close together -- is caching working correctly there? | 20:14 |
| *** yamahata has joined #openstack-infra | 20:14 | |
| *** lakshmiS has quit IRC | 20:14 | |
| *** zeih has joined #openstack-infra | 20:15 | |
| mordred | jeblair: it's just at startup | 20:16 |
| mordred | oh - wait. I responded poorly | 20:16 |
| *** matrohon has joined #openstack-infra | 20:16 | |
| mordred | flavor getextraspecs will only run at startup, they're followup calls to the flavor list | 20:16 |
| mordred | the server list seems close together ... we might be missing a config option ... one sec | 20:17 |
| jeblair | ok, so that's just something we should have the provider manager de-dup. | 20:17 |
| *** BigWillie has joined #openstack-infra | 20:17 | |
| jeblair | but is not high-priority since it's only at startup (or reconfigure, probably) | 20:17 |
| *** dprince has quit IRC | 20:17 | |
| *** jamesmcarthur has joined #openstack-infra | 20:18 | |
| *** lakshmiS has joined #openstack-infra | 20:18 | |
| jeblair | http://paste.openstack.org/show/492157/ | 20:18 |
| Shrews | oh that's neat | 20:19 |
| *** piet has joined #openstack-infra | 20:20 | |
| *** zeih has quit IRC | 20:21 | |
| *** Sukhdev has quit IRC | 20:21 | |
| *** baoli has quit IRC | 20:22 | |
| jeblair | so active questions: 1) caching in serverlist; 2) caching in portlist; 3) the more-than-one-port exception | 20:22 |
| jeblair | (re 2: i'm seeing a lot of portlist tasks as well) | 20:23 |
| *** automagically has left #openstack-infra | 20:23 | |
| jeblair | i suspect the tripleo cloud is out of commission due to #3 | 20:23 |
| *** ZZelle has quit IRC | 20:23 | |
| *** lakshmiS has quit IRC | 20:23 | |
| *** ddieterly[away] is now known as ddieterly | 20:24 | |
| *** fawadkhaliq has quit IRC | 20:24 | |
| *** vgridnev has quit IRC | 20:25 | |
| Shrews | i'm trying to figure out where that error message from #3 is coming from | 20:25 |
| *** fawadkhaliq has joined #openstack-infra | 20:25 | |
| * morgan waves at Shrews. | 20:25 | |
| Shrews | ah, there it is | 20:25 |
| morgan | mostly cause Shrews is there. | 20:25 |
| morgan | jeblair: i might have a couple questions for you and/or mordred re nodepool and caching soon. | 20:26 |
| Shrews | morgan: howdy howdy | 20:26 |
| jeblair | Shrews: http://paste.openstack.org/show/492158/ now with more trackbacks :) | 20:26 |
| *** esikachev has joined #openstack-infra | 20:27 | |
| *** ybathia has joined #openstack-infra | 20:27 | |
| jeblair | i have received an out-of-band message from mordred saying clouds.yaml is missing cache.expiration.server: 5 | 20:27 |
| *** Sukhdev has joined #openstack-infra | 20:28 | |
| *** maishsk has quit IRC | 20:28 | |
| jeblair | i'll look into that while Shrews looks into the tripleo port thing | 20:28 |
| *** vgridnev has joined #openstack-infra | 20:29 | |
| *** vgridnev has quit IRC | 20:29 | |
| *** weshay_ has quit IRC | 20:32 | |
| clarkb | tripleo hosts have two networks each | 20:33 |
| clarkb | which means two ports | 20:33 |
| *** nadya_ has quit IRC | 20:33 | |
| anteaya | I'm still enjoying Easter Monday and good thing too as I probably wouldn't otherwise have witnessed the beginning of the spring flood in my basement | 20:33 |
| anteaya | I had hoped I had missed it this year, but nope | 20:34 |
| anteaya | based on the speed of the water coming in I expect I'll be up every hour all night long | 20:34 |
| anteaya | my attendance tomorrow is questionable | 20:34 |
| dims | anteaya ouch! | 20:34 |
| anteaya | yeah, I just have to stay ahead of the water and replace and dry towels all night | 20:35 |
| pc_m | For neutron, is there a way to tell what commit on master, corresponds to the commit on stable/mitaka? | 20:35 |
| *** Nikolay_St has joined #openstack-infra | 20:37 | |
| amuller | pc_m: gerrit search works with commit hashes | 20:37 |
| amuller | pc_m: then you click on the patch, then click on its change ID | 20:37 |
| amuller | pc_m: it will bring up the patch including any cherry picks | 20:37 |
| pc_m | amuller: not working with a specific commit/review | 20:38 |
| *** maishsk has joined #openstack-infra | 20:38 | |
| Shrews | clarkb: ahhh. so we need to pass which network to use for the fip. how would we determine that? | 20:38 |
| pc_m | amuller: We have a CI system that is failing with the latest on master. I think (confirming) it works with stable/mitaka neutron. | 20:39 |
| fungi | pc_m: that behavior depends on the backport keeping the change-id commit message header from the master commit | 20:39 |
| fungi | the searching on the change-id depends on it, i mean | 20:39 |
| pc_m | amuller: Want to select the commit on master that corresponds to stable/mitaka and do a git bisect to find the commit that introduced the failure. | 20:39 |
| clarkb | Shrews: one of the networks should be annotated public? I forget | 20:40 |
| pleia2 | anteaya: yikes, sorry to hear | 20:40 |
| *** jsavak has quit IRC | 20:40 | |
| fungi | pc_m: i see, you want to start from the point in master where syable/mitaka branched? | 20:40 |
| *** asettle has joined #openstack-infra | 20:40 | |
| *** Goneri has quit IRC | 20:40 | |
| amuller | pc_m: why don't you go to stable/mitaka branch, look at HEAD, write down the commit has, go back to master, then git bisect with that patch as the 'low' and master HEAD as the 'high' | 20:40 |
| amuller | commit hash* | 20:41 |
| pc_m | fungi: yes | 20:41 |
| fungi | amuller: stable/mitaka may have diverged after branching | 20:41 |
| amuller | fungi: true | 20:41 |
| pc_m | fungi: hmmm | 20:41 |
| pc_m | is there a way to find the point that stable/mitaka branched? | 20:42 |
| *** Sukhdev has quit IRC | 20:42 | |
| pc_m | on master | 20:42 |
| fungi | you're looking for the last common ancestor, i think | 20:42 |
| *** sdake_ has joined #openstack-infra | 20:43 | |
| fungi | i think git merge-base is the tool you probably want | 20:43 |
| *** sdake has quit IRC | 20:43 | |
| *** matrohon has quit IRC | 20:43 | |
| pc_m | fungi: Thanks will look at that. | 20:43 |
| fungi | though the idea of a "last common ancestor" is somewhat fuzzy since git is a dag | 20:44 |
| pc_m | fungi: So looks like git merge-base stable/mitaka...HEAD | 20:45 |
| fungi | `git merge-base origin/stable/mitaka origin/master` says dd4f1253c951d78a5b497680dfb31317ba469a58 which was the merge for "De-dup conntrack deletions before running them" | 20:46 |
| fungi | 12 days ago | 20:46 |
| amrith | fungi, ping ... | 20:46 |
| fungi | amrith: hey there | 20:46 |
| amrith | hiya | 20:46 |
| amrith | I got to the bottom of one py27 failure | 20:47 |
| fungi | details on the ironic unit test issues? | 20:47 |
| amrith | Routes released 2.3 today | 20:47 |
| amrith | and that broke some of trove's gate jobs | 20:47 |
| fungi | fun! | 20:47 |
| amrith | couple of issues there | 20:47 |
| amrith | getting a list | 20:47 |
| *** jamesmcarthur has quit IRC | 20:47 | |
| amrith | this is the trove issue | 20:47 |
| amrith | https://bugs.launchpad.net/trove/+bug/1563028 | 20:47 |
| openstack | Launchpad bug 1563028 in Trove "Routes version 2.3 broke the way we register routes" [Critical,In progress] - Assigned to Amrith (amrith) | 20:47 |
| amrith | it was exposed because of this (also trove issue) https://bugs.launchpad.net/trove/+bug/1563038 | 20:48 |
| openstack | Launchpad bug 1563038 in Trove "fix tox.ini in Trove to respect upper-constraints.txt" [High,New] - Assigned to Amrith (amrith) | 20:48 |
| fungi | oh lovely | 20:48 |
| amrith | this means that if someone bumps upper-constraints.txt and allows Routes 2.3, then some other projects may also trip over this | 20:48 |
| amrith | I sent this http://openstack.markmail.org/thread/by6prcthd546wjzg to the ML | 20:49 |
| amrith | as a heads-up | 20:49 |
| pc_m | fungi looks like one uses triple dot. | 20:49 |
| *** vgridnev has joined #openstack-infra | 20:49 | |
| amrith | So, if you see a failure that looks like: | 20:50 |
| amrith | File "/home/jenkins/workspace/gate-trove-python27-db/.tox/py27/local/lib/python2.7/site-packages/routes/mapper.py", line 168, in connect | 20:50 |
| amrith | 2016-03-28 18:58:54.390 | routename, path = args | 20:50 |
| amrith | 2016-03-28 18:58:54.390 | ValueError: need more than 1 value to unpack | 20:50 |
| amrith | then you want to check the bugs above :( | 20:50 |
| pc_m | fungi: git log stable/mitaka...HEAD | 20:50 |
| *** kzaitsev_mb has joined #openstack-infra | 20:50 | |
| fungi | amrith: thanks--will be on the lookout. we definitely don't seem to be capping routes in global-requirements.txt | 20:50 |
| amrith | fungi, it is capped in upper-constraints.txt | 20:50 |
| jeblair | clarkb, Shrews: we don't have the 'public' annotation set on either of the tripleo networks; however, i don't think that would solve this problem | 20:51 |
| amrith | but trove wasn't using that in tox.ini like (for example) neutron is | 20:51 |
| fungi | amrith: right, that's less of a cap and more of a "this is what we're testing with today" list | 20:51 |
| amrith | which is why we were the canary in the coal mine. | 20:51 |
| jeblair | clarkb, Shrews, mordred: i'm going to update https://etherpad.openstack.org/p/nodepool-shade with our current issues | 20:51 |
| *** mylu has quit IRC | 20:51 | |
| fungi | pc_m: intersting, the manpage i have for git-merge-base uses a space separator between ref names | 20:51 |
| fungi | pc_m: also i wouldn't trust stable/mitaka and HEAD to be in sync with origin/stable/mitaka and origin/master, better to be explicit | 20:52 |
| pc_m | fungi: yeha, I see someone saying you can do git log A...B | 20:52 |
| pc_m | fungi: Good IDea | 20:52 |
| fungi | you can, that gives you the list of commits from a to b | 20:52 |
| *** achanda has quit IRC | 20:52 | |
| fungi | though supposed to be .. not ... afaik | 20:52 |
| *** BigWillie has quit IRC | 20:52 | |
| *** baoli has joined #openstack-infra | 20:52 | |
| mordred | ola! | 20:52 |
| amrith | anyway fungi ... just wanted to give you a quick update on this. will follow up on the other issue from the email thread. toodle doo for now ... have to go head into traffic. | 20:52 |
| fungi | mordred: your plane landed in spain/. | 20:53 |
| fungi | ? | 20:53 |
| * mordred has found chair | 20:53 | |
| mordred | fungi: salt lake city | 20:53 |
| mordred | fungi: same thing really | 20:53 |
| fungi | mordred: funny how salt-lakers sound like they're speaking spanish | 20:53 |
| fungi | amrith: thanks again | 20:53 |
| pc_m | fungi thanks! | 20:54 |
| *** esikachev has quit IRC | 20:54 | |
| *** jsavak has joined #openstack-infra | 20:54 | |
| jeblair | mordred, Shrews: so if #3 is going to take some time, i reckon we ought to roll back. i can just check out the old commit on nodepool.o.o and reinstall and restart manually. | 20:55 |
| *** eil397 has joined #openstack-infra | 20:55 | |
| Shrews | jeblair: the thing is, if there is more than one network, we need to be telling it which network to use. i do not know how to determine that info | 20:55 |
| *** kzaitsev_mb has quit IRC | 20:55 | |
| mordred | Shrews: yah | 20:56 |
| mordred | so - my thought is that we use one of the "I have told you what my network is" settings | 20:56 |
| jeblair | could we re-use nodepool's "public" network attribute for this? it seems like the case to me that if we are attaching floating ips in nodepool, that's always going to be on the "public" network.... | 20:57 |
| *** MarkAtwood has quit IRC | 20:57 | |
| *** efoley has joined #openstack-infra | 20:57 | |
| mordred | that would actually break a different logic | 20:57 |
| Shrews | mordred: there might be a shade bug, too, where this is still going to happen b/c we do not pass the fixed address through to the _get_free_fixed_port() call | 20:58 |
| mordred | which is that for internap we specify the 'public' network since there are two and we don't know the difference | 20:58 |
| mordred | and shade says "do I have an ip on the network you've told me is public? if so, we're good. if not, we need a floating ip" | 20:58 |
| mordred | jeblair: so I actually think we should use the internal_network setting | 20:59 |
| mordred | which is how you tell shade which network is you "private" network | 20:59 |
| mordred | we could also add a whole new thing of "which network should your floating ips NAT to" | 20:59 |
| *** fawadkhaliq has quit IRC | 20:59 | |
| * mordred has no strong feelings and is open to discussing shed paint | 20:59 | |
| efoley | Hi, are there some issues with jenkins? I'm getting random failures on https://review.openstack.org/#/c/294969/1... different tests are failing on each jenkins run. | 21:00 |
| Shrews | jeblair: my recommendation is rollback | 21:00 |
| *** fawadkhaliq has joined #openstack-infra | 21:00 | |
| *** amuller has left #openstack-infra | 21:00 | |
| fungi | sounds like this is probably a shed which should be painted in comfort after rolling back, then trying again later, agreed | 21:00 |
| *** baoli has quit IRC | 21:00 | |
| mordred | yah. I think each of these are very understandable and tractable problems | 21:00 |
| fungi | rushing a new feature into shade so we can get tripleo back online seems like not a great set of constraints to impose on the dev process | 21:01 |
| mordred | bah. it's how all good software is writen - for some values of good, software, all and written | 21:01 |
| jeblair | mordred: how would using public break internap? internap will have the ip already so we still won't need to add a floating ip. | 21:01 |
| mordred | it'll break tripleo | 21:01 |
| jeblair | (i'm performing the rollback now) | 21:01 |
| mordred | just differently | 21:01 |
| mordred | because the do-i-need-a-floating-ip logic will stop at "yes, you have an ip from the network you've defined as public" | 21:02 |
| *** ericksonsantos has joined #openstack-infra | 21:02 | |
| *** jamesmcarthur has joined #openstack-infra | 21:03 | |
| *** |-paul-| has joined #openstack-infra | 21:03 | |
| jeblair | mordred: we have an ip on that network before creating a floating ip? | 21:03 |
| *** thorst has quit IRC | 21:03 | |
| *** IlyaG has quit IRC | 21:03 | |
| *** achanda has joined #openstack-infra | 21:04 | |
| jeblair | [oldpool restarted] | 21:04 |
| *** Sukhdev has joined #openstack-infra | 21:04 | |
| *** IlyaG has joined #openstack-infra | 21:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add server and port caching settings for nodepool https://review.openstack.org/298432 | 21:04 |
| jeblair | oh i was writing that | 21:05 |
| mordred | jeblair: yes. configuring a public network is the way we tell shade that even though it cannot infer which of the two networks is public and directly routable, this is the one | 21:05 |
| *** rockyg has joined #openstack-infra | 21:05 | |
| mordred | jeblair: oh, sorry | 21:05 |
| * jeblair updates etherpad to assign that task to mordred | 21:05 | |
| *** Sukhdev has quit IRC | 21:05 | |
| jeblair | mordred: what ip address do we have assigned to a tripleo node on the ostensibly public network before we get a floating ip? | 21:06 |
| *** baoli has joined #openstack-infra | 21:07 | |
| mordred | jeblair: let me start over - I can't come up with a way to answer that question that is not more confusing | 21:07 |
| jeblair | ok | 21:07 |
| *** Sukhdev has joined #openstack-infra | 21:08 | |
| *** thorst has joined #openstack-infra | 21:08 | |
| mordred | jeblair: configuring which network is public has a current meaning - that meaning is that a network labeled in config as public is a network from which one expects to get directly attached IP addresses | 21:08 |
| *** kzaitsev_mb has joined #openstack-infra | 21:08 | |
| mordred | jeblair: so if we configure a network as public, and shade finds that the server has an ip from that network, it will consider its job done, becaue the server has an ip from the public network | 21:09 |
| jeblair | mordred: that is not its meaning in nodepool | 21:09 |
| jeblair | mordred: in nodepool that means "this network is attached to the internet; therefore, the nodepool server should be able to talk to ip addresses on that network (howevery they are obtained)" | 21:10 |
| mordred | gotcha. so we are definitely talking about different things. that's good - it will make this better | 21:10 |
| mordred | I'm talking about the external_network setting in clouds.yaml, which has the above semantics | 21:11 |
| jeblair | mordred: okay. i did say "re-use nodepool's "public" network attribute for this" :) | 21:11 |
| jeblair | mordred: i still don't understand even so why we would have an ip on a network that requires floating ips before we have created a floating ip | 21:12 |
| clarkb | thsts a neutronism | 21:12 |
| clarkb | because nat | 21:12 |
| *** thorst has quit IRC | 21:12 | |
| _ody | Ok. Can't tell if anything strange is happening so I am just going to mention it...cloning anything from git.openstack.org is taking me 15 ore more and occasionally just fails. | 21:13 |
| *** aysyd has quit IRC | 21:13 | |
| bkero | _ody: are you comcasting? | 21:13 |
| _ody | bkero: No. | 21:13 |
| bkero | Ah, I'm also having connectivity issues to multiple places today | 21:14 |
| bkero | (on comcast) | 21:14 |
| _ody | Going out HE IPv4 right now. | 21:14 |
| pabelanger | _ody: bkero: Know issue today, see http://status.openstack.org//elastic-recheck/#1298006 | 21:14 |
| *** EricGonczer_ has quit IRC | 21:14 | |
| *** EricGonczer_ has joined #openstack-infra | 21:15 | |
| *** tqtran has quit IRC | 21:16 | |
| *** ldnunes has quit IRC | 21:16 | |
| *** fitoduarte has joined #openstack-infra | 21:16 | |
| bkero | well then | 21:17 |
| bkero | pabelanger: thanks | 21:17 |
| *** eharney has quit IRC | 21:17 | |
| *** tlbr has joined #openstack-infra | 21:18 | |
| _ody | bkero: Does that link also just hang or come back with no data for you? | 21:18 |
| *** julim has quit IRC | 21:18 | |
| fungi | _ody: pabelanger: as far as i know that should have been limited to rackspace's ord region | 21:19 |
| *** piet has quit IRC | 21:19 | |
| bkero | _ody: it sits there with no apparent update for a while, then shows graphs | 21:19 |
| *** aeng has joined #openstack-infra | 21:19 | |
| bkero | presumably the gifs are generated on pageload and they're not too quick | 21:19 |
| fungi | most recent hit on that query seems to be several hours ago | 21:20 |
| fungi | if you add the build_node column you'll see the last hit for those errors was on a worker which ran in rax-ord, before we turned out use of that region down to 0 | 21:21 |
| *** ociuhandu has quit IRC | 21:21 | |
| fungi | er, turned our use of that region down to 0 | 21:21 |
| *** prometheanfire has quit IRC | 21:21 | |
| *** prometheanfire has joined #openstack-infra | 21:22 | |
| dougwig | my nodepool suddenly decided it needed to start assigning floating ips on a network that doesn't support them. known issue? | 21:23 |
| *** _ari_ has quit IRC | 21:23 | |
| fungi | dougwig: did you restart after https://review.openstack.org/226751 merged? | 21:24 |
| fungi | dougwig: if so, you may want to roll that back. a revert is on the way | 21:24 |
| dougwig | fungi: yes, server just rebooted. | 21:24 |
| dougwig | fungi: ahh, ok, thanks. | 21:24 |
| * dougwig goes to disable puppet cron and rollback. | 21:25 | |
| *** thorst has joined #openstack-infra | 21:26 | |
| Shrews | dougwig: that's interesting. mordred, perhaps we should have a config option to turn off/on auto_ip? ^^^ | 21:27 |
| *** EricGonczer_ has quit IRC | 21:27 | |
| *** sdake_ is now known as sdake | 21:27 | |
| *** bpokorny has quit IRC | 21:27 | |
| efoley | Hi, are there some issues with jenkins? I'm getting random failures on https://review.openstack.org/#/c/294969/1... different tests are failing on each jenkins run. | 21:27 |
| *** salv-orlando has joined #openstack-infra | 21:27 | |
| dougwig | Shrews: used to be that marking the network 'public' punted that; did that change? | 21:28 |
| *** sridhar_ram1 is now known as sridhar_ram | 21:29 | |
| *** thorst has quit IRC | 21:30 | |
| *** bpokorny has joined #openstack-infra | 21:30 | |
| fungi | efoley: helps to characterize how they're failing and look for commonality | 21:31 |
| *** jamesmcarthur has quit IRC | 21:31 | |
| Shrews | dougwig: well, the meaning of that word was just discussed in scrollback. i think we need some shade changes before trying this again | 21:31 |
| *** jamesmcarthur has joined #openstack-infra | 21:31 | |
| efoley | fungi: It's the same testcase that fails, but different tests each time | 21:32 |
| dougwig | Shrews: ok. fwiw, my use case is a provider net that's directly accessible, which do not support floating ips (which I personally think is good, because using floating IPs in the common case is maddening.) | 21:32 |
| fungi | efoley: strangely, the change is modifying a devstack plugin, yet its the unit tests failing | 21:33 |
| *** jsavak has quit IRC | 21:33 | |
| *** ybathia has quit IRC | 21:33 | |
| efoley | fungi: Worked perfectly before, and now... random failures! That's what's weird about it | 21:33 |
| _ody | fungi: All my clones from git.openstack.org are still taking forever and/or failing so there might me something else going on. I've switched to sourcing github and everthing is fine to do local testing again. | 21:33 |
| dougwig | fungi: many thanks, humming along now. | 21:34 |
| pabelanger | _ody: fungi: which protocol are you using? http(s) / git ? | 21:34 |
| fungi | _ody: could it be ipv6 connectivity that's your problem, or are you on an ipv4-only connection? | 21:34 |
| *** piet has joined #openstack-infra | 21:34 | |
| *** jamesmcarthur has quit IRC | 21:34 | |
| fungi | _ody: any silly corporate proxies to contend with? | 21:35 |
| *** salv-orlando has quit IRC | 21:35 | |
| *** ociuhandu has joined #openstack-infra | 21:35 | |
| *** regXboi has quit IRC | 21:35 | |
| *** doug-fish has quit IRC | 21:36 | |
| *** Sukhdev has quit IRC | 21:36 | |
| *** achanda has quit IRC | 21:37 | |
| *** Sukhdev has joined #openstack-infra | 21:37 | |
| *** salv-orlando has joined #openstack-infra | 21:38 | |
| *** tqtran has joined #openstack-infra | 21:39 | |
| _ody | pabelanger: https | 21:40 |
| _ody | fungi: Was on ipv4 only, inside and outside an openstack instance. No proxies. | 21:40 |
| zaro | fungi: git-review wip on push feature is just a generic way to support Gerrit's feature to set labels on push. that's a feature in gerrit | 21:41 |
| *** salv-orlando has quit IRC | 21:42 | |
| *** esikachev has joined #openstack-infra | 21:42 | |
| *** tiswanso has quit IRC | 21:43 | |
| *** zeih has joined #openstack-infra | 21:43 | |
| *** ddieterly is now known as ddieterly[away] | 21:44 | |
| pabelanger | _ody: So, as a data point, we've seen poor https performance until https with centos6: https://bugzilla.redhat.com/show_bug.cgi?id=1237395 | 21:44 |
| openstack | bugzilla.redhat.com bug 1237395 in git "extremley slow https clone performance" [Unspecified,Assigned] - Assigned to pstodulk | 21:44 |
| fungi | zaro: yep, i haven't looked at it in a while. it doesn't actually say "work in progress" or hard-code any specific labels/values anywhere now, i guess? | 21:44 |
| *** salv-orlando has joined #openstack-infra | 21:45 | |
| *** cfarquhar has quit IRC | 21:45 | |
| fungi | zaro: better not to call it a "wip on push" feature in that case, just "vote on push" or something | 21:45 |
| zaro | fungi: yeah, it's just setting vote on existing labels in gerrit. | 21:45 |
| zaro | commit msg is probably a little misleading | 21:46 |
| fungi | we happen to use a custom label+value to indicate work in progress, but i don't want that encoded in git-review anywhere | 21:46 |
| zaro | fungi: +1 there, and it's not in that change. | 21:47 |
| *** cfarquhar has joined #openstack-infra | 21:48 | |
| *** cfarquhar has quit IRC | 21:48 | |
| *** cfarquhar has joined #openstack-infra | 21:48 | |
| zaro | yep, even the example uses Code-Review label. only mention of wip is in the commit message | 21:48 |
| fungi | that said, let's not land more feature-like changes until after we tag a release, and just flush lingering bug fixes and test improvements first | 21:48 |
| *** zeih has quit IRC | 21:49 | |
| jeblair | yeah, i think it might pass the sniff test since it's using a generic feature in gerrit, and it's not available some other way (if you want to set a label on push, you would need git-review to support this). | 21:49 |
| fungi | efoley: i was going to try to recreate that failure on my system, but seems to need libvirt to be able to run unit tests | 21:49 |
| jeblair | but also, not urgent before tagging :) | 21:49 |
| *** |-paul-| has quit IRC | 21:50 | |
| *** ybathia has joined #openstack-infra | 21:50 | |
| jeblair | zaro: do you know if there is support for a similar "post this message on push" feature? | 21:50 |
| jeblair | zaro: i'd like to be able to run: git review -m 'This patchset is just a rebase on current master.' | 21:50 |
| zaro | jeblair: hmm, not that i have seen. let me take a look again. | 21:51 |
| *** smarcet has quit IRC | 21:51 | |
| fungi | efoley: you're running it like `tox -r -e py27` right? to make sure you don't have any old pyc cruft in a preexisting tox virtualenv? | 21:51 |
| *** ddieterly[away] is now known as ddieterly | 21:51 | |
| efoley | fungi: I recreate the environment each time, just to be safe | 21:52 |
| fungi | _ody: i wonder if we have some sort of blackhole situation going on somewhere between you and the git servers, maybe path mtu discovery issues or something which aren't affecting your path to github | 21:52 |
| *** yamahata has quit IRC | 21:53 | |
| *** jamesmcarthur has joined #openstack-infra | 21:53 | |
| *** yamahata has joined #openstack-infra | 21:54 | |
| *** mriedem has quit IRC | 21:55 | |
| fungi | _ody: anyway, i just tested a fresh clone of nova from my home network halfway across the usa and it took less than 60 seconds to complete, but traceroute also says my home isp has a peering relationship with rackspace in equinix near the same data center where the git servers are | 21:55 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: WIP increase TEMPEST_OS_TEST_TIMEOUT https://review.openstack.org/298446 | 21:56 |
| *** annegentle has quit IRC | 21:56 | |
| fungi | _ody: scratch that, i forgot to use traceroute6. i was actually cloning over my gre tunnel from hurricane electric | 21:56 |
| fungi | also seems to peer in the same equinix facility though | 21:57 |
| *** doug-fish has joined #openstack-infra | 21:57 | |
| _ody | I'll look again later, possibly from home after I get done at the gym. For now I am making progress just using github so I am happy with that. | 21:58 |
| *** thorst has joined #openstack-infra | 21:58 | |
| fungi | _ody: more details will be appreciated when you have any. we definitely want this reachable from everywhere so appreciate the heads up that you're having issues with it | 21:58 |
| *** burgerk_ has joined #openstack-infra | 21:58 | |
| *** annegentle has joined #openstack-infra | 21:59 | |
| fungi | AJaeger: pabelanger: nova other-requirements.txt example seems to be passing all jobs on the latest patchset https://review.openstack.org/200672 | 22:01 |
| *** burgerk has quit IRC | 22:02 | |
| *** Sukhdev has quit IRC | 22:03 | |
| fungi | efoley: i'm manually attempting to recreate the failure you're seeing on a held ubuntu trusty node now | 22:04 |
| pabelanger | fungi: excellent | 22:04 |
| efoley | fungi: Awesome! | 22:04 |
| *** achanda has joined #openstack-infra | 22:04 | |
| *** fitoduarte has quit IRC | 22:05 | |
| zaro | jeblair: unfortunately i don't see that feature. have added a feature request, https://code.google.com/p/gerrit/issues/detail?id=4015 which i should be able to work on sometime soon. | 22:05 |
| *** camunoz has joined #openstack-infra | 22:07 | |
| jeblair | zaro: neat, that would be cool :) | 22:08 |
| *** burgerk_ has quit IRC | 22:08 | |
| *** esikachev has quit IRC | 22:09 | |
| *** dizquierdo has quit IRC | 22:09 | |
| *** IlyaG has quit IRC | 22:10 | |
| *** apoorvad has quit IRC | 22:11 | |
| *** tonytan4ever has quit IRC | 22:14 | |
| fungi | efoley: manual run failed collectd_ceilometer.tests.test_config.TestConfig.test_missing_value raising AssertionError: error(u'Configuration parameter %s not set.', u'OS_AUTH_URL') call not found | 22:14 |
| fungi | efoley: is that one of the failures you've been encountering? | 22:14 |
| *** dimtruck is now known as zz_dimtruck | 22:15 | |
| fungi | efoley: ran again and got an AssertionError: 10 != 1 in collectd_ceilometer.tests.test_config.TestConfig.test_invalid_value which does look like one of the errors i saw in your logs | 22:16 |
| fungi | efoley: this is with the tip of the collectd-ceilometer-plugin master branch, so definitely looks unrelated to your patch | 22:16 |
| *** cloudtrainme has quit IRC | 22:16 | |
| *** LaT0rtue has quit IRC | 22:16 | |
| *** tpsilva has quit IRC | 22:17 | |
| *** IlyaG has joined #openstack-infra | 22:17 | |
| zaro | fungi: when you ran flush-cache to attempt to fix ec2-api project do you remember if you flush all caches? | 22:18 |
| fungi | zaro: i used "gerrit flush-caches" | 22:19 |
| openstackgerrit | Nikhil Komawar proposed openstack-infra/irc-meetings: Change Quota WG time https://review.openstack.org/298454 | 22:19 |
| *** kzaitsev_mb has quit IRC | 22:20 | |
| fungi | zaro: i did not add --all | 22:20 |
| *** jamesmcarthur has quit IRC | 22:21 | |
| *** amrith is now known as _amrith_ | 22:21 | |
| *** thorst has quit IRC | 22:21 | |
| fungi | zaro: though https://review.openstack.org/Documentation/cmd-flush-caches.html seems to imply they're synonymous | 22:21 |
| *** ddieterly is now known as ddieterly[away] | 22:23 | |
| fungi | efoley: i ran again and this time it passed. i suspect you have ordering/isolation issues between your tests and depending on the PYTHONHASHSEED value it gets exposed in different ways (or not at all) | 22:23 |
| *** thorst has joined #openstack-infra | 22:24 | |
| zaro | fungi: yep, code also says default is --all | 22:25 |
| *** Sukhdev has joined #openstack-infra | 22:25 | |
| efoley | fungi: thanks. How do I go about fixing this? | 22:25 |
| fungi | efoley: tox sets a random hash seed, so my guess is either you're running this with fewer worker threads or getting lucky and hitting an ordering where it's happening in just the right order to be able to succeed | 22:26 |
| fungi | efoley: does the "worker balance" output at the end of your local runs indicate 4 worker threads? | 22:26 |
| *** IlyaG has quit IRC | 22:27 | |
| zaro | fungi: the error doesn't seem like its the same as the diff timeout issue. i'm guessing it's probably a timing issue that caused this problem way before we saw the MissingObjectException. | 22:27 |
| *** ddieterly[away] is now known as ddieterly | 22:28 | |
| efoley | fungi: It uses 24 workers. So this is a paralellism issue | 22:28 |
| zaro | i'm just gonna post the error on Gerrit ML to see if anybody else has seen this issue. | 22:28 |
| *** ddieterly has quit IRC | 22:29 | |
| *** rockyg has quit IRC | 22:29 | |
| *** markvoelker has quit IRC | 22:29 | |
| fungi | efoley: yes, i'm betting that almost all of the tests are running in a separate worker thread for you then, as you have 24 threads for 25 tests as opposed to in the gate we only run 4 threads in parallel and so some of the jobs are getting serialized | 22:30 |
| fungi | zaro: thanks for looking | 22:30 |
| fungi | zaro: i have a feeling it's due to some sort of momentary race between a change to the repo on disk and gerrit reading something from it, so may be very hard to reproduce | 22:31 |
| *** harlowja has joined #openstack-infra | 22:32 | |
| *** harlowja has quit IRC | 22:32 | |
| *** thorst has quit IRC | 22:32 | |
| fungi | efoley: er, i mean some of the tests are getting serialized | 22:33 |
| *** harlowja has joined #openstack-infra | 22:33 | |
| *** doug-fish has quit IRC | 22:37 | |
| *** gordc has quit IRC | 22:43 | |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Cap Routes < 2.3 https://review.openstack.org/298458 | 22:43 |
| *** zeih has joined #openstack-infra | 22:44 | |
| *** sputnik13 has joined #openstack-infra | 22:48 | |
| *** fawadkhaliq has quit IRC | 22:48 | |
| *** fawadkhaliq has joined #openstack-infra | 22:49 | |
| *** zeih has quit IRC | 22:49 | |
| *** esker has joined #openstack-infra | 22:49 | |
| *** piet has quit IRC | 22:50 | |
| *** maishsk has quit IRC | 22:51 | |
| *** sdague has quit IRC | 22:53 | |
| *** harlowja has quit IRC | 22:54 | |
| *** harlowja has joined #openstack-infra | 22:54 | |
| *** tongli has quit IRC | 22:56 | |
| *** david-lyle_ is now known as david-lyle | 22:57 | |
| *** eil397 has left #openstack-infra | 22:58 | |
| *** IlyaG has joined #openstack-infra | 22:58 | |
| *** jamesmcarthur has joined #openstack-infra | 22:58 | |
| *** maishsk has joined #openstack-infra | 22:59 | |
| *** annegentle has quit IRC | 22:59 | |
| ianw | 2016-03-28 21:13:15.430 | 2016-03-26 16:59 <- timestamp | image build date. presumably images aren't rolling out to rax? mostly i'm wondering if running on performance node | 22:59 |
| ianw | hmm, actually we cat cpuinfo somewhere | 22:59 |
| *** rbrndt_ has quit IRC | 22:59 | |
| *** akshai has quit IRC | 23:00 | |
| *** bpokorny_ has joined #openstack-infra | 23:02 | |
| craige | o/ | 23:02 |
| *** bpokorny_ has quit IRC | 23:02 | |
| *** esker has quit IRC | 23:02 | |
| *** bpokorny_ has joined #openstack-infra | 23:03 | |
| *** esker has joined #openstack-infra | 23:03 | |
| *** ybathia has quit IRC | 23:04 | |
| *** efoley_ has joined #openstack-infra | 23:04 | |
| *** bpokorny has quit IRC | 23:04 | |
| *** tphummel has quit IRC | 23:05 | |
| *** jamesmcarthur has quit IRC | 23:05 | |
| *** tphummel has joined #openstack-infra | 23:05 | |
| *** tonytan4ever has joined #openstack-infra | 23:05 | |
| *** efoley has quit IRC | 23:05 | |
| jroll | are there docs on building a bare-trusty image (nearly) identical to infra's? | 23:07 |
| *** vgridnev has quit IRC | 23:07 | |
| jroll | pointer to the code is an acceptable answer here, to | 23:07 |
| jroll | too* | 23:07 |
| *** LaT0rtue has joined #openstack-infra | 23:08 | |
| *** hichihara has joined #openstack-infra | 23:08 | |
| *** mriedem has joined #openstack-infra | 23:08 | |
| clarkb | jroll: we stopped using bare-trusty so no :) the project-config repo has a tools/build-image script that needs updating to use the ubuntu minimal elements | 23:08 |
| *** sputnik13 has quit IRC | 23:08 | |
| jroll | clarkb: oh, what's the new name? | 23:09 |
| clarkb | jroll: if you s/ubuntu/ubuntu-minimal/ and add in the simple-init element that will get you ehat we use | 23:09 |
| clarkb | jroll: just 'ubuntu-trusty' | 23:09 |
| clarkb | common image for all your ubuntu needs | 23:09 |
| fungi | jroll: yeah, we're now running devstack-based jobs and all other jobs on the same images rather than having one image for devstack and another for everything else | 23:10 |
| fungi | makes things _much_ simpler | 23:10 |
| jroll | clarkb: so tools/build-image, DISTRO=ubuntu-minimal, add simple-init, add whatever element drops in ssh keys, upload to my provider, done? | 23:10 |
| fungi | just took a long time to lay all the groundwork | 23:10 |
| *** esker has quit IRC | 23:10 | |
| jroll | fungi: I'm well aware, been debugging them on and off since the switch, just didn't know the name :) | 23:10 |
| clarkb | jroll: yup be warned its a pretty big image | 23:10 |
| *** bpokorny_ has quit IRC | 23:11 | |
| ianw | hmm, tempest runs +300 more jobs on centos7-minimal images than !minimal images | 23:11 |
| *** maishsk has quit IRC | 23:11 | |
| fungi | ianw: s/jobs/tests/? if so, awesome? | 23:11 |
| *** yuanying has joined #openstack-infra | 23:11 | |
| *** apoorvad has joined #openstack-infra | 23:11 | |
| jroll | clarkb: thanks man | 23:11 |
| ianw | sorry, yeah, tests. not sure if it's a bug or a feature yet | 23:11 |
| fungi | ianw: hopefully it's not missing something that's supposed to be making it skip tests on that platform | 23:12 |
| ianw | yeah, or we've inadvertently enabled something | 23:12 |
| *** ybathia has joined #openstack-infra | 23:13 | |
| *** apoorvad has quit IRC | 23:13 | |
| *** apoorvad has joined #openstack-infra | 23:13 | |
| ianw | but fedora 23 seems to run about 50 minutes on the performance nodes, which is about exactly what i would expect ... about 5 minutes longer than ubuntu & can be attributed to pip cache | 23:14 |
| *** esker has joined #openstack-infra | 23:16 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:17 | |
| ianw | oh, it's neutron | 23:18 |
| *** annegentle has joined #openstack-infra | 23:19 | |
| *** annegentle has quit IRC | 23:19 | |
| fungi | oh? what about neutron specifically? | 23:19 |
| clarkb | neutron changes the test count | 23:20 |
| clarkb | and runtime they both go up | 23:20 |
| *** thorst has joined #openstack-infra | 23:20 | |
| fungi | oh, this was comparing tempest neutron and !neutron jobs? | 23:20 |
| fungi | if so, right not a surprise at all | 23:20 |
| *** esker has quit IRC | 23:20 | |
| *** maishsk has joined #openstack-infra | 23:20 | |
| *** salv-orlando has quit IRC | 23:21 | |
| *** kzaitsev_mb has quit IRC | 23:22 | |
| *** amitgandhinz has quit IRC | 23:22 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Update build-image.sh for new images, add EXTRA_ELEMENTS https://review.openstack.org/298463 | 23:22 |
| jroll | clarkb: hope that's helpful ^ | 23:22 |
| *** asettle has quit IRC | 23:24 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add E-R query for bug 1312016 https://review.openstack.org/297914 | 23:24 |
| openstack | bug 1312016 in neutron "nova libvirtError: Unable to add bridge brqxxx-xx port tapxxx-xx: Device or resource busy" [Medium,Fix released] https://launchpad.net/bugs/1312016 - Assigned to Andreas Scheuring (andreas-scheuring) | 23:24 |
| *** harlowja has quit IRC | 23:25 | |
| *** harlowja has joined #openstack-infra | 23:25 | |
| pvaneck | Hello infra/puppet core. Looking to get https://review.openstack.org/#/c/281737/ merged. It has two +2s, just need the Workflow+1 :) | 23:27 |
| *** Qiming has joined #openstack-infra | 23:29 | |
| *** mylu has joined #openstack-infra | 23:31 | |
| _ody | fungi: Did a quick mtr of my connection to git.openstack.org, http://paste.openstack.org/show/492177/. | 23:35 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add E-R query for bug 1312016 https://review.openstack.org/297914 | 23:35 |
| openstack | bug 1312016 in neutron "nova libvirtError: Unable to add bridge brqxxx-xx port tapxxx-xx: Device or resource busy" [Medium,Fix released] https://launchpad.net/bugs/1312016 - Assigned to Andreas Scheuring (andreas-scheuring) | 23:35 |
| fungi | pvaneck: looks like jeblair was holding off approving that waiting for your response, so he may want to approve it | 23:35 |
| _ody | Interesting though, items with packet loss there doesn't have packet loss when going over a comcast link. | 23:35 |
| *** sputnik13 has joined #openstack-infra | 23:35 | |
| *** esker has joined #openstack-infra | 23:36 | |
| jroll | just realized most of this image will likely be the plethora of openstack repos it is caching, heh | 23:36 |
| fungi | jroll: yes, that is precisely most of it | 23:36 |
| * jroll leaves that out | 23:37 | |
| pvaneck | fungi: yea, i'll wait for him | 23:37 |
| ianw | hmm, no smoking gun -> https://etherpad.openstack.org/p/centos7-minimal-runtimes . will get some more data points | 23:37 |
| fungi | _ody: the "loss" on the interim hops looks like it's probably just throttling icmp responses or filtering the probes rather than responding when their ttl is exceeded. since there's no packet loss to git.o.o according to mtr that's probably not indicative of any issue | 23:39 |
| *** gildub has joined #openstack-infra | 23:39 | |
| pleia2 | fungi: it only took all day (ok, I didn't work on it all day) but it turns out their openstackid URL changed, current one in zanata is giving a 404, once I have the right one I should be able to fix it up \o/ | 23:40 |
| fungi | pleia2: what did it change from/to? | 23:40 |
| *** achanda has quit IRC | 23:41 | |
| fungi | rather, what is zanata trying and failing to use? | 23:41 |
| *** LaT0rtue has quit IRC | 23:41 | |
| pleia2 | fungi: following up with the user to figure out what their current real one is | 23:41 |
| fungi | oh! | 23:41 |
| fungi | got it | 23:41 |
| pleia2 | trying to use https://openstackid.org/nikita.burtsev which doesn't exist for some reason | 23:41 |
| * pleia2 shrugs | 23:41 | |
| fungi | so not (necessarily) an issue with openstackid.org | 23:41 |
| pleia2 | no, maybe they changed their name at some point | 23:41 |
| pleia2 | so it just doesn't match anymore | 23:42 |
| fungi | yeah, that's been a concern of mine ever since i first saw that openstackid was relying on human names for the openids | 23:42 |
| fungi | names change often, you need something immutable | 23:42 |
| *** knikolla has quit IRC | 23:44 | |
| *** zz_dimtruck is now known as dimtruck | 23:44 | |
| pleia2 | especially if you're like me (I am better at computers than people) | 23:45 |
| fungi | _ody: i don't seem to be able to reach pdx-626c2-core-vlan16-corp6.ops.puppetlabs.net so can't easily construct the return route to your system from the git server. is there an ip address on the machine you're testing from, to which i can try to traceroute? | 23:45 |
| *** NobodyCam has quit IRC | 23:46 | |
| fungi | though i do hope more of your systems are named after bleach characters | 23:47 |
| _ody | heh. No just my laptop | 23:47 |
| _ody | Our outbound NAT IP is 192.69.65.12. | 23:47 |
| *** tonytan4ever has quit IRC | 23:47 | |
| fungi | cool, i'll trace back to that | 23:48 |
| fungi | _ody: surprisingly, the return route goes through comcast | 23:48 |
| fungi | so this may in fact be related to what bkero mentioned earlier | 23:49 |
| *** sputnik13 has quit IRC | 23:49 | |
| fungi | your packets are reaching rackspace via hurricane electric, and then rackspace is choosing to send the replies back via comcast | 23:49 |
| _ody | heh. Good to know. | 23:50 |
| *** nelsnelson has joined #openstack-infra | 23:50 | |
| *** fawadkhaliq has quit IRC | 23:50 | |
| fungi | i mean, route selection changes minute to minute so who knows what it'll be tomorrow. bgp is a fickle mistress | 23:50 |
| *** fawadkhaliq has joined #openstack-infra | 23:50 | |
| _ody | We went ahead and raised a support request with rax. | 23:51 |
| *** mylu has quit IRC | 23:51 | |
| *** nelsnels_ has quit IRC | 23:51 | |
| fungi | but at least some of the internet thinks your network is best reached through comcast | 23:51 |
| *** davideagnello has quit IRC | 23:51 | |
| fungi | and spectrumnet | 23:51 |
| fungi | and opusnet | 23:51 |
| *** fawadkhaliq has quit IRC | 23:52 | |
| *** fawadkhaliq has joined #openstack-infra | 23:52 | |
| fungi | fwiw my traceroute from home to you (charter) takes hurricane electric | 23:53 |
| _ody | opusnet is our "other" provider which we do advertise on so that seem normal for part of the internet. | 23:54 |
| *** LaT0rtue has joined #openstack-infra | 23:54 | |
| *** EricGonczer_ has joined #openstack-infra | 23:54 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:58 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Change Quota WG time https://review.openstack.org/298454 | 23:58 |
| *** ajmiller has quit IRC | 23:58 | |
| *** IlyaG has quit IRC | 23:59 | |
| *** fawadkhaliq has quit IRC | 23:59 | |
| *** fawadkhaliq has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!