| *** shashank_hegde has quit IRC | 00:01 | |
| *** piet has quit IRC | 00:03 | |
| *** salv-orlando has joined #openstack-infra | 00:06 | |
| *** erlon has quit IRC | 00:06 | |
| *** sputnik13 has joined #openstack-infra | 00:07 | |
| *** piet has joined #openstack-infra | 00:08 | |
| *** dgonzalez has quit IRC | 00:08 | |
| openstackgerrit | Motohiro/Yuanying Otsuka proposed openstack-infra/project-config: Add Magnum functional test for k8s ironic https://review.openstack.org/337386 | 00:10 |
|---|---|---|
| *** salv-orlando has quit IRC | 00:13 | |
| *** dgonzalez has joined #openstack-infra | 00:15 | |
| *** gouthamr has quit IRC | 00:15 | |
| *** kus has joined #openstack-infra | 00:17 | |
| *** kus has quit IRC | 00:17 | |
| *** piet has quit IRC | 00:21 | |
| *** Goneri has quit IRC | 00:21 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel and ORM https://review.openstack.org/322307 | 00:24 |
| *** piet has joined #openstack-infra | 00:25 | |
| *** yamahata has quit IRC | 00:28 | |
| *** lvdongbing has joined #openstack-infra | 00:29 | |
| *** dimtruck is now known as zz_dimtruck | 00:34 | |
| *** karthikp_ has joined #openstack-infra | 00:39 | |
| *** karthikp_ has quit IRC | 00:47 | |
| *** karthikp_ has joined #openstack-infra | 00:51 | |
| *** sarob has quit IRC | 00:53 | |
| *** piet has quit IRC | 00:59 | |
| *** piet has joined #openstack-infra | 00:59 | |
| *** karthikp_ has quit IRC | 01:02 | |
| *** rook- has quit IRC | 01:04 | |
| *** jamielennox is now known as jamielennox|away | 01:09 | |
| *** karthikp_ has joined #openstack-infra | 01:09 | |
| *** salv-orlando has joined #openstack-infra | 01:10 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 01:12 |
| *** piet has quit IRC | 01:14 | |
| *** piet has joined #openstack-infra | 01:14 | |
| *** karthikp_ has quit IRC | 01:17 | |
| *** karthikp_ has joined #openstack-infra | 01:18 | |
| *** zhurong has joined #openstack-infra | 01:22 | |
| *** amotoki has quit IRC | 01:28 | |
| *** salv-orlando has quit IRC | 01:28 | |
| *** piet has quit IRC | 01:29 | |
| *** karthikp_ has quit IRC | 01:33 | |
| *** sputnik13 has quit IRC | 01:33 | |
| *** mixos has joined #openstack-infra | 01:33 | |
| *** karthikp_ has joined #openstack-infra | 01:42 | |
| *** jamielennox|away is now known as jamielennox | 01:43 | |
| *** karthikp_ has quit IRC | 01:45 | |
| *** amotoki has joined #openstack-infra | 01:46 | |
| *** karthikp_ has joined #openstack-infra | 01:46 | |
| *** sputnik13 has joined #openstack-infra | 01:47 | |
| *** amotoki has quit IRC | 01:51 | |
| *** yanyanhu has joined #openstack-infra | 01:51 | |
| *** jamielennox is now known as jamielennox|away | 01:53 | |
| *** Daisy has joined #openstack-infra | 01:55 | |
| *** salv-orlando has joined #openstack-infra | 01:56 | |
| *** rfolco_ has quit IRC | 01:57 | |
| *** amotoki has joined #openstack-infra | 01:58 | |
| *** karthik__ has joined #openstack-infra | 02:02 | |
| *** karthik__ has quit IRC | 02:03 | |
| *** amotoki has quit IRC | 02:03 | |
| *** karthikp_ has quit IRC | 02:03 | |
| *** karthikp_ has joined #openstack-infra | 02:03 | |
| *** Daisy has quit IRC | 02:06 | |
| *** Daisy has joined #openstack-infra | 02:06 | |
| *** Daisy has quit IRC | 02:11 | |
| *** karthikp_ has quit IRC | 02:11 | |
| *** ramishra has joined #openstack-infra | 02:12 | |
| *** asettle has joined #openstack-infra | 02:13 | |
| *** salv-orlando has quit IRC | 02:13 | |
| *** zz_dimtruck is now known as dimtruck | 02:14 | |
| *** karthikp_ has joined #openstack-infra | 02:15 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: run git gc instead of repack for git repos https://review.openstack.org/334714 | 02:15 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Remove duplicate cron job https://review.openstack.org/334715 | 02:17 |
| *** asettle has quit IRC | 02:19 | |
| *** Guest55765 has quit IRC | 02:19 | |
| *** furlongm has joined #openstack-infra | 02:20 | |
| *** caowei has joined #openstack-infra | 02:24 | |
| *** Daisy has joined #openstack-infra | 02:27 | |
| *** AL34N1X has joined #openstack-infra | 02:28 | |
| *** sputnik13 has quit IRC | 02:29 | |
| *** karthikp_ has quit IRC | 02:29 | |
| *** karthikp_ has joined #openstack-infra | 02:30 | |
| *** hongbin has joined #openstack-infra | 02:30 | |
| *** AL34N1X has quit IRC | 02:34 | |
| *** amotoki has joined #openstack-infra | 02:35 | |
| *** dimtruck is now known as zz_dimtruck | 02:36 | |
| *** salv-orlando has joined #openstack-infra | 02:40 | |
| *** jamielennox|away is now known as jamielennox | 02:43 | |
| *** AJaeger has quit IRC | 02:43 | |
| *** AJaeger has joined #openstack-infra | 02:44 | |
| *** karthik__ has joined #openstack-infra | 02:45 | |
| *** mixos has quit IRC | 02:46 | |
| *** PollyZ has joined #openstack-infra | 02:46 | |
| *** karthikp_ has quit IRC | 02:48 | |
| *** Daisy has quit IRC | 02:54 | |
| *** DaiDV has joined #openstack-infra | 02:54 | |
| *** Daisy has joined #openstack-infra | 02:54 | |
| *** DaiDV has quit IRC | 02:55 | |
| *** ramishra has quit IRC | 02:56 | |
| *** salv-orlando has quit IRC | 02:58 | |
| *** Daisy_ has joined #openstack-infra | 02:59 | |
| *** coolsvap has joined #openstack-infra | 02:59 | |
| *** karthikp_ has joined #openstack-infra | 03:00 | |
| *** sputnik13 has joined #openstack-infra | 03:01 | |
| *** Daisy has quit IRC | 03:03 | |
| *** karthik__ has quit IRC | 03:03 | |
| *** ramishra has joined #openstack-infra | 03:04 | |
| *** rcernin has joined #openstack-infra | 03:09 | |
| *** piet_ has joined #openstack-infra | 03:10 | |
| *** roxanagh_ has joined #openstack-infra | 03:11 | |
| *** DaiDV has joined #openstack-infra | 03:17 | |
| *** sputnik13 has quit IRC | 03:18 | |
| *** ramishra has quit IRC | 03:20 | |
| *** ramishra has joined #openstack-infra | 03:22 | |
| *** salv-orlando has joined #openstack-infra | 03:24 | |
| *** amotoki has quit IRC | 03:25 | |
| *** piet_ has quit IRC | 03:26 | |
| *** baoli has joined #openstack-infra | 03:28 | |
| *** baoli_ has joined #openstack-infra | 03:30 | |
| *** baoli has quit IRC | 03:33 | |
| *** DaiDV has quit IRC | 03:33 | |
| *** sputnik13 has joined #openstack-infra | 03:34 | |
| *** Daisy_ has quit IRC | 03:37 | |
| *** Daisy has joined #openstack-infra | 03:37 | |
| *** salv-orlando has quit IRC | 03:40 | |
| *** Daisy has quit IRC | 03:42 | |
| *** DaiDV has joined #openstack-infra | 03:44 | |
| *** baoli_ has quit IRC | 03:45 | |
| *** shashank_hegde has joined #openstack-infra | 03:55 | |
| *** ramishra has quit IRC | 03:59 | |
| *** ramishra has joined #openstack-infra | 04:00 | |
| *** EricGonczer_ has joined #openstack-infra | 04:01 | |
| *** xarses_ is now known as xarses | 04:04 | |
| *** salv-orlando has joined #openstack-infra | 04:06 | |
| *** Daisy has joined #openstack-infra | 04:06 | |
| *** oanson has joined #openstack-infra | 04:08 | |
| *** hongbin has quit IRC | 04:08 | |
| *** timello has quit IRC | 04:09 | |
| *** amotoki has joined #openstack-infra | 04:10 | |
| *** Daisy has quit IRC | 04:11 | |
| *** EricGonczer_ has quit IRC | 04:12 | |
| *** EricGonczer_ has joined #openstack-infra | 04:12 | |
| *** EricGonczer_ has quit IRC | 04:13 | |
| *** shashank_hegde has quit IRC | 04:14 | |
| *** timello has joined #openstack-infra | 04:16 | |
| *** Daisy has joined #openstack-infra | 04:18 | |
| *** garyk has joined #openstack-infra | 04:18 | |
| *** salv-orlando has quit IRC | 04:18 | |
| *** aeng has quit IRC | 04:22 | |
| *** Daisy has quit IRC | 04:23 | |
| *** Daisy has joined #openstack-infra | 04:24 | |
| *** aeng has joined #openstack-infra | 04:24 | |
| *** roxanagh_ has quit IRC | 04:24 | |
| *** zhurong has quit IRC | 04:26 | |
| *** flepied has joined #openstack-infra | 04:27 | |
| *** Daisy has quit IRC | 04:29 | |
| *** psachin has joined #openstack-infra | 04:29 | |
| *** oanson has quit IRC | 04:29 | |
| *** Daisy has joined #openstack-infra | 04:30 | |
| *** roxanagh_ has joined #openstack-infra | 04:30 | |
| *** DaiDV has quit IRC | 04:33 | |
| *** Daisy has quit IRC | 04:35 | |
| *** Daisy has joined #openstack-infra | 04:35 | |
| *** roxanagh_ has quit IRC | 04:40 | |
| *** Daisy has quit IRC | 04:40 | |
| *** savihou has quit IRC | 04:43 | |
| *** roxanagh_ has joined #openstack-infra | 04:43 | |
| *** oanson has joined #openstack-infra | 04:45 | |
| *** salv-orlando has joined #openstack-infra | 04:45 | |
| *** roxanagh_ has quit IRC | 04:47 | |
| *** amotoki has quit IRC | 04:48 | |
| *** oanson has quit IRC | 04:53 | |
| *** Daisy has joined #openstack-infra | 05:00 | |
| *** Daisy_ has joined #openstack-infra | 05:02 | |
| *** zhurong has joined #openstack-infra | 05:03 | |
| *** PollyZ has quit IRC | 05:03 | |
| *** salv-orlando has quit IRC | 05:05 | |
| *** Daisy has quit IRC | 05:05 | |
| *** claudiub has joined #openstack-infra | 05:07 | |
| *** salv-orlando has joined #openstack-infra | 05:10 | |
| *** abregman has joined #openstack-infra | 05:14 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 05:16 |
| *** shashank_hegde has joined #openstack-infra | 05:18 | |
| *** ociuhandu has joined #openstack-infra | 05:24 | |
| *** abregman has quit IRC | 05:24 | |
| *** abregman has joined #openstack-infra | 05:25 | |
| *** xarses has quit IRC | 05:27 | |
| *** xarses has joined #openstack-infra | 05:28 | |
| *** xarses has quit IRC | 05:28 | |
| *** xarses has joined #openstack-infra | 05:29 | |
| *** ociuhandu has quit IRC | 05:32 | |
| *** e0ne has joined #openstack-infra | 05:32 | |
| *** solumuser__ has joined #openstack-infra | 05:33 | |
| *** caowei has quit IRC | 05:33 | |
| *** savihou has joined #openstack-infra | 05:34 | |
| *** pt_15 has quit IRC | 05:36 | |
| *** amotoki has joined #openstack-infra | 05:36 | |
| *** e0ne has quit IRC | 05:37 | |
| *** yamamoto_ has joined #openstack-infra | 05:38 | |
| *** itisha has quit IRC | 05:40 | |
| *** xarses has quit IRC | 05:40 | |
| *** savihou has quit IRC | 05:40 | |
| *** savihou has joined #openstack-infra | 05:41 | |
| *** xarses has joined #openstack-infra | 05:41 | |
| *** xarses has quit IRC | 05:41 | |
| *** coolsvap has quit IRC | 05:42 | |
| *** xarses has joined #openstack-infra | 05:42 | |
| *** roxanagh_ has joined #openstack-infra | 05:44 | |
| *** PollyZ has joined #openstack-infra | 05:44 | |
| *** Nikolay_St has joined #openstack-infra | 05:45 | |
| *** pt_15 has joined #openstack-infra | 05:46 | |
| *** itisha has joined #openstack-infra | 05:47 | |
| *** roxanagh_ has quit IRC | 05:48 | |
| *** asettle has joined #openstack-infra | 05:48 | |
| *** coolsvap has joined #openstack-infra | 05:52 | |
| *** asettle has quit IRC | 05:54 | |
| *** tonytan4ever has quit IRC | 05:55 | |
| *** Srinu has joined #openstack-infra | 05:56 | |
| Srinu | Hi | 05:57 |
| Srinu | Can anyone know how the slaves in jenkins master sends the response to the jenkins master whether the slave will broadcaste or unicaste the response | 05:58 |
| *** Ravikiran_K has joined #openstack-infra | 06:01 | |
| *** rcernin has quit IRC | 06:01 | |
| *** amotoki has quit IRC | 06:02 | |
| Srinu | Ravikiran_K: hi | 06:03 |
| *** mrmartin has joined #openstack-infra | 06:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 06:07 |
| *** oferby has joined #openstack-infra | 06:10 | |
| AJaeger | infra-root, we still see timeouts in rax-iad. | 06:14 |
| AJaeger | jhesketh, can you help, please? http://logs.openstack.org/15/312315/11/gate/gate-swift-docs/6f9a5f0/console.html is an example | 06:14 |
| *** oanson has joined #openstack-infra | 06:15 | |
| Srinu | AJaeger: hi | 06:16 |
| *** Sukhdev has joined #openstack-infra | 06:16 | |
| *** Daisy_ has quit IRC | 06:18 | |
| *** Daisy has joined #openstack-infra | 06:18 | |
| *** pgadiya has joined #openstack-infra | 06:20 | |
| *** coolsvap has quit IRC | 06:20 | |
| AJaeger | Srinu: sorry, I have no experience with sysadmin of Jenkins - best you ask on the third party channel. | 06:21 |
| *** coolsvap has joined #openstack-infra | 06:22 | |
| *** coolsvap has quit IRC | 06:25 | |
| *** hichihara has joined #openstack-infra | 06:25 | |
| *** hogepodge has quit IRC | 06:25 | |
| *** coolsvap has joined #openstack-infra | 06:25 | |
| *** slaweq has quit IRC | 06:26 | |
| *** hogepodge has joined #openstack-infra | 06:27 | |
| *** amotoki has joined #openstack-infra | 06:31 | |
| *** slaweq has joined #openstack-infra | 06:32 | |
| *** rcernin has joined #openstack-infra | 06:33 | |
| *** salv-orlando has quit IRC | 06:34 | |
| Srinu | AJaeger: can you suggest me how to use one zuul with multiple jenkins master | 06:35 |
| *** flepied has quit IRC | 06:36 | |
| AJaeger | Srinu: See above for my only advise that I can give. | 06:36 |
| *** salv-orlando has joined #openstack-infra | 06:36 | |
| *** ccamacho|afk is now known as ccamacho | 06:36 | |
| *** hdaniel has joined #openstack-infra | 06:37 | |
| *** salv-orlando has quit IRC | 06:41 | |
| *** isaacb has joined #openstack-infra | 06:42 | |
| *** roxanagh_ has joined #openstack-infra | 06:44 | |
| *** mikelk has joined #openstack-infra | 06:45 | |
| *** pcaruana has joined #openstack-infra | 06:46 | |
| *** andreas_s has joined #openstack-infra | 06:47 | |
| *** roxanagh_ has quit IRC | 06:49 | |
| *** PollyZ has quit IRC | 06:54 | |
| *** tonytan4ever has joined #openstack-infra | 06:55 | |
| *** hdaniel has quit IRC | 06:57 | |
| *** tonytan4ever has quit IRC | 07:00 | |
| *** amotoki has quit IRC | 07:01 | |
| *** amotoki has joined #openstack-infra | 07:04 | |
| *** salv-orlando has joined #openstack-infra | 07:07 | |
| *** Sukhdev has quit IRC | 07:08 | |
| *** tesseract- has joined #openstack-infra | 07:09 | |
| openstackgerrit | Dong Ma proposed openstack-infra/project-config: Install nodejs 6.x on Xenial https://review.openstack.org/337481 | 07:11 |
| *** eezhova has joined #openstack-infra | 07:13 | |
| *** vgridnev has quit IRC | 07:16 | |
| *** jpich has joined #openstack-infra | 07:16 | |
| *** kevinbenton has quit IRC | 07:17 | |
| *** nibalizer has quit IRC | 07:18 | |
| *** mikelk has quit IRC | 07:19 | |
| *** lennyb has quit IRC | 07:19 | |
| *** beisner has quit IRC | 07:19 | |
| *** salv-orlando has quit IRC | 07:19 | |
| *** mikelk has joined #openstack-infra | 07:20 | |
| *** lennyb has joined #openstack-infra | 07:20 | |
| *** beisner has joined #openstack-infra | 07:20 | |
| *** kevinbenton has joined #openstack-infra | 07:20 | |
| *** amotoki has quit IRC | 07:21 | |
| *** flepied has joined #openstack-infra | 07:21 | |
| *** vgridnev has joined #openstack-infra | 07:22 | |
| *** nhandler has quit IRC | 07:22 | |
| yolanda | good morning | 07:24 |
| *** nhandler has joined #openstack-infra | 07:24 | |
| *** hdaniel has joined #openstack-infra | 07:24 | |
| *** karthikp_ has quit IRC | 07:26 | |
| *** nibalizer has joined #openstack-infra | 07:26 | |
| *** jlanoux has joined #openstack-infra | 07:31 | |
| *** ifarkas has joined #openstack-infra | 07:32 | |
| *** amotoki has joined #openstack-infra | 07:36 | |
| therve | We're getting some weird failures in Heat test jobs | 07:37 |
| therve | http://logs.openstack.org/50/331850/5/check/gate-heat-python34-db/2967746/console.html for example | 07:37 |
| therve | Any idea what may be happening? | 07:37 |
| AJaeger | therve: on rax-iad? | 07:37 |
| therve | AJaeger, yep | 07:38 |
| AJaeger | rax-iad has networking problems ;( | 07:38 |
| therve | Looks stuck in apt-get update | 07:38 |
| therve | Ah, okay | 07:38 |
| AJaeger | is any infra-root around to file a ticket? | 07:38 |
| *** shashank_hegde has quit IRC | 07:39 | |
| AJaeger | Is there a way to figure out how large the rax-iad problem is? I'm considering to propose disabling that cloud for now. | 07:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Diable rax-iad https://review.openstack.org/337513 | 07:45 |
| AJaeger | in case we want to shutdown rax-iad, here's a change ^ | 07:45 |
| *** salv-orlando has joined #openstack-infra | 07:46 | |
| *** sflanigan has quit IRC | 07:46 | |
| *** zeih has joined #openstack-infra | 07:47 | |
| *** zeih has quit IRC | 07:49 | |
| *** eezhova has quit IRC | 07:51 | |
| *** chlong has quit IRC | 07:52 | |
| *** hashar has joined #openstack-infra | 07:53 | |
| AJaeger | So, looking at current zuul status: Yes, there are some rax-iad timeouts but does not seem so critical yet that we should disable it. I'll still leave my patch up... | 07:53 |
| *** mrmartin has quit IRC | 07:53 | |
| *** zeih has joined #openstack-infra | 07:55 | |
| *** abregman is now known as abregman|mtg | 07:55 | |
| *** amotoki has quit IRC | 07:57 | |
| *** hdaniel has quit IRC | 07:57 | |
| *** eezhova has joined #openstack-infra | 07:57 | |
| openstackgerrit | Alexander Saprykin proposed openstack-infra/project-config: Add CI jobs for murano-plugin-networking-sfc https://review.openstack.org/337113 | 07:57 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** zeih has quit IRC | 08:01 | |
| *** tonytan4ever has joined #openstack-infra | 08:03 | |
| *** dtardivel has joined #openstack-infra | 08:03 | |
| *** zeih has joined #openstack-infra | 08:04 | |
| *** chlong has joined #openstack-infra | 08:04 | |
| *** salv-orlando has quit IRC | 08:04 | |
| *** tonytan4ever has quit IRC | 08:08 | |
| *** chem` has joined #openstack-infra | 08:10 | |
| *** zhurong has quit IRC | 08:11 | |
| *** shardy has joined #openstack-infra | 08:12 | |
| *** derekh has joined #openstack-infra | 08:15 | |
| *** asettle has joined #openstack-infra | 08:19 | |
| *** wanghua has quit IRC | 08:25 | |
| *** rvba has joined #openstack-infra | 08:27 | |
| *** rvba has quit IRC | 08:27 | |
| *** rvba has joined #openstack-infra | 08:27 | |
| *** salv-orlando has joined #openstack-infra | 08:31 | |
| *** lucas|afk is now known as lucasagomes | 08:31 | |
| *** roxanagh_ has joined #openstack-infra | 08:33 | |
| *** sorantis has joined #openstack-infra | 08:33 | |
| kozhukalov | team, could you please add me to shotgun-release group https://review.openstack.org/#/admin/groups/1381,members? I am the author of the patch https://review.openstack.org/#/c/298615/ | 08:35 |
| *** roxanagh_ has quit IRC | 08:37 | |
| yolanda | hi kozhukalov , sure | 08:38 |
| yolanda | done | 08:38 |
| *** acoles_ is now known as acoles | 08:38 | |
| kozhukalov | yolanda: thanks a lot | 08:38 |
| *** makowals has joined #openstack-infra | 08:40 | |
| *** makowals has quit IRC | 08:40 | |
| *** yolanda has quit IRC | 08:41 | |
| *** esikachev has joined #openstack-infra | 08:42 | |
| *** makowals has joined #openstack-infra | 08:42 | |
| *** yolanda has joined #openstack-infra | 08:42 | |
| *** ccamacho is now known as ccamacho|afk | 08:42 | |
| *** makowals has quit IRC | 08:43 | |
| *** abregman|mtg is now known as abregman | 08:43 | |
| *** makowals has joined #openstack-infra | 08:43 | |
| *** amotoki has joined #openstack-infra | 08:43 | |
| *** Daisy has quit IRC | 08:44 | |
| *** electrofelix has joined #openstack-infra | 08:44 | |
| *** Daisy has joined #openstack-infra | 08:44 | |
| *** claudiub|2 has joined #openstack-infra | 08:45 | |
| *** dingyichen has quit IRC | 08:45 | |
| *** makowals has quit IRC | 08:46 | |
| *** claudiub has quit IRC | 08:48 | |
| *** salv-orlando has quit IRC | 08:48 | |
| *** Hal has joined #openstack-infra | 08:55 | |
| *** Hal is now known as Guest28713 | 08:56 | |
| *** john-davidge has joined #openstack-infra | 08:57 | |
| *** amotoki has quit IRC | 08:58 | |
| *** john-dav_ has joined #openstack-infra | 08:58 | |
| asettle | I gathered :;p | 08:59 |
| *** john-davidge has quit IRC | 09:01 | |
| *** hichihara has quit IRC | 09:02 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Cleanup IRONIC_DEPLOY_DRIVER_ISCSI_WITH_IPA variable https://review.openstack.org/337560 | 09:02 |
| *** ccamacho|afk is now known as ccamacho | 09:06 | |
| *** solumuser__ has quit IRC | 09:10 | |
| *** solumuser__ has joined #openstack-infra | 09:10 | |
| *** salv-orlando has joined #openstack-infra | 09:15 | |
| *** sshnaidm has quit IRC | 09:17 | |
| *** solumuser__ has quit IRC | 09:17 | |
| *** solumuser__ has joined #openstack-infra | 09:20 | |
| *** mhickey has joined #openstack-infra | 09:20 | |
| *** yolanda has quit IRC | 09:21 | |
| *** john-dav_ is now known as john-davidge | 09:21 | |
| *** makowals has joined #openstack-infra | 09:22 | |
| *** yolanda has joined #openstack-infra | 09:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 09:23 |
| openstackgerrit | Merged openstack-infra/storyboard: Add example commands for the Story api https://review.openstack.org/332946 | 09:24 |
| *** asettle has quit IRC | 09:24 | |
| *** asettle has joined #openstack-infra | 09:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 09:25 |
| *** savihou has quit IRC | 09:27 | |
| *** sambetts|afk is now known as sambetts | 09:27 | |
| azvyagintsev | Hi folks, looks like i'm doing something wrong with nodepool...i have simple config : http://paste.openstack.org/show/525588/ . nodepool succsesfully spawn vm, but for some reason - don't assign FIP. and tries connect to internal netw. ( from debug stub in ssh_connect function -DEBUG nodepool.utils: ZZZ2: ssh connect going to : root 192.168.111.103) | 09:29 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Remove all jobs running on rh1 https://review.openstack.org/337573 | 09:29 |
| *** asettle has quit IRC | 09:30 | |
| *** salv-orlando has quit IRC | 09:30 | |
| *** jlanoux has quit IRC | 09:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Make Zaqar integration job voting https://review.openstack.org/337152 | 09:32 |
| *** jlanoux has joined #openstack-infra | 09:32 | |
| *** solumuser__ has quit IRC | 09:33 | |
| hughsaunders | Hey all, I vaguely remember there being a tool for creating resources (instances, volumes, networks) etc, then checking those resources are still ok at a later stage eg after an upgrade. However I can't remember the name. Did/does such a thing exist? | 09:35 |
| AJaeger | hughsaunders: grenade is our upgrade tester - best ask on #openstack-qa | 09:36 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Fix dependency on keystone keys https://review.openstack.org/334951 | 09:37 |
| *** eezhova has quit IRC | 09:41 | |
| *** asettle has joined #openstack-infra | 09:41 | |
| *** solumuser__ has joined #openstack-infra | 09:42 | |
| *** yolanda has quit IRC | 09:43 | |
| *** yolanda has joined #openstack-infra | 09:45 | |
| *** arif-ali has quit IRC | 09:48 | |
| *** arif-ali has joined #openstack-infra | 09:50 | |
| *** eezhova has joined #openstack-infra | 09:51 | |
| *** jpich has quit IRC | 09:54 | |
| *** jpich has joined #openstack-infra | 09:54 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Remove all jobs running on rh1 https://review.openstack.org/337573 | 09:54 |
| *** lvdongbing has quit IRC | 09:55 | |
| *** salv-orlando has joined #openstack-infra | 09:56 | |
| *** savihou has joined #openstack-infra | 10:00 | |
| *** solumuser__ has quit IRC | 10:06 | |
| *** Ravikiran_K has quit IRC | 10:10 | |
| *** sorantis has quit IRC | 10:11 | |
| azvyagintsev | all nodepool guru's sleep's? ) | 10:12 |
| *** ociuhandu has joined #openstack-infra | 10:12 | |
| *** salv-orlando has quit IRC | 10:16 | |
| jamespage | morning all | 10:21 |
| *** roxanagh_ has joined #openstack-infra | 10:21 | |
| jamespage | if there is anyone around with a bit of time to review: | 10:21 |
| jamespage | https://review.openstack.org/#/q/owner:james.page%2540canonical.com+status:open+project:openstack-infra/project-config | 10:21 |
| *** linuxmaniac_ has joined #openstack-infra | 10:21 | |
| jamespage | that would be much appreciated :-) | 10:21 |
| *** sshnaidm has joined #openstack-infra | 10:22 | |
| *** yanyanhu has quit IRC | 10:22 | |
| *** linuxmaniac has quit IRC | 10:22 | |
| *** M-docaedo_vector has quit IRC | 10:22 | |
| mordred | jamespage: what's a layer? | 10:23 |
| *** notnownikki has joined #openstack-infra | 10:24 | |
| jamespage | mordred, a layer supports sharing of common functionality or configuration across multiple charms | 10:24 |
| mordred | jamespage: cool - I don't think I knew about those | 10:25 |
| jamespage | mordred, its new-ish | 10:25 |
| jamespage | mordred, https://jujucharms.com/docs/devel/developer-layers | 10:25 |
| mordred | jamespage: btw - the layers and interfaces patches seem to conflict? | 10:25 |
| jamespage | mordred, hmm - lemme look - that's quite possible | 10:25 |
| mordred | jamespage: might be worth stacking them since they might hit the same place in one of the files or something | 10:25 |
| *** roxanagh_ has quit IRC | 10:25 | |
| jamespage | mordred, I'll do that now - thanks for the feedback | 10:27 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute090.chocolate metadata for better tracking https://review.openstack.org/337595 | 10:27 |
| mordred | jamespage: sure thing - I'll keep an eye out for the next version | 10:27 |
| mordred | rcarrillocruz: ^^ 'compute090.chocolate' is never gonna get old | 10:28 |
| openstackgerrit | Merged openstack-infra/project-config: Add python35 jobs https://review.openstack.org/336272 | 10:28 |
| rcarrillocruz | heh :D | 10:28 |
| *** sorantis has joined #openstack-infra | 10:29 | |
| *** sarob has joined #openstack-infra | 10:30 | |
| openstackgerrit | Roman Podoliaka proposed openstack-infra/project-config: WIP: oslo.db: make py35 jobs voting https://review.openstack.org/337597 | 10:30 |
| *** Daisy has quit IRC | 10:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Define coverage jobs when needed https://review.openstack.org/336796 | 10:33 |
| *** Daisy has joined #openstack-infra | 10:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Add client-publish-job group https://review.openstack.org/336787 | 10:33 |
| *** zeih has quit IRC | 10:34 | |
| *** sarob has quit IRC | 10:34 | |
| *** piet has joined #openstack-infra | 10:36 | |
| AJaeger | #status log we now have python35 jobs enabled | 10:37 |
| openstackstatus | AJaeger: finished logging | 10:37 |
| rcarrillocruz | \o/ | 10:37 |
| *** erlon has joined #openstack-infra | 10:37 | |
| *** Daisy has quit IRC | 10:38 | |
| mordred | \o/ indeed | 10:38 |
| *** dims has joined #openstack-infra | 10:38 | |
| *** M-docaedo_vector has joined #openstack-infra | 10:38 | |
| mordred | Zara: just playing with boards in storyboard at the moment - it seems the best way to add a task to a board is to search for its title, yeah? | 10:38 |
| *** rossella_s has quit IRC | 10:39 | |
| *** rossella_s has joined #openstack-infra | 10:40 | |
| Zara | mordred: yeah. :) we'd like to add tasks to boards from story pages at some point... but it might take us a while to get there. | 10:40 |
| * SotK likes to search for the story title and expand the task list, since I normally remember story titles better than task titles | 10:40 | |
| *** berendt has joined #openstack-infra | 10:40 | |
| Zara | or that :D | 10:41 |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add repositories for Charm interfaces https://review.openstack.org/334502 | 10:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove kite configuration https://review.openstack.org/297738 | 10:42 |
| *** salv-orlando has joined #openstack-infra | 10:43 | |
| jamespage | AJaeger, mordred: eek - I just rebased the review that you both +2'ed - sorry | 10:43 |
| AJaeger | jamespage: bad luck ;) That pushes you at the end of the queue again... | 10:44 |
| AJaeger | jamespage: will review again... | 10:44 |
| jamespage | thanks | 10:44 |
| jamespage | the two reviews are stacked now so no longer conflict | 10:45 |
| openstackgerrit | Merged openstack-infra/project-config: Added missed export in sahara-cli job https://review.openstack.org/337177 | 10:45 |
| AJaeger | jamespage: tehre's a bracnh origin/no-ignore-readme for 'https://github.com/openstack-charmers/charm-layer-openstack': - do you really want this? | 10:45 |
| AJaeger | rcarrillocruz, mordred: Could I get a second +2 on https://review.openstack.org/#/c/332173/ , please? | 10:48 |
| *** eezhova has quit IRC | 10:48 | |
| jamespage | AJaeger, dropped that branch - obsolete - nice spot | 10:48 |
| rcarrillocruz | done | 10:49 |
| *** itisha has quit IRC | 10:50 | |
| *** zeih has joined #openstack-infra | 10:50 | |
| *** rodrigods has quit IRC | 10:51 | |
| *** rodrigods has joined #openstack-infra | 10:51 | |
| AJaeger | thanks, rcarrillocruz | 10:51 |
| AJaeger | jamespage: our tests tell me about that - see http://logs.openstack.org/80/334880/2/check/gate-project-config-gerrit/9efc734/ | 10:52 |
| *** yolanda has quit IRC | 10:52 | |
| *** itisha has joined #openstack-infra | 10:52 | |
| jamespage | AJaeger, I see | 10:52 |
| jamespage | AJaeger, I guess I missed it because it does not throw an error - tricky | 10:53 |
| *** deadnull has joined #openstack-infra | 10:54 | |
| *** dizquierdo has joined #openstack-infra | 10:54 | |
| AJaeger | jamespage: it's perfectly fine to have it if desired. Therefore reviewing of teh branches is needed... | 10:54 |
| AJaeger | WE throw errors for situations that are really broken. | 10:54 |
| *** krtaylor has quit IRC | 10:55 | |
| *** psilvad has quit IRC | 10:55 | |
| *** pt_15 has quit IRC | 10:56 | |
| *** yolanda has joined #openstack-infra | 10:56 | |
| *** jlanoux has quit IRC | 10:57 | |
| *** jlanoux has joined #openstack-infra | 10:58 | |
| jamespage | AJaeger, ack | 11:00 |
| AJaeger | jamespage: reviewed the other - instead of python-jobs, why not use pep8 directly in projects.yaml? | 11:00 |
| *** salv-orlando has quit IRC | 11:01 | |
| AJaeger | clarkb, we're down to 8145 jobs in JJB... | 11:02 |
| AJaeger | (and that includes the new python35 ones) | 11:02 |
| jamespage | AJaeger, will be adding unit test targets; hence the use of python-charm-interface-jobs to allow for that to be added later | 11:02 |
| AJaeger | jamespage: ah, ok. | 11:03 |
| jamespage | AJaeger, is that the best way to implement that? appreciate my knowledge here is somewhat limited... | 11:03 |
| AJaeger | derekh: should we merge your rh1 change quickly? | 11:03 |
| AJaeger | jamespage: yeah, that looked fine | 11:03 |
| jamespage | good - grok still works then :-) | 11:04 |
| *** salv-orlando has joined #openstack-infra | 11:04 | |
| azvyagintsev | Repeating help-call: if there is anyone around with knowledge about nodepool ? need help with FIP | 11:05 |
| *** salv-orl_ has joined #openstack-infra | 11:05 | |
| derekh | AJaeger: we have about 4 or 5 hours before I'm taking down the cloud, so before then would be good, I can get one or two others to sanity check it first if you want | 11:06 |
| AJaeger | mordred, rcarrillocruz : derekh is speaking about https://review.openstack.org/337573 - reviews are welcome | 11:09 |
| AJaeger | derekh: then let's take it down soon so that it's idle... | 11:09 |
| *** salv-orlando has quit IRC | 11:10 | |
| mordred | AJaeger, derekh: lgtm | 11:10 |
| derekh | AJaeger: ack, will get a second person from #tripleo to look at it ASAP, then I think we're good to go, I'll ping you when its ready | 11:11 |
| AJaeger | derekh: mordred just approved - do you want to WIP it? | 11:12 |
| mordred | whoops - sorry | 11:12 |
| mordred | I can remove my +1 too | 11:12 |
| derekh | AJaeger: mordred its ok,, if there is anything missed I'll follow up | 11:12 |
| AJaeger | works for me... | 11:12 |
| mordred | okie | 11:12 |
| openstackgerrit | Ildiko Vancsa proposed openstack-infra/project-config: Add install-guide job to Aodh https://review.openstack.org/337357 | 11:15 |
| *** sorantis has quit IRC | 11:16 | |
| yolanda | hi mordred, around? care reviewing a set of patches with puppet-infracloud topic? | 11:16 |
| *** notnownikki has quit IRC | 11:17 | |
| *** acabot has quit IRC | 11:17 | |
| *** piet has quit IRC | 11:18 | |
| *** asettle has quit IRC | 11:20 | |
| mordred | yolanda: still waking up a little ... | 11:21 |
| *** asettle has joined #openstack-infra | 11:21 | |
| yolanda | come on... :) | 11:21 |
| * AJaeger sends an espresso to mordred | 11:22 | |
| *** zeih has quit IRC | 11:23 | |
| AJaeger | again a timeout in rax-iad - http://logs.openstack.org/73/332173/4/gate/gate-project-config-layout/bfd39a2/ ;( | 11:24 |
| *** weshay has joined #openstack-infra | 11:25 | |
| *** asettle has quit IRC | 11:26 | |
| *** ociuhandu has quit IRC | 11:27 | |
| *** zeih has joined #openstack-infra | 11:27 | |
| *** markusry has joined #openstack-infra | 11:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 11:30 |
| *** eezhova has joined #openstack-infra | 11:31 | |
| *** zeih has quit IRC | 11:32 | |
| *** jed56 has joined #openstack-infra | 11:33 | |
| *** yolanda has quit IRC | 11:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove kite configuration https://review.openstack.org/297738 | 11:36 |
| *** markusry has quit IRC | 11:36 | |
| rcarrillocruz | hmm | 11:36 |
| rcarrillocruz | AJaeger: remember the issue from yesterday about my job on centos missing virtualenv ? | 11:36 |
| rcarrillocruz | i held a node | 11:36 |
| rcarrillocruz | looks like it doesn't get installed cleanly within the job, an uninstall/install from pip afterwards fixes it | 11:37 |
| rcarrillocruz | http://paste.openstack.org/show/525953/ | 11:37 |
| rcarrillocruz | very odd | 11:37 |
| *** moravec has quit IRC | 11:37 | |
| *** tpsilva has joined #openstack-infra | 11:39 | |
| *** moravec has joined #openstack-infra | 11:39 | |
| *** gordc has joined #openstack-infra | 11:44 | |
| Srinu | fungi: hi | 11:45 |
| Srinu | jeblair: hi | 11:45 |
| Srinu | clarkb:hi | 11:45 |
| Srinu | mordred:hi | 11:46 |
| frickler | AJaeger: clarkb: python35-db seems broken: http://logs.openstack.org/06/317306/12/check/gate-nova-python35-db-nv/2c4e88a/console.html#_2016-07-05_10_57_44_044523 | 11:46 |
| *** ldnunes has joined #openstack-infra | 11:46 | |
| frickler | are there other jobs using mysql on xenial? | 11:47 |
| *** asettle has joined #openstack-infra | 11:47 | |
| *** eezhova has quit IRC | 11:48 | |
| *** markusry has joined #openstack-infra | 11:48 | |
| *** lucasagomes is now known as lucas|hungry | 11:48 | |
| frickler | neutron-python35 seems fine from the start, yay | 11:49 |
| *** aysyd has joined #openstack-infra | 11:51 | |
| *** psilvad has joined #openstack-infra | 11:53 | |
| *** ccamacho is now known as ccamacho|lunch | 11:53 | |
| *** eezhova has joined #openstack-infra | 11:53 | |
| *** markusry has quit IRC | 11:54 | |
| *** zeih has joined #openstack-infra | 11:54 | |
| *** amotoki has joined #openstack-infra | 11:56 | |
| *** yamamoto_ has quit IRC | 11:57 | |
| *** mrmartin has joined #openstack-infra | 11:58 | |
| *** psilvad has quit IRC | 11:59 | |
| *** psilvad has joined #openstack-infra | 11:59 | |
| AJaeger | rcarrillocruz: strange... | 12:00 |
| *** zeih has quit IRC | 12:01 | |
| AJaeger | frickler: interesting. Seems we need to do some extra work for Xenial here. Let's flag this for clarkb and others to look into ^ | 12:01 |
| *** markusry has joined #openstack-infra | 12:02 | |
| AJaeger | rcarrillocruz: we do have networking problems on rax-iad - is your held node on rax-iad and might networking problems explain the issue? | 12:03 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP 'All Boards and Worklists' Page https://review.openstack.org/279753 | 12:03 |
| *** hferenc has quit IRC | 12:03 | |
| rcarrillocruz | no, cos i also got that issue on osic, ovh, etc | 12:03 |
| rcarrillocruz | :/ | 12:03 |
| *** psilvad has quit IRC | 12:04 | |
| *** tonytan4ever has joined #openstack-infra | 12:04 | |
| *** BobBall_AWOL is now known as BobBall | 12:05 | |
| AJaeger | ;( | 12:05 |
| *** rhallisey has joined #openstack-infra | 12:06 | |
| *** raildo-afk is now known as raildo | 12:07 | |
| AJaeger | rcarrillocruz: I have no idea abuot this, hope others can help you... | 12:08 |
| *** rfolco_ has joined #openstack-infra | 12:08 | |
| *** zeih has joined #openstack-infra | 12:08 | |
| rcarrillocruz | yep, i'll await for clarkb and pabelanger , it's strange cos i would assume this would happen to other jobs using centos cos the failure occurs during devstack bringup | 12:08 |
| *** roxanagh_ has joined #openstack-infra | 12:09 | |
| *** tonytan4ever has quit IRC | 12:09 | |
| *** psilvad has joined #openstack-infra | 12:10 | |
| openstackgerrit | Mathieu Mitchell proposed openstack-infra/project-config: Introduce experimental job to test TinyIPA with less RAM https://review.openstack.org/327994 | 12:11 |
| *** yolanda has joined #openstack-infra | 12:11 | |
| *** chlong has quit IRC | 12:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove all jobs running on rh1 https://review.openstack.org/337573 | 12:12 |
| *** sorantis has joined #openstack-infra | 12:13 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Update to use quintupleo https://review.openstack.org/336309 | 12:13 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Make ovb testenv more user friendly https://review.openstack.org/337637 | 12:13 |
| AJaeger | frickler, clarkb: There's a release note entry about mysql, is that relevant to the problem? https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#MySQL_5.7 | 12:13 |
| *** roxanagh_ has quit IRC | 12:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Consistent ERROR messages https://review.openstack.org/332173 | 12:14 |
| frickler | AJaeger: not directly, but from testing locally, "mysql -h localhost" would work, while the used variant "mysql -h 127.0.0.1" doesn't. | 12:16 |
| AJaeger | frickler: care to send a patch for project-config - file jenkins/jobs/macros.yaml? | 12:18 |
| *** openstackgerrit has quit IRC | 12:19 | |
| *** AL34N1X has joined #openstack-infra | 12:19 | |
| *** openstackgerrit has joined #openstack-infra | 12:19 | |
| *** AL34N1X has quit IRC | 12:19 | |
| *** rhallisey has quit IRC | 12:19 | |
| *** AL34N1X has joined #openstack-infra | 12:20 | |
| AJaeger | frickler, clarkb : It's strange that localhost works but 127.0.0.1 not ;( | 12:20 |
| openstackgerrit | Dimitri Mazmanov proposed openstack-infra/project-config: Add python-kingbirdclient to the project list https://review.openstack.org/337254 | 12:20 |
| *** _prad_ has joined #openstack-infra | 12:21 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Update ci report script to only include rh2 jobs https://review.openstack.org/337641 | 12:21 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/release-tools: update the script to expire old bug reports https://review.openstack.org/337642 | 12:21 |
| AJaeger | can any TC member ( mordred, jeblair, ttx ? ) explain to me how a repo can be part of one team but not part of official repos? I'm confused by Fuel CCP status: They have an approved spec but still don't want repos as official part... https://review.openstack.org/335584 | 12:23 |
| *** rhallisey has joined #openstack-infra | 12:23 | |
| openstackgerrit | Alexander Saprykin proposed openstack-infra/project-config: Add CI jobs for murano-plugin-networking-sfc https://review.openstack.org/337113 | 12:24 |
| *** dizquierdo has quit IRC | 12:25 | |
| ttx | AJaeger: we don't force them to declare all the repositories a given official project team is working on | 12:26 |
| ttx | i.e. they can have repositories on the side that the same group of people works on... | 12:26 |
| ttx | Not that I recommend it, but I could see that happening | 12:27 |
| *** dprince has joined #openstack-infra | 12:27 | |
| AJaeger | ttx, I see - ok, so be it... Thanks. I would have expected teams to adopt them and retire. Gives them much more leeway IMHO to be official... | 12:27 |
| *** zeih has quit IRC | 12:28 | |
| *** _ari_ has joined #openstack-infra | 12:28 | |
| pabelanger | morning | 12:28 |
| ttx | AJaeger: if the alternative is having them stand up such repos on GitHub, I prefer the current state | 12:28 |
| *** moravec has quit IRC | 12:29 | |
| AJaeger | ttx, yeah, hosting on github would be worse | 12:29 |
| *** sarob has joined #openstack-infra | 12:31 | |
| *** notnownikki has joined #openstack-infra | 12:31 | |
| openstackgerrit | James Slagle proposed openstack-infra/project-config: Add TripleO 2-node job https://review.openstack.org/326095 | 12:31 |
| *** EricGonczer_ has joined #openstack-infra | 12:31 | |
| *** zeih has joined #openstack-infra | 12:32 | |
| *** moravec has joined #openstack-infra | 12:32 | |
| *** claudiub has joined #openstack-infra | 12:33 | |
| *** aviau has quit IRC | 12:34 | |
| *** aviau has joined #openstack-infra | 12:34 | |
| AJaeger | morning, pabelanger | 12:34 |
| AJaeger | pabelanger: rcarrillocruz has a problem with centos and virtualenv, see scrollback from an hour ago - do you have any ideas? | 12:35 |
| *** sarob has quit IRC | 12:35 | |
| mordred | yolanda: you have your old hpe.com address in the root email alias | 12:36 |
| yolanda | mordred, i thought i updated :( | 12:37 |
| *** claudiub|2 has quit IRC | 12:37 | |
| pabelanger | AJaeger: rcarrillocruz: Possible we are hitting https://review.openstack.org/#/c/327472/ now? | 12:37 |
| pabelanger | There was a new release of DIB recently | 12:37 |
| AJaeger | pabelanger, mordred : I see quite a few networking issues (manifests as timeouts), all of them in rax-iad. Can you open a ticket with Rackspace? Or what's the best way forward here? | 12:38 |
| *** _prad_ is now known as _\prad | 12:38 | |
| *** _\prad has quit IRC | 12:38 | |
| *** chem` is now known as chem | 12:39 | |
| pabelanger | I know fungi has an ongoing ticket open, we should check if that is the same region | 12:39 |
| pabelanger | AJaeger: do we have a review open to disable rax-iad? | 12:40 |
| AJaeger | yes, https://review.openstack.org/#/c/337513 | 12:40 |
| AJaeger | I'm not sure yet whether it occurs often enough - is there a way to figure out how often we have fails in rax-iad? | 12:41 |
| AJaeger | The review contains some logs to timeouts - I didn't copy all, just a few of them... | 12:41 |
| *** ociuhandu has joined #openstack-infra | 12:41 | |
| pabelanger | we should be able to track async timeouts in logstash | 12:42 |
| pabelanger | but I've never tried | 12:42 |
| AJaeger | neither have I ;( | 12:42 |
| *** EricGonczer_ has quit IRC | 12:42 | |
| AJaeger | and no time to investigate further. | 12:42 |
| *** pradk- has joined #openstack-infra | 12:43 | |
| *** chlong has joined #openstack-infra | 12:43 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/project-config: Fix mysql setup for Xenial jobs https://review.openstack.org/337648 | 12:45 |
| frickler | AJaeger: ^ that would need to be tested on both versions, though, trusty and xenial | 12:46 |
| *** rlandy has joined #openstack-infra | 12:46 | |
| *** amotoki has quit IRC | 12:47 | |
| AJaeger | thanks, frickler. Let's see who can test it. clarkb, could you check, please ? ^ | 12:47 |
| *** Nikolay_St has quit IRC | 12:47 | |
| frickler | AJaeger: maybe you also want to send a followup to clarkb's mail, stating that the new jobs are now there, but the db variant still needs fixing? | 12:48 |
| *** chem` has joined #openstack-infra | 12:49 | |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| AJaeger | good idea, I'm on it. | 12:50 |
| *** lucas|hungry is now known as lucasagomes | 12:50 | |
| rcarrillocruz | let me see | 12:50 |
| rcarrillocruz | pabelanger: ^ | 12:50 |
| pabelanger | derekh: can you manually delete 1e5b0379-cb44-4359-b692-094d279d1843 and 6c580640-d6aa-452f-b42a-5e66fc8407b8 in tripleo-test-cloud-rh1 | 12:52 |
| *** mrmartin has quit IRC | 12:52 | |
| pabelanger | that is the last bits still in nodepool | 12:52 |
| pabelanger | images have been deleted too | 12:53 |
| derekh | pabelanger: yup, will do | 12:53 |
| *** chem has quit IRC | 12:53 | |
| AJaeger | frickler: mail written - and commented also on your change.... | 12:53 |
| *** jed56 has quit IRC | 12:55 | |
| *** sarob has joined #openstack-infra | 12:55 | |
| *** matt-borland has joined #openstack-infra | 12:56 | |
| rcarrillocruz | pabelanger: could be related, on my held node i do see /etc/yum.conf with the python-virtualenv package excluded, just as the element from that change does | 12:56 |
| *** jed56 has joined #openstack-infra | 12:57 | |
| pabelanger | rcarrillocruz: Ya, I am not sure what the review was trying to do actually. I just remember seeing it in my inbox | 12:58 |
| openstackgerrit | Merged openstack-infra/release-tools: List system dependencies for running common tests https://review.openstack.org/335804 | 12:58 |
| *** solumuser__ has joined #openstack-infra | 12:58 | |
| rcarrillocruz | and my change did not have that problem a couple weeks ago | 12:58 |
| rcarrillocruz | so dates match at least | 12:58 |
| *** asaprykin has joined #openstack-infra | 12:59 | |
| *** sarob has quit IRC | 12:59 | |
| asaprykin | Hello, please review patch to project-config https://review.openstack.org/#/c/337113/ | 13:01 |
| *** ianychoi has quit IRC | 13:01 | |
| *** jcoufal has joined #openstack-infra | 13:02 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Change networking-ovn python34 job to voting https://review.openstack.org/335605 | 13:02 |
| *** claudiub has quit IRC | 13:02 | |
| rcarrillocruz | hmm | 13:04 |
| rcarrillocruz | so | 13:04 |
| rcarrillocruz | if the change installs python-virtualenv with a package | 13:04 |
| rcarrillocruz | to later overwrite it with pip | 13:04 |
| derekh | pabelanger: done | 13:05 |
| rcarrillocruz | i guess we could just remove the tools/fixup_stuff.sh on devstack that removes pip system package since we already have pip on the image | 13:05 |
| rcarrillocruz | thoughts? | 13:05 |
| *** julim has joined #openstack-infra | 13:05 | |
| *** yolanda has quit IRC | 13:05 | |
| *** tonytan4ever has joined #openstack-infra | 13:05 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Change networking-ovn python34 job to voting https://review.openstack.org/335605 | 13:06 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP 'All Boards and Worklists' Page https://review.openstack.org/279753 | 13:06 |
| rcarrillocruz | i believe removing the system-package yet it's been already overwritten by pip, to then do again a pip install -U is what leads the system in a funky state for virtualenv | 13:06 |
| *** yolanda has joined #openstack-infra | 13:07 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Remove unused variable https://review.openstack.org/337656 | 13:08 |
| *** amotoki has joined #openstack-infra | 13:08 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove tripleo-jobs.py https://review.openstack.org/337657 | 13:09 |
| *** andreykurilin has joined #openstack-infra | 13:09 | |
| Srinu | hi | 13:09 |
| Srinu | http://paste.openstack.org/show/525997/ | 13:09 |
| Srinu | any clue for ^^^ | 13:09 |
| andreykurilin | AJaeger: hi! Could you look at https://review.openstack.org/#/c/333226/ ?) | 13:10 |
| *** tonytan4ever has quit IRC | 13:10 | |
| AJaeger | andreykurilin: is that package in requirements repo? | 13:10 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Update ci report script to only include rh2 jobs https://review.openstack.org/337641 | 13:11 |
| *** dizquierdo has joined #openstack-infra | 13:12 | |
| andreykurilin | AJaeger: no, but there are projects which are not uses g-r and can use pytest. As an alternative, I can rename `pytest_results.html` to `whatever_you_want_results.html`, so any possible tool will be able to post reports(testr_results.html and nose_results.html are parsed in infra-scripts, so there is no possibility to use their name for other tools) | 13:13 |
| Srinu | we are facing frequent slave went offline issue with this setup http://paste.openstack.org/show/525997/ | 13:13 |
| *** dizquierdo has quit IRC | 13:13 | |
| *** ccamacho|lunch is now known as ccamacho | 13:13 | |
| andreykurilin | AJaeger: btw, nosetests is not in g-r too, but it is possible to publish results for it | 13:14 |
| AJaeger | andreykurilin: I'll comment on the review but will stay neutral on it. | 13:14 |
| *** tlian has joined #openstack-infra | 13:15 | |
| andreykurilin | AJaeger: so what about add some neutral report name which can be published and will no be parsed by jenkins/scripts/run-tox.sh ? | 13:15 |
| andreykurilin | AJaeger: just "test_results.html" or something like that | 13:16 |
| AJaeger | andreykurilin: better ask somebody else from the team. If clarkb, jeblair, mordred, or fungi approve it, I'm happy. From what I know, I cannot give a positive review - but also don't want to give a -1. | 13:16 |
| andreykurilin | AJaeger: ok! thanks | 13:17 |
| andreykurilin | clarkb, jeblair, mordred, fungi : hi! | 13:17 |
| *** pgadiya has quit IRC | 13:17 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add valid set of options parameters to convert_xml https://review.openstack.org/337381 | 13:17 |
| mordred | andreykurilin: hi! | 13:18 |
| andreykurilin | mordred: Can we discuss https://review.openstack.org/#/c/333226/ ?) | 13:18 |
| mordred | andreykurilin: so - we focus on testr/subunit and not nose/py.test for a reason | 13:18 |
| mordred | that is that subunit provides a streamable machine-parsable output format | 13:18 |
| mordred | and we have more and more tooling built up that processes that | 13:18 |
| rcarrillocruz | yah | 13:19 |
| rcarrillocruz | thta patch causes the issue | 13:19 |
| rcarrillocruz | pabelanger: if you hold a centos node | 13:19 |
| mordred | while we could certainly do what you suggest and have a "neutral" name ... I wouldn't want people to think that using py.test or nose are in some way "supported" | 13:19 |
| rcarrillocruz | remove python-virtualenv from /etc/yum.conf | 13:19 |
| rcarrillocruz | yum install python-virtualenv | 13:19 |
| rcarrillocruz | pip install -U virtualenv | 13:19 |
| rcarrillocruz | yum remove python-virtualenv | 13:19 |
| rcarrillocruz | that leaves some bits of virtualenv in there | 13:20 |
| rcarrillocruz | yet the virtualenv.py is not | 13:20 |
| andreykurilin | mordred: but nose is supported by infra scripts | 13:20 |
| rcarrillocruz | causing the my test to fail | 13:20 |
| mordred | andreykurilin: our nose support is historical and we'd delete it the second that swift would get off of nose | 13:20 |
| rcarrillocruz | this is the failure by following above steps: | 13:20 |
| *** zhurong has joined #openstack-infra | 13:20 | |
| rcarrillocruz | http://paste.openstack.org/show/525998/ | 13:21 |
| AJaeger | rcarrillocruz: glad to hear that pabelanger's instinct was good° | 13:21 |
| rcarrillocruz | indeed :-) | 13:21 |
| rcarrillocruz | as you can see, some stuff is removed (virtualenv.py), but some other is not (virtualenv dist-info + virtualenv_support) | 13:22 |
| mordred | andreykurilin: I mean, I have no problem with your patch from a practical basis - mostly just worried that people will think that means that using nose/py.test is more supported than it is. we'll never be able to do anything other than simply publish the html that either tool might output | 13:22 |
| *** Srinu has quit IRC | 13:22 | |
| mordred | andreykurilin: and if one of the projects that was using nose/py.test wanted to become more "official" but lived during their unofficial period using not-testr, it'll be hard on them when they move | 13:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Return current router if there are no updates https://review.openstack.org/337661 | 13:23 |
| rcarrillocruz | now i'm not sure how to proceed | 13:23 |
| rcarrillocruz | if we should revert + regenerate new dibs | 13:23 |
| *** mdrabe has joined #openstack-infra | 13:23 | |
| andreykurilin | mordred: in case if project is signed on g-r, it will unable to use pytest and neutral name, so it should not confuse them. but if project is not use g-r, why not allow to publish "custom" reports? | 13:23 |
| *** esberglu has joined #openstack-infra | 13:23 | |
| rcarrillocruz | or touch devstack, so it doesn't remove the python-virtualenv package (in the end, the image has it, installed from source) | 13:23 |
| rcarrillocruz | otoh, that may break devstack runs on centos that are not build by our dibs | 13:24 |
| AJaeger | andreykurilin: why not integrate nicely in testr/subunit than doing a suboptimal setup with pytest? | 13:24 |
| rcarrillocruz | so :S | 13:24 |
| mordred | andreykurilin: well, it's not _just_ about g-r ... the common interface in openstack is testr, not py.test or nose, and it is not desirable to add py.test or nose to anything that is official. that said - I don't have personal issues with what the patch does in support of non-official projects | 13:24 |
| *** rubasov has joined #openstack-infra | 13:24 | |
| mordred | I just don't want to increase the confusion | 13:24 |
| *** penguinolog has joined #openstack-infra | 13:25 | |
| andreykurilin | There are some problems with testr + windows/macos | 13:25 |
| *** burgerk has joined #openstack-infra | 13:25 | |
| mordred | so ... | 13:25 |
| andreykurilin | mordred: and pytest doesn't have such problems | 13:25 |
| andreykurilin | :) | 13:25 |
| mordred | 2 things: a) we don't support either platform b) if there are problems and they are important, we should fix them | 13:25 |
| mordred | andreykurilin: pytest is not an option for openstack broadly | 13:26 |
| andreykurilin | mordred: why?) | 13:26 |
| mordred | we have _way_ too much invested in subunit | 13:26 |
| *** eezhova has quit IRC | 13:26 | |
| andreykurilin | mordred: how it is related? pytest has subunit plugin too - https://pypi.python.org/pypi/pytest-subunit | 13:27 |
| mordred | interesting. I had not seen that plugin | 13:27 |
| mordred | however | 13:27 |
| *** chem` has quit IRC | 13:28 | |
| mordred | the thing is - openstack uses testr. it's not 'better' to have 2 test runners, so a discussion about py.test should be about moving all 1000 repos from testr to py.test | 13:28 |
| *** chem` has joined #openstack-infra | 13:28 | |
| mordred | that is a _very_ large amount of work for not a huge amount of gain, no? | 13:28 |
| *** armax has joined #openstack-infra | 13:28 | |
| mordred | (I'll go look at pytest-subunit just on general principle) | 13:29 |
| andreykurilin | mordred: I can understand "huge work" when we talk about porting webob to flask or something like that. But pytest, testr, nosetests are just one line in tox.ini file | 13:29 |
| *** xyang1 has joined #openstack-infra | 13:29 | |
| mordred | they're not though. they are huamn workflow too | 13:29 |
| AJaeger | andreykurilin: if it's just a single line, why are you not switching? ;) | 13:31 |
| mordred | the humans in openstack have been consistently using testr for about 4 years. switching tools for people, and the tools that automation runs, and that scripts consume, and all of that is non-trivial | 13:31 |
| Shrews | mordred: rcarrillocruz: i'm on the fence with 337661 | 13:31 |
| andreykurilin | AJaeger: already done https://github.com/openstack/rally/blob/master/tox.ini#L21 | 13:32 |
| AJaeger | mordred: thanks for jumping in here! I'm happy to learn | 13:32 |
| Shrews | mordred: rcarrillocruz: returning None let's the user see if any changes were applied (and means we should move the bug fix to os_router). | 13:32 |
| rcarrillocruz | Shrews: checking | 13:32 |
| Shrews | mordred: rcarrillocruz: but always returning a dict seems more consistent | 13:32 |
| AJaeger | andreykurilin: I meant using testr | 13:32 |
| mordred | andreykurilin: I think he means "why don't you move back to testr since that's what openstack uses and rally is an openstack project" | 13:32 |
| AJaeger | what mordred said ^ | 13:33 |
| mordred | andreykurilin: and actually, rally is one of the programs where using testr/subunit is even more important, because we'd be more likely to do want to do long-term trend analysis on rally output | 13:33 |
| rcarrillocruz | Shrews: so, returning {} instead of None | 13:33 |
| rcarrillocruz | ? | 13:33 |
| mordred | like, unittest for swift aren't _that_ interesting for trends compared to things like tempest or rally runs | 13:33 |
| andreykurilin | because we have problems with running unit tests on local env | 13:33 |
| andreykurilin | testr != subunit | 13:34 |
| Shrews | rcarrillocruz: no, returning the current router | 13:34 |
| rcarrillocruz | ah | 13:34 |
| *** raildo has quit IRC | 13:34 | |
| rcarrillocruz | hmm | 13:34 |
| andreykurilin | mordred: I started work on subunit-based output for rally, but have no time to finish it. BUT it is not related to how launch unit tests for rally | 13:34 |
| Shrews | rcarrillocruz: mordred: for reference .... https://github.com/ansible/ansible-modules-core/issues/3065 | 13:34 |
| mordred | andreykurilin: then we should focus on fixing those if we can... and if they are unfixable bugs, then clearly the tool that openstack uses is deficient and we should change it for everyone | 13:34 |
| *** raildo-afk has joined #openstack-infra | 13:35 | |
| Shrews | rcarrillocruz: mordred: last two comments in the PR is the important one | 13:35 |
| mordred | andreykurilin: like, this is the reason we're so adamant about sharing tools - it's so that energy in fixing issues can be shared across projects. if rally by itself switches ot py.test, then any work on making things better in either py.test or testr are work that we aren't sharing | 13:35 |
| Shrews | s | 13:35 |
| mordred | Shrews: thanks -it's up in my browser to look at | 13:36 |
| *** burgerk has quit IRC | 13:36 | |
| *** raildo-afk is now known as raildo | 13:36 | |
| rcarrillocruz | eugh, that's an ugly bug | 13:36 |
| andreykurilin | mordred: testr has a bug which requires to launch py34 before py27. The report for this bug was published long time ago, but no one wants to fix it. If no one have time to fix this bug, I don't think that support for macos/windows will be added soon | 13:38 |
| *** links has joined #openstack-infra | 13:38 | |
| pabelanger | rcarrillocruz: revert will take some work, since we also need to release diskimage-builder to pypi too | 13:38 |
| pabelanger | rcarrillocruz: not to say that is the right path, I would confirm with ianw first however | 13:39 |
| penguinolog | zuul is too sloooooow... | 13:39 |
| AJaeger | ianw is on vacation, his alias right now is "ianw_pto" | 13:39 |
| *** Daisy has joined #openstack-infra | 13:39 | |
| *** cardeois has joined #openstack-infra | 13:39 | |
| *** rbrndt has joined #openstack-infra | 13:39 | |
| penguinolog | py27 checker is running >35 minutes and still "in progress" | 13:40 |
| mordred | andreykurilin: I hear that - but if the case is that testr is somehow unsuitable for openstack to use, then the suggestion should be to change what openstack uses. to my knowledge, that suggestion has not been made | 13:40 |
| rcarrillocruz | Shrews: so, i'm checking other objects update methods, and I don't see returning None if there's nothing to update as a pattern | 13:40 |
| *** Goneri has joined #openstack-infra | 13:40 | |
| AJaeger | penguinolog: which change? | 13:40 |
| *** thiagop has joined #openstack-infra | 13:40 | |
| rcarrillocruz | as an end user of shade, i think it's ok to expect an object to show what happened, and if nothing happened just the curr router | 13:40 |
| penguinolog | AJaeger, https://review.openstack.org/337491 - telnet://104.239.228.13:8088 | 13:40 |
| *** eezhova has joined #openstack-infra | 13:40 | |
| penguinolog | telnet is not connecting | 13:40 |
| rcarrillocruz | so i'm good fixing this in shade | 13:40 |
| mordred | andreykurilin: http://governance.openstack.org/reference/cti/python_cti.html#unittest-running the PTI lists testrepository explcitly | 13:41 |
| Shrews | rcarrillocruz: yeah, i just checked too. update_subnet() returns None, but that is the only one I see | 13:41 |
| rcarrillocruz | having said that, will that also imply some cornercase tweak on os_router | 13:41 |
| rcarrillocruz | ? | 13:41 |
| penguinolog | penguinolog@phoenix ~ $ telnet 104.239.228.13:8088telnet: 104.239.228.13:8088: Name or service not known | 13:41 |
| penguinolog | 104.239.228.13:8088: Unknown host | 13:41 |
| AJaeger | penguinolog: line 2 of that has "Building remotely on ubuntu-trusty-rax-iad-2222220 in workspace /home/jenkins/workspace/gate-fuel-devops-python27" | 13:41 |
| rcarrillocruz | pabelanger: ianw_pto is on leave, yeah? | 13:41 |
| Shrews | rcarrillocruz: i'm afraid of breaking others who may be depending on None... perhaps that is reason enough to keep it as-is | 13:41 |
| AJaeger | penguinolog: we have networking problem in rax-iad ;( Nothing to do with Zuul. Zuul is fast ;) | 13:41 |
| Shrews | rcarrillocruz: and fixing os_router to check for None | 13:42 |
| *** garyk has quit IRC | 13:42 | |
| clarkb | AJaeger: It didnt even occur to me that the db macro might not work, probably because I am still used to the DB setup happening in image | 13:42 |
| mordred | andreykurilin: if the issues in testr are unfixable or are not being fixed, then I would suggest starting the discussion to get the TC to change the policy on what OpenStack uses. but until that time, we use the thing that we've all decided that we use | 13:42 |
| penguinolog | AJaeger, thanks for information. Will wait... | 13:42 |
| clarkb | mordred: as an aside py.tests parallel testing support is funky last time I looked at it | 13:43 |
| Shrews | rcarrillocruz: i think i just convinced myself to leave it as is and fix the module | 13:43 |
| *** eharney has joined #openstack-infra | 13:43 | |
| andreykurilin | mordred: If I start new threat about "move from testr to something else", this link will be posted in first reply and most of folks will said "you should fix issues in testr instead" | 13:43 |
| mordred | clarkb: same here - but I also haven't looked at it in a while, and jeblair will be the first to tell you that so is testr's :) | 13:43 |
| andreykurilin | *will say | 13:43 |
| rcarrillocruz | being strict, is an pi breakage, so... | 13:43 |
| *** ayoung has joined #openstack-infra | 13:43 | |
| clarkb | mordred: well testrs works py.tests did not but sure | 13:43 |
| *** zeih has quit IRC | 13:44 | |
| *** markusry has quit IRC | 13:44 | |
| AJaeger | clarkb: there's a change by frickler for the db setup - what do you think of it? | 13:44 |
| *** markusry has joined #openstack-infra | 13:44 | |
| clarkb | AJaeger: trying to load it now | 13:44 |
| rcarrillocruz | clarkb: when you get a sec, can you look an issue on centos dib instances issues? | 13:44 |
| *** Daisy has quit IRC | 13:44 | |
| mordred | andreykurilin: if the issues are unfixable or unfixable in an amount of time that would be greater than the time it would take to make sure that py.test can do what is needed, I think that's a perfectly valid basis for the conversion and would be happy to say so | 13:44 |
| *** links has quit IRC | 13:45 | |
| pabelanger | rcarrillocruz: oh, not sure. I can find out | 13:45 |
| *** hichihara has joined #openstack-infra | 13:45 | |
| mordred | andreykurilin: but you're right, just like I have, that will be the first respnse, beuase there are a lot of interrelated requirements and stakeholders in openstack - and making a switch in something so fundamental gets harder than you might think | 13:45 |
| mordred | but if there are serious issues, then we should have a serious discussion about the topic | 13:46 |
| jamespage | AJaeger, thanks for the reviews | 13:46 |
| *** claudiub has joined #openstack-infra | 13:46 | |
| *** zeih has joined #openstack-infra | 13:46 | |
| *** jsavak has joined #openstack-infra | 13:47 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 13:47 |
| *** jtomasek_ has joined #openstack-infra | 13:48 | |
| clarkb | rcarrillocruz: have more info/context? | 13:48 |
| rcarrillocruz | my change fails tests on centos consistently | 13:48 |
| rcarrillocruz | http://logs.openstack.org/35/332435/6/gate/gate-ansible-role-cloud-launcher-dsvm-ansible-func-centos-7/a46ff96/console.html | 13:48 |
| andreykurilin | mordred: ok, so I'll start a thread when have time for it and listen how bad my suggestion is :) | 13:48 |
| *** r-mibu has quit IRC | 13:48 | |
| mordred | andreykurilin: :) | 13:49 |
| rcarrillocruz | i've reproduced this on a centos held node | 13:49 |
| *** r-mibu has joined #openstack-infra | 13:49 | |
| clarkb | AJaeger: frickler that seems fine though I am not sure I fully grok the relationship between mysql-prep and mysql-db yet | 13:49 |
| clarkb | rcarrillocruz: and why do we think it is an image issue? | 13:49 |
| rcarrillocruz | if you install python-virtualenv package, then overwrite it with pip from source, then remove it from system packages it leaves it in a bad state | 13:49 |
| rcarrillocruz | http://paste.openstack.org/show/525953/ | 13:49 |
| clarkb | ah yes becaus ecentos is broken | 13:49 |
| rcarrillocruz | cos that is done by this change: https://review.openstack.org/#/c/327472/7 | 13:50 |
| andreykurilin | mordred: thanks for discussion and training for mailing-fight:) | 13:50 |
| clarkb | yay centos/fedora python | 13:50 |
| mordred | andreykurilin: I'll happily respond to the thread and say that having the discussion is not crazy. you are not the only one to express frustration with testr or the rate of problems being fixed. however, there are other things, such as how parallel testing works, etc, that _all_ need to be considered | 13:50 |
| *** itisha has quit IRC | 13:50 | |
| *** dizquierdo has joined #openstack-infra | 13:50 | |
| mordred | andreykurilin: thanks for being willing to have your side of the discussion - it's a hard topic :) | 13:50 |
| *** Daisy has joined #openstack-infra | 13:50 | |
| *** PollyZ has joined #openstack-infra | 13:50 | |
| rcarrillocruz | so i'm not sure if we should revert ^ or tweak devstack to not remove the pckage. TBH, i think latter is quicker but wrong, cos it may break on other centos that are not using our dib images | 13:50 |
| *** zeih has quit IRC | 13:51 | |
| clarkb | rcarrillocruz: I think you need to talk to dib then | 13:51 |
| *** zeih has joined #openstack-infra | 13:51 | |
| rcarrillocruz | ianw_pto looks on vaca per his nick, greghaynes you around ? | 13:52 |
| clarkb | also doesnt py.test not use standard python unittest? | 13:53 |
| rcarrillocruz | i'm surprised if this is not hitting other people on dsvm centos jobs | 13:53 |
| *** xarses has quit IRC | 13:53 | |
| mordred | clarkb: no, you can use standard unittest with py.test these days | 13:53 |
| clarkb | mordred: oh good | 13:53 |
| clarkb | that was the other issue with using it when I looked | 13:53 |
| clarkb | I wasnt going to rewrote literally every test | 13:54 |
| rubasov | hello | 13:55 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 13:55 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add Carlos Camacho to TripleO blogs feed https://review.openstack.org/336904 | 13:55 |
| *** EricGonczer_ has joined #openstack-infra | 13:55 | |
| *** Daisy has quit IRC | 13:55 | |
| rubasov | am I at the right channel if I'm looking for a gerrit admin? | 13:55 |
| *** links has joined #openstack-infra | 13:56 | |
| clarkb | rubasov: yes | 13:56 |
| rubasov | clarkb: thanks | 13:56 |
| rubasov | I have the same problem as described here: https://bugs.launchpad.net/openstack-ci/+bug/1394191 | 13:56 |
| openstack | Launchpad bug 1394191 in OpenStack Core Infrastructure "can't be added as a gerrit reviewer " [Medium,Fix released] - Assigned to Jeremy Stanley (fungi) | 13:56 |
| *** tonytan4ever has joined #openstack-infra | 13:56 | |
| *** Julien-zte has joined #openstack-infra | 13:56 | |
| *** roxanagh_ has joined #openstack-infra | 13:57 | |
| rubasov | in short: if somebody tries to invite me for a review on review.openstack.org he gets this error: Bence Romsics <bence.romsics@ericsson.com> does not identify a registered user or group | 13:57 |
| fungi | clarkb: the point of the mysql-prep macro is to prepare mysql with known admin credentials so that jobs can use it however they see fit. the mysql-db macro is a transitional macro purely to provide a database and account that some jobs were expecting even though they could have made their own instead | 13:57 |
| clarkb | fungi: ya but does mysql-db run mysql-prep? | 13:57 |
| fungi | clarkb: nope, they're independent. you need to run mysql-prep first, then mysql-db. the hope is that projects relying on mysql-db will updat etheir jobs in time so we can just drop that macro entirely | 13:58 |
| rubasov | I'd like to ask somebody to "disable the second account" just like in the ticket | 13:58 |
| *** links has quit IRC | 13:59 | |
| fungi | rubasov: i'm checking the database now to see how many accounts we have for you | 13:59 |
| EmilienM | yolanda: can you -A and +A again please ? https://review.openstack.org/#/c/337209/ | 13:59 |
| clarkb | fungi: AJaeger frickler ok in that case we should update the secpnd 127.0.0.1 since they are not srrictly tied together | 13:59 |
| rubasov | fungi: thanks, I wouldn't be surprised if I had two, I had some complications when I registered first | 13:59 |
| pabelanger | rcarrillocruz: looks like there is another reported issue related to you: https://review.openstack.org/#/c/336586/ | 14:00 |
| pabelanger | rcarrillocruz: not the same, but a regression related to the new code | 14:00 |
| rcarrillocruz | yay | 14:00 |
| *** roxanagh_ has quit IRC | 14:01 | |
| pabelanger | rcarrillocruz: I'd say, get a revert up and confirm with greghaynes. Then we can do a point release and maybe push new images to tomorrow | 14:01 |
| rcarrillocruz | ++ | 14:01 |
| *** Daisy has joined #openstack-infra | 14:02 | |
| *** isaacb has quit IRC | 14:03 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add valid set of options parameters to convert_xml https://review.openstack.org/337381 | 14:03 |
| *** cardeois_ has joined #openstack-infra | 14:03 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack/diskimage-builder: Revert "Pre-install pip/virtualenv packages" https://review.openstack.org/337677 | 14:03 |
| *** hashar is now known as hasharAway | 14:04 | |
| rcarrillocruz | erm, yeah, via a change, sec | 14:04 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add valid set of options parameters to convert_xml https://review.openstack.org/337381 | 14:04 |
| fungi | rubasov: yep, two accounts (8150 and 15554). the newer one (15554/ebenrom from march of last year) seems to be getting used for things. the older one (8150/bence.romsics which seems to be from sometime in mid 2013) has probably been dormant for a while. it looks like the second account probably happened when you stopped logging into launchpad with your gmail.com address and used your ericsson.com | 14:05 |
| fungi | address instead | 14:05 |
| *** woodster_ has joined #openstack-infra | 14:05 | |
| rubasov | fungi: could you please delete the one with the gmail.com address? | 14:06 |
| fungi | rubasov: i can disable it, yes. just a sec | 14:06 |
| rubasov | fungi: that's probably the older and unused | 14:06 |
| *** cardeois has quit IRC | 14:06 | |
| *** Daisy has quit IRC | 14:07 | |
| *** jsavak has quit IRC | 14:07 | |
| fungi | rubasov: okay, the older 8150 account is now disabled and i've reassociated its openid with the newer 15554 account just in case you accidentally try to log in with it again | 14:08 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP 'All Boards and Worklists' Page https://review.openstack.org/279753 | 14:08 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add valid set of options parameters to convert_xml https://review.openstack.org/337381 | 14:08 |
| rubasov | fungi: thanks a lot for your help! | 14:08 |
| *** hongbin has joined #openstack-infra | 14:08 | |
| clarkb | fungi: when you are dome wrangling accounts maybe you can review 337648 | 14:09 |
| fungi | sounds exciting | 14:10 |
| andreykurilin | mordred: btw, it looks like you are familiar with subunit, so be ready, I'll ping you someday when subunit-output for rally will be ready:) | 14:10 |
| EmilienM | AJaeger: thx | 14:11 |
| AJaeger | you're welcome, EmilienM | 14:11 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack/diskimage-builder: Revert "Pre-install pip/virtualenv packages" https://review.openstack.org/337686 | 14:12 |
| clarkb | fungi: AJaeger frickler we should also test that on trusty and centos before merging | 14:12 |
| clarkb | I can help with that once on a proper computer | 14:12 |
| *** claudiub|2 has joined #openstack-infra | 14:12 | |
| fungi | clarkb: okay, maybe not _that_ exciting. after all, its largest prime factor is only 9 bits long | 14:13 |
| AJaeger | clarkb: agreed. | 14:13 |
| *** claudiub has quit IRC | 14:13 | |
| *** permalac has quit IRC | 14:14 | |
| AJaeger | clarkb: that's why I didn't leave any further comments on it - I can't help with testing on that, and it should be tested... | 14:14 |
| *** cardeois_ has quit IRC | 14:15 | |
| * AJaeger still does not understand why this helps, this looks odd | 14:15 | |
| *** cardeois_ has joined #openstack-infra | 14:15 | |
| *** burgerk has joined #openstack-infra | 14:15 | |
| clarkb | AJaeger: it is due to mysqls authentication being source specific | 14:16 |
| clarkb | localhost likely uses the local socket and 127.0.0.1 uses tcp | 14:16 |
| fungi | mysql's users table has entries specific to the access mechanism. localhost connects over a socket | 14:16 |
| fungi | er, what clarkb just said | 14:16 |
| rcarrillocruz | greghaynes: when you are around https://review.openstack.org/337686 . That change breaks dsvm jobs , when tools/fixup_stuff.sh removes virtualenv system package and installs it from pip again, you can see the issue on http://logs.openstack.org/35/332435/6/check/gate-ansible-role-cloud-launcher-dsvm-ansible-func-centos-7/e1c39ef/logs/devstacklog.txt.gz | 14:17 |
| AJaeger | Thanks, the socket for localhost explains it! | 14:17 |
| AJaeger | Worth noting in the review... | 14:17 |
| fungi | chances are mysql is configured not to listen on the loopback in xenial. has anyone looked? | 14:17 |
| AJaeger | fungi, apparently: So, we could limit to 5x12 support but need to cover both regions. | 14:18 |
| *** zhurong has quit IRC | 14:18 | |
| clarkb | fungi: I have not, assuming frickler did but dont know | 14:18 |
| AJaeger | wrong paste;( | 14:18 |
| *** Daisy has joined #openstack-infra | 14:18 | |
| fungi | if it is listening on the loopback, then it's probably just missing a users table entry for root@127.0.0.1 | 14:18 |
| AJaeger | So, we could limit to 5x12 support but need to cover both regions. | 14:18 |
| AJaeger | trying again: https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#MySQL_5.7 | 14:19 |
| frickler | fungi: mysql does listen, but with -h localhost it connects via unix socket and the permissions are different then | 14:19 |
| mordred | andreykurilin: fwiw, I have just tested - the xdist parallel plugin works, as does the subunit plugin - but they do not work at the same time | 14:19 |
| frickler | this only affects setting up the admin account using the root account | 14:19 |
| AJaeger | fungi, clarkb https://anonscm.debian.org/cgit/pkg-mysql/mysql.git/tree/debian/NEWS | 14:19 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add rcarrillocruz as infra-root https://review.openstack.org/337692 | 14:20 |
| *** mriedem has joined #openstack-infra | 14:20 | |
| fungi | frickler: yes, i was asking specifically whether it's missing a root@127.0.0.1 in the users table, or simply not listening on the loopback at all | 14:20 |
| mordred | andreykurilin: also, ctrl-c does not actually abort running the tests - I had to use kill to kill an run | 14:20 |
| andreykurilin | mordred: thanks for research. I can answer you as future replies to my thread: "it is opensource, you can fix it:)" lol | 14:21 |
| *** dmellado has quit IRC | 14:21 | |
| mordred | heh | 14:21 |
| *** piet has joined #openstack-infra | 14:21 | |
| *** dmellado has joined #openstack-infra | 14:21 | |
| *** oanson has quit IRC | 14:21 | |
| frickler | fungi: the former, see the link AJaeger just posted | 14:22 |
| * mordred imagines a future where andreykurilin and mordred sit in a room with a ton of alcohol and come out the other side having fixed both py.test and testr | 14:22 | |
| dhellmann | are we seeing any unusual delay between approving patches and having them queued up in the gate? I don't see a big event backlog on the zuul status page, but I approved https://review.openstack.org/#/c/336505/ about 15 minutes ago and I don't see it being processed yet | 14:22 |
| fungi | frickler: until we confirm that _all_ our supported platforms will accept empty root password auth on the socket, we can't assume this is a safe change (also, that account configuration routine was copied from devstack, so chances are that needs fixing or already has been) | 14:22 |
| andreykurilin | mordred: actually, ctrl+c works unstable in tox+testr too. Sometimes it doesn't kill processes | 14:22 |
| *** esikachev has quit IRC | 14:22 | |
| andreykurilin | mordred: heh | 14:22 |
| dhellmann | sigh, nm, that's at the end of a patch series | 14:22 |
| *** pilgrimstack1 has joined #openstack-infra | 14:23 | |
| *** Daisy has quit IRC | 14:23 | |
| *** pilgrimstack has quit IRC | 14:24 | |
| *** EricGonczer_ has quit IRC | 14:24 | |
| *** lucasagomes has quit IRC | 14:25 | |
| clarkb | fungi: its not wmpty is it? I think the first mysqladmin line sets a passwd | 14:25 |
| mordred | clarkb: I want wmpty to be a new word | 14:26 |
| notnownikki | wmpty == accidentally left empty | 14:26 |
| clarkb | mordred: it is on android | 14:27 |
| pabelanger | mordred: fungi: clarkb: Wouldn't mind your thoughts on 337286. Basically adding reprepro bits to our release slave so we can create packaging repos for the packaging-deb team. If we don't want to use our release slave, I'm open to another (new?) one. | 14:27 |
| fungi | clarkb: yep, it does. i'm trying to find a log demonstrating this behavior but frickler didn't include one in gerrit | 14:28 |
| fungi | do we have a demo of how this is failing currently? | 14:28 |
| clarkb | fungi: let me dig up a link | 14:28 |
| AJaeger | fungi, http://logs.openstack.org/06/317306/12/check/gate-nova-python35-db-nv/2c4e88a/console.html#_2016-07-05_10_57_44_044523 | 14:29 |
| AJaeger | from scroll-back | 14:29 |
| fungi | AJaeger: oh, thanks. i'll link that in a review comment | 14:29 |
| *** isaacb has joined #openstack-infra | 14:29 | |
| *** dmellado has quit IRC | 14:29 | |
| *** Daisy has joined #openstack-infra | 14:29 | |
| clarkb | fungi: http://logs.openstack.org/63/323963/2/check/gate-neutron-python35-nv/ca3482e/ | 14:30 |
| *** jamesmcarthur has joined #openstack-infra | 14:30 | |
| clarkb | oh AJaeger has a more direct link | 14:30 |
| *** dmellado has joined #openstack-infra | 14:30 | |
| fungi | okay, so the mysqladmin call seems to be working, but isn't adding loopback interface access | 14:31 |
| *** lucasagomes has joined #openstack-infra | 14:31 | |
| fungi | i guess it's just updating the password for root via socket | 14:31 |
| fungi | and not root via network | 14:32 |
| *** kgiusti has joined #openstack-infra | 14:32 | |
| clarkb | oh wait neutron unittests dont db anymore? weird | 14:32 |
| *** eezhova has quit IRC | 14:33 | |
| *** sigmavirus_away is now known as sigmavirus | 14:33 | |
| *** sigmavirus is now known as bops | 14:33 | |
| mtreinish | clarkb: they do that in their functional tests. It's been that way for a while | 14:33 |
| *** bops is now known as sigmavirus | 14:34 | |
| AJaeger | fungi, I have to leave soon, so a short topic switch: I've noticed a lot of timeouts in rax-iad again today and put up a change to disable rax-iad just in case we want to go there. That change contains also as review comments a couple of log files that failed: https://review.openstack.org/#/c/337513/ . I don't know what percentage is failing, I just checked that every fail was in rax-iad. | 14:34 |
| *** dkehn__ is now known as dkehnp | 14:34 | |
| mtreinish | clarkb: http://git.openstack.org/cgit/openstack/neutron/tree/neutron/tests/functional/db | 14:34 |
| *** dkehn_ is now known as dkehn | 14:34 | |
| *** Daisy has quit IRC | 14:34 | |
| *** dkehnp is now known as dkehn_ | 14:34 | |
| clarkb | mtreinish: cool likely a mich better plqce for them | 14:35 |
| openstackgerrit | Henry Gessau proposed openstack-infra/reviewday: Add neutron-lib project to neutron review dashboard https://review.openstack.org/337698 | 14:36 |
| *** xenogear has joined #openstack-infra | 14:36 | |
| *** jesusaur has quit IRC | 14:37 | |
| *** mdrabe has quit IRC | 14:37 | |
| *** mgould has quit IRC | 14:38 | |
| *** mdrabe has joined #openstack-infra | 14:38 | |
| *** mgould has joined #openstack-infra | 14:38 | |
| *** rossella_s has quit IRC | 14:39 | |
| *** tonytan4ever has quit IRC | 14:39 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add valid set of options parameters to convert_xml https://review.openstack.org/337381 | 14:39 |
| *** jesusaur has joined #openstack-infra | 14:39 | |
| *** rossella_s has joined #openstack-infra | 14:39 | |
| *** tonytan4ever has joined #openstack-infra | 14:39 | |
| pabelanger | AJaeger: I think 326095 is ready to land now. Having a 2-node centos job is better then 3 to start with | 14:39 |
| isaacb | mordred, clarkb, pabelanger: Can you please review: https://review.openstack.org/#/c/321480/ | 14:41 |
| openstackgerrit | Brent Eagles proposed openstack-infra/tripleo-ci: Exit with non-zero status on create and update errors https://review.openstack.org/319352 | 14:41 |
| andreykurilin | mordred: sorry, but I could not do otherwise :) | 14:41 |
| *** jtomasek_ has quit IRC | 14:41 | |
| andreykurilin | mordred: https://twitter.com/andrey_kurilin/status/750338383222804480 | 14:41 |
| anteaya | fungi: um, I'm looking at the meeting agenda for today and I seeing an agenda item posted in open discussion, which personally I always consider odd as if someone has an agenda item I see it as deserving of a place in the agenda | 14:41 |
| anteaya | fungi: do you mind if I move the agenda item out of open discussion and then I can add my item after it? | 14:42 |
| fungi | anteaya: yep, i don't know why people do that. i assume it's just a mistake and i generally correct it when i'm cleaning up the agenda | 14:42 |
| anteaya | I would like to add an agenda item and don't want to jump the queue | 14:42 |
| anteaya | I will fix and append my item below, thank you | 14:42 |
| anteaya | jhesketh: ^^ | 14:42 |
| *** ddieterly has joined #openstack-infra | 14:43 | |
| fungi | frickler: have you tested this yet? based on my reading, the default configuration change i'm reading is that it no longer uses password authentication, instead it checks the calling local user opening the socket | 14:43 |
| *** itisha has joined #openstack-infra | 14:43 | |
| fungi | for root specifically | 14:43 |
| frickler | fungi: I tested on trusty and xenial, works fine for me there | 14:44 |
| mordred | andreykurilin: :) | 14:44 |
| *** PollyZ has quit IRC | 14:44 | |
| fungi | frickler: have you tried just dropping the -u and -p options entirely as well? it looks like that's supposed to work | 14:44 |
| mordred | pabelanger: so - your thinking is that you'll just run the reprepro commands on the release slave and then do the vos release? | 14:45 |
| *** PollyZ has joined #openstack-infra | 14:45 | |
| fungi | frickler: anyway, i don't think we can assume that some jobs aren't using the root account, so the fix is going to be making sure we also set up the root account for access via 127.0.0.1, i think | 14:45 |
| *** Daisy has joined #openstack-infra | 14:46 | |
| *** raildo is now known as raildo-afk | 14:47 | |
| *** raildo-afk is now known as raildo | 14:47 | |
| openstackgerrit | dane-fichter proposed openstack-infra/project-config: Add cursive library https://review.openstack.org/337709 | 14:47 |
| clarkb | it looks like mysqladmin password updates existing grant table rows | 14:48 |
| clarkb | so xenial mysql likely missing those rows by default | 14:48 |
| fungi | yep, i'm adding suggested queries in a review comment now | 14:49 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Remove unused variable https://review.openstack.org/337656 | 14:49 |
| *** jamesmcarthur has quit IRC | 14:50 | |
| pabelanger | mordred: Right, either that node or another. I cannot use mirror-update.o.o today since zuul-launcher can't contact it | 14:50 |
| *** Daisy has quit IRC | 14:51 | |
| *** piet has quit IRC | 14:51 | |
| *** bunting has quit IRC | 14:51 | |
| mordred | pabelanger: yah. makes sense to me - and looks good | 14:52 |
| *** esikachev has joined #openstack-infra | 14:53 | |
| *** dane-fichter has joined #openstack-infra | 14:53 | |
| *** pilgrimstack1 has quit IRC | 14:53 | |
| pabelanger | mordred: I'll get up the other 1/2 of the patch later today | 14:53 |
| *** bunting has joined #openstack-infra | 14:53 | |
| *** chem` is now known as chem | 14:53 | |
| *** akshai has joined #openstack-infra | 14:53 | |
| *** pilgrimstack has joined #openstack-infra | 14:54 | |
| mordred | \o/ | 14:55 |
| fungi | mordred: mysql specialist feedback on 337648 requested | 14:56 |
| mordred | uhoh | 14:56 |
| AJaeger | bbl | 14:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Disable rax-iad https://review.openstack.org/337513 | 14:56 |
| mordred | fungi: anything in particular? or just want the sanity check? | 14:57 |
| *** dprince has quit IRC | 14:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Disable rax-iad temporarily https://review.openstack.org/337513 | 14:57 |
| *** Daisy has joined #openstack-infra | 14:57 | |
| mordred | fungi: +2 from me | 14:58 |
| *** dprince has joined #openstack-infra | 14:58 | |
| fungi | mordred: yeah, sanity check that granting for root@% works like i expect. apparently in xenial, they scaled root back to only be root@localhost (identified by socket) | 14:58 |
| fungi | so hoping my suggestion on that change is sane | 14:58 |
| fungi | my inline comment on it, specifically | 14:59 |
| *** psachin has quit IRC | 14:59 | |
| *** piet has joined #openstack-infra | 15:00 | |
| dane-fichter | Reading the project creator's guide. Can anyone explain where/when the git repo is created if you don't specify an upstream source? | 15:00 |
| mordred | fungi: I do not see an inline comment from you | 15:00 |
| *** cardeois__ has joined #openstack-infra | 15:01 | |
| *** xarses has joined #openstack-infra | 15:01 | |
| *** jamesmcarthur has joined #openstack-infra | 15:01 | |
| clarkb | dane-fichter: it just creates a new repo with a .gitreview file | 15:01 |
| clarkb | mordred: at https://review.openstack.org/#/c/337648/1/jenkins/jobs/macros.yaml | 15:02 |
| *** jamesmcarthur has quit IRC | 15:02 | |
| *** Daisy has quit IRC | 15:02 | |
| dane-fichter | clarkb: at what point in the process does this occur? who creates the new repo? | 15:02 |
| *** jamesmcarthur has joined #openstack-infra | 15:02 | |
| *** Daisy has joined #openstack-infra | 15:03 | |
| jeblair | i do see it | 15:03 |
| rcarrillocruz | an automated script does | 15:03 |
| rcarrillocruz | dane-fichter: ^ | 15:03 |
| *** cardeois_ has quit IRC | 15:04 | |
| dane-fichter | rcarillocruz: when the initial porject-config patch merges? | 15:04 |
| openstackgerrit | Merged openstack-infra/project-config: Retire puppet-openstack-release-tools https://review.openstack.org/337209 | 15:05 |
| pabelanger | dane-fichter: about 15-30mins after the patch is merged | 15:05 |
| *** ddieterly is now known as ddieterly[away] | 15:05 | |
| dane-fichter | pabelanger: And then you configure the gerrit groups and clone the repo from stackforge's github in order to merge the first commit? | 15:06 |
| rcarrillocruz | it's not a hook on merge, but a cron script iirc | 15:06 |
| pabelanger | dane-fichter: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py is the code that handles new projects | 15:06 |
| rcarrillocruz | the clone is done on project creation | 15:06 |
| pabelanger | right | 15:06 |
| openstackgerrit | Jef Mallal proposed openstack-infra/jenkins-job-builder: Add support for retry in multijob plugin https://review.openstack.org/329169 | 15:07 |
| rcarrillocruz | so are the gerrit groups | 15:07 |
| rcarrillocruz | once all is done, we just add you as a member of the needed group | 15:07 |
| dane-fichter | hmm ok | 15:07 |
| rcarrillocruz | which in turn allows you to add other members to the group | 15:07 |
| *** makowals has quit IRC | 15:07 | |
| pabelanger | It would be nice if we added something like upstream into our yaml file but for the initial user into gerrit | 15:08 |
| pabelanger | then our bot normalize it everyday | 15:08 |
| clarkb | pabelanger: we srt of intentionally let those groups be self managed, if we want t oset it once at creation thats fine but I wouldn't enforce after that | 15:08 |
| *** aysyd has quit IRC | 15:09 | |
| pabelanger | clarkb: Agreed, at creation would be all I would want to do | 15:09 |
| pabelanger | going to hack on that this week, see how hard it would be | 15:09 |
| *** solumuser__ has quit IRC | 15:10 | |
| jeblair | yeah, adding temporary data like that is imperfect, but considering the volume of requests related to that, i'd support doing it. | 15:10 |
| rcarrillocruz | ++ | 15:10 |
| *** hichihara has quit IRC | 15:10 | |
| *** aysyd has joined #openstack-infra | 15:10 | |
| fungi | yeah, we'd avoided it in the past because it's a piece of information which makes no sense after the initial creation of teh group, but i guess if we clean those up at teh same time we clean up the upstream urls (which are usually in the same vein) it seems not completely terrible | 15:10 |
| *** isaacb has quit IRC | 15:11 | |
| *** Jeffrey4l_ has quit IRC | 15:11 | |
| *** roxanagh_ has joined #openstack-infra | 15:11 | |
| mordred | ++ | 15:12 |
| mordred | fungi: responded | 15:12 |
| *** scottda has joined #openstack-infra | 15:12 | |
| *** scottda is now known as scottda_vacation | 15:13 | |
| fungi | mordred: thanks! | 15:13 |
| *** Daisy has quit IRC | 15:13 | |
| clarkb | frickler: ^ are you able to update that today or should someone adopt the change to get it in soon? | 15:13 |
| *** wfoster has quit IRC | 15:15 | |
| *** jamesmcarthur has quit IRC | 15:15 | |
| *** roxanagh_ has quit IRC | 15:16 | |
| greghaynes | rcarrillocruz: if we can get some kind of confirmation that the revert fixes the issue then im fine merging | 15:16 |
| greghaynes | rcarrillocruz: the issue is that virtualenv is just not installed afterwards or ? | 15:16 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Project api https://review.openstack.org/337730 | 15:16 |
| clarkb | greghaynes: the issue is that centos/fedora don't use /usr/local for pip installed things so you install system package to /usr, install pip package to /usr then uninstall system package from /usr which uninstalls everything including the pip package | 15:17 |
| clarkb | greghaynes: if you don't preinstall the system package then the delete fails because package is not installed | 15:17 |
| greghaynes | clarkb: oh. who does the uninstall? | 15:17 |
| clarkb | (even though the pip package is on disk at /usr) | 15:17 |
| clarkb | greghaynes: I think rcarrillocruz was saying devstack does that | 15:17 |
| *** markusry has quit IRC | 15:18 | |
| greghaynes | nice | 15:18 |
| *** jamesmcarthur has joined #openstack-infra | 15:18 | |
| frickler | clarkb: I was just trying to do some more testing, seems getting root@127.0.0.1 to work on xenial is non-trivial | 15:18 |
| * fungi is starting to wonder why we don't just install pip into a virtualenv and call it directly there in all our jobs | 15:18 | |
| clarkb | frickler: woo, ty for testing :) | 15:18 |
| frickler | clarkb: but if you want to take over the patch, I'm fine with that | 15:19 |
| clarkb | frickler: mostly just thinking you are near the end of your day if I remember correctly and didn't want this to be left alone for half a day | 15:19 |
| fungi | oh, right, then it would install everything into that virtualenv | 15:19 |
| greghaynes | rcarrillocruz: clarkb I think ianw_pto's comments explain why you want to do it the way https://review.openstack.org/#/c/337686/1/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/01-install-pip explains. We probably shouldnt up and break the world by changing that behavior though | 15:19 |
| fungi | d'oh | 15:19 |
| rcarrillocruz | greghaynes: if you hold a node on centos and do this: yum install python-virtualenv; pip install virtualenv; yum remove python-virtualenv would show you the issue | 15:19 |
| rcarrillocruz | http://paste.openstack.org/show/525998/ | 15:19 |
| rcarrillocruz | ^ | 15:19 |
| clarkb | frickler: but feel free to keep working it, maybe just let us know when you have to drop off and one of us can pick it up in the interim | 15:19 |
| rcarrillocruz | if you hold a centos node you can follow those steps and see | 15:19 |
| *** Daisy has joined #openstack-infra | 15:19 | |
| rcarrillocruz | you may have to change /etc/yum.conf tho, as the change itself excludes python-virtualenv | 15:20 |
| clarkb | another alternative is to stop running the package delete | 15:20 |
| greghaynes | Yea, the missing bit was that some consumers of the images perform a yum remove | 15:20 |
| rcarrillocruz | devstack has tools/fixup_stuff.sh , that does remove the python-virtualenv package to install it from pip | 15:20 |
| greghaynes | clarkb: yes, I think thats what we want, but we can probably revert then try and work with devstack | 15:21 |
| rcarrillocruz | but the dib image already has installed the pakcage and overwrote it with pip package | 15:21 |
| rcarrillocruz | it's messy | 15:21 |
| *** esikachev has quit IRC | 15:21 | |
| clarkb | or to make sure the remove happens before any pip installs | 15:21 |
| anteaya | <-- running errands, expect to be back for the last bit of the infra meeting | 15:22 |
| greghaynes | wait, so devstack must do a remove and then a pip install against but that install doesnt notice the entrypoint missing? | 15:22 |
| greghaynes | s/against/again | 15:22 |
| clarkb | greghaynes: I think the problem is that it does the remove in suhc a way that the subsequent install has no code to back it so it fails | 15:23 |
| clarkb | because yum is deleting all of those files | 15:23 |
| greghaynes | yes, but there must also be a pip install later on, otherwise devstack just does a yum remove of virtualenv and then expects it to be there | 15:24 |
| *** jamesmcarthur has quit IRC | 15:24 | |
| *** sorantis has quit IRC | 15:24 | |
| *** jamesmcarthur has joined #openstack-infra | 15:24 | |
| *** Daisy has quit IRC | 15:24 | |
| clarkb | yes in devstack/tools/fixup_stuff.sh at the end of the file is where we break I think | 15:25 |
| clarkb | if we remove the second to last line in that I believe everything will just magically work | 15:25 |
| rcarrillocruz | yup | 15:25 |
| clarkb | on debuntu and red hat thingies | 15:25 |
| clarkb | thingee: ^ you are now a linux distro | 15:26 |
| *** pilgrimstack has quit IRC | 15:26 | |
| jeblair | better not run systemd | 15:26 |
| *** pilgrimstack has joined #openstack-infra | 15:26 | |
| clarkb | in other news horizon wins the py35 cookie | 15:27 |
| clarkb | their tests appear to just work :) | 15:27 |
| fungi | horizon knows its cookies | 15:27 |
| greghaynes | yea, so the pip install must not notice that it needs to re-install everything. It seems like devstack should do the same thing dib does and maybe check state before the removal due to ianw's comments | 15:27 |
| greghaynes | its a bit safer | 15:27 |
| rcarrillocruz | anyway, i'm stepping for a bit and will be back for meeting | 15:27 |
| clarkb | haven't found any other "server" projects in that same boat but in their defense horizon doesn't have a db to worry about | 15:27 |
| *** Julien-zte has quit IRC | 15:27 | |
| greghaynes | I am fine reverting for now to unbreak stuff though and working with devstack later | 15:28 |
| rcarrillocruz | i'll check it out later greghaynes clarkb | 15:28 |
| frickler | clarkb: neutron is fine too, merged my patch for py35 just in time ;) | 15:28 |
| rcarrillocruz | brb | 15:28 |
| clarkb | frickler: oh the example I saw failed, maybe the one I saw ran just before that merge | 15:28 |
| *** yolanda has quit IRC | 15:28 | |
| clarkb | frickler: also possible that the change in quetion is broken for py35 http://logs.openstack.org/63/323963/2/check/gate-neutron-python35-nv/ca3482e/console.html#_2016-07-05_13_21_10_463912 | 15:29 |
| *** esikachev has joined #openstack-infra | 15:29 | |
| frickler | clarkb: http://logs.openstack.org/68/337368/2/check/gate-neutron-python35-nv/54d282c/ is fine | 15:29 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Complex priorities UI in stories https://review.openstack.org/312666 | 15:29 |
| frickler | clarkb: yours is failing for py27 and py34, too | 15:30 |
| *** yolanda has joined #openstack-infra | 15:30 | |
| clarkb | frickler: aha then that is likely the issue | 15:30 |
| openstackgerrit | dane-fichter proposed openstack-infra/project-config: Add cursive library https://review.openstack.org/337709 | 15:30 |
| openstackgerrit | Artur Zarzycki proposed openstack-infra/project-config: Add new project Fuel CCP https://review.openstack.org/335584 | 15:31 |
| frickler | clarkb: so with the db patch, I don't think I can make more progress with it anytime soon. I'd propose to merge it as is as the first iteration and see what will work after that and what doesn't | 15:31 |
| greghaynes | rcarrillocruz: commented on https://review.openstack.org/#/c/337686/1 | 15:31 |
| clarkb | fungi: AJaeger ^ what do you think? I am happy with that any tests that rely on 127.0.0.1 loopback being available will shake themselves out once we get that in | 15:32 |
| fungi | clarkb: i'm okay giving it a shot, just need to be ready to emergency revert if it causes db setup to start breaking for voting jobs on, e.g., centos/fedora/whatever | 15:33 |
| *** openstackgerrit has quit IRC | 15:33 | |
| *** mrmartin has joined #openstack-infra | 15:33 | |
| *** openstackgerrit has joined #openstack-infra | 15:33 | |
| clarkb | fungi: ya that should only happen if we refuse root@localhost connections which would be really odd | 15:33 |
| *** amotoki has quit IRC | 15:34 | |
| fungi | clarkb: or if the mysqladmin password change isn't setting a password for socket connections on some platform | 15:34 |
| clarkb | I can really quickly test it if we ant to be extra careful | 15:35 |
| *** jed56 has quit IRC | 15:35 | |
| pabelanger | looks like we need to update our sshd config on xenial. I am seeing the following from our root user: | 15:35 |
| clarkb | I can boot our images and run the code and make sure that I can login as that test user after | 15:35 |
| *** dprince has quit IRC | 15:35 | |
| pabelanger | Jul 5 08:35:42 wheel-mirror-ubuntu-xenial-amd64 sshd[18012]: Authentication tried for root with correct key but not from a permitted host (host=2001:4800:7818:101:3c21:a454:23ed:4072, ip=2001:4800:7818:101:3c21:a454:23ed:4072). | 15:35 |
| pabelanger | I confirmed the public key is correct | 15:35 |
| *** zeih has quit IRC | 15:35 | |
| fungi | i did a lot of rigorous testing of those macros on all our supported platforms back when i wrote them, also tested running multiple times to make sure they're idempotent | 15:36 |
| *** jamesmcarthur has quit IRC | 15:36 | |
| *** dprince has joined #openstack-infra | 15:36 | |
| *** ddieterly[away] is now known as ddieterly | 15:36 | |
| clarkb | I am not worried about idempotency in this change | 15:36 |
| fungi | pabelanger: maybe connecting via ipv6 instead? | 15:36 |
| pabelanger | looks like http://git.openstack.org/cgit/openstack-infra/puppet-ssh/tree/templates/sshd_config.erb#n89 is not resolving dns properly | 15:36 |
| clarkb | the only thing that should matter is whether or not we allow connections on the local socket | 15:36 |
| fungi | pabelanger: er, i mean ipv4 | 15:37 |
| clarkb | fungi: do you know why you didn't use the local socket? | 15:37 |
| pabelanger | fungi: possible, i can check that | 15:37 |
| fungi | clarkb: i didn't use the unix socket because devstack didn't use the unix socket and the puppet-mysql module didn't appear to do socket either, so i was attempting to make this as close as possible to what we had previously been running | 15:38 |
| clarkb | fungi: gotcha, so it wasn't due to a known issue with socket useability across platforms | 15:38 |
| *** sshnaidm is now known as sshnaidm|afk | 15:38 | |
| clarkb | but maybe those tools chose it because of one | 15:38 |
| fungi | clarkb: it might have been, but not on my part | 15:38 |
| clarkb | ya | 15:38 |
| clarkb | ok let me run some tests of these macros with the change on our imgaes | 15:39 |
| fungi | i was trying not to introduce any new unknowns at the time | 15:39 |
| *** chem has quit IRC | 15:39 | |
| *** null0 has joined #openstack-infra | 15:40 | |
| *** harlowja has joined #openstack-infra | 15:40 | |
| *** null0 has left #openstack-infra | 15:40 | |
| *** chem has joined #openstack-infra | 15:41 | |
| *** rook- has joined #openstack-infra | 15:41 | |
| *** mikelk has quit IRC | 15:41 | |
| *** jamesmcarthur has joined #openstack-infra | 15:41 | |
| *** mrmartin has quit IRC | 15:42 | |
| *** sshnaidm|afk has quit IRC | 15:43 | |
| clarkb | dtroyer: osc feature request, make specifying a netowrk for server create simpler | 15:43 |
| *** zz_dimtruck is now known as dimtruck | 15:43 | |
| mordred | clarkb: ++ | 15:45 |
| fungi | clarkb: pabelanger: zigo: this comment from Mithrandir reflects some of the concerns we had earlier about relying on httpredir.d.o... https://lists.debian.org/debian-devel/2016/07/msg00090.html | 15:45 |
| *** lezbar has joined #openstack-infra | 15:45 | |
| *** gyee has joined #openstack-infra | 15:45 | |
| mordred | clarkb, dtroyer: "openstack server create --net=my-network" or something similar would be my vote | 15:46 |
| clarkb | mordred: ya and keep --nic complicated stuff for people that want >1 nic or need fancy stuff | 15:46 |
| clarkb | fungi: and yet its the default? | 15:46 |
| *** vhosakot has joined #openstack-infra | 15:47 | |
| pabelanger | fungi: I thought we pinned to a debian mirror, let me check | 15:47 |
| fungi | clarkb: i don't think it's the default, but that may also depend on what you mean by "default" (default option presented when debian-installer prompts asking what mirror you want to use?) | 15:47 |
| mordred | clarkb: ++ | 15:47 |
| pabelanger | fungi: Oh, we did that only in nodepool.yaml. We should also do the change for configure_mirror.sh | 15:48 |
| fungi | clarkb: at least last time i installed, it tried to match me with a mirror in the same country as the locale i specified | 15:48 |
| pabelanger | since changing it, our debian-jessie DIB has not failed to build | 15:48 |
| clarkb | fungi: default that debootstrap is writing out | 15:48 |
| pabelanger | DIB_DISTRIBUTION_MIRROR: http://ftp.us.debian.org/debian/ what we are using today | 15:49 |
| clarkb | pabelanger: probably a decent choice for all but ovh gra1 | 15:49 |
| fungi | clarkb: i thought debootstrap copied the hostname from your sources.list outside the chroot if you didn't specify on eon the command line, but maybe that too has changed | 15:49 |
| pabelanger | ya | 15:49 |
| pabelanger | clarkb: guess it is time to mirror debian to AFS? | 15:49 |
| *** abregman has quit IRC | 15:50 | |
| pabelanger | don't think it would take much work to get it going now | 15:50 |
| fungi | clarkb: so... ftp.us.debian.org != httpredir.debian.org | 15:50 |
| *** jamesmcarthur has quit IRC | 15:51 | |
| fungi | ftp.us.d.o is just a dns round-robin across multiple official mirrors in the usa | 15:51 |
| *** amotoki has joined #openstack-infra | 15:51 | |
| *** amotoki has quit IRC | 15:51 | |
| *** amitgandhinz has joined #openstack-infra | 15:51 | |
| fungi | httpredir.debian.org is a global load balancer that tries to redirect you to the nearest mirror | 15:51 |
| *** Daisy has joined #openstack-infra | 15:52 | |
| EmilienM | I'm trying to see why zuul says the job fails on http://logs.openstack.org/99/336699/4/check/gate-puppet-tripleo-puppet-lint/6764f4f/console.html#_2016-07-05_15_28_20_098721 | 15:52 |
| *** esikachev has quit IRC | 15:52 | |
| EmilienM | sounds like a timeout thing again | 15:52 |
| pabelanger | EmilienM: http://logs.openstack.org/99/336699/4/check/gate-puppet-tripleo-puppet-lint/6764f4f/_zuul_ansible/ansible_log.txt is better | 15:52 |
| pabelanger | EmilienM: yes, networking issues in rax-iad today | 15:52 |
| pabelanger | I think we have a patch up to disable rax-iad | 15:53 |
| EmilienM | ok | 15:53 |
| pabelanger | https://review.openstack.org/#/c/337513/ | 15:53 |
| pabelanger | fungi: are you up to speed on the failures of the last 24 hours? ^ | 15:54 |
| clarkb | fungi: frickler AJaeger http://paste.openstack.org/show/526029/ is what I am going to use to test this on our images, how does that look? | 15:54 |
| *** jamesmcarthur has joined #openstack-infra | 15:54 | |
| clarkb | fungi: yes we switched intentionally to the us mirror from httpredir.debian.org | 15:54 |
| clarkb | fungi: because it was using httpredir.debian.org by default somewhere (debootstrap iirc) and that failed frequently in dib builds | 15:54 |
| *** rdx_matt has joined #openstack-infra | 15:55 | |
| fungi | clarkb: i suspect dib's debian-minimal element might have been getting set to that for misguided reasons | 15:55 |
| clarkb | fungi: that is also possible | 15:55 |
| clarkb | I might be confusing debootstrap element with debootstrap itself | 15:55 |
| *** yolanda has quit IRC | 15:55 | |
| clarkb | jeblair: pabelanger for testing this mysql stuff should I be using the zuul user or the jenkins user? | 15:56 |
| fungi | clarkb: bingo http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/debian-minimal/environment.d/10-debian-minimal.bash#n3 | 15:56 |
| fungi | clarkb: so it's the "default" dib has chosen for us | 15:56 |
| fungi | not a default behavior of debootstrap | 15:56 |
| *** yolanda has joined #openstack-infra | 15:56 | |
| *** adrian_otto has joined #openstack-infra | 15:56 | |
| jeblair | clarkb, pabelanger: i believe we're still using jenkins | 15:57 |
| *** Daisy has quit IRC | 15:57 | |
| fungi | yeah, afaik we don't have jobs running as "zuul" on nodes | 15:57 |
| fungi | though i left a comment on the change to start using the "zuul" account in gerrit that it should be safe to un-wip since we restarted zuul on friday | 15:58 |
| clarkb | cloudnull: I have an instance that appears ot be suffering from the ssh connection issues in osic. clarkb-test-xenial | 15:58 |
| frickler | clarkb: that paste looks fine to me | 15:59 |
| cloudnull | clarkb: looking now | 15:59 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 16:00 |
| clarkb | cloudnull: it is also possible that I suffer from pebkac but let me know what you find :) | 16:00 |
| fungi | clarkb: yep, that basically looks like the script i ran on lots of node representatives back when i first wrote those macros | 16:00 |
| *** jamesmcarthur has quit IRC | 16:00 | |
| *** cardeois__ is now known as cardeois | 16:00 | |
| fungi | but with -h localhost instead of -h 127.0.0.1 | 16:01 |
| *** sambetts is now known as sambetts|afk | 16:01 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Make py35 jobs voting on tempest https://review.openstack.org/337748 | 16:01 |
| *** Daisy has joined #openstack-infra | 16:03 | |
| *** trown is now known as trown|lunch | 16:03 | |
| adrian_otto | is there anyone here who can force-end a meeting that was accidentally left open in #openstack-meeting-alt ? | 16:04 |
| hongbin | We need to start the magnum meeting, but the last person forgot to end the meeting there | 16:04 |
| pleia2 | good morning | 16:05 |
| hongbin | AJaeger: ^^ | 16:05 |
| clarkb | adrian_otto: hongbin you should be able to after the prevous meeting runs for 6 0minutes | 16:05 |
| adrian_otto | apparently they started late, because meetbot is ignoring us. | 16:05 |
| *** sputnik13_ has joined #openstack-infra | 16:06 | |
| *** sdake has joined #openstack-infra | 16:06 | |
| pleia2 | should be able to at :08:43 | 16:06 |
| pleia2 | a couple more minutes | 16:06 |
| *** jamesmcarthur has joined #openstack-infra | 16:07 | |
| *** andreas_s has quit IRC | 16:07 | |
| pleia2 | ah, good, they closed it out :) | 16:07 |
| EmilienM | pabelanger: tripleo & puppet gates are failing a lot of times because of RAX issue :/ | 16:08 |
| EmilienM | can we consider to approve https://review.openstack.org/#/c/337513/ and restart nodepool ? | 16:09 |
| dmsimard | EmilienM: +1, this is painful | 16:09 |
| *** jlanoux has quit IRC | 16:09 | |
| clarkb | fungi: ok, none of this works on centos/fedora from what I can see due to mariadb != mysql | 16:10 |
| clarkb | I am about to test that with 127.0.0.1 to make sure this isn't a regression | 16:10 |
| *** sdake has quit IRC | 16:11 | |
| *** akshai has quit IRC | 16:11 | |
| mordred | clarkb: oh lovely | 16:11 |
| pabelanger | EmilienM: waiting on fungi to comment | 16:11 |
| clarkb | ya it doesn't work on centos/fedora anyways | 16:12 |
| *** roxanagh_ has joined #openstack-infra | 16:12 | |
| * clarkb restricts testing to debuntu for now | 16:12 | |
| *** Daisy has quit IRC | 16:13 | |
| *** ifarkas has quit IRC | 16:14 | |
| *** jamesmcarthur has quit IRC | 16:14 | |
| *** flepied has quit IRC | 16:15 | |
| *** sputnik13_ has quit IRC | 16:15 | |
| *** roxanagh_ has quit IRC | 16:16 | |
| *** sputnik13 has quit IRC | 16:17 | |
| clarkb | fungi: also doesn't appear to be idempotent due to ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) on the mysqladmin command | 16:19 |
| *** Daisy has joined #openstack-infra | 16:20 | |
| clarkb | it will run once without error on trusty and debian jessie but not twice | 16:20 |
| clarkb | (note thats not the new localhost, thats the built in mysqladmin uses localhost by default behavior) | 16:20 |
| *** aarefiev has quit IRC | 16:21 | |
| cloudnull | clarkb: can you keep that no IP instance up for a while? I have a few other folks looking into right now, and if they don't find something I will look into myself in a bit. | 16:21 |
| clarkb | cloudnull: I can, and thank you | 16:21 |
| *** aarefiev has joined #openstack-infra | 16:23 | |
| *** Daisy has quit IRC | 16:25 | |
| *** jamesmcarthur has joined #openstack-infra | 16:25 | |
| *** savihou has quit IRC | 16:25 | |
| *** Apoorva has joined #openstack-infra | 16:25 | |
| clarkb | cloudnull: hrm, I am beginning to wonder if this is pebkac since I launched a second instance and it too is suffering that way (but my trusty, debian, centos and fedora boots didn't so I dunno) | 16:25 |
| *** Daisy has joined #openstack-infra | 16:25 | |
| *** amitgandhinz has quit IRC | 16:26 | |
| clarkb | cloudnull: I will keep digging on my side as well | 16:26 |
| *** mtanino has joined #openstack-infra | 16:26 | |
| pleia2 | nice, I have a friend who work in an office a block away and they're complaining about the fire alarm testing in my building | 16:26 |
| pleia2 | how do you think I feel /o\ | 16:26 |
| *** amitgandhinz has joined #openstack-infra | 16:26 | |
| *** jpich has quit IRC | 16:26 | |
| pleia2 | it's been a noisy morning here | 16:26 |
| *** markusry has joined #openstack-infra | 16:27 | |
| *** karthikp_ has joined #openstack-infra | 16:27 | |
| *** yolanda has quit IRC | 16:28 | |
| *** sshnaidm|afk has joined #openstack-infra | 16:28 | |
| *** tphummel has joined #openstack-infra | 16:28 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Log stderr when creating and destroying envs https://review.openstack.org/337775 | 16:28 |
| *** rfolco_ has quit IRC | 16:29 | |
| *** rfolco has joined #openstack-infra | 16:29 | |
| *** Apoorva_ has joined #openstack-infra | 16:29 | |
| *** Daisy has quit IRC | 16:30 | |
| *** yolanda has joined #openstack-infra | 16:31 | |
| *** Daisy has joined #openstack-infra | 16:31 | |
| *** amitgandhinz has quit IRC | 16:31 | |
| *** rcernin has quit IRC | 16:31 | |
| *** amitgandhinz has joined #openstack-infra | 16:32 | |
| *** asettle has quit IRC | 16:32 | |
| fungi | clarkb: oh, yeah it was just the mysql-db macro where i expected idempotency, not mysql-prep. sorry | 16:32 |
| *** Apoorva has quit IRC | 16:33 | |
| *** tesseract- has quit IRC | 16:33 | |
| clarkb | pleia2: at last they take safety seriously? | 16:33 |
| *** Swami has joined #openstack-infra | 16:33 | |
| fungi | clarkb: as for it not working on centos, it worked on centos 6 but maybe since dropping python 2.6 testing we no longer have projects that expect database setup on centos in their unit tests | 16:33 |
| pleia2 | clarkb: indeed :) it's done once a year right on schedule | 16:33 |
| *** pcaruana has quit IRC | 16:33 | |
| *** hogepodge has quit IRC | 16:34 | |
| clarkb | fungi: I think 6 -> 7 was the mysql to mariadb switch too | 16:34 |
| clarkb | so that makes sense | 16:34 |
| clarkb | fungi: I am happy with this on trusty and debian jessie. Not having good luck getting our xenial images to boot in an ssh'able manner | 16:35 |
| clarkb | fungi: hopeflly going to hvae that sorted soon and we can approve | 16:35 |
| fungi | so anyway, sounds like 337648 is safe to approve | 16:35 |
| clarkb | yes should be safe for trusty and debian | 16:35 |
| *** jamesmcarthur has quit IRC | 16:35 | |
| pleia2 | yeah, centos7 switched to mariadb | 16:35 |
| fungi | we could in theory extend those macros to support mariadb as well | 16:35 |
| *** cody-somerville has joined #openstack-infra | 16:35 | |
| *** cody-somerville has quit IRC | 16:35 | |
| *** cody-somerville has joined #openstack-infra | 16:35 | |
| fungi | but i wouldn't bother until someone comes along actually needing it | 16:36 |
| clarkb | ++ | 16:36 |
| clarkb | mordred: since you are the only -1 on https://review.openstack.org/#/c/337648/1 what d oyou think about approving it as is then sorting out further issues if they crop up? | 16:36 |
| *** rdx_matt has quit IRC | 16:37 | |
| *** rdx_matt has joined #openstack-infra | 16:37 | |
| *** Apoorva_ has quit IRC | 16:38 | |
| *** Apoorva has joined #openstack-infra | 16:38 | |
| *** amitgandhinz has quit IRC | 16:39 | |
| *** amitgandhinz has joined #openstack-infra | 16:39 | |
| *** AL34N1X has quit IRC | 16:39 | |
| *** AL34N1X has joined #openstack-infra | 16:40 | |
| *** hogepodge has joined #openstack-infra | 16:41 | |
| *** Daisy has quit IRC | 16:41 | |
| *** chem has quit IRC | 16:41 | |
| clarkb | cloudnull: gah I think it was pebkac, I didn't set config drive up but when I did then it works | 16:41 |
| *** chem has joined #openstack-infra | 16:41 | |
| clarkb | mordred: greghaynes ^ possible bug in glean on xenial, doesn't work with dhcp automagically | 16:42 |
| greghaynes | nice | 16:42 |
| clarkb | will need cloudnullto confirm that things are looking good on their side | 16:42 |
| *** rdx_matt has quit IRC | 16:42 | |
| clarkb | I am going to guess a boot order issue with writing out the default dhcp config | 16:43 |
| *** AL34N1X has quit IRC | 16:44 | |
| *** tonytan_brb has joined #openstack-infra | 16:44 | |
| *** AL34N1X has joined #openstack-infra | 16:45 | |
| *** berendt has quit IRC | 16:45 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Make py35 jobs voting on tempest https://review.openstack.org/337748 | 16:45 |
| *** jamesmcarthur has joined #openstack-infra | 16:45 | |
| clarkb | ooh bindep issues with php on xenial | 16:46 |
| * clarkb will sort that out too | 16:46 | |
| *** tonytan4ever has quit IRC | 16:46 | |
| fungi | anybody happen to know why "workers" is so high now on the zuul job queue graph? | 16:46 |
| *** Daisy has joined #openstack-infra | 16:47 | |
| fungi | i wonder if we're reporting the wrong values for that, or if that no longer represents what we think it did | 16:47 |
| *** lucasagomes is now known as lucas|afk | 16:48 | |
| fungi | anyway, we're not seriously behind in job backlog, so i'm okay with us temporarily disabling rax-iad as long as someone is opening a ticket with rackspace to figure out what's happening and has a plan for how to tell when it's safe to reenable it again | 16:49 |
| *** hogepodge has quit IRC | 16:50 | |
| *** jamesmcarthur has quit IRC | 16:50 | |
| fungi | https://status.rackspace.com/ doesn't indicate anything out of the ordinary for iad/dfw, just a resolved disruption in ord from ~12 hours ago | 16:51 |
| *** Daisy has quit IRC | 16:52 | |
| clarkb | fungi: I seem to recall jeblair saying the workers accounting was known buggy | 16:53 |
| *** zhurong has joined #openstack-infra | 16:56 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:58 | |
| clarkb | fungi: any idea how the php5 things in fallback other requirements manage to get past xenial testing? | 16:58 |
| *** adrian_otto1 has joined #openstack-infra | 16:58 | |
| fungi | clarkb: nope, but logs should tell us | 16:59 |
| *** Daisy has joined #openstack-infra | 16:59 | |
| pabelanger | I'd be curious to see that log too | 16:59 |
| clarkb | there is no php5-cli or php5-mcrypt on xenial, its all php7 | 16:59 |
| pleia2 | yeah, 5->7 should be breaking a lot of things | 16:59 |
| *** shashank_hegde has joined #openstack-infra | 16:59 | |
| clarkb | I think I have a fix for bindep, testing it on xenial now. But am confused at how xenial tests pass because we run the bindep tests against this | 17:00 |
| *** AL34N1X has quit IRC | 17:00 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Log stderr when creating and destroying envs https://review.openstack.org/337775 | 17:01 |
| clarkb | oh nope this fix won | 17:01 |
| clarkb | t work on xenial | 17:01 |
| clarkb | er trusty | 17:01 |
| clarkb | xenial is nice enough to have php-foo -> php7.0-foo mappings but trusty is't for mcrypt | 17:02 |
| pabelanger | http://logs.openstack.org/30/335230/2/gate/gate-project-config-bindep-fallback-ubuntu-xenial/68f8c80/console.html.gz seems to show php5 being installed on xenial | 17:02 |
| fungi | clarkb: http://logs.openstack.org/30/335330/1/gate/gate-project-config-bindep-fallback-ubuntu-xenial/425084c/console.html.gz#_2016-06-29_21_17_16_815980 "Setting up php5-common (5.6.17+dfsg-3ubuntu1) ... | 17:02 |
| *** adrian_otto has quit IRC | 17:02 | |
| *** ayoung has quit IRC | 17:02 | |
| *** hogepodge has joined #openstack-infra | 17:02 | |
| * pleia2 headscratch | 17:02 | |
| clarkb | huh | 17:02 |
| *** Apoorva_ has joined #openstack-infra | 17:03 | |
| clarkb | we aren't setting up our xenial instances with trusty packages are we in the configure mirror script? | 17:03 |
| *** Daisy has quit IRC | 17:03 | |
| *** piet has quit IRC | 17:03 | |
| clarkb | let me push up my fix for xenial and see if that works too | 17:04 |
| pleia2 | something must be weird here, these packages definitely aren't in xenial | 17:04 |
| *** piet has joined #openstack-infra | 17:04 | |
| *** Daisy has joined #openstack-infra | 17:04 | |
| fungi | "Setting up php5-cli (5.6.17+dfsg-3ubuntu1)" and yet that's not a current version anywhere... http://packages.ubuntu.com/php5-cli | 17:04 |
| *** maishsk has joined #openstack-infra | 17:05 | |
| *** PollyZ has quit IRC | 17:05 | |
| fungi | newer than wily's | 17:05 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use proper php packages on xenial https://review.openstack.org/337795 | 17:05 |
| clarkb | oh wait derp one more thing to update | 17:05 |
| * pleia2 nods | 17:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use proper php packages on xenial https://review.openstack.org/337795 | 17:06 |
| clarkb | there, that should be more "correct" | 17:06 |
| *** Nikolay_St has joined #openstack-infra | 17:06 | |
| *** Apoorva has quit IRC | 17:07 | |
| *** AL34N1X has joined #openstack-infra | 17:07 | |
| fungi | "Get:120 http://mirror.ord.rax.openstack.org/ubuntu xenial/main amd64 php5-cli amd64 5.6.17+dfsg-3ubuntu1 [2250 kB] | 17:07 |
| *** DaiDV has joined #openstack-infra | 17:07 | |
| dane-fichter | Anyone know what the error "*-docs-tags-only not defined" means? | 17:07 |
| clarkb | I wonder if we synced the package and then it went away but we never lceaned it out? | 17:07 |
| *** karthikp_ has quit IRC | 17:07 | |
| clarkb | dane-fichter: is that from a job log? | 17:07 |
| fungi | clarkb: no, this would have to be _in_ the dists index for xenial | 17:07 |
| *** karthikp_ has joined #openstack-infra | 17:08 | |
| clarkb | fungi: which we build ourselves with reprepro | 17:08 |
| fungi | clarkb: ahh, right | 17:08 |
| dane-fichter | clarkb: yes, it's the job logs for the project-config check I believe | 17:08 |
| clarkb | dane-fichter: without more context I can't be completely sure but it sounds like you have asked zuul to use a job name that isn't created by jjb | 17:09 |
| *** Daisy has quit IRC | 17:09 | |
| AJaeger | dane-fichter: I'll review and comment | 17:09 |
| *** permalac has joined #openstack-infra | 17:09 | |
| *** abregman has joined #openstack-infra | 17:09 | |
| clarkb | fungi: I am going to go ahead and approve frickler's change for mysql things now despite the mordred -1 | 17:09 |
| *** PollyZ has joined #openstack-infra | 17:09 | |
| *** yolanda has quit IRC | 17:09 | |
| clarkb | I think that mordred's comments are a good follow up | 17:09 |
| fungi | clarkb: wfm | 17:09 |
| dane-fichter | AJaeger: Thanks, this is my first attempt at setting up a project | 17:10 |
| clarkb | and I guess I should look into why our xenial mirror is funny | 17:10 |
| clarkb | though not entirely sure where to get started on that | 17:10 |
| *** Daisy has joined #openstack-infra | 17:10 | |
| *** abregman has quit IRC | 17:10 | |
| *** derekh has quit IRC | 17:10 | |
| *** linuxmaniac_ has left #openstack-infra | 17:11 | |
| AJaeger | dane-fichter: should your project be an official project under the big tent? | 17:11 |
| *** jamesmcarthur has joined #openstack-infra | 17:11 | |
| AJaeger | dane-fichter: is it part of any existing team? | 17:11 |
| pabelanger | clarkb: I can check that out if you want | 17:12 |
| clarkb | pabelanger: yes more eyeballs would be nice | 17:12 |
| pabelanger | dumpunreferenced should show us if there are problems | 17:12 |
| dane-fichter | AJaeger: No. It's not meant to be an official project. Basically I need to create a lib that Glance and Nova can use. | 17:12 |
| clarkb | pabelanger: I am not sure that reprepro considers that there are any problems | 17:12 |
| dane-fichter | AJaeger: And I need to integrate w/ Gerrit | 17:12 |
| AJaeger | dane-fichter: in that case: you cannot publish on docs.o.o, I'll comment in the review with details. | 17:13 |
| dane-fichter | AJaeger: What does "publish on docs" refer to? | 17:13 |
| AJaeger | dane-fichter: we publish on docs.openstack.org only content for official projects. | 17:14 |
| *** abregman has joined #openstack-infra | 17:14 | |
| *** PollyZ has quit IRC | 17:14 | |
| *** tqtran has joined #openstack-infra | 17:14 | |
| dane-fichter | AJaeger: Ok. Yes that's not necessary for my project. It's initial purpose is just a lib for some common code. | 17:15 |
| *** Daisy has quit IRC | 17:15 | |
| dane-fichter | AJaeger: Thanks for your help | 17:15 |
| AJaeger | dane-fichter: remove those lines and your tests should pass | 17:15 |
| *** abregman has quit IRC | 17:16 | |
| *** Daisy has joined #openstack-infra | 17:16 | |
| *** jamesmcarthur has quit IRC | 17:16 | |
| pleia2 | clarkb, pabelanger, I was hoping it was something simple like a -backports or cloud archive serving up something weird in xenial, but no luck so far | 17:16 |
| clarkb | http://mirror.dfw.rax.openstack.org/ubuntu/lists/ubuntu_xenial_main_amd64_Packages doesn't include it | 17:16 |
| *** rcernin has joined #openstack-infra | 17:17 | |
| clarkb | at least firefox reports it doesn't but that file is "large" maybe I should wget | grep | 17:17 |
| *** hogepodge has quit IRC | 17:17 | |
| *** bnemec has quit IRC | 17:19 | |
| pabelanger | xenial|universe|amd64 pool/universe/p/php5/php5-cgi_5.6.17+dfsg-3ubuntu1_amd64.deb | 17:19 |
| pabelanger | is what I see in reprepro | 17:19 |
| pabelanger | sorry, wrong file | 17:19 |
| pabelanger | xenial|universe|amd64 pool/universe/p/php5/php5-common_5.6.17+dfsg-3ubuntu1_amd64.deb | 17:19 |
| *** ddieterly is now known as ddieterly[away] | 17:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix mysql setup for Xenial jobs https://review.openstack.org/337648 | 17:20 |
| fungi | clarkb: pabelanger: pleia2: it's looking like reprepro doesn't remove obsolete packages | 17:20 |
| *** EricGonczer_ has joined #openstack-infra | 17:21 | |
| *** Daisy has quit IRC | 17:21 | |
| *** DaiDV has quit IRC | 17:21 | |
| *** AL34N1X has quit IRC | 17:21 | |
| fungi | and the log indicates we added it to the repo back on april 2 when we initialized the archive | 17:21 |
| pleia2 | aha | 17:21 |
| pleia2 | I thought it was removed from the repo in february, but that might have just been when removal was initiated | 17:22 |
| pabelanger | I would expect them to show up in unreferenced | 17:22 |
| pabelanger | at some point in time | 17:22 |
| pleia2 | yep, finally removed April 6 https://launchpad.net/ubuntu/xenial/amd64/php5 | 17:22 |
| *** shashank_hegde has quit IRC | 17:23 | |
| fungi | pabelanger: unreferenced what? | 17:23 |
| *** PollyZ has joined #openstack-infra | 17:23 | |
| pleia2 | fungi: good sluething | 17:23 |
| *** hogepodge has joined #openstack-infra | 17:23 | |
| pabelanger | fungi: reprepro unreferenced | 17:23 |
| pleia2 | they did a lot of late-in-the-release cleanup with this LTS, I guess this is the downside of creating the mirror before release :\ | 17:24 |
| pabelanger | is the command to listed packages that are no longer needed | 17:24 |
| fungi | aha | 17:24 |
| clarkb | they are in conflicts with of some packages does that count as a reference? | 17:25 |
| *** karthikp_ has quit IRC | 17:25 | |
| dane-fichter | AJaeger: I'm a bit confused. One of your comments recommends removing the openstack-client-publish-jobs line and the other comment says to add it. | 17:25 |
| *** ccamacho has quit IRC | 17:26 | |
| *** karthikp_ has joined #openstack-infra | 17:26 | |
| *** bnemec has joined #openstack-infra | 17:27 | |
| AJaeger | dane-fichter: Remove it. I wanted to say: If it's an official project, you would use openstack-client-publish-jobs instead of openstack-publish-jobs here to fix it. | 17:27 |
| dane-fichter | AJaeger: Thanks. | 17:27 |
| *** trown|lunch is now known as trown | 17:27 | |
| *** PollyZ has quit IRC | 17:28 | |
| fungi | pabelanger: i guess i need to kinit before reprepro dumpunreferenced can create /afs/.openstack.org/mirror/ubuntu/db/lockfile ? | 17:28 |
| *** EricGonczer_ has quit IRC | 17:28 | |
| *** hogepodge has quit IRC | 17:28 | |
| fungi | pabelanger: don't want to go running things that could adversely affect our mirroring | 17:28 |
| *** EricGonczer_ has joined #openstack-infra | 17:29 | |
| pabelanger | fungi: Ya, you need to wrap your reprepro commands with k5start | 17:29 |
| AJaeger | clarkb: btw. we need php5 only for one or two repos, let me check which ones... | 17:29 |
| *** abregman has joined #openstack-infra | 17:29 | |
| *** PollyZ has joined #openstack-infra | 17:29 | |
| *** mgould is now known as mgould|afk | 17:30 | |
| fungi | pabelanger: any idea why our cron jobs don't have to k5start? | 17:30 |
| pabelanger | fungi: our mirror scripts handle that | 17:30 |
| fungi | pabelanger: oh! reprepro-mirror-update is a wrapper | 17:31 |
| pabelanger | yar | 17:31 |
| *** csomerville has joined #openstack-infra | 17:31 | |
| *** shardy has quit IRC | 17:31 | |
| fungi | looks like /var/run/reprepro/mirror.ubuntu.unreferenced-files is currently empty | 17:32 |
| *** electrofelix has quit IRC | 17:32 | |
| AJaeger | clarkb: openstack-infra/groups - and openstackid has already other-requirements. | 17:32 |
| AJaeger | clarkb: so, we might be able to just remove the php5 files... | 17:33 |
| EmilienM | we're having a high percent of failures in different CIs, can we review https://review.openstack.org/#/c/337513/ ? | 17:33 |
| fungi | pabelanger: and similarly, this is returning no output at all: `k5start -t -f /etc/reprepro.keytab service/reprepro -- reprepro --confdir /etc/reprepro/ubuntu dumpunreferenced` | 17:34 |
| *** tonytan_brb has quit IRC | 17:34 | |
| pabelanger | fungi: reprepro checkupdate is showing some interesting info. I see php5 listed, but not sure what that means. Going to look into the source more | 17:34 |
| pabelanger | fungi: Ya, that is the strange thing, I would expect it to be listed there | 17:34 |
| pabelanger | so something must still be using it | 17:34 |
| *** cody-somerville has quit IRC | 17:34 | |
| fungi | EmilienM: i'm hesitant to disable a region in nodepool unless someone has sufficient details to report to the provider so they can track down the issue and some idea of how to tell when it's safe to start using it again | 17:35 |
| fungi | EmilienM: rackspace isn't reporting any network issues in that region | 17:36 |
| *** rajinir has joined #openstack-infra | 17:36 | |
| *** hogepodge has joined #openstack-infra | 17:36 | |
| *** oanson has joined #openstack-infra | 17:37 | |
| AJaeger | fungi, I collected log files - but that's all I can do today... | 17:37 |
| *** maishsk has quit IRC | 17:37 | |
| fungi | AJaeger: did they show anything useful? are they jobs where we're also doing traceroutes at the beginning? | 17:38 |
| fungi | the last time i tried to report "network connectivity problems" to rackspace, they basically said there's nothing they can do unless we have a live instance exhibiting the exact network issue claimed so that they can observe it in their network. they apparently have no capacity for post-mortem troubleshooting of network issues | 17:39 |
| AJaeger | fungi, it's the project-config layout job, some pep8 jobs - but there was one or two dsvm as well that I stumbled upon | 17:39 |
| AJaeger | fungi, I added some links to the review | 17:39 |
| *** karthik__ has joined #openstack-infra | 17:39 | |
| AJaeger | fungi, so we could see a layout check hanging... | 17:39 |
| AJaeger | fungi, telnet://104.239.162.241:8088 | 17:40 |
| AJaeger | that one is hanging | 17:40 |
| fungi | i'll hold it now | 17:40 |
| AJaeger | but timing out pretty soon, fungi | 17:40 |
| AJaeger | change https://review.openstack.org/337795 | 17:40 |
| fungi | i've held ubuntu-precise-rax-iad-2234896/104.239.162.241 | 17:41 |
| AJaeger | clarkb, fungi: Alternative for php5: Let's merge https://review.openstack.org/#/c/298600 and remove php5 from the bindep fallback list | 17:41 |
| *** karthikp_ has quit IRC | 17:41 | |
| *** ldnunes has quit IRC | 17:42 | |
| pabelanger | fungi: pleia2: Maybe we should delete xenial-main on mirror-update and reinitial reprepro for it | 17:43 |
| *** ldnunes has joined #openstack-infra | 17:43 | |
| pleia2 | pabelanger: ++ | 17:43 |
| pleia2 | I think one of the php5 packages comes from universe, so we might need to check that | 17:44 |
| fungi | pabelanger: that's only a stop-gap measure though. we need some means of ensuring that packages deleted from official mirrors get removed from ours as well | 17:44 |
| *** deadnull has quit IRC | 17:44 | |
| *** acoles is now known as acoles_ | 17:44 | |
| fungi | AJaeger: looking on that held worker, the last line currently in .tox/zuul/log/zuul-0.log is "Installing setuptools, pip, wheel..." | 17:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/groups: Add other-requirements.txt https://review.openstack.org/298600 | 17:45 |
| openstackgerrit | dane-fichter proposed openstack-infra/project-config: Add cursive library https://review.openstack.org/337709 | 17:45 |
| openstackgerrit | Tim Burke proposed openstack-infra/lodgeit: Drop support for pre-2.5 Python https://review.openstack.org/337811 | 17:46 |
| *** shashank_hegde has joined #openstack-infra | 17:46 | |
| AJaeger | fungi, so timeout accessing our pypi mirror? | 17:46 |
| *** adrian_otto1 has quit IRC | 17:46 | |
| *** jamesmcarthur has joined #openstack-infra | 17:47 | |
| *** zhurong has quit IRC | 17:47 | |
| *** adrian_otto has joined #openstack-infra | 17:48 | |
| fungi | AJaeger: so far i see no direct evidence of it. these jobs just stop for a while and then get killed due to a timeout but i don't see any mention of a network error in them | 17:48 |
| fungi | i also don't actually see tox running any longer on this held node | 17:48 |
| pleia2 | fungi: I agree that it's a stop-gap, it should be somewhat rare in a stable release for things to disappear, but when they do there is usually a pretty serious reason that we need to be aware of, but for now I'd say our xenial tests aren't reflecting reality so we need to break some eggs asap | 17:49 |
| AJaeger | fungi, let me dig a bit... | 17:49 |
| pabelanger | fungi: Agreed. If we need to dig more into this, I'd rather spend the effort switching to rsync and writing our validation scripts for afs. | 17:49 |
| fungi | AJaeger: so it _seems_ like something could have just died when the job was running and ansible doesn't realize it | 17:49 |
| fungi | there are no children of the "sshd: jenkins@notty" process, and the only thing running python right now is "/tmp/ansible_9Dxcyg/ansible_module_zuul_console.py" | 17:50 |
| *** HeOS has quit IRC | 17:51 | |
| *** EricGonczer_ has quit IRC | 17:51 | |
| *** asettle has joined #openstack-infra | 17:51 | |
| dane-fichter | AJaeger: Can you describe a bit what the first patch will need to entail? As far as I know, it'll just be the cookiecutter process described in the project creator's guide. | 17:51 |
| fungi | pabelanger: pleia2: i'm fine with reinitializing the xenial mirror as a stopgap measure | 17:52 |
| AJaeger | fungi, http://logs.openstack.org/50/331850/5/check/gate-heat-python34-db/2967746/console.html#_2016-07-05_05_07_46_523231 | 17:53 |
| AJaeger | fungi, I saw some real timeout but can't find the log file anymore, will dig more... | 17:53 |
| *** EricGonczer_ has joined #openstack-infra | 17:55 | |
| fungi | AJaeger: has it just been timing out on things that talk to mirror.iad.rax.openstack.org? wondering if maybe we're having an issue with that server instead | 17:55 |
| *** hogepodge has quit IRC | 17:55 | |
| *** notnownikki has quit IRC | 17:55 | |
| *** karthik__ has quit IRC | 17:55 | |
| pabelanger | fungi: pleia2: let me read up on how to do it | 17:55 |
| AJaeger | fungi, from yesterday: | 17:56 |
| AJaeger | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-07-04.log.html#t2016-07-04T20:45:08 | 17:56 |
| AJaeger | http://logs.openstack.org/79/336779/7/check/gate-puppet-openstack-integration-puppet-lint/cd126af/console.html#_2016-07-04_20_35_30_903758 | 17:56 |
| AJaeger | EmilienM: do you have log files from today for rax-iad? | 17:57 |
| fungi | AJaeger: comparing cacti graphs, load average on that server is much higher than its peers | 17:57 |
| AJaeger | fungi, might be as well. | 17:57 |
| AJaeger | I saw lots of timeouts on the layout job - but no clear indication what failed | 17:58 |
| fungi | oh, and it has some serious i/o errors in its logs | 17:58 |
| EmilienM | AJaeger: not handy but I can find some | 17:58 |
| fungi | AJaeger: it seems to have started here... [Mon Jul 4 16:19:53 2016] INFO: task jbd2/dm-0-8:1971 blocked for more than 120 seconds. | 17:59 |
| fungi | after that, lots of kernel panics, i/o errors, et cetera | 17:59 |
| AJaeger | dane-fichter: just adding the cookiecutter should be enough - just keep in mind that this first change needs to pass all the tests you have setup. Nothing will merge before all tests pass . The cookiecutter should set everything up properly | 17:59 |
| AJaeger | fungi, ohhh. That would explain a lot! | 17:59 |
| fungi | and /dev/xvdb seems particularly unhappy | 17:59 |
| fungi | i think that's its afs cache volume | 18:00 |
| fungi | chances are we lost network connectivity briefly on the backend to that cinder volume | 18:00 |
| fungi | and now it's all sorts of unhappy trying to update the afs cache | 18:00 |
| AJaeger | fungi, hope that rax-iad works again once the mirror is working fine. | 18:00 |
| fungi | [Mon Jul 4 16:29:04 2016] EXT4-fs (dm-0): Remounting filesystem read-only | 18:01 |
| AJaeger | So, let me abandon my change. | 18:01 |
| fungi | i'm going to see what i can do to recover this | 18:01 |
| *** jamesmca_ has joined #openstack-infra | 18:01 | |
| *** makowals has joined #openstack-infra | 18:01 | |
| AJaeger | EmilienM: not needed anymore - thanks | 18:01 |
| *** pvaneck has joined #openstack-infra | 18:01 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: switch to python-db-jobs https://review.openstack.org/335551 | 18:01 |
| *** makowals has quit IRC | 18:02 | |
| fungi | stopping apache2 and openafs-client on it | 18:02 |
| *** jamesmcarthur has quit IRC | 18:02 | |
| clarkb | sorry I had to pop out to make dinner (yay slow cooker) | 18:02 |
| clarkb | AJaeger: that is also fine by me if we want to remove packages from fallback | 18:03 |
| AJaeger | clarkb: I prefer removing two packages from fallback that are only used by a single repo. | 18:03 |
| frickler | clarkb: fungi: AJaeger: nova-python35-db seems to be well underway at telnet://104.130.73.19:8088, can't wait to see the final result :) | 18:03 |
| AJaeger | klasse! | 18:04 |
| AJaeger | clarkb, could you review https://review.openstack.org/#/c/298600/ , please? I'll propose the php5 removal now.... | 18:04 |
| *** Apoorva_ has quit IRC | 18:05 | |
| fungi | ugh. `lsof /afs` isn't returning. i may have to reboot (possibly hard reboot) this machine | 18:05 |
| *** Apoorva has joined #openstack-infra | 18:05 | |
| fungi | trying to reboot now | 18:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove php5 packages https://review.openstack.org/337816 | 18:06 |
| fungi | oh, hey, rackspace notified us with a support ticket too: A CBS storage node in IAD has gone offline and is temporarily unavailable. The following volumes on your have been impacted by the outage: | 18:06 |
| fungi | ac285e54-10f4-4625-8898-7d70c01cd02a mirror.iad.rax.openstack.org/main01 SATA | 18:06 |
| AJaeger | frickler, clarkb: success for the nova-python35-db job! | 18:06 |
| fungi | as expected, soft reboot is going nowhere. i'm initiating a hard reboot now | 18:08 |
| AJaeger | fungi: glad that they noticed ;) | 18:08 |
| *** maishsk has joined #openstack-infra | 18:09 | |
| AJaeger | fungi, so now everythign in rax-iad will fail? | 18:09 |
| pabelanger | fungi: pleia2: k5start -t -f /etc/reprepro.keytab service/reprepro -- timeout -k 2m 30m reprepro --confdir /etc/reprepro/ubuntu removefilter xenial 'Section' should be the command to remove xenial packages | 18:09 |
| fungi | oh, slick, we have an ubuntu "splash screen" at boot on this server instead of an actual console log | 18:09 |
| pabelanger | just waiting for it to complete | 18:09 |
| fungi | how "attractive" | 18:09 |
| fungi | okay, server's back up again. giving it a health check | 18:10 |
| fungi | looks like afs is _probably_ happy again | 18:10 |
| *** jamesmca_ has quit IRC | 18:10 | |
| AJaeger | pabelanger: my alternative for php5: Remove them, see https://review.openstack.org/#/c/298600/ and https://review.openstack.org/337816 | 18:10 |
| fungi | i'm successfully browsing http://mirror.iad.rax.openstack.org/ | 18:11 |
| AJaeger | yeah! | 18:12 |
| *** jamesmcarthur has joined #openstack-infra | 18:12 | |
| *** kzaitsev_mb has quit IRC | 18:12 | |
| fungi | #status notice Job instability resulting from a block storage connectivity error on mirror.iad.rax.openstack.org has been corrected; jobs running in rax-iad should be more reliable again. | 18:13 |
| openstackstatus | fungi: sending notice | 18:13 |
| pabelanger | AJaeger: Cool, lets hold off before we land 337816, I want to see if our regenerated repo exposes the problem | 18:13 |
| *** jamesmcarthur has quit IRC | 18:14 | |
| -openstackstatus- NOTICE: Job instability resulting from a block storage connectivity error on mirror.iad.rax.openstack.org has been corrected; jobs running in rax-iad should be more reliable again. | 18:14 | |
| clarkb | AJaeger: 298600 has my +2 | 18:14 |
| AJaeger | pabelanger: feel free to WIP 337816 as long as needed. | 18:15 |
| *** tonytan4ever has joined #openstack-infra | 18:15 | |
| AJaeger | clarkb: thanks. | 18:15 |
| *** ociuhandu has quit IRC | 18:15 | |
| clarkb | AJaeger: I can abandon my change to fallback reqs | 18:15 |
| *** chem has quit IRC | 18:15 | |
| pabelanger | AJaeger: +3 on 600 | 18:15 |
| openstackstatus | fungi: finished sending notice | 18:16 |
| fungi | okay, i'm going to crawl into a hole for the next 45 minutes and prep for our team meeting, unless there are any other grave emergencies which need attending to | 18:16 |
| AJaeger | fungi, not anymore ;) Thanks | 18:16 |
| *** ddieterly[away] is now known as ddieterly | 18:16 | |
| * clarkb catches up on xenial repo things | 18:16 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Project api https://review.openstack.org/337730 | 18:16 |
| *** ayoung has joined #openstack-infra | 18:16 | |
| clarkb | pabelanger: pleia2 we probably shouldn't rely on that though, any package could be supplanted at any time right? granted not very likely with an LTS release | 18:17 |
| clarkb | where that == make a new mirror | 18:17 |
| *** chem has joined #openstack-infra | 18:17 | |
| pabelanger | Ya, I am pretty surprise reprepro isn't doing the right thing. | 18:18 |
| pabelanger | makes we want to work on rsync support sooner | 18:18 |
| *** sarob has joined #openstack-infra | 18:19 | |
| pabelanger | xenial mirror process currently running | 18:19 |
| * clarkb is excited that xenial things are mostly working | 18:20 | |
| pabelanger | clarkb: fungi: pleia2: speaking of reprepro, any objections to 337286 which adds reprepro bits to our release slave | 18:21 |
| clarkb | jeblair: https://review.openstack.org/#/c/336311/ has been updated, do we want to push forward with that as the gearman registration with trusty/xenial preferred option? | 18:22 |
| clarkb | I guess I can bring that bakc up in today's meeting | 18:22 |
| openstackgerrit | dane-fichter proposed openstack-infra/project-config: Add cursive library https://review.openstack.org/337709 | 18:22 |
| *** yamamoto has quit IRC | 18:23 | |
| *** TaylorHuang has joined #openstack-infra | 18:23 | |
| *** sarob has quit IRC | 18:23 | |
| AJaeger | clarkb: https://review.openstack.org/#/c/326095 adds a centos7-2 multinode job. It looks fine from my point but since you did some multinode setup, could you review it, please? | 18:23 |
| clarkb | fungi: can we add trusty/xenial to today's agenda? and based on discussion on that I can write the minispec on how we will handle host selection | 18:24 |
| clarkb | AJaeger: sure | 18:24 |
| *** hogepodge has joined #openstack-infra | 18:26 | |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| clarkb | pabelanger: fungi jeblair re using the zuul user, looks like we needt to update some of the ready scripts for nodepool, then build new images first? | 18:26 |
| *** TaylorH has quit IRC | 18:26 | |
| clarkb | multinode_setup.sh for example | 18:27 |
| pabelanger | clarkb: ya, we still need to do that | 18:27 |
| *** sarob has joined #openstack-infra | 18:28 | |
| pabelanger | what we should do is add zuul bits along side jenkins, in our dib elements for now | 18:28 |
| *** jamesmcarthur has quit IRC | 18:28 | |
| pabelanger | then, once 100% remove jenkins | 18:28 |
| clarkb | yes | 18:28 |
| pabelanger | I can take a peak into that now | 18:28 |
| openstackgerrit | Merged openstack-infra/groups: Add other-requirements.txt https://review.openstack.org/298600 | 18:28 |
| *** sarob has quit IRC | 18:28 | |
| *** sarob has joined #openstack-infra | 18:29 | |
| clarkb | AJaeger: looks like this job template is made of magic sauce. The type parameter feeds into tripleoci things. So I can't really speak to whether or not what they are doig will work but the infra side looks fine | 18:30 |
| *** csomerville has quit IRC | 18:30 | |
| pleia2 | clarkb: yeah, it's possible, but pretty unlikely with an lts (and usually for security/major bug, which would be a problem) | 18:31 |
| *** cody-somerville has joined #openstack-infra | 18:31 | |
| clarkb | pleia2: ya, but also we reprepro things that aren't ubuntu LTSes | 18:31 |
| * pleia2 nods | 18:31 | |
| clarkb | ceph is the current thing but also pabelanger and zigo are working on it for testing packages of openstack | 18:31 |
| *** jamesmcarthur has joined #openstack-infra | 18:32 | |
| fungi | clarkb: add anything you like to today's agenda. i think it's still on the short side | 18:32 |
| *** csomerville has joined #openstack-infra | 18:33 | |
| AJaeger | clarkb: magic? Cool! | 18:35 |
| clarkb | AJaeger: it takes a 'type' parameter then everything else is handled outside of jjb | 18:35 |
| fungi | we could all use a little more magic in our lives, just not sure i want mine coming in job-template form | 18:35 |
| *** cody-somerville has quit IRC | 18:36 | |
| *** jamesmcarthur has quit IRC | 18:36 | |
| *** tphummel has quit IRC | 18:37 | |
| *** asettle has quit IRC | 18:37 | |
| *** jamesmcarthur has joined #openstack-infra | 18:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Add ceph-proxy project to Juju Charms https://review.openstack.org/334658 | 18:41 |
| *** thorst_ has joined #openstack-infra | 18:41 | |
| *** thorst_ has quit IRC | 18:41 | |
| *** oanson has quit IRC | 18:41 | |
| *** thorst_ has joined #openstack-infra | 18:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Prepare to replace jenkins user with zuul user https://review.openstack.org/337821 | 18:43 |
| pabelanger | clarkb: ^ | 18:43 |
| *** abregman is now known as abregman|afk | 18:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Add TripleO 2-node job https://review.openstack.org/326095 | 18:44 |
| clarkb | pabelanger: one concern with that is disk space on smaller /'s | 18:44 |
| *** mrmartin has joined #openstack-infra | 18:44 | |
| clarkb | pabelanger: we should check how big https://review.openstack.org/#/c/337821/1/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos is and maybe hardlink those dir contents together | 18:44 |
| pabelanger | Ya, didn't think of that | 18:44 |
| pabelanger | ++ | 18:45 |
| *** Sukhdev has joined #openstack-infra | 18:45 | |
| openstackgerrit | Brent Eagles proposed openstack-infra/tripleo-ci: Exit with non-zero status on create and update errors https://review.openstack.org/319352 | 18:47 |
| *** jamesmcarthur has quit IRC | 18:48 | |
| clarkb | pabelanger: couple comments, one that definitely need to be addressed | 18:48 |
| pabelanger | looking | 18:50 |
| *** jamesmcarthur has joined #openstack-infra | 18:52 | |
| *** hogepodge has quit IRC | 18:53 | |
| *** jswarren__ has joined #openstack-infra | 18:53 | |
| *** sputnik13 has joined #openstack-infra | 18:54 | |
| *** Sukhdev has quit IRC | 18:55 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel and ORM https://review.openstack.org/322307 | 18:56 |
| *** piet has quit IRC | 18:56 | |
| jeblair | clarkb: +2 336311 | 18:57 |
| *** jswarren_ has quit IRC | 18:57 | |
| * jeblair pretends to be a numbers station | 18:57 | |
| *** sarob has quit IRC | 18:58 | |
| anteaya | fungi: I added a last minute item about editing storyboard comments | 18:58 |
| *** mhickey has quit IRC | 18:58 | |
| anteaya | I'd like to discuss it | 18:59 |
| jhesketh | Morning | 19:00 |
| anteaya | morning jhesketh | 19:00 |
| fungi | anteaya: thanks | 19:00 |
| anteaya | jhesketh: I moved you agenda item onto the agenda | 19:00 |
| *** hogepodge has joined #openstack-infra | 19:00 | |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| anteaya | fungi: thank you | 19:00 |
| *** jamesmcarthur has quit IRC | 19:00 | |
| pleia2 | meeting time | 19:00 |
| jhesketh | anteaya: okay thanks. It's not overly important so was happy to leave it if we get time | 19:01 |
| anteaya | okay sure, I just didn't want to jump the queue since it was on the agenda before me | 19:01 |
| anteaya | and I had items | 19:01 |
| *** piet has joined #openstack-infra | 19:03 | |
| *** lennyb has quit IRC | 19:04 | |
| *** mhickey has joined #openstack-infra | 19:04 | |
| Zara | numbers stations are so cool and creepy :D | 19:08 |
| *** tphummel has joined #openstack-infra | 19:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Prepare to replace jenkins user with zuul user https://review.openstack.org/337821 | 19:10 |
| *** jamesmcarthur has joined #openstack-infra | 19:12 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add new posix element. https://review.openstack.org/294748 | 19:13 |
| *** sarob has joined #openstack-infra | 19:17 | |
| *** jswarren__ has quit IRC | 19:19 | |
| *** berendt has joined #openstack-infra | 19:19 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add new db api method to get recent successes by metadata https://review.openstack.org/333460 | 19:19 |
| *** jswarren__ has joined #openstack-infra | 19:19 | |
| *** BobBall is now known as BobBall_AWOL | 19:19 | |
| *** julim has quit IRC | 19:19 | |
| *** jamesmcarthur has quit IRC | 19:20 | |
| *** chem` has joined #openstack-infra | 19:22 | |
| *** dizquierdo has quit IRC | 19:23 | |
| *** Sukhdev has joined #openstack-infra | 19:23 | |
| *** piet has quit IRC | 19:23 | |
| *** yamamoto has joined #openstack-infra | 19:23 | |
| *** lennyb has joined #openstack-infra | 19:23 | |
| *** dimtruck is now known as zz_dimtruck | 19:24 | |
| AJaeger | Zara, do you join us on #openstack-meeting? | 19:25 |
| *** chem has quit IRC | 19:25 | |
| AJaeger | thanks, Zara | 19:25 |
| AJaeger | fungi asked for zaro not Zara - sorry. | 19:26 |
| Zara | haha, as it happens it's also relevant to me. thank you :) | 19:26 |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack/os-client-config: Adding new DreamHost Cloud cluster https://review.openstack.org/337832 | 19:26 |
| AJaeger | Zara: ;) | 19:26 |
| *** dane-fichter has quit IRC | 19:26 | |
| *** hogepodge has quit IRC | 19:27 | |
| *** dtardivel has quit IRC | 19:27 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Spec for changing the block device handling: local loop https://review.openstack.org/337833 | 19:27 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: have the ptl command parse the liaison wiki page https://review.openstack.org/334067 | 19:28 |
| *** hogepodge has joined #openstack-infra | 19:29 | |
| *** jamesmcarthur has joined #openstack-infra | 19:29 | |
| *** karthikp_ has joined #openstack-infra | 19:31 | |
| *** gyee has quit IRC | 19:31 | |
| *** EricGonczer_ has quit IRC | 19:31 | |
| *** coolsvap has quit IRC | 19:32 | |
| *** mdrabe has quit IRC | 19:33 | |
| *** EricGonczer_ has joined #openstack-infra | 19:33 | |
| *** jamesmcarthur has quit IRC | 19:35 | |
| *** yamamoto has quit IRC | 19:35 | |
| *** mdrabe has joined #openstack-infra | 19:35 | |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack/os-client-config: Adding new DreamHost Cloud cluster https://review.openstack.org/337832 | 19:36 |
| *** tphummel has quit IRC | 19:37 | |
| *** inc0 has joined #openstack-infra | 19:37 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:39 | |
| *** ddieterly is now known as ddieterly[away] | 19:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Raise ubuntu-xenial min-ready https://review.openstack.org/337838 | 19:40 |
| kgiusti | Hi infra - there seems to be a problem with the configuration of the Centos7 gates. Can't run devstack. | 19:41 |
| kgiusti | Looks like the python-virtualenv package has been installed over the pip version | 19:42 |
| clarkb | kgiusti: yes rcarrillocruz and greghaynes were looking into that | 19:43 |
| kgiusti | Devstack removes the rpm version, and clobbers the pip version: | 19:43 |
| *** karthikp_ has quit IRC | 19:43 | |
| clarkb | I think devstack can just stop removing the rpm version | 19:43 |
| rcarrillocruz | yup | 19:43 |
| *** hogepodge has quit IRC | 19:43 | |
| rcarrillocruz | greghaynes: i'll amend the commit and push a patchset shortly | 19:43 |
| *** zz_dimtruck is now known as dimtruck | 19:43 | |
| greghaynes | rcarrillocruz: ok, thanks | 19:43 |
| rcarrillocruz | kgiusti: https://review.openstack.org/#/c/337686/ | 19:43 |
| kgiusti | clarkb: ah, kewl. | 19:43 |
| *** hogepodge has joined #openstack-infra | 19:44 | |
| rcarrillocruz | kgiusti: i was surprised it was just me hitting it :-) | 19:44 |
| AJaeger | kgiusti: thanks for looking into it and telling us! | 19:44 |
| kgiusti | rcarrillocruz: I've got some oslo.messaging jobs that run on Centos7 | 19:44 |
| kgiusti | rcarrillocruz: it's an... exclusive club :) | 19:45 |
| rcarrillocruz | heh | 19:45 |
| *** julim has joined #openstack-infra | 19:45 | |
| kgiusti | rcarrillocruz: thanks for taking care of that. Much appreciated. | 19:45 |
| *** karthikp_ has joined #openstack-infra | 19:45 | |
| rcarrillocruz | np | 19:45 |
| *** amitgandhinz has quit IRC | 19:45 | |
| *** amitgandhinz has joined #openstack-infra | 19:46 | |
| rcarrillocruz | ah | 19:46 |
| rcarrillocruz | cinerama , greghaynes : you do track bugs on dib via launchpad | 19:47 |
| rcarrillocruz | ? | 19:47 |
| greghaynes | rcarrillocruz: yes, theres a launchpad | 19:47 |
| rcarrillocruz | oh, ok | 19:47 |
| rcarrillocruz | let me open a LP and reference it on the commit | 19:47 |
| *** jamesmcarthur has joined #openstack-infra | 19:47 | |
| fungi | kgiusti: yeah, the rpm and pip itself fight over the same files, so removing either removes both | 19:48 |
| *** roxanagh_ has joined #openstack-infra | 19:48 | |
| *** Sukhdev has quit IRC | 19:48 | |
| *** EricGonczer_ has quit IRC | 19:49 | |
| kgiusti | fungi: sounds like my two kids..... | 19:49 |
| *** dprince has quit IRC | 19:50 | |
| *** roxanagh_ has quit IRC | 19:52 | |
| pabelanger | Zara: I haven't looked, but will storyboard support an ACL system for the APIs? | 19:54 |
| *** karthikp_ has quit IRC | 19:55 | |
| *** jamesmcarthur has quit IRC | 19:56 | |
| *** karthikp_ has joined #openstack-infra | 19:56 | |
| SotK | pabelanger: I don't believe there is any plan for it beyond what currently exists | 19:57 |
| openstackgerrit | Mike Dorman proposed openstack-infra/logstash-filters: Improve folding of multiline messages https://review.openstack.org/337843 | 19:57 |
| AJaeger | dougwig, armax, mestery : Could you review the next days these neutron related changes, please? https://review.openstack.org/#/c/330926/ https://review.openstack.org/#/c/336927 https://review.openstack.org/#/c/335624/ | 19:57 |
| pabelanger | SotK: thanks | 19:57 |
| armax | AJaeger: ack | 19:57 |
| SotK | (that is, authenticated users can create and edit stories, comments, and tasks (and delete tasks)) | 19:58 |
| SotK | oh, worklists and boards too | 19:58 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack/diskimage-builder: Revert "Pre-install pip/virtualenv packages" https://review.openstack.org/337686 | 19:58 |
| *** asettle has joined #openstack-infra | 19:58 | |
| *** asettle has quit IRC | 19:58 | |
| SotK | only a comment's author can edit a comment currently though | 19:58 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Publish os-performance-tools docs https://review.openstack.org/317005 | 19:59 |
| rcarrillocruz | greghaynes , cinerama : ^ | 20:00 |
| rcarrillocruz | new patchset for the dib virtualenv revert | 20:00 |
| *** abregman|afk is now known as abregman | 20:00 | |
| rcarrillocruz | opened LP and linked it on the commit | 20:00 |
| *** Sukhdev has joined #openstack-infra | 20:03 | |
| *** ddieterly[away] is now known as ddieterly | 20:03 | |
| *** jamesmcarthur has joined #openstack-infra | 20:03 | |
| *** adrian_otto has quit IRC | 20:06 | |
| *** Apoorva has quit IRC | 20:07 | |
| *** dimtruck is now known as zz_dimtruck | 20:09 | |
| *** Apoorva has joined #openstack-infra | 20:09 | |
| *** jamesmcarthur has quit IRC | 20:10 | |
| *** karthikp_ has quit IRC | 20:11 | |
| *** karthikp_ has joined #openstack-infra | 20:11 | |
| *** jamesmcarthur has joined #openstack-infra | 20:12 | |
| *** john-davidge has quit IRC | 20:12 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Revert "Pre-install pip/virtualenv packages" https://review.openstack.org/337686 | 20:13 |
| *** zz_dimtruck is now known as dimtruck | 20:14 | |
| *** tphummel has joined #openstack-infra | 20:14 | |
| clarkb | ok time for lunch things, will work on getting that spec written and pushed after | 20:14 |
| *** jcoufal_ has joined #openstack-infra | 20:17 | |
| *** jcoufal_ has quit IRC | 20:17 | |
| *** piet has joined #openstack-infra | 20:18 | |
| *** kgiusti has left #openstack-infra | 20:19 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| rcarrillocruz | clarkb , fungi : i sent an email to the DC and NET managers at HPE , so they can give more insight about the infracloud servers move to the new datacenter | 20:20 |
| rcarrillocruz | as we agreed last week | 20:20 |
| *** jcoufal has quit IRC | 20:20 | |
| clarkb | rcarrillocruz: ty | 20:21 |
| *** karthikp_ has quit IRC | 20:21 | |
| *** Kaiyan has joined #openstack-infra | 20:22 | |
| rcarrillocruz | bnemec: you're right | 20:22 |
| rcarrillocruz | haven't thought about that, re: putting a job log on the commit | 20:22 |
| rcarrillocruz | how about pasting that on the bug, yeah? | 20:23 |
| rcarrillocruz | so it doesn't get lost | 20:23 |
| bnemec | rcarrillocruz: Yeah, just something to pull the relevant log output into a permanent location. | 20:23 |
| *** mrmartin has quit IRC | 20:24 | |
| rcarrillocruz | ah well | 20:24 |
| rcarrillocruz | i already did | 20:24 |
| rcarrillocruz | pasted the relevant bits on the bug already | 20:24 |
| rcarrillocruz | i'll just remove the job log from the commit | 20:24 |
| fungi | rcarrillocruz: thanks! | 20:26 |
| *** jamesmcarthur has joined #openstack-infra | 20:27 | |
| *** _ari_ has quit IRC | 20:27 | |
| *** EricGonczer_ has joined #openstack-infra | 20:28 | |
| *** inc0 has quit IRC | 20:29 | |
| *** hogepodge has quit IRC | 20:29 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Timeline api https://review.openstack.org/337854 | 20:30 |
| *** hogepodge has joined #openstack-infra | 20:31 | |
| *** eharney has quit IRC | 20:32 | |
| *** PollyZ_ has joined #openstack-infra | 20:33 | |
| *** Guest28713 has quit IRC | 20:34 | |
| *** abregman has quit IRC | 20:34 | |
| *** dimtruck is now known as zz_dimtruck | 20:35 | |
| rcarrillocruz | clarkb, pabelanger : greghaynes just approved the revert on dib for the pip/virtualenv issue | 20:35 |
| rcarrillocruz | just a fyi, in case you can trigger dib builds when that lands | 20:35 |
| rcarrillocruz | i'm going away now | 20:35 |
| rcarrillocruz | ttyl | 20:35 |
| Kaiyan | Hello! There is a review https://review.openstack.org/#/c/336634/ by OpenStack Proposal Bot changed copyright which is wrong. How should I handle this? Can I -1 it? | 20:36 |
| clarkb | rcarrillocruz: we need a new release forst | 20:36 |
| *** PollyZ has quit IRC | 20:36 | |
| pabelanger | agreed | 20:36 |
| *** markusry has quit IRC | 20:37 | |
| clarkb | Kaiyan: why is it wrong? | 20:37 |
| rcarrillocruz | ah right, you pull it from pip | 20:37 |
| rcarrillocruz | k | 20:37 |
| pabelanger | I won't be able to watch the rebuilds today | 20:37 |
| *** Nikolay_St has quit IRC | 20:37 | |
| pabelanger | but can confirm morning DIBs are working | 20:37 |
| EmilienM | pabelanger: fyi, first gate-tripleo-ci-centos-7-nonha-multinode in experimental queue now :) | 20:37 |
| *** karthikp_ has joined #openstack-infra | 20:37 | |
| rcarrillocruz | weee | 20:37 |
| rcarrillocruz | we got traction from kevin coker | 20:38 |
| rcarrillocruz | (now really off) | 20:38 |
| pabelanger | EmilienM: review? | 20:38 |
| EmilienM | pabelanger: https://review.openstack.org/#/c/324777/ | 20:38 |
| clarkb | Kaiyan: its writing a file that was written in 2013 so copyright ahould be correct there | 20:38 |
| pabelanger | EmilienM: node is building now | 20:39 |
| *** mhickey has quit IRC | 20:40 | |
| Kaiyan | clarkb: thanks for checking! but we've been making changes on that file and the company's name changed too. So maybe it should be 2013, 2015 hmmm | 20:40 |
| *** jamesmcarthur has quit IRC | 20:40 | |
| clarkb | Kaiyan: ehat changes, the proposal bot should give you an exact copy of what it has | 20:40 |
| clarkb | Kaiyan: so there shpuldnt be any other copyright involved | 20:40 |
| *** PollyZ has joined #openstack-infra | 20:41 | |
| *** jamesmcarthur has joined #openstack-infra | 20:42 | |
| *** PollyZ has quit IRC | 20:42 | |
| pabelanger | EmilienM: http://paste.openstack.org/show/526079/ | 20:44 |
| pabelanger | that is the failure on centos-7-node-2 | 20:44 |
| pabelanger | so we need to fix up our firewall commands | 20:44 |
| *** raildo is now known as raildo-afk | 20:44 | |
| EmilienM | slagle: fyi | 20:44 |
| *** PollyZ_ has quit IRC | 20:44 | |
| Kaiyan | clarkb: in github repo, it shows in history that 2015 is when people created that file hmmm | 20:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Make su use a login shell when collecting logs https://review.openstack.org/337859 | 20:45 |
| Kaiyan | clarkb: https://github.com/openstack/monasca-common/commits/master/setup.py | 20:45 |
| mordred | Kaiyan: yah - doesn't matter ... the copyright is on the content, not on the previous content | 20:45 |
| *** hogepodge has quit IRC | 20:45 | |
| mordred | Kaiyan: so approving that change as is is fine/correct | 20:45 |
| mordred | it looks weird because of diff view | 20:45 |
| mordred | but this is just a wholesale copying of a pre-existing different file in to place | 20:46 |
| slagle | pabelanger: EmilienM : ack, will have a look | 20:46 |
| *** dfflanders has joined #openstack-infra | 20:46 | |
| clarkb | mordred: Kaiyan arguably the old copyright was wrong... | 20:46 |
| clarkb | since its derivative of the standard which is in proposal bot | 20:46 |
| openstackgerrit | Eric Brown proposed openstack-infra/project-config: Enable py35 voting for Bandit project https://review.openstack.org/337862 | 20:46 |
| *** PollyZ has joined #openstack-infra | 20:47 | |
| mordred | yah. but in any case, in this situation, even though it looks weird, the results are correct | 20:47 |
| *** hogepodge has joined #openstack-infra | 20:47 | |
| *** zz_dimtruck is now known as dimtruck | 20:47 | |
| Sam-I-Am | whats the deal if i constantly get a server unavail when trying to review a specific file in a patch? | 20:49 |
| *** pradk- has quit IRC | 20:49 | |
| Sam-I-Am | one file works, the other file doesnt | 20:49 |
| Kaiyan | mordred: yeah... its just weird why the proposal bot is changing the copyright... | 20:49 |
| clarkb | Sam-I-Am: is the file massive? | 20:50 |
| Sam-I-Am | clarkb: https://review.openstack.org/#/c/318532/ second file | 20:50 |
| Sam-I-Am | short answer: no | 20:51 |
| *** jamesmcarthur has quit IRC | 20:51 | |
| clarkb | Sam-I-Am: the diff opens for me, what do you mean by review a file? | 20:51 |
| Sam-I-Am | well, clicking on that file | 20:52 |
| Sam-I-Am | er, the link to it | 20:52 |
| pabelanger | slagle: clarkb: So it looks like the iptables service is not started under centos-7 by default | 20:53 |
| pabelanger | which is odd | 20:53 |
| pabelanger | and makes some sense, since we do some chroot magic for services in dib | 20:53 |
| clarkb | ya worked for me then, I wonder if just general.GC slowness again | 20:53 |
| Sam-I-Am | clarkb: would that link hit something different than the other file (which works) ? | 20:54 |
| *** EricGonczer_ has quit IRC | 20:54 | |
| clarkb | Sam-I-Am: only at the git level, different object files maybe | 20:54 |
| *** gyee has joined #openstack-infra | 20:55 | |
| Sam-I-Am | its definitely consistent | 20:55 |
| clarkb | pabelanger: we disable starting in the chroot but should enable on boot | 20:55 |
| *** EricGonczer_ has joined #openstack-infra | 20:56 | |
| pabelanger | clarkb: right, for some reason, it is not starting on boot. Looking at logs now | 20:56 |
| clarkb | is it possibly firewalld problems again? | 20:56 |
| pabelanger | maybe | 20:56 |
| *** jamesmcarthur has joined #openstack-infra | 20:56 | |
| *** trown is now known as trown|outtypewww | 20:56 | |
| pabelanger | good news is, once I started the service multinode script worked as expected | 20:56 |
| *** karthikp_ has quit IRC | 20:58 | |
| *** adrian_otto has joined #openstack-infra | 20:59 | |
| *** rhallisey has quit IRC | 21:00 | |
| pabelanger | clarkb: http://paste.openstack.org/show/526082/ | 21:00 |
| Kaiyan | Thanks! | 21:00 |
| pabelanger | looks like puppet isn't enabling it | 21:00 |
| *** Kaiyan has quit IRC | 21:00 | |
| pabelanger | let me look into it | 21:01 |
| *** Sukhdev has quit IRC | 21:01 | |
| *** rhallisey has joined #openstack-infra | 21:01 | |
| *** csomerville has quit IRC | 21:01 | |
| *** csomerville has joined #openstack-infra | 21:01 | |
| *** Goneri has quit IRC | 21:02 | |
| *** Sukhdev has joined #openstack-infra | 21:02 | |
| *** maishsk has quit IRC | 21:04 | |
| fungi | clarkb: also on a related point, the git gc change doesn't seem to have had any impact on the size of the nova repo on gerrit (which is now up to 48gb on disk as of earlier today). i haven't looked yet to see if it's failing to run or really just ineffective for some reason | 21:04 |
| openstackgerrit | Eric Brown proposed openstack-infra/project-config: Enable py35 voting for Bandit project https://review.openstack.org/337862 | 21:04 |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Tasks api https://review.openstack.org/337879 | 21:05 |
| fungi | looks like our cron is still just running git repack -afd | 21:05 |
| *** rhallisey has quit IRC | 21:05 | |
| fungi | so i guess the change hasn't taken effect | 21:05 |
| *** jamesmcarthur has quit IRC | 21:07 | |
| fungi | yeah, looks like the crontab hasn't been updated at all | 21:07 |
| *** asettle has joined #openstack-infra | 21:07 | |
| *** psilvad has quit IRC | 21:07 | |
| *** blahblah has joined #openstack-infra | 21:07 | |
| clarkb | Sam-I-Am: it does open for me so not consistent across browsers at least | 21:09 |
| *** hogepodge has quit IRC | 21:09 | |
| *** jamesmcarthur has joined #openstack-infra | 21:09 | |
| pabelanger | clarkb: I see the issue: http://git.openstack.org/cgit/openstack-infra/puppet-iptables/tree/manifests/init.pp#n26 | 21:09 |
| *** tonytan4ever has quit IRC | 21:09 | |
| pabelanger | we don't setup the service properly when in a chroot | 21:09 |
| clarkb | I wonder if unbound suffers the same.issue | 21:11 |
| openstackgerrit | Josh Chase proposed openstack-infra/git-review: Fix log command used to count refs to be submitted. https://review.openstack.org/337883 | 21:12 |
| pabelanger | clarkb: likely, I see: 2016-07-05 13:34:43,997 INFO nodepool.image.build.centos-7: Notice: /Stage[main]/Ntp::Service/Service[ntp]/enable: enable changed 'false' to 'true' | 21:13 |
| pabelanger | which is what I expect | 21:13 |
| *** hogepodge has joined #openstack-infra | 21:14 | |
| *** ldnunes has quit IRC | 21:14 | |
| fungi | clarkb: aha! zaro didn't change http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/cron.pp and we dodn | 21:15 |
| fungi | didn't spot it | 21:15 |
| *** wfoster has joined #openstack-infra | 21:16 | |
| *** chem` has quit IRC | 21:16 | |
| *** jamesmcarthur has quit IRC | 21:20 | |
| *** jamesmcarthur has joined #openstack-infra | 21:20 | |
| clarkb | did we change anything? | 21:20 |
| *** mikelk has joined #openstack-infra | 21:21 | |
| openstackgerrit | Eric Brown proposed openstack-infra/project-config: Enable py35 voting for Bandit project https://review.openstack.org/337862 | 21:21 |
| *** aaltman has joined #openstack-infra | 21:22 | |
| fungi | clarkb: openstack_project::gerrit was changed via https://review.openstack.org/329566, but the only cron job in there for gerrit which got altered was a completely unused (by us) one for the local mirror | 21:22 |
| *** blahblah has quit IRC | 21:22 | |
| *** matt-borland has quit IRC | 21:23 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/storyboard: Add example commands for the Systeminfo api https://review.openstack.org/337888 | 21:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-iptables: Ensure service logic run regardless of using chroot https://review.openstack.org/337889 | 21:26 |
| pabelanger | clarkb: ^should be our fix for iptables | 21:27 |
| notmorgan | jeblair: ok so to recap, looking at gear - going to make unique and name do some auto-translate from bytes() to str() | 21:27 |
| notmorgan | jeblair: but args/data will remain explicit. | 21:28 |
| notmorgan | jeblair: since we can't know if this is supposed to be a str or not. | 21:28 |
| *** gouthamr has joined #openstack-infra | 21:28 | |
| clarkb | pabelanger: we should quickly run that in a throwaway build host to make sure the chroot puppet doesnt affect the build host (think nodepool) firewall | 21:28 |
| notmorgan | it could be legit packed data structs | 21:28 |
| notmorgan | clarkb: also will be revisiting alembic this week now that i am over the flu. | 21:28 |
| notmorgan | clarkb: i did not forget/drop it :) | 21:28 |
| pabelanger | clarkb: agreed | 21:29 |
| *** EricGonczer_ has quit IRC | 21:29 | |
| clarkb | notmorgan: ok | 21:29 |
| fungi | rcarrillocruz: the reply from kevin makes it sound like they won't need to be re-networked? | 21:29 |
| fungi | if so, that's excellent | 21:29 |
| notmorgan | clarkb: i also think i have a way to test the change - :) might need a couple minor changes to the code, but should be a single isolated test not needing the whole stack of things. | 21:30 |
| *** jamesmcarthur has quit IRC | 21:31 | |
| *** EricGonczer_ has joined #openstack-infra | 21:31 | |
| *** jamesmcarthur has joined #openstack-infra | 21:32 | |
| *** aysyd has quit IRC | 21:36 | |
| *** roxanagh_ has joined #openstack-infra | 21:36 | |
| *** EricGonczer_ has quit IRC | 21:37 | |
| *** roxanagh_ has quit IRC | 21:41 | |
| *** ddieterly is now known as ddieterly[away] | 21:42 | |
| *** sigmavirus is now known as sigmavirus_away | 21:42 | |
| openstackgerrit | Merged openstack/diskimage-builder: Revert "Pre-install pip/virtualenv packages" https://review.openstack.org/337686 | 21:42 |
| clarkb | greghaynes: ^ can has release? | 21:43 |
| *** mriedem is now known as mriedem_away | 21:43 | |
| clarkb | fungi: were you ging to propose the necessary change for gerrit git gc? | 21:43 |
| *** Sukhdev has quit IRC | 21:44 | |
| jeblair | notmorgan: re gear: yes, i think that's the thing to do -- at least for starters. i think there may be some things we can do to make dealing with args/data easier (you had an idea in your email and i think i suggested a slight change in mine), but that can be a followup. the name/unique is 90% of the issue. | 21:44 |
| notmorgan | jeblair: yeah. | 21:44 |
| *** amitgandhinz has quit IRC | 21:44 | |
| notmorgan | jeblair: exactly, just confirming and sending out a response to the ML thread as well. | 21:45 |
| notmorgan | so it's on record outside of IRC. then hacking on things (alembic, should be today, then gear) | 21:45 |
| *** Sukhdev has joined #openstack-infra | 21:45 | |
| *** amitgandhinz has joined #openstack-infra | 21:45 | |
| notmorgan | i think I might need to put a temp. upper bound on zuul since some encode things may have snuck in. | 21:45 |
| jeblair | w00t. i have found doing this in email to be very helpful. | 21:45 |
| fungi | clarkb: yeah, though it's complicated. since we're doing that in the gerrit puppet module we need to introduce it in a non-disruptive way for downstream consumers | 21:46 |
| notmorgan | jeblair: yeah, it's nice to have a record. | 21:46 |
| notmorgan | but it also is nice to have faster-than-email convos | 21:46 |
| notmorgan | :) | 21:46 |
| jeblair | yes and yes! | 21:46 |
| notmorgan | (hah i admit i was staring at the email when i asked you these things in IRC :) | 21:46 |
| clarkb | fungi: ok I won't dig into it then and will actually write this spec now | 21:46 |
| fungi | clarkb: i'm thinking of wrapping the existing cron jobs there in a conditional with the new ones as an option | 21:46 |
| fungi | jeblair: i'm sure you're still catching up on your e-mail, but i have flagged this message so i'd remember to let you know that you may want to reply http://lists.openstack.org/pipermail/openstack-dev/2016-July/098718.html | 21:49 |
| *** tonytan4ever has joined #openstack-infra | 21:49 | |
| *** jamesmcarthur has quit IRC | 21:49 | |
| *** jamesmcarthur has joined #openstack-infra | 21:55 | |
| jeblair | fungi: thanks | 21:55 |
| *** amitgandhinz has quit IRC | 21:56 | |
| *** gouthamr_ has joined #openstack-infra | 21:56 | |
| *** xyang1 has quit IRC | 21:58 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Add channel openstack-swg https://review.openstack.org/337899 | 21:58 |
| *** mikelk has quit IRC | 21:58 | |
| *** gouthamr has quit IRC | 22:00 | |
| *** HeOS has joined #openstack-infra | 22:01 | |
| *** Apoorva_ has joined #openstack-infra | 22:03 | |
| greghaynes | clarkb: yep | 22:03 |
| *** cardeois has quit IRC | 22:03 | |
| *** cardeois has joined #openstack-infra | 22:04 | |
| *** ddieterly[away] is now known as ddieterly | 22:06 | |
| *** Apoorva has quit IRC | 22:06 | |
| *** shashank_hegde has quit IRC | 22:06 | |
| *** mdrabe has quit IRC | 22:08 | |
| *** cardeois has quit IRC | 22:09 | |
| *** pradk has quit IRC | 22:10 | |
| *** ddieterly is now known as ddieterly[away] | 22:11 | |
| *** jamesmcarthur has quit IRC | 22:12 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/infra-specs: Spec for testing Newton on Xenial transition https://review.openstack.org/337905 | 22:14 |
| clarkb | fungi: ^ there we go | 22:14 |
| mgagne | so I'm trying to run Zuul in test mode on my mac and it fails with AttributeError: 'module' object has no attribute 'EPOLLET', referring to this line: https://github.com/openstack-infra/gear/blob/master/gear/__init__.py#L2429 | 22:15 |
| mgagne | would there be a way to make it mac compatible? | 22:15 |
| *** Apoorva has joined #openstack-infra | 22:16 | |
| *** jamesmcarthur has joined #openstack-infra | 22:16 | |
| *** Apoorva_ has quit IRC | 22:16 | |
| fungi | mgagne: even if there were, there'd be no way for us to guarantee it stays that way since we can't test it | 22:16 |
| clarkb | iirc the way would be to use select rather than poll which has performance implications | 22:16 |
| *** Sukhdev has quit IRC | 22:16 | |
| mordred | mgagne: not easily - osx doesn't have epoll - you have to program for osx in this differently | 22:16 |
| mordred | speaking of ... | 22:17 |
| mgagne | that's unfortunate, especially since test mode shouldn't, IMO, requires use of epoll... | 22:17 |
| *** ddieterly[away] is now known as ddieterly | 22:17 | |
| mgagne | I just want to test a config, not run the full server | 22:17 |
| clarkb | mgagne: part of testing is making sure your polling works | 22:17 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add 1.18.1 releasenotes https://review.openstack.org/337907 | 22:18 |
| mgagne | clarkb: well, depends on what you are testing | 22:18 |
| mgagne | clarkb: I'm testing that projects/jobs defined in Zuul are found in Jenkins | 22:18 |
| greghaynes | clarkb: ok, once that reno change merges I can tag a release | 22:18 |
| mgagne | I just don't want to recode something from scratch because Zuul thinks it should test polling too | 22:19 |
| fungi | it should be possible to extract out the layout parser and call it independent of the scheduler daemon | 22:19 |
| fungi | having the ability to independently parse zuul layouts seems like a useful addition | 22:20 |
| *** piet has quit IRC | 22:20 | |
| mgagne | so I suspect loading Zuul classes in a custom script and trying to get the layout parsed without actually running the daemon is the way to go | 22:22 |
| *** piet has joined #openstack-infra | 22:22 | |
| *** gouthamr_ is now known as gouthamr | 22:23 | |
| clarkb | a new cli command for that would probably make sense | 22:23 |
| * clarkb looks at zuul code real quick | 22:23 | |
| mgagne | well, I thought that "test_config" was actually testing the config (only) | 22:23 |
| clarkb | scheduler._parseConfig is what would need to be exposed I think | 22:24 |
| *** asettle has quit IRC | 22:24 | |
| clarkb | mgagne: its not because the code path is ^ | 22:24 |
| jeblair | mgagne: actually, gear is intended to be osx compatible when used as a client library, but not as a server | 22:25 |
| jeblair | mgagne: apparently it isn't, because as fungi rightly points out, we don't have a way to test it | 22:25 |
| *** jamesmcarthur has quit IRC | 22:25 | |
| jeblair | mgagne: but a patcch to gear to avoid loading the server class (or similar fix) on osx would be fine | 22:26 |
| *** Julien-zte has joined #openstack-infra | 22:26 | |
| *** erlon has quit IRC | 22:26 | |
| mgagne | yea, trying to understand how it actually works so I know where to patch | 22:26 |
| *** cody-somerville has joined #openstack-infra | 22:27 | |
| jeblair | mgagne: ok. i'm saying the simplest solution may be a patch to gear rather than a patch to zuul | 22:27 |
| *** purp_too is now known as purp | 22:27 | |
| mgagne | well, I was under the impression it was not possible due to the fact it isn't tested against osx | 22:28 |
| *** somerville32 has joined #openstack-infra | 22:29 | |
| jeblair | mgagne: i don't think a patch to change away from epoll in geard is appropriate -- but a patch to fail gracefully and keep working as a client is. | 22:30 |
| *** csomerville has quit IRC | 22:30 | |
| *** cody-somerville has quit IRC | 22:32 | |
| mgagne | I feel it is starting to fall outside my field of competences ^^' | 22:33 |
| *** jamesmcarthur has joined #openstack-infra | 22:34 | |
| clarkb | pabelanger: is the spike on the nodepool graph related to centos-7-2-node not ready scripting properly? | 22:34 |
| clarkb | I will get centos-7 images rebuilt once dib release is installed on nodepool.o.o | 22:35 |
| *** salv-orl_ has quit IRC | 22:36 | |
| *** ddieterly is now known as ddieterly[away] | 22:36 | |
| *** Julien-zte has quit IRC | 22:36 | |
| *** ddieterly[away] has quit IRC | 22:36 | |
| *** vhosakot has quit IRC | 22:36 | |
| *** salv-orlando has joined #openstack-infra | 22:37 | |
| *** karthikp_ has joined #openstack-infra | 22:38 | |
| clarkb | project-config roots https://review.openstack.org/#/c/337838/1 is probably a good idea considering our ramp up of testing on xenial | 22:40 |
| clarkb | s/roots/core/ my brain no worky | 22:40 |
| *** somerville32 has quit IRC | 22:42 | |
| *** salv-orlando has quit IRC | 22:42 | |
| *** piet has quit IRC | 22:43 | |
| *** somerville32 has joined #openstack-infra | 22:43 | |
| clarkb | mtreinish: for the preseeded subunit streams does jenkins just need to be able to read those files? | 22:44 |
| *** jamesmcarthur has quit IRC | 22:44 | |
| *** jamesmcarthur has joined #openstack-infra | 22:45 | |
| mtreinish | clarkb: it's d-g. It does a testr load on them | 22:46 |
| mtreinish | I forget which user does the op | 22:46 |
| mtreinish | clarkb: heh, we sudo it: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n696 | 22:46 |
| *** Qiming has quit IRC | 22:47 | |
| *** ggillies_ is now known as ggillies | 22:47 | |
| mtreinish | that being said I need to push a release before we land that | 22:47 |
| *** Apoorva_ has joined #openstack-infra | 22:47 | |
| clarkb | ya I started looking there too | 22:47 |
| *** Apoorva has quit IRC | 22:47 | |
| clarkb | we also -d test it | 22:47 |
| *** thorst_ has quit IRC | 22:47 | |
| *** Qiming has joined #openstack-infra | 22:47 | |
| clarkb | so we need read and execute on dir and read on the files but not write on any of them | 22:47 |
| *** rlandy has quit IRC | 22:48 | |
| *** rbrndt has quit IRC | 22:48 | |
| *** piet has joined #openstack-infra | 22:48 | |
| *** thorst has joined #openstack-infra | 22:48 | |
| *** rhallisey has joined #openstack-infra | 22:50 | |
| *** EricGonczer_ has joined #openstack-infra | 22:50 | |
| clarkb | mtreinish: you may be intersted in https://review.openstack.org/#/c/337821/2 as it affects that stuff | 22:51 |
| *** jamesmcarthur has quit IRC | 22:52 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/puppet-translation_checksite: Bump run_devstack timeout from 3200 to 7200 secs https://review.openstack.org/337912 | 22:55 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/puppet-translation_checksite: Bump run_devstack timeout from 3600 to 7200 secs https://review.openstack.org/337912 | 22:55 |
| *** thorst has quit IRC | 22:57 | |
| *** amitgandhinz has joined #openstack-infra | 22:57 | |
| *** gordc has quit IRC | 22:59 | |
| *** itisha has quit IRC | 23:00 | |
| *** EricGonczer_ has quit IRC | 23:01 | |
| mgagne | ok, script to validate Zuul layout: check ordering and test layout against list of Jenkins jobs (found in a file): https://gist.github.com/mgagne/d3d7c4e0cec4d770b5f0b95baba5d4eb | 23:01 |
| mgagne | it basically extract the logic from Zuul | 23:01 |
| *** amitgandhinz has quit IRC | 23:02 | |
| *** karthikp_ has quit IRC | 23:02 | |
| *** dprince has joined #openstack-infra | 23:04 | |
| *** xarses has quit IRC | 23:05 | |
| *** claudiub|2 has quit IRC | 23:05 | |
| *** salv-orlando has joined #openstack-infra | 23:06 | |
| notmorgan | mordred, jeblair, clarkb: how crazy in unit testing do we want for alembic. I can go as far as to implement (or source/copy) https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/test_migrations.py#L278 and verify that models in-fact match the migrations | 23:08 |
| notmorgan | but it's either grown a test-depends on oslo.db [easiest] or a lot of included/copied code. | 23:09 |
| notmorgan | and i know we strive to not have oslo* deps in nodepool (I'd prefer not to start now) | 23:09 |
| clarkb | notmorgan: I think what I had in mind was 1) make sure migrations even run and 2) that the resulting db schema matches what we have defined in the code | 23:09 |
| notmorgan | clarkb: yes. that second bit is what i'm working on | 23:09 |
| mtreinish | clarkb: you probably want to have the migration tests run with data too | 23:09 |
| notmorgan | clarkb: it's actually *really* complex to do that check. | 23:09 |
| clarkb | since migrations need not actually match | 23:09 |
| clarkb | aiui | 23:10 |
| notmorgan | right. | 23:10 |
| mtreinish | that's something that's often overlooked | 23:10 |
| clarkb | mtreinish: ya we have enough test data we could seed with | 23:10 |
| notmorgan | checking Model vs Schema is a complex task. | 23:10 |
| *** shashank_hegde has joined #openstack-infra | 23:10 | |
| clarkb | since we already load up the db pretty well with datas | 23:10 |
| notmorgan | and really should be done explicitly not just with "does the data look right" | 23:10 |
| clarkb | notmorgan: so my fear is that when we add our second migration and now there is a delta in production all of a sudden its an outage due to mismatch | 23:10 |
| clarkb | since normal testin will just do the splat a proper schema down from scratch each time right? | 23:11 |
| notmorgan | clarkb: i am 100% agreeing with you. just looking for the level of validation we need. | 23:11 |
| notmorgan | simply because with SQL-A, it is complex. DDL reflected data is not always the same as what is defined in the model, and doesn't 100% map | 23:11 |
| mtreinish | notmorgan: the oslo.db fixtures do make it easy to run the schema vs model diff. Although they're implemented in an annoying way | 23:11 |
| clarkb | notmorgan: would it be possible to describe tables and compare? I guess that the tricky thing is in a db agnostic way | 23:11 |
| notmorgan | mtreinish: nodepool has no oslo* deps | 23:11 |
| mtreinish | notmorgan: right, I agree with that for a runtime dep. It was a mistake to use it on subunit2sql | 23:12 |
| mtreinish | but as a test dep it shouldn't be that invasive | 23:12 |
| notmorgan | clarkb, jeblair: if we're ok with growing an oslo.db test-dep, this is VERY easy | 23:12 |
| *** salv-orlando has quit IRC | 23:12 | |
| clarkb | I don't have a lot of experience in the doing it right of this problem. Just in the doing it wrong and the fallout that results :) | 23:12 |
| notmorgan | clarkb: yah. | 23:13 |
| jeblair | i would not like to grow an oslo.db dep | 23:13 |
| jeblair | even for testing | 23:13 |
| jeblair | and i think we should not invest heavily in this, since, if things go well, we will be removing it all anyway | 23:13 |
| *** PollyZ has quit IRC | 23:13 | |
| notmorgan | clarkb: it just falls into "do we want to re-implement this" [why is this not in sql-a directly, (psssst zzzeek) | 23:13 |
| *** Swami has quit IRC | 23:13 | |
| notmorgan | jeblair: if we don't mind skipping FK checking, i can do this in ~40lines or so. | 23:14 |
| jeblair | so i think unit testing of "it runs without raising an exception" plus "it is run before we do other unit/functional tests" would make me happy | 23:14 |
| notmorgan | and it'll be mostly mysql-specific. | 23:14 |
| clarkb | I would be happy with an explict walk of the migrations and a sanity check at the end | 23:14 |
| notmorgan | but if we want full coverage on schema vs models, it is a lot of work. | 23:14 |
| clarkb | even if it isn't a complete schema validation | 23:14 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add valid set of options parameters to convert_xml https://review.openstack.org/337381 | 23:15 |
| mtreinish | clarkb: that's easy to do without oslo.db | 23:15 |
| jeblair | notmorgan: does not having any foreign keys make this any easier? :) | 23:15 |
| clarkb | Then changes to schemas can update/add sanity checks for their bit | 23:15 |
| notmorgan | ok i'll validate most everything and pass on the FK/relationship parts. | 23:15 |
| jeblair | notmorgan: because we don't have any :) | 23:15 |
| notmorgan | jeblair: oh wait we have no FKs.. easy then | 23:15 |
| mtreinish | clarkb: just copy: http://git.openstack.org/cgit/openstack-infra/subunit2sql/tree/subunit2sql/tests/migrations/test_migrations.py | 23:15 |
| notmorgan | no relationships = waaaay easier | 23:15 |
| jeblair | notmorgan: subnodes have an informal fk with nodes, but that's it, and it's not in the schema | 23:15 |
| notmorgan | meh, i'll just do a skip | 23:15 |
| jeblair | for reasons best discussed over beer | 23:15 |
| notmorgan | right and done then | 23:16 |
| notmorgan | easy. | 23:16 |
| *** piet has quit IRC | 23:16 | |
| *** thiagop has quit IRC | 23:17 | |
| *** rcernin has quit IRC | 23:18 | |
| cinerama | hi it's me. can we get a glean release? iirc any infra core should be able to tag | 23:21 |
| *** pvaneck has quit IRC | 23:21 | |
| *** dprince has quit IRC | 23:22 | |
| *** EricGonczer_ has joined #openstack-infra | 23:23 | |
| *** pvaneck has joined #openstack-infra | 23:23 | |
| *** hasharAway has quit IRC | 23:24 | |
| *** Apoorva has joined #openstack-infra | 23:24 | |
| *** roxanagh_ has joined #openstack-infra | 23:24 | |
| * clarkb wonders if new glean release will fix xenial | 23:24 | |
| cinerama | oh? | 23:25 |
| openstackgerrit | Merged openstack/diskimage-builder: Add 1.18.1 releasenotes https://review.openstack.org/337907 | 23:25 |
| cinerama | anyway i don't think it's been released since march? | 23:26 |
| *** Goneri has joined #openstack-infra | 23:26 | |
| *** Apoorva_ has quit IRC | 23:26 | |
| clarkb | cinerama: default dhcp on startup does not work with glean on xenial from what I can see. I have to attach a config drive then it works | 23:26 |
| clarkb | mordred: you around? you would be best person to deal with glean | 23:27 |
| * clarkb kind of wants to rebuild centos7 images before breaking everything with new glean >_> | 23:27 | |
| cinerama | huh | 23:27 |
| *** Apoorva has quit IRC | 23:29 | |
| *** Apoorva has joined #openstack-infra | 23:29 | |
| *** csomerville has joined #openstack-infra | 23:29 | |
| *** hongbin has quit IRC | 23:29 | |
| *** somerville32 has quit IRC | 23:29 | |
| *** roxanagh_ has quit IRC | 23:29 | |
| *** shashank_hegde has quit IRC | 23:30 | |
| *** Sukhdev has joined #openstack-infra | 23:31 | |
| openstackgerrit | Stephanie Miller proposed openstack/diskimage-builder: Fix proliant-tools dependencies https://review.openstack.org/337918 | 23:31 |
| greghaynes | clarkb: 1.18.1 diskimage-builder tag pushed | 23:32 |
| clarkb | ty ty | 23:33 |
| clarkb | cinerama: I am digging through the glean diff now | 23:33 |
| cinerama | greghaynes, awesome now i can land one of these "vintage" bifrost changes | 23:33 |
| cinerama | they're not old, they're just queue aged for flavor | 23:33 |
| greghaynes | cinerama: hah, glad to see less vintage in the review queue | 23:34 |
| *** permalac has quit IRC | 23:35 | |
| *** berendt has quit IRC | 23:36 | |
| *** sarob has quit IRC | 23:38 | |
| *** xarses has joined #openstack-infra | 23:38 | |
| *** sarob has joined #openstack-infra | 23:39 | |
| *** adrian_otto has quit IRC | 23:40 | |
| *** dfflanders_ has joined #openstack-infra | 23:41 | |
| *** tonytan4ever has quit IRC | 23:41 | |
| EmilienM | can someone explain why we don't have stackviz installed on centos7 nodes? http://logs.openstack.org/98/337898/1/check/gate-puppet-openstack-integration-3-scenario002-tempest-centos-7/e37f395/console.html#_2016-07-05_22_36_21_753076 | 23:42 |
| EmilienM | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/stackviz/install.d/89-prepare-stackviz#n24 | 23:42 |
| EmilienM | we should have the element: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n745 | 23:43 |
| mtreinish | clarkb: the py35 jobs are using xenial right? That means a newer version of mysql | 23:44 |
| mtreinish | :( looks like the subunit2sql migrations aren't working on whatever comes there | 23:44 |
| clarkb | cinerama: looks like writing network stuff for rhel, debuntu, and gentoo is all changed as is reading from the config drive | 23:44 |
| *** adrian_otto has joined #openstack-infra | 23:44 | |
| clarkb | cinerama: not sure I am comfortable doing a release | 23:44 |
| clarkb | mtreinish: yes and yes | 23:44 |
| clarkb | mtreinish: mysql 5.7 | 23:45 |
| cinerama | clarkb: hmm | 23:45 |
| mtreinish | clarkb: sigh, I guess that's what I get for using mariadb locally and never testing mysql | 23:45 |
| EmilienM | timothyb89: fyi about stackviz ^ | 23:45 |
| clarkb | cinerama: including removal of sorting the interfaces list for consistent ordering | 23:46 |
| mtreinish | SpamapS: want to fix: http://logs.openstack.org/11/327411/1/check/gate-subunit2sql-python35-db-nv/d0310f4/console.html#_2016-07-05_23_35_55_378838 it's in your migration :) | 23:46 |
| *** dfflanders_ has quit IRC | 23:47 | |
| clarkb | cinerama: I wonder if the removal of sorted things for debian is a bug | 23:47 |
| *** tpsilva has quit IRC | 23:48 | |
| mtreinish | EmilienM: pabelanger might have pulled it off the centos7 nodes. I know he was having trouble with it before. But I wasn't really paying too much attention | 23:48 |
| EmilienM | mtreinish: mhh good to know | 23:49 |
| clarkb | sambetts|afk: do you know why you removed the sorted iteration for debian interfaces in https://review.openstack.org/#/c/283726/4 ? | 23:50 |
| *** darvon has quit IRC | 23:54 | |
| *** thorst has joined #openstack-infra | 23:54 | |
| *** hosanai has joined #openstack-infra | 23:55 | |
| clarkb | heh also we overwrite vendor_info_file with vendor_data_file if they both exist, which is correct? | 23:56 |
| *** adrian_otto has quit IRC | 23:57 | |
| *** adrian_otto has joined #openstack-infra | 23:57 | |
| *** Goneri has quit IRC | 23:57 | |
| clarkb | building centos7 image now | 23:57 |
| *** amitgandhinz has joined #openstack-infra | 23:58 | |
| clarkb | mtreinish: do you happen to know which of the two json files for config drive vendor data/info is more "correct" | 23:58 |
| SpamapS | mtreinish: oh noes | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!