| *** Qiming has quit IRC | 00:02 | |
| *** _ari_ has joined #openstack-infra | 00:03 | |
| *** _ari_ has quit IRC | 00:04 | |
| *** ajmiller has joined #openstack-infra | 00:07 | |
| *** thorst has quit IRC | 00:07 | |
| *** zaro has quit IRC | 00:08 | |
| *** amotoki has quit IRC | 00:08 | |
| *** sflanigan has joined #openstack-infra | 00:09 | |
| *** csaba has quit IRC | 00:10 | |
| *** amotoki has joined #openstack-infra | 00:11 | |
| *** amotoki has quit IRC | 00:14 | |
| *** zaro has joined #openstack-infra | 00:17 | |
| *** baoli has quit IRC | 00:18 | |
| *** asettle has joined #openstack-infra | 00:18 | |
| *** salv-orlando has joined #openstack-infra | 00:18 | |
| *** _denisra has joined #openstack-infra | 00:20 | |
| openstackgerrit | Merged openstack-infra/project-config: The bash script has moved from tests/ to tools/ https://review.openstack.org/289043 | 00:21 |
|---|---|---|
| openstackgerrit | Richard Jones proposed openstack-infra/infra-specs: Releasing xstatic packages https://review.openstack.org/253296 | 00:22 |
| *** salv-orlando has quit IRC | 00:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix cache-bindep breakage on Fedora / CentOS https://review.openstack.org/289101 | 00:24 |
| pabelanger | ianw: if you have a moment to review ^ That enables cache-bindep on fedora | 00:24 |
| pabelanger | but it also exposed another issue with gate coverage on other-requirements.txt | 00:25 |
| pabelanger | fungi: For you too ^ | 00:25 |
| openstackgerrit | Merged openstack-infra/project-config: Make python3 gate voting in kolla https://review.openstack.org/289045 | 00:25 |
| pabelanger | TL;DR because we don't gate other-requirements.txt on centos or fedora for all projects, it is possible for cache-bindep to try and download invalid package names | 00:26 |
| pabelanger | because we loop over all git repos looking at other-requirements.txt | 00:26 |
| ianw | pabelanger: makes sense ... it's still using "yum" at the end there | 00:26 |
| pabelanger | ianw: good eye, I can change that now | 00:27 |
| ianw | i think i looked at the output of the experimental job and it worked fine | 00:27 |
| fungi | pabelanger: i've been thinking about that, and in retrospect i've come to the conclusion that we're never going to be able to enforce 100% coverage for testing other-requirements.txt additions to every project for every platform we might decide to support, so instead the cache loop probably needs to have a way to skip missing matches and move on | 00:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix cache-bindep breakage on Fedora / CentOS https://review.openstack.org/289101 | 00:28 |
| pabelanger | fungi: Ya, I was thinking about that too. | 00:28 |
| pabelanger | I plan on experimenting on it tomorrow | 00:28 |
| fungi | unfortunately my original design tried to increase efficiency by taking the union of all lists and only passing them to the package manager in a single shot | 00:28 |
| *** ddieterly[away] has quit IRC | 00:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 00:29 |
| fungi | which i think is counter to being able to have a broken project and just ignore it | 00:29 |
| openstackgerrit | Merged openstack-infra/project-config: Neutron multinode dvr - add missing PROJECTS variable https://review.openstack.org/288987 | 00:30 |
| *** [1]Thelo has joined #openstack-infra | 00:30 | |
| ianw | pabelanger / fungi : i'm pretty sure that, as a consequence of "-y", yum & dnf will both just ignore missing packages | 00:30 |
| fungi | however, firing up a package install process over and over for every single project that adds an other-requirements.txt file is also likely to take an insanely long time if the number of them grows substantially | 00:30 |
| pabelanger | okay, have to run, night time for kids, if I have time later tonight I'll play with it, otherwise tomorrow | 00:30 |
| ianw | this is an issue in devstack where we run the output through awk to pick up failures like that | 00:30 |
| fungi | ianw: yep, need to look for a similar flag to apt-get i think | 00:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add FIP graphs to OSIC grafyaml https://review.openstack.org/288967 | 00:30 |
| fungi | pabelanger: have a good evening1 | 00:31 |
| fungi | ! | 00:31 |
| ianw | fungi: it's never been clear if it's a bug or a feature. for this, a feature, for devstack, a bug :) | 00:31 |
| *** [HeOS] has quit IRC | 00:31 | |
| *** thorst has joined #openstack-infra | 00:33 | |
| *** Thelo has quit IRC | 00:33 | |
| *** [1]Thelo is now known as Thelo | 00:33 | |
| fungi | in most cases i consider it a bug to at least not have a mode which returns nonzero if the package manager install command can't satisfy your entire list of requests | 00:35 |
| fungi | doesn't have to be the default mode, but something you can at least turn on | 00:36 |
| *** amotoki has joined #openstack-infra | 00:37 | |
| *** amotoki has quit IRC | 00:39 | |
| *** angdraug has joined #openstack-infra | 00:44 | |
| *** gildub has joined #openstack-infra | 00:47 | |
| *** dimtruck is now known as zz_dimtruck | 00:47 | |
| *** tiswanso has joined #openstack-infra | 00:52 | |
| *** zz_dimtruck is now known as dimtruck | 00:53 | |
| *** tiswanso has quit IRC | 01:03 | |
| *** Sukhdev has quit IRC | 01:05 | |
| *** Qiming has joined #openstack-infra | 01:07 | |
| *** claudiub has quit IRC | 01:11 | |
| *** denisra has quit IRC | 01:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't cache inactive repos https://review.openstack.org/288958 | 01:12 |
| *** denisra has joined #openstack-infra | 01:12 | |
| *** dims has quit IRC | 01:26 | |
| *** asettle has quit IRC | 01:28 | |
| *** oomichi_ has joined #openstack-infra | 01:32 | |
| *** Jeffrey4l has joined #openstack-infra | 01:37 | |
| *** sdake has joined #openstack-infra | 01:37 | |
| *** amotoki has joined #openstack-infra | 01:37 | |
| *** Jeffrey4l has quit IRC | 01:40 | |
| *** Jeffrey4l has joined #openstack-infra | 01:40 | |
| *** asettle has joined #openstack-infra | 01:40 | |
| *** ddieterly has joined #openstack-infra | 01:41 | |
| *** asettle has quit IRC | 01:43 | |
| *** salv-orlando has joined #openstack-infra | 01:43 | |
| *** asettle has joined #openstack-infra | 01:43 | |
| *** markvoelker has quit IRC | 01:43 | |
| *** sdake_ has joined #openstack-infra | 01:44 | |
| *** sdake has quit IRC | 01:44 | |
| *** angdraug has quit IRC | 01:45 | |
| *** jamesmcarthur has joined #openstack-infra | 01:49 | |
| *** salv-orlando has quit IRC | 01:50 | |
| *** sdake has joined #openstack-infra | 01:52 | |
| *** amotoki has quit IRC | 01:52 | |
| *** _ari_ has joined #openstack-infra | 01:54 | |
| *** sdake_ has quit IRC | 01:55 | |
| *** sdake has quit IRC | 01:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove experimental bindep infra jobs https://review.openstack.org/288868 | 01:58 |
| *** tiswanso has joined #openstack-infra | 02:00 | |
| *** annegentle has joined #openstack-infra | 02:06 | |
| *** amotoki has joined #openstack-infra | 02:11 | |
| *** asettle has quit IRC | 02:12 | |
| *** baoli has joined #openstack-infra | 02:14 | |
| *** tiswanso has quit IRC | 02:15 | |
| *** baoli has quit IRC | 02:19 | |
| *** baoli has joined #openstack-infra | 02:19 | |
| *** baoli has quit IRC | 02:21 | |
| *** annegentle has quit IRC | 02:25 | |
| *** dimtruck is now known as zz_dimtruck | 02:25 | |
| *** baoli has joined #openstack-infra | 02:25 | |
| *** nikhil has joined #openstack-infra | 02:25 | |
| *** annegentle has joined #openstack-infra | 02:27 | |
| *** annegentle has quit IRC | 02:27 | |
| *** annegentle has joined #openstack-infra | 02:32 | |
| *** sdake has joined #openstack-infra | 02:33 | |
| *** annegentle has quit IRC | 02:37 | |
| *** milky has joined #openstack-infra | 02:38 | |
| *** yamamoto has quit IRC | 02:39 | |
| *** hichihara has joined #openstack-infra | 02:39 | |
| *** milky has quit IRC | 02:40 | |
| *** sdake has quit IRC | 02:40 | |
| *** annegentle has joined #openstack-infra | 02:40 | |
| *** thorst has quit IRC | 02:43 | |
| *** milky has joined #openstack-infra | 02:45 | |
| *** milky has quit IRC | 02:45 | |
| *** chlong has joined #openstack-infra | 02:45 | |
| *** milky has joined #openstack-infra | 02:46 | |
| *** milky has quit IRC | 02:46 | |
| *** jamesmcarthur has quit IRC | 02:51 | |
| *** ddieterly has quit IRC | 02:51 | |
| *** zz_dimtruck is now known as dimtruck | 02:52 | |
| *** annegentle has quit IRC | 02:53 | |
| *** yonglihe has joined #openstack-infra | 02:53 | |
| *** annegentle has joined #openstack-infra | 02:58 | |
| *** tricksters has quit IRC | 02:59 | |
| *** ajmiller has quit IRC | 03:00 | |
| *** akshai has quit IRC | 03:01 | |
| *** roxanaghe has joined #openstack-infra | 03:03 | |
| *** hdd has joined #openstack-infra | 03:04 | |
| *** hdd has quit IRC | 03:07 | |
| *** Jeffrey4l has quit IRC | 03:09 | |
| *** tiswanso has joined #openstack-infra | 03:11 | |
| *** hdd has joined #openstack-infra | 03:12 | |
| *** gongysh has joined #openstack-infra | 03:12 | |
| *** annegentle has quit IRC | 03:13 | |
| *** pahuang has quit IRC | 03:14 | |
| *** roxanaghe has quit IRC | 03:15 | |
| *** Jeffrey4l has joined #openstack-infra | 03:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.5.0 https://review.openstack.org/236815 | 03:18 |
| *** yuanying has quit IRC | 03:22 | |
| *** tiswanso has quit IRC | 03:24 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Add XStatic version checking job https://review.openstack.org/289123 | 03:26 |
| *** dimtruck is now known as zz_dimtruck | 03:29 | |
| *** zz_dimtruck is now known as dimtruck | 03:29 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Add experimental gate to kolla-mesos https://review.openstack.org/289042 | 03:32 |
| *** pahuang has joined #openstack-infra | 03:32 | |
| *** x00350071 is now known as xiangxinyong | 03:34 | |
| *** amrith is now known as _amrith_ | 03:35 | |
| *** rguillebert has quit IRC | 03:37 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add XStatic version checking job https://review.openstack.org/289123 | 03:39 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add experimental job to test xstatic packaging https://review.openstack.org/289125 | 03:39 |
| *** yuanying has joined #openstack-infra | 03:40 | |
| *** yamamoto has joined #openstack-infra | 03:40 | |
| *** dimtruck is now known as zz_dimtruck | 03:42 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Add experimental gate to kolla-mesos https://review.openstack.org/289042 | 03:43 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Add python3 non-voting job to kolla-mesos https://review.openstack.org/289046 | 03:43 |
| *** yuanying has quit IRC | 03:44 | |
| *** harlowja_at_home has joined #openstack-infra | 03:45 | |
| *** sdake has joined #openstack-infra | 03:51 | |
| openstackgerrit | LisaLi proposed openstack-infra/project-config: Enable import backup in tempest https://review.openstack.org/289128 | 03:52 |
| *** sdake has quit IRC | 03:52 | |
| *** sdake has joined #openstack-infra | 03:54 | |
| *** ayoung has quit IRC | 03:54 | |
| *** gongysh has quit IRC | 03:57 | |
| *** hdd has quit IRC | 03:58 | |
| *** jpr has quit IRC | 04:00 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add experimental job to test xstatic packaging https://review.openstack.org/289125 | 04:01 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add XStatic version checking job https://review.openstack.org/289123 | 04:01 |
| *** yuanying has joined #openstack-infra | 04:08 | |
| *** gongysh has joined #openstack-infra | 04:08 | |
| *** k4n0 has joined #openstack-infra | 04:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Run full tempest on cenots/fedora https://review.openstack.org/289130 | 04:09 |
| tonyb | c | 04:10 |
| *** sdake has quit IRC | 04:11 | |
| *** sdake has joined #openstack-infra | 04:12 | |
| *** harlowja_at_home has quit IRC | 04:14 | |
| *** links has joined #openstack-infra | 04:15 | |
| *** tiswanso has joined #openstack-infra | 04:20 | |
| *** sdake has quit IRC | 04:21 | |
| *** cloudtrainme has quit IRC | 04:22 | |
| *** jamesmcarthur has joined #openstack-infra | 04:25 | |
| *** esikachev has joined #openstack-infra | 04:26 | |
| *** salv-orlando has joined #openstack-infra | 04:26 | |
| *** tiswanso has quit IRC | 04:27 | |
| *** Sukhdev has joined #openstack-infra | 04:27 | |
| *** roxanaghe has joined #openstack-infra | 04:29 | |
| *** salv-orlando has quit IRC | 04:30 | |
| *** esikachev has quit IRC | 04:31 | |
| *** roxanaghe has quit IRC | 04:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Run full tempest on CentOS/Fedora https://review.openstack.org/289130 | 04:33 |
| *** noorul has joined #openstack-infra | 04:34 | |
| *** roxanaghe has joined #openstack-infra | 04:34 | |
| noorul | jhesketh: hi | 04:36 |
| noorul | jhesketh: I was watching https://www.youtube.com/watch?v=HlpJhqgGL34 | 04:36 |
| jhesketh | noorul: Hello | 04:37 |
| jhesketh | oh I hope it's coherent | 04:37 |
| noorul | jhesketh: In the end someone asked about gihub integration, and you said that it is not yet ready | 04:37 |
| noorul | jhesketh: I am looking to use zuul with Bitbucket | 04:37 |
| noorul | jhesketh: Bitbuckets PR system is kind of similar to Github | 04:37 |
| noorul | jhesketh: That video is 7 months old | 04:38 |
| noorul | jhesketh: Any progress on github integration? | 04:38 |
| *** roxanaghe has quit IRC | 04:39 | |
| jhesketh | noorul: so a lot of the connection stuff has been abstracted, which will make it easier to integrate with github (and also bitbucket) | 04:39 |
| jhesketh | unfortunately github isn't finished yet | 04:39 |
| noorul | Is there a branch that I can take a look? | 04:40 |
| jhesketh | noorul: heh, just pulling that up for you :-) | 04:40 |
| jhesketh | https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+branch:master+topic:github-integration | 04:40 |
| *** treaki_ has joined #openstack-infra | 04:40 | |
| noorul | jhesketh: Thank you! | 04:43 |
| jhesketh | noorul: anytime :-) | 04:44 |
| *** treaki__ has quit IRC | 04:44 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Add XStatic version checking job https://review.openstack.org/289123 | 04:46 |
| StevenK | tonyb: https://bitbucket.org/pypa/setuptools/pull-requests/179/shift-requirement-parsing-inside/diff | 04:46 |
| tonyb | StevenK: Thanks | 04:46 |
| noorul | jhesketh: And regarding replacing launchpad with redmine, how easy it will be? | 04:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Fix spurious = in dib-python readme https://review.openstack.org/286991 | 04:47 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix cloud-init-disable-resizefs README title https://review.openstack.org/286999 | 04:49 |
| jhesketh | noorul: we're not trying to do that so I'm not entirely sure sorry | 04:51 |
| jhesketh | noorul: zuul doesn't really talk to launchpad though, only gerrit, so it's not an issue from that point of view | 04:51 |
| noorul | jhesketh: Oh I see | 04:51 |
| jhesketh | noorul: craige may be able to comment about migrating off launchpad though | 04:52 |
| *** kushal has joined #openstack-infra | 04:52 | |
| *** bpokorny has joined #openstack-infra | 04:54 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix spurious = in dib-python readme https://review.openstack.org/286991 | 04:59 |
| *** jamesmcarthur has quit IRC | 05:05 | |
| *** claudiub has joined #openstack-infra | 05:07 | |
| *** claudiub has quit IRC | 05:09 | |
| *** claudiub has joined #openstack-infra | 05:09 | |
| *** chlong has quit IRC | 05:12 | |
| *** mtanino has quit IRC | 05:13 | |
| *** asettle has joined #openstack-infra | 05:18 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Update create_endpoint() https://review.openstack.org/289051 | 05:18 |
| *** asettle has quit IRC | 05:22 | |
| *** tiswanso has joined #openstack-infra | 05:24 | |
| *** roxanaghe has joined #openstack-infra | 05:25 | |
| *** amotoki has quit IRC | 05:25 | |
| *** tiswanso has quit IRC | 05:29 | |
| *** chlong has joined #openstack-infra | 05:29 | |
| *** harlowja_at_home has joined #openstack-infra | 05:31 | |
| *** bpokorny has quit IRC | 05:32 | |
| *** zz_dimtruck is now known as dimtruck | 05:35 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/infra-specs: Create elections.openstack.org https://review.openstack.org/287577 | 05:36 |
| openstackgerrit | Craige McWhirter proposed openstack-infra/system-config: Added RemoteUser Provider & OpenID configs https://review.openstack.org/254066 | 05:37 |
| openstackgerrit | Tony Breeds proposed openstack-infra/infra-specs: Create elections.openstack.org https://review.openstack.org/287577 | 05:38 |
| *** jaosorior has joined #openstack-infra | 05:43 | |
| *** roxanaghe has quit IRC | 05:43 | |
| *** __oanson has joined #openstack-infra | 05:43 | |
| *** harlowja_at_home has quit IRC | 05:47 | |
| *** rcernin has joined #openstack-infra | 05:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert some tox jobs to ubuntu-trusty https://review.openstack.org/289040 | 05:47 |
| *** Jeffrey4l has quit IRC | 05:54 | |
| *** rcernin has quit IRC | 05:57 | |
| *** abregman|afk has joined #openstack-infra | 06:02 | |
| *** amotoki has joined #openstack-infra | 06:06 | |
| *** bpokorny has joined #openstack-infra | 06:06 | |
| *** bpokorny has quit IRC | 06:07 | |
| *** Jeffrey4l has joined #openstack-infra | 06:08 | |
| *** kiran-r has joined #openstack-infra | 06:11 | |
| *** oomichi_ has quit IRC | 06:19 | |
| ianw | AJaeger: when you have a minute -> http://docs.openstack.org/developer/bashate/ gives 403 ... which is an odd error i thought. i released a 0.5.0 which i was under the impression should have updated docs site. possibly the problem is me | 06:23 |
| ianw | any suggestions? | 06:23 |
| ianw | i've accessed gerrit from several networks today, and it seems to have been a bit slower than usual from all of them | 06:24 |
| AJaeger | ianw: let's have a look at the post job... | 06:24 |
| *** tiswanso has joined #openstack-infra | 06:25 | |
| AJaeger | ianw: http://logs.openstack.org/16/16e4e0b46cfa412a463fdffe764653e59e32f1d7/release/bashate-docs/e8e466d/console.html | 06:25 |
| AJaeger | ianw: http://docs.openstack.org/developer/bashate/0.5.0/ exists | 06:26 |
| *** dguitarbite has quit IRC | 06:26 | |
| *** esikachev has joined #openstack-infra | 06:27 | |
| AJaeger | and the omissions is a logical error, you should have approved and merged https://review.openstack.org/#/c/288644/ first ;) | 06:27 |
| AJaeger | ianw: so problem was mine, I introduced in my last change a logic error and bashate confirms that logic error ;( | 06:28 |
| ianw | AJaeger: ahh, nice combination of my problem and not my problem :) | 06:28 |
| ianw | AJaeger: so probably have to do like a point release to get that to happen again? | 06:30 |
| AJaeger | you missed that knowledge ;) | 06:30 |
| AJaeger | ianw: Or you can ask me to move the content ;) | 06:30 |
| ianw | AJaeger: that sounds easier :) every day I learn something new! | 06:30 |
| * AJaeger moves the content, give me 5 mins... | 06:31 | |
| *** esikachev has quit IRC | 06:31 | |
| ianw | AJaeger: super low priority, whenever | 06:32 |
| *** tiswanso has quit IRC | 06:32 | |
| AJaeger | Either now or never ;) http://docs.openstack.org/developer/bashate/ is up | 06:33 |
| AJaeger | My ftp client moved instead of copied, so you have no 0.5.0 specific historic information | 06:33 |
| *** sridhar_ram has joined #openstack-infra | 06:34 | |
| *** dguitarbite has joined #openstack-infra | 06:38 | |
| *** _nadya_ has joined #openstack-infra | 06:40 | |
| *** dimtruck is now known as zz_dimtruck | 06:41 | |
| *** abregman|afk is now known as abregman | 06:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul: Cloner: Don't fall back on infrastructure failure https://review.openstack.org/288409 | 06:48 |
| *** sridhar_ram has quit IRC | 06:50 | |
| *** amotoki has quit IRC | 07:00 | |
| *** salv-orlando has joined #openstack-infra | 07:01 | |
| AJaeger | infra-root, if anybody is still/already around: I suggest to restart gerrit, it's getting slower and I get again 502s | 07:02 |
| *** gildub has quit IRC | 07:02 | |
| yolanda | hi AJaeger, i will be available in 15 min | 07:02 |
| AJaeger | yolanda: drink your morning coffee first ;) | 07:02 |
| *** amotoki has joined #openstack-infra | 07:03 | |
| *** asettle has joined #openstack-infra | 07:04 | |
| *** asettle has quit IRC | 07:09 | |
| *** salv-orlando has quit IRC | 07:12 | |
| yolanda | #status gerrit is going to be restarted due to bad performane | 07:19 |
| openstackstatus | yolanda: unknown command | 07:19 |
| yolanda | #status alert gerrit is going to be restarted due to bad performance | 07:20 |
| openstackstatus | yolanda: sending alert | 07:20 |
| yolanda | #status ok | 07:20 |
| *** esikachev has joined #openstack-infra | 07:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Set tags-only docs publishing for client jobs, not server jobs https://review.openstack.org/288644 | 07:22 |
| -openstackstatus- NOTICE: gerrit is going to be restarted due to bad performance | 07:22 | |
| *** ChanServ changes topic to "gerrit is going to be restarted due to bad performance" | 07:22 | |
| *** fedexo has quit IRC | 07:22 | |
| *** armax has joined #openstack-infra | 07:25 | |
| *** _nadya_ has quit IRC | 07:25 | |
| openstackstatus | yolanda: finished sending alert | 07:25 |
| openstackstatus | yolanda: sending ok | 07:25 |
| *** |-paul-| has joined #openstack-infra | 07:26 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 07:27 | |
| *** nmagnezi has joined #openstack-infra | 07:27 | |
| AJaeger | thanks, yolanda | 07:27 |
| openstackstatus | yolanda: finished sending ok | 07:28 |
| *** tiswanso has joined #openstack-infra | 07:29 | |
| *** chlong has quit IRC | 07:30 | |
| AJaeger | yolanda: once https://review.openstack.org/#/c/286778/ has merged, please re-approve https://review.openstack.org/#/c/28677 . armax will ask here. This is for removing constraints jobs from neutron | 07:31 |
| armax | AJaeger: I am going to bed in a bit | 07:31 |
| armax | ihar would take over soon, I guess | 07:31 |
| AJaeger | armax: understood. | 07:32 |
| *** sdake has joined #openstack-infra | 07:33 | |
| *** hichihara has quit IRC | 07:35 | |
| *** gildub has joined #openstack-infra | 07:39 | |
| *** tiswanso has quit IRC | 07:41 | |
| *** csaba has joined #openstack-infra | 07:43 | |
| *** zeih has joined #openstack-infra | 07:44 | |
| AJaeger | yolanda: 286777 is supposed to merge in an hour... | 07:44 |
| *** kaisers has quit IRC | 07:47 | |
| *** bradm has quit IRC | 07:48 | |
| *** zeih has quit IRC | 07:48 | |
| *** kaisers has joined #openstack-infra | 07:48 | |
| *** bradm has joined #openstack-infra | 07:49 | |
| AJaeger | yolanda: I'm offline for the rest of the day, bye for now. | 07:49 |
| *** amotoki has quit IRC | 07:50 | |
| *** jed56 has joined #openstack-infra | 07:53 | |
| *** jd__ has quit IRC | 07:56 | |
| *** jd__ has joined #openstack-infra | 07:56 | |
| *** cznewt has quit IRC | 08:05 | |
| *** mikelk has joined #openstack-infra | 08:07 | |
| *** scheuran has joined #openstack-infra | 08:09 | |
| *** cznewt has joined #openstack-infra | 08:11 | |
| *** salv-orlando has joined #openstack-infra | 08:11 | |
| *** Sukhdev has quit IRC | 08:12 | |
| *** amotoki has joined #openstack-infra | 08:13 | |
| *** salv-orlando has quit IRC | 08:19 | |
| *** BobBall_AWOL is now known as BobBall | 08:19 | |
| *** yaume has joined #openstack-infra | 08:19 | |
| *** _nadya_ has joined #openstack-infra | 08:20 | |
| *** vincentll has joined #openstack-infra | 08:21 | |
| *** sdake has quit IRC | 08:21 | |
| *** jlanoux has joined #openstack-infra | 08:23 | |
| *** mrmartin has joined #openstack-infra | 08:24 | |
| *** arxcruz has joined #openstack-infra | 08:25 | |
| *** pcaruana has joined #openstack-infra | 08:25 | |
| *** kushal has quit IRC | 08:30 | |
| *** Hal has joined #openstack-infra | 08:33 | |
| *** Hal is now known as Guest68457 | 08:34 | |
| *** tiswanso has joined #openstack-infra | 08:37 | |
| *** lezbar has joined #openstack-infra | 08:38 | |
| *** fhubik has joined #openstack-infra | 08:38 | |
| *** aloga_ is now known as aloga | 08:38 | |
| *** acabot has quit IRC | 08:43 | |
| *** daemontool_ has joined #openstack-infra | 08:43 | |
| *** daemontool has quit IRC | 08:47 | |
| *** tiswanso has quit IRC | 08:48 | |
| *** tricksters has joined #openstack-infra | 08:52 | |
| *** dingyichen has quit IRC | 08:54 | |
| *** jordanP has joined #openstack-infra | 08:54 | |
| *** rossella_s has joined #openstack-infra | 08:57 | |
| *** dizquierdo has joined #openstack-infra | 08:57 | |
| *** esikachev has quit IRC | 08:58 | |
| *** hashar has joined #openstack-infra | 08:58 | |
| *** jaosorior has quit IRC | 09:00 | |
| *** jaosorior has joined #openstack-infra | 09:00 | |
| *** __oanson is now known as oanson | 09:00 | |
| *** amotoki has quit IRC | 09:03 | |
| *** dguitarbite has quit IRC | 09:03 | |
| *** salv-orlando has joined #openstack-infra | 09:04 | |
| openstackgerrit | Julien Danjou proposed openstack-dev/pbr: package: fix wrong catch in email parsing https://review.openstack.org/289227 | 09:06 |
| *** fhubik has quit IRC | 09:07 | |
| *** shardy has joined #openstack-infra | 09:11 | |
| *** ifarkas has joined #openstack-infra | 09:12 | |
| *** max_lobur has joined #openstack-infra | 09:14 | |
| *** sputnik13_ has joined #openstack-infra | 09:14 | |
| *** jd__ has quit IRC | 09:16 | |
| *** jd__ has joined #openstack-infra | 09:16 | |
| *** kushal has joined #openstack-infra | 09:18 | |
| *** kushal has quit IRC | 09:18 | |
| *** kushal has joined #openstack-infra | 09:18 | |
| *** max_lobur has joined #openstack-infra | 09:19 | |
| *** dizquierdo has quit IRC | 09:19 | |
| openstackgerrit | Julien Danjou proposed openstack-dev/pbr: package: fix wrong catch in email parsing https://review.openstack.org/289227 | 09:21 |
| *** Guest68457 has quit IRC | 09:23 | |
| *** Hal has joined #openstack-infra | 09:25 | |
| *** Hal is now known as Guest92187 | 09:25 | |
| *** fhubik has joined #openstack-infra | 09:25 | |
| *** fhubik is now known as fhubik_brb | 09:25 | |
| openstackgerrit | Moshe Levi proposed openstack/diskimage-builder: Add lshw package to ironic-agent https://review.openstack.org/289233 | 09:32 |
| *** asettle has joined #openstack-infra | 09:33 | |
| *** jistr has joined #openstack-infra | 09:33 | |
| *** dguitarbite has joined #openstack-infra | 09:37 | |
| *** asettle has quit IRC | 09:37 | |
| *** Jeffrey4l has quit IRC | 09:37 | |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Update overtest to 0.11.0 https://review.openstack.org/281912 | 09:43 |
| *** electrofelix has joined #openstack-infra | 09:44 | |
| *** derekh has joined #openstack-infra | 09:44 | |
| *** tiswanso has joined #openstack-infra | 09:45 | |
| *** max_lobur has quit IRC | 09:49 | |
| *** dtantsur|afk is now known as dtantsur | 09:50 | |
| *** fhubik_brb is now known as fhubik | 09:51 | |
| *** fhubik has quit IRC | 09:51 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite https://review.openstack.org/274701 | 09:54 |
| *** tpsilva has joined #openstack-infra | 09:54 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/system-config: Manifest for configuration translation checksite https://review.openstack.org/276466 | 09:54 |
| yolanda | hi AJaeger, back to work i'll take a look at the changes | 09:55 |
| *** Daisy has joined #openstack-infra | 09:55 | |
| *** fhubik has joined #openstack-infra | 09:56 | |
| *** amotoki has joined #openstack-infra | 09:56 | |
| *** tiswanso has quit IRC | 09:57 | |
| *** ihrachys has joined #openstack-infra | 10:00 | |
| *** Daisy has quit IRC | 10:00 | |
| *** bogdando has quit IRC | 10:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Use non-constraint jobs for neutron/master https://review.openstack.org/286777 | 10:02 |
| *** jtomasek_ has joined #openstack-infra | 10:02 | |
| *** noorul has quit IRC | 10:04 | |
| *** sfinucan has joined #openstack-infra | 10:11 | |
| *** vdrok_ is now known as vdrok | 10:11 | |
| *** e0ne has joined #openstack-infra | 10:15 | |
| *** daemontool_ has quit IRC | 10:16 | |
| *** Qiming has quit IRC | 10:16 | |
| *** rguillebert has joined #openstack-infra | 10:19 | |
| *** max_lobur has joined #openstack-infra | 10:19 | |
| *** ociuhandu has quit IRC | 10:21 | |
| *** fhubik is now known as fhubik_brb | 10:22 | |
| *** fhubik_brb is now known as fhubik | 10:22 | |
| *** esikachev has joined #openstack-infra | 10:24 | |
| *** amotoki has quit IRC | 10:25 | |
| jamespage | AJaeger, hey - would you have time to look at https://review.openstack.org/#/c/287676/ ? | 10:27 |
| jamespage | (morning all) | 10:27 |
| dulek | Hi guys! My unit test change (https://review.openstack.org/248768) wasn't tested in Tempest. I just want to confirm that this is intentional and not some bug. It seems super cool to exclude UT changes from integration tests. :) | 10:27 |
| *** max_lobur has quit IRC | 10:27 | |
| *** sputnik13_ has quit IRC | 10:29 | |
| *** esikachev has quit IRC | 10:29 | |
| *** [1]Thelo has joined #openstack-infra | 10:30 | |
| *** andreykurilin__ has joined #openstack-infra | 10:31 | |
| *** Thelo has quit IRC | 10:32 | |
| *** [1]Thelo is now known as Thelo | 10:32 | |
| *** christian_ has joined #openstack-infra | 10:32 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use IP addresses compliant with rfc5737 https://review.openstack.org/289250 | 10:34 |
| *** daemontool has joined #openstack-infra | 10:35 | |
| *** notnownikki has joined #openstack-infra | 10:35 | |
| *** jtomasek_ has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:39 | |
| *** dims_ has joined #openstack-infra | 10:41 | |
| *** _nadya_ has quit IRC | 10:42 | |
| *** chlong has joined #openstack-infra | 10:46 | |
| *** gongysh has quit IRC | 10:49 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/project-config: WIP: add job to test Nova with latest libvirt code https://review.openstack.org/289255 | 10:50 |
| *** tiswanso has joined #openstack-infra | 10:53 | |
| openstackgerrit | Teemu Patja proposed openstack/python-jenkins: Adds support for deleting builds https://review.openstack.org/289257 | 10:53 |
| openstackgerrit | Teemu Patja proposed openstack/python-jenkins: Adds support for wiping out a job's workspace https://review.openstack.org/289258 | 10:53 |
| *** hogepodge has quit IRC | 10:56 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add IP addresses compliant with rfc5737 https://review.openstack.org/289250 | 11:00 |
| *** tiswanso has quit IRC | 11:00 | |
| *** gildub has quit IRC | 11:03 | |
| *** gildub has joined #openstack-infra | 11:04 | |
| *** Qiming has joined #openstack-infra | 11:05 | |
| *** e0ne has quit IRC | 11:06 | |
| *** e0ne has joined #openstack-infra | 11:06 | |
| *** dims_ has quit IRC | 11:09 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Add XStatic version checking job https://review.openstack.org/289123 | 11:09 |
| lxsli | Morning, http://codesearch.openstack.org/ says it's dead, what's up please? | 11:10 |
| *** tojuvone_ is now known as tojuvone | 11:12 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Add experimental job to test xstatic packaging https://review.openstack.org/289125 | 11:12 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Adding the build-discarder property https://review.openstack.org/273952 | 11:13 |
| *** christian_ has quit IRC | 11:13 | |
| *** sdague has joined #openstack-infra | 11:14 | |
| *** jcoufal has joined #openstack-infra | 11:15 | |
| yolanda | hi lxsli going to take a look | 11:17 |
| lxsli | yolanda: sweet thanks :) | 11:18 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: IPv4 network isolation testing for HA and Ceph https://review.openstack.org/288163 | 11:19 |
| openstackgerrit | Teemu Patja proposed openstack/python-jenkins: Adds support for deleting builds https://review.openstack.org/289257 | 11:19 |
| *** christian_ has joined #openstack-infra | 11:20 | |
| *** acruz has joined #openstack-infra | 11:22 | |
| *** dims_ has joined #openstack-infra | 11:22 | |
| *** arxcruz has quit IRC | 11:22 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: neutron: added back python3 unit test job to master https://review.openstack.org/289277 | 11:23 |
| yolanda | lxsli so hound is crashing , i'm seeing some panic errors | 11:23 |
| yolanda | trying to debug | 11:23 |
| lxsli | thank you | 11:23 |
| ihrachys | AJaeger: ^ we removed python34 job from neutron gate. need to get it back. | 11:24 |
| *** ldnunes has joined #openstack-infra | 11:25 | |
| ihrachys | yolanda: ^ | 11:26 |
| *** kushal has quit IRC | 11:27 | |
| *** dizquierdo has joined #openstack-infra | 11:28 | |
| *** yamamoto has joined #openstack-infra | 11:28 | |
| *** kushal has joined #openstack-infra | 11:28 | |
| *** fhubik is now known as fhubik_brb | 11:28 | |
| *** chlong has quit IRC | 11:29 | |
| *** christian_ has quit IRC | 11:30 | |
| *** hogepodge has joined #openstack-infra | 11:34 | |
| yolanda | lxsli back, there was some corrupted index | 11:35 |
| yolanda | ihrachys, i haven't started my reviews today sorry, will take a look in a while | 11:36 |
| ihrachys | yolanda: ok, I hope nothing breaking the job getting merged today :) | 11:36 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches https://review.openstack.org/288506 | 11:40 |
| yolanda | ihrachys, that will need another +2 and an extra +1 from a neutron liaison | 11:41 |
| ihrachys | yolanda: aye. let me drag someone. :) | 11:41 |
| ihrachys | mestery: are you up for an endorcement? | 11:42 |
| lxsli | yolanda: still getting 503 though | 11:42 |
| yolanda | oh i tested and was fine for a while | 11:43 |
| *** chlong has joined #openstack-infra | 11:43 | |
| yolanda | checking again | 11:43 |
| ihrachys | mestery: I mean, not political endorsement :P, just an infra patch: https://review.openstack.org/#/c/289277/ | 11:43 |
| *** _nadya_ has joined #openstack-infra | 11:43 | |
| *** aysyd has joined #openstack-infra | 11:44 | |
| yolanda | lxsli, checking again, because i get too many open files error now, it may need some tweaking | 11:44 |
| *** erlon has joined #openstack-infra | 11:45 | |
| *** gongysh has joined #openstack-infra | 11:46 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches https://review.openstack.org/288506 | 11:47 |
| yolanda | it's up again but i'll need to check ulimits on that server | 11:47 |
| *** placko_ has joined #openstack-infra | 11:47 | |
| *** _nadya_ has quit IRC | 11:48 | |
| daemontool | clarkb, yolanda when you get the chance can you take a look at https://jenkins02.openstack.org/job/gate-freezer-api-requirements/3/console ? | 11:48 |
| daemontool | we are getting the error | 11:48 |
| daemontool | 2016-03-07 11:32:06.269 | File "/usr/local/jenkins/slave_scripts/project-requirements-change.py", line 260, in <module> | 11:48 |
| daemontool | 2016-03-07 11:32:06.270 | main() | 11:48 |
| daemontool | 2016-03-07 11:32:06.270 | File "/usr/local/jenkins/slave_scripts/project-requirements-change.py", line 231, in main | 11:48 |
| daemontool | 2016-03-07 11:32:06.271 | if req.extras: | 11:48 |
| daemontool | 2016-03-07 11:32:06.272 | AttributeError: 'Requirement' object has no attribute 'extras' | 11:48 |
| *** salv-orlando has quit IRC | 11:49 | |
| yolanda | daemontool, i think i can't help you with that, seems a problem with requirements | 11:49 |
| yolanda | you may ping some requirement cores | 11:49 |
| *** salv-orlando has joined #openstack-infra | 11:49 | |
| Anticimex | i have a question on jenkins-job-builder: i tried to make a macro out of the property slack, and the publisher slack, but I get: "jenkins_jobs.errors.JenkinsJobsException: Unknown entry point or macro 'slack' for component type: 'property'." for these | 11:50 |
| * Anticimex looking into code atm | 11:50 | |
| *** aysyd has quit IRC | 11:50 | |
| daemontool | yolanda, ty | 11:50 |
| Anticimex | (jjb 1.4.0) | 11:50 |
| *** salv-orlando has quit IRC | 11:51 | |
| Anticimex | ah... that's the problem, doc I was reading is from master | 11:51 |
| *** salv-orlando has joined #openstack-infra | 11:51 | |
| lxsli | yolanda: I have search results! Thanks! | 11:51 |
| *** aysyd has joined #openstack-infra | 11:51 | |
| *** salv-orlando has quit IRC | 11:51 | |
| electrofelix | zxiiro: have a query on https://review.openstack.org/240360 when you get a chance | 11:51 |
| *** salv-orlando has joined #openstack-infra | 11:53 | |
| dims_ | daemontool : your best bet is to ping folks on #openstack-stable, i think bknudson and mriedem were looking at options (https://review.openstack.org/#/q/file:project-requirements-change.py) | 11:53 |
| *** baoli has quit IRC | 11:54 | |
| daemontool | dims_, thanks a lot! | 11:54 |
| *** _nadya_ has joined #openstack-infra | 11:56 | |
| *** tiswanso has joined #openstack-infra | 11:56 | |
| *** jtomasek_ has joined #openstack-infra | 11:57 | |
| *** fhubik_brb is now known as fhubik | 11:59 | |
| *** sdague has quit IRC | 11:59 | |
| *** lucasagomes is now known as lucas-hungry | 12:03 | |
| *** _nadya_ has quit IRC | 12:04 | |
| *** fhubik has quit IRC | 12:05 | |
| Anticimex | jenkins-job-builder's doc doesn't currently build | 12:06 |
| *** tiswanso has quit IRC | 12:06 | |
| *** amotoki has joined #openstack-infra | 12:06 | |
| *** Daisy has joined #openstack-infra | 12:07 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swap-partition.yaml environment https://review.openstack.org/289085 | 12:09 |
| *** _nadya_ has joined #openstack-infra | 12:11 | |
| *** baoli has joined #openstack-infra | 12:12 | |
| *** baoli has quit IRC | 12:12 | |
| *** rfolco has joined #openstack-infra | 12:13 | |
| *** _amrith_ is now known as amrith | 12:14 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add Due Dates to boards https://review.openstack.org/284279 | 12:16 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Display a modal when a card is clicked on https://review.openstack.org/284280 | 12:16 |
| *** jtomasek_ has quit IRC | 12:16 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 12:19 |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Upgrade Drupal Core to 7.43 https://review.openstack.org/289301 | 12:22 |
| Anticimex | now jenkins-job-builder doc/ make html soon works | 12:22 |
| *** jpr has joined #openstack-infra | 12:24 | |
| *** kushal has quit IRC | 12:24 | |
| *** fhubik has joined #openstack-infra | 12:30 | |
| mrmartin | yolanda: is everything ok with the gating? this patch: https://review.openstack.org/#/c/289301/1 simply not showing up in the zuul status queue: http://status.openstack.org/zuul/ and I not received any notification emails... | 12:31 |
| mrmartin | ok, it is there now | 12:31 |
| mrmartin | it was just slooow | 12:31 |
| yolanda | ah, patience... | 12:32 |
| *** gordc has joined #openstack-infra | 12:32 | |
| yolanda | no notification is received until the job completes the check | 12:32 |
| *** salv-orlando has quit IRC | 12:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Run full tempest on CentOS/Fedora https://review.openstack.org/289130 | 12:37 |
| *** esikachev has joined #openstack-infra | 12:39 | |
| *** kushal has joined #openstack-infra | 12:39 | |
| *** |-paul-| has quit IRC | 12:40 | |
| openstackgerrit | Merged openstack-infra/groups: Upgrade Drupal Core to 7.43 https://review.openstack.org/289301 | 12:41 |
| *** dprince has joined #openstack-infra | 12:42 | |
| *** esikachev has quit IRC | 12:43 | |
| *** sdague has joined #openstack-infra | 12:43 | |
| *** weshay has joined #openstack-infra | 12:43 | |
| *** ddieterly has joined #openstack-infra | 12:45 | |
| *** [HeOS] has joined #openstack-infra | 12:46 | |
| mrmartin | yolanda: it's ok it was just slow | 12:47 |
| *** Daisy has quit IRC | 12:48 | |
| *** ildikov has quit IRC | 12:48 | |
| *** gildub has quit IRC | 12:48 | |
| *** thorst has joined #openstack-infra | 12:49 | |
| *** ddieterly has quit IRC | 12:50 | |
| *** rhallisey has joined #openstack-infra | 12:51 | |
| *** kgiusti has joined #openstack-infra | 12:52 | |
| *** matrohon has joined #openstack-infra | 12:52 | |
| *** _nadya_ has quit IRC | 12:53 | |
| *** baoli has joined #openstack-infra | 12:55 | |
| *** salv-orlando has joined #openstack-infra | 12:56 | |
| *** ddieterly has joined #openstack-infra | 12:57 | |
| *** nikhil has quit IRC | 12:58 | |
| *** baoli has quit IRC | 12:58 | |
| *** baoli has joined #openstack-infra | 12:59 | |
| *** baoli has quit IRC | 12:59 | |
| *** abregman_ has joined #openstack-infra | 12:59 | |
| *** baoli has joined #openstack-infra | 12:59 | |
| *** abregman has quit IRC | 13:02 | |
| *** jaypipes has joined #openstack-infra | 13:02 | |
| *** Kiall has quit IRC | 13:03 | |
| *** Kiall has joined #openstack-infra | 13:03 | |
| *** tiswanso has joined #openstack-infra | 13:03 | |
| *** Kiall has quit IRC | 13:03 | |
| *** Kiall has joined #openstack-infra | 13:03 | |
| *** amrith is now known as _amrith_ | 13:03 | |
| *** gongysh has quit IRC | 13:04 | |
| *** sc68cal has joined #openstack-infra | 13:05 | |
| *** ddieterly is now known as ddieterly[away] | 13:09 | |
| *** lucas-hungry is now known as lucasagomes | 13:10 | |
| *** zhurong has joined #openstack-infra | 13:10 | |
| *** akshai has joined #openstack-infra | 13:11 | |
| *** _nadya_ has joined #openstack-infra | 13:16 | |
| *** Jeffrey4l has joined #openstack-infra | 13:16 | |
| openstackgerrit | Merged openstack-infra/project-config: gerrit: add puppet-neutron-core https://review.openstack.org/288753 | 13:17 |
| *** tiswanso has quit IRC | 13:17 | |
| *** asettle has joined #openstack-infra | 13:20 | |
| sdague | fungi: is there a reason this is not approved - https://review.openstack.org/#/c/275975 ? | 13:21 |
| *** dims_ has quit IRC | 13:25 | |
| *** [HeOS] has quit IRC | 13:27 | |
| *** links has quit IRC | 13:28 | |
| *** keedya has joined #openstack-infra | 13:28 | |
| *** edmondsw has joined #openstack-infra | 13:29 | |
| *** ddieterly[away] is now known as ddieterly | 13:29 | |
| *** nmagnezi has quit IRC | 13:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Enabling test_volume_boot_pattern for gluster jobs https://review.openstack.org/240412 | 13:31 |
| openstackgerrit | Merged openstack-infra/project-config: Add pypy build to Anchor https://review.openstack.org/218729 | 13:31 |
| openstackgerrit | Merged openstack-infra/project-config: neutron: added back python3 unit test job to master https://review.openstack.org/289277 | 13:31 |
| openstackgerrit | Merged openstack-infra/project-config: Remove constraint jobs from neutron-{fwaas,lbaas} https://review.openstack.org/288991 | 13:32 |
| openstackgerrit | Merged openstack-infra/project-config: Fix cache-bindep breakage on Fedora / CentOS https://review.openstack.org/289101 | 13:32 |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-bindep-fallback-fedora-23 voting https://review.openstack.org/289081 | 13:32 |
| openstackgerrit | Merged openstack-infra/project-config: Convert specs-jobs to ubuntu-trusty https://review.openstack.org/288596 | 13:32 |
| openstackgerrit | Merged openstack-infra/project-config: Translation script: honor comments for translators https://review.openstack.org/289038 | 13:33 |
| openstackgerrit | Merged openstack-infra/project-config: Convert windmill jobs to bindep https://review.openstack.org/288600 | 13:33 |
| *** pradk has joined #openstack-infra | 13:35 | |
| *** keedya has quit IRC | 13:36 | |
| *** keedya has joined #openstack-infra | 13:36 | |
| pabelanger | morning | 13:37 |
| rcarrillocruz | o/ | 13:37 |
| *** keedya has quit IRC | 13:37 | |
| *** keedya has joined #openstack-infra | 13:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Change kolla gate timeout to 90 minutes https://review.openstack.org/287321 | 13:38 |
| openstackgerrit | Merged openstack-infra/project-config: Rename experimental jobs https://review.openstack.org/287683 | 13:38 |
| sdague | anteaya: I wanted to double check with you on ensuring my -1 was right here - https://review.openstack.org/#/c/286113/ | 13:38 |
| *** ddieterly is now known as ddieterly[away] | 13:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Setup translation for networking-midonet https://review.openstack.org/287654 | 13:38 |
| sdague | because I think that's the current policy on claiming openstack early | 13:38 |
| openstackgerrit | Merged openstack-infra/project-config: Promote gate-neutron-fwaas-dsvm-tempest from experimental to non-voting https://review.openstack.org/287996 | 13:39 |
| *** ddieterly[away] has quit IRC | 13:39 | |
| *** kushal has quit IRC | 13:39 | |
| *** ZZelle has joined #openstack-infra | 13:40 | |
| *** aysyd has quit IRC | 13:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove non-voting jobs from the gate pipeline. https://review.openstack.org/287464 | 13:40 |
| openstackgerrit | Merged openstack-infra/project-config: Update groups for charm projects https://review.openstack.org/287676 | 13:40 |
| *** aysyd has joined #openstack-infra | 13:41 | |
| *** kushal has joined #openstack-infra | 13:41 | |
| anteaya | sdague: I agree with you | 13:44 |
| anteaya | an openstack service is the tc's decision | 13:44 |
| Anticimex | having trouble using jenkins-job-builder with macro definitions. now using jjb master from git.openstack.org. as soon as include a macro in a job, i'm getting "duplicate definition" of them | 13:44 |
| anteaya | a service that does foo is what anyone can claim | 13:45 |
| sdague | anteaya: ok, great. Felt like I needed to make sure I wasn't off base on that | 13:45 |
| anteaya | yup that makes sense to me | 13:45 |
| anteaya | I support your reasoning their | 13:45 |
| anteaya | and fair to have a second opinion | 13:45 |
| anteaya | thanks | 13:46 |
| sdague | anteaya: thanks | 13:46 |
| *** erlon has quit IRC | 13:46 | |
| lucasagomes | hi all, I think the py34 gate is broken due installing pillow on ubuntu, see: http://logs.openstack.org/90/289290/1/check/gate-ironic-python34/5969325/console.html#_2016-03-07_12_45_06_824 | 13:47 |
| *** nmagnezi has joined #openstack-infra | 13:49 | |
| *** tiswanso has joined #openstack-infra | 13:50 | |
| zxiiro | electrofelix: +1 I think that's a good idea and would make it very clear which options a version applies to. | 13:51 |
| pabelanger | lucasagomes: I believe we landed a patch over the weekend. When did that build happen? | 13:51 |
| lucasagomes | pabelanger, few minutes ago: https://review.openstack.org/#/c/289290/ | 13:51 |
| lucasagomes | pabelanger, so something may have changed and it sneaked in again | 13:52 |
| pabelanger | lucasagomes: looks like today... Hmm, maybe new dib didn't make it to OVH. Try a recheck please | 13:52 |
| *** kushal has quit IRC | 13:52 | |
| lucasagomes | pabelanger, rechecked, I let you know | 13:52 |
| pabelanger | lucasagomes: https://review.openstack.org/#/c/288738/ and https://review.openstack.org/#/c/288805/ were the fixes | 13:52 |
| pabelanger | I'll confirm with fungi and clarkb that OVH was uploaded | 13:53 |
| lucasagomes | pabelanger, thanks for the links | 13:53 |
| lucasagomes | ack, I will keep an eye on the recheck (or new patches) see if the error have reappeared | 13:53 |
| zxiiro | electrofelix: can you review this one again https://review.openstack.org/#/c/261620/ I'd really like to try to get this in soon as it'll make some work I want me intern to work on more easy. | 13:54 |
| pabelanger | lucasagomes: Ya, new image should have been uploaded yesterday | 13:54 |
| lucasagomes | pabelanger, alright, strange then... cause https://review.openstack.org/#/c/289290/ was uploaded today (~an hour ago) | 13:55 |
| *** tiswanso has quit IRC | 13:55 | |
| *** akshai has quit IRC | 13:55 | |
| pabelanger | lucasagomes: possible the jenkins slave is stale, using old image. A recheck should land you on a cloud forcing a new node build and the latest dib | 13:55 |
| pabelanger | lucasagomes: if you docs job worked, then the patch landed | 13:55 |
| fungi | pabelanger: looks like ovh-gra1 updated but ovh-bhs1 has not | 13:55 |
| *** akshai has joined #openstack-infra | 13:56 | |
| pabelanger | fungi: Ah, good to know | 13:56 |
| fungi | latest bhs-1 ubuntu-trusty image is over 6 days old | 13:56 |
| pabelanger | eep | 13:56 |
| fungi | and indeed, the log linked above ran in bhs1 | 13:57 |
| *** sdake has joined #openstack-infra | 13:57 | |
| fungi | i'll get the current images deleted there | 13:57 |
| lucasagomes | fungi, pabelanger thank you both for the info | 13:58 |
| *** kiran-r has quit IRC | 13:58 | |
| pabelanger | lucasagomes: Yup, so expect some flakyness until fungi pushes the latest upload | 13:58 |
| *** rvasilets__ has joined #openstack-infra | 13:58 | |
| Anticimex | traced the duplication of entries in module_eps to https://github.com/openstack-infra/jenkins-job-builder/blob/master/jenkins_jobs/registry.py#L208 - if macro is defined | 13:58 |
| lucasagomes | pabelanger, no problem! | 13:58 |
| Anticimex | or rather, if macro is referenced from a job definition, in my case | 13:58 |
| *** e0ne has quit IRC | 13:58 | |
| *** rlandy has joined #openstack-infra | 13:59 | |
| fungi | okay, we'll now cease building new ubuntu-trusty images in ovh-bhs1 until an updated one is uploaded. this should fix projects using pillow | 14:00 |
| *** e0ne has joined #openstack-infra | 14:00 | |
| *** ildikov has joined #openstack-infra | 14:00 | |
| fungi | yolanda: lxsli: hound crashes trying to reindex nova, hitting a too many open files condition with its git subprocess. someone probably needs to test altering the initscript to up the max files for it with ulimit | 14:03 |
| anteaya | I have tried to create a hound test instance for myself but have been unsuccessful | 14:04 |
| yolanda | fungi i updated the ulimit on the server manually to make it work , but haven't fixed that properly yet | 14:04 |
| fungi | or to figure out if it having that many open file handles is indicative of a bug somewhere which needs to be fixed | 14:04 |
| anteaya | I can't get puppet to get the hound module to run | 14:04 |
| *** _ari_ has joined #openstack-infra | 14:04 | |
| *** amitgandhinz has joined #openstack-infra | 14:04 | |
| *** abregman_ is now known as abregman | 14:08 | |
| *** ociuhandu has joined #openstack-infra | 14:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Mutex the trusty wheel mirror/release jobs https://review.openstack.org/275975 | 14:10 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Preserve data type when constructing new container https://review.openstack.org/289343 | 14:11 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Use provided mapping object type instead OrderedDict https://review.openstack.org/289344 | 14:11 |
| *** ddieterly has joined #openstack-infra | 14:11 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Take parameter order from yaml in trigger_parameterized_builds https://review.openstack.org/193660 | 14:11 |
| *** boris-42 has joined #openstack-infra | 14:12 | |
| *** woodster_ has joined #openstack-infra | 14:12 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 14:13 |
| Anticimex | ok guys, what's the workflow for sending reviews to e.g. jenkins-job-builder? | 14:13 |
| Anticimex | i have 3 doc fixes (to allow 'make html' to work again), and 1 fix to unbreak macros | 14:14 |
| Anticimex | project 'openstack-ci' @ launchpad doesn't accept bugs, as 'openstack' seems to do | 14:14 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 14:14 |
| *** Daisy has joined #openstack-infra | 14:15 | |
| openstackgerrit | Mauro Rodrigues proposed openstack-infra/devstack-gate: Make TEMPEST_OS_TEST_TIMEOUT configurable for ironic jobs https://review.openstack.org/283077 | 14:16 |
| zxiiro | waynr: can you review this patch https://review.openstack.org/261620/ when you get a chance? I think it satisfies your previous reviews. I'm hoping to get this merged in soon. | 14:18 |
| waynr | zxiiro: looking now | 14:19 |
| *** eharney has joined #openstack-infra | 14:20 | |
| *** sc68cal has quit IRC | 14:22 | |
| pabelanger | Anticimex: http://docs.openstack.org/infra/manual/developers.html explains our workflow for OpenStack projects. | 14:22 |
| *** sdague has quit IRC | 14:23 | |
| pabelanger | Anticimex: also, we don't use make html for docs, you should be able to just do: tox -edocs in the root directory | 14:23 |
| zxiiro | waynr: thanks! | 14:24 |
| zxiiro | electrofelix: thanks! | 14:24 |
| Anticimex | pabelanger: thanks | 14:24 |
| Anticimex | pabelanger: well, there are three syntax errors there now in the python docstring source. unclosed tags and the like | 14:25 |
| *** placko_ has quit IRC | 14:25 | |
| Anticimex | pabelanger: i'm familiar with the workflow on regular openstack projects such as puppet-nova or neutron. but i'm failing to find the corresponding launchpad project for jjb | 14:26 |
| pabelanger | Anticimex: fair enough, I was just noting how documentation is build for projects. We actually do significant unit testing for it, so I would be surprised if building docs was broken | 14:26 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Allow passing project_id to create_network https://review.openstack.org/288166 | 14:26 |
| pabelanger | Anticimex: we don't use launchpad for -infra projects, they are in storyboard.openstack.org | 14:26 |
| Anticimex | ah, the future | 14:27 |
| pabelanger | Anticimex: look at the README.rst for JJB | 14:27 |
| *** jpeeler has joined #openstack-infra | 14:27 | |
| *** jamesmcarthur has joined #openstack-infra | 14:27 | |
| *** markvoelker has joined #openstack-infra | 14:28 | |
| Anticimex | pabelanger: well, tox -edocs on jjb exits with return code 0, those three errors show up as warnings (which make html errors on). nonetheless, broken source | 14:28 |
| Anticimex | pabelanger: thanks! | 14:29 |
| *** tiswanso has joined #openstack-infra | 14:29 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Allow passing project_id to create_network https://review.openstack.org/288166 | 14:29 |
| pabelanger | Anticimex: okay, then something is broken. Warnings should cause our jobs to fail | 14:30 |
| *** mtanino has joined #openstack-infra | 14:30 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add tls-proxy as an available service https://review.openstack.org/188436 | 14:30 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Allow passing project_id to create_network https://review.openstack.org/288166 | 14:32 |
| *** yamamoto has quit IRC | 14:32 | |
| *** jaosorior is now known as jaosorior_climbi | 14:32 | |
| *** markvoelker has quit IRC | 14:33 | |
| pabelanger | Anticimex: so, https://review.openstack.org/#/c/229951 is the reason we didn't catch the breakage. | 14:33 |
| pabelanger | going to work on landing that | 14:33 |
| openstackgerrit | Martin Millnert proposed openstack-infra/jenkins-job-builder: Fix 3 doc warnings https://review.openstack.org/289360 | 14:34 |
| lxsli | yolanda: codesearch has broken again - just reporting, I don't need it right now | 14:35 |
| *** jamesmcarthur has quit IRC | 14:35 | |
| yolanda | lxsli it needs more efforts, to find a permanent solution | 14:35 |
| *** ccrouch has quit IRC | 14:35 | |
| rcarrillocruz | codesearch as in hound ? | 14:36 |
| *** _amrith_ is now known as amrith | 14:36 | |
| anteaya | rcarrillocruz: yes | 14:36 |
| rcarrillocruz | :/ | 14:36 |
| *** ccrouch has joined #openstack-infra | 14:36 | |
| lxsli | ah I see fungi's message now | 14:36 |
| *** kevinsho_ has joined #openstack-infra | 14:36 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add helper function to convert xml https://review.openstack.org/261620 | 14:36 |
| rcarrillocruz | what's up with it, anything i can do to help? | 14:37 |
| yolanda | rcarrillocruz, it fails with max simulteaneous files opened | 14:37 |
| *** markvoelker has joined #openstack-infra | 14:37 | |
| rcarrillocruz | ah | 14:37 |
| rcarrillocruz | so it's a tweaking game now | 14:37 |
| yolanda | i managed to make it work temporarily with ulimit, but that's just temporary | 14:37 |
| *** nelsnels_ has quit IRC | 14:38 | |
| rcarrillocruz | is ulimit set on openstack_project::server ? | 14:38 |
| lxsli | where's the code? | 14:38 |
| rcarrillocruz | lxsli: i assume it would be puppet-hound | 14:38 |
| rcarrillocruz | sec | 14:38 |
| yolanda | there is an ulimit module we can use | 14:38 |
| yolanda | i've only seen that used on jenkins slaves | 14:38 |
| Anticimex | pabelanger: ack. submitted review 289360 up there | 14:38 |
| rcarrillocruz | http://git.openstack.org/cgit/openstack-infra/puppet-hound | 14:38 |
| Anticimex | my next bug/change proposal is a bit more severe (macros broken?) | 14:38 |
| lxsli | https://github.com/etsy/hound | 14:39 |
| fungi | i think the reindex runs as root and so would need a ulimit call in the initscript to apply to the parent process environment, but i could be wrong about that | 14:39 |
| *** esikachev has joined #openstack-infra | 14:39 | |
| *** ihrachys has quit IRC | 14:40 | |
| rcarrillocruz | fungi: is anyone looking at it? | 14:40 |
| rcarrillocruz | i could take a peek if nobody is | 14:40 |
| *** zz_dimtruck is now known as dimtruck | 14:40 | |
| openstackgerrit | Nisha Agarwal proposed openstack/diskimage-builder: Add psmisc to the packages for ironic-agent https://review.openstack.org/289364 | 14:40 |
| *** ccrouch1 has joined #openstack-infra | 14:41 | |
| fungi | rcarrillocruz: not sure how deeply anyone else is into it yet. i looked at it long enough to identify from the logs that it's when the reindex process reaches nova it bombs out on hitting max open files with a git subprocess | 14:41 |
| *** jamesmcarthur has joined #openstack-infra | 14:41 | |
| lxsli | yolanda: fungi: stab in the dark but - might reducing max-concurrent-indexers to 1 help? | 14:42 |
| *** mriedem has joined #openstack-infra | 14:42 | |
| fungi | it's possible--i'm not intimately familiar with the setup there | 14:42 |
| yolanda | i'm not very familiar with hound as well | 14:42 |
| fungi | could also just be that the repos there need some periodic garbage collecting/packing | 14:42 |
| lxsli | literally just looking at the github | 14:42 |
| lxsli | on the plus side - it's broken now, can't get worse :) | 14:43 |
| yolanda | it has a limit of 1024 files, that seem to be the default, and hound is exhausting it | 14:43 |
| lxsli | (famous last words) | 14:43 |
| *** ccrouch has quit IRC | 14:43 | |
| *** asselin has joined #openstack-infra | 14:43 | |
| nibalizer | yolanda: so you bumped it manually | 14:43 |
| nibalizer | ? | 14:43 |
| *** tiswanso has quit IRC | 14:43 | |
| nibalizer | and it still hits max files? | 14:43 |
| *** esikachev has quit IRC | 14:43 | |
| yolanda | nibalizer on my session , to test, but not on the service | 14:43 |
| fungi | pabelanger: AJaeger: just a heads up, i have a feeling this is caused by the move to ubuntu-trusty http://lists.openstack.org/pipermail/openstack-dev/2016-March/088542.html | 14:43 |
| yolanda | bumping to 4096 made it work, we can add that to init.d script | 14:44 |
| yolanda | although there may be smartest solutions such as tweaking indexes | 14:44 |
| *** markvoelker_ has joined #openstack-infra | 14:44 | |
| fungi | pabelanger: AJaeger: i've followed up to that thread with some details, hoping one of the fuel devs will help us figure out if there's something we're missing or doing differently in the postgres setup script compared to what the puppet class on bare-trusty resulted in | 14:44 |
| nibalizer | yolanda: i think bumping to 4096 is fine | 14:44 |
| pabelanger | fungi: AJaeger: Sure, I can start poking around | 14:45 |
| nibalizer | can you write that patch? im in a car but can approve when i get out | 14:45 |
| yolanda | yep | 14:45 |
| fungi | yolanda: yeah, i have no objection to quadrupling it | 14:45 |
| *** dimtruck is now known as zz_dimtruck | 14:45 | |
| *** asselin__ has quit IRC | 14:45 | |
| *** hdd has joined #openstack-infra | 14:46 | |
| *** nwkarsten has joined #openstack-infra | 14:46 | |
| *** markvoelker has quit IRC | 14:47 | |
| *** sridhar_ram has joined #openstack-infra | 14:48 | |
| *** Daisy has quit IRC | 14:48 | |
| *** annegentle has joined #openstack-infra | 14:48 | |
| yolanda | i tweaked manually on the hound init script on server, going to add that to emergency hosts in ansible until we get a proper fix | 14:49 |
| *** mrmartin has quit IRC | 14:54 | |
| *** dims has joined #openstack-infra | 14:54 | |
| *** mtanino has quit IRC | 14:55 | |
| anteaya | rcarrillocruz: I had tried to take a look at it on friday by trying to set up a hound instance but was unsuccessful in getting puppet to start hound | 14:55 |
| *** pradk_ has joined #openstack-infra | 14:55 | |
| rcarrillocruz | anteaya: k, i can take a look | 14:55 |
| *** pradk has quit IRC | 14:56 | |
| anteaya | while I'm happy for the fix if 4096 is it, I would like to learn how to get a puppet module to run | 14:56 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:56 | |
| *** pradk_ is now known as pradk | 14:56 | |
| anteaya | rcarrillocruz: well I'm more saying that I would like to know how to get one of our modules to run | 14:56 |
| anteaya | if I can't learn how I'm kind of useless to the team | 14:56 |
| anteaya | unless the team is fine with that | 14:56 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Test Puppet Parser Future - Do not merge https://review.openstack.org/286732 | 14:56 |
| rcarrillocruz | anteaya: i know, but i need to take a look before sharing | 14:56 |
| *** ihrachys has joined #openstack-infra | 14:56 | |
| anteaya | rcarrillocruz: I know that is fine | 14:57 |
| rcarrillocruz | i'll let you know what i find | 14:57 |
| anteaya | rcarrillocruz: thanks | 14:57 |
| *** rbrndt_ has joined #openstack-infra | 14:57 | |
| pabelanger | fungi: If I am looking at this correctly, fuel seems to be relying on the postgres user to manage to database, not openstack_citest | 14:57 |
| anteaya | rcarrillocruz: the friday logs have my paste wtih my config plus the command I was trying | 14:57 |
| pabelanger | fungi: so the fix would be to migrate to openstack_citest to manage the database | 14:58 |
| rcarrillocruz | cool, thx | 14:58 |
| pabelanger | fungi: doing a test now | 14:58 |
| EmilienM | hello infra, if there is any Gerrit admin here, please include puppet-manager-core to https://review.openstack.org/#/admin/groups/1316,members | 14:59 |
| *** salv-orlando has quit IRC | 14:59 | |
| *** hdd has quit IRC | 15:00 | |
| *** salv-orlando has joined #openstack-infra | 15:00 | |
| *** jamesmcarthur has quit IRC | 15:00 | |
| *** baoli has quit IRC | 15:00 | |
| *** baoli has joined #openstack-infra | 15:01 | |
| *** thiagop has joined #openstack-infra | 15:02 | |
| *** ayoung has joined #openstack-infra | 15:03 | |
| *** jpr has quit IRC | 15:05 | |
| *** amitgandhinz has quit IRC | 15:06 | |
| yolanda | fungi, rcarrillocruz, more things going on with hound ... panic: runtime error: slice bounds out of range | 15:06 |
| *** amitgandhinz has joined #openstack-infra | 15:07 | |
| rcarrillocruz | hmm, o-k | 15:07 |
| *** ddecapit has joined #openstack-infra | 15:07 | |
| *** ddecapit is now known as DuaneDeC7 | 15:07 | |
| rcarrillocruz | i spinned a vm and installing puppet modules and all prior to bring it up hound | 15:07 |
| rcarrillocruz | once i have it up, will share steps with anteaya, then look at ulimit | 15:07 |
| yolanda | that's a go error, not very clear | 15:08 |
| rcarrillocruz | oh | 15:08 |
| rcarrillocruz | they use go | 15:08 |
| rcarrillocruz | :/ | 15:08 |
| yolanda | fun! | 15:08 |
| anteaya | rcarrillocruz: yeah, hound uses go | 15:08 |
| fungi | pabelanger: so it's possible we're not granting as broad permissions to the openstack_citest account now as we did with the puppet module, i suppose. any suggestions as to what else we should add to that pgsql setup macro? | 15:08 |
| *** zeih has joined #openstack-infra | 15:08 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: ignore existing default branch when setting new default branch https://review.openstack.org/289384 | 15:10 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: make args to creating stable branch more explicit https://review.openstack.org/289385 | 15:10 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add script to generate branching instructions https://review.openstack.org/289386 | 15:10 |
| *** ildikov has quit IRC | 15:11 | |
| lxsli | yolanda: that sounds like a regular IndexOutOfBoundsException to me | 15:11 |
| *** ccrouch1 has quit IRC | 15:11 | |
| yolanda | lxsli, memory problems? | 15:11 |
| lxsli | a slice is a collection dereference with a range, EG arr[1:2] | 15:11 |
| lxsli | so probably a coding error? AFAICT hound relies on Go's path/filepath module for indexing though | 15:13 |
| lxsli | not started reading that yet | 15:13 |
| *** nelsnelson has joined #openstack-infra | 15:13 | |
| yolanda | i set concurrent indexers to 1 as well, hound doesn't seem to have much more config options | 15:13 |
| *** ajmiller has joined #openstack-infra | 15:13 | |
| *** markvoelker_ has quit IRC | 15:13 | |
| *** markvoelker has joined #openstack-infra | 15:13 | |
| *** markvoelker has quit IRC | 15:14 | |
| *** markvoelker has joined #openstack-infra | 15:14 | |
| clarkb | fungi the gist of the old perms was test user owned test db and could do whatevwr it wanted to it and it had creat perms to create new dbs that it owned and could do whatever it wanted with those | 15:16 |
| fungi | clarkb: yep, that's what i tried to do with pgsql-prep at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n651 but it's possible either a) i missed something, or b) the puppet class for pgsql perms was doing more than we strictly intended | 15:17 |
| pabelanger | fungi: Hmm, would have to test, and see the overhead, but maybe call puppet from JJB to setup the database again. | 15:18 |
| pabelanger | not sure honestly | 15:18 |
| *** markvoelker has quit IRC | 15:18 | |
| fungi | the fiddly bit with those macros is that they may not be bug-compatible with our use of the equivalent puppet classes | 15:19 |
| *** markvoelker has joined #openstack-infra | 15:19 | |
| *** markvoelker has quit IRC | 15:19 | |
| *** nwkarsten has quit IRC | 15:19 | |
| fungi | but i don't want to drag puppet in as a dependency of unit test jobs if i can help it | 15:19 |
| *** nwkarsten has joined #openstack-infra | 15:19 | |
| jeblair | EmilienM: done! | 15:20 |
| EmilienM | jeblair++ | 15:20 |
| *** DuaneDeC7 has quit IRC | 15:21 | |
| *** abregman_ has joined #openstack-infra | 15:22 | |
| *** DuaneDeC7 has joined #openstack-infra | 15:22 | |
| *** jpr has joined #openstack-infra | 15:23 | |
| *** jlanoux has quit IRC | 15:23 | |
| anteaya | jeblair: have you a minute for a zuul memory leak discussion in -meeting for the third-party meeting? | 15:23 |
| jeblair | anteaya: ack | 15:24 |
| anteaya | thank you | 15:24 |
| clarkb | fungi: I dont think wr handed out superuser previously, I suppose if they are checking for too many permissions that might break now | 15:24 |
| *** nwkarsten has quit IRC | 15:25 | |
| *** sdake_ has joined #openstack-infra | 15:25 | |
| *** abregman has quit IRC | 15:25 | |
| fungi | oh, perhaps | 15:26 |
| fungi | this might explain why only fuel has hit it so far | 15:26 |
| fungi | since it looks like they have a fair amount of custom routines to do basically the same thing we're doing in the macros | 15:27 |
| *** piet has joined #openstack-infra | 15:27 | |
| *** piet has quit IRC | 15:27 | |
| *** amotoki has quit IRC | 15:27 | |
| clarkb | postgres acls are really rich and dont suffer from thr need to do that like in mysql if you just need to create and delete dbs | 15:28 |
| *** k4n0 has quit IRC | 15:28 | |
| *** sdake has quit IRC | 15:29 | |
| *** piet has joined #openstack-infra | 15:30 | |
| *** annegent_ has joined #openstack-infra | 15:30 | |
| *** noorul has joined #openstack-infra | 15:30 | |
| *** yamamoto has joined #openstack-infra | 15:33 | |
| fungi | it's possible i tried to mimic our workarounds for mysql's shortcomings too | 15:33 |
| fungi | but we also agreed it was no longer unsafe to crack those wide open once we started running unit tests on single-use workers | 15:33 |
| *** annegentle has quit IRC | 15:33 | |
| *** ddieterly is now known as ddieterly[away] | 15:35 | |
| clarkb | ya I dont think its a safety issue | 15:35 |
| *** jamesmcarthur has joined #openstack-infra | 15:35 | |
| *** Qiming has quit IRC | 15:36 | |
| openstackgerrit | Martin Oemke proposed openstack-infra/project-config: Enable Translation for tacker https://review.openstack.org/274855 | 15:36 |
| *** mtanino has joined #openstack-infra | 15:37 | |
| *** sfinucan has quit IRC | 15:37 | |
| *** sfinucan has joined #openstack-infra | 15:37 | |
| *** ddieterly[away] is now known as ddieterly | 15:38 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/project-config: WIP: add job to test Nova with latest libvirt code https://review.openstack.org/289255 | 15:38 |
| *** yamamoto has quit IRC | 15:38 | |
| fungi | also i'd like to eventually be able to scale back which jobs are using the db macros to just the projects which need them, and drop the database creation macros entirely since the jobs should be able to do that themselves with sufficient access now | 15:39 |
| *** sridhar_ram has quit IRC | 15:40 | |
| fungi | so this in theory isn't going to stick around forever anyway | 15:40 |
| *** nmagnezi has quit IRC | 15:40 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 15:41 |
| *** regXboi has joined #openstack-infra | 15:41 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: dump a release tracking dashboard https://review.openstack.org/288739 | 15:42 |
| asselin | anteaya, mmedvede just noticed line 37 says "#TODO: Update these to real e-mail addresses" | 15:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migrate misc JJB to ubuntu-trusty https://review.openstack.org/289404 | 15:44 |
| *** Sukhdev has joined #openstack-infra | 15:47 | |
| *** _nadya_ has quit IRC | 15:47 | |
| mmedvede | asselin: meaning there are correct instructions already? | 15:47 |
| *** jecarey has joined #openstack-infra | 15:48 | |
| *** dprince has quit IRC | 15:49 | |
| *** kiran-r has joined #openstack-infra | 15:49 | |
| asselin | mmedvede, yes | 15:49 |
| rcarrillocruz | anteaya: http://paste.openstack.org/show/489556/ | 15:49 |
| *** kencjohnston has joined #openstack-infra | 15:49 | |
| rcarrillocruz | that's what i followed to bring up a hound in a vm | 15:49 |
| rcarrillocruz | after puppet completes, hound is running and indexing repos | 15:50 |
| *** baoli has quit IRC | 15:50 | |
| *** harlowja_at_home has joined #openstack-infra | 15:51 | |
| *** baoli has joined #openstack-infra | 15:51 | |
| *** DuaneDeC_ has joined #openstack-infra | 15:52 | |
| yolanda | it crashed again | 15:54 |
| anteaya | asselin: ah thank you, I had missed that | 15:55 |
| anteaya | asselin: good eye | 15:55 |
| anteaya | rcarrillocruz: thank you | 15:55 |
| * anteaya looks at rcarrillocruz's paste | 15:56 | |
| *** nwkarsten has joined #openstack-infra | 15:56 | |
| *** DuaneDeC7 has quit IRC | 15:56 | |
| *** sdague has joined #openstack-infra | 15:56 | |
| anteaya | rcarrillocruz: how did you get go installed? | 15:56 |
| *** nwkarsten has quit IRC | 15:56 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 15:57 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:57 | |
| *** nwkarsten has joined #openstack-infra | 15:57 | |
| anteaya | it appears my local.pp was too long | 15:57 |
| openstackgerrit | Martin Oemke proposed openstack-infra/infra-manual: Fix path to jenkins/jobs/projects.yaml https://review.openstack.org/289415 | 15:57 |
| anteaya | as well my puppet apply command looks very different | 15:57 |
| anteaya | rcarrillocruz: thanks I'll try your approach | 15:58 |
| rcarrillocruz | the hound module installs it | 15:58 |
| anteaya | okay | 15:58 |
| rcarrillocruz | if you go to /etc/puppet/modules/hound/manifests/init.pp | 15:58 |
| anteaya | rcarrillocruz: would you like to create a readme with these instructions? | 15:58 |
| anteaya | and I can review | 15:58 |
| rcarrillocruz | you'll see there a package golang resource | 15:58 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list https://review.openstack.org/286062 | 15:58 |
| anteaya | right now the puppet-hound readme is a tad thin | 15:58 |
| *** DuaneDeC7 has joined #openstack-infra | 15:58 | |
| *** jpr has quit IRC | 15:58 | |
| anteaya | rcarrillocruz: right, yes, that makes sense | 15:58 |
| *** zeih has quit IRC | 15:59 | |
| *** ericksonsantos has joined #openstack-infra | 15:59 | |
| anteaya | while it may seem obvious to some, I would need a readme with this in it | 15:59 |
| anteaya | thanks for taking the time to help me with this | 15:59 |
| fungi | yolanda: i'm curious if you try to manually run a repack loop, whether that fixes it... like we do daily on our git servers at http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/git_backend.pp#n123 | 16:00 |
| fungi | though it may take a while to complete | 16:00 |
| *** pots79 is now known as pots | 16:00 | |
| fungi | no idea if there's already a similar cron job on the codesearch server | 16:01 |
| rcarrillocruz | anteaya: the instructions are not really hound specific, i do that workflow everytime i need to test some puppetry from openstack infra, it's already documented here: | 16:01 |
| rcarrillocruz | http://docs.openstack.org/infra/system-config/sysadmin.html | 16:01 |
| *** amitgandhinz has quit IRC | 16:02 | |
| *** nwkarsten has quit IRC | 16:02 | |
| *** DuaneDeC_ has quit IRC | 16:02 | |
| mtreinish | fungi, yolanda: it looks like the subunit gearm worker OOM'd again... | 16:02 |
| anteaya | rcarrillocruz: that may be the case | 16:02 |
| anteaya | however I was unsuccessful whilst following that document | 16:02 |
| anteaya | I belive it assumes a level of puppet paradigms that I don't possess | 16:03 |
| *** DuaneDeC7 has quit IRC | 16:03 | |
| *** amitgandhinz has joined #openstack-infra | 16:03 | |
| mtreinish | fungi, yolanda: if you could restart it on subunit-worker01.o.o | 16:03 |
| anteaya | however if you don't feel a puppet-hound readme would be helpful, that is fine | 16:03 |
| fungi | mtreinish: the plan was to rebuild on a bigger machine, right? | 16:03 |
| anteaya | rcarrillocruz: thanks again for helping me | 16:03 |
| mtreinish | fungi: well we were going to add subunit-worker02.o.o to the pool | 16:03 |
| *** DuaneDeC7 has joined #openstack-infra | 16:03 | |
| mtreinish | which should help, so scale horizontally instead | 16:03 |
| rcarrillocruz | np | 16:04 |
| fungi | mtreinish: i've restarted it now | 16:04 |
| mtreinish | fungi: but nibalizer and I never quite finished getting it running on subunit-worker02.o.o | 16:04 |
| mtreinish | we were debugging some puppet issues before I called it EOD on friday | 16:04 |
| *** annegent_ has quit IRC | 16:04 | |
| fungi | ahh, okay. so "in progress" still | 16:04 |
| *** ccrouch has joined #openstack-infra | 16:05 | |
| mtreinish | fungi: I think it's in a good state to start the worker daemon on subunit-worker02.o.o | 16:05 |
| *** nikhil has joined #openstack-infra | 16:06 | |
| mtreinish | heh, although the puppet needs some cleanup still (something is creating /etc/logstash and we have it set to absent now) | 16:06 |
| fungi | mtreinish: are unit test jobs getting indexed in there yet? | 16:06 |
| mtreinish | subunit2sql? yeah, we turned that on last month | 16:06 |
| fungi | was going to recommend it on the cinder test timing thread | 16:06 |
| fungi | mainly because i worry they have some faulty assumptions there | 16:07 |
| mtreinish | fungi: http://status.openstack.org/openstack-health/#/job/gate-nova-python27?groupKey=project&resolutionKey=hour | 16:07 |
| *** bgmccollum has left #openstack-infra | 16:07 | |
| mtreinish | fungi: well if you want cinder unit tests: http://status.openstack.org/openstack-health/#/job/gate-cinder-python27?groupKey=project&resolutionKey=hour | 16:08 |
| *** simondodsley has joined #openstack-infra | 16:08 | |
| mtreinish | fungi: also is the worker still running? I don't see anything being added to the db | 16:08 |
| *** annegentle has joined #openstack-infra | 16:09 | |
| fungi | mtreinish: no, the start seems to have returned without error, but i don't see a process running for it. debugging now | 16:11 |
| *** yolanda has quit IRC | 16:11 | |
| fungi | it didn't immediately oom at least... last oom for it was 6 days ago | 16:12 |
| clarkb | likely tge pid lockfile | 16:12 |
| clarkb | since ooms tend to not clean that out | 16:12 |
| fungi | there's no /var/run/jenkins-subunit-worker-A/jenkins-subunit-worker-A.pid | 16:12 |
| mtreinish | hmm, well things seem to have died on the 4th | 16:12 |
| *** yolanda has joined #openstack-infra | 16:12 | |
| mtreinish | maybe it wasn't an oom but another fault | 16:12 |
| mtreinish | oh, puppet | 16:12 |
| rcarrillocruz | fungi: that's a good point, re: git repack | 16:12 |
| mtreinish | nibalizer and I changed the init script | 16:12 |
| mtreinish | I bet that screwed up on worker01 | 16:13 |
| fungi | aha | 16:13 |
| rcarrillocruz | yolanda: where you see the exception, in /var/log/hound.log i assume? | 16:13 |
| yolanda | yes | 16:13 |
| fungi | mtreinish: yep, the initscript is still looking for /var/run/jenkins-subunit-worker-A/jenkins-subunit-worker-A.pid | 16:13 |
| rcarrillocruz | any condition that triggers it, or just leave it running and i'll see the same thing on my vm | 16:13 |
| rcarrillocruz | ? | 16:13 |
| *** yaume_ has joined #openstack-infra | 16:13 | |
| *** DuaneDeC7 has quit IRC | 16:13 | |
| fungi | rcarrillocruz: we can probably safely copy that log somewhere, though go tracebacks are waaaaay too huge to fit on paste.o.o | 16:14 |
| yolanda | logs are not super useful there. What i've seen is that with ulimit, the too many files error disappear, but then we get that slice error | 16:15 |
| mtreinish | fungi: I don't think I changed that part of it, we just did s/logstash/subunit iirc | 16:15 |
| * mtreinish pulls up the patch | 16:15 | |
| mtreinish | fungi: https://review.openstack.org/#/c/288706/ | 16:15 |
| *** esker has joined #openstack-infra | 16:15 | |
| *** yaume has quit IRC | 16:16 | |
| *** esker has quit IRC | 16:16 | |
| *** ccrouch has quit IRC | 16:17 | |
| *** rossella_s has quit IRC | 16:17 | |
| yolanda | rcarrillocruz, hound works fine with all the indexed projects in your vm? i think 8g may be too small | 16:18 |
| rcarrillocruz | well | 16:18 |
| rcarrillocruz | it's still going... | 16:18 |
| anteaya | fungi: you sent out the last round of summit invites already? | 16:18 |
| *** hdd has joined #openstack-infra | 16:18 | |
| rcarrillocruz | just wondering when i'd hit the same problem | 16:19 |
| anteaya | I just spotted: https://review.openstack.org/#/c/289408/2 | 16:19 |
| fungi | anteaya: yes, last week | 16:19 |
| *** ccrouch has joined #openstack-infra | 16:19 | |
| anteaya | fungi: thought so, I'll tell bswartz | 16:19 |
| yolanda | free command in codesearch shows a high usage of memory, only 600mb free, but top only shows a 4% consumption of hound | 16:19 |
| fungi | anteaya: though the more important use for that change is so that they get the ability to vote for the tc and manila ptl (if there's an election), so that's probably still soon enough | 16:20 |
| *** sbalukoff has quit IRC | 16:20 | |
| anteaya | yes | 16:20 |
| anteaya | when are the rolls created? | 16:20 |
| anteaya | has that been determined yet? | 16:20 |
| fungi | anteaya: the day before the first election | 16:21 |
| anteaya | okay thanks | 16:21 |
| *** fhubik has quit IRC | 16:21 | |
| *** dprince has joined #openstack-infra | 16:22 | |
| fungi | my new script makes project-team rolls automagically, so we no longer need to know which races to expect for ptl to be able to save unnecessary manual aggregation work | 16:22 |
| anteaya | yay! | 16:22 |
| *** rossella_s has joined #openstack-infra | 16:22 | |
| anteaya | less unnecessary manual aggregation | 16:22 |
| fungi | so i'll be able to provide the officials with a tarball with rolls for every big-tent team and they can just ignore the majority of them and use the ones they need for whichever elections come up | 16:23 |
| anteaya | awesome | 16:24 |
| anteaya | well done on the script work! | 16:24 |
| fungi | thanks--this will be the first election to use it, so i'm hoping all goes well | 16:24 |
| rcarrillocruz | yolanda: i have about half mem used on my instance | 16:25 |
| rcarrillocruz | but it's a 11G tho | 16:25 |
| rcarrillocruz | around 6G free | 16:25 |
| fungi | anteaya: i'll be able to provide detailed structured data about all eligible voters (in yaml) plus autogenerated deduplicated lists of their preferred e-mail addresses suitable for feeding directly to civs | 16:25 |
| yolanda | i wonder if that's some index corruption | 16:25 |
| *** kevinsho_ has quit IRC | 16:27 | |
| fungi | anteaya: it does some particularly rigorous iterating through all addresses of all voters to try and spot duplicate accounts where there may even be only indirect ties of non-preferred addresses in common between them | 16:27 |
| *** rhedlind has joined #openstack-infra | 16:27 | |
| anteaya | fungi: apparently the eligibility period for the elecorate is closed: https://wiki.openstack.org/wiki/PTL_Elections_March_2016 | 16:28 |
| anteaya | fungi: I hope the scripts work well too | 16:28 |
| anteaya | fancy | 16:28 |
| anteaya | very nice | 16:29 |
| anteaya | so bswartz's patch to govenance won't result in that extra-atc getting a chance to vote | 16:29 |
| fungi | i've already been using them to generate the csv files for summit invites, so they're at least working well to that end so far | 16:29 |
| anteaya | nice | 16:29 |
| *** jaosorior_climbi is now known as jaosorior | 16:29 | |
| anteaya | according the the wikipage the eligibility period is March 4, 2015 00:00 UTC to March 3, 2016 23:59 UTC | 16:30 |
| anteaya | my calandar says it is march 7th | 16:30 |
| fungi | missing some useful features for that (particularly tracking changes to the output between runs, which i've been having to do manually for invite batches after the first one for now) but other than that no issues | 16:30 |
| *** dguitarbite has quit IRC | 16:30 | |
| anteaya | that's awesome | 16:30 |
| *** bpokorny has joined #openstack-infra | 16:30 | |
| fungi | anteaya: oh, if the eligibility period has ended, then i guess that extra-atcs addition for manila won't be effective until next cycle | 16:30 |
| anteaya | that is the way I am interpreting what I am reading | 16:31 |
| clarkb | are extra atcs not evaluated on when their contribution was? | 16:31 |
| *** annegentle has quit IRC | 16:32 | |
| anteaya | their timestamp starts when the tc approves their inclusion | 16:32 |
| anteaya | it is for 365 days according to the by-laws | 16:32 |
| anteaya | but we go by month | 16:32 |
| anteaya | I'm guessing for convience | 16:32 |
| *** oomichi_ has joined #openstack-infra | 16:33 | |
| sdague | btw, university standard policy of blocking all non http ports is .... challenging | 16:33 |
| *** abregman_ has quit IRC | 16:33 | |
| *** jamesmcarthur has quit IRC | 16:34 | |
| *** asettle has quit IRC | 16:34 | |
| *** sdake_ is now known as sdake | 16:35 | |
| *** yamamoto has joined #openstack-infra | 16:35 | |
| fungi | clarkb: it's a loose evaluation since it may not be code contribution | 16:35 |
| fungi | it's basically up to each ptl | 16:35 |
| *** ddieterly is now known as ddieterly[away] | 16:36 | |
| fungi | for example, the i18n ptl for some reason decided to only count translations from the previous 6 months rather than a year (though that may be a one-time bootstrapping issue dur to the limited amount of history they have in zanata) | 16:36 |
| *** jamesmcarthur has joined #openstack-infra | 16:37 | |
| clarkb | right but it sounds like if the change to say "this person contributed last cycle" merged today it wouldn't be considered | 16:38 |
| clarkb | even though the assertion is that the work was done last cycle | 16:38 |
| *** ddieterly[away] is now known as ddieterly | 16:39 | |
| *** asettle has joined #openstack-infra | 16:39 | |
| *** yamahata has quit IRC | 16:39 | |
| clarkb | in other news, the vast majority of the green stripe on the nodepool graph is bare-trusty | 16:40 |
| clarkb | we should proably reduce the min ready on those at this point and maybe delete the older ones | 16:40 |
| *** esikachev has joined #openstack-infra | 16:40 | |
| *** yamamoto has quit IRC | 16:40 | |
| pabelanger | clarkb: Ya, I was trying to look into that. I think we are in good shape to reduce them. | 16:40 |
| fungi | yeah, reducing them will probably work the old ones out of rotation fairly quickly still | 16:41 |
| fungi | may not need to do any manual deletes to free up the quota | 16:41 |
| clarkb | huh min ready is already 8 for them | 16:41 |
| clarkb | which is low compared to the ~100 we have ready | 16:41 |
| fungi | sounds more like a leak somewhere then | 16:41 |
| clarkb | ya | 16:42 |
| fungi | maybe a jenkins master ceased emitting zmq or something | 16:42 |
| pabelanger | our currently topic for the bindep migration: https://review.openstack.org/#/q/status:open+topic:bindep | 16:42 |
| pabelanger | aside with fuel-web this morning, I didn't see any other failures | 16:42 |
| clarkb | its distributed across jenkins masters so I don't think that is it | 16:42 |
| clarkb | I wonder if we had demand that was switched out by a jjb update? | 16:43 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for novaclient bug 1553597 https://review.openstack.org/289440 | 16:43 |
| openstack | bug 1553597 in python-novaclient "test_trigger_crash_dump_in_res continues failing" [Undecided,In progress] https://launchpad.net/bugs/1553597 - Assigned to Ken'ichi Ohmichi (oomichi) | 16:43 |
| *** emmenemoi has joined #openstack-infra | 16:43 | |
| *** asettle has quit IRC | 16:43 | |
| jeblair | clarkb, fungi: we should restart zuul to fix the demand problem | 16:44 |
| jeblair | clarkb, fungi: nodepool is likely associating the wrong nodes with demand based on obsolete gearman function registrations | 16:44 |
| *** vincentll has quit IRC | 16:44 | |
| fungi | oh, great point | 16:45 |
| anteaya | yeah zuul is into swap again | 16:45 |
| fungi | it sees the jobs registered for both | 16:45 |
| emmenemoi | Hello, I'm trying to push a patch to the kolla project, but git review throws an error: "fatal: ICLA contributor agreement requires current contact information." | 16:45 |
| *** esikachev has quit IRC | 16:45 | |
| fungi | emmenemoi: did you follow all the steps in http://docs.openstack.org/infra/manual/developers.html#account-setup | 16:46 |
| *** david_lyle has quit IRC | 16:46 | |
| emmenemoi | I also have an error on review.openstack.org: "Server Error - Cannot store contact information" | 16:46 |
| *** david_lyle has joined #openstack-infra | 16:46 | |
| fungi | emmenemoi: most common cause for those errprs is not following the part of the instructions to set up a foundation member account first | 16:47 |
| *** jpr has joined #openstack-infra | 16:47 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix 3 doc warnings https://review.openstack.org/289360 | 16:48 |
| openstackgerrit | Deklan Dieterly proposed openstack-infra/project-config: Modify freezer job for tempest tests https://review.openstack.org/289442 | 16:48 |
| emmenemoi | ok I'll check again | 16:48 |
| *** oomichi_ has quit IRC | 16:49 | |
| *** Daisy has joined #openstack-infra | 16:49 | |
| clarkb | fungi: btw https://review.openstack.org/#/c/289054/2 adds in zerofree support to dib | 16:49 |
| *** oomichi_ has joined #openstack-infra | 16:49 | |
| *** e0ne has quit IRC | 16:49 | |
| *** claudiub has quit IRC | 16:49 | |
| fungi | emmenemoi: and if you were following developer setup instructions from somewhere other than that link, please let me know so i can get them removed/redirected to our official instructions | 16:49 |
| tristanC | anteaya: fungi: tbh, eligibility timeframe is a best guess, we tried to make it match mitaka feature freeze. However the atc rolls needs to be generated on March 17 (which is the documented limit to configure gerrit preferred email address). Does that works for you ? | 16:49 |
| *** yaume_ has quit IRC | 16:50 | |
| *** matrohon has quit IRC | 16:50 | |
| fungi | tristanC: yeah, it works out. i think the surprise was that manila tried to add an extra-atcs entry today | 16:50 |
| fungi | which is after the eligibility window | 16:50 |
| *** cschwede has quit IRC | 16:50 | |
| *** yaume has joined #openstack-infra | 16:50 | |
| fungi | so no idea if that makes it into the electorate or not | 16:51 |
| *** mikelk has quit IRC | 16:51 | |
| *** dguitarbite has joined #openstack-infra | 16:51 | |
| *** fedexo has joined #openstack-infra | 16:51 | |
| *** gnuoy_ has joined #openstack-infra | 16:51 | |
| *** gnuoy_ has quit IRC | 16:51 | |
| mordred | infra-root: y'all ok with the shade release now? I just got an ansible-related report of things breaking because of the neutronclient thing | 16:51 |
| yolanda | ++ | 16:52 |
| clarkb | mordred: I am though I have to disappear in the middle of the day to visit the dentist and other errands | 16:52 |
| tristanC | fungi: noted, I'll check that with tonyb | 16:52 |
| mordred | clarkb: shouldn't be a problem - everything is going to work perfectly :) | 16:52 |
| jeblair | mordred: ++ | 16:52 |
| *** yaume_ has joined #openstack-infra | 16:52 | |
| nibalizer | mordred: ++ | 16:52 |
| mordred | tag pushed | 16:53 |
| rcarrillocruz | \o/ | 16:53 |
| mordred | Sam-I-Am: ^^ as soon as 1.5.0 releases to PyPI you should be good | 16:53 |
| *** Daisy has quit IRC | 16:53 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: add IPv6 gate job https://review.openstack.org/289445 | 16:53 |
| clarkb | mordred: fungi we should restart nodepool and nodepool builder once all image builds are done too | 16:54 |
| clarkb | to pick up new shade (double check new shade is installed first) | 16:54 |
| mordred | clarkb: ++ | 16:55 |
| mordred | it shold make the swift uploads betterer | 16:55 |
| *** yaume has quit IRC | 16:55 | |
| mordred | Shrews: oh - btw ^^ pushed shade tg | 16:55 |
| clarkb | ianw: looks like we can't remove that trove image from our image cache, maybe we can at least get them to build a smaller image? | 16:55 |
| clarkb | 500MB for mysql seems large, but I havne't looked into how they are building it | 16:55 |
| Shrews | mordred: ++ | 16:56 |
| *** armax has quit IRC | 16:56 | |
| anteaya | clarkb: happy dentisting | 16:56 |
| *** noorul has quit IRC | 16:56 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for novaclient bug 1553597 https://review.openstack.org/289440 | 16:56 |
| *** oomichi_ has quit IRC | 16:56 | |
| openstack | bug 1553597 in python-novaclient "test_trigger_crash_dump_in_res continues failing" [Undecided,In progress] https://launchpad.net/bugs/1553597 - Assigned to Ken'ichi Ohmichi (oomichi) | 16:56 |
| anteaya | tristanC: okay thanks for the explanation | 16:57 |
| fungi | mordred: yeah, now seems like a fine time for a shade release | 16:57 |
| anteaya | so did zuul get a restart? | 16:58 |
| anteaya | or is that in progress? | 16:58 |
| fungi | clarkb: it trove is using dib to build that, maybe the zerofree element will improve the situation dramatically | 16:58 |
| fungi | anteaya: i don't believe anyone has restarted zuul yet | 16:59 |
| anteaya | ah | 16:59 |
| clarkb | ya that may be, I wonder if they are installing extra things that have large dep lists | 16:59 |
| clarkb | that may actually just be what mysql does too :/ | 16:59 |
| anteaya | okay well I'm supportive of a zuul restart, as it is in swap, if anyone is counting votes | 16:59 |
| *** jaosorior has quit IRC | 17:00 | |
| emmenemoi | @fungi: no no, I always follow "first level sources" when possible. So I followed instructions on openstack site. But I remember something regarding openstack foundation, maybe I though it wasn't mandatory. | 17:00 |
| *** jaosorior has joined #openstack-infra | 17:00 | |
| fungi | mtreinish: so the initscript says the logfile should be /var/log/subunit2sql/subunit-worker-A-debug.log but the /var/log/subunit2sql doesn't even exist there | 17:01 |
| *** salv-orlando has quit IRC | 17:01 | |
| fungi | so maybe still some puppet issues to work out? | 17:01 |
| mtreinish | ugh, that's probably the issue | 17:01 |
| mtreinish | fungi: although we didn't create the old log dir in the puppet | 17:01 |
| anteaya | emmenemoi: okay thanks for letting us know what instructions you are following | 17:01 |
| nibalizer | fungi: mtreinish are you working on worker01 ? | 17:01 |
| nibalizer | or worker02? | 17:01 |
| anteaya | emmenemoi: it is mandatory to have a foundation account | 17:01 |
| nibalizer | if worker02 is functional I kindof want to just junk 01 and replace it or build 03 | 17:02 |
| mtreinish | nibalizer: heh, both :) | 17:02 |
| anteaya | emmenemoi: and I belive the instructions tell you what kind of foundation account you need to have | 17:02 |
| nibalizer | ah ok | 17:02 |
| fungi | nibalizer: not "working" per se, just investigating in between the other half dozen things i'm juggling | 17:02 |
| emmenemoi | sure | 17:02 |
| fungi | nibalizer: if you have more context and available bandwidth now, it's all yours | 17:02 |
| nibalizer | sure | 17:03 |
| nibalizer | i can work with mtreinish on that | 17:03 |
| emmenemoi | that's why I'm surprised not to have a community account :) | 17:03 |
| *** ifarkas has quit IRC | 17:03 | |
| Sam-I-Am | mordred: cool | 17:03 |
| emmenemoi | sorry. Foundation account | 17:03 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-hound: Add ulimit max-open-files param to puppet-hound https://review.openstack.org/289451 | 17:03 |
| rcarrillocruz | yolanda: ^ | 17:03 |
| rcarrillocruz | not sure if leaving default on instance with 2048 | 17:04 |
| rcarrillocruz | see how it goes | 17:04 |
| yolanda | i think it will be ok for our current usage | 17:04 |
| rcarrillocruz | if not enough, i push a change for system-config site.pp to 4096 | 17:04 |
| fungi | rcarrillocruz: so were you able to reproduce the max files crash? | 17:04 |
| *** piet has quit IRC | 17:04 | |
| yolanda | i have set manually to 1500 | 17:04 |
| fungi | on a clean deployment? | 17:04 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 17:04 |
| rcarrillocruz | fungi: unfortunately, the max files crash i haven't hit it on a clean deployment | 17:05 |
| rcarrillocruz | (so far...) | 17:05 |
| *** yamahata has joined #openstack-infra | 17:05 | |
| *** bgmccollum has joined #openstack-infra | 17:05 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Ensure worker log dir is created https://review.openstack.org/289452 | 17:06 |
| fungi | rcarrillocruz: so that does lend credence to the possibility of the repos getting crufty over time without a periodic repack | 17:06 |
| mtreinish | nibalizer, fungi: ^^^ | 17:06 |
| *** zhurong has quit IRC | 17:07 | |
| nibalizer | mtreinish: are the perms on that right? | 17:07 |
| nibalizer | seems like if its root:root then the subunit user wouldn't be able to log there | 17:08 |
| fungi | mtreinish: er, does it need to be owned by subunit? | 17:08 |
| mtreinish | oh, good call. I should have actually thought about it instead of jsut blindly copying and pasting | 17:08 |
| fungi | yeah, what i was just noticing when i went back to look in the initscript | 17:08 |
| fungi | USER=subunit | 17:08 |
| rcarrillocruz | fungi: i think the max files issue is separate to the slice out of bounds. Per yolanda, looks like the max files is fixed with the ulimit, the slice out of bounds due to crufty repos | 17:08 |
| rcarrillocruz | yolanda: can you confirm | 17:08 |
| rcarrillocruz | ? | 17:08 |
| yolanda | rcarrillocruz, yes, two different issues | 17:09 |
| rcarrillocruz | ++ | 17:09 |
| yolanda | first i was seeing the limit of concurrent open files | 17:09 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Ensure worker log dir is created https://review.openstack.org/289452 | 17:09 |
| mtreinish | fungi, nibalizer: ^^^ take 2 | 17:09 |
| yolanda | when i upped that, a new error raised , a go error related with indexing | 17:09 |
| fungi | rcarrillocruz: or possibly a corrupt index from an earlier aborting of the update at a random spot because of the max-files bug | 17:09 |
| *** piet has joined #openstack-infra | 17:09 | |
| yolanda | it was fixed by trashing the index and regenerated again | 17:09 |
| fungi | yolanda: sounds like that's confirmed then | 17:10 |
| rcarrillocruz | ah, that makes sense | 17:10 |
| fungi | so anyway, the max open files issue may be due to not repacking the repos regularly, and hound may also not be very transactional in its index updating | 17:10 |
| *** fawadkhaliq has joined #openstack-infra | 17:11 | |
| anteaya | dims: ^^ | 17:11 |
| *** apoorvad has joined #openstack-infra | 17:12 | |
| *** ashleighfarnham has joined #openstack-infra | 17:12 | |
| *** eharney has quit IRC | 17:12 | |
| rcarrillocruz | yay | 17:13 |
| rcarrillocruz | puppet-lint | 17:13 |
| rcarrillocruz | \o/ | 17:13 |
| yolanda | your friend :) | 17:13 |
| anteaya | I'm glad a linter made you happy | 17:13 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Update create_endpoint() https://review.openstack.org/289051 | 17:14 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Update create_endpoint() https://review.openstack.org/289051 | 17:15 |
| * Anticimex would appreciate if someone familiar with JJB took a look at https://review.openstack.org/#/c/289373/ | 17:15 | |
| *** ddieterly is now known as ddieterly[away] | 17:15 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-hound: Add ulimit max-open-files param to puppet-hound https://review.openstack.org/289451 | 17:15 |
| clarkb | Anticimex: aiui that behavior is intentional since jjb added a tox builder after we lardy had a tox macro and broke us | 17:17 |
| clarkb | Anticimex: at least the "let macro override an entrypoint" behavior is intentional | 17:18 |
| emmenemoi | @fungi : sorry, seems to work now. | 17:18 |
| *** sgordon has quit IRC | 17:18 | |
| emmenemoi | review submitted | 17:18 |
| *** ddieterly[away] is now known as ddieterly | 17:18 | |
| fungi | emmenemoi: great! | 17:18 |
| fungi | happy hacking! | 17:18 |
| clarkb | I have skimmed hte trove imae elements and they don't seem to be doing anything terrible | 17:21 |
| clarkb | guessing that the python + mysql dep tree is rather large :/ | 17:21 |
| fungi | clarkb: Anticimex: in fact, they broke us that way twice. we reverted the tox builder addition once, then a different contributor submitted a review to add it again a couple months later and the reviewers had already forgotten the earlier revert. so after the second revert they added that override behavior | 17:21 |
| pleia2 | good morning | 17:21 |
| anteaya | morning pleia2 | 17:22 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Ensure worker log dir is created https://review.openstack.org/289452 | 17:23 |
| *** ihrachys has quit IRC | 17:25 | |
| *** gyee has joined #openstack-infra | 17:25 | |
| *** eharney has joined #openstack-infra | 17:26 | |
| *** salv-orlando has joined #openstack-infra | 17:27 | |
| yolanda | infra-root, can you review https://review.openstack.org/285433 ? that's a fix for glean that already has a +2 from Monty | 17:27 |
| yolanda | to fix hostnames | 17:27 |
| *** harlowja_at_home has quit IRC | 17:28 | |
| emmenemoi | my proposition have 2 conflicts (I took too long to push). Is there any doc to manage conflicts in gerrit ? | 17:29 |
| anteaya | emmenemoi: do you see a button called rebase? | 17:30 |
| emmenemoi | sure | 17:30 |
| anteaya | or better yet what do you mean by conflicts | 17:30 |
| anteaya | can you share the url for the patch? | 17:30 |
| emmenemoi | https://review.openstack.org/#/c/289462/ | 17:30 |
| *** nwkarsten has joined #openstack-infra | 17:31 | |
| anteaya | emmenemoi: were do you see what you are calling conflicts? | 17:31 |
| anteaya | where* | 17:31 |
| emmenemoi | "conflicts with(2)" on the top right | 17:31 |
| anteaya | oh yeah there is no way you well ever not have something there | 17:32 |
| *** annegentle has joined #openstack-infra | 17:32 | |
| anteaya | emmenemoi: go to http://status.openstack.org/zuul/ | 17:32 |
| anteaya | emmenemoi: put 289462 in the text box filter | 17:32 |
| anteaya | and watch your patch as it runs the check tests | 17:33 |
| emmenemoi | ok | 17:33 |
| anteaya | once the tests have finished running, jenkins will comment with logs of the tests | 17:33 |
| zaro | morning | 17:33 |
| emmenemoi | thanks | 17:33 |
| Anticimex | clarkb, fungi, ok. my problem is described in the storyboard. i'm all ears on how proper syntax looks, because i spent some 3 hours trying different ways, but i consistently provoke the duplication of entry points which bails out the registry.py | 17:33 |
| anteaya | emmenemoi: read what jenkins posts, and if you need help understanding just ask | 17:34 |
| anteaya | emmenemoi: and you are welcome | 17:34 |
| anteaya | morning zaro | 17:34 |
| fungi | emmenemoi: yeah, that just means there are multiple proposed changes which touch similar lines to your change. if one of them merges before yours then you'll likely need to rebase your change at some point. if your change merges first then they'll need to rebase theirs eventually | 17:34 |
| fungi | but doesn't imply any action needed for now (other than as an indication that you might review those other changes, especially since one might already be trying to fix the same bug as yours) | 17:35 |
| emmenemoi | ok fungi. That's what I needed to ear: I could amend my code to maybe avoid some conflicts | 17:35 |
| Anticimex | clarkb, fungi: storyboard, i.e. https://storyboard.openstack.org/#!/story/2000503 | 17:35 |
| emmenemoi | thanks | 17:35 |
| *** sdake_ has joined #openstack-infra | 17:35 | |
| *** yamamoto has joined #openstack-infra | 17:37 | |
| *** tsg has joined #openstack-infra | 17:37 | |
| *** sdake has quit IRC | 17:38 | |
| *** annegentle has quit IRC | 17:38 | |
| *** akshai has quit IRC | 17:39 | |
| *** akshai has joined #openstack-infra | 17:40 | |
| *** salv-orlando has quit IRC | 17:40 | |
| *** armax has joined #openstack-infra | 17:40 | |
| *** salv-orlando has joined #openstack-infra | 17:41 | |
| *** yamamoto has quit IRC | 17:41 | |
| *** jaosorior has quit IRC | 17:41 | |
| *** jaosorior has joined #openstack-infra | 17:42 | |
| *** jecarey has quit IRC | 17:42 | |
| *** andreykurilin__ has quit IRC | 17:43 | |
| *** Swami has joined #openstack-infra | 17:43 | |
| fungi | Anticimex: thanks--i mostly defer to zaro, zxiiro and pelix on triaging issues like that | 17:44 |
| fungi | i've long since lost adequate context on some of the more recent changes around that part of the implementation | 17:45 |
| *** angdraug has joined #openstack-infra | 17:45 | |
| mtreinish | fungi, nibalizer: http://puppetboard.openstack.org/report/subunit-worker02.openstack.org/5f320323ae02edf33553edb7e866698f85a853a4 and http://puppetboard.openstack.org/report/subunit-worker01.openstack.org/8ab8ae6ed65621c62118e7bf2850b0d15adb68dd | 17:46 |
| *** jaosorior has quit IRC | 17:46 | |
| mtreinish | we should be good to try starting the workers again | 17:46 |
| * mtreinish crosses fingers | 17:47 | |
| fungi | Anticimex: did this regress recently, or is this your first time trying to use it? | 17:47 |
| anteaya | rcarrillocruz: thank you, hound appears to be hounding on my vm | 17:48 |
| rcarrillocruz | np | 17:48 |
| rcarrillocruz | otoh, if you want to also check hound on web interface | 17:49 |
| rcarrillocruz | you may need to do some amendments | 17:49 |
| nibalizer | mtreinish: ok | 17:49 |
| rcarrillocruz | you need to put proper /etc/hosts | 17:49 |
| rcarrillocruz | so the vhost gets created correctly by apache | 17:49 |
| nibalizer | okay i started it on 01 | 17:49 |
| rcarrillocruz | but it's a secondary thing, not needed for debugging this issue in particular | 17:50 |
| anteaya | I don't know what that means, proper /etc/hosts | 17:50 |
| nibalizer | and on 02 | 17:50 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-hound: Add ulimit max-open-files param to puppet-hound https://review.openstack.org/289451 | 17:50 |
| anteaya | but thanks | 17:50 |
| mtreinish | nibalizer: cool, it looks like it's working through the backlog | 17:51 |
| *** nwkarsten has quit IRC | 17:51 | |
| *** salv-orlando has quit IRC | 17:51 | |
| mtreinish | or at least 4 runs so far :) | 17:51 |
| *** salv-orlando has joined #openstack-infra | 17:51 | |
| *** eharney_ has joined #openstack-infra | 17:51 | |
| *** nwkarsten has joined #openstack-infra | 17:52 | |
| *** salv-orl_ has joined #openstack-infra | 17:52 | |
| anteaya | rcarrillocruz: did you encounter the too many open files error on your test vm? | 17:53 |
| mtreinish | nibalizer: also, it looks like subunit-worker02.o.o is missing from cacti. Did I need to add something besides the line in the hiera data? | 17:53 |
| rcarrillocruz | not yet | 17:53 |
| rcarrillocruz | i'm leaving it running for a while just in case | 17:53 |
| rcarrillocruz | pleia2: amended commit message, thx | 17:54 |
| anteaya | rcarrillocruz: I'm tailing /var/log/hound.log hoping it will show up there | 17:54 |
| pleia2 | rcarrillocruz: thank you :) | 17:54 |
| *** rossella_s has quit IRC | 17:55 | |
| *** scheuran has quit IRC | 17:55 | |
| *** eharney has quit IRC | 17:55 | |
| *** lucasagomes is now known as lucas-dinner | 17:56 | |
| *** salv-orlando has quit IRC | 17:56 | |
| *** piet has quit IRC | 17:56 | |
| *** nwkarsten has quit IRC | 17:56 | |
| *** fedexo has quit IRC | 17:57 | |
| *** amrith is now known as _amrith_ | 17:57 | |
| anteaya | rcarrillocruz: got an error: http://paste.openstack.org/show/489576/ | 17:58 |
| *** zz_dimtruck is now known as dimtruck | 17:58 | |
| *** _amrith_ is now known as amrith | 17:58 | |
| *** tphummel has joined #openstack-infra | 17:59 | |
| *** fawadkhaliq has quit IRC | 17:59 | |
| clarkb | I have to pop out nowish, I don't think zuul has been restarted yet but probably should be soon then we can clear out the extra bare-trusty machines | 17:59 |
| anteaya | this was after the indexes were all built and it tried to run the server: http://paste.openstack.org/show/489577/ | 17:59 |
| rcarrillocruz | oh nice | 17:59 |
| *** fawadkhaliq has joined #openstack-infra | 18:00 | |
| anteaya | clarkb: happy errands | 18:00 |
| *** eharney_ is now known as eharney | 18:00 | |
| anteaya | clarkb: yeah I'm still looking for a zuul restart candidate | 18:00 |
| *** cschwede has joined #openstack-infra | 18:00 | |
| zaro | fungi: was gerrit restarted recently? | 18:00 |
| openstackgerrit | Merged openstack-infra/storyboard: Archive cards instead of deleting them https://review.openstack.org/285392 | 18:00 |
| *** acruz has quit IRC | 18:01 | |
| pleia2 | zaro: at 7:20 UTC | 18:01 |
| rcarrillocruz | brb | 18:01 |
| *** angdraug has quit IRC | 18:01 | |
| *** yaume_ has quit IRC | 18:03 | |
| anteaya | then it fails to destroy the index for too many open files | 18:03 |
| *** yaume has joined #openstack-infra | 18:03 | |
| anteaya | http://paste.openstack.org/show/489578/ | 18:03 |
| anteaya | rcarrillocruz: I'll try to apply your patch and run again | 18:03 |
| anteaya | how do I tell a service running under puppet to stop? | 18:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add script to generate branching instructions https://review.openstack.org/289386 | 18:04 |
| Anticimex | fungi: first time i try it | 18:05 |
| *** hashar has quit IRC | 18:05 | |
| Anticimex | fungi: but well, it works in tag-release 1.4.0 (but this release lacks 'slack') | 18:05 |
| Anticimex | fungi: at any rate, it has regressed after 1.4.0, that much is clear | 18:06 |
| *** ayoung has quit IRC | 18:06 | |
| *** tricksters has quit IRC | 18:06 | |
| *** sfinucan has quit IRC | 18:06 | |
| *** elo has joined #openstack-infra | 18:06 | |
| *** doug-fish has quit IRC | 18:07 | |
| *** kiran-r has quit IRC | 18:08 | |
| *** sputnik13 has joined #openstack-infra | 18:08 | |
| anteaya | if I have made a change to a puppet-module can I just run puppet apply again without stopping the service first to pick up the change? | 18:08 |
| *** doug-fish has joined #openstack-infra | 18:08 | |
| zaro | pleia2: thanks | 18:08 |
| anteaya | I'll give folks a minute to say "don't do that" before I do that | 18:08 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 18:08 |
| anteaya | okay I'm going to run puppet apply again | 18:09 |
| anteaya | and hope it restarts hound with rcarrillocruz's downloaded patch in place | 18:09 |
| *** salv-orl_ has quit IRC | 18:10 | |
| *** salv-orlando has joined #openstack-infra | 18:10 | |
| fungi | Anticimex: i'm mostly asking since it's possible if it's an _extremely_ recent regression that it may also be impacting us, since we continuously deploy the master branch tip of jjb on our masters. i haven't had a chance to check our logs to find out if we're hitting it too, or perhaps if our configuration just isn't complex enough to expose the problem (we use very few jenkins plugins in our jobs) | 18:10 |
| *** sridhar_ram has joined #openstack-infra | 18:10 | |
| *** markvoelker has joined #openstack-infra | 18:10 | |
| *** _nadya_ has joined #openstack-infra | 18:11 | |
| *** dtantsur is now known as dtantsur|afk | 18:11 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Don't override the private_net settings for the tenant https://review.openstack.org/289495 | 18:11 |
| *** yamahata has quit IRC | 18:11 | |
| *** harlowja has joined #openstack-infra | 18:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix typo with $YUM variable https://review.openstack.org/289497 | 18:11 |
| Anticimex | fungi: right, quite possible i suppose | 18:12 |
| pabelanger | fungi: clarkb: that will fix 289101 which was merged overnight and broke centos7 DIBs | 18:12 |
| pabelanger | fungi: clarkb: see http://nodepool.openstack.org/dib.devstack-centos7.log | 18:13 |
| *** doug-fish has quit IRC | 18:13 | |
| *** markvoelker has quit IRC | 18:13 | |
| *** markvoelker has joined #openstack-infra | 18:13 | |
| Anticimex | fungi: i can do some basic triaging and see if i can narrow down when it broke | 18:13 |
| *** ihrachys has joined #openstack-infra | 18:14 | |
| Anticimex | fungi: jjb breaks on that sample code, there's not much of plugins in action? | 18:14 |
| *** salv-orlando has quit IRC | 18:14 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 18:15 |
| *** sdake_ has quit IRC | 18:17 | |
| AJaeger | fungi, pabelanger: I'm back online ... | 18:17 |
| pabelanger | AJaeger: WB | 18:17 |
| *** sarob has joined #openstack-infra | 18:17 | |
| AJaeger | fungi, https://review.openstack.org/289059 creates a tox-db ubuntu-trusty job and uses it - what do you think? | 18:17 |
| *** salv-orlando has joined #openstack-infra | 18:18 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Added possibility to exclude project combinations https://review.openstack.org/175233 | 18:19 |
| fungi | AJaeger: per that ml thread earlier today, fuel-web is apparently relying on postgres in their unit tests | 18:19 |
| *** angdraug has joined #openstack-infra | 18:19 | |
| *** salv-orl_ has joined #openstack-infra | 18:19 | |
| *** salv-orl_ has quit IRC | 18:19 | |
| *** salv-orlando has quit IRC | 18:19 | |
| fungi | though they're covered for now by the fact that we always do db setup in the unit test builders | 18:19 |
| AJaeger | fungi, just read it - thanks | 18:20 |
| *** salv-orlando has joined #openstack-infra | 18:20 | |
| fungi | so they go onto the pile with nova and ironic as projects i know of doing that | 18:20 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Don't override the private_net settings for the tenant https://review.openstack.org/289495 | 18:20 |
| *** dimtruck is now known as zz_dimtruck | 18:21 | |
| AJaeger | ok | 18:21 |
| AJaeger | fungi, 289059 is for *generic* tox environments | 18:21 |
| fungi | AJaeger: yep, noted. just wanted to bring up fuel-web while that was reminding me | 18:21 |
| AJaeger | ok | 18:22 |
| *** salv-orlando has quit IRC | 18:22 | |
| *** kevinsho_ has joined #openstack-infra | 18:22 | |
| Anticimex | fungi: commit 8bb2fa5b80b33c13671e1cccec9f26fe71639082 introduces the error | 18:22 |
| Anticimex | (haven't checked code yet) | 18:22 |
| *** salv-orlando has joined #openstack-infra | 18:22 | |
| *** annegentle has joined #openstack-infra | 18:22 | |
| AJaeger | also, here's a change to finalize neutron-constraints removal, could a project-config core +2, please? | 18:23 |
| Anticimex | if i read that right. i'll triple check a bit | 18:23 |
| *** Sukhdev has quit IRC | 18:24 | |
| fungi | AJaeger: so i guess we should let the mistral and oslo.db teams know these job names are about to change? | 18:24 |
| AJaeger | do you have a generic mail already written? If not, what about adding it there? | 18:25 |
| *** kevinsho_ has quit IRC | 18:25 | |
| Zara | storyboard 'quote comment' button in review, for the intersted: https://review.openstack.org/#/c/288944/ . there's also a 'permalink to comment' button: https://review.openstack.org/#/c/289078/ | 18:25 |
| Zara | *interested | 18:25 |
| fungi | AJaeger: i don't yet, because the thread i want to start first is about a plan to get rid of opportunistic db testing in our ci | 18:25 |
| *** ildikov has joined #openstack-infra | 18:26 | |
| anteaya | Zara: is there a screenshot of what quote comment would look like? | 18:26 |
| Anticimex | fungi: git diff 48e34564daa358ae60fa7ddb6f8dbe367c06f895 8bb2fa5b80b33c13671e1cccec9f26fe71639082 | 18:26 |
| fungi | AJaeger: since jobs continuing to pass and quietly skipping database usage when we try to cut over to new templates is dangerous | 18:26 |
| anteaya | Zara: and so far I have not had the time to get consensous on the items I said I would from the storyboard meetup | 18:27 |
| *** doug-fish has joined #openstack-infra | 18:27 | |
| anteaya | Zara: we are having a hard time having group conversations at present | 18:27 |
| fungi | AJaeger: and i also don't want to announce open season on projects adding other-requirements.txt files until we solve not needing to add database packages to them when they're just wanting to run unit tests and don't actually have any that interact with a database (which is most projects) | 18:27 |
| AJaeger | fungi, I see. I hope my other open changes do not break any of this - I tried to check for database usage. | 18:27 |
| AJaeger | fungi, agreed on that | 18:28 |
| Anticimex | those are two adjacent commithashes in 'git log', so basically the 8bb2fa introduces it | 18:28 |
| *** kevinsho_ has joined #openstack-infra | 18:28 | |
| AJaeger | fungi, so, what's the best way to move forward with ubuntu-trusty? Should I wait a bit before sending further changes? | 18:28 |
| fungi | AJaeger: also we need to solve/decide on the package caching in our image builds as to whether we find better coverage for testing per-project other-requirements.txt files on less common platforms or whether we make all platforms ignore failures to cache packages they can't find | 18:29 |
| Anticimex | now i haven't tried every single commit since 8bb2fa5b80b33c13671e1cccec9f26fe71639082 - can't rule out it's like someone said: "it broke, got fixed, then broke again" | 18:29 |
| AJaeger | fungi, not sure I can help much today on either of these. | 18:30 |
| Anticimex | i'm too much of a noob at git to see why the web diffs of github/cgit both show a single diff, but console diff between those two commits show massive amount of changes | 18:31 |
| fungi | AJaeger: right, just explaining reasons for not having an announcement yet, since the only thing to really announce is when projects can safely/sanely begin individually controlling their distro dependencies | 18:31 |
| *** ddieterly is now known as ddieterly[away] | 18:32 | |
| fungi | AJaeger: in the meantime, we probably just need to reach out to projects if we're doing something that's going to rename their jobs | 18:32 |
| *** sambetts is now known as sambetts|afk | 18:32 | |
| AJaeger | fungi, ok, I reach out to oslo.db and mistral... | 18:33 |
| fungi | AJaeger: or i can ping ptls/liaisons on irc and get them to +1 the proposed changes which rename jobs | 18:33 |
| *** kzaitsev_mb has joined #openstack-infra | 18:34 | |
| Zara | anteaya: screenshot in the comments here: https://storyboard.openstack.org/#!/story/2000504 | 18:34 |
| Zara | anteaya: that's okay :) | 18:34 |
| Zara | anteaya: thank you for keeping me updated | 18:34 |
| Anticimex | in jjb, git diff d1c9845..5b9f4fd will show some indexes that shows the diff of registry.py specifically. tho those are not commitids (?) | 18:34 |
| anteaya | Zara: I love what you are doing, keep doing that | 18:35 |
| AJaeger | fungi, I'm on it... | 18:35 |
| anteaya | Zara: sorry I am missing the section called comments | 18:36 |
| anteaya | I see four links on the page, one to an etherpad, three to gerrit reviews | 18:36 |
| AJaeger | dims: could you review https://review.openstack.org/#/c/289059 , please? It changes gate-oslo.db-tox-db-mysql-python's name and uses ubuntu-trusty as image. | 18:36 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:37 | |
| Zara | anteaya: the comments are currently part of the events timeline | 18:37 |
| Anticimex | ok, here's the commit that breaks my case: https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/commit/jenkins_jobs/registry.py?id=449bb24f8f40866eb57404b8e4c5e77b529a996b -- https://review.openstack.org/#/c/252335/ | 18:37 |
| *** yamamoto has joined #openstack-infra | 18:37 | |
| Zara | (we want to give users the option to visually separate them out in the future... but still rather up in the air) | 18:37 |
| anteaya | okay so I had to click the button for more comments | 18:37 |
| *** nwkarsten has joined #openstack-infra | 18:37 | |
| anteaya | Zara: thanks I found it | 18:37 |
| AJaeger | anteaya: do you have time to review https://review.openstack.org/#/c/288506/ , please? That removes constraints from neutron completely | 18:38 |
| anteaya | AJaeger: I will do my best, I still don't understand constraints | 18:38 |
| AJaeger | anteaya: that change removes the constraints specific handling, neutron is using the normal targets now - also for liberty. | 18:38 |
| *** kencjohnston has quit IRC | 18:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix typo with $YUM variable https://review.openstack.org/289497 | 18:39 |
| AJaeger | anteaya: It also removes all the constraint templates from zuul/layout, we don't need them anymore ;) | 18:39 |
| anteaya | Zara: the screenshot is great thank you! so glad storyboard can render images | 18:39 |
| *** ihrachys has quit IRC | 18:39 | |
| fungi | Anticimex: we're definitely not hitting it since that merged months ago and jjb master is still working for our configs | 18:39 |
| anteaya | AJaeger: oh | 18:39 |
| anteaya | AJaeger: well that should be easy | 18:39 |
| *** zz_dimtruck is now known as dimtruck | 18:40 | |
| Zara | anteaya: thank you! I really want it to be able to resize them automatically (I have to do it manually at the moment, bit fiddly.) but one thing at a time! :) | 18:40 |
| Anticimex | fungi: nod. makes me wonder how to unbreak the related code | 18:41 |
| anteaya | you are doing well at knocking down your one things at a time | 18:41 |
| *** [HeOS] has joined #openstack-infra | 18:41 | |
| *** jistr has quit IRC | 18:41 | |
| anteaya | AJaeger: thank you | 18:42 |
| AJaeger | thanks, anteaya. Once that's in we can cleanup jenkins and remove constraint jobs ;) | 18:43 |
| *** yamamoto has quit IRC | 18:43 | |
| Zara | thanks :) | 18:43 |
| fungi | Anticimex: hopefully zaro, zxiiro or pelix have some insight into the breakage you're encountering, and whether or not https://review.openstack.org/252335 needs reverting | 18:43 |
| *** kencjohnston has joined #openstack-infra | 18:44 | |
| Anticimex | i don't think revert is appropriate given that it passed all tests cases, but figuring out what happened seems clever | 18:44 |
| Anticimex | i still have not been able to define a macro with post that merge. i guess the test cases are more fit for the introduced code, so to speak | 18:45 |
| *** sgordon has joined #openstack-infra | 18:46 | |
| anteaya | AJaeger: yay | 18:48 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Remove gate-ironic-dsvm-functional from experimental queue for devstack https://review.openstack.org/289515 | 18:49 |
| mriedem | mtreinish: ^ | 18:49 |
| *** davideagnello has joined #openstack-infra | 18:49 | |
| anteaya | AJaeger: so constraints aren't a thing anymore? | 18:49 |
| anteaya | did they fall out of fashion like certain shades of green? | 18:49 |
| mtreinish | mriedem: +1 | 18:49 |
| *** ildikov has quit IRC | 18:49 | |
| AJaeger | anteaya: we changed jobs so that the "normal" jobs are constraints enabled - and projects can enable them in their setup one by one without any change to project-config | 18:50 |
| zxiiro | Anticimex: are you sure it used to work? I always thought macros and defaults didn't work in regular "job" sections and you had to use "job-template" if you needed those features. | 18:50 |
| *** ildikov has joined #openstack-infra | 18:50 | |
| fungi | oh, i missed that this was a job and not a job-template | 18:50 |
| zxiiro | Anticimex: I'm likely wrong though. In my projects we don't use "job" configuration. we always use "job-template" for everything | 18:50 |
| anteaya | AJaeger: ah okay | 18:50 |
| AJaeger | anteaya: so, instead of gate-neutron-XXX-constraints, we have gate-neutron-XXX and that sets up constraints by default. | 18:50 |
| anteaya | AJaeger: thank you | 18:50 |
| *** pvaneck has joined #openstack-infra | 18:51 | |
| *** dizquierdo has quit IRC | 18:51 | |
| fungi | zxiiro: we definitely use macros in some of our jobs which aren't instantiated from job-templates, but we may not rely on defaults there | 18:51 |
| AJaeger | anteaya: nova, neutron, cinder, glance have changed their tox.ini files for this already. Other projects can follow at their pace... | 18:51 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Create new partitioning-sfdisk element. https://review.openstack.org/259881 | 18:51 |
| Anticimex | zxiiro: i'm not sure it used to work. let me check the cases i went through | 18:51 |
| anteaya | AJaeger: right so it is opt in at the project level based on how they configure their tox.ini | 18:51 |
| *** markvoelker has quit IRC | 18:51 | |
| AJaeger | anteaya: only think missing: post and release jobs cannot use constraints since zuul-cloner does not support these queues yet. jesusaur is on it... | 18:51 |
| openstackgerrit | Merged openstack-infra/project-config: Use non-constraint jobs for all neutron branches https://review.openstack.org/288506 | 18:51 |
| AJaeger | anteaya: correct | 18:51 |
| anteaya | AJaeger: awesome work | 18:51 |
| anteaya | AJaeger: cool, thanks for the update | 18:52 |
| AJaeger | anteaya: sdague did the hard work for nova and I just cleaned up ;) | 18:52 |
| anteaya | I appreciate it | 18:52 |
| anteaya | thanks sdague for the hard work | 18:52 |
| fungi | anteaya: AJaeger: the best part about constraints via tox.ini (now that it's cleared the pti with the tc) is that the switch is entirely self-testing | 18:52 |
| anteaya | thanks AJaeger for cleaning up (as you do so often and so well) | 18:52 |
| anteaya | fungi: yay, self-testing! | 18:52 |
| AJaeger | fungi: self-service and self-testing ;) | 18:53 |
| fungi | indeed. less work for us | 18:53 |
| yolanda | SpamapS, i addressed the dib element, renaming that | 18:53 |
| zxiiro | Anticimex: I suspect if you switched your definition to a job-template it will work. As for if regular "job" sections should allow you to use macros/defaults I don't know since I never used that. I tried defaults in job sections in the past and they never get parsed so I assumed you had to be explicit in regular job sections. | 18:53 |
| yolanda | also i'll need to do the corresponding change on lvm element, to depend properly on new namign | 18:54 |
| *** salv-orlando has quit IRC | 18:54 | |
| *** salv-orlando has joined #openstack-infra | 18:54 | |
| yolanda | SpamapS, but i wanted your thoughts on lvm element. I am finding that allowing this element to just build a single volume is not fitting my needs, and i need to construct multiple volumes for the different partitions | 18:54 |
| yolanda | i was thinking in passing a set comma separated values in some multi-line parameter, to allow to create the volumes with the settings we need, to be more flexible | 18:55 |
| fungi | zxiiro: Anticimex: yes, i believe we always use a job-template if we expect it or any of its macros to require parameter expansion | 18:55 |
| Anticimex | zxiiro: after https://review.openstack.org/252335 i get this error: jenkins_jobs.errors.JenkinsJobsException: Duplicate entry point found for component type: 'property', 'property',name: 'my-test-property-macro' | 18:55 |
| Anticimex | as reported in the bug | 18:55 |
| Anticimex | before that, i get: | 18:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove constraints jobs https://review.openstack.org/288994 | 18:55 |
| AJaeger | anteaya: this is the cleanup ^ | 18:55 |
| *** salv-orl_ has joined #openstack-infra | 18:56 | |
| *** _nadya_ has quit IRC | 18:56 | |
| *** ildikov has quit IRC | 18:56 | |
| Anticimex | zxiiro: just kidding -- yes, before https://review.openstack.org/252335 it actually works (if you add the macro) | 18:57 |
| *** ddieterly[away] is now known as ddieterly | 18:57 | |
| *** salv-orl_ has quit IRC | 18:57 | |
| *** asselin has quit IRC | 18:57 | |
| *** salv-orlando has quit IRC | 18:57 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:57 | |
| crinkle | yolanda: mordred what do you think about merging 267817 and 267863 since the clouds are off and will be redeployed anyways? | 18:57 |
| zxiiro | Anticimex: sounds like a regression then. I'd talk to electrofelix since he wrote the patch. | 18:58 |
| *** salv-orlando has joined #openstack-infra | 18:58 | |
| yolanda | about upgrading to liberty? fine for me | 18:58 |
| *** ddieterly is now known as ddieterly[away] | 18:58 | |
| Anticimex | zxiiro: i double checked the documentation, which does say it's ok to put macros on Jobs | 18:58 |
| Anticimex | http://docs.openstack.org/infra/jenkins-job-builder/definition.html#macro | 18:58 |
| Anticimex | zxiiro: cool, thanks for quick response | 18:58 |
| zxiiro | Anticimex: great, thanks for checking. In the meantime I guess your workaround is to convert it to a job-template. | 18:59 |
| mordred | crinkle: ++ | 18:59 |
| Anticimex | yeah, i can move those pieces to job-template if that works | 18:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for docs jobs https://review.openstack.org/289034 | 18:59 |
| Anticimex | zxiiro: thanks | 18:59 |
| anteaya | AJaeger: +2, thank you | 19:00 |
| AJaeger | thanks for review, anteaya ! | 19:00 |
| anteaya | thanks for doing the work | 19:00 |
| AJaeger | ;) | 19:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Initial commit for RPM afs mirror https://review.openstack.org/289520 | 19:00 |
| crinkle | yolanda: mordred want to +a for me then? | 19:00 |
| anteaya | rcarrillocruz: with your patch applied to my hound test instance I don't see any more "too many open files" errors in the log files as I tail them | 19:01 |
| mordred | crinkle: done | 19:01 |
| crinkle | mordred: ty | 19:01 |
| anteaya | rcarrillocruz: so thank you | 19:01 |
| *** pradk_ has joined #openstack-infra | 19:01 | |
| Anticimex | zxiiro: seems macros-on-job-template get same error | 19:02 |
| *** rockyg has joined #openstack-infra | 19:02 | |
| mordred | clarkb: I have confirmed shade is updated on nodepool - but I have not restarted anything | 19:02 |
| Anticimex | zxiiro: my initial workaround (though incorrect) was https://review.openstack.org/#/c/289373/ , i'll keep that for some days :] | 19:02 |
| *** dimtruck is now known as zz_dimtruck | 19:03 | |
| *** doug-fis_ has joined #openstack-infra | 19:03 | |
| *** pradk has quit IRC | 19:03 | |
| *** doug-fi__ has joined #openstack-infra | 19:04 | |
| *** doug-f___ has joined #openstack-infra | 19:05 | |
| *** flepied1 has joined #openstack-infra | 19:06 | |
| *** rvasilets__ has quit IRC | 19:06 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add a quote button to comments https://review.openstack.org/288944 | 19:06 |
| *** doug-fish has quit IRC | 19:06 | |
| *** doug-fish has joined #openstack-infra | 19:06 | |
| *** kzaitsev_mb has quit IRC | 19:07 | |
| *** doug-fis_ has quit IRC | 19:08 | |
| *** doug-fi__ has quit IRC | 19:09 | |
| *** flepied has quit IRC | 19:10 | |
| *** doug-f___ has quit IRC | 19:10 | |
| *** pradk has joined #openstack-infra | 19:10 | |
| *** pradk_ has quit IRC | 19:11 | |
| *** ccrouch1 has joined #openstack-infra | 19:12 | |
| zxiiro | Anticimex: what's the minimal reproducible configuration? I just tried this and it worked http://pastebin.com/pfnaCQQh | 19:12 |
| *** ccrouch has quit IRC | 19:12 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Upgrade from kilo to liberty https://review.openstack.org/267817 | 19:13 |
| rcarrillocruz | anteaya: ++ | 19:14 |
| *** amrith is now known as _amrith_ | 19:15 | |
| *** tphummel has quit IRC | 19:15 | |
| openstackgerrit | Merged openstack-infra/system-config: Upgrade infra-cloud to liberty https://review.openstack.org/267863 | 19:15 |
| *** tphummel has joined #openstack-infra | 19:16 | |
| *** gordc has quit IRC | 19:16 | |
| *** esker has joined #openstack-infra | 19:16 | |
| *** sbalukoff has joined #openstack-infra | 19:17 | |
| *** shardy_ has joined #openstack-infra | 19:18 | |
| *** shardy has quit IRC | 19:18 | |
| *** yamahata has joined #openstack-infra | 19:18 | |
| *** _amrith_ is now known as amrith | 19:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert -tarball jobs to use ubuntu-trusty https://review.openstack.org/289035 | 19:21 |
| Anticimex | zxiiro: http://pastebin.com/pEvi3gTk works at 48e34564daa358ae60fa7ddb6f8dbe367c06f895 of jjb, but not from 8bb2fa5b80b33c13671e1cccec9f26fe71639082 | 19:21 |
| *** Sukhdev has joined #openstack-infra | 19:23 | |
| anteaya | rcarrillocruz: what were you saying about what I should do to /etc/hosts so I can browse to my test instance? | 19:23 |
| *** ociuhandu has quit IRC | 19:24 | |
| anteaya | rcarrillocruz: I wasn't able to figure it out | 19:24 |
| *** vgridnev has joined #openstack-infra | 19:24 | |
| rcarrillocruz | babysitting my son, if you'll be online in a couple hours i can walk thru it | 19:24 |
| rcarrillocruz | by that time, my wife should be around | 19:24 |
| anteaya | rcarrillocruz: thanks, enjoy time with your son | 19:24 |
| anteaya | yup, good call on priorities | 19:25 |
| anteaya | still looking for an infra-root candidate able to restart zuul: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 19:28 |
| fungi | anteaya: i was considering waiting for today's load to die off, since we still seem to be keeping up even with a glut of bare-trusty workers we're not making efficient use of | 19:29 |
| *** angdraug has quit IRC | 19:29 | |
| anteaya | fungi: fair enough | 19:29 |
| anteaya | I'll stop the barkering | 19:30 |
| jeblair | same here :) | 19:30 |
| anteaya | okey dokey | 19:30 |
| anteaya | thanks | 19:30 |
| fungi | a zuul restart means we lose jobs running or queued for any of the pipelines besides check and gate | 19:30 |
| anteaya | ah | 19:30 |
| anteaya | I thought re reenqued them all | 19:30 |
| fungi | which punts post and check-tripleo pretty hard in particular | 19:30 |
| zxiiro | Anticimex: I copy pasted your code and reinstalled JJB from latest master, still works for me: http://pastebin.com/aBry1rgT | 19:30 |
| jeblair | it looks like we only barely touched peak capacity for maybe 1-2 hours today (and likely would have been much less without the extra nodes) | 19:30 |
| *** sdake has joined #openstack-infra | 19:31 | |
| anteaya | yeah | 19:31 |
| jeblair | a far cry from last week | 19:31 |
| anteaya | true | 19:31 |
| anteaya | okay if you are keeping an eye on it, that is sufficent for my needs | 19:31 |
| AJaeger | yeah, looks really idle compared to last week at this time ;) | 19:31 |
| anteaya | thanks | 19:31 |
| fungi | jeblair: yep, between ff being officially on and osic and vexxhost going into production i think we're clear of the worst of it | 19:31 |
| *** jordanP has quit IRC | 19:32 | |
| Anticimex | zxiiro: hrmpf, strange. pm | 19:32 |
| *** akshai has quit IRC | 19:32 | |
| AJaeger | zuul's last reconfigured is a few hours old - is that a fallout from anteaya's observation? | 19:32 |
| anteaya | is osic in production now? | 19:32 |
| fungi | anteaya: as of the weekend, yep. 100 nodes capacity for now i think? | 19:33 |
| AJaeger | anteaya: yes! Check grafana | 19:33 |
| anteaya | oh great | 19:33 |
| fungi | AJaeger: did we recently merge layout changes which you would expect to see reflected in a zuul reconfigure in the past few hours? | 19:33 |
| anteaya | low on the error node launch attempts, well done osic | 19:34 |
| fungi | anteaya: also their workers are roughly on par with rackspace's performance-wise, even at half the vcpu count | 19:34 |
| *** sridhar_ram1 has joined #openstack-infra | 19:35 | |
| *** ociuhandu has joined #openstack-infra | 19:35 | |
| fungi | though current theory is this is because we're basically the only tenant on a mostly empty cloud of 1k host machines at the moment | 19:35 |
| anteaya | impressive | 19:35 |
| jeblair | fungi: do they oversubscribe? | 19:35 |
| fungi | jeblair: who knows? | 19:35 |
| AJaeger | fungi: https://review.openstack.org/#/c/288506/ - 44 mins ago | 19:35 |
| Anticimex | ok.. | 19:35 |
| jeblair | fungi: maybe intel just knows how to spec a machine | 19:35 |
| fungi | right now we have enough quota to put one vm on one out of ten of their hosts | 19:35 |
| Anticimex | fungi, and others, that bug report can be rejected.. i had messed up my venv with a pip install of 1.4.0 from pypi mixed with jjb from master | 19:36 |
| fungi | and the only other two tenants are very likely not even exercising the cloud yet | 19:36 |
| jeblair | fungi: iirc they have 2 other tenants right now? i don't know the others are doing anything tho | 19:36 |
| fungi | jeblair: right | 19:36 |
| Anticimex | installed in an obvious broken way (python setup.py install) | 19:36 |
| fungi | jeblair: ibm and mirantis, both of whom were probably delayed by the same dns/cert issue we were | 19:36 |
| openstackgerrit | Merged openstack-infra/project-config: Remove gate-ironic-dsvm-functional from experimental queue for devstack https://review.openstack.org/289515 | 19:37 |
| fungi | and are probably not as agile as we are in terms of immediately blasting our workload onto it | 19:37 |
| *** baoli has quit IRC | 19:37 | |
| *** kencjohnston has quit IRC | 19:37 | |
| *** akshai has joined #openstack-infra | 19:37 | |
| jeblair | we are agile(TM)! | 19:37 |
| anteaya | ha ha ha | 19:37 |
| anteaya | clarkb is agile? | 19:37 |
| *** sridhar_ram has quit IRC | 19:37 | |
| * fungi meant in the traditional english sense of the word, not the corportate-approved buzzwordy sense | 19:38 | |
| *** doug-fish has quit IRC | 19:38 | |
| fungi | corportate! i made a new word | 19:38 |
| AJaeger | ;) | 19:38 |
| anteaya | quick, trademark | 19:38 |
| pabelanger | fungi: AJaeger: Are we ready to reduce bare-trusty to 1 min-ready? | 19:38 |
| AJaeger | pabelanger: let me count job usage... | 19:38 |
| fungi | pabelanger: probably not that low yet, i think there are still quite a few jobs which run on it even if they're not as many as the ubuntu-trusty jobs now | 19:39 |
| Anticimex | zxiiro: thanks for taking the time... :s | 19:39 |
| fungi | pabelanger: but AJaeger's count will get us perspective on how far we've gotten | 19:39 |
| AJaeger | pabelanger: 2773 jobs still use bare-trusty | 19:39 |
| *** rfolco has quit IRC | 19:39 | |
| *** mrmartin has joined #openstack-infra | 19:39 | |
| pabelanger | AJaeger: that's surprising | 19:39 |
| fungi | granted, those may also be much less frequently-run jobs, or shorter-running jobs | 19:39 |
| jeblair | or not at all run jobs? | 19:40 |
| AJaeger | pabelanger: https://review.openstack.org/#/c/289035/ is another 900 jobs | 19:40 |
| fungi | this is also jobs defined in jjb, many of which may not even appear in zuul's layout | 19:40 |
| zxiiro | Anticimex: mind updating the docs to suggest installing via "python setup.py sdist && pip install dist/jenkins-job-builder-.....tar.gz" rather than installing via setup.py install? I think the current docs recommendation causes this issue you've run into. | 19:40 |
| AJaeger | https://review.openstack.org/#/c/289034/ reduce by 800 | 19:40 |
| *** yamamoto has joined #openstack-infra | 19:40 | |
| AJaeger | jeblair: yes, includes not run jobs. | 19:40 |
| AJaeger | these are all jobs setup in JJB | 19:40 |
| Anticimex | zxiiro: i'll take a look | 19:40 |
| AJaeger | so, reviews for 288994, 289035, 289034 are welcome - to brings us down to 1000 configured jobs | 19:41 |
| clarkb | zxiiro: Anticimex you don't even need to sdist, just pip install . | 19:42 |
| anteaya | how do we calculate how many not at all run jobs we might have? | 19:42 |
| zxiiro | clarkb: interesting... i never thought of that | 19:43 |
| AJaeger | anteaya: I did: tox -e zuul;cd .test/jenkins-job-builder/.test/new/out |grep bare-trusty * |wc -l | 19:43 |
| Anticimex | clarkb: that was even smoother | 19:43 |
| AJaeger | anteaya: or what do you mean with "not at all run" ? | 19:43 |
| jeblair | anteaya: len(set(jobs created by jjb) - set(job names that appear in zuul layout.yaml)) | 19:43 |
| anteaya | ah thank you | 19:44 |
| clarkb | did zuul get restarted? I may be able to do that real quick now if it needs doing | 19:44 |
| anteaya | AJaeger: does your command do the same thing as what jeblair just said? | 19:44 |
| *** yamamoto has quit IRC | 19:44 | |
| anteaya | AJaeger: it looks like it does | 19:45 |
| *** ddieterly[away] is now known as ddieterly | 19:45 | |
| jeblair | clarkb: no, fungi or i were going to just wait since we're well under capacity | 19:45 |
| clarkb | jeblair: ah rgr | 19:45 |
| AJaeger | anteaya: no, mine only checks "jobs created by jjb" | 19:45 |
| fungi | i need to step away for a bit and get in a walk while the weather's nice. will be back in ~ an hour | 19:45 |
| AJaeger | jeblair: any reason not to +W 288994? The constraints removal... I can +W myself | 19:45 |
| jeblair | AJaeger: i was still poking around the chain to make sure i understood all related patches | 19:45 |
| AJaeger | fungi, enjoy the walk! | 19:46 |
| AJaeger | jeblair: 288994 is not realted logically with the other - I just wanted to avoid merge conflicts | 19:46 |
| anteaya | fungi: enjoy your walk | 19:46 |
| jeblair | AJaeger: ok | 19:46 |
| *** salv-orlando has quit IRC | 19:47 | |
| *** rfolco has joined #openstack-infra | 19:47 | |
| AJaeger | thanks for review, jeblair | 19:47 |
| jeblair | np | 19:48 |
| AJaeger | complete set of open changes for ubuntu-trusty: https://review.openstack.org/#/q/status:open+topic:bindep | 19:48 |
| jeblair | those are easy | 19:48 |
| *** piet has joined #openstack-infra | 19:48 | |
| *** vgridnev has quit IRC | 19:48 | |
| *** annegentle has quit IRC | 19:48 | |
| *** rfolco has quit IRC | 19:48 | |
| AJaeger | clarkb: 289034 and 289035 will need manual JJB runs I fear... | 19:49 |
| clarkb | AJaeger: ya probably, I am around for the next hour and a half or so then afk again | 19:49 |
| clarkb | I can kick them off and babysit this afternoon | 19:49 |
| AJaeger | thanks, clarkb ! | 19:49 |
| *** rfolco has joined #openstack-infra | 19:50 | |
| *** rockyg has quit IRC | 19:50 | |
| jeblair | should we just try to push through all the ubuntu-trusty jobs then manually jjb? | 19:50 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Bug fix: Do not fail on routers with no ext gw https://review.openstack.org/289538 | 19:50 |
| *** _nadya_ has joined #openstack-infra | 19:51 | |
| *** annegentle has joined #openstack-infra | 19:51 | |
| pabelanger | I have another 2.5 hours to help out | 19:51 |
| *** jtomasek_ has joined #openstack-infra | 19:51 | |
| AJaeger | one hour for me... | 19:51 |
| clarkb | I just approved the two | 19:51 |
| AJaeger | but I reviewed them already - or wrote them ;) | 19:51 |
| *** doug-fish has joined #openstack-infra | 19:53 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Bug fix: Do not fail on routers with no ext gw https://review.openstack.org/289538 | 19:53 |
| clarkb | are there more changes than the ones that have been approved? | 19:53 |
| openstackgerrit | Martin Millnert proposed openstack-infra/jenkins-job-builder: Change recommended src-install method https://review.openstack.org/289539 | 19:53 |
| jeblair | clarkb: topic:bindep | 19:54 |
| AJaeger | clarkb: complete set of open changes for ubuntu-trusty: https://review.openstack.org/#/q/status:open+topic:bindep | 19:54 |
| *** kzaitsev_mb has joined #openstack-infra | 19:54 | |
| clarkb | oh there are chnages for devstack-trusty now | 19:54 |
| openstackgerrit | Merged openstack-infra/project-config: Remove constraints jobs https://review.openstack.org/288994 | 19:54 |
| jeblair | AJaeger: what's the plan for "nodb"? it seems backwards to specify that we *don't* want a db... | 19:54 |
| clarkb | are we ready for that switch? have we rerun the experimental jobs against ubuntu-trusty recently? | 19:54 |
| AJaeger | training-labs only - that one was wrong | 19:55 |
| *** ihrachys has joined #openstack-infra | 19:55 | |
| AJaeger | jeblair: We use gate-X-tox-ENV and I changed to gate-X-tox-nodb-ENV | 19:55 |
| anteaya | AJaeger: what is the status of the issue that fungi felt was the source of the error that fuel-web was experiencing? http://lists.openstack.org/pipermail/openstack-dev/2016-March/088542.html | 19:55 |
| AJaeger | And to gate-X-tox-db-ENV | 19:55 |
| *** ccrouch1 has quit IRC | 19:55 | |
| *** sarob has quit IRC | 19:55 | |
| AJaeger | And once all tox jobs are renamed, we can rename gate-X-tox-nodb-ENV to gate-X-tox-ENV again... | 19:56 |
| AJaeger | But for now I didn't want to mass migrate all of them at once | 19:56 |
| AJaeger | other ideas welcome... | 19:56 |
| jeblair | AJaeger: by not doing it all at once, won't that mean even more manual jjb runs? | 19:56 |
| clarkb | AJaeger: pabelanger https://review.openstack.org/#/c/289404/1 replaces parameterized nodes with ubuntu-trusty, I don't think we want to do that? | 19:57 |
| *** zeih has joined #openstack-infra | 19:57 | |
| AJaeger | jeblair: those are setup manually, no extra jobs for that | 19:57 |
| AJaeger | See https://review.openstack.org/#/c/289059/ the just renames the two jobs, no new ones created | 19:57 |
| *** _nadya_ has quit IRC | 19:58 | |
| clarkb | AJaeger: pabelanger same with https://review.openstack.org/#/c/289036/1 | 19:58 |
| *** hashar has joined #openstack-infra | 19:58 | |
| *** ccrouch has joined #openstack-infra | 19:58 | |
| AJaeger | clarkb: mmh, I was under the impression that everybody uses node: bare-trusty in projects.yaml, so that this has no effect | 19:59 |
| *** kzaitsev_mb has quit IRC | 19:59 | |
| jeblair | AJaeger: what about other projects that do database tests (eg, nova unit tests?) | 20:00 |
| clarkb | AJaeger: that may be, but we should just change the projects.yaml in that case | 20:00 |
| *** apoorvad has quit IRC | 20:01 | |
| AJaeger | clarkb: that's the same fungi did in Id65b7e106d62be92467c18bcb93c9d5da716242f | 20:01 |
| openstackgerrit | Merged openstack-infra/project-config: Use ubuntu-trusty for docs jobs https://review.openstack.org/289034 | 20:01 |
| *** apoorvad has joined #openstack-infra | 20:01 | |
| clarkb | hrm not sure I like that | 20:01 |
| clarkb | but maybe this is necessary to transition? | 20:01 |
| *** mrmartin has quit IRC | 20:01 | |
| AJaeger | clarkb: then -1 and let's discuss with fungi... | 20:02 |
| jeblair | we're just going to have to put that back when we go to xenial | 20:02 |
| jeblair | (we are doomed to repeat our past if we forget it :) | 20:02 |
| clarkb | AJaeger: yup comments left | 20:02 |
| AJaeger | jeblair: those repos need database setup - that's why I'm doing small steps. | 20:02 |
| jeblair | AJaeger: so does nova? | 20:03 |
| nibalizer | mtreinish: we are probably missing subunit02 from cacti because cacti doesn't run puppet right now because it is slow | 20:03 |
| jeblair | nibalizer: did we not merge your cron? | 20:03 |
| AJaeger | nova does in py27 tests if I remember correctly | 20:03 |
| *** rockyg has joined #openstack-infra | 20:04 | |
| jeblair | AJaeger: ah, you're saying this is just for non-standard jobs... | 20:04 |
| AJaeger | jeblair: fungi converted the standard jobs already - see backscroll as well. | 20:05 |
| *** doug-fish has quit IRC | 20:05 | |
| AJaeger | the py27 jobs uses databases but he wants to go back to the drawing board that we do this correctly | 20:05 |
| jeblair | okay. that will not be a simple proposition. | 20:05 |
| *** _nadya_ has joined #openstack-infra | 20:05 | |
| *** doug-fish has joined #openstack-infra | 20:06 | |
| AJaeger | see http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2016-03-07T18:25:57 | 20:06 |
| AJaeger | and since somebody asked about fuel-web: They have fixed their setup now... | 20:06 |
| jeblair | AJaeger: yeah, i was there for that, but that's fungi wanting to start a discussion; it doesn't give me much more context to what's in his mind. :) | 20:07 |
| AJaeger | but I fear we will run in a few more of these isssues | 20:07 |
| openstackgerrit | Merged openstack-infra/project-config: Convert -tarball jobs to use ubuntu-trusty https://review.openstack.org/289035 | 20:07 |
| mtreinish | nibalizer: ah, ok | 20:07 |
| *** |-paul-| has joined #openstack-infra | 20:07 | |
| AJaeger | jeblair: maybe he thinks about this during his walk... | 20:07 |
| *** rguillebert has quit IRC | 20:07 | |
| jeblair | AJaeger: but back to the tox-db/nodb -- do enough jobs need db setup like this to warrant a special job template? | 20:07 |
| AJaeger | zuul is still not reconfigured - over an hour now | 20:08 |
| nibalizer | https://review.openstack.org/#/c/284466/ looks like i fail at puppet | 20:08 |
| AJaeger | jeblair: I haven't checked all of them | 20:08 |
| *** simondodsley has quit IRC | 20:09 | |
| AJaeger | just started for the tox-db with the first two to see that everything works fine | 20:09 |
| AJaeger | And I've seen some more, so there should be at least a dozen (from memory) | 20:09 |
| *** rockyg has quit IRC | 20:09 | |
| jeblair | AJaeger: okay, let me see if i understand the plan: current job definition (tox-env) has db setup. you will change each of those to either 'tox-db-env' or 'tox-nodb-env' as you work through them all. this lets you work piecemeal and we can tell which ones have been evaluated or not. then at the end, tox-nodb-env gets renamed to tox-env. | 20:10 |
| jeblair | AJaeger: is that right? | 20:10 |
| AJaeger | looking at names we have 6 with mysql in it and 6 with postgresql in it | 20:10 |
| *** doug-fis_ has joined #openstack-infra | 20:10 | |
| AJaeger | jeblair: yes | 20:10 |
| openstackgerrit | Major Hayden proposed openstack/requirements: Raise python-openstackclient cap to <1.7.0 https://review.openstack.org/289546 | 20:10 |
| jeblair | AJaeger: okay, thanks for catching me up. sounds good to me. :) | 20:10 |
| *** doug-fish has quit IRC | 20:11 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Fix apache failure https://review.openstack.org/288763 | 20:11 |
| AJaeger | jeblair: let me update https://etherpad.openstack.org/p/bindep with that information... | 20:12 |
| *** david-lyle_ has joined #openstack-infra | 20:12 | |
| jeblair | okay, i'm +2 on all the bindep changes that clarkb is not -1 on. | 20:12 |
| openstackgerrit | Brad Behle proposed openstack-infra/project-config: Create new networking-ovn-native-l3-nv check job https://review.openstack.org/289547 | 20:12 |
| *** david_lyle has quit IRC | 20:13 | |
| jeblair | i did not +3 anything; will let clarkb work through if he wants, otherwise i see no reason not to +3 them now to get them into the batch jjb runs | 20:13 |
| clarkb | jeblair: I say go for the +3 | 20:13 |
| clarkb | except are you talking about devstack-trusty too? | 20:13 |
| clarkb | becuase I don't know if we have retested those on the current images | 20:13 |
| *** abregman has joined #openstack-infra | 20:14 | |
| *** yaume has quit IRC | 20:14 | |
| jeblair | clarkb: the only one i saw was https://review.openstack.org/288597 which AJaeger mentioned was correcting a mistake, but i will double check | 20:14 |
| AJaeger | clarkb: the training-labs is indeed fixing a mistake, the two jobs are the same, one for post job as publishing, the other for gate. They should be same node | 20:15 |
| openstackgerrit | Brad Behle proposed openstack-infra/project-config: Create new networking-ovn-native-l3-nv check job https://review.openstack.org/289547 | 20:15 |
| jeblair | clarkb: i double checked; ^ is correct | 20:15 |
| AJaeger | clarkb: or did you find anything else? | 20:15 |
| clarkb | https://review.openstack.org/#/c/288877/ for example | 20:15 |
| clarkb | but is that just the unittests? | 20:16 |
| clarkb | I was going off of titles | 20:16 |
| clarkb | and yup looks like just unittests | 20:16 |
| jeblair | yep | 20:16 |
| jeblair | (more like "convert bare-trusty-using jobs in the devstack* projects to ubuntu-trusty") | 20:17 |
| AJaeger | clarkb: yes, that converts only explicit bare-trusty to ubuntu-trusty | 20:17 |
| *** abregman is now known as abregman|nb | 20:17 | |
| AJaeger | clarkb: if it changes devstack-trusty, please -2 ;) | 20:17 |
| clarkb | yup those are fine, go for it | 20:17 |
| jeblair | https://review.openstack.org/288597 and https://review.openstack.org/288694 could use a +3 | 20:17 |
| jeblair | the others i +3d | 20:18 |
| *** harlowja has quit IRC | 20:18 | |
| clarkb | looking | 20:18 |
| *** ddieterly has quit IRC | 20:20 | |
| clarkb | jeblair: AJaeger should https://review.openstack.org/#/c/288694/1 remove the job templates and the group too? | 20:20 |
| AJaeger | clarkb, jeblair: I was wondering about that as well - we could remove the complete file... | 20:21 |
| *** sridhar_ram1 is now known as sridhar_ram | 20:22 | |
| jeblair | i didn't know if anyone wanted to keep it around for a future experiment. but that's what git is for, right? so i'm marginally in favor of removing it. | 20:22 |
| * AJaeger will remove it then... | 20:23 | |
| AJaeger | fungi, pabelanger, clarkb, jeblair updated https://etherpad.openstack.org/p/bindep with information about current status - feel free to add to it. | 20:23 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 20:24 |
| *** zeih has quit IRC | 20:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove experimental-bindep-jobs from repos https://review.openstack.org/288694 | 20:25 |
| AJaeger | jeblair, clarkb ^ | 20:25 |
| AJaeger | jeblair: since https://review.openstack.org/#/c/275975/ merged, there has been no reconfigure of zuul - is there a problem with zuul? https://review.openstack.org/#/c/288506/ merged two hours ago but still no reconfigure | 20:26 |
| *** david-lyle_ is now known as david-lyle | 20:26 | |
| jeblair | AJaeger: i can look when i get back from lunch | 20:27 |
| AJaeger | jeblair: enjoy lunch - thanks | 20:28 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add element to force loading of the 3ware module https://review.openstack.org/289552 | 20:28 |
| notmyname | last week we did a python-swiftclient release, but the (docs?) CI was down at the time and the dev docs weren't built/uploaded. can someone (infra-root) requeue that job so that the docs are updated, please? | 20:29 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add element to force loading of the 3ware module https://review.openstack.org/289552 | 20:29 |
| *** rguillebert has joined #openstack-infra | 20:29 | |
| *** [1]Thelo has joined #openstack-infra | 20:30 | |
| AJaeger | correction to notmyname: fix was broken publishing, fixed today with https://review.openstack.org/#/c/288644/ - this should have affected all client projects published since 11th of February ;( | 20:31 |
| notmyname | thanks AJaeger | 20:31 |
| *** anthonyper has quit IRC | 20:32 | |
| *** Thelo has quit IRC | 20:32 | |
| *** [1]Thelo is now known as Thelo | 20:32 | |
| *** anthonyper has joined #openstack-infra | 20:33 | |
| *** claudiub has joined #openstack-infra | 20:37 | |
| *** vgridnev has joined #openstack-infra | 20:37 | |
| *** Sukhdev has quit IRC | 20:38 | |
| *** ZZelle_ has joined #openstack-infra | 20:38 | |
| *** asettle has joined #openstack-infra | 20:40 | |
| *** claudiub has quit IRC | 20:40 | |
| *** claudiub has joined #openstack-infra | 20:40 | |
| *** yamamoto has joined #openstack-infra | 20:41 | |
| *** esikachev has joined #openstack-infra | 20:42 | |
| *** jtomasek_ has quit IRC | 20:43 | |
| *** asettle has quit IRC | 20:44 | |
| *** esikachev has quit IRC | 20:46 | |
| *** yamamoto has quit IRC | 20:46 | |
| clarkb | looks like some of the project-config changes are failing on the fallback-fedora-23 job | 20:47 |
| openstackgerrit | Merged openstack-infra/project-config: ubuntu-trusty: misc,openstack-planet,... https://review.openstack.org/288874 | 20:47 |
| *** baoli has joined #openstack-infra | 20:47 | |
| clarkb | AJaeger: notmyname so the job that needs to be rerun is the docs publishing job? | 20:47 |
| AJaeger | clarkb: yes | 20:47 |
| *** tphummel has quit IRC | 20:47 | |
| AJaeger | clarkb: they fail in ovh-bhs1 like https://jenkins05.openstack.org/job/gate-bindep-fallback-fedora-23/11/console - the image is old there | 20:48 |
| clarkb | AJaeger: ah, ya I think bhs has been having troubles | 20:48 |
| *** baoli has quit IRC | 20:48 | |
| AJaeger | could you remove the fedora-23 image from that cloud? we earlier removed the ubuntu-trusty image there as well... | 20:48 |
| clarkb | AJaeger: ya I can do that | 20:48 |
| *** baoli has joined #openstack-infra | 20:48 | |
| AJaeger | and this is rax-ord - same problem https://jenkins04.openstack.org/job/gate-bindep-fallback-fedora-23/8/console | 20:48 |
| AJaeger | pabelanger: ^ | 20:49 |
| *** tphummel has joined #openstack-infra | 20:49 | |
| AJaeger | clarkb: the image might be old as well, could check rax-ord, please? | 20:49 |
| AJaeger | "/usr/bindep-env/bin/bindep: No such file or directory" indicates old image | 20:49 |
| pabelanger | AJaeger: Ya, looks like stale images | 20:49 |
| clarkb | notmyname: https://git.openstack.org/cgit/openstack/python-swiftclient/tag/?h=3.0.0 is the tag? | 20:50 |
| pabelanger | AJaeger: I'm also working on some optimizations not to run bindep-fallback on every project-config commit | 20:50 |
| notmyname | clarkb: yes. 3.0.0 | 20:50 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Only unset proxy for deploy command https://review.openstack.org/289556 | 20:50 |
| AJaeger | pabelanger: right now I'd like to see them run so that we catch problems early ;) | 20:51 |
| clarkb | AJaeger: those fedora-23 images are deleted in bhs1 | 20:51 |
| AJaeger | thanks, clarkb | 20:51 |
| clarkb | now to retrigger python-swiftclient-docs on 3.0.0 | 20:51 |
| anteaya | AJaeger: have you a moment to look at this paste: http://paste.openstack.org/show/489589/ | 20:51 |
| anteaya | AJaeger: I don't think I understood your equation properly | 20:51 |
| AJaeger | anteaya: the "grep bare-trusty * |wc -l" filters out only the jobs with bare-trusty in it. | 20:52 |
| clarkb | I am just going to trigger the job so that we don't make new artifacts (which is what will happen if I enqueue the ref to the release pipeline) | 20:52 |
| AJaeger | anteaya: 8985 is the total number of jobs | 20:52 |
| *** esikachev has joined #openstack-infra | 20:53 | |
| AJaeger | no, 8961 is the total number of jobs, see first line | 20:53 |
| AJaeger | you do not have a number for jobs used in zuul | 20:53 |
| openstackgerrit | Dolph Mathews proposed openstack-infra/project-config: Add #osic to accessbot https://review.openstack.org/289557 | 20:53 |
| *** salv-orlando has joined #openstack-infra | 20:54 | |
| anteaya | how do I find the number of jobs used in zuul? | 20:54 |
| *** mrmartin has joined #openstack-infra | 20:55 | |
| anteaya | what does number of jobs generated mean then? | 20:55 |
| *** yamahata has quit IRC | 20:55 | |
| anteaya | this isn't important | 20:55 |
| AJaeger | that's the number generated by JJB | 20:55 |
| anteaya | but it does seem to be coming up lately | 20:55 |
| AJaeger | all jobs configured in any jenkins - so jobs, that could be triggered | 20:55 |
| anteaya | 8961 is the number of jobs generated by JJB? | 20:55 |
| AJaeger | yes | 20:55 |
| *** yamahata has joined #openstack-infra | 20:55 | |
| AJaeger | anteaya: run tox -e zuul and grep for jobs in the output again - similar like you grepped for non-voting last week | 20:56 |
| clarkb | AJaeger: notmyname ok assuming I got the command right that is now queued/running | 20:56 |
| *** hichihara has joined #openstack-infra | 20:56 | |
| anteaya | yeah taht was over 12000 jobs | 20:56 |
| notmyname | clarkb: thanks | 20:56 |
| anteaya | as it sorts by pipeline | 20:56 |
| AJaeger | anteaya: we want unique jobs for this calculation | 20:57 |
| AJaeger | anteaya: not worth investigating IMHO, just curiosity... | 20:57 |
| anteaya | okay I'll stop then | 20:58 |
| clarkb | AJaeger: notmyname and done http://logs.openstack.org/0f/0fea74f7edc9b21bdd4046cbcae1d8e0a7662a6a/release/python-swiftclient-docs/manual-run-1/console.html | 20:58 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 20:58 | |
| anteaya | I am trying to understand this and am not having much luck at all | 20:58 |
| clarkb | you all should probably check the docs look correct now | 20:58 |
| AJaeger | thanks, clarkb ! | 20:59 |
| *** stevemar has quit IRC | 20:59 | |
| anteaya | I think the various defintions of jobs are not being conveyed to my understanding | 20:59 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 20:59 |
| notmyname | clarkb: AJaeger: http://docs.openstack.org/developer/python-swiftclient/ doesn't seem to have updated | 20:59 |
| anteaya | as for unique jobs I would argue that 8961 is the number of unique jobs | 20:59 |
| AJaeger | anteaya: correct. and zuul includes duplicates | 21:00 |
| AJaeger | notmyname: let me double check... | 21:00 |
| clarkb | AJaeger: notmyname http://logs.openstack.org/0f/0fea74f7edc9b21bdd4046cbcae1d8e0a7662a6a/release/python-swiftclient-docs/manual-run-1/console.html#_2016-03-07_20_56_25_754 it moved the / content to /3.0.0 | 21:00 |
| AJaeger | clarkb: you need docs-tags-only job instead of docs | 21:01 |
| *** stevemar has joined #openstack-infra | 21:01 | |
| clarkb | ah ok, I asked earlier | 21:01 |
| *** amrith is now known as _amrith_ | 21:01 | |
| clarkb | AJaeger: can you tell me what the exact job name is? | 21:02 |
| AJaeger | clarkb: sorry, missed that ;( | 21:02 |
| AJaeger | python-swiftclient-docs-tags-only | 21:02 |
| *** kencjohnston has joined #openstack-infra | 21:02 | |
| clarkb | ok that is queued and should be done shortly | 21:03 |
| *** annegentle has quit IRC | 21:03 | |
| *** esker has quit IRC | 21:04 | |
| *** esker has joined #openstack-infra | 21:04 | |
| AJaeger | anteaya: a job can run in several pipelines and for different repositories, so you need to get the unique jobs out of zuul. Just extract all jobs, run through sort and uniq | 21:04 |
| clarkb | gear.NoConnectedServersError: No connected Gearman servers | 21:04 |
| clarkb | thats weird considering this just worked with that other job | 21:04 |
| clarkb | jeblair: ^ any ideas on why `sudo /opt/zuul/tools/trigger-job.py --url=http://review.openstack.org/p --job python-swiftclient-docs-tags-only --project openstack/python-swiftclient --pipeline release --newrev 0fea74f7edc9b21bdd4046cbcae1d8e0a7662a6a --refname refs/tags/3.0.0 --logpath 0f/0fea74f7edc9b21bdd4046cbcae1d8e0a7662a6a/release/python-swiftclient-docs-tags-only/manual-run-1` would fail? | 21:06 |
| *** yaume has joined #openstack-infra | 21:06 | |
| anteaya | AJaeger: okay | 21:06 |
| *** dims has quit IRC | 21:07 | |
| * AJaeger waves good bye! | 21:07 | |
| *** sdake has quit IRC | 21:07 | |
| anteaya | AJaeger: thank you, good night | 21:07 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 21:07 |
| clarkb | the script does do a wait for server against gearman before submitting the job | 21:08 |
| *** asettle has joined #openstack-infra | 21:08 | |
| clarkb | must be timing out before gearman is able to respond to the submit? | 21:09 |
| clarkb | which might mean the jjb updates are running | 21:09 |
| *** baoli has quit IRC | 21:09 | |
| *** sdague has quit IRC | 21:09 | |
| *** rfolco has quit IRC | 21:09 | |
| *** baoli has joined #openstack-infra | 21:09 | |
| *** e0ne has joined #openstack-infra | 21:10 | |
| *** Sukhdev has joined #openstack-infra | 21:10 | |
| *** sarob has joined #openstack-infra | 21:10 | |
| *** baoli has quit IRC | 21:10 | |
| *** baoli has joined #openstack-infra | 21:11 | |
| *** esker has quit IRC | 21:11 | |
| ianw | system-config people !clarkb : can anyone take a look at this workaround for broken pip packages for fedora -> https://review.openstack.org/#/c/285876/ | 21:14 |
| *** sarob has quit IRC | 21:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Convert devstac* to ubuntu-trusty https://review.openstack.org/288877 | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Convert elastic-recheck to ubuntu-trusty https://review.openstack.org/288878 | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config: Convert grenade to use ubuntu-trusty https://review.openstack.org/288879 | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config: Convert job-builder to use ubuntu-trusty https://review.openstack.org/288881 | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config: Convert first DB using tox jobs to ubuntu-trusty https://review.openstack.org/289059 | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config: Training-labs: Use ubuntu-trusty https://review.openstack.org/288597 | 21:16 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:16 | |
| *** angdraug has joined #openstack-infra | 21:17 | |
| *** jcoufal has quit IRC | 21:17 | |
| *** vgridnev has quit IRC | 21:19 | |
| *** ayoung has joined #openstack-infra | 21:20 | |
| *** mrmartin has quit IRC | 21:20 | |
| clarkb | ok I have to pop out again, I can run jjb on the masters when back later this afternoon if no one beats me to it | 21:20 |
| clarkb | remmeber don't do more than 1 or 2 masters at a time | 21:20 |
| ianw | clarkb: yeah, i believe that image is built via dib. maybe zerofree will help? :) i'll take a look, but probably ~500mb is about right for a thick-ish cloud image with mysql and everything that requires | 21:21 |
| clarkb | ianw: ya | 21:21 |
| anteaya | AJaeger: 8575 unique jobs according to zuul: http://paste.openstack.org/show/489601/ | 21:21 |
| ianw | clarkb: i added some notes on it too, from what i found https://review.openstack.org/289092 | 21:22 |
| *** sarob has joined #openstack-infra | 21:25 | |
| *** shardy_ has quit IRC | 21:27 | |
| *** weshay has quit IRC | 21:28 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 21:29 |
| *** kevinsho_ has quit IRC | 21:30 | |
| *** _nadya_ has quit IRC | 21:30 | |
| *** baoli has quit IRC | 21:32 | |
| *** zz_dimtruck is now known as dimtruck | 21:32 | |
| *** baoli has joined #openstack-infra | 21:32 | |
| *** dimtruck is now known as zz_dimtruck | 21:33 | |
| *** baoli has quit IRC | 21:33 | |
| *** baoli has joined #openstack-infra | 21:33 | |
| *** e0ne has quit IRC | 21:34 | |
| hashar | ianw: hello! Thank you for you reviews on nodepool (and other projects). I am trying to catch up :D | 21:34 |
| *** kzaitsev_mb has joined #openstack-infra | 21:35 | |
| *** annegentle has joined #openstack-infra | 21:35 | |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Remove test skip for Cinder Ceph import backup https://review.openstack.org/289578 | 21:35 |
| *** derekh has quit IRC | 21:36 | |
| anteaya | is there anything a non-root can do to help delete floating ips in bluebox? http://grafana.openstack.org/dashboard/db/nodepool-blue-box | 21:36 |
| anteaya | we don't seem to be able to get any breathing room there | 21:36 |
| *** hongbin has joined #openstack-infra | 21:37 | |
| fungi | okay, i'm back and catching up. AJaeger: i'm good with the proposed plan for the db vs nodb versions of those generic "tox" template jobs. unit tests are going to be a much tougher nut to crack | 21:38 |
| SpamapS | yolanda: I'm out on leave a few more days, so my presence is sporadic. ;) | 21:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Run cacti's node generation from cron https://review.openstack.org/284466 | 21:40 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests https://review.openstack.org/288840 | 21:40 |
| hongbin | Hi, I want to know how to setup a proposal bot to sync up the requirements.txt in magnum-ui. Is here the right channel to get help? | 21:40 |
| anteaya | you add your project to the requirements subscription file | 21:41 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:41 | |
| anteaya | hongbin: add your project to this file: http://git.openstack.org/cgit/openstack/requirements/tree/projects.txt | 21:42 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:42 | |
| hongbin | anteaya: get that. Thanks! | 21:42 |
| anteaya | then proposal bot will offer patches for any changes that affect your project | 21:42 |
| nibalizer | hongbin: the magic is in ./jenkins/scripts/propose_update.sh in project-config | 21:42 |
| fungi | jeblair: AJaeger: thinking back, the stale zuul reconfiguration could possibly be related to the change i approved today for adding the new mutexes in its layout. the comments said zuul had been restarted since the necessary support merged, but i haven't checked back in on it yet. will start looking at debug logs now | 21:43 |
| *** yamamoto has joined #openstack-infra | 21:43 | |
| *** e0ne has joined #openstack-infra | 21:43 | |
| *** kzaitsev_mb has quit IRC | 21:43 | |
| jeblair | fungi: i just started looking at that; zuul is running since feb13, the change merged on feb5 | 21:44 |
| fungi | well, that's good at least | 21:45 |
| jeblair | MultipleInvalid: extra keys not allowed @ data['jobs'][388]['mutex'] | 21:45 |
| jeblair | and yet... | 21:45 |
| fungi | gah | 21:45 |
| *** |-paul-| has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Use ubuntu-trusty for checklang https://review.openstack.org/288990 | 21:46 |
| *** yamamoto has quit IRC | 21:47 | |
| jeblair | fungi: the git repo is current, but the installed zuul still bombs on that (even in config test mode) | 21:48 |
| jeblair | fungi: i suspect that means it hasn't installed in a while | 21:48 |
| jeblair | anteaya: this brings our supposition about whether we're running the memory leak fix into question | 21:48 |
| fungi | yeah, was just looking at that. pbr freeze claims zuul==2.1.1.dev111 # git sha 2055dcd | 21:48 |
| jeblair | Zuul version: 2.1.1.dev111 | 21:48 |
| jeblair | which agrees with the status page ^ | 21:48 |
| jeblair | git describe says 2.1.0-132-g0a6a0c4 | 21:49 |
| fungi | 16 commits since that sha in the local repo on the server | 21:49 |
| openstackgerrit | Hongbin Lu proposed openstack/requirements: Add magnum-ui to projects.txt https://review.openstack.org/289584 | 21:49 |
| *** esikachev has quit IRC | 21:49 | |
| fungi | so basically we're running zuul from december | 21:49 |
| fungi | now onto the next question... | 21:49 |
| jeblair | anteaya: indeed, that means we are *not* running the fix | 21:50 |
| *** baoli has quit IRC | 21:50 | |
| jeblair | fungi: shall i "pip install /opt/zuul" and see what happens? | 21:50 |
| *** kzaitsev_mb has joined #openstack-infra | 21:50 | |
| fungi | jeblair: as a workaround, i'm on board with that | 21:51 |
| jeblair | or even pip install -U | 21:51 |
| jeblair | error in setup command: Error parsing /tmp/pip-1cK5R4-build/setup.cfg: ImportError: No module named testrepository | 21:51 |
| fungi | geez | 21:51 |
| jeblair | File "/usr/local/lib/python2.7/dist-packages/pbr/testr_command.py", line 47, in <module> | 21:51 |
| jeblair | from there | 21:51 |
| anteaya | jeblair: oh my | 21:51 |
| jeblair | is there some pbr weirdness? | 21:51 |
| jeblair | mordred: ^ this mean anything to you? | 21:52 |
| openstackgerrit | Merged openstack-infra/project-config: Convert some tox jobs to ubuntu-trusty https://review.openstack.org/289040 | 21:52 |
| anteaya | jeblair: I believe mmedvede said he was running zuul dev123 and still had memory leak issues | 21:52 |
| anteaya | let me re-read the logs | 21:52 |
| fungi | jeblair: mordred: a little searching turned up a fix in pbr https://review.openstack.org/#/c/124652/5/pbr/packaging.py | 21:52 |
| *** harlowja has joined #openstack-infra | 21:52 | |
| fungi | jeblair: so perhaps we continuously deployed a broken pbr? | 21:53 |
| anteaya | 15:35:33 <mmedvede> yes. For the record, zuul ui is reporting 'Zuul version: 2.1.1.dev123' | 21:53 |
| anteaya | http://eavesdrop.openstack.org/meetings/third_party/2016/third_party.2016-03-07-15.00.log.html | 21:53 |
| jeblair | fungi: yeah, it's uncapped in zuul, so it probably upgraded at some point | 21:54 |
| jeblair | fungi: it's possible that everything we deploy with pbr is similarly wedged | 21:54 |
| jeblair | this is :( | 21:54 |
| mmedvede | yes, I am running zuul with the alleged fix applied, and it did crash at least once because it VM ran out of ram. Did not send anything to mailing list because I do not have much evidence, other than zuul-server process consuming 95% of ram after about 3-4 days | 21:55 |
| anteaya | mmedvede: well sending what you have can't hurt | 21:56 |
| anteaya | mmedvede: it appears you are running newer zuul than infra | 21:56 |
| anteaya | jhesketh: hello, might you be about? | 21:57 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Add NFS devstack plugin https://review.openstack.org/289589 | 21:57 |
| *** dims has joined #openstack-infra | 21:58 | |
| fungi | jeblair: so i'm going to guess that separately pip installing pbr and then trying again will work around this, since i don't know whether pip install -U is taking setup_requires into account | 21:58 |
| openstackgerrit | Merged openstack-infra/infra-ansible: Ignore <playbook>.retry files on git https://review.openstack.org/280195 | 21:59 |
| krotscheck | mordred: Did you have time to look at that keystone discovery problem I ran into in shade? | 21:59 |
| fungi | jeblair: as it may be trying to use broken pbr before it gets around to upgrading pbr in the context of the pip install -U zuul | 21:59 |
| jeblair | fungi: likely so. i'd really like to raise this to the attention of lifeless and mordred, as PBR wedging itself is decidedly uncool. | 21:59 |
| fungi | jeblair: 100% agreement on that | 22:00 |
| jeblair | like, i'm wondering if we should pick a version of pbr and start globally pinning it in infra | 22:00 |
| jeblair | ok, zuul is upgraded | 22:00 |
| jhesketh | Morning | 22:00 |
| anteaya | yay! | 22:00 |
| jhesketh | anteaya: I am now | 22:00 |
| anteaya | jhesketh: morning | 22:00 |
| anteaya | how nice to see you | 22:00 |
| anteaya | after you have had some coffee | 22:00 |
| anteaya | and settled in | 22:01 |
| jeblair | fungi: i reckon we should go ahead and restart it. that will solve several problems. | 22:01 |
| fungi | jeblair: was that hacky workaround what managed it, or did you have to do even worse? | 22:01 |
| *** dims has quit IRC | 22:01 | |
| jeblair | fungi: just reinstalled pbr | 22:01 |
| fungi | k | 22:01 |
| jhesketh | anteaya: coffee is in hand! | 22:01 |
| jhesketh | :-) | 22:01 |
| anteaya | it appears that mmedvede is running your fix zuul memory leak patch, yet still has memory leak issues | 22:01 |
| fungi | jeblair: yes, i'm more inclined to think an immediate restart is in order since we currently can't update configuration | 22:01 |
| jeblair | fungi: i'll perform the restart now then. | 22:01 |
| fungi | thanks jeblair! | 22:01 |
| anteaya | we discussed it today in the third party meeting: http://eavesdrop.openstack.org/meetings/third_party/2016/third_party.2016-03-07-15.00.log.html | 22:01 |
| *** david-lyle has quit IRC | 22:02 | |
| anteaya | jhesketh: 15:23:15 <mmedvede> I have one topic - wondered if anyone else has the same problem - zuul memory leak | 22:02 |
| *** angdraug_ has joined #openstack-infra | 22:02 | |
| jhesketh | anteaya: yes, I saw the highlights this morning... I have some changes that may help, but they are waiting on some feedback | 22:02 |
| *** david-lyle has joined #openstack-infra | 22:02 | |
| anteaya | jhesketh: do you have a list of changes? | 22:02 |
| jeblair | jhesketh: i looked but did not see them. | 22:02 |
| jhesketh | just pulling that up | 22:02 |
| anteaya | or perhaps an etherpad to organize them | 22:02 |
| anteaya | thank you | 22:02 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add missing PROJECT definition https://review.openstack.org/289590 | 22:02 |
| jhesketh | jeblair, anteaya: https://review.openstack.org/#/c/276395/4 | 22:03 |
| jhesketh | jeblair: this needs some more discussion: https://review.openstack.org/#/c/281166/2 | 22:03 |
| anteaya | mmedvede: ^^ do join us | 22:03 |
| fungi | clarkb: on the topic of switching parameterized nodes in jobs to non-parameterized, that was for the sake of transitioning. just switching the preferred node types in projects.yaml would have been intractable on a per-job basis (the file would have likely grown by a factor of 10x or more) | 22:03 |
| jhesketh | mmedvede: are you around? | 22:03 |
| jhesketh | anteaya: with the 3rd party, I'm wondering how often they reload their config.. because the cache doesn't get cleared unless it is reloaded | 22:04 |
| anteaya | jhesketh: that is a good question | 22:04 |
| anteaya | one which I did not ask | 22:04 |
| jhesketh | so it's possible doing some kind of cache management woudl help, but it doesn't answer why the memory usage is so much higher than it used to be | 22:04 |
| anteaya | I too await the answer | 22:04 |
| anteaya | yes, memory usage is higher than before | 22:05 |
| anteaya | before being last november | 22:05 |
| *** angdraug has quit IRC | 22:05 | |
| jhesketh | I think the memory usage also grew recently due to the CRD project checking (or something like that), not just the connection stuff (or in other words it's difficult to see which one the actual problem is from) | 22:05 |
| *** rhallisey has quit IRC | 22:06 | |
| anteaya | CRD? | 22:06 |
| fungi | cross-repo dependencies | 22:06 |
| anteaya | ah | 22:06 |
| fungi | i.e., "Depends-On:" | 22:06 |
| anteaya | yup | 22:07 |
| *** yamahata has quit IRC | 22:07 | |
| *** tpsilva has quit IRC | 22:07 | |
| *** gnuoy_ has joined #openstack-infra | 22:07 | |
| * anteaya puts away her "I can't acronym" shirt | 22:07 | |
| jeblair | that of course landed much longer ago than november. i think you're thinking of the non-existent project change | 22:07 |
| jhesketh | jeblair: yes, that one, sorry | 22:07 |
| mmedvede | jhesketh: yes, around | 22:07 |
| anteaya | mmedvede: awesome thank you | 22:07 |
| mmedvede | was working on email to zuul thread | 22:07 |
| jhesketh | but it's certainly compounded by connections, (or is hte only cause) | 22:08 |
| jeblair | jhesketh: i didn't realize you updated https://review.openstack.org/276395 :) | 22:08 |
| fungi | which dealt with some of the fallout of the crd implementation for deployments which wanted to only test a subset of projects | 22:08 |
| jhesketh | mmedvede: cool. Firstly, very sorry for the troubles you're having :-( | 22:08 |
| jhesketh | mmedvede: do you know how often you reload? | 22:08 |
| mmedvede | jhesketh: no problem, memory leaks are kind of expected :) | 22:08 |
| fungi | i do wish memory leaks weren't expected | 22:09 |
| mmedvede | jhesketh: do you mean how often I restart the service, or reload the config? | 22:09 |
| jhesketh | mmedvede: reload the config | 22:09 |
| *** dprince has quit IRC | 22:09 | |
| mmedvede | fungi: unfortunately humans are bad at writing leak-free code. So I would expect it as long as humans do it :) | 22:10 |
| jeblair | 2016-03-07 22:10:17,393 DEBUG zuul.Scheduler: Project None not found | 22:10 |
| jeblair | that is disturbing | 22:10 |
| mmedvede | jhesketh: not often, our configuration is rather static | 22:10 |
| fungi | uhh, yeah. project None? | 22:10 |
| jhesketh | mmedvede: okay, that's what I suspected... | 22:10 |
| *** e0ne has quit IRC | 22:11 | |
| mmedvede | I actually had a graph from some time before. let me see if I can post it to imgr | 22:11 |
| fungi | jeblair: is that in reference to a gerrit event with an empty project field (or no project field)? | 22:11 |
| jhesketh | mmedvede: even before the connection stuff the cache was never cleared unless there was a reload... however it also didn't fill up as quickly | 22:11 |
| jeblair | fungi: i have no idea | 22:11 |
| jeblair | which is making my extremely frustrated | 22:12 |
| *** gnuoy_ has quit IRC | 22:12 | |
| jeblair | fungi: would you do me a favor and stop the jjb runs clarkb started? | 22:12 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests https://review.openstack.org/288840 | 22:12 |
| fungi | oh wow, there were a ton of those entries | 22:12 |
| mmedvede | jhesketh: I was aware that reload should clear it up, so I remember checking ram use before/after config change, and it did not change much. Maybe I was using incorrect method to test it | 22:12 |
| fungi | jeblair: yep, killing them asap. just a moment | 22:13 |
| jeblair | fungi: oh, it's directly related to 2016-03-07 22:10:17,380 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent ref-replicated None> | 22:13 |
| jeblair | did we trade a moderately annoying log bug for a severe log bug? | 22:13 |
| jhesketh | mmedvede: so those fixes in review should help the reloading (ie actually free up some memory - I hope), but I suspect it'll still use too much memory | 22:13 |
| jeblair | jhesketh: too much memory is very different than memory leak :) | 22:14 |
| jhesketh | that's true | 22:14 |
| *** amitgandhinz has quit IRC | 22:14 | |
| *** thorst has quit IRC | 22:14 | |
| fungi | jeblair: checked all jenkins masters and killed any running updates i could find | 22:14 |
| mmedvede | yes, I am fine to bump VM ram, given ram use would level out at some point | 22:15 |
| *** kgiusti has left #openstack-infra | 22:15 | |
| *** stevelle has joined #openstack-infra | 22:15 | |
| jeblair | fungi: oh, sorry, i just meant suspend... oh well :) | 22:16 |
| fungi | jeblair: so yes, the ref-replicated events are definitely a nuisance. have you pondered a first-level filter on the events to drop any on the floor which match (or maybe don't match) a particular name regex before enqueuing them for deeper processing? | 22:16 |
| fungi | jeblair: suspend isn't too necessary since they will resume mostly from where they let off | 22:17 |
| fungi | because of maintaining a cache | 22:17 |
| jeblair | fungi: don't they write their cache only at the end? | 22:17 |
| *** thorst has joined #openstack-infra | 22:17 | |
| *** salv-orlando has quit IRC | 22:17 | |
| *** salv-orlando has joined #openstack-infra | 22:18 | |
| fungi | oh, i thought it was written at termination even if they hit errors, but maybe the exception handler for that follows a significantly different code path from the signal handler | 22:18 |
| fungi | i've at least seen jjb runs which encounter errors and abort on their own resume later | 22:19 |
| jeblair | ok | 22:19 |
| *** aysyd has quit IRC | 22:19 | |
| *** kevinsho_ has joined #openstack-infra | 22:19 | |
| fungi | anyway, what's done is done. i'll ctrl-z them next time | 22:19 |
| jeblair | have we changed the gearman plugin recently? | 22:19 |
| fungi | looking | 22:19 |
| *** stevelle has left #openstack-infra | 22:20 | |
| *** doug-fis_ has quit IRC | 22:20 | |
| fungi | i don't recall updating it but i'll verify versions installed | 22:20 |
| jeblair | because indeed the jjb runs were making zuul pathologically slow, and this seems to be a moderately new behavior | 22:20 |
| *** doug-fish has joined #openstack-infra | 22:20 | |
| fungi | i had thought maybe it was due to memory pressure since zuul hadn't been restarted in a while, but yeah if you saw it again right after a restart :/ | 22:21 |
| fungi | jenkins07 has 0.1.1.6.0314ab1 installed | 22:22 |
| *** thorst has quit IRC | 22:22 | |
| fungi | that's pretty old | 22:22 |
| jeblair | hrm | 22:22 |
| fungi | current release is 0.1.3 | 22:22 |
| fungi | 0.1.2 was 7 months ago | 22:22 |
| *** psanchez has quit IRC | 22:23 | |
| craige | o/ | 22:23 |
| *** doug-fish has quit IRC | 22:23 | |
| jeblair | gate-grenade-dsvm-multinode is not registered | 22:23 |
| fungi | that 0314ab1 commit was from may, a couple commits before 0.1.2 was tagged | 22:23 |
| *** doug-fish has joined #openstack-infra | 22:24 | |
| craige | If anyone feels like hitting it up, the latest iteration of the Maniphest migration is here: http://os01.mcwhirter.io/ - fixes some bugs, perhaps not other's as I've not checked them all off yet. | 22:24 |
| craige | mtreinish ^^ | 22:24 |
| anteaya | jhesketh: can you use a common topic with your memory-leak patches | 22:24 |
| jeblair | fungi: what jenkins masters were running jjb? | 22:24 |
| anteaya | jhesketh: perhaps something with memory or leak in it? | 22:24 |
| fungi | jeblair: 01 and 02 were what i found, and 02 looked like it finished just as i got ready to kill it | 22:24 |
| fungi | i'll check the plugin versions on those | 22:24 |
| *** slagle has quit IRC | 22:24 | |
| craige | I'll be at the bu smash in about an hour if you're still good to go today jhesketh. | 22:24 |
| *** stevebaker has quit IRC | 22:25 | |
| *** doug-fish has quit IRC | 22:26 | |
| fungi | jeblair: 01 and 02 are also running that same version of the plugin | 22:26 |
| fungi | so all consistent, it seems | 22:26 |
| *** doug-fish has joined #openstack-infra | 22:26 | |
| jeblair | maybe the change is due to the ubuntu-trusty switch | 22:26 |
| mtreinish | craige: http://i.imgur.com/5l3OQ8h.png | 22:26 |
| jeblair | 8k functions registered.... | 22:26 |
| jhesketh | anteaya: yep, will do | 22:27 |
| craige | oooh, thanks mtreinish. At what point did that come up? | 22:27 |
| jhesketh | craige: yep, happy to help | 22:27 |
| anteaya | jhesketh: thank you | 22:27 |
| fungi | jeblair: that many even after the restart? | 22:28 |
| mtreinish | craige: I think I hit manage on the side bar after joining the project | 22:28 |
| * craige tries it. | 22:28 | |
| mtreinish | nope workboard, not manage | 22:28 |
| mtreinish | I knew I hit something | 22:28 |
| *** hongbin has left #openstack-infra | 22:28 | |
| mtreinish | I was just clicking things | 22:28 |
| * anteaya pictures mtreinish mashing the keyboard | 22:28 | |
| fungi | jeblair: i suppose we can hunt down pending cleanup patches for experimental bindep jobs and all the old opt-in constraints jobs and try to whittle that total back down | 22:29 |
| craige | Damn, can't replicate it that way mtreinish :-/ | 22:29 |
| mtreinish | anteaya: heh, that's why I use a model m | 22:29 |
| clarkb | I didnt start any jjb runs fwiw | 22:29 |
| *** regXboi has quit IRC | 22:29 | |
| jeblair | clarkb: oh i thought you did | 22:30 |
| mtreinish | craige: it's pretty consistent for me, everytime I hit workboard on subunit2sql I get that error | 22:30 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Remove test skip for Cinder Ceph import backup https://review.openstack.org/289578 | 22:30 |
| craige | Kepe clickin mtreinish :-) I'd like to reproduce the errors too. | 22:30 |
| mtreinish | craige: I wonder if it's permission based | 22:30 |
| craige | okay, cool. | 22:30 |
| jeblair | fungi: https://review.openstack.org/288694 ? | 22:30 |
| mtreinish | I assume your super admin | 22:30 |
| craige | Probably. | 22:30 |
| craige | SUPER! | 22:30 |
| mtreinish | s/your/you're | 22:30 |
| anteaya | mtreinish: :) | 22:31 |
| *** cbader has quit IRC | 22:31 | |
| *** Sukhdev has quit IRC | 22:31 | |
| fungi | jeblair: also AJaeger was shopping for reviews on his constraints jobs cleanup patch. seeing if i can find it | 22:32 |
| *** kencjohnston has quit IRC | 22:32 | |
| jeblair | fungi: i +3d it https://review.openstack.org/288994 | 22:32 |
| anteaya | merged i believe | 22:32 |
| fungi | oh, excellent | 22:32 |
| craige | Made you an admin mtreinish. Does tat change anything? | 22:33 |
| *** claudiub has quit IRC | 22:33 | |
| fungi | so the other outstanding bit is that if some jenkins masters have updated definitions and others have old definitions for some of the bindep job transition, then that could be doubling function registration for those | 22:33 |
| *** jtomasek_ has joined #openstack-infra | 22:33 | |
| fungi | but obviously trying to catch too many of those up at once seems to be impacting zuul performance now | 22:33 |
| jeblair | fungi: yeah. i think after 288694 lands we should kick off manual jjb updates, probably sequentially, suffer through that for a while, then restart zuul again. after that, things should be a bit better. | 22:34 |
| mtreinish | craige: still get the exception, but now I have a red bar: http://i.imgur.com/UwHv8TE.png | 22:34 |
| *** flip214 has quit IRC | 22:34 | |
| *** thorst has joined #openstack-infra | 22:34 | |
| *** flip214 has joined #openstack-infra | 22:34 | |
| anteaya | mtreinish: congratulations | 22:35 |
| anteaya | fungi: you mention of a walk has me keen to go outside myself | 22:35 |
| fungi | jeblair: one option might be to offline update the masters? we're not running at capacity (much) today, so could probably stand to gracefully take a few offline at a time | 22:35 |
| anteaya | mmedvede: thanks for your help in the zuul memory leak issue | 22:35 |
| jhesketh | jeblair: did you see the options I put in the discussion on https://review.openstack.org/#/c/281166/2 | 22:35 |
| anteaya | back in a while | 22:35 |
| fungi | anteaya: it's lovely out (here anyway), so i highly recommend it! | 22:35 |
| anteaya | fungi: still some sunshine here, won't last much longer but going to catch some | 22:36 |
| fungi | jeblair: if that makes sense, i can put 01 and 02 into prepare for shutdown, maybe as zuul catches up on its current backlog i can safely take more masters offline at once too | 22:36 |
| jeblair | fungi: ++ especially the slow masters :) | 22:37 |
| *** [HeOS] has quit IRC | 22:37 | |
| fungi | yeah, wanted to do them first in fact | 22:37 |
| craige | Managed to replicate that error mtreinish . | 22:37 |
| fungi | okay, 01 and 02 are on their way to quiescence | 22:38 |
| jeblair | jhesketh: is that an optimization to avoid clearing the cache? | 22:38 |
| jeblair | jhesketh: or is that a memory-leak bugfix? | 22:38 |
| *** jamielennox|away is now known as jamielennox | 22:38 | |
| mtreinish | craige: cool | 22:38 |
| *** thorst has quit IRC | 22:39 | |
| jhesketh | jeblair: it's an optimisation | 22:39 |
| jeblair | jhesketh: i'd like to just rework the caching system completely in zuulv3 if that's okay? | 22:39 |
| craige | Added it to the eterpad. | 22:40 |
| craige | mtreinish ^^ | 22:40 |
| *** asettle has quit IRC | 22:41 | |
| jhesketh | jeblair: that's fine, but I think we still don't want to throw away the cache at every reload (which is what we're currently doing) | 22:41 |
| jeblair | jhesketh: i agree, but that's such a fundamental change that i want to make it in zuulv3 | 22:41 |
| jeblair | i'd really like for v2 to just be stable while we work on it | 22:42 |
| *** angdraug_ has quit IRC | 22:42 | |
| *** angdraug has joined #openstack-infra | 22:42 | |
| jhesketh | jeblair: not reloading zuul.conf isn't that big of a change.. | 22:43 |
| *** asettle has joined #openstack-infra | 22:43 | |
| *** ihrachys has quit IRC | 22:43 | |
| jhesketh | jeblair: or just copying the cache object into the new connection | 22:43 |
| jeblair | jhesketh: not dumping the cache is | 22:43 |
| *** mriedem is now known as mriedem_away | 22:44 | |
| *** yamamoto_ has joined #openstack-infra | 22:45 | |
| jhesketh | jeblair: right, so you're suggesting keeping v2 losing its cache on every reload and needing to rebuild it? | 22:45 |
| *** camunoz has joined #openstack-infra | 22:45 | |
| *** yaume has quit IRC | 22:46 | |
| jeblair | jhesketh: yes. i want v2 to stabilize (eg, fix the memory leak) so we can work on v3. this stuff is hard, and i don't want to do it twice. | 22:46 |
| craige | It appears to be all projects have the same problem with "Workboard" mtreinish. Thanks for finding it. | 22:47 |
| *** thiagop has quit IRC | 22:48 | |
| *** yamamoto_ has quit IRC | 22:51 | |
| *** jamesmcarthur has quit IRC | 22:51 | |
| *** doug-fish has quit IRC | 22:53 | |
| *** fawadkhaliq has quit IRC | 22:54 | |
| *** fawadkhaliq has joined #openstack-infra | 22:56 | |
| *** annegentle has quit IRC | 22:56 | |
| jeblair | we still don't have any devstack-trusty-2-node nodes | 22:57 |
| SamYaple | How quickly after Xenial releases will we get to switch our gates over too it? | 22:59 |
| SamYaple | I am very anxious to do so | 22:59 |
| *** doug-fish has joined #openstack-infra | 23:01 | |
| jeblair | fungi: do you have a script to clean up leaked bluebox fips? | 23:01 |
| *** jtomasek_ has quit IRC | 23:01 | |
| *** jamesmcarthur has joined #openstack-infra | 23:01 | |
| *** jtomasek has quit IRC | 23:02 | |
| *** jpr has quit IRC | 23:04 | |
| fungi | jeblair: a one-liner shell for loop, but i can run it now | 23:05 |
| *** eharney has quit IRC | 23:05 | |
| fungi | i'm collecting a list of them now | 23:05 |
| *** hashar has quit IRC | 23:06 | |
| fungi | hopefully soon someone will have available time to collect some logs of the leaking as it occurs and get up with bluebox to troubleshoot from their end | 23:06 |
| *** asettle has quit IRC | 23:07 | |
| *** gildub has joined #openstack-infra | 23:07 | |
| *** Sukhdev_ has joined #openstack-infra | 23:07 | |
| *** bgmccollum is now known as recheckbot | 23:08 | |
| *** nelsnelson has quit IRC | 23:09 | |
| *** tsg has quit IRC | 23:09 | |
| *** annegentle has joined #openstack-infra | 23:09 | |
| *** piet has quit IRC | 23:10 | |
| *** dingyichen has joined #openstack-infra | 23:10 | |
| EmilienM | hello Infra, can I have a review to fix login.defs on centos/fedora images please ? https://review.openstack.org/#/c/288244/ | 23:10 |
| *** jamesmcarthur has quit IRC | 23:11 | |
| *** harlowja has quit IRC | 23:13 | |
| *** jamesmcarthur has joined #openstack-infra | 23:13 | |
| *** mtreinish has quit IRC | 23:13 | |
| *** salv-orlando has quit IRC | 23:13 | |
| *** salv-orlando has joined #openstack-infra | 23:14 | |
| fungi | EmilienM: on 288244 i object to the assertion that i dropped SYS_UID_MIN and SYS_GID_MIN from the file. i copied it directly from an official centos 6.x image, and merely added SYS_UID_MAX and SYS_GID_MAX and increased the values of UID_MIN and GID_MIN | 23:14 |
| *** harlowja has joined #openstack-infra | 23:14 | |
| *** mtreinish has joined #openstack-infra | 23:14 | |
| fungi | i did not remove any settings from the file we had previously been defaulting to on that platform | 23:14 |
| *** _ari_ has quit IRC | 23:15 | |
| EmilienM | fungi: ok, should I update something? | 23:15 |
| fungi | i will admit that it's possible that newer centos have started explicitly setting SYS_*_MIN and SYS_*_MAX | 23:15 |
| EmilienM | they do, I checked on centos7 | 23:16 |
| fungi | okay, just pointing out that this was from centos 6 (at the time we introduced the change there wasn't a centos 7 or else it was new enough we didn't have it running anywhere yet) | 23:16 |
| EmilienM | fungi: there is no problem at all | 23:17 |
| fungi | i checked the fedora releases at the same time (18 or 19 if memory serves) and it was identical to what i copied from centos | 23:17 |
| EmilienM | I'm just trying to fix that, our CI is running OpenStack packages (keystone, etc) and it's currently exposed to races like this | 23:17 |
| fungi | so anyway, this is likely a shift in behavior of the platforms since we added the file, not anything i intentionally removed | 23:17 |
| clarkb | how can that race? isnt puppet single threaded? | 23:18 |
| EmilienM | fungi: I know it was not intentional | 23:18 |
| fungi | EmilienM: okay, just responding to your comment on the change where you said "I'm not sure exactly why Jeremy dropped the values" | 23:19 |
| EmilienM | clarkb: before that patch, we have CI failures that happen randomly where rabbitmq is installed *before* keystone and steal its UID | 23:19 |
| *** markvoelker_ has joined #openstack-infra | 23:19 | |
| clarkb | right but keystone should install after and get a different uid | 23:20 |
| clarkb | they are installed one after the other by the puppet yum provider iirc | 23:20 |
| EmilienM | that's wrong, you can install keystone before rabbitmq if you don't use notifications | 23:20 |
| EmilienM | yes they are | 23:20 |
| clarkb | byt in that case rabbitmq should pick a different uid | 23:20 |
| fungi | in which case rabbit would get a different uid though, right? | 23:20 |
| EmilienM | but rabbitmq should not steal keystone's UID | 23:21 |
| clarkb | fungi: yes exaclty. this sounds like broken packages | 23:21 |
| mordred | crinkle: keystone has implemented a new way of bootstrapping for mitaka that does not involve needing admin-token | 23:21 |
| EmilienM | it should not take a reserved UID | 23:21 |
| fungi | i agree, it shouldn't. why does it? | 23:21 |
| mordred | EmilienM: ^^ you too | 23:21 |
| crinkle | mordred: yup it's very exciting | 23:21 |
| mordred | cool. just wanted to make sure you'd seen it | 23:21 |
| *** piet has joined #openstack-infra | 23:21 | |
| EmilienM | mordred: keystone-manage bootstrap? | 23:22 |
| *** xarses has joined #openstack-infra | 23:22 | |
| fungi | jeblair: we definitely leaked our full 50 fip quota in bluebox again, after only a few days of having cleaned it up | 23:22 |
| fungi | i'm removing them all now | 23:22 |
| jeblair | fungi: thanks. for some reason nodepool only wants to build devstack-trusty-2-node instances there | 23:22 |
| EmilienM | mordred: puppet-keystone already support & test it by default | 23:22 |
| crinkle | EmilienM: it does? | 23:23 |
| EmilienM | crinkle: of course ! | 23:23 |
| EmilienM | in mitaka only though | 23:23 |
| *** zhurong has joined #openstack-infra | 23:23 | |
| *** tphummel has quit IRC | 23:23 | |
| EmilienM | crinkle: https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L436-L443 | 23:23 |
| EmilienM | https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L981-L989 | 23:23 |
| xarses | I'm probably a broken record, so don't hate me, but what's the status on the NOT_REGISTERED errors? | 23:23 |
| mordred | EmilienM: wow | 23:24 |
| EmilienM | fungi, clarkb : I think rabbitms packaging is using a dynamic UID/GID | 23:25 |
| *** tphummel has joined #openstack-infra | 23:25 | |
| clarkb | dynamic is fine | 23:25 |
| EmilienM | using dynamic is fine when you set SYS_UID_MIN and SYS_GID_MIN in login.defs | 23:25 |
| clarkb | as long as it picks when nothing else is picking | 23:25 |
| fungi | xarses: should clear up at any moment. i just deleted the cruft which was causing us to be over quota in the one location nodepool wanted to build teh workers for that | 23:25 |
| mordred | SamYaple: I'm hoping we can start having xenial nodes availabe not-too-long after it's released so that we can start testing | 23:25 |
| *** zhurong has quit IRC | 23:25 | |
| mordred | SamYaple: however, moving the entire gate will take some effort/testing | 23:26 |
| mordred | and will not begin until newton | 23:26 |
| *** asettle has joined #openstack-infra | 23:26 | |
| xarses | fungi: TYVM | 23:26 |
| EmilienM | clarkb: if https://review.openstack.org/#/c/288244/ is good for you, please review when you have time | 23:26 |
| mordred | SamYaple: since mitaka has been developed against trusty | 23:26 |
| EmilienM | mordred: is it the bootstrap thing, right? if yes, we support it, and mostly because there is no other choice in Mitaka. | 23:26 |
| fungi | EmilienM: i was led from earlier versions of the config file to believe that there were useradd defaults if you didn't explicitly override them in that file | 23:26 |
| clarkb | EmilienM: I am saying it should be unnecessary, there is a bug in packaging | 23:26 |
| EmilienM | mordred: when they did the switch, it broke our CI and we had to do it quickly | 23:27 |
| clarkb | *shouldnt | 23:27 |
| *** piet has quit IRC | 23:27 | |
| mordred | EmilienM: hah. nice way to get you to update quickly | 23:27 |
| EmilienM | clarkb: which bug? | 23:27 |
| clarkb | EmilienM: the keystone vs rabbiy thibg? | 23:27 |
| clarkb | or did I mixup cjamges | 23:27 |
| EmilienM | clarkb: keystone use static UID, rabbit uses dynamic. If you don't set SYS_UID_MIN and SYS_GID_MIN in login.defs, you expose dynamic UIDS packages to steal statics, right? | 23:28 |
| *** aspiers has quit IRC | 23:28 | |
| EmilienM | maybe I misunderstand | 23:28 |
| clarkb | that implies keystone package is broken | 23:29 |
| clarkb | dont use static uids if they can be stolen | 23:29 |
| clarkb | fix the package | 23:29 |
| EmilienM | clarkb: https://git.fedorahosted.org/cgit/setup.git/tree/uidgid | 23:29 |
| EmilienM | I did not write that | 23:29 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:30 | |
| EmilienM | that's how RDO works, we have static UID/GID | 23:30 |
| *** piet has joined #openstack-infra | 23:30 | |
| *** Jeffrey4l has quit IRC | 23:30 | |
| EmilienM | but not for rabbitmq apparently, and a lot of services | 23:31 |
| SamYaple | mordred: does that mean I could set it for some kolla gates though? Really wanting to use some newer kernel bits | 23:31 |
| SamYaple | EmilienM: RDO uses static UID/GID?! | 23:31 |
| SamYaple | have you never had conflicts doing that? | 23:31 |
| EmilienM | SamYaple: for some packages, yes https://git.fedorahosted.org/cgit/setup.git/tree/uidgid | 23:32 |
| EmilienM | SamYaple: not afik, if your login.defs is ok | 23:32 |
| mordred | SamYaple: once it's ready, yeah | 23:32 |
| mordred | SamYaple: just know that it'll be a non-zero amount of work for us to get it up and supported, so it might be a little while after the release before it's actually a possibility | 23:32 |
| EmilienM | clarkb: consider https://review.openstack.org/#/c/288244/ as a sync to recent system update. Current login.defs is not updated anyway | 23:32 |
| mordred | SamYaple: however, I did already start poking at making sure that dib can make xenial images | 23:33 |
| EmilienM | clarkb: my patch add the missing bits to have the same file as in centos7 or recent fedora | 23:33 |
| mordred | SamYaple: and the current push to get to ubuntu-trusty instead of bare-trusty will help _alot_ to reduce teh surface area of te problem | 23:33 |
| greghaynes | mordred: crinkle put up a bugfix for that and AFAICT it was the only bug for xenial to work | 23:33 |
| greghaynes | mordred: did you confirm xenial works with dib too though? | 23:33 |
| greghaynes | because I havent | 23:34 |
| *** Qiming has joined #openstack-infra | 23:34 | |
| mordred | greghaynes: I have built xenial images | 23:34 |
| greghaynes | w00t | 23:34 |
| mordred | greghaynes: I do not rembemer if I was able to boot them - but I think I was able to? | 23:34 |
| *** pahuang has quit IRC | 23:34 | |
| SamYaple | mordred: I would like to switch Kolla over to 16.04 in base images and base distro ASAP. so let me know if I can help with testing/patches/reviews | 23:35 |
| greghaynes | haha | 23:35 |
| mordred | greghaynes: you need to do 'ln -s /usr/share/debootstrap/scripts/gutsy /usr/share/debootstrap/scripts/xenial' | 23:35 |
| mordred | greghaynes: for debootstrap to know how to react | 23:35 |
| greghaynes | mordred: ah, right | 23:35 |
| mordred | greghaynes: (on the build host) | 23:35 |
| SamYaple | i mean xenial cant be that different that trusty/percise | 23:35 |
| mordred | greghaynes: if you do that, it pretty much works | 23:36 |
| mordred | SamYaple: ah, but it can | 23:36 |
| SamYaple | ive been using the same script since 10.04 | 23:36 |
| mordred | SamYaple: xenial is systemd | 23:36 |
| greghaynes | yea, I forgot about how debootstrap wants to know of each release | 23:36 |
| SamYaple | yea i get that | 23:36 |
| SamYaple | i should probably dig into dib more | 23:36 |
| mordred | greghaynes: also - I _think_ we need to update systemd detection | 23:36 |
| fungi | EmilienM: ideally we should make sure the new state of that file is as identical as possible to whatever centos 7's default is, but with a gap so that the 2000-2999 range won't get any accounts (system or otherwise) autocreated within it | 23:36 |
| EmilienM | fungi, clarkb: if you don't want the patch in openstack-infra, just let me know and give me a valid reason so I can report the issues in RDO. But I'll fix the file in our Puppet OpenStack CI scripts, something I don't like because probably somebody else will hit this issue too | 23:36 |
| greghaynes | mordred: oh? | 23:36 |
| mordred | greghaynes: I wanna say that I ran in to a case where it was detecting the wrong thing | 23:36 |
| mordred | greghaynes: I think. I can't remember. it was a weekend thing a little while ago | 23:37 |
| mordred | greghaynes: I'll hit it again soon if you don't get to it | 23:37 |
| greghaynes | mordred: ok, awesome | 23:37 |
| EmilienM | fungi: my patch make it identical, I'm running centos7 | 23:37 |
| *** doug-fish has quit IRC | 23:37 | |
| SamYaple | mordred: the dib stuff, does it run on 14.04 is that the issue? | 23:37 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests https://review.openstack.org/288840 | 23:37 |
| fungi | EmilienM: perfect, then my +2 still stands | 23:38 |
| fungi | and thanks | 23:38 |
| EmilienM | fungi: thanks for considering the patch | 23:38 |
| mordred | SamYaple: no - dib builds in a chroot of the distro in question | 23:39 |
| mordred | SamYaple: but dib is, itself, a set of run-parts scripts to get the image into the state you want | 23:39 |
| mordred | SamYaple: and if they're making choices based on prior understanding of ubuntu that may have changed | 23:39 |
| mordred | SamYaple: then it can be work | 23:40 |
| mordred | more specifically - what I'm really talking about here is the simple-init element | 23:40 |
| fungi | EmilienM: so just to clarify, the puppet-keystone module hard-sets the keystone uid/gid to something less than 200? | 23:40 |
| EmilienM | fungi: no, we rely on packaging | 23:40 |
| mordred | which sets up things for handling config-drive and network interfaces on boot | 23:40 |
| EmilienM | we don't manage POSIX resources | 23:40 |
| EmilienM | we think packages already do that so we don't want to override it | 23:40 |
| mordred | SamYaple: so the real thing that is most likely to need attention for the gate is making sure that the simple-init element in dib works | 23:40 |
| mordred | with xenial | 23:40 |
| fungi | EmilienM: oh, the rdo keystone package creates a user <200? | 23:40 |
| EmilienM | fungi: that's official uidgid package https://git.fedorahosted.org/cgit/setup.git/tree/uidgid | 23:40 |
| EmilienM | in Fedora | 23:41 |
| EmilienM | fungi: yes | 23:41 |
| fungi | ouch | 23:41 |
| fungi | oh! | 23:41 |
| mordred | SamYaple: if you want to help with that - it's easy enough to build an ubuntu-minial simple-init image with xenial and then start troubleshooting if it works on clouds that ahve dHCP and also on clouds that only have static interfaces in config-drive | 23:41 |
| fungi | so fedora actually has a standardized uid/gid registry for "common" system accounts? that's rather interesting | 23:41 |
| mordred | wow | 23:42 |
| fungi | treating it like /etc/services | 23:42 |
| SamYaple | mordred: i do currently do all my kolla-dev on xenial in my private cloud | 23:42 |
| SamYaple | the image i built there does dhpc proper like | 23:42 |
| fungi | it's a novel approach to the problem i guess | 23:42 |
| *** asettle has quit IRC | 23:42 | |
| SamYaple | so ill have to dig into this dib stuff | 23:42 |
| mordred | SamYaple: did you use ubuntu-minimal and simple-init/glean? :) | 23:42 |
| SamYaple | nope | 23:42 |
| mordred | ah. | 23:42 |
| mordred | you did not use dib | 23:43 |
| mordred | you shoudl learn to love dib | 23:43 |
| SamYaple | thats it | 23:43 |
| EmilienM | fungi: because of that, we don't do anything in puppet related to users/groups | 23:43 |
| mordred | it'll change your life | 23:43 |
| EmilienM | fungi: we had so much issues in the patch | 23:43 |
| EmilienM | in the past* | 23:43 |
| SamYaple | you say that but ive got this script ive been using for so long... and it just does what i want. debootstrap does the heavy lifting | 23:43 |
| fungi | EmilienM: in that case, i assume the default SYS_UID_MIN is something below 200 and so they raised it in login.defs explicitly rather than adjusting teh compiled-in lower bound when they introduced their static uid/gid registry | 23:43 |
| anteaya | lots of things change my life | 23:43 |
| EmilienM | fungi: I don't know if there is actually a default | 23:44 |
| anteaya | some even come with the label "progress" attached | 23:44 |
| anteaya | sometimes I question the definition of that label | 23:44 |
| fungi | EmilienM: well, there's a "default" regardless of whether it's implicit or explicit, it just may default to the lowest possible non-root uid/gid (i.e., 1) | 23:44 |
| EmilienM | fungi: right... | 23:45 |
| anteaya | I enjoyed my walk | 23:45 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add experimental job to test xstatic packaging https://review.openstack.org/289125 | 23:45 |
| anteaya | didn't even have to use my flashlight | 23:45 |
| anteaya | jhesketh: I question whether than syntax will work in zuul/layout.yaml: https://review.openstack.org/#/c/289125/4/zuul/layout.yaml | 23:47 |
| anteaya | that syntax* | 23:47 |
| fungi | EmilienM: the comments in the debian/ubuntu version of the file suggest an implicit default lowe bound of 100, but i have no idea if that's consistent with centos/fedora compile time defaults | 23:47 |
| EmilienM | fungi: I took the centos7 file, ran a diff and apply the diff in this patch | 23:48 |
| *** Daisy has joined #openstack-infra | 23:48 | |
| jhesketh | anteaya: it is quite likely I fat fingered something | 23:48 |
| *** hdd has quit IRC | 23:48 | |
| jhesketh | anteaya: but I believe it's correct | 23:48 |
| *** akshai has quit IRC | 23:49 | |
| *** dizquierdo has joined #openstack-infra | 23:49 | |
| fungi | EmilienM: and it looks like debian/ubuntu may also have a static registry of sorts, since i see some common system accounts in low values below 100 which generally aren't created by packages and would be part of the base account set, so i guess this is actually a common solution just red hat ran out of values under 100 for their registry and had to increase the minimum in configuration | 23:49 |
| jhesketh | anteaya: look at line 509 | 23:49 |
| EmilienM | fungi: that's also what I understood | 23:49 |
| fungi | there are just so few on non-rh platforms generally that i don't think of them as much of a registry | 23:50 |
| * anteaya looks at line 509 | 23:50 | |
| anteaya | jhesketh: that is a zuul template | 23:51 |
| anteaya | yours is a job name | 23:51 |
| anteaya | I don't think the two mash together well | 23:51 |
| anteaya | but we can wait for jenkins | 23:51 |
| fungi | jenkins01 and 02 are almost idle finally. as soon as these last couple jobs complete i'll shut them down and update their configs | 23:51 |
| anteaya | yay | 23:51 |
| *** HeOS has joined #openstack-infra | 23:52 | |
| *** salv-orl_ has joined #openstack-infra | 23:52 | |
| *** Daisy has quit IRC | 23:53 | |
| fungi | zuul's pretty quiet too, so hopefully whatever load i generate when they come back online and start registering their jobs again will be easily absorbed | 23:53 |
| *** Daisy has joined #openstack-infra | 23:53 | |
| *** abregman|nb has quit IRC | 23:53 | |
| *** keedya has quit IRC | 23:54 | |
| *** annegentle has quit IRC | 23:54 | |
| *** dizquierdo has quit IRC | 23:55 | |
| openstackgerrit | James Polley proposed openstack-dev/pbr: Correct typo - s/enabeld/enabled/ https://review.openstack.org/289623 | 23:55 |
| jhesketh | anteaya: 2742 is another example | 23:55 |
| jhesketh | anteaya: basically it's how zuul says "run this job first, and if it passes, then run the following" | 23:55 |
| *** salv-orlando has quit IRC | 23:55 | |
| jhesketh | jeblair: do you have a few moments to discuss some zuulv3 design points? | 23:56 |
| jeblair | jhesketh: surej | 23:56 |
| jhesketh | jeblair: so the ansible launcher you started uses the zuul merger directly.. is the intention for the launcher to replace the mergers? | 23:57 |
| fungi | clarkb: so anyway, that seems to be the explanation. fedora kept adding static gid/uid reservations under the compile-time default minimum of 100 until they ran out of available numbers, then raised the lower bound in configuration only to accommodate new static assignments between 100 and 200 | 23:57 |
| tchaypo | lifeless: Nakato: https://review.openstack.org/#/c/289623/ | 23:57 |
| *** Daisy has quit IRC | 23:58 | |
| fungi | clarkb: so if you don't carry their new in-config defaults, you end up with useradd stepping on system accounts in that new range | 23:58 |
| *** HeOS has quit IRC | 23:58 | |
| anteaya | jhesketh: sorry I'm not able to see what you are seeing: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n2742 | 23:58 |
| EmilienM | fungi: exactly our case in Puppet OpenStack CI | 23:58 |
| anteaya | jhesketh: but finish your conversation first | 23:59 |
| *** ashleighfarnham has quit IRC | 23:59 | |
| jhesketh | anteaya: ah, my checkout is different... let me find it | 23:59 |
| jhesketh | anteaya: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n2705 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
