| *** yamahata has quit IRC | 00:20 | |
| *** e0ne has joined #openstack-infra | 00:20 | |
| *** Sukhdev has quit IRC | 00:23 | |
| *** e0ne has quit IRC | 00:24 | |
| tchaypo | l/me fights the git | 00:25 |
|---|---|---|
| * tchaypo fights the irc | 00:25 | |
| *** matsuhashi has joined #openstack-infra | 00:27 | |
| *** nosnos has joined #openstack-infra | 00:43 | |
| *** sarob has joined #openstack-infra | 00:47 | |
| *** sarob has quit IRC | 00:52 | |
| *** bknudson has left #openstack-infra | 01:01 | |
| tchaypo | All my reviews are failing after 7-8 minutes. If I hadn't seen the chatter in here this morning I'd be very sad. | 01:05 |
| *** yamahata has joined #openstack-infra | 01:06 | |
| *** fifieldt has joined #openstack-infra | 01:15 | |
| *** e0ne has joined #openstack-infra | 01:18 | |
| *** yamahata has quit IRC | 01:21 | |
| *** yamahata has joined #openstack-infra | 01:22 | |
| *** e0ne has quit IRC | 01:23 | |
| *** otter768 has quit IRC | 01:23 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Capture *.failure logs from $BASE/status/stack https://review.openstack.org/91965 | 01:40 |
| bodepd | mordred: I've got some inconsistent comments on I tend to gesticulate wildly when I get excited about something. | 01:42 |
| bodepd | damnit, that was a failed copy/paste :) | 01:43 |
| bodepd | mordred: https://review.openstack.org/#/c/96522/ | 01:44 |
| *** sarob has joined #openstack-infra | 01:47 | |
| *** otherwiseguy has joined #openstack-infra | 01:47 | |
| *** sarob has quit IRC | 01:52 | |
| *** otherwiseguy has quit IRC | 01:52 | |
| *** dims has quit IRC | 01:57 | |
| *** dims has joined #openstack-infra | 01:59 | |
| *** dims has quit IRC | 02:00 | |
| *** trinaths has joined #openstack-infra | 02:05 | |
| *** otter768 has joined #openstack-infra | 02:06 | |
| *** achuprin has quit IRC | 02:08 | |
| *** trinaths has quit IRC | 02:10 | |
| harlowja_ | hi guys, i just tagged a taskflow release, is there usually a > 10 minute delay on the push to pypi? | 02:11 |
| nibalizer | bodepd: got my +1 | 02:12 |
| *** amotoki has joined #openstack-infra | 02:14 | |
| *** e0ne has joined #openstack-infra | 02:18 | |
| *** sarob has joined #openstack-infra | 02:20 | |
| *** achuprin has joined #openstack-infra | 02:23 | |
| *** e0ne has quit IRC | 02:23 | |
| *** oomichi has joined #openstack-infra | 02:25 | |
| *** sarob has quit IRC | 02:25 | |
| *** bhuvan has quit IRC | 02:31 | |
| bodepd | nibalizer: is there an official policy on what requires CLAs? | 02:33 |
| *** otherwiseguy has joined #openstack-infra | 02:42 | |
| *** weshay has quit IRC | 02:42 | |
| nibalizer | bodepd: not that I'm aware of, but I'm still comming on to the project, others are too, probably why you're getting inconsistency | 02:44 |
| *** sarob has joined #openstack-infra | 02:47 | |
| *** fifieldt has quit IRC | 02:47 | |
| *** e0ne has joined #openstack-infra | 02:48 | |
| *** e0ne has quit IRC | 02:50 | |
| *** sarob has quit IRC | 02:51 | |
| *** otherwiseguy has quit IRC | 02:52 | |
| nibalizer | bodepd: so infra/confing doesn't require a CLA so a request like yours wouldn't require one | 02:52 |
| nibalizer | https://review.openstack.org/#/admin/projects/openstack-infra/config | 02:52 |
| *** bhuvan has joined #openstack-infra | 02:58 | |
| *** gokrokve has joined #openstack-infra | 03:09 | |
| *** gokrokve has quit IRC | 03:14 | |
| *** harlowja_at_home has joined #openstack-infra | 03:15 | |
| *** e0ne has joined #openstack-infra | 03:18 | |
| *** e0ne has quit IRC | 03:20 | |
| *** harlowja_at_home has quit IRC | 03:23 | |
| *** lbragstad has joined #openstack-infra | 03:23 | |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/config: Update elastic-recheck link on index.html https://review.openstack.org/97134 | 03:26 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/config: Update elastic-recheck link on index.html https://review.openstack.org/97134 | 03:31 |
| pleia2 | sneaky tabs | 03:31 |
| *** nosnos has quit IRC | 03:34 | |
| *** talluri has joined #openstack-infra | 03:35 | |
| *** mmaglana has joined #openstack-infra | 03:39 | |
| *** lcheng_ has joined #openstack-infra | 03:42 | |
| *** harlowja_ is now known as harlowja_away | 03:47 | |
| *** sarob has joined #openstack-infra | 03:47 | |
| *** otter768 has quit IRC | 03:48 | |
| mordred | bodepd: sorry for the confusion - I think we probably need to get jeblair to weigh in | 03:50 |
| *** chuckC has quit IRC | 03:50 | |
| *** miqui has quit IRC | 03:51 | |
| *** talluri has quit IRC | 03:51 | |
| *** lcheng_ has quit IRC | 03:52 | |
| *** sarob has quit IRC | 03:52 | |
| *** chuckC has joined #openstack-infra | 03:52 | |
| *** talluri has joined #openstack-infra | 03:53 | |
| *** zhiyan_ is now known as zhiyan | 03:54 | |
| *** harlowja_at_home has joined #openstack-infra | 03:57 | |
| *** Sukhdev has joined #openstack-infra | 04:01 | |
| *** Ryan_Lane has joined #openstack-infra | 04:09 | |
| *** Ryan_Lane has quit IRC | 04:09 | |
| *** harlowja_at_home has quit IRC | 04:13 | |
| *** e0ne has joined #openstack-infra | 04:18 | |
| mordred | sdague: when you're around, could you help explain this to me: https://review.openstack.org/#/c/94713/1/tox.ini ? | 04:18 |
| mordred | sdague: namely, what's up with the python -m nova.openstack.common.lockutils - if that's a thing that's needed in the test suite, why isn't it in a fixture? | 04:19 |
| mordred | sdague: oh - nevermind. I read the code | 04:19 |
| mordred | I really wish we didn't have to paste that into tox.ini - but also, fair | 04:20 |
| *** e0ne has quit IRC | 04:23 | |
| *** nosnos has joined #openstack-infra | 04:26 | |
| clarkb | mordred: its a terrible terruible thing that should diaf | 04:28 |
| clarkb | there is no reason for it to exist | 04:28 |
| openstackgerrit | A change was merged to openstack-infra/config: Add nodepool_prod.pp to documentation https://review.openstack.org/97115 | 04:33 |
| *** trinaths has joined #openstack-infra | 04:35 | |
| *** harlowja_at_home has joined #openstack-infra | 04:36 | |
| *** mmaglana has quit IRC | 04:37 | |
| *** yfried has quit IRC | 04:38 | |
| * StevenK stabs setuputils 4.0 | 04:39 | |
| StevenK | TypeError: 'unicode' does not have the buffer interface is now impacting local tests in newly created environments | 04:39 |
| openstackgerrit | Fathi Boudra proposed a change to openstack-infra/jenkins-job-builder: authorization property: add extended read support https://review.openstack.org/97097 | 04:40 |
| *** gokrokve has joined #openstack-infra | 04:41 | |
| *** sarob has joined #openstack-infra | 04:44 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Don't delete tarballs in run-wheel.sh https://review.openstack.org/97141 | 04:44 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Generate sha1 and md5 sum file for wheels https://review.openstack.org/97142 | 04:44 |
| clarkb | mordred: dhellmann ^ that will fix at least one bug with the wheels change | 04:45 |
| sarob | Mordred: I'd like to create storyboard project training-guides | 04:47 |
| *** sarob_ has joined #openstack-infra | 04:47 | |
| harlowja_at_home | clarkb, thx for the quick fix | 04:47 |
| *** e0ne has joined #openstack-infra | 04:48 | |
| sarob | Mordred: what's the process to get up and running? | 04:48 |
| *** sarob__ has joined #openstack-infra | 04:49 | |
| *** sarob_ has quit IRC | 04:51 | |
| *** sarob__ has quit IRC | 04:54 | |
| *** e0ne has quit IRC | 05:02 | |
| *** W00dy has joined #openstack-infra | 05:03 | |
| *** W00dy is now known as Guest37560 | 05:03 | |
| *** gokrokve has quit IRC | 05:06 | |
| *** yamahata has quit IRC | 05:06 | |
| *** basha has joined #openstack-infra | 05:06 | |
| *** yamahata has joined #openstack-infra | 05:06 | |
| *** vkdrao has joined #openstack-infra | 05:08 | |
| openstackgerrit | Fathi Boudra proposed a change to openstack-infra/jenkins-job-builder: general: add support for childCustomWorkspace https://review.openstack.org/97108 | 05:09 |
| *** harlowja_at_home has quit IRC | 05:26 | |
| openstackgerrit | Frederic Lepied proposed a change to openstack-infra/config: glanceclient: make the Python 3.3 gate voting https://review.openstack.org/96370 | 05:33 |
| *** ildikov has quit IRC | 05:35 | |
| *** sarob has quit IRC | 05:36 | |
| *** masayuki_ has joined #openstack-infra | 05:39 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon https://review.openstack.org/95716 | 05:46 |
| *** sarob has joined #openstack-infra | 05:47 | |
| *** Guest37560 has quit IRC | 05:50 | |
| *** sarob has quit IRC | 05:51 | |
| trinaths | Hi Good Morning.. | 05:55 |
| *** Sukhdev has quit IRC | 06:01 | |
| mattoliverau | Morning trinaths | 06:02 |
| *** _nadya_ has joined #openstack-infra | 06:07 | |
| *** mkoderer has joined #openstack-infra | 06:09 | |
| *** ildikov has joined #openstack-infra | 06:09 | |
| *** bhuvan has quit IRC | 06:13 | |
| *** sarob has joined #openstack-infra | 06:19 | |
| *** jlibosva has joined #openstack-infra | 06:23 | |
| *** sarob has quit IRC | 06:24 | |
| *** yfried has joined #openstack-infra | 06:25 | |
| *** _nadya_ has quit IRC | 06:26 | |
| *** _nadya_ has joined #openstack-infra | 06:29 | |
| *** _nadya_ has quit IRC | 06:30 | |
| *** Guest22618 has quit IRC | 06:31 | |
| *** Sukhdev has joined #openstack-infra | 06:33 | |
| *** _nadya_ has joined #openstack-infra | 06:34 | |
| *** lcheng_ has joined #openstack-infra | 06:36 | |
| *** Sukhdev has quit IRC | 06:39 | |
| *** jcoufal has joined #openstack-infra | 06:39 | |
| *** _nadya_ has quit IRC | 06:39 | |
| *** andreykurilin has joined #openstack-infra | 06:39 | |
| *** _nadya_ has joined #openstack-infra | 06:41 | |
| *** _nadya_ has quit IRC | 06:42 | |
| *** otherwiseguy has joined #openstack-infra | 06:44 | |
| *** yfried has quit IRC | 06:44 | |
| *** sarob has joined #openstack-infra | 06:47 | |
| *** otherwiseguy has quit IRC | 06:48 | |
| *** afazekas has quit IRC | 06:49 | |
| *** hashar has joined #openstack-infra | 06:51 | |
| *** sarob has quit IRC | 06:51 | |
| *** marun has joined #openstack-infra | 06:52 | |
| *** salv-orlando has joined #openstack-infra | 06:55 | |
| *** e0ne has joined #openstack-infra | 06:56 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/gerritlib: Apply cookiecutter defaults https://review.openstack.org/94844 | 06:57 |
| *** wenlock has quit IRC | 06:59 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: cleaning up index.rst file https://review.openstack.org/96407 | 07:02 |
| *** praneshp has joined #openstack-infra | 07:06 | |
| *** marun is now known as marun_afk | 07:09 | |
| *** basha has quit IRC | 07:11 | |
| *** Longgeek has joined #openstack-infra | 07:16 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix up fetching jquery.visibility https://review.openstack.org/91316 | 07:17 |
| *** e0ne has quit IRC | 07:18 | |
| *** rgerganov has joined #openstack-infra | 07:18 | |
| Longgeek | Joshua Hesketh：ping | 07:22 |
| mattoliverau | Longgeek: if you want to ping him it'll probably be better to ping his irc nick so he gets an alert :) jhesketh: your being pinged! | 07:25 |
| jhesketh | oh hiya | 07:25 |
| jhesketh | Longgeek: mattoliverau's tactic got my attention.. what's up :-) | 07:26 |
| fifieldt_ | fungi, jeblair - just saw your pings here. assume you'll read the patch response (in essence: it's intentional, and desired by docs team too) | 07:26 |
| *** fifieldt_ is now known as fifieldt | 07:26 | |
| hashar | jhesketh: it is great to see you get added to the Zuul core team (i.e. +2 / approve). Your reviews have always been very helpful. | 07:26 |
| Longgeek | mattoliverau： ok | 07:26 |
| jhesketh | thanks hashar :-) | 07:26 |
| *** shardy has joined #openstack-infra | 07:27 | |
| Longgeek | jhesketh: Your patch https://review.openstack.org/#/c/91316/6/etc/status/fetch-dependencies.sh the https://raw.githubusercontent.com/mathiasbynens/jquery-visibility/master/jquery-visibility.js url incorrect. | 07:28 |
| jhesketh | Longgeek: How so? | 07:29 |
| Longgeek | jhesketh: The correct url is https://github.com/mathiasbynens/jquery-visibility/blob/master/jquery-visibility.js | 07:30 |
| *** salv-orlando has quit IRC | 07:31 | |
| jhesketh | Longgeek: Okay, but if you then click on "Raw" in github it takes you to the URL I was using | 07:31 |
| jhesketh | we want to wget the raw file, otherwise we'll pull in all of githubs html | 07:31 |
| Longgeek | jhesketh: See http://paste.openstack.org/show/82390/ | 07:33 |
| mattoliverau | Night all! | 07:34 |
| *** andreykurilin has quit IRC | 07:35 | |
| jhesketh | Longgeek: I ran the same command and it works for me http://paste.openstack.org/show/82391/ | 07:35 |
| jhesketh | Longgeek: what happens if you run curl without --silent? | 07:36 |
| jhesketh | seeya mattoliverau! | 07:36 |
| *** otherwiseguy has joined #openstack-infra | 07:38 | |
| Longgeek | jhesketh: ok, here is my question, sorry | 07:38 |
| *** otherwiseguy has quit IRC | 07:42 | |
| *** matrohon has joined #openstack-infra | 07:44 | |
| jhesketh | Longgeek: what was your question sorry? | 07:45 |
| Longgeek | jhesketh:My https proxy server is not working properly | 07:46 |
| jhesketh | ah ok | 07:46 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add require-approvals to EventFilter representation https://review.openstack.org/88570 | 07:46 |
| *** ociuhandu has quit IRC | 07:46 | |
| *** sarob has joined #openstack-infra | 07:47 | |
| *** sarob has quit IRC | 07:51 | |
| *** ihrachyshka has joined #openstack-infra | 07:52 | |
| *** thomasbiege has joined #openstack-infra | 07:54 | |
| trinaths | mattoliverau: good night | 07:54 |
| *** Guest22618 has joined #openstack-infra | 07:58 | |
| *** yfried has joined #openstack-infra | 07:59 | |
| *** ildikov has quit IRC | 08:02 | |
| *** thomasbiege has quit IRC | 08:04 | |
| *** ildikov has joined #openstack-infra | 08:10 | |
| *** SlickN1k has joined #openstack-infra | 08:11 | |
| *** spiffxp_ has joined #openstack-infra | 08:12 | |
| *** lcheng_ has quit IRC | 08:14 | |
| *** vponomaryov has joined #openstack-infra | 08:15 | |
| *** SlickN1k has quit IRC | 08:15 | |
| *** spiffxp_ has quit IRC | 08:15 | |
| therve | I think there is an issue with the latest setuptools release? | 08:17 |
| therve | The gate is about to explode | 08:17 |
| *** apevec has joined #openstack-infra | 08:19 | |
| apevec | hi, this gate failure: http://logs.openstack.org/56/95656/5/gate/gate-glance-pep8/0676bbe/console.html | 08:20 |
| apevec | happened after setuptools 4.0 was released | 08:21 |
| *** dizquierdo has joined #openstack-infra | 08:22 | |
| therve | apevec, Yeah. I think we need < 3.6 | 08:23 |
| therve | Or 3.7 | 08:23 |
| therve | 3 releases in 1h sounds suspicious | 08:24 |
| *** _afazekas has joined #openstack-infra | 08:24 | |
| *** basha has joined #openstack-infra | 08:25 | |
| therve | https://pypi.python.org/packages/source/s/setuptools/ fwiw | 08:26 |
| *** yfried has quit IRC | 08:26 | |
| *** ominakov has joined #openstack-infra | 08:28 | |
| apevec | yeah, also releasing on Sunday evening... | 08:29 |
| zigo | Fun fun fun ... | 08:29 |
| therve | https://bitbucket.org/pypa/setuptools/issue/213/regression-setuptools-37-installation | 08:29 |
| mordred | #status alert setuptools upstream has broken the world. it's a known issue. we're hoping that a solution materializes soon | 08:29 |
| openstackstatus | mordred: sending alert | 08:29 |
| -openstackstatus- NOTICE: setuptools upstream has broken the world. it's a known issue. we're hoping that a solution materializes soon | 08:31 | |
| *** ChanServ changes topic to "setuptools upstream has broken the world. it's a known issue. we're hoping that a solution materializes soon" | 08:31 | |
| zigo | Also, 3 releases in 57 minutes ... | 08:31 |
| *** otherwiseguy has joined #openstack-infra | 08:32 | |
| zigo | mordred: Is there a way to blacklist setuptools >= 3.7 globally? | 08:32 |
| mordred | zigo: nope | 08:32 |
| zigo | :/ | 08:32 |
| hashar | don't you use a manually updated pypi mirror? | 08:32 |
| *** habib has joined #openstack-infra | 08:33 | |
| mordred | zigo: we don't express dependencies on setuptools because it breaks things | 08:33 |
| zigo | Right. | 08:33 |
| mordred | hashar: we don't - and setuptools is, I believe, special anway | 08:33 |
| hashar | :-( | 08:33 |
| openstackstatus | mordred: finished sending alert | 08:33 |
| mordred | we kinda have to exist in a world where setuptools works | 08:33 |
| hashar | that sounds reasonable | 08:33 |
| mordred | well, except for when it breaks ;) | 08:34 |
| zigo | mordred: How about hacking our pypi mirror to not include blacklisted versions? | 08:34 |
| tchaypo | Hack hack hack | 08:34 |
| tchaypo | That sounds nasty | 08:35 |
| mordred | we could - but the thing is that openstack is broken in the wild | 08:35 |
| mordred | like, it's actually broken currently | 08:35 |
| mordred | so we could hack aroudn it - but it will be broken on people's laptops | 08:35 |
| *** salv-orlando has joined #openstack-infra | 08:35 | |
| *** _afazekas is now known as afazekas | 08:36 | |
| mordred | (also, I'm at a conference and don't have tons of attention, so I don't want to start hacking things without others around) | 08:36 |
| *** pblaho has joined #openstack-infra | 08:36 | |
| *** andreaf has joined #openstack-infra | 08:37 | |
| openstackgerrit | Peter Belanyi proposed a change to openstack-infra/config: Add jshint job for tuskar-ui https://review.openstack.org/96473 | 08:39 |
| afazekas | mordred: Isn't simple global-requirements change enough to prevent these issues both on the gate and on the laptops? | 08:41 |
| therve | afazekas, " we don't express dependencies on setuptools because it breaks things" | 08:41 |
| *** otherwiseguy has quit IRC | 08:42 | |
| apevec | yeah, it is not in global-requirements | 08:42 |
| *** fbo has joined #openstack-infra | 08:43 | |
| afazekas | Is one of the local pip.conf (something like that) allows to disable installing a particular version of a package ? | 08:44 |
| *** srenatus has joined #openstack-infra | 08:45 | |
| *** sarob has joined #openstack-infra | 08:47 | |
| *** sarob has quit IRC | 08:52 | |
| *** yfried has joined #openstack-infra | 08:53 | |
| *** salv-orlando has quit IRC | 08:53 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard-webclient: Fixes misaligned comments on very long lines https://review.openstack.org/96128 | 09:01 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard: Added new priority changed event for tasks https://review.openstack.org/96151 | 09:04 |
| *** yfried has quit IRC | 09:06 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added priority on task row https://review.openstack.org/96156 | 09:07 |
| *** dizquierdo has quit IRC | 09:10 | |
| *** IvanBerezovskiy has joined #openstack-infra | 09:11 | |
| *** yolanda has joined #openstack-infra | 09:12 | |
| yolanda | hi, getting this error on a job i try to check, on every test script: TypeError: 'unicode' does not have the buffer interface | 09:13 |
| yolanda | is that something related with the bug you talk about in the status of the chat? | 09:13 |
| tchaypo | yolanda: yep, that's it | 09:14 |
| *** dkranz has joined #openstack-infra | 09:14 | |
| SlickNik | yolanda: Yes, I believe it's related to the setuptools bug that's breaking us. | 09:15 |
| SlickNik | yolanda: https://bugs.launchpad.net/openstack-ci/+bug/1325423 FYI | 09:15 |
| uvirtbot | Launchpad bug 1325423 in openstack-ci "Setuptools 3.7 and newer breaks installation of many packages" [Undecided,New] | 09:15 |
| yolanda | yes, looks the same | 09:15 |
| *** praneshp has quit IRC | 09:16 | |
| *** ociuhandu has joined #openstack-infra | 09:16 | |
| *** skolekonov has joined #openstack-infra | 09:18 | |
| *** salv-orlando has joined #openstack-infra | 09:19 | |
| *** mrda is now known as mrda-away | 09:21 | |
| *** SlickN1k has joined #openstack-infra | 09:34 | |
| *** SlickN1k has quit IRC | 09:34 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add docs-on-rtfd to rally https://review.openstack.org/95523 | 09:35 |
| *** SlickN1k has joined #openstack-infra | 09:35 | |
| *** yfried has joined #openstack-infra | 09:36 | |
| Kiall | Interesting... Upstream setuptools author is doing dev on Windows and Python 3, doesn't seem to be able to test linux / python 2.. I'm surprised it doesn't break more often ;) | 09:36 |
| *** spiffxp_ has joined #openstack-infra | 09:37 | |
| tchaypo | so I hate to throw a spanner into the works, but I'm wondering if all the failures we're seeing are down to setuptools | 09:38 |
| *** otherwiseguy has joined #openstack-infra | 09:38 | |
| tchaypo | but happily I think I'm wrong. | 09:39 |
| tchaypo | http://logs.openstack.org/38/94838/4/check-tripleo/check-tripleo-undercloud-precise/b57ab98/console.html does seem to have failed because "E: Unable to locate package ubuntu-cloud-keyring" | 09:39 |
| tchaypo | but that seems to have been a one off, the other failures are because of setuptools | 09:39 |
| *** yfried_ has joined #openstack-infra | 09:40 | |
| tchaypo | do we have anything to show it happening in packages other than pbr? | 09:40 |
| *** yfried has quit IRC | 09:42 | |
| Kiall | tchaypo: they are, it's a setuptools bug | 09:43 |
| Kiall | https://bitbucket.org/pypa/setuptools/issue/213/regression-setuptools-37-installation | 09:43 |
| *** otherwiseguy has quit IRC | 09:43 | |
| Kiall | oh | 09:43 |
| Kiall | misread | 09:43 |
| Kiall | I've seen that error on tripleo builds recently too | 09:44 |
| tchaypo | yeah, but as far as i can tell they all break installing pbr | 09:44 |
| *** yfried_ has quit IRC | 09:44 | |
| tchaypo | I'd love to find more examples of the breakage to add to the bug | 09:45 |
| tchaypo | so far it's just two packages, pbr and python-daemon | 09:45 |
| srenatus | https://review.openstack.org/#/c/95521/ chef cookbook stuff | 09:46 |
| srenatus | could this be related? | 09:46 |
| srenatus | (I wonder, the tests shouldn't fail...) | 09:46 |
| *** sarob has joined #openstack-infra | 09:47 | |
| Kiall | srenatus: they don't look obviously related, and being ruby I'd expect setuptools isn't involved | 09:47 |
| tchaypo | virtualenvwrapper | 09:48 |
| *** rlandy has joined #openstack-infra | 09:48 | |
| Kiall | Oh, I don't see any reference to that in the logs | 09:50 |
| srenatus | Kiall: true, just all three explode with internet access failing... bundler, too. | 09:51 |
| *** _nadya_ has joined #openstack-infra | 09:51 | |
| *** dizquierdo has joined #openstack-infra | 09:51 | |
| *** sarob has quit IRC | 09:51 | |
| Kiall | So, in the python-daemon setup.py, I get some funky characters in my editor: | 09:54 |
| *** ildikov has quit IRC | 09:54 | |
| Kiall | Just the 1, on a line of it's own in the middle of the file. | 09:54 |
| *** ildikov_ has joined #openstack-infra | 09:54 | |
| Kiall | pbr doesn't have anything similar though.. | 09:55 |
| Kiall | (the again, pbr does all sorts of setup.py magic) | 09:55 |
| *** ildikov_ has quit IRC | 09:56 | |
| *** ildikov_ has joined #openstack-infra | 09:56 | |
| *** Longgeek has quit IRC | 09:57 | |
| *** Longgeek has joined #openstack-infra | 09:58 | |
| tchaypo | I'm hoping to find one of the pypa packages that does it | 09:59 |
| tchaypo | virtualenvwrqpper is as close as I've managed so far :p | 09:59 |
| *** _nadya_ has quit IRC | 09:59 | |
| Kiall | Yea - Installing all packages from global-requirements now, at least, those that don't depend on PBR | 09:59 |
| Kiall | Hopefully another one that he can test with bails | 10:00 |
| *** _nadya_ has joined #openstack-infra | 10:00 | |
| *** mrmartin has joined #openstack-infra | 10:03 | |
| *** ildikov_ has quit IRC | 10:03 | |
| *** talluri has quit IRC | 10:05 | |
| *** talluri has joined #openstack-infra | 10:05 | |
| *** ildikov has joined #openstack-infra | 10:06 | |
| *** matrohon has quit IRC | 10:07 | |
| *** talluri_ has joined #openstack-infra | 10:08 | |
| *** habib has quit IRC | 10:09 | |
| tchaypo | dangnabbit. virtualenvwrapper uses pbr? | 10:09 |
| *** talluri has quit IRC | 10:10 | |
| tchaypo | I don't think it's that though. | 10:10 |
| Kiall | Everything that depends on pbr is failing | 10:10 |
| tchaypo | http://paste.openstack.org/show/82404/ | 10:10 |
| tchaypo | I'd expect to see it getting to the point where it looks at the requirements and tries to install them | 10:11 |
| tchaypo | if it was just a transitive error from the pbr failure | 10:11 |
| Kiall | Yea, pbr get's loaded *really* early, so you won't see it as a normal dep | 10:11 |
| tchaypo | oh | 10:11 |
| tchaypo | sadness. | 10:11 |
| tchaypo | dangnabbit. I don't suppose python-daemon depends on pbr? | 10:11 |
| Kiall | I think MySQL-python just triggered it, not sure, got the same error, but it also failed to find mysql_config | 10:12 |
| Kiall | No, python-daemon does not use PBR | 10:12 |
| Kiall | damn .. MySQL-python passed after install libmysql | 10:13 |
| Kiall | Literally every package in global-requirements.txt that depends on PBR is failing ;) | 10:13 |
| *** talluri_ has quit IRC | 10:13 | |
| Kiall | Everything else is so far succeeding (slowly making my way through the list, every fail means re-downloading all the previous packages ;)) | 10:14 |
| Kiall | Okay - every global-requirements.txt entry bar those that depends on PBR install and work just fine | 10:16 |
| tchaypo | well that certainly suggests another thing we can attack | 10:17 |
| Kiall | Yea, if PBR can work around it... | 10:19 |
| tchaypo | I feel bad about saying "we" now | 10:20 |
| tchaypo | since it's 8:20pm here and I'm close to signing off and leaving this for other people to solve | 10:20 |
| tchaypo | I hate it when people say "we" when they mean "anyone but me" | 10:21 |
| Kiall | lol | 10:21 |
| tchaypo | I'm running through http://freepythontips.wordpress.com/2013/07/30/20-python-libraries-you-cant-live-without/ to see if i can find another package that has a problem | 10:22 |
| tchaypo | but about half of them are external and I cbfed doing the --allow-external --allow-unverified dance | 10:23 |
| *** thomasbiege has joined #openstack-infra | 10:23 | |
| *** salv-orlando has quit IRC | 10:24 | |
| *** talluri has joined #openstack-infra | 10:25 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added angular-momentjs to dependencies https://review.openstack.org/96713 | 10:26 |
| *** salv-orlando has joined #openstack-infra | 10:26 | |
| *** Guest22618 has quit IRC | 10:27 | |
| *** talluri has quit IRC | 10:28 | |
| tchaypo | I give up. | 10:29 |
| *** talluri has joined #openstack-infra | 10:30 | |
| *** e0ne has joined #openstack-infra | 10:35 | |
| Kiall | Interestingly - rebuilding python-daemon's sdist and it installs perfectly | 10:35 |
| Kiall | Actually, pip install'ing the downloaded python-daemon tar.gz works, just not when downloaded from pypi | 10:37 |
| *** thomasbiege has quit IRC | 10:38 | |
| *** yamahata has quit IRC | 10:39 | |
| *** thomasbiege has joined #openstack-infra | 10:39 | |
| openstackgerrit | Antoine Musso proposed a change to stackforge/python-jenkins: Speed up job existence tests by fetching less info https://review.openstack.org/89589 | 10:43 |
| *** sarob has joined #openstack-infra | 10:47 | |
| *** e0ne has quit IRC | 10:51 | |
| openstackgerrit | Stephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue https://review.openstack.org/97196 | 10:52 |
| *** sarob has quit IRC | 10:52 | |
| *** e0ne has joined #openstack-infra | 10:52 | |
| *** jistr has joined #openstack-infra | 10:55 | |
| openstackgerrit | Stephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue https://review.openstack.org/97196 | 10:58 |
| *** viktors has joined #openstack-infra | 11:00 | |
| mrmartin | SergeyLukjanov are you here? | 11:02 |
| *** yfried_ has joined #openstack-infra | 11:14 | |
| *** Longgeek has quit IRC | 11:18 | |
| *** zhiyan is now known as zhiyan_ | 11:19 | |
| *** zhiyan_ is now known as zhiyan | 11:20 | |
| *** zzelle has joined #openstack-infra | 11:23 | |
| *** _nadya_ has quit IRC | 11:25 | |
| *** salv-orlando has quit IRC | 11:25 | |
| *** salv-orlando has joined #openstack-infra | 11:27 | |
| *** thomasbiege has quit IRC | 11:28 | |
| *** sandywalsh has joined #openstack-infra | 11:29 | |
| *** markmc has joined #openstack-infra | 11:32 | |
| *** hashar is now known as hasharLunch | 11:32 | |
| *** Longgeek has joined #openstack-infra | 11:37 | |
| *** otherwiseguy has joined #openstack-infra | 11:39 | |
| *** yamahata has joined #openstack-infra | 11:41 | |
| *** mburned_out is now known as mburned | 11:42 | |
| *** otherwiseguy has quit IRC | 11:43 | |
| *** sarob has joined #openstack-infra | 11:47 | |
| *** e0ne has quit IRC | 11:49 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/infra-specs: Added specification for storyboard story tags https://review.openstack.org/97211 | 11:50 |
| *** Guest22618 has joined #openstack-infra | 11:50 | |
| *** sarob has quit IRC | 11:51 | |
| *** basha has quit IRC | 11:55 | |
| *** yfried_ has quit IRC | 11:56 | |
| *** basha has joined #openstack-infra | 11:56 | |
| *** BAKfr has joined #openstack-infra | 11:58 | |
| *** e0ne_ has joined #openstack-infra | 11:58 | |
| *** basha has quit IRC | 11:59 | |
| *** hasharLunch is now known as hashar | 12:01 | |
| *** ArxCruz has joined #openstack-infra | 12:03 | |
| openstackgerrit | Stephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue https://review.openstack.org/97196 | 12:07 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add docs-on-rtfd to rally https://review.openstack.org/95523 | 12:08 |
| *** pdmars has joined #openstack-infra | 12:09 | |
| *** mrmartin has quit IRC | 12:10 | |
| *** pdmars has quit IRC | 12:11 | |
| *** dims has joined #openstack-infra | 12:12 | |
| *** adalbas has joined #openstack-infra | 12:12 | |
| *** weshay has joined #openstack-infra | 12:13 | |
| *** pdmars has joined #openstack-infra | 12:16 | |
| *** jgallard has joined #openstack-infra | 12:16 | |
| *** sandywalsh has quit IRC | 12:18 | |
| *** matsuhashi has quit IRC | 12:22 | |
| *** ildikov_ has joined #openstack-infra | 12:22 | |
| SergeyLukjanov | mrmartin, I'm here now | 12:22 |
| *** dims has quit IRC | 12:23 | |
| *** ildikov has quit IRC | 12:23 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-dev/cookiecutter: Added a doc build target to tox.ini https://review.openstack.org/92019 | 12:24 |
| *** mbacchi has joined #openstack-infra | 12:26 | |
| *** aysyd has joined #openstack-infra | 12:26 | |
| vponomaryov | SergeyLukjanov: hello, if you have sme time, could you please review this: https://review.openstack.org/#/c/97022/ ? | 12:27 |
| sdague | SergeyLukjanov: any change you could punch the new setuptools out of the mirror? | 12:30 |
| sdague | otherwise I think we're going to be broken until it's fixed | 12:30 |
| SergeyLukjanov | vponomaryov, I'll take a look | 12:30 |
| SergeyLukjanov | sdague, morning | 12:30 |
| *** talluri has quit IRC | 12:30 | |
| *** dims has joined #openstack-infra | 12:30 | |
| SergeyLukjanov | sdague, AFAIK we're not able to limit setuptools | 12:31 |
| SergeyLukjanov | from scrollback | 12:31 |
| sdague | SergeyLukjanov: not in global requirements | 12:31 |
| SergeyLukjanov | [12:32:38] <zigo> mordred: Is there a way to blacklist setuptools >= 3.7 globally? | 12:31 |
| SergeyLukjanov | [12:32:43] <@mordred> zigo: nope | 12:31 |
| *** talluri has joined #openstack-infra | 12:31 | |
| sdague | but I thought we could pull it from the mirror | 12:31 |
| Kiall | sdague: mordred was saying that even if we remove it from our mirror, everyone's laptops etc are still broke | 12:31 |
| *** Alexei_987 has joined #openstack-infra | 12:31 | |
| sdague | Kiall: sure | 12:31 |
| Kiall | (But.. That's better than 100% broke ;)) | 12:31 |
| *** talluri has quit IRC | 12:31 | |
| DinaBelova | Kiall, sdague - but it this case we'll have the gate fixed | 12:31 |
| DinaBelova | yep | 12:31 |
| sdague | but then you don't have the inability for anyone to do code reviews because jenkins will fail everything | 12:32 |
| SergeyLukjanov | sdague, I'm not sure that it will help for enough time, probably even all images will be needed to rebuild | 12:32 |
| *** dims has quit IRC | 12:32 | |
| *** talluri has joined #openstack-infra | 12:32 | |
| sdague | SergeyLukjanov: well, we could do that as well :) | 12:32 |
| sdague | basically everything is currently on fire | 12:32 |
| *** dims has joined #openstack-infra | 12:32 | |
| SergeyLukjanov | oops, I see "/srv/static/docs-draft is using 91.0% of 1.32TB" | 12:33 |
| SergeyLukjanov | fungi, is it ok? ^^ | 12:33 |
| sdague | I actually think if we aren't going to fix it, we should turn off zuul, because right now, it's not doing anything useful, except burning nodes and failing people's tests | 12:33 |
| *** salv-orlando has quit IRC | 12:34 | |
| *** dkliban_afk is now known as dkliban | 12:34 | |
| sdague | DinaBelova: is there an openstack-ci bug for this? | 12:34 |
| sdague | so I can create an elastic recheck query on it | 12:34 |
| SergeyLukjanov | sdague, is setuptools 4.0 work ok? | 12:35 |
| sdague | SergeyLukjanov: no idea | 12:36 |
| sdague | I'm just trying to avoid a giant amount of recheck in the gate | 12:36 |
| SergeyLukjanov | sdague, I can remove >=3.7 from the mirror | 12:36 |
| sdague | SergeyLukjanov: I think you are also right that you need to kick off the nodepool rebuild as well | 12:36 |
| *** talluri has quit IRC | 12:36 | |
| SergeyLukjanov | sdague, the only issue is that I'm not sure how to do it safe | 12:37 |
| srenatus | If anyone could have a look at this before I'm publicly ridiculed in a hangout this afternoon, I'd be glad... https://review.openstack.org/#/c/97196/ | 12:37 |
| sdague | SergeyLukjanov: gotcha, well fungi should be up soon I think | 12:37 |
| DinaBelova | sdague, there was bug first created for the tempest - https://bugs.launchpad.net/ceilometer/+bug/1325514 | 12:37 |
| uvirtbot | Launchpad bug 1325514 in tempest "'unicode' does not have the buffer interface" [Undecided,New] | 12:37 |
| DinaBelova | after that folks have added other projects to it too | 12:37 |
| sdague | DinaBelova: ok, I'm going to remove all the other projects except openstack-ci | 12:38 |
| DinaBelova | sdague, ok | 12:38 |
| *** salv-orlando has joined #openstack-infra | 12:38 | |
| *** maxbit has joined #openstack-infra | 12:39 | |
| *** habib has joined #openstack-infra | 12:39 | |
| SergeyLukjanov | done: sudo rm -rf setuptools-4.0* | 12:39 |
| SergeyLukjanov | the latest available setuptools in mirror is 3.6 | 12:40 |
| DinaBelova | hehe, let's try to recheck no bug :D | 12:40 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: added bug for setuptools 3.7 breaking the world https://review.openstack.org/97221 | 12:40 |
| sdague | sweet, I can't get an er bug through, because... setuptools | 12:41 |
| *** AJaeger has joined #openstack-infra | 12:42 | |
| *** AJaeger has joined #openstack-infra | 12:42 | |
| mtreinish | sdague: nice | 12:42 |
| DinaBelova | sdague, probably recheck will help? as SergeyLukjanov removed the 3.7+ from the mirror | 12:43 |
| DinaBelova | just after you proposed the error I guess | 12:43 |
| *** radez_g0n3 is now known as radez | 12:43 | |
| YorikSar | I see first successful jobs for Neutron. | 12:44 |
| *** mwagner_lap has quit IRC | 12:44 | |
| YorikSar | You can start recheck flood ;) | 12:44 |
| samuelmz_ | AJaeger, ping | 12:45 |
| AJaeger | Hi samuelmz_ | 12:45 |
| *** rfolco has joined #openstack-infra | 12:45 | |
| samuelmz_ | AJaeger, hi :) | 12:45 |
| samuelmz_ | AJaeger, https://review.openstack.org/#/c/97219/ | 12:45 |
| samuelmz_ | AJaeger, please take a look at this | 12:45 |
| samuelmz_ | AJaeger, and thanks for having confirmed this bug | 12:45 |
| DinaBelova | sdague, it looks like SergeyLukjanov's fix works | 12:46 |
| AJaeger | Sam-I-Am, will do later. Thakns for the patch! | 12:46 |
| AJaeger | samuelmz_, I need to check whether we need to add the information elsewhere... | 12:46 |
| SergeyLukjanov | jeblair, mordred, fungi, clarkb, I've removed setuptools >= 3.7 from mirror to tmp fix this issue | 12:47 |
| *** sarob has joined #openstack-infra | 12:47 | |
| *** rlandy_ has joined #openstack-infra | 12:47 | |
| sdague | DinaBelova: not for me :( | 12:47 |
| salv-orlando | extra day off for everybody! | 12:48 |
| YorikSar | sdague: Which change request failed? | 12:48 |
| DinaBelova | YorikSar, https://review.openstack.org/#/c/97221/1 | 12:48 |
| apevec | DinaBelova, sdague - there was setuptools bug reported earlier https://bugs.launchpad.net/openstack-ci/+bug/1325423 | 12:48 |
| uvirtbot | Launchpad bug 1325423 in openstack-ci "Setuptools 3.7 and newer breaks installation of many packages" [Undecided,New] | 12:48 |
| samuelmz_ | AJaeger, yes, could be at 'Install the clients > Installing from packages' | 12:48 |
| DinaBelova | apevec, oh, sorry - did not find it earlier | 12:48 |
| samuelmz_ | AJaeger, http://docs.openstack.org/icehouse/install-guide/install/apt/content/install_clients.html | 12:48 |
| *** dprince has joined #openstack-infra | 12:49 | |
| AJaeger | samuelmz_, I'll check later tonight - thanks | 12:49 |
| samuelmz_ | AJaeger, ok, thanks | 12:49 |
| *** rlandy has quit IRC | 12:49 | |
| YorikSar | Hm... I guess not everybody uses pypi mirror... | 12:50 |
| SergeyLukjanov | only projects defined in os/requirements/projects.txt could be fixe | 12:51 |
| SergeyLukjanov | sdague ^^ | 12:51 |
| *** russellb has joined #openstack-infra | 12:51 | |
| *** sarob has quit IRC | 12:51 | |
| sdague | SergeyLukjanov: gotcha | 12:53 |
| sdague | is there a different mirror for infra jobs? I thought we controlled those as well | 12:53 |
| *** doug-fish has joined #openstack-infra | 12:54 | |
| dhellmann | good morning, all | 12:55 |
| anteaya | jeblair: regarding the gertty tracebacks, I was browsing into a patchset and looking at diffs | 12:56 |
| anteaya | jeblair: I had entered no comments for review | 12:56 |
| *** rlandy__ has joined #openstack-infra | 12:57 | |
| SergeyLukjanov | dhellmann, morning | 12:57 |
| *** ihrachyshka has quit IRC | 12:57 | |
| anteaya | jesusaurus: okay I will ask about puppet-livegrep, so far I have gotten crickets on it so hopefully this week folks will be willing to weigh in | 12:57 |
| SergeyLukjanov | sdague, no, we remove this code when migrated to the projects.txt mirror chooser AFAIK | 12:57 |
| *** bauzas has joined #openstack-infra | 12:57 | |
| openstackgerrit | Ana Krivokapic proposed a change to openstack/requirements: Add Pint for Horizon https://review.openstack.org/97224 | 12:57 |
| fungi | looking back over the setuptools discussion now | 12:57 |
| sdague | SergeyLukjanov: gotach | 12:57 |
| SergeyLukjanov | fungi, short info | 12:57 |
| anteaya | trinaths: have you checked the logs on your server like I asked you too? | 12:58 |
| SergeyLukjanov | fungi, I've removed bad version from the mirror, it fix projects defined in projects.txt | 12:58 |
| anteaya | trinaths: did you find out why the server only renders pages sometimes and not consistently? | 12:58 |
| SergeyLukjanov | fungi, it works and we see some green jobs now in gate | 12:58 |
| SergeyLukjanov | fungi, non-related question, how can I force push something? :) | 12:58 |
| bauzas | SergeyLukjanov: thanks for your work :) | 12:58 |
| anteaya | morning dhellmann | 12:58 |
| *** rlandy_ has quit IRC | 12:59 | |
| fungi | SergeyLukjanov: you have to add your gerrit account temporarily to https://review.openstack.org/#/admin/groups/26,members and then when you review a change, delete any verified -2 or workflow -1 votes, add verified +2 in addition to your workflow +1, and then hit the "publish and submit" button | 13:00 |
| fungi | oh, and make sure there's a code review +2 vote as well | 13:01 |
| SergeyLukjanov | fungi, okay, thx, It could be needed someday | 13:01 |
| *** rdopiera has joined #openstack-infra | 13:01 | |
| fungi | and then remove yourself from Project Bootstrappers immediately once you're done, to avoid any unpleasant mistakes | 13:01 |
| fungi | i also usually make sure i leave a review comment with my approval when i do that explaining the reason i had to bypass all gating | 13:02 |
| SergeyLukjanov | fungi, sure, that makes sense | 13:02 |
| SergeyLukjanov | fungi, will it auto reset related chain in gate? | 13:03 |
| fungi | SergeyLukjanov: it won't, that change will simply fail to merge since it's already merged | 13:03 |
| *** miqui has joined #openstack-infra | 13:03 | |
| YorikSar | fungi: btw, wouldn't Cerdic's --submit-immediately be useful in this case? | 13:03 |
| fungi | SergeyLukjanov: but it's assumed that basically everything in the gate is doing that anyway if you're stuck bypassing tests completely | 13:03 |
| SergeyLukjanov | fungi, it works too, I mean that force push will not break everything | 13:04 |
| *** yfried_ has joined #openstack-infra | 13:04 | |
| fungi | YorikSar: possibly, except that we have to grant ourselves permission to use it, and by the time we've done that having a special option to git-review to skip pressing the submit button isn't really much of a time saver | 13:05 |
| fungi | YorikSar: we don't leave any of our accounts with the necessary permissions to use that under normal circumstances | 13:05 |
| YorikSar | fungi: Oh, ok. | 13:06 |
| fungi | SergeyLukjanov: well, when you say force push i assume you're talking about merging a change directly from gerrit bypassing all gating, not actually git push --force (which is bad, bad, bad) | 13:06 |
| YorikSar | fungi: How do you add yourself to a group? Is it in UI? | 13:06 |
| fungi | YorikSar: i believe it can be done via an api call in the new version of gerrit, but i'm used to using the webui for that since it was the only way in 2.4 and we've not been on 2.9 long at all | 13:07 |
| YorikSar | fungi: Although even if it's done not from UI automation would do more harm than good here, I think. | 13:07 |
| fungi | s/2.9/s.8/ | 13:07 |
| fungi | something. moar coffee | 13:07 |
| *** ArxCruz has quit IRC | 13:07 | |
| fungi | YorikSar: yes, at least for openstack infra, we make this sort of exception extremely deliberate, so having a few additional steps to prevent you from doing it accidentally is more of a feature than a bug | 13:08 |
| sdague | on the upside - https://github.com/jaraco/setuptools/pull/7 | 13:08 |
| sdague | so the fix was merged into upstream 13 minutes ago | 13:08 |
| *** julim has joined #openstack-infra | 13:08 | |
| sdague | hopefully we'll get a new release shortly | 13:08 |
| fungi | sdague: awesome! i was about to ask how, exactly, new setuptools was breaking us | 13:08 |
| sdague | fungi: basically, doesn't look like it was tested well on python2 | 13:08 |
| * dhellmann is confused to see a github pull request for a bug filed on bitbucket | 13:09 | |
| sdague | dhellmann: yeh, no idea | 13:09 |
| fungi | sdague: srsly... nobody uses python 2 these days. we should have been pure py3k so long ago, totally our fault ;) | 13:09 |
| *** julim has quit IRC | 13:09 | |
| *** mriedem has joined #openstack-infra | 13:09 | |
| dhellmann | have we already talked about helping pypa set up testing? | 13:09 |
| YorikSar | fungi: How about we migrate all infra stuff to 3.4? | 13:10 |
| dhellmann | as in, using stackforge? | 13:10 |
| sdague | well, from the thread it looks like the maintainer is basically on windows and python3. And the changelog seems to indicate python2 breaks a couple times in the last year on releases | 13:10 |
| *** yamahata has quit IRC | 13:10 | |
| *** ihrachyshka has joined #openstack-infra | 13:10 | |
| fungi | dhellmann: yes, i believe dstufft is wanting to run his own copies (understandably) and has been working on getting some of it running recently | 13:10 |
| *** ArxCruz has joined #openstack-infra | 13:10 | |
| dhellmann | fungi: cool | 13:10 |
| *** samuelmz_ is now known as samuelmz | 13:10 | |
| *** yamahata has joined #openstack-infra | 13:10 | |
| fungi | if the current setuptools maintainer develops entirely in python 3 on windows, that actually explains a lot ;) | 13:11 |
| SergeyLukjanov | fungi, sure, nothing about git push -f, I'm never doing it :) | 13:11 |
| dhellmann | fungi: indeed | 13:11 |
| dstufft | I've messed around with it a little bit, but haven't spent a great deal of time on it yet | 13:11 |
| *** smarcet has joined #openstack-infra | 13:11 | |
| dstufft | setuptools does use travis to test itself as well | 13:12 |
| Longgeek | fungi: Good morning | 13:12 |
| fungi | dstufft: just not in a way that would have caught https://bitbucket.org/pypa/setuptools/issue/213 i guess | 13:12 |
| dstufft | yea well | 13:12 |
| dstufft | I think setuptools in the same boat as pip there, where a lot of stuff isn't tested and/or is hard to test | 13:13 |
| dhellmann | dstufft: If there's an existing test suite, would you be interested in patches that tried installing common packages? | 13:13 |
| fungi | Longgeek: good morning! (or good night where you live maybe?) | 13:13 |
| *** trinaths has quit IRC | 13:13 | |
| *** mbacchi has quit IRC | 13:13 | |
| *** eharney has quit IRC | 13:13 | |
| Longgeek | fungi, yes, night. | 13:14 |
| dstufft | dhellmann: to which project? | 13:14 |
| dhellmann | dstufft: setuptools, I guess? | 13:14 |
| dstufft | dhellmann: so jaraco is the person to aska bout that | 13:15 |
| dstufft | he's the setuptools maintainer | 13:15 |
| dhellmann | dstufft: ok | 13:15 |
| Longgeek | fungi: Upstream of the ci system is mainly used openstack_projects this puppet modules is not it? | 13:15 |
| dstufft | dhellmann: hes in #pypa-dev fwiw | 13:15 |
| dstufft | dhellmann: also FWIW, one of the things I want to do is test a bunch of common/known good patches in a seperate test suite, eventually | 13:15 |
| dstufft | once I get around to setting up our CI | 13:16 |
| dhellmann | dstufft: that sounds good, too | 13:16 |
| Longgeek | fungi: I found that although the upstream jenkins home directory is /home/jenkins, but many places are defined /var/lib/jenkins | 13:17 |
| fungi | Longgeek: if i understand your question correctly, http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project is the module we use to configure our servers for the openstack developer and community infrastructure. that module uses some other modules in http://git.openstack.org/cgit/openstack-infra/config/tree/ and some modules from puppetforge | 13:17 |
| dstufft | it's the common not enough time problem :[ | 13:17 |
| *** nosnos has quit IRC | 13:18 | |
| fungi | Longgeek: so the other modules in http://git.openstack.org/cgit/openstack-infra/config/tree/ are designed to be possibly used for more generic system configuration, for example to set up a gerrit server or a jenkins master the way gerrit or jenkins developers might expect rather than the way openstack infra needs to run its copies | 13:18 |
| *** lcostantino has joined #openstack-infra | 13:18 | |
| rdopiera | sdague: hey, do you want to talk about that xstatic package? | 13:19 |
| fungi | SergeyLukjanov: are we ready to sound the all-clear signal with statusbot now? | 13:20 |
| SergeyLukjanov | fungi, stackforge/* projects aren't working still | 13:21 |
| SergeyLukjanov | fungi, openstack/* projects should work ok | 13:21 |
| fungi | SergeyLukjanov: ahh, okay it's just fixed on our mirror for now. got it | 13:21 |
| SergeyLukjanov | we could add it to the notice | 13:21 |
| *** dripton has joined #openstack-infra | 13:21 | |
| SergeyLukjanov | I can try to make my first notice ;) | 13:21 |
| fungi | SergeyLukjanov: so we should probably update the topic to include a link to the setuptools bug for people wanting to track progress | 13:21 |
| dstufft | There is a 3.7.1 | 13:21 |
| dstufft | and a 4.0.1 | 13:22 |
| dstufft | that is supposed to fix pbr | 13:22 |
| fungi | wow--fast! | 13:22 |
| SergeyLukjanov | yay! | 13:22 |
| fungi | i guess let's wait a few minutes to see if stackforge tests recover, and then we can send an ok message to all channels with a bug number devs can recheck against | 13:22 |
| SergeyLukjanov | fungi, +1 | 13:23 |
| SergeyLukjanov | fungi, do you have a correct bug id? | 13:23 |
| SergeyLukjanov | https://bugs.launchpad.net/openstack-ci/+bug/1325423 looks marked duplicated | 13:23 |
| fungi | i haven't started looking yet, though i suspect Alex_Gaynor opened one | 13:23 |
| uvirtbot | Launchpad bug 1325423 in openstack-ci "Setuptools 3.7 and newer breaks installation of many packages (dup-of: 1325514)" [Undecided,New] | 13:23 |
| uvirtbot | Launchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New] | 13:23 |
| sdague | rdopiera: sure | 13:24 |
| SergeyLukjanov | sdague, Alex_Gaynor, what's the correct bug id for setuptools issue? | 13:24 |
| SergeyLukjanov | for rechecks | 13:24 |
| sdague | though I think I left my concerns pretty clear in the review | 13:24 |
| sdague | SergeyLukjanov: I duped it into 1325514 | 13:25 |
| fungi | SergeyLukjanov: looks like we've settled on 1325514 | 13:25 |
| *** vkdrao has quit IRC | 13:25 | |
| fungi | yeah, what sdague said | 13:25 |
| sdague | because that's the ER | 13:25 |
| rdopiera | sdague: I'm not sure what is the required action then | 13:25 |
| SergeyLukjanov | sdague, okay | 13:25 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-dsvm-netron job to rally job group https://review.openstack.org/97234 | 13:25 |
| rdopiera | sdague: just forget about the whole horizon split thing? | 13:25 |
| *** homeless has joined #openstack-infra | 13:25 | |
| sdague | rdopiera: so if there is a bug in that pypi package, who's going to fix it? | 13:25 |
| sdague | code that hasn't released since 2011 is very concerning | 13:26 |
| rdopiera | sdague: it has been updated as recently as a month ago | 13:26 |
| sdague | rdopiera: https://pypi.python.org/pypi/XStatic/0.0.2 | 13:26 |
| sdague | that's what I found when I went looking | 13:27 |
| rdopiera | sdague: it is being actively used by MoinMoin 2 wiki project, and maintained by the project lead | 13:27 |
| rdopiera | sdague: yeah, that's because the changes were only in the docs, because that project contains almost no code | 13:27 |
| rdopiera | sdague: it's a "done" project, it won't grow more features | 13:27 |
| sdague | rdopiera: then why is it self proclaimed as alpha? | 13:28 |
| sdague | those things don't add up | 13:28 |
| SergeyLukjanov | fungi, notice message draft: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on https://bugs.launchpad.net/openstack-ci/+bug/1325514 | 13:29 |
| uvirtbot | Launchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New] | 13:29 |
| rdopiera | sdague: he just set it as alpha at the beginning and never changed it | 13:29 |
| fungi | SergeyLukjanov: i'd just use the bug number rather than the url (uvirtbot will happily provide people with the url right after that anyway) | 13:30 |
| SergeyLukjanov | fungi, ok | 13:30 |
| sdague | rdopiera: well that also doesn't speak well to it being well maintained | 13:30 |
| fungi | "... recheck on bug 1325514" | 13:30 |
| *** bknudson has joined #openstack-infra | 13:30 | |
| uvirtbot | Launchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New] https://launchpad.net/bugs/1325514 | 13:30 |
| SergeyLukjanov | fungi, notice message draft: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on 1325514 | 13:30 |
| sdague | when we specify a global requirement we are basically telling all distros everywhere that they need to package this code and support it | 13:30 |
| therve | dstufft, Is 4.0.1 published somewhere? | 13:30 |
| sdague | and telling them that's the case with something self marked as alpha is not the business we want to be in :( | 13:31 |
| SergeyLukjanov | fungi, notice message draft, correction: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug https://bugs.launchpad.net/openstack-ci/+bug/1325514 | 13:31 |
| uvirtbot | Launchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New] | 13:31 |
| *** jcoufal has quit IRC | 13:31 | |
| sdague | so if the maintainer is active ask them to cut a 1.0, change the maturity flags, then that would be different | 13:31 |
| fungi | SergeyLukjanov: looks great | 13:31 |
| rdopiera | sdague: I already did | 13:31 |
| *** maxbit has quit IRC | 13:31 | |
| dstufft | therve: pypi | 13:31 |
| rdopiera | sdague: thanks | 13:32 |
| SergeyLukjanov | fungi, ok, so, waiting for confirmation from stackforge projects | 13:32 |
| fungi | therve: https://pypi.python.org/packages/source/s/setuptools/setuptools-4.0.1.tar.gz#md5=190b1d4470de9bae0b4414353e14700d is linked from https://pypi.python.org/simple/setuptools/ | 13:32 |
| therve | Ah just needed some hardcore refresh | 13:33 |
| fungi | therve: also a wheel and a zip | 13:33 |
| mordred | fungi: ooh. upstream fixed? | 13:33 |
| *** mrmartin has joined #openstack-infra | 13:33 | |
| fungi | mordred: seems that way | 13:33 |
| fungi | time will tell | 13:33 |
| *** bogdando has quit IRC | 13:34 | |
| mordred | woot | 13:34 |
| SergeyLukjanov | mordred, status ok notice is ready too :) waiting for the changes in overloaded check queue | 13:34 |
| gilliard | hi -infra. Just checking in to see if nodepool's still having trouble booting on HP cloud? | 13:35 |
| gilliard | I added a lot more info to bug 1324934 | 13:35 |
| uvirtbot | Launchpad bug 1324934 in nova "Neutron port leak when connection is dropped during port create in instance boot." [Undecided,In progress] https://launchpad.net/bugs/1324934 | 13:35 |
| mordred | dstufft: I've been meaning to get stuff in our CI systems to let you guys know if/when your changes are going to break us too... | 13:36 |
| mordred | dstufft: but also, you know, ENOTIME | 13:36 |
| dstufft | mordred: heh | 13:36 |
| dstufft | yea | 13:36 |
| dstufft | I need a minion | 13:36 |
| dstufft | or two | 13:36 |
| therve | Confirmed working for me | 13:36 |
| dstufft | so much stuff is broken | 13:36 |
| dstufft | :( | 13:36 |
| *** markmc has quit IRC | 13:37 | |
| dstufft | and it's like, well do I work on some CI, or do I work on things that'll clean up some of the other brokeness | 13:37 |
| mordred | yup | 13:38 |
| *** pcrews has joined #openstack-infra | 13:42 | |
| *** rlandy_ has joined #openstack-infra | 13:42 | |
| *** signed8bit has joined #openstack-infra | 13:42 | |
| fungi | gilliard: i just reran the script looking for disassociated neutron ports and only turned up 5 since friday, so at least that particular problem seems to be fairly infrequent (though our weekend volume is also much lower than weekdays) | 13:44 |
| *** wenlock has joined #openstack-infra | 13:44 | |
| *** rlandy__ has quit IRC | 13:45 | |
| *** rlandy__ has joined #openstack-infra | 13:46 | |
| *** sarob has joined #openstack-infra | 13:47 | |
| *** mbacchi has joined #openstack-infra | 13:48 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/devstack-gate: Allow to configure git base URL https://review.openstack.org/95901 | 13:48 |
| *** signed8bit has quit IRC | 13:48 | |
| gilliard | fungi: that's good. We haven't done anything to mitigate it yet. So instances are booting OK? | 13:49 |
| fungi | gilliard: i'm looking at some stats now to see what the overall situation is there | 13:49 |
| *** rlandy_ has quit IRC | 13:49 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Add md5 checksum to groups tarball publisher https://review.openstack.org/97238 | 13:50 |
| fungi | gilliard: looks like we are still having some percentage of nodes booting into an error state in nova | 13:50 |
| fungi | gilliard: | 13:50 |
| fungi | http://graphite.openstack.org/render/?from=-8hours&title=HP%20Cloud%201.1%20East&height=495&width=885&_salt=1400700116.702&target=color(alias(summarize(sumSeries(stats_counts.nodepool.launch.provider.hpcloud-b*.ready)%2C%20%221h%22)%2C%20%27Ready%27)%2C%20%2700ff22%27)&target=color(alias(summarize(sumSeries(stats_counts.nodepool.launch.provider.hpcloud-b*.error.*)%2C%20%221h%22)%2C%20%27Error%27)%2C% | 13:50 |
| fungi | 20%27ff0000%27)&target=sumSeries(stats.gauges.nodepool.target.*.devstack-precise.hpcloud-b*.building)&target=sumSeries(stats.gauges.nodepool.target.*.devstack-precise.hpcloud-b*.delete)&target=sumSeries(stats.gauges.nodepool.target.*.devstack-precise.hpcloud-b*.ready)&target=sumSeries(stats.gauges.nodepool.target.*.devstack-precise.hpcloud-b*.used) | 13:51 |
| fungi | eek huge url | 13:51 |
| mrmartin | SergeyLukjanov: hi, what I need to do restart gating check of this failed patch: https://review.openstack.org/#/c/96421/ ? | 13:51 |
| *** sarob has quit IRC | 13:51 | |
| gilliard | fungi: OK thanks I will see what more I can see. | 13:52 |
| gilliard | it's not going to be "port allocation quota exceeded" any more I guess. | 13:52 |
| *** zz_gondoi is now known as gondoi | 13:53 | |
| fungi | mrmartin: looks like the projects alphabetization check failed to retrieve a needed package from pypi.python.org (probably due to a transient network error somewhere on the internet). i'll reapprove it | 13:53 |
| mrmartin | fungi: thanx | 13:53 |
| fungi | mrmartin: it's rechecking and will head into the gate shortly if there are no new problems with it | 13:54 |
| *** sballe has joined #openstack-infra | 13:55 | |
| *** sballe has quit IRC | 13:56 | |
| *** signed8bit has joined #openstack-infra | 13:58 | |
| SergeyLukjanov | mrmartin, /me was afk, fungi thx for help | 13:58 |
| *** jcoufal has joined #openstack-infra | 14:00 | |
| openstackgerrit | Longgeek proposed a change to openstack-infra/config: Modify jenkins home directory https://review.openstack.org/97240 | 14:00 |
| *** mfer has joined #openstack-infra | 14:00 | |
| *** wenlock has quit IRC | 14:01 | |
| *** wenlock has joined #openstack-infra | 14:01 | |
| *** mrmartin has quit IRC | 14:02 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Generate sha1 and md5 sum file for wheels https://review.openstack.org/97142 | 14:03 |
| anteaya | Longgeek: why do you want to modify the jenkins home directory? | 14:03 |
| anteaya | Longgeek: and as I said Friday, we tend to discuss in channel whether or not something actually is a bug before creating a report | 14:04 |
| Longgeek | anteaya: Why jenkins :: master the use of the /var/lib/jenkins? | 14:04 |
| anteaya | that is a good question | 14:04 |
| *** mrmartin has joined #openstack-infra | 14:05 | |
| anteaya | in future please start by asking a question, just like the question you just asked | 14:05 |
| anteaya | saves us a lot of time | 14:05 |
| Longgeek | ok | 14:05 |
| anteaya | thank you | 14:05 |
| *** jaypipes has joined #openstack-infra | 14:05 | |
| anteaya | hey jaypipes | 14:06 |
| anteaya | do you have much on your agenda today? | 14:06 |
| anteaya | I could use some of your time before our third-party meeting today | 14:06 |
| Longgeek | jenkins::master use /var/lib/jenkins, but jenkins::jenkinsuser use /home/jenkins | 14:06 |
| ruhe | AFAIK /var/lib/jenkins is a default home for the jenkins user | 14:06 |
| jaypipes | anteaya: unfortunately yes :( | 14:06 |
| anteaya | jaypipes: okay, go you | 14:06 |
| anteaya | Longgeek: yes, you have asked the question, well done | 14:06 |
| anteaya | Longgeek: now we wait | 14:07 |
| *** julim has joined #openstack-infra | 14:07 | |
| anteaya | Longgeek: we wait for someone who knows the answer to read your question and respond | 14:07 |
| anteaya | Longgeek: that person is not me | 14:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't delete tarballs in run-wheel.sh https://review.openstack.org/97141 | 14:07 |
| Longgeek | anteaya: ok | 14:07 |
| anteaya | Longgeek: and we don't continually ask the same question, because that gets annoying | 14:07 |
| anteaya | Longgeek: thanks | 14:07 |
| Longgeek | anteaya: ok, I will not ask. | 14:08 |
| anteaya | Longgeek: ask by all means, but ask once | 14:08 |
| *** BobBall has quit IRC | 14:08 | |
| *** wenlock has quit IRC | 14:09 | |
| *** jgallard has quit IRC | 14:09 | |
| *** jgallard has joined #openstack-infra | 14:10 | |
| ruhe | SergeyLukjanov: fungi: fyi i see at least two stackforge projects with green py27/pep8 jobs: http://logs.openstack.org/06/97206/1/check/gate-solum-python27/9efea4c/ and https://jenkins03.openstack.org/job/gate-murano-pep8/14/console | 14:12 |
| *** afazekas has quit IRC | 14:12 | |
| ruhe | SergeyLukjanov: fungi: also local run of "tox -e py27" also was successful | 14:12 |
| SergeyLukjanov | ruhe, nice, thanks for the update | 14:12 |
| SergeyLukjanov | fungi, time for notice? | 14:12 |
| *** mriedem has quit IRC | 14:12 | |
| SergeyLukjanov | fungi: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug 1325514 | 14:13 |
| uvirtbot | Launchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New] https://launchpad.net/bugs/1325514 | 14:13 |
| openstackgerrit | A change was merged to openstack-infra/config: Generate sha1 and md5 sum file for wheels https://review.openstack.org/97142 | 14:13 |
| *** basha has joined #openstack-infra | 14:13 | |
| fungi | SergeyLukjanov: that's cool with me | 14:13 |
| SergeyLukjanov | heh, bracking statusbot | 14:13 |
| SergeyLukjanov | breaking* | 14:13 |
| SergeyLukjanov | #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug 1325514 | 14:14 |
| openstackstatus | SergeyLukjanov: sending ok | 14:14 |
| *** mriedem has joined #openstack-infra | 14:14 | |
| *** sballe has joined #openstack-infra | 14:15 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add docs-on-rtfd to rally https://review.openstack.org/95523 | 14:15 |
| *** mwagner_lap has joined #openstack-infra | 14:15 | |
| *** sarob has joined #openstack-infra | 14:15 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/ | https://git.openstack.org/cgit/openstack-infra/config/tree/" | 14:15 | |
| -openstackstatus- NOTICE: setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug 1325514 | 14:15 | |
| *** sarob_ has joined #openstack-infra | 14:16 | |
| fungi | Longgeek: part of the problem is that there are two different places the jenkins account gets used. the jenkins service running on the master and the jenkins slave agent running on slaves. it seemed reasonable to let the slaves have the jenkins user's homedir in /home but the server tends to do a lot of things under its account on a master which fits better in /var/lib | 14:16 |
| *** BobBall has joined #openstack-infra | 14:17 | |
| *** primemin1sterp has joined #openstack-infra | 14:17 | |
| fungi | Longgeek: so we set the default to /home/jenkins in the jenkinsuser class, but then override it to /var/lib/jenkins in the master class | 14:17 |
| *** mgagne has joined #openstack-infra | 14:17 | |
| openstackstatus | SergeyLukjanov: finished sending ok | 14:17 |
| fungi | Longgeek: the correctness of those assumptions can certainly be debated, and regardless i think a comment in the master class manifest explaining that would probably be a good idea | 14:18 |
| *** pdmars has quit IRC | 14:18 | |
| *** primemin1sterp has quit IRC | 14:19 | |
| *** blamar has joined #openstack-infra | 14:19 | |
| *** pdmars has joined #openstack-infra | 14:19 | |
| *** _nadya_ has joined #openstack-infra | 14:19 | |
| *** primeministerp has quit IRC | 14:19 | |
| *** sarob has quit IRC | 14:19 | |
| *** primeministerp has joined #openstack-infra | 14:19 | |
| fungi | Longgeek: i'll update the bug with some additional details | 14:19 |
| SergeyLukjanov | fungi, hm, that's strange, but there is no openstackstatus bot in sahara channel :( | 14:20 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add #openstack-sahara to status bot https://review.openstack.org/97247 | 14:20 |
| SergeyLukjanov | fungi, please, take a look on it ^^ | 14:20 |
| ruhe | were there any changes which would hide devstacklog.txt.gz from job results? | 14:20 |
| Longgeek | fungi: ok | 14:20 |
| *** sarob_ has quit IRC | 14:20 | |
| ruhe | here is an example: http://logs.openstack.org/57/94657/3/check/gate-murano-devstack-dsvm/4f891e5/logs/ | 14:20 |
| fungi | ruhe: what do you mean hide from job results? | 14:20 |
| fungi | looking | 14:20 |
| anteaya | sdague: I gave http://goo.gl/dblfeg a whirl, since i am not core, I have interesting results in the -2 and +2 sections | 14:21 |
| ruhe | here is results for the same patch http://logs.openstack.org/57/94657/1/check/gate-murano-devstack-dsvm/5e81f0a/logs/ . it has devstacklog.txt.gz | 14:21 |
| ruhe | this one is ~1 week earlier | 14:21 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Run tox by stack user in rally-cli job https://review.openstack.org/97249 | 14:23 |
| anteaya | sdague: I got things I reviewed that anybody -2'd or +2'd | 14:23 |
| fungi | SergeyLukjanov: i think you rebased 97247 incorrectly or something | 14:24 |
| *** Longgeek has quit IRC | 14:24 | |
| *** Longgeek has joined #openstack-infra | 14:25 | |
| SergeyLukjanov | fungi, today isn't my day | 14:25 |
| *** Longgeek has quit IRC | 14:25 | |
| *** otherwiseguy has joined #openstack-infra | 14:25 | |
| *** Longgeek has joined #openstack-infra | 14:25 | |
| *** Longgeek has quit IRC | 14:25 | |
| *** Longgeek has joined #openstack-infra | 14:25 | |
| ruhe | fungi: btw it seems like all devstack jobs miss actual devstack logs. here is another example http://logs.openstack.org/22/93322/1/check/check-tempest-dsvm-full/6c58992/logs/ | 14:25 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add #openstack-sahara to status bot https://review.openstack.org/97247 | 14:26 |
| SergeyLukjanov | fungi, ^^ | 14:26 |
| SergeyLukjanov | fungi, fixed one ;) | 14:26 |
| sdague | anteaya: firefox? | 14:27 |
| *** afazekas has joined #openstack-infra | 14:27 | |
| fungi | ruhe: i wonder if something changed in the cleanup function which is causing it to fail to copy the logs... looking at change history in devstack-gate now for clues | 14:27 |
| sdague | anteaya: can you jump to -qa, we're discussing there | 14:27 |
| *** jgrimm has joined #openstack-infra | 14:27 | |
| ruhe | fungi: thank you! | 14:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix translation setup for api-site https://review.openstack.org/97085 | 14:28 |
| *** beekneemech is now known as bnemec | 14:28 | |
| fungi | ruhe: my guess is a regression introduced by http://git.openstack.org/cgit/openstack-infra/devstack-gate/commit/?id=56bf367 | 14:28 |
| fungi | ruhe: er, that was the merge. actual commit was http://git.openstack.org/cgit/openstack-infra/devstack-gate/commit/?id=bd45571 | 14:29 |
| fungi | sdague: ianw: ^ | 14:29 |
| fungi | ruhe: yep, that change dropped the 'sudo cp $BASE/new/devstacklog.txt $NEWLOGTARGET/' and none of what it added accounts for it either | 14:31 |
| openstackgerrit | Tim Kelsey proposed a change to openstack-infra/gerritbot: Adding the a configuration option for gerritbots PID file. https://review.openstack.org/97250 | 14:31 |
| fungi | i think just readding that one line ought to solve it... doing now | 14:31 |
| *** olaph has joined #openstack-infra | 14:32 | |
| phschwartz | morning infra | 14:33 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Copy devstacklog.txt https://review.openstack.org/97251 | 14:33 |
| fungi | ruhe: ianw: sdague: ^ | 14:33 |
| anteaya | morning phschwartz | 14:34 |
| ruhe | fungi: woohoo! thank you, you alwasy fix things :) | 14:34 |
| fungi | ruhe: any day i fix more things than i break is a good day, in my opinion | 14:35 |
| anteaya | fungi: have you had time to break anything today? | 14:36 |
| anteaya | I can't see that you have | 14:36 |
| fungi | anteaya: probably plenty, i just haven't found out yet | 14:36 |
| anteaya | ha ha ha | 14:36 |
| anteaya | I'm still reading weekend scrollback | 14:37 |
| sc68cal | Is there an IRC channel for devstack? | 14:37 |
| sc68cal | I feel bad asking in -infra or -qa | 14:37 |
| anteaya | sc68cal: -qa | 14:37 |
| openstackgerrit | Stephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue https://review.openstack.org/97196 | 14:37 |
| fungi | sc68cal: #openstack-qa is probably the more effective option | 14:37 |
| sc68cal | thx :) | 14:37 |
| anteaya | don't feel bad sc68cal | 14:37 |
| fungi | though they might say the same thing about #-infra ;) | 14:37 |
| *** maxbit_ has joined #openstack-infra | 14:38 | |
| *** afazekas has quit IRC | 14:38 | |
| anteaya | fortunately due to overlap, folks don't ping pong much before they find a chatty person | 14:38 |
| fungi | sc68cal: these are the people you likely want to pester about devstack patches... https://review.openstack.org/#/admin/groups/50,members | 14:38 |
| sc68cal | fungi: thanks :) | 14:39 |
| *** maxbit_ has quit IRC | 14:39 | |
| *** maxbit has joined #openstack-infra | 14:39 | |
| *** andreaf has quit IRC | 14:40 | |
| *** pblaho has quit IRC | 14:40 | |
| anteaya | fungi: turns out he needs to talk to me, atm | 14:41 |
| SergeyLukjanov | fungi, +2'd waiting for job results | 14:41 |
| openstackgerrit | Stephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue https://review.openstack.org/97196 | 14:42 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/infra-specs: StoryBoard search spec https://review.openstack.org/97254 | 14:42 |
| Alex_Gaynor | Are there an absence of whichever buiders are used for ht pep8, 2.7, and docs jobs? | 14:42 |
| *** mrmartin has quit IRC | 14:44 | |
| fungi | Alex_Gaynor: not specifically, from the look of it, we're just generally starved for workers at the moment due to the earlier setuptools failures (everything's getting rechecked now) | 14:45 |
| *** wenlock has joined #openstack-infra | 14:45 | |
| Alex_Gaynor | fungi: makes sense, thanks | 14:46 |
| *** mbacchi has quit IRC | 14:46 | |
| *** trinaths has joined #openstack-infra | 14:46 | |
| fungi | Alex_Gaynor: looks like we're about an hour wait in the check pipeline for non-devstack workers at the moment | 14:47 |
| *** w_ has joined #openstack-infra | 14:47 | |
| *** skolekonov has quit IRC | 14:47 | |
| Alex_Gaynor | fungi: I'm imagining a butcher, and there's a bunch of folks just milling about with those tickets they give you, and complaining :-) | 14:47 |
| fungi | Alex_Gaynor: and just under that wait time for devstack workers too | 14:47 |
| fungi | Alex_Gaynor: yes, next up is #93576,5 | 14:48 |
| openstackgerrit | Tim Kelsey proposed a change to openstack-infra/gerritbot: Adding a configuration option for gerritbots PID file path. https://review.openstack.org/97250 | 14:48 |
| *** [1]trinaths has joined #openstack-infra | 14:49 | |
| *** olaph has quit IRC | 14:49 | |
| *** afazekas has joined #openstack-infra | 14:50 | |
| *** w_ is now known as olaph | 14:50 | |
| *** tkelsey has joined #openstack-infra | 14:50 | |
| *** pblaho has joined #openstack-infra | 14:50 | |
| *** jlibosva has quit IRC | 14:51 | |
| *** mbacchi has joined #openstack-infra | 14:51 | |
| *** david-lyle has joined #openstack-infra | 14:51 | |
| *** trinaths has quit IRC | 14:51 | |
| *** [1]trinaths is now known as trinaths | 14:51 | |
| *** atiwari has joined #openstack-infra | 14:53 | |
| *** duran has joined #openstack-infra | 14:53 | |
| *** vhoward has joined #openstack-infra | 14:55 | |
| *** e0ne_ has quit IRC | 14:57 | |
| cody-somerville | Hey. Is it possible to tell nodepool to pick an az? | 14:57 |
| *** e0ne has joined #openstack-infra | 14:57 | |
| *** hogepodge has joined #openstack-infra | 14:58 | |
| fungi | cody-somerville: by using a node label which is specific to a particular az | 14:58 |
| cody-somerville | How do I make a node label specific to a particular az? | 14:59 |
| fungi | cody-somerville: in nodepool.yaml, create a label which only references one provider name | 14:59 |
| *** skolekonov has joined #openstack-infra | 14:59 | |
| cody-somerville | How do I make that provider only use one az though? | 15:00 |
| fungi | cody-somerville: each az is a separate "provider" in nodepool parlance | 15:00 |
| openstackgerrit | Longgeek proposed a change to openstack-infra/config: Modify jenkins home directory https://review.openstack.org/97240 | 15:00 |
| cody-somerville | fungi: No it ain't. | 15:00 |
| cody-somerville | fungi: or at least I don't see one specified :) | 15:01 |
| cody-somerville | fungi: What is the configuration option? | 15:01 |
| *** skolekonov has quit IRC | 15:01 | |
| fungi | cody-somerville: for instance, see the two tripleo "providers" in http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb (each of which is actually a separate "region" in the tripleo cloud) | 15:01 |
| krotscheck | StoryBoard meeting in #openstack-meeting-3 | 15:01 |
| *** e0ne has quit IRC | 15:02 | |
| anteaya | fungi: the status of nodes in delete state seems to be staying steady, rather than following the curve of nodes in use, this tends to be my indicator that some nodes may be stuck in a delete state | 15:02 |
| fungi | cody-somerville: right now there are "tripleo-test-cloud" and "tripleo-test-cloud-rh1" providers defined, which are the hp and rh regions of the tripleo cloud respectively | 15:03 |
| cody-somerville | I want to pick availability zone. | 15:03 |
| fungi | cody-somerville: similarly, hpcloud-az1/az2/az3 "providers" for the three hpcloud 1.0 west availability zones | 15:03 |
| cody-somerville | That's for US East. | 15:03 |
| cody-somerville | and those are actually regions | 15:04 |
| cody-somerville | HP Cloud used different regions to provide different "availability zones" in 1.0. | 15:04 |
| cody-somerville | HP Cloud has capacity issue in AZ2 of US East (1.1, aka 13.5) and it appears nodepool is only creating instances in that az because that's the default is not specified otherwise. | 15:04 |
| *** e0ne has joined #openstack-infra | 15:05 | |
| fungi | cody-somerville: ahh, you're asking about hpcloud 1.1. you should have been more specific. we were informed that there was one tenant quota covering that entire cloud and that hpcloud would simply round-robin or otherwise distribute nova boot requests to where there was available capacity. so we just treat it as a single provider currently | 15:06 |
| cody-somerville | It turns out it actually just puts it all in az2 as it seems. | 15:07 |
| cody-somerville | :( | 15:07 |
| *** w_ has joined #openstack-infra | 15:07 | |
| *** lcostantino has quit IRC | 15:08 | |
| fungi | cody-somerville: if that's not actually the case after all, then we probably need to start using multiple region-b providers representing different availability zones and set separate max-servers levels for each in nodepool | 15:08 |
| fungi | mordred: clarkb: jeblair: ^ fun! | 15:08 |
| cody-somerville | I don't see how to set what az it actually uses? | 15:08 |
| tteggel | fungi: cody-somerville is correct | 15:08 |
| *** eharney has joined #openstack-infra | 15:08 | |
| * tteggel goe hunting for the docs | 15:09 | |
| *** yfried_ has quit IRC | 15:09 | |
| Alex_Gaynor | fungi: I think we might be more than just behind, even the top of the check pipeline has unstarted jobs | 15:09 |
| fungi | tteggel: it's possible this has changed since the original beta account we had before region b was ga | 15:09 |
| *** olaph has quit IRC | 15:10 | |
| fungi | Alex_Gaynor: those usually represent a job which got started, lost because of a jenkins-specific failure (node went offline or something) and is queued to be automatically retried | 15:10 |
| Alex_Gaynor | fungi: No, I mean, basically the entire pipeline, /even/ the top has unstarted jobs | 15:11 |
| fungi | Alex_Gaynor: due to the queueing order, however, those don't get serviced until other changes which were already waiting for workers when that failed | 15:11 |
| *** mbacchi has quit IRC | 15:11 | |
| *** mbacchi has joined #openstack-infra | 15:12 | |
| fungi | Alex_Gaynor: the top several in the check pipeline all seem to have workers, there are a couple nova changes behind it which look like they probably lost a devstack worker and are waiting on their turn to get the lost jobs retried. behind those are a lot of changes waiting on centos6 nodes where they can run py26 jobs | 15:13 |
| *** talluri has joined #openstack-infra | 15:13 | |
| *** rfolco has quit IRC | 15:14 | |
| fungi | Alex_Gaynor: looking in the gate pipeline, which gets priority over check, we also seem to be running low on available precise-py3k nodes to run pypy and python33 jobs | 15:14 |
| cody-somerville | fungi: Looking at the code, nodepool does not have the capability to specify an az. | 15:14 |
| fungi | cody-somerville: yeah, i have no idea what the nova server parameter is for that in the python api, but i assume there is one we can add and then expose in nodepool configuration | 15:15 |
| trinaths | Hi Stackers!, You are able to access the CI logs server at http://freescaleci.fh2web.com/ ... Kindly help me with your comments. | 15:15 |
| fungi | cody-somerville: from nodepool's perspective, without some significant refactoring anyway, i assume we'd just call them separate providers (one for each az) | 15:16 |
| Alex_Gaynor | fungi: mm, I guess it is just centos and precise-py3k then, thanks | 15:16 |
| fungi | cody-somerville: it doesn't help for terminology that hpcloud previously called its regions "availability zones" i guess, so i didn't realize there was a distinction | 15:16 |
| cody-somerville | ++ | 15:17 |
| *** esker has joined #openstack-infra | 15:17 | |
| Kiall | fungi: Yea, we fixed the terminology with the new regions.. The old naming was painful | 15:19 |
| *** w_ is now known as olaph | 15:20 | |
| openstackgerrit | Roman Podoliaka proposed a change to openstack-infra/config: Upload tripleo-heat-templates releases to PyPi https://review.openstack.org/97265 | 15:20 |
| tteggel | fungi: cody-somerville: so hpcloud 1.1 is using OS-EXT-AZ extension | 15:21 |
| tteggel | no longer the weird az-as-region with its own endpoint thing | 15:22 |
| *** mmaglana has joined #openstack-infra | 15:24 | |
| *** morganfainberg is now known as morganfainberg_Z | 15:25 | |
| srenatus | Is there any way to pass more information from a jenkins job to the gerrit message? I.e. "FAILED: part 1" if the check consists of several parts? | 15:25 |
| *** basha has quit IRC | 15:26 | |
| anteaya | srenatus: hi what is the context for your question? | 15:27 |
| anteaya | srenatus: are you working with you own ci system seperate from openstack's? | 15:27 |
| srenatus | anteaya: I'm pondering the idea of moving the chef cookbooks' tests to a per-repo fashion: https://blueprints.launchpad.net/openstack-chef/+spec/per-repo-tests | 15:28 |
| srenatus | anteaya: but then we'd lose (compared to before) the quick visual feedback of "chef-lint: FAILURE, chef-spec: SUCCESS" | 15:29 |
| srenatus | anteaya: I'd like to figure out if it'd be possible to get that back | 15:29 |
| anteaya | I don't know enough about your workflow to offer an opinion | 15:29 |
| anteaya | for instance I have no idea what strainers are | 15:30 |
| anteaya | do you have a sample patch I can view? | 15:30 |
| anteaya | fungi: I have to conclude we have nodes stuck in a delete state somewhere | 15:31 |
| srenatus | anteaya: strainer is a tool to specify tests suites (tools, rather) to be run against a cookbook: https://github.com/stackforge/cookbook-openstack-network/blob/master/Strainerfile | 15:31 |
| srenatus | anteaya: and I'd love to have this file be the one source of truth about which tests are run for this cookbook | 15:31 |
| anteaya | srenatus: do you have a sample patch in gerrit that I can view? | 15:31 |
| srenatus | anteaya: https://review.openstack.org/#/c/97217/ | 15:32 |
| * anteaya clicks | 15:32 | |
| srenatus | anteaya: it's all failures for some other reason, but | 15:32 |
| srenatus | if we changed the jenkins-part (i.e. the job builder yaml) to only one job, "strainer test", running all three parts, it'd only be one, accumulated output: success/failure | 15:33 |
| mordred_phone | cody-somerville: you mean it doesn't round robin if not specified? that's assinine | 15:35 |
| tteggel | mordred_phone: nope. we lost that argument early on | 15:35 |
| anteaya | srenatus: yes I see the three tests, lint, style and unit: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml | 15:35 |
| tteggel | mordred_phone: because cross AZ compute volume attachment doesn't work | 15:36 |
| mordred_phone | tteggel: whoever you lost to was wrong | 15:36 |
| * mordred_phone cries | 15:36 | |
| anteaya | srenatus: I'm failing to see the efficienty for rolling the three tests into one | 15:36 |
| fungi | srenatus: there is work underway to replace jenkins with a more flexible job runner which can return more specific status information on task completion. right now we get "pass" or "fail" so there's no easy way to break that out into a list of discrete states for sub-tasks | 15:36 |
| fungi | anteaya: what makes you think there are nodes stuck? | 15:36 |
| srenatus | fungi: thanks! | 15:37 |
| srenatus | fungi: good to know | 15:37 |
| anteaya | fungi: the purple line is staying straight | 15:37 |
| srenatus | fungi: any details? :) | 15:37 |
| anteaya | fungi: which is consistent with all other times nodes are stuck in delete | 15:37 |
| *** rgerganov has quit IRC | 15:37 | |
| anteaya | fungi: the in use line is fluxuating and the delete state portion of the graph is not | 15:37 |
| hashar | hello! Any Zuul guru ever heard of Zuul being stuck in a loop while reporting change? It looped with an exception that a job already got reported (trace: http://paste.openstack.org/show/82452/ ) | 15:38 |
| fungi | anteaya: the one doesn't follow from the other. i haven't had a moment to look yet, but i suspect that the large number of nodes in a building state are rapidly transitioning directly to delete state because they fail to complete | 15:38 |
| mordred_phone | so I have a cross az quota, but I still have to point to azs | 15:38 |
| mordred_phone | that's going to be ugly for quota awareness | 15:38 |
| anteaya | fungi: yes that was your assessment the last time I saw the same thing | 15:39 |
| tteggel | mordred_phone: yup. we are trying to reopen this dicussion internally for precisely these reasons | 15:39 |
| anteaya | fungi: since you had been busy with other things, I wanted it to be on your radar, since we do have such a backlog in the queues right now | 15:39 |
| tteggel | mordred_phone: although the likely outcome is that you will be forced to specify AZ | 15:39 |
| anteaya | hashar: zuul will loop if it can't report the change and then will kick off a new test since there is no feedback reported | 15:40 |
| mordred_phone | tteggel: whee! :) | 15:40 |
| anteaya | hashar: like if there is a comment on patchset 5 and patchset 6 is the latest patchset | 15:40 |
| hashar | anteaya: yeah because it received a new patchset so it dequeue the previous event | 15:41 |
| anteaya | right, so it won't report | 15:41 |
| anteaya | because voting is closed on that patchset | 15:41 |
| anteaya | hashar: is it happening now? | 15:42 |
| jeblair | annegentle: regarding https://review.openstack.org/#/c/96334/ I don't really understand the idea of publishing something from an unofficial (stackforge) project on the official openstack website | 15:42 |
| jeblair | annegentle: i think that's a line we should not cross | 15:42 |
| hashar | anteaya: in this case I had only one patchset but the change was in two pipelines (test and gate). So might be a race condition, I will look at the debug log | 15:42 |
| hashar | anteaya: na I killed Zuul, it was stuck and no more processing any event hehe | 15:42 |
| jeblair | fungi, fifieldt, mordred_phone: ^ | 15:42 |
| tteggel | mordred_phone: there's no good answer unless we have a scheduler that can work with cinder, nova and neutron | 15:42 |
| anteaya | hashar: kk | 15:42 |
| tteggel | mordred_phone: otherwise people who want AZ affinity of resources (which you really do with a volume) are screwed | 15:43 |
| hashar | anteaya: was just wondering whether it has been a known issue. Will investigate. Thank you! | 15:43 |
| openstackgerrit | Ana Krivokapic proposed a change to openstack/requirements: Add Pint for Horizon https://review.openstack.org/97224 | 15:43 |
| anteaya | hashar: an edgecase that has popped up occasionally, I think we have the known cases fixed but are interested to hear details of other triggers | 15:43 |
| *** e0ne_ has joined #openstack-infra | 15:44 | |
| mordred_phone | jeblair: looking | 15:44 |
| mordred_phone | jeblair: yes. I agree | 15:44 |
| mordred_phone | if we want it to be a thing, it should get adopted by docs for real ... or it should grow another home | 15:45 |
| mordred_phone | or a new program | 15:45 |
| fungi | mordred_phone: that's what sarob seems to be implying in his comment... that they want to start a training program which this will eventually be incubated in | 15:46 |
| *** SumitNaiksatam has quit IRC | 15:46 | |
| *** SumitNaiksatam has joined #openstack-infra | 15:47 | |
| *** amotoki_ has joined #openstack-infra | 15:47 | |
| fungi | so not going to be part of the docs program ever, i guess, which would explain why they don't want to incubate in docs (in case the training program isn't approved by the tc?) | 15:48 |
| *** e0ne has quit IRC | 15:48 | |
| *** salv-orlando has quit IRC | 15:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix groups-dev drush dependency problem https://review.openstack.org/96421 | 15:50 |
| *** gyee has joined #openstack-infra | 15:53 | |
| dhellmann | clarkb: are you still up for testing alpha releases today, or should we wait for a day when we don't have huge breaking issues like setuptools? | 15:54 |
| anteaya | dhellmann: I do believe setuptools is actually fixed now | 15:55 |
| *** social__ has joined #openstack-infra | 15:55 | |
| anteaya | dhellmann: we are straining under the weight of rechecks though | 15:55 |
| dhellmann | anteaya: yeah, that's what I meant :-) | 15:55 |
| anteaya | :D | 15:55 |
| anteaya | mighty kind of you | 15:55 |
| *** salv-orlando has joined #openstack-infra | 15:55 | |
| *** gokrokve has joined #openstack-infra | 15:56 | |
| *** rdopiera has quit IRC | 15:57 | |
| *** zhiyan is now known as zhiyan_ | 15:57 | |
| *** amotoki_ has quit IRC | 15:59 | |
| mriedem | anyone know what thstack is? | 15:59 |
| mriedem | thstack-ci | 15:59 |
| *** amotoki_ has joined #openstack-infra | 15:59 | |
| mriedem | Longgeek: ^ | 16:00 |
| Longgeek | mriedem: hi. | 16:00 |
| Longgeek | mriedem: my ci. | 16:00 |
| mriedem | Longgeek: let's talk in #openstack-nova | 16:00 |
| *** sarob has joined #openstack-infra | 16:02 | |
| *** talluri has quit IRC | 16:04 | |
| *** gokrokve has quit IRC | 16:04 | |
| *** e0ne_ has quit IRC | 16:04 | |
| zaro | morning | 16:04 |
| *** IvanBerezovskiy has left #openstack-infra | 16:05 | |
| anteaya | morning zaro | 16:06 |
| wenlock | /wave zaro | 16:06 |
| *** salv-orlando has quit IRC | 16:07 | |
| sarob | fungi: any other questions on https://review.openstack.org/#/c/96334/ | 16:07 |
| srenatus | anteaya, fungi: thanks for your input above (the jenkins job feedback issue) | 16:08 |
| *** ewindisch has quit IRC | 16:09 | |
| anteaya | srenatus: np | 16:09 |
| *** viktors is now known as viktors|afk | 16:12 | |
| fungi | sarob: at the moment, my primary concern is with the slippery slope of allowing stackforge projects, which by definition are not under the governance of the openstack technical committee, to publish documentation on an official openstack website | 16:13 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Don't gate pecan on barbican for now. https://review.openstack.org/97285 | 16:13 |
| ryanpetrello | any chance I could convince one of you to push ^ through? | 16:14 |
| ryanpetrello | I goofed up on the git repo URL for barbican and now pecan patchsets are failing -_- | 16:14 |
| ryanpetrello | <— dummy | 16:14 |
| *** packet has joined #openstack-infra | 16:14 | |
| ryanpetrello | actually, I think I did *that* wrong, too... | 16:15 |
| ryanpetrello | sec... | 16:15 |
| sarob | fungi: we have been planning on becoming a separate project since we started last summer | 16:15 |
| *** dizquierdo has quit IRC | 16:16 | |
| fungi | ryanpetrello: sure, but the patch is wrong. see inline comments | 16:16 |
| ryanpetrello | yep | 16:16 |
| ryanpetrello | fixing now | 16:16 |
| fungi | ryanpetrello: oh, you just noticed it anyway | 16:16 |
| *** ominakov has quit IRC | 16:16 | |
| fungi | sarob: cool. do you have plans for a website you're going to publish it on? | 16:17 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Don't gate pecan on barbican for now. https://review.openstack.org/97285 | 16:17 |
| sarob | fungi: right where it is under docs.openstack.org | 16:17 |
| sarob | fungi: annegentle can attest to planning | 16:18 |
| anteaya | jaypipes krtaylor so this is what I have so far for today: https://wiki.openstack.org/wiki/Meetings/ThirdParty#Agenda_for_next_meeting | 16:18 |
| fungi | ryanpetrello: see my second inline comment on the first patchset of that change. leaving it non-voting in check is fine, but there's no reason to have it in the gate pipeline since nobody will see the results | 16:18 |
| sarob | fungi: we will continue to work with the documentation team, as we are reusing much of the material | 16:18 |
| fungi | sarob: i understand, i'm just wondering where we define the criteria for which unofficial projects publish on docs.openstack.org if it's no longer reserved for official documentation, so i know how to answer the other stackforge projects which ask that question | 16:19 |
| jaypipes | anteaya: looks good. I will not be attending today's meeting, however... | 16:19 |
| anteaya | fungi: the delete line for nodes is dropping now, thanks | 16:19 |
| anteaya | jaypipes: sorry to hear that | 16:19 |
| ryanpetrello | got it | 16:19 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Don't gate pecan on barbican for now. https://review.openstack.org/97285 | 16:19 |
| sarob | fungi: i need to bring on new training developers and giving 5-6 new people core reviewing under docs wasnt considered | 16:19 |
| anteaya | jaypipes: I was hoping you could help answer some questions during the highlight an account section | 16:19 |
| anteaya | jaypipes: if I send some folks your way, when might you be available for questions? | 16:20 |
| sarob | fungi: its a chicken egg problem | 16:20 |
| *** UtahDave has joined #openstack-infra | 16:20 | |
| sarob | fungi: we are working towards incubated | 16:20 |
| sarob | fungi: but i need consistent contribution | 16:20 |
| jaypipes | anteaya: if someone can add questions to an etherpad somewhere, I can answer them later on. | 16:20 |
| sarob | fungi: more contributors mean more core reviews | 16:20 |
| anteaya | jaypipes: great, I will set that up and attach it to the meeting agenda | 16:21 |
| anteaya | thank you | 16:21 |
| jaypipes | np | 16:21 |
| sarob | fungi: more core reviewers mean i need a separate repo from docs | 16:21 |
| *** dkliban is now known as dkliban_lunch | 16:21 | |
| *** sballe has quit IRC | 16:22 | |
| anteaya | I am going to go for a walk before the third-party meeting | 16:22 |
| anteaya | back in about an hour | 16:22 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/gerritlib: Apply cookiecutter defaults https://review.openstack.org/94844 | 16:22 |
| fungi | sarob: well, this might be the time to make an incubation proposal to the tc, because as a project we have generally accepted publishing documentation for incubated projects within official programs to the official publication sites | 16:22 |
| sarob | fungi: im good with that | 16:23 |
| fungi | sarob: assuming the idea is to have "openstack training" as an official program? | 16:23 |
| *** adalbas has quit IRC | 16:23 | |
| dhellmann | sdague: I'm wondering if there's a way to disable "foreach" for a section in gerrit-dash-creator so I can have a section that links to reviews outside of oslo for syncs that are happening | 16:23 |
| sarob | fungi: official not so sure | 16:23 |
| sarob | fungi: one step at a time | 16:24 |
| sarob | fungi: i need to ramp up about another 6-8 developers | 16:24 |
| sarob | fungi: our contribution rate is probably off for incubated right now | 16:25 |
| *** msabramo has joined #openstack-infra | 16:25 | |
| sarob | fungi: id like to get these new developers, that know training, but not openstack ci | 16:25 |
| sarob | fungi: ramped up with as few changes in the near future as possible | 16:26 |
| sarob | fungi: can we focus on this first step first? | 16:26 |
| fungi | sarob: ultimately, continuing to publish it to an official openstack site when it's not currently incubated is something which should be discussed at a higher level within the openstack project than this code review change. i think the proper venue for that discussion is the tc. if we drop the publish job from the new project request for now, i have no other significant concerns | 16:27 |
| ryanpetrello | thanks, fungi | 16:27 |
| sarob | fungi: between annegentle and tfifield, we should have pretty good coverage on if publishing short term from a stackforge repo is okay? | 16:27 |
| sarob | fungi: sure we can discuss with TC as well | 16:28 |
| fungi | sarob: i think they can act as advocates to the tc for that exception if they feel that way, yes | 16:28 |
| *** julim has quit IRC | 16:28 | |
| sarob | fungi: okay, ill discuss at next tc meet | 16:29 |
| jeblair | sarob: i think you should aim for getting the program going first | 16:29 |
| jeblair | sarob: i think publishing unofficial docs on the official site is putting the cart before the horse | 16:29 |
| *** zehicle_at_dell has quit IRC | 16:29 | |
| *** zehicle_at_dell has joined #openstack-infra | 16:30 | |
| jeblair | sarob: but fortunately, the bar for creating a new program isn't that high | 16:30 |
| sarob | jeblair: its being published there now | 16:30 |
| jeblair | sarob: we've made it a lot harder for new projects to get on the 'core' track, but programs that aren't aiming for that don't have nearly as many hurdles | 16:31 |
| sarob | jeblair: we want to bring on new developers and expand the core reviewer | 16:31 |
| *** jgallard has quit IRC | 16:31 | |
| jeblair | sarob: yeah, and i think a new program will help that | 16:31 |
| sarob | jeblair: i have the new developers lined up already | 16:31 |
| fungi | it's being published there as part of the documentation program, but this is a request for a project outside of the documentation program publishing to the same place, and without an official openstack program to govern it | 16:31 |
| sarob | jeblair: im aiming to expand the people invovled without muddling docs | 16:32 |
| *** marcoemorais has joined #openstack-infra | 16:32 | |
| *** mmaglana has quit IRC | 16:32 | |
| jeblair | sarob: ++ | 16:32 |
| sarob | jeblair: if annegentle is okay with adding new core reviewers into docs for over the summer | 16:32 |
| sarob | jeblair: then breaking off into a new repo can wait | 16:33 |
| *** mmaglana has joined #openstack-infra | 16:33 | |
| fungi | sarob: well, it can be broken off into a new repo within the docs program too, if they agree for it to still fall under their umbrella | 16:33 |
| sarob | jeblair: i dont want to confuse the new people coming onboard | 16:33 |
| sarob | fungi: i thought that was the case | 16:33 |
| *** _nadya_ has quit IRC | 16:34 | |
| fungi | sarob: but in that case it would be openstack/openstack-training-guides instead of stackforge/openstack-training-guides | 16:34 |
| sarob | fungi: there is no reason to start a new program outside of docs | 16:34 |
| sarob | fungi ah, okay | 16:34 |
| *** BAKfr has quit IRC | 16:35 | |
| sarob | fungi tfifield and i didnt think of that | 16:35 |
| sarob | fungi: so should i just adjust the patch? | 16:36 |
| *** shakamunyi has joined #openstack-infra | 16:36 | |
| fungi | sarob: that's primarily a question for annegentle | 16:36 |
| sarob | fungi: add annegentle as additional reviewer? | 16:36 |
| fungi | sarob: but yes that would be one way | 16:36 |
| sarob | fungi: /me sigh | 16:36 |
| sarob | arrg | 16:36 |
| * sarob sigh | 16:37 | |
| sarob | fungi: its never one way | 16:37 |
| *** mmaglana has quit IRC | 16:37 | |
| sdague | dhellmann: am I doing it unconditionally? | 16:37 |
| sdague | realistically you probably always want status:open in there anyway | 16:37 |
| sarob | fungi: ill confer with tfifield and annegentle, and then come back to the patch | 16:38 |
| dhellmann | sdague: yeah, I wanted to override the projects list for one case (see https://github.com/sdague/gerrit-dash-creator/pull/1) but I decided to make that a separate dashboard | 16:38 |
| sarob | fungi: thanks for the feedback | 16:38 |
| clarkb | o/ | 16:38 |
| fungi | sarob: okay, great. let me know what you figure out | 16:38 |
| *** adalbas has joined #openstack-infra | 16:39 | |
| sarob | jeblair: sound about right to have training as separate project but under docs program? | 16:39 |
| sarob | fungi: roger | 16:39 |
| clarkb | fungi: cody-somerville I think we should avoid specifying AZs unless we can confirm that the scheduling given us is actually wrong | 16:40 |
| sdague | dhellmann: lgtm, merged | 16:40 |
| clarkb | it is supposed to schedule things properly | 16:40 |
| sarob | any infra online that can speak to storyboard access? | 16:40 |
| jeblair | sarob: yep, that's an option. programs can have as many projects as they need. | 16:40 |
| clarkb | dhellmann: I am up for testing it today. Found one bug yesterday after harlowja_away tagged taskflow | 16:40 |
| clarkb | catching up this morning to see if the fix merged | 16:41 |
| jeblair | sarob: re storyboard: yes. :) for what project? | 16:41 |
| sarob | jeblair: training-guides ;) | 16:41 |
| fungi | sarob: yep, for storyboard to add your project it's just an additional parameter in the review.projects.yaml file | 16:41 |
| *** radez is now known as radez_g0n3 | 16:41 | |
| sarob | fungi: ah, righto | 16:41 |
| fungi | sarob: and then when the review does merge, you'll have a corresponding project in storyboard | 16:41 |
| jeblair | sarob: yeah, that should be okay. two things: | 16:41 |
| dhellmann | sdague: https://wiki.openstack.org/wiki/Oslo#Review_Links | 16:41 |
| fungi | sarob: so that 96334 change is the appropriate place to add it | 16:42 |
| dhellmann | clarkb: I saw that change earlier and left some comments | 16:42 |
| *** marun_afk is now known as marun | 16:42 | |
| jeblair | sarob: a) if you stay in the docs program, can you run that by annegentle? she may want all docs-related projects to use the same tracker | 16:42 |
| jeblair | sarob: b) we're not actually recommending storyboard for production use by openstack projects yet. i'd actually strongly discourage it. | 16:42 |
| sarob | jeblair: i'm torn actually | 16:43 |
| fungi | sarob: jeblair makes very good points ;) | 16:43 |
| sarob | jeblair: its bringing on non-openstack people | 16:43 |
| jeblair | sarob: we haven't managed to do significant dogfooding in infrastructure yet; i would recommend at least waiting until we can injure ourselves first | 16:43 |
| clarkb | dhellmann: thanks. I just copy pasta'd that code from the run-tarball.sh script | 16:43 |
| sarob | jeblair: these are training developers, not python developers | 16:43 |
| dhellmann | clarkb: yeah, not worth a -1 | 16:43 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Gate barbicanclient on PyPy and Python 3.3 https://review.openstack.org/97288 | 16:43 |
| dhellmann | clarkb: I don't see that change any more, was it merged? | 16:44 |
| fungi | sarob: storyboard is definitely still short a few features which a lot of projects are likely to want too | 16:44 |
| sarob | jeblair: storyboard will be alot easier to train them on | 16:44 |
| jeblair | sarob: i think i can anticipate many of the things you're going to say about launchpad, but i'm worried about storyboard being worse for new devs | 16:44 |
| clarkb | dhellmann: it was merged | 16:44 |
| jeblair | sarob: and storyboard is going to change _radically_ several more times before we really start to use it | 16:44 |
| clarkb | dhellmann: https://review.openstack.org/#/c/97142/1 | 16:44 |
| dhellmann | clarkb: cool | 16:44 |
| *** morganfainberg_Z is now known as morganfainberg | 16:44 | |
| *** Guest22618 has quit IRC | 16:44 | |
| sdague | dhellmann: cool | 16:45 |
| sarob | jeblair: i know launchpad well, but i have trained about 50 people so far as new openstack devs | 16:45 |
| clarkb | jeblair: did the gertty stack get that missing DB migration? | 16:45 |
| sarob | jeblair: launchpad blueprints is one of the biggest confusing parts | 16:45 |
| jeblair | sarob: so i actually think it might hurt more than it would help (as much as i want everyone to just use storyboard now (!!!) i don't want to put people off because we're asking them to use something unfinished and which will fundamentally change | 16:45 |
| *** habib has quit IRC | 16:45 | |
| dhellmann | clarkb: unfortunately I have a block of meetings starting shortly, so it may be a while before I can trigger any more tests. If harlowja_away is already tagging taskflow and there were problems, should we continue testing with that lib to "fix" the release? | 16:46 |
| sdague | dhellmann: as you are a mac person, do you happen to have IE installed somewhere? | 16:46 |
| dhellmann | sdague: I'm not sure there's a version of IE that runs on OS X any more. | 16:46 |
| sdague | I'd like to figure out if the qa review url that breaks in firefox works in IE | 16:46 |
| sdague | oh, I thought there still was | 16:46 |
| *** trinaths has quit IRC | 16:46 | |
| sarob | jeblair: i though we were just going to add new features | 16:46 |
| morganfainberg | dhellmann, sdague , there isn't | 16:46 |
| sdague | morganfainberg: ok | 16:46 |
| *** rwsu has joined #openstack-infra | 16:46 | |
| sarob | jeblair: thats what i got from the session | 16:46 |
| morganfainberg | sdague, most people use VMs if they need IE | 16:46 |
| sdague | being on Linux, it's never been here :) | 16:46 |
| sarob | jeblair: i misunderstand? | 16:47 |
| morganfainberg | sdague, yeah hear that. | 16:47 |
| clarkb | dhellmann: ya we can try retriggering taskflow jobs | 16:47 |
| clarkb | dhellmann: rather than tagging something new | 16:47 |
| jeblair | sarob: some of those new features may require significant changes. the "priority" system is something that is still way up in the air | 16:47 |
| sarob | jeblair: hmm, okay. i will wait then | 16:48 |
| Alex_Gaynor | So this graph looks ungood: http://bit.ly/1nI5HkW | 16:48 |
| sarob | jeblair: i was leaning that way, but after refstack joined, i changed my mind | 16:48 |
| clarkb | Alex_Gaynor: we don't have enough slaves | 16:49 |
| fungi | Alex_Gaynor: yeah, at the moment we're using almost nothing in hpcloud, and trying to get by only on rackspace | 16:49 |
| Alex_Gaynor | clarkb: is one of our clouds having a bad morning? It seems way worse than normal | 16:49 |
| Alex_Gaynor | I gather that's not going well. | 16:50 |
| clarkb | Alex_Gaynor: its two things. we turned off one cloud that will be turned off today | 16:50 |
| *** blamar has quit IRC | 16:50 | |
| clarkb | and the replacement for said cloud is giving us a hard time | 16:50 |
| *** trinaths has joined #openstack-infra | 16:50 | |
| Alex_Gaynor | If there's anything I can do to help on the rackspace side, let me know. | 16:50 |
| fungi | Alex_Gaynor: hpcloud 1.0 is apparently finally being turned off, so we stopped using it over the weekend. hpcloud 1.1 is struggling to boot new nodes, supposedly because all requests are getting scheduled to one az | 16:51 |
| *** hashar has quit IRC | 16:51 | |
| fungi | though i'm not sure if that's the only reason, it's been brought up as a reason | 16:51 |
| clarkb | fungi: is that really why its failing? | 16:51 |
| *** afazekas has quit IRC | 16:51 | |
| clarkb | it wasn't clear to me from cody-somerville that tis was related | 16:51 |
| *** jistr has quit IRC | 16:52 | |
| zigo | sdague: I'm having an issue with tempest (with the "1" tag). It's doing "Non-zero exit code (2) from test listing." before I even got a chance to run tests. What can I do to investigate where the issue is? | 16:52 |
| fungi | clarkb: may have been tangential, but i assumed given the timing that it was related | 16:52 |
| jesusaurus | clarkb: i believe that nodepool is hardcoded to use an az thats having extra trouble | 16:52 |
| jesusaurus | at least thats my understanding | 16:52 |
| clarkb | jesusaurus: it isn't hard coded to use any AZ | 16:52 |
| clarkb | jesusaurus: nova client/server are supposed to schedule an AZ for you | 16:53 |
| zigo | The relevant output seems to be: unittest.loader.LoadTestsFailure.tempest.api.compute.v3.flavors.test_flavors_negative\x1eH\xd0J\xb3 `I\x02\x17text/plain;charset=utf8\rimport errorsH\xd1tempest.api.compute.admin.test_migrations | 16:53 |
| clarkb | jogo: ^ can you fill us in on that? iirc you explained it to us when we added these regions to nodepool | 16:53 |
| zigo | However, that file seems fine ... | 16:53 |
| jeblair | sarob: oddly enough, refstack is not part of an official program, so i have a bit less of an opinion there. (that refstack is not in an official program is perhaps something that should be addressed) | 16:54 |
| sdague | zigo: probably ping mtreinish in -qa | 16:54 |
| *** harlowja_away is now known as harlowja_ | 16:54 | |
| fungi | is the os-ext-az extension one which comes standard in novaclient, or something we would need to add another piece of software to support? | 16:54 |
| zigo | Cheers. | 16:54 |
| jesusaurus | clarkb: oh, huh | 16:54 |
| mtreinish | zigo: that sounds like the testr import error thing, probably a config file issue | 16:54 |
| clarkb | Alex_Gaynor: looking at numbers rax is mostly happy. Other than bumping quotas I am not sure we need rax to do anything. and I am not sure we need to bump quotas | 16:54 |
| *** andreaf has joined #openstack-infra | 16:54 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: added bug for setuptools 3.7 breaking the world https://review.openstack.org/97221 | 16:54 |
| tteggel | fungi: supported in default novaclient | 16:54 |
| zigo | mtreinish: So, by default, the tempest.conf.sample doesn't work? | 16:55 |
| *** mmaglana has joined #openstack-infra | 16:55 | |
| tteggel | fungi: need to pass availability_zone param to server create method | 16:55 |
| fungi | tteggel: okay, so assuming we're using latest released novaclient we can in theory add support to twiddle that from nodepool | 16:55 |
| mtreinish | zigo: no you need to add things like creds so tempest knows how to talk to the cloud | 16:55 |
| clarkb | tteggel: well reality may be we need to but we shouldn't need to | 16:55 |
| harlowja_ | clarkb is there anything i need to do to retrigger the pypi taskflow tag stuff/job? | 16:55 |
| jogo | clarkb: yeah if you don't specify an AZ you will be assigned one | 16:56 |
| zigo | mtreinish: I did add keystone creds, yes, but that's not enough? | 16:56 |
| Alex_Gaynor | clarkb: how are the different worker types distributed between the different providers? | 16:56 |
| clarkb | fungi: yes we can twiddle it in novaclient but we specifically chose not to because nova will do it for you | 16:56 |
| zigo | mtreinish: I wanted to try with only keystone stuff ... | 16:56 |
| jogo | clarkb: its something you can specify to get better spread accross failure domains | 16:56 |
| *** _nadya_ has joined #openstack-infra | 16:56 | |
| *** atiwari has quit IRC | 16:56 | |
| fungi | clarkb: well, at least we were told nova would do it for us | 16:56 |
| fungi | clarkb: but ultimately i think it's going to be up to hpcloud whether they make that happen or not | 16:56 |
| clarkb | fungi: yes, I just want to understand if nova is actually failing us here so that we can hopefully fix upstream nova | 16:57 |
| zigo | mtreinish: I was running "testr run tempest.services.identity" | 16:57 |
| clarkb | it may be the the nova scheduler is scheduling us in the only az with room | 16:57 |
| zigo | mtreinish: FYI, I'm trying to package tempest in Debian... | 16:57 |
| clarkb | or some other contraint is influencnig the system | 16:57 |
| fungi | clarkb: certainly entirely possible | 16:57 |
| jeblair | clarkb, fungi, mordred, SergeyLukjanov: can you review https://review.openstack.org/#/c/94440/ when you get a chance? | 16:57 |
| tteggel | clarkb: hpcloud explicitly chooses to schedule you to az2 if you don't specify | 16:57 |
| mtreinish | zigo: ok, yeah the thing is if the config is incomplete and we have a bug that requires something at import then when test discovery is run it'll fail | 16:58 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Fixes misaligned comments on very long lines https://review.openstack.org/96128 | 16:58 |
| mtreinish | which is probably what you're hitting | 16:58 |
| zigo | mtreinish: Ok, then what is mandatory to edit? | 16:58 |
| clarkb | tteggel: oh well wtf | 16:58 |
| tteggel | clarkb: this is due to a constraint in our cinder backend that doesn't allow cross-az volume attach | 16:58 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added priority on task row https://review.openstack.org/96156 | 16:59 |
| tteggel | clarkb: and without a scheduler that can do az affinity of volumes / compute then a key use case for a lot of users becomes painful | 16:59 |
| *** tkelsey has quit IRC | 16:59 | |
| mtreinish | zigo: the bare minimum should be the keystone creds, auth url and the enabled services. But, depending on the services enabled you have other things needed. For example nova tests require image and flavor ids | 17:00 |
| mtreinish | zigo: I don't think we've ever mapped out exactly what the min set of options was to run tempest | 17:00 |
| zigo | mtreinish: Thanks, will try then. | 17:00 |
| mtreinish | zigo: that being said you probably hit a bug because we don't want to be dependent on a config file at import time | 17:01 |
| mtreinish | it makes things like unit testing hard | 17:01 |
| clarkb | jeblair: yup I can | 17:01 |
| clarkb | fungi: have we started working on a nodepool patch yet? | 17:01 |
| *** srenatus has quit IRC | 17:01 | |
| fungi | clarkb: not yet, pending further confirmation that was the necessary solution | 17:02 |
| *** lcostantino has joined #openstack-infra | 17:02 | |
| fungi | clarkb: though i don't think the actual feature would be hard to add | 17:02 |
| *** srenatus has joined #openstack-infra | 17:02 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added new priority changed event for tasks https://review.openstack.org/96151 | 17:02 |
| *** atiwari has joined #openstack-infra | 17:02 | |
| mtreinish | zigo: what happens when you try to manually import the file testr spit out in the python interpreter? | 17:02 |
| clarkb | harlowja_: nope we will need to trigger it on the gearman bus directly. Once we get slave availability sorted I can stab at rerunning te jobs | 17:02 |
| *** matjazp has joined #openstack-infra | 17:02 | |
| harlowja_ | clarkb sounds great, thx :) | 17:03 |
| SergeyLukjanov | jeblair, looking | 17:03 |
| *** julim has joined #openstack-infra | 17:03 | |
| clarkb | fungi: ya I don't think so either. More an issue of how we want to do the assignment of AZ | 17:03 |
| clarkb | fungi: also iirc networks are cross AZ so not really sure what benefit there is of having AZs at all | 17:03 |
| *** arnaud__ has joined #openstack-infra | 17:03 | |
| *** ominakov has joined #openstack-infra | 17:03 | |
| zigo | mtreinish: It just works ... | 17:03 |
| clarkb | tteggel: ^ is that true or do we need to create routers and networks per AZ? when I created them it seemed to not care what the AZ was | 17:04 |
| tteggel | router and networks are cross-az. volumes are not. | 17:04 |
| clarkb | thanks | 17:04 |
| clarkb | fungi: jeblair what if we specify a list of AZ to the nodepool provider and just pick them at random when booting nodes? | 17:05 |
| fungi | clarkb: so given that we don't use cinder for nodepool, yes i think that would make sense | 17:05 |
| clarkb | or would it be more flexible (for volume things) to have a different provider per AZ? | 17:05 |
| clarkb | and go to 15 providers for 1.1 east? | 17:05 |
| *** ArxCruz has quit IRC | 17:05 | |
| mtreinish | zigo: yeah that definitely sounds like a config issue. The other thing you can do is try to manually run it with unittest or testtools.run. So something like 'python -m unittest module' | 17:06 |
| zigo | Thanks. | 17:06 |
| jeblair | clarkb: so images are shared across az's, volumes are not. right? what about networks? | 17:07 |
| *** ihrachyshka has quit IRC | 17:07 | |
| clarkb | jeblair: images and networks are cross AZ | 17:07 |
| clarkb | volumes are not. As we use nodepool today without volumes we should be fine just picking an AZ at random | 17:07 |
| fungi | clarkb: and i guess if we later encounter a provider where these things are not cross-az, then we just create separate providers with singleton az lists | 17:07 |
| clarkb | fungi: ya | 17:08 |
| *** rfolco has joined #openstack-infra | 17:08 | |
| clarkb | that gives us more flexibility imo | 17:08 |
| *** ihrachyshka has joined #openstack-infra | 17:08 | |
| mtreinish | lifeless: when you're around later what do you think about doing something like: http://bazaar.launchpad.net/~treinish/subunit/trace-on-import-error/revision/241 | 17:08 |
| *** esker has quit IRC | 17:08 | |
| mtreinish | lifeless: at least until we have a real fix for http://bugs.python.org/issue19746 | 17:09 |
| *** amotoki has quit IRC | 17:09 | |
| *** bhuvan has joined #openstack-infra | 17:09 | |
| *** esker has joined #openstack-infra | 17:09 | |
| *** esker has quit IRC | 17:09 | |
| *** esker has joined #openstack-infra | 17:10 | |
| *** HenryG has quit IRC | 17:10 | |
| *** HenryG has joined #openstack-infra | 17:10 | |
| *** amotoki_ has quit IRC | 17:11 | |
| jeblair | clarkb: yeah, i think on balance we should probably do your list suggestion then. | 17:11 |
| *** _nadya_ has quit IRC | 17:12 | |
| *** talluri has joined #openstack-infra | 17:12 | |
| clarkb | jeblair: left a comment on the specs change. can you see if that makes sense? | 17:12 |
| jesusaurus | clarkb: images are shared cross az? | 17:12 |
| clarkb | jesusaurus: yes | 17:13 |
| *** ewindisch has joined #openstack-infra | 17:14 | |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Support background sync of missing refs https://review.openstack.org/97066 | 17:14 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add dependency navigation https://review.openstack.org/97067 | 17:14 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Remove stray debug line https://review.openstack.org/97068 | 17:14 |
| jeblair | clarkb: so sorry about the missing migration! | 17:14 |
| clarkb | jeblair: its ok I tested on the other computer :) | 17:15 |
| clarkb | fungi: jeblair: I suppose I should start working on an AZ list then? | 17:15 |
| jeblair | clarkb: do you want to do the np change or shall i? | 17:15 |
| clarkb | I can do it. One last question. Should subnodes live in the same AZ? I am leaning towards yes | 17:16 |
| jeblair | clarkb: sounds reasonable | 17:16 |
| *** mmaglana has quit IRC | 17:17 | |
| fungi | yeah, probably, for sanity more than anything i think (i expect it would make the change smaller) | 17:17 |
| *** ihrachyshka has quit IRC | 17:19 | |
| *** mmaglana has joined #openstack-infra | 17:20 | |
| jesusaurus | though subnodes being in other azs would uncover more issues | 17:21 |
| jeblair | jesusaurus: i think on balance we probably want them topologically close. | 17:22 |
| fungi | jesusaurus: well, with the current nodepool design a subnode is useless without the head node to which it's associated, so if an az goes offline then we | 17:22 |
| fungi | 're better off losing them all together | 17:22 |
| jesusaurus | jeblair: good point | 17:23 |
| clarkb | jeblair: fungi: actually sharing AZs with subnodes may make the code more complicated. Nodepool._run() iterates in a way that decouples the two sets | 17:23 |
| fungi | thinking to the distant future where we have these in use and doing something, the more distributed between azs a multi-node group is, the more groups we lose to sudden loss of a single az | 17:23 |
| jesusaurus | fungi: i wasnt thinking of errors from losing an az, i was thinking about assumptions of being on the same subnet | 17:23 |
| clarkb | actually wait, I have a node object there, if we assign an AZ to it that should be fine | 17:23 |
| jeblair | clarkb: yeah that. :) | 17:23 |
| bodepd | jeblair: to CLA or not to CLA for openstack-ifra? | 17:27 |
| *** arunkant has joined #openstack-infra | 17:27 | |
| fungi | bodepd: generally not | 17:27 |
| bodepd | fungi: ok, I'm not changing it again after this :) | 17:28 |
| *** fbo is now known as fbo_away | 17:28 | |
| *** Ajaeger1 has joined #openstack-infra | 17:28 | |
| fungi | bodepd: there are a few projects in openstack-infra which specify receive.requireContributorAgreement but the majority do not | 17:30 |
| jeblair | apparently 33% of our repos require the cla. we're a bit inconsistent | 17:30 |
| *** dims has quit IRC | 17:30 | |
| *** atiwari has quit IRC | 17:30 | |
| jeblair | there are hysterical raisins for that. | 17:31 |
| fungi | bodepd: and for something being split out of openstack-infra/config, we shouldn't add receive.requireContributorAgreement because http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/gerrit/acls/openstack-infra/config.config also lacks it | 17:31 |
| jeblair | fungi: that's as good an argument as any. :) | 17:31 |
| * fungi would come up with other convincing arguments too if needed ;) | 17:32 | |
| *** nati_ueno has joined #openstack-infra | 17:32 | |
| *** gokrokve has joined #openstack-infra | 17:32 | |
| arunkant | hi...seeing this build failure when change was set to merge..https://review.openstack.org/#/c/95842/4 It appear to hit some known bugs as mentioned by elastic-recheck. How to address this? | 17:32 |
| jeblair | bodepd: that work for you? | 17:33 |
| *** denis_makogon_ has joined #openstack-infra | 17:35 | |
| *** atiwari has joined #openstack-infra | 17:35 | |
| fungi | mordred_phone: dhellmann: if pbr>=0.6,<1.0 is already specified in setup_requires for a project, should we still be adding it to requrements.txt as well? | 17:35 |
| *** amcrn has joined #openstack-infra | 17:35 | |
| *** timrc is now known as timrc-afk | 17:35 | |
| *** zzelle_ has joined #openstack-infra | 17:36 | |
| mordred_phone | fungi: depends | 17:36 |
| mordred_phone | if the project uses pbr at runtjme | 17:36 |
| fungi | mordred_phone: ahh, okay. so the specs project definitely doesn't have a run-time dependency on that | 17:37 |
| fungi | noted | 17:37 |
| mordred_phone | fungi: that said. setup_requires should not be versioned | 17:37 |
| bodepd | jeblair: yes. it sounds like a decision. I woudl have preferred CLAs, but I'm much happier with a decision | 17:37 |
| anteaya | jeblair fungi that makes life difficult for me reviewing acl patches | 17:37 |
| fungi | mordred_phone: adding a note about that as well | 17:37 |
| *** denis_makogon_ is now known as denis_makogon | 17:37 | |
| *** maxbit has quit IRC | 17:38 | |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-pip as a unique project https://review.openstack.org/96522 | 17:38 |
| fungi | bodepd: you prefer our cla? i'm moderately frightened | 17:38 |
| jeblair | anteaya: yeah, it's tricky and i don't think we know what the right answer is. we may need to turn them all on for infra projects some day. or, hopefully, we may get to turn them all off. | 17:38 |
| anteaya | jeblair: stackforge projects have requireCLA = true | 17:39 |
| *** dkliban_lunch is now known as dkliban | 17:39 | |
| jeblair | anteaya: i don't consider the CLA a review criteria for stackforge projects | 17:39 |
| anteaya | I don't ask stackforge projects to change it | 17:39 |
| anteaya | if they have it or not I just leave it | 17:39 |
| *** sandywalsh_ has quit IRC | 17:39 | |
| anteaya | but bodepd had it = true and was -1'd for it | 17:39 |
| fungi | hopefully only stackforge projects which are trying to become official components of openstack itself are bothering with applying our cla | 17:39 |
| anteaya | I don't do that to stackforge projects | 17:40 |
| anteaya | that is what I assume as well, fungi | 17:40 |
| jeblair | that's not what i assume, but it is what i hope. | 17:40 |
| anteaya | assume/hope | 17:40 |
| anteaya | either way, whatever they have, I just leave it if it is stackforge | 17:41 |
| fungi | yes, i ASSUME many of then do it as a cargo-cult behavior | 17:41 |
| anteaya | anything under openstack, I request CLA = true in the acl | 17:41 |
| bodepd | fungi: CLAs are there to help resolve copyright concerns and other lagal nonsense. | 17:41 |
| bodepd | I'm happy with anything that makes code more legit from a legal perspective | 17:41 |
| fungi | bodepd: some clas are, i'm unconvinced our cla is | 17:42 |
| bodepd | I'm also becoming a suit these days :) | 17:42 |
| anteaya | and i appreciate your quick turn around on your patches, bodepd | 17:42 |
| jeblair | our cla is legal nonsense | 17:42 |
| anteaya | that maybe the case and there is a process for that | 17:42 |
| bodepd | fungi: well, I guess part of that was me assuming that the OS CLA had some value other than generating legal revenue ;) | 17:42 |
| anteaya | but in regards to what projects should do about it, my divinding line has been the use of the openstack namespace | 17:42 |
| *** _nadya_ has joined #openstack-infra | 17:43 | |
| jeblair | bodepd: i'm pretty sure that's its only value :/ | 17:43 |
| fungi | anteaya: yes, that's what i've been doing as well | 17:43 |
| anteaya | so getting -1'd for including it when using the openstack namespace confuses me | 17:43 |
| anteaya | since I now have no direction on how to review future patches | 17:43 |
| anteaya | and "ask in channel" become a bottleneck | 17:44 |
| jeblair | bodepd: in all honesty, in your capacity as a suit if you are considering CLAs for open source projects, i would be happy to share some resources and introduce you to some folks who hold very sensible, legally supportable, non-revenue-generating opinions on the subject. | 17:44 |
| fungi | anteaya: what openstack namespace? or did you mean you were doing it for openstack-infra and openstack-dev too? | 17:44 |
| anteaya | yes | 17:44 |
| anteaya | I have been | 17:44 |
| *** gokrokve has quit IRC | 17:45 | |
| jeblair | bodepd: i care a lot about free software and i want the contribution barrier to be minimal, which is why i prefer no CLAs or something like the kernel DCO model | 17:45 |
| *** gokrokve has joined #openstack-infra | 17:45 | |
| jesusaurus | jeblair: is that the signed-off-by process? | 17:45 |
| jeblair | jesusaurus: yes | 17:45 |
| fungi | anteaya: ahh, i'd only been highlighting it as considered necessary for actual runnable code and libraries (but not documentation-only projects) explicitly within the openstack/* namespace (treating it optional in stackforge and suggesting against it in openstack-dev and openstack-infra but still optional) | 17:46 |
| jeblair | jesusaurus: (dco == developer certificate of origin; https://www.kernel.org/doc/Documentation/SubmittingPatches section 12) | 17:46 |
| anteaya | fungi: why suggesting against it in the -dev and -infra spaces? | 17:47 |
| fungi | anteaya: i hadn't seen any implication that it was strictly required except for software components of openstack itself | 17:48 |
| anteaya | why would it not be required for openstack-infra and -dev? | 17:49 |
| fungi | and we generally put things in openstack-dev or openstack-infra which are supporting cast (used to develop and test openstack or to support its community and ecosystem, but not which would be something people would run and call an openstack cloud or an official channel for controlling one) | 17:49 |
| anteaya | yes, that is what I understood to be the case | 17:50 |
| anteaya | though the infra projects are all in the governance/programs.yaml repo | 17:50 |
| anteaya | or should be | 17:51 |
| anteaya | and openstack-dev/devstack is a program | 17:51 |
| fungi | i fail to understand what being under the governance of the tc has to do with a contributor license agreement | 17:51 |
| clarkb | ok tox is running on what I have so far. | 17:51 |
| *** timrc-afk is now known as timrc | 17:51 | |
| anteaya | that is the reference I use for what constitutes an official openstack program | 17:51 |
| anteaya | openstack-infra is an openstack program | 17:52 |
| fungi | sure, official openstack program is broader than "something i would run and market as an openstack cloud" | 17:52 |
| *** _nadya_ has quit IRC | 17:52 | |
| anteaya | so what is the scope of the cla? | 17:52 |
| anteaya | and its expected usage | 17:53 |
| *** ihrachyshka has joined #openstack-infra | 17:53 | |
| anteaya | I had thought it was to be used with openstack programs | 17:53 |
| fungi | the lawyers who drafted the cla expressed concern about the openstack product (the cloud software competing with amazon et al) being compromised by bad actors within the contributing community, effectively tainting it for other member companies who also want to run and sell services on it | 17:53 |
| anteaya | okay | 17:55 |
| fungi | e.g. foocorp wants some additional assurance that barinc won't sneakily introduce intellectual property into nova which prevents them from being able to continue to make money off of it | 17:55 |
| *** nati_ueno has quit IRC | 17:55 | |
| anteaya | hence the third-party party | 17:55 |
| fungi | and i'm as of yet unconvinced it even does that | 17:55 |
| *** nati_ueno has joined #openstack-infra | 17:55 | |
| anteaya | it doesn't | 17:55 |
| anteaya | so in anycase, we have a set of rules which we are trying to change | 17:56 |
| anteaya | and in the meantime we are still stuck with | 17:56 |
| anteaya | and what I do, I would prefer to do consistently | 17:56 |
| fungi | anyway, it's been my understanding that only components of an openstack cloud, maybe also dependent libraries and official client interfaces, are of interest to those who want the cla enforced | 17:56 |
| *** dims has joined #openstack-infra | 17:56 | |
| *** apevec has quit IRC | 17:56 | |
| anteaya | so that if I post a review to bodepd he doesn't get -1'd down the road for impliementing the thing I asked him to change | 17:56 |
| *** ihrachyshka has quit IRC | 17:57 | |
| anteaya | fungi: what documentation could I read to validate that, do you know? | 17:57 |
| *** salv-orlando has joined #openstack-infra | 17:57 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Add an option to mark uploaded patchsets as WIP https://review.openstack.org/96528 | 17:57 |
| fungi | anteaya: you can reread the cla and form your own opinions as to what it says, or consult your legal counsel. you are free to disagree and/or -1 any changes you see fit | 17:58 |
| fungi | i'm merely trying to explain my interpretation of the same document | 17:58 |
| anteaya | so there is no documentation outside the cla that indicates its scope or use, just the cla? | 17:58 |
| anteaya | fungi: I understand | 17:58 |
| anteaya | and I appreciate it | 17:58 |
| *** sandywalsh has joined #openstack-infra | 17:58 | |
| *** mbacchi has quit IRC | 17:59 | |
| fungi | anteaya: also https://wiki.openstack.org/wiki/Governance/Foundation/Bylaws#ARTICLE_VII._INTELLECTUAL_PROPERTY_POLICY | 18:00 |
| *** Ryan_Lane has joined #openstack-infra | 18:00 | |
| *** nati_ueno has quit IRC | 18:00 | |
| anteaya | thanks | 18:00 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists https://review.openstack.org/97303 | 18:00 |
| clarkb | jeblair: fungi ^ | 18:00 |
| *** dims has quit IRC | 18:01 | |
| *** dkranz has quit IRC | 18:01 | |
| fungi | anteaya: it's sort of an ouroboros, in that the bylaws define that the foundation will accept contributions under a cla, and the cla grants contributions to the foundation | 18:01 |
| *** marcoemorais1 has joined #openstack-infra | 18:01 | |
| *** marcoemorais1 has quit IRC | 18:02 | |
| clarkb | note I don't haev amysql daemon on my laptop (and don't really want one) so we should make suer that passes unittests | 18:02 |
| *** marcoemorais1 has joined #openstack-infra | 18:02 | |
| *** blamar has joined #openstack-infra | 18:02 | |
| *** marcoemorais has quit IRC | 18:02 | |
| *** dkranz has joined #openstack-infra | 18:04 | |
| anteaya | trinaths: please join us in #openstack-meeting for the third-party meeting | 18:05 |
| anteaya | trinaths: we need to address some of your questions | 18:05 |
| fungi | clarkb: would it make sense to add an availability-zone list to one of the test yaml configs, at a minimum (or better, a test that they work)? | 18:06 |
| *** srenatus has quit IRC | 18:06 | |
| jeblair | fungi: i just got an interesting bounce message from static with output from the log archive maintenance script | 18:06 |
| *** srenatus has joined #openstack-infra | 18:06 | |
| *** julim has quit IRC | 18:06 | |
| clarkb | probably. I suppose I should find a better test machine and do that | 18:07 |
| * fungi looks | 18:07 | |
| clarkb | fungi: though the mocking is rudimentary iirc | 18:07 |
| fungi | jeblair: bounce for my address? | 18:07 |
| sdague | anteaya: yeh, I kicked off the legal thread / hornet nest this weekend because of our conversation on it on friday | 18:08 |
| jeblair | fungi: i didn't pay attention to whose bounce; it was a msg too big bounce -- it was the contents of the original message i found interesting | 18:08 |
| *** ihrachyshka has joined #openstack-infra | 18:08 | |
| anteaya | sdague: yeah, I felt that | 18:08 |
| *** zul has quit IRC | 18:08 | |
| jeblair | fungi: i think we all got the bounce though, so you should be able to see it | 18:09 |
| *** julim has joined #openstack-infra | 18:09 | |
| jeblair | fungi: the bounce is Message-Id: <E1WrW01-0003ja-PR@static.openstack.org> | 18:09 |
| fungi | jeblair: yeah, reading it now | 18:09 |
| fungi | jeblair: looks like it tried to remove directories which were not yet empty | 18:10 |
| *** SumitNaiksatam has quit IRC | 18:10 | |
| fungi | rereading the current script now and checking some of the examples on disk to see if i can spot why | 18:11 |
| fungi | -type d -not -empty -mtime +1 -execdir rmdir {} \; | 18:12 |
| sdague | anteaya: my -1s on the infra split out I thought were pretty well reasoned in that there was not a CLA in the parent repo, so it didn't seem like there should be on in the split outs | 18:12 |
| *** mbacchi has joined #openstack-infra | 18:12 | |
| fungi | that doesn't look right. maybe i inadvertently introduced/orphaned a stray -not in that | 18:12 |
| fungi | checking | 18:12 |
| *** yfried has joined #openstack-infra | 18:12 | |
| *** zul has joined #openstack-infra | 18:13 | |
| *** pblaho has quit IRC | 18:13 | |
| *** dims has joined #openstack-infra | 18:13 | |
| fungi | yep! | 18:13 |
| fungi | patch on the way | 18:13 |
| fungi | somehow that wasn't exposed when i tested it manually | 18:13 |
| mordred | clarkb: nice patch | 18:14 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add project for zuul https://review.openstack.org/96542 | 18:14 |
| mordred | jeblair, clarkb: ^^ I know it doesn't happen often - but since we've got it in place other places now successfully .. should we alembic nodepool? | 18:14 |
| *** nati_ueno has joined #openstack-infra | 18:14 | |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-pip as a unique project https://review.openstack.org/96522 | 18:15 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add project for jenkins https://review.openstack.org/96544 | 18:16 |
| jeblair | mordred: oh. hm. probably not a bad idea. | 18:17 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add project for zuul https://review.openstack.org/96542 | 18:17 |
| mordred | jeblair: k. cool. I'll put that on my plane hacking list if nobody gets to it | 18:18 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Fix empty directory removal for log expiration https://review.openstack.org/97307 | 18:18 |
| fungi | jeblair: ^ | 18:18 |
| lbragstad | sdague: mrodden and I had a quick question for you on the whole gerrit-powered-agendas subject. You had some ideas about publish to the wiki correct? | 18:18 |
| *** _nadya_ has joined #openstack-infra | 18:19 | |
| sdague | lbragstad: I was just going to suggest be ghetto and mechanize | 18:19 |
| clarkb | mordred: jeblair do you know if the sqlalchemy <-> mysql translation of None to and from the mysql type does the right thing? | 18:20 |
| jeblair | mordred: you'll note that i did not do it the same way in gertty as storyboard does. i thought it would be worth starting from first principles in gertty because i found the storyboard implementation a bit hard to follow. i have learned quite a bit in the process but have not yet had a chance to apply what i learned to storybard. | 18:20 |
| clarkb | I am assuming it will set teh value to null and everything will be fine | 18:20 |
| jeblair | clarkb: None==null yes | 18:20 |
| mordred | jeblair: awesome. I'll look at gertty then | 18:20 |
| *** marcoemorais1 has quit IRC | 18:20 | |
| *** marcoemorais has joined #openstack-infra | 18:20 | |
| lbragstad | sdague: ok, currently, the way the team has it, the publish functionality is something that is controlled by the end-user... which I don't think is necessarily correct since it should only publish to the openstack wiki on post jenkins jobs, after an approval... right? | 18:21 |
| jeblair | storybard, btw, was an englishman who wrote a bunch of plays during the elizibethan era | 18:21 |
| sdague | lbragstad: that seems reasonable | 18:21 |
| jeblair | clarkb: reviewing now | 18:21 |
| lbragstad | sdague: so would we do that with a python task or... any other good ideas on how to best accomplish that with a jenkins post job? | 18:21 |
| *** e0ne has joined #openstack-infra | 18:22 | |
| mkoderer | jeblair: mtreinish: hi are you around for a short hangout session? | 18:22 |
| mkoderer | about the mid cycle summit.. | 18:22 |
| sdague | lbragstad: now you are beyond my general knowledge on how to do that best | 18:22 |
| sdague | but plenty of folks in here should know | 18:22 |
| bodepd | the fact that there was a merge conflict b/c someone was able to get puppet-yum excepted before me really indicates I'm doing something wrong :) | 18:22 |
| jeblair | mkoderer: oh, sorry, i don't use google hangouts. happy to irc or phone call though. | 18:23 |
| lbragstad | sdague: ok, sounds good, thanks! | 18:23 |
| mkoderer | jeblair: ok, phone is also ok for me | 18:23 |
| *** ihrachyshka has quit IRC | 18:25 | |
| *** _nadya_ has quit IRC | 18:25 | |
| *** ihrachyshka has joined #openstack-infra | 18:25 | |
| mtreinish | mkoderer: yeah I'm around | 18:25 |
| *** _nadya_ has joined #openstack-infra | 18:25 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists https://review.openstack.org/97303 | 18:25 |
| clarkb | jeblair: fungi ^ now with tests that I haven't run locally | 18:26 |
| *** sarob_ has joined #openstack-infra | 18:26 | |
| *** ihrachyshka has quit IRC | 18:26 | |
| fungi | clarkb: i know someone who will run them for you | 18:26 |
| mkoderer | mtreinish: ok cool.. we can do it via IRC.. I just thought it would be faster with voice :) | 18:26 |
| *** amcrn has quit IRC | 18:27 | |
| *** amcrn has joined #openstack-infra | 18:28 | |
| *** ociuhandu has quit IRC | 18:28 | |
| clarkb | fungi: unfortunately that someone is so far behind :) | 18:28 |
| *** matjazp has quit IRC | 18:28 | |
| *** sarob has quit IRC | 18:29 | |
| *** homeless_ has joined #openstack-infra | 18:30 | |
| *** _nadya_ has quit IRC | 18:30 | |
| mordred | clarkb: here's hoping that better distribution across the three azs will help the errors | 18:30 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: features: do not drop ceilometer when using grenade https://review.openstack.org/96518 | 18:31 |
| mrodden | lbragstad: i'd suggest prototyping it as python | 18:32 |
| mrodden | and we can figure out what belongs where later | 18:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Add #openstack-sahara to status bot https://review.openstack.org/97247 | 18:33 |
| mordred | clarkb: WOW. the current error spike is super impressive | 18:33 |
| lbragstad | mrodden: ok | 18:33 |
| *** homeless has quit IRC | 18:33 | |
| jeblair | clarkb: okay, finally finished leaving comments on PS1 of your change :) | 18:33 |
| mrodden | not sure if moin moin has an actual API or if we're gonna have to write a web crawler | 18:33 |
| clarkb | jeblair: looking | 18:33 |
| tteggel | mordred: https://community.hpcloud.com/status/incident/2612 ;( | 18:34 |
| tteggel | mordred: resolved now | 18:34 |
| *** ildikov_ is now known as ildikov_afk | 18:35 | |
| mordred | tteggel: neat. honestly, the spike is still impressive :) | 18:36 |
| *** matjazp has joined #openstack-infra | 18:36 | |
| fungi | mrodden: lbragstad: we've been on mediawiki for over a year now. moin moin is history | 18:37 |
| mordred | jeblair: can we april fools a storybard install? | 18:37 |
| *** [1]trinaths has joined #openstack-infra | 18:37 | |
| *** matjazp has quit IRC | 18:38 | |
| *** zzelle has quit IRC | 18:38 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tempest jobs to dib and os-* tools https://review.openstack.org/94827 | 18:39 |
| *** rlandy_ has joined #openstack-infra | 18:41 | |
| *** trinaths has quit IRC | 18:41 | |
| *** [1]trinaths is now known as trinaths | 18:41 | |
| *** ihrachyshka has joined #openstack-infra | 18:41 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists https://review.openstack.org/97303 | 18:42 |
| clarkb | jeblair: ^ that addresses your comments | 18:42 |
| *** rlandy__ has quit IRC | 18:43 | |
| *** srenatus has quit IRC | 18:43 | |
| *** srenatus has joined #openstack-infra | 18:44 | |
| *** fbo_away is now known as fbo | 18:44 | |
| *** reed has joined #openstack-infra | 18:46 | |
| *** homeless_ has quit IRC | 18:49 | |
| jeblair | clarkb: thanks, lgtm except can you add a sentence to the docs about the new option? | 18:49 |
| *** andreykurilin has joined #openstack-infra | 18:49 | |
| jeblair | (sorry i forgot that the first time) | 18:49 |
| *** openstackstatus has quit IRC | 18:50 | |
| anteaya | openstackstatus has quit | 18:51 |
| clarkb | jeblair: sure | 18:51 |
| *** openstackstatus has joined #openstack-infra | 18:51 | |
| *** ChanServ sets mode: +v openstackstatus | 18:51 | |
| *** rlandy__ has joined #openstack-infra | 18:51 | |
| *** bauzas has quit IRC | 18:52 | |
| *** homeless has joined #openstack-infra | 18:53 | |
| *** rlandy_ has quit IRC | 18:54 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists https://review.openstack.org/97303 | 18:55 |
| clarkb | and done | 18:55 |
| *** hogepodge has quit IRC | 18:55 | |
| openstackgerrit | A change was merged to openstack/requirements: Move oslotest to test catalog https://review.openstack.org/96822 | 18:58 |
| *** nati_uen_ has joined #openstack-infra | 18:58 | |
| *** _nadya_ has joined #openstack-infra | 18:58 | |
| *** nati_uen_ has quit IRC | 18:59 | |
| *** mrda-away is now known as mrda | 18:59 | |
| *** hashar has joined #openstack-infra | 18:59 | |
| *** nati_uen_ has joined #openstack-infra | 18:59 | |
| *** esker has quit IRC | 19:00 | |
| *** nati_ueno has quit IRC | 19:02 | |
| anteaya | jeblair mtreinish so when the question came up, can third-party ci folks attend the infra/qa meetup, I told them the intent is to encourage new core reviewers and if they are willing to commit to learning how to review 30 patches in 30 days (as a minimum) they could request to attend | 19:02 |
| anteaya | jeblair mtreinish and if they wanted to learn how to set up their systems, they needed to ask questions and participate in the meeting space on irc | 19:03 |
| pleia2 | everyone should attend our meetings :) | 19:03 |
| *** tristanC has quit IRC | 19:03 | |
| *** lttrl has joined #openstack-infra | 19:03 | |
| Longgeek | yes. | 19:04 |
| anteaya | pleia2: meetup | 19:04 |
| anteaya | http://eavesdrop.openstack.org/meetings/third_party/2014/third_party.2014-06-02-18.00.log.html | 19:04 |
| pleia2 | anteaya: ooh, the one next month | 19:04 |
| anteaya | timestamp 18:10:03 | 19:04 |
| anteaya | yeah | 19:04 |
| jeblair | clarkb: AttributeError: 'TestNodepool' object has no attribute 'aseertEqual' | 19:04 |
| anteaya | I knew there was going to be confusion and there was | 19:04 |
| fungi | aseert! aseert! | 19:04 |
| anteaya | but i think I dealt with it clearly | 19:04 |
| *** trinaths has quit IRC | 19:04 | |
| *** sanjiva has joined #openstack-infra | 19:05 | |
| *** lcostantino has quit IRC | 19:05 | |
| mtreinish | anteaya: I'm not sure about putting hard restrictions on it. But, it's definitely not about learning how to setup a third party ci system. | 19:05 |
| anteaya | mtreinish: I needed to put something around it | 19:06 |
| anteaya | mtreinish: if you don't want to enforce that, that is your call | 19:06 |
| mtreinish | we're going to have a day or 2 of bootstrapping to explain the architecture of infra and qa, and how they work around it | 19:06 |
| *** esker has joined #openstack-infra | 19:06 | |
| clarkb | jeblair: gah thank you | 19:06 |
| anteaya | but they read it as, we can show up and get our systems working | 19:06 |
| pleia2 | speaking of meeting meetings, might be a little late tomorrow because I have an appointment in the morning | 19:06 |
| anteaya | mtreinish: since that is how they see everythign | 19:06 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists https://review.openstack.org/97303 | 19:06 |
| anteaya | pleia2: k, hope it is good news | 19:07 |
| *** yfried has quit IRC | 19:07 | |
| *** yfried has joined #openstack-infra | 19:07 | |
| mtreinish | yeah, that's definitely not what it is | 19:07 |
| anteaya | mtreinish: nod | 19:07 |
| anteaya | yeah these folks want things and have no immediate use for it | 19:07 |
| anteaya | I'm glad they are speaking up | 19:07 |
| *** duran has quit IRC | 19:07 | |
| anteaya | sometimes it is just frustrating | 19:08 |
| anteaya | I'm going to take another walk before it rains | 19:08 |
| bookwar | anteaya: actually i am considering to participate, because I want to learn more about how openstack infrastructure works | 19:08 |
| anteaya | bookwar: know that this is not designed to answer questions about how to set up your third party system | 19:08 |
| anteaya | as long as you are prepared to follow the agenda and sprit of the meetup you can request to attend | 19:09 |
| Longgeek | anteaya: I think I have closed the start job Comments. | 19:09 |
| anteaya | Longgeek: test your system on the sandbox repo | 19:10 |
| Longgeek | anteaya: ok | 19:10 |
| *** bauzas has joined #openstack-infra | 19:10 | |
| anteaya | I want to see at least 5 comments from your system on the sandbox repo before we discuss anything else | 19:10 |
| anteaya | I played all sorts of whack a mole with third party people demanding to attend the neutron tempest code sprint who had no interest in actually contributing to neutron or tempest that I have no interest in doing that again | 19:11 |
| *** marcoemorais has quit IRC | 19:12 | |
| *** wenlock has quit IRC | 19:13 | |
| zaro | fungi: i was going to pick this up again. wanted to double check with you whether there's anything larger required than just cleaning from your suggestions. https://review.openstack.org/#/c/47937 | 19:13 |
| anteaya | didn't even make it 10 metres (rain) | 19:13 |
| Longgeek | anteaya: Has confirmed the closure | 19:13 |
| anteaya | Longgeek: post the url of one of the sandbox patches | 19:13 |
| *** otherwiseguy has quit IRC | 19:14 | |
| *** praneshp has joined #openstack-infra | 19:14 | |
| *** radez_g0n3 is now known as radez | 19:15 | |
| *** tristanC has joined #openstack-infra | 19:15 | |
| Longgeek | anteaya: patch link: https://review.openstack.org/#/c/96714/ | 19:16 |
| *** mbacchi has quit IRC | 19:16 | |
| *** andreaf has quit IRC | 19:17 | |
| *** arnaud__ has quit IRC | 19:19 | |
| anteaya | Longgeek: okay so well done you have successfully gotten yourself to one commet per patchset | 19:20 |
| anteaya | Longgeek: do you remember what you did to accomplish that? | 19:20 |
| *** sarob_ has quit IRC | 19:20 | |
| Longgeek | anteaya: yes | 19:20 |
| anteaya | good, write it down | 19:20 |
| Longgeek | anteaya: remember | 19:20 |
| anteaya | you are going to present it at next week's meeting | 19:20 |
| clarkb | jeblair: fungi: I am going to wander and find lunch then an outdoor hacking spot | 19:21 |
| *** sarob has joined #openstack-infra | 19:21 | |
| clarkb | will be afkish for the next bit. feel free to edit my nodepool change if you need to | 19:21 |
| anteaya | noone spoke up to help you when I asked, so now you can teach them all how to do it | 19:21 |
| Longgeek | anteaya: No problem | 19:21 |
| anteaya | Longgeek: now what projects do you want to test? | 19:21 |
| *** yfried has quit IRC | 19:21 | |
| anteaya | pick one to start | 19:21 |
| *** yfried has joined #openstack-infra | 19:21 | |
| Longgeek | anteaya: devstack | 19:21 |
| *** _nadya_ has quit IRC | 19:21 | |
| anteaya | great | 19:21 |
| clarkb | also will need to whip up the corresponding infra/config change to add the az's | 19:22 |
| *** lcostantino has joined #openstack-infra | 19:22 | |
| anteaya | go into #openstack-qa and look for mtreinish | 19:22 |
| Longgeek | anteaya: Write more appropriate in where? | 19:22 |
| *** marcoemorais has joined #openstack-infra | 19:22 | |
| anteaya | Longgeek: make your personal notes and you will present them at next week's third-party meeting | 19:22 |
| Longgeek | anteaya: ok | 19:22 |
| anteaya | ask mtreinish if he will allow you to test and comment on devstack | 19:23 |
| anteaya | show him your sandbox repo | 19:23 |
| anteaya | address any questions or concerns he has | 19:23 |
| anteaya | and if you don't know how, ask me | 19:23 |
| Longgeek | anteaya, Good, thank you. | 19:23 |
| anteaya | Longgeek: if he let's comment on devstack patches, stay with only that project before adding any others | 19:24 |
| anteaya | come and talk to me before adding any other projects to what you test | 19:24 |
| Longgeek | anteaya: ok, clear | 19:25 |
| *** sarob has quit IRC | 19:25 | |
| *** sarob_ has joined #openstack-infra | 19:25 | |
| mtreinish | anteaya: heh, I'm not ptl of devstack :) | 19:25 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add availability zones to hpcloud region b https://review.openstack.org/97327 | 19:25 |
| *** nati_uen_ has quit IRC | 19:25 | |
| clarkb | jeblair: fungi mordred ^ that is the corresponding config chaneg and now to find food | 19:26 |
| *** kaitlin-farr has joined #openstack-infra | 19:26 | |
| anteaya | mtreinish: though dtroyer is away | 19:26 |
| Longgeek | mtreinish: Who i ptl of devstack. | 19:26 |
| clarkb | you can check the az names with `nova availability-zone-list` | 19:26 |
| anteaya | mtreinish: and I am getting him into the habit of going into another channel and seeking permission to test | 19:26 |
| *** marcoemorais1 has joined #openstack-infra | 19:26 | |
| *** marcoemorais1 has quit IRC | 19:27 | |
| *** nati_uen_ has joined #openstack-infra | 19:27 | |
| *** marcoemorais2 has joined #openstack-infra | 19:27 | |
| *** marcoemorais has quit IRC | 19:27 | |
| fungi | zaro: 47937 may need some updating to reflect changes which were introduced as part of the gerrit upgrade. also it's been tabled long enough we should have infra and the vmt confirm this is still the direction we want to go (just so we don't waste more of your time). aside from that, yeah really just cleanup | 19:28 |
| sdague | so my question is mostly I don't know what thstack.com is | 19:28 |
| anteaya | sdague: good question | 19:28 |
| sdague | so I'm not sure why it would be running on devstack changes | 19:28 |
| anteaya | nether do I | 19:28 |
| *** xyang1 has joined #openstack-infra | 19:29 | |
| anteaya | but at least it isn't commenting twice per patchset, which was my goal | 19:29 |
| sdague | which was the reason I raised the question last week | 19:29 |
| anteaya | yup | 19:29 |
| anteaya | nobody knows what it is testing | 19:29 |
| anteaya | and I have been so busy teaching ettiquette I haven't had time to ask | 19:30 |
| *** talluri has quit IRC | 19:30 | |
| xyang1 | anteaya: hi, so my question is if we have multiple CI systems, should we have multiple upstream service accounts, or just one? | 19:30 |
| anteaya | well for our sake it is much easier if you can do just one ci account | 19:30 |
| *** andreaf has joined #openstack-infra | 19:31 | |
| anteaya | and it will be easier for you if you have a dedicated person to stay on irc to monitor your account and deal with issues and questions as they arise | 19:31 |
| fungi | xyang1: multiple ci systems won't be able to vote independently of one another if they use the same account | 19:31 |
| xyang1 | sure, so we should be able to submit test results from multiple CI systems, using the same account, right? that | 19:31 |
| anteaya | xyang1: what projects do you want to test? | 19:32 |
| xyang1 | the reason for multiple CI systems is that we have multiple cinder drivers | 19:32 |
| xyang1 | each driver will be tested at a different CI system | 19:32 |
| anteaya | how many cinder drivers do you have? | 19:32 |
| xyang1 | 4 | 19:32 |
| fungi | if they're testing different things and aren't controlled by the same zuul or gerrit trigger plugin instance, then one succeeding on a patchset after the other fails on the same patchset may mask failure results | 19:32 |
| anteaya | ah very true | 19:32 |
| anteaya | sigh | 19:32 |
| anteaya | this is going to get nice and confusing | 19:33 |
| xyang1 | we don't have one central place that can test all 4 drivers for Juno | 19:33 |
| xyang1 | hopefully we can consolidate in the future | 19:33 |
| zaro | fungi: what's the path to confirmation? | 19:34 |
| xyang1 | each Cinder driver works with a different storage platform. So results could be different | 19:34 |
| fungi | on the other hand if they're just four jobs testing four different things, you should try to engineer it so that they can combine reporting if possible (for example upstream zuul runs multiple jobs on a change, possibly using different jenkins masters entirely, then waits for them all to complete before posting an aggregated vote based on the results of all the jobs it ran) | 19:34 |
| *** homeless_ has joined #openstack-infra | 19:35 | |
| xyang1 | fungi: there will be 4 CI systems at different labs | 19:36 |
| fungi | zaro: remind me tomorrow morning, and lets try to get it onto the infra team meeting agenda as a quick sync-up topic. i'll make sure to confirm with the rest of the vmt today, though ttx won't be around to answer until tomorrow i think | 19:36 |
| fungi | xyang1: yeah, and if you can't run one zuul instance somewhere which has access to talk to systems at all 4 of those labs, then that would be a bit of a blocker for using a shared account for them | 19:36 |
| fungi | xyang1: though it's perhaps something to shoot for as a future goal | 19:37 |
| clarkb | fungi: in theory if all 4 can talk to gerrit they can talk to a central zuul | 19:37 |
| *** homeless has quit IRC | 19:37 | |
| fungi | clarkb: yep | 19:37 |
| clarkb | fungi: they are both outbound connections | 19:37 |
| clarkb | I neglected to do a gertty sync this morning | 19:37 |
| clarkb | I may just give up on that and do lunch anyways | 19:38 |
| sdague | are things just flushing in zuul at this point, and we just need to wait for backlog to clear? | 19:38 |
| lifeless | mtreinish: tht looks like a no-op to me :) | 19:38 |
| lifeless | mtreinish: that said, the layer to carry a local fix is testtools and there is a PR there | 19:39 |
| clarkb | sdague: we don't have enough nodes | 19:39 |
| sdague | clarkb: gotcha | 19:39 |
| lifeless | mtreinish: I'd like to see someone put up the fix in Python and us carry a backport, rather than us carry a workaround without the upstream fix being proposed. | 19:39 |
| clarkb | sdague: and the slope on the waiting jobs graph is very positive | 19:39 |
| fungi | clarkb: true, if all the jenkins masters at xyang1's different labs can reach the gearman service on a common zuul, then that should be sufficient | 19:39 |
| xyang1 | fungi: so we should go with 4 accounts for now? and in the future we'd like to consolidate everything in one central place | 19:39 |
| sdague | clarkb: yeh, is that reaction to everything exploding and having to recheck the 451 failures from setup tools? | 19:39 |
| clarkb | sdague: that + hpcloud 1.0 being turned off | 19:40 |
| sdague | or are we looking generally over capacity again? | 19:40 |
| sdague | ah... yeh, that would be an issue | 19:40 |
| clarkb | and 1.1 is unable to provide consistent capacity. A potential fix is up for review now | 19:40 |
| lifeless | bye bye diablo | 19:40 |
| sdague | should we start pruning jobs? | 19:40 |
| clarkb | sdague: not yet. if the nodepool change makes tis better we should be ableto catch up | 19:40 |
| sdague | like the fact that nova is kicking off 2 extra silent jobs on every rev | 19:40 |
| clarkb | we will essentially double our node count | 19:41 |
| sdague | ok, cool | 19:41 |
| *** gyee has quit IRC | 19:41 | |
| mtreinish | lifeless: ok yeah that makes sense | 19:41 |
| clarkb | sdague: we might need to prune if this lasts a whiel | 19:41 |
| lifeless | mtreinish: so yeah, if you want to move it forward - work upstream, I'll happily review there (and help find python-cores) and once you have something /working/ and plausible, we'll carry a copy of it in testtools | 19:42 |
| sdague | clayg: ok, let me know if you need some help selecting stuff to prune | 19:42 |
| *** trinaths has joined #openstack-infra | 19:43 | |
| *** gabriel-bezerra has joined #openstack-infra | 19:43 | |
| *** wenlock has joined #openstack-infra | 19:44 | |
| *** [1]trinaths has joined #openstack-infra | 19:44 | |
| *** e0ne has quit IRC | 19:46 | |
| fungi | xyang1: if you think you can make it work with one account, then that's fine. otherwise request four accounts for the equipment running tests in each of your labs (preferably in a single request) and make sure to include a brief explanation of why you're currently unable to do it with one account (since that's almost certain to get asked again otherwise) | 19:46 |
| fungi | sdague: tab-complete fail? | 19:47 |
| xyang1 | fungi: sure. We are still in the early stage of setting them up. I think we are likely going to request 4 accounts for Juno. moving forward, we hope to consolidate with 1. Thanks! | 19:47 |
| *** trinaths has quit IRC | 19:47 | |
| *** [1]trinaths is now known as trinaths | 19:47 | |
| sdague | fungi: yep | 19:49 |
| sdague | it's unacceptable that there are 3 cla<tab> nicks in here :) | 19:49 |
| anteaya | your the only sd<tab> | 19:50 |
| anteaya | lucky guy | 19:50 |
| *** ominakov has quit IRC | 19:50 | |
| *** nati_uen_ has quit IRC | 19:51 | |
| *** nati_ueno has joined #openstack-infra | 19:51 | |
| fungi | there are only 5 nicks here starting with f (and i'm the only fu, which sounds like a joke but i assure you it's not) | 19:52 |
| *** _nadya_ has joined #openstack-infra | 19:52 | |
| sdague | heh | 19:52 |
| anteaya | ha ha ha | 19:52 |
| *** ArxCruz has joined #openstack-infra | 19:52 | |
| anteaya | it was worth you saying it though | 19:52 |
| * fungi enjoys the thought that when anyone replies to him, they're probably starting by typing fu[tab] | 19:53 | |
| anteaya | fungi: I know I do | 19:53 |
| pleia2 | hehe | 19:54 |
| *** ominakov has joined #openstack-infra | 19:54 | |
| dstufft | fungi: lol I never noticed that | 19:55 |
| *** nati_ueno has quit IRC | 19:56 | |
| pleia2 | fungi: I'm coming to debconf this year (yay!) so if you're planning on doing a "how openstack ci can make debian more awesomer" kind of thing, I'd love to help out | 19:56 |
| fungi | pleia2: ooh, i hadn't even looked to see when the cfp opens/closes | 19:57 |
| fungi | (or whether i'm due to be somewhere else at the same time) | 19:58 |
| *** openstackgerrit has quit IRC | 19:58 | |
| clarkb | pleia2 at psu? | 19:59 |
| pleia2 | clarkb: yep | 19:59 |
| fungi | clarkb: yup | 19:59 |
| pleia2 | fungi: I don't think it has, but I don't keep as close an eye on it as I probably should :) | 19:59 |
| jesusaurus | clarkb: oh, is debconf the conf theyre gonna have at psu? | 20:00 |
| fungi | jesusaurus: yes | 20:00 |
| jesusaurus | cool | 20:00 |
| clarkb | late august it will be nice then | 20:02 |
| mrodden | do we have any nice graphing/reporting tools that could help me answer the question "how long on average do the various chef cookbook jobs each take to run?" | 20:02 |
| mrodden | i could look at the jenkins reports but is there a way to get that aggregated over all jenkins masters? | 20:03 |
| clarkb | mrodden graphite.o.o | 20:04 |
| mrodden | wow cool... | 20:04 |
| *** nati_ueno has joined #openstack-infra | 20:06 | |
| *** gondoi is now known as zz_gondoi | 20:06 | |
| fungi | jesusaurus: clarkb: how accessible is psu from downtown? | 20:06 |
| jeblair | clarkb, fungi: assuming the latest np tests pass, shall i ninja-merge it and the config change? | 20:07 |
| jeblair | (the latest np tests do pass) | 20:07 |
| fungi | jeblair: i think so, given the wait time | 20:07 |
| fungi | and then we need a (probably not-so-graceful) nodepool restart and database update before we add a change to do the thing with the thing, yeah? | 20:07 |
| tchaypo | So we have working setup tools again? | 20:07 |
| jeblair | fungi: yup | 20:08 |
| clarkb | jesusaurus it is downtown | 20:08 |
| fungi | tchaypo: as of the 3.7.1 and 4.0.1 releases earlier today | 20:08 |
| tchaypo | Woo | 20:08 |
| jeblair | i'm guessing gerritbot is dead | 20:08 |
| tchaypo | This is lovely news to wake up to | 20:08 |
| fungi | gah, i can restart it | 20:08 |
| *** amcrn has quit IRC | 20:08 | |
| clarkb | er fungi: it is downtown | 20:09 |
| *** sarob_ has quit IRC | 20:09 | |
| jeblair | those changes have merged | 20:09 |
| fungi | clarkb: smith memorial student union specifically? | 20:09 |
| *** mbacchi has joined #openstack-infra | 20:09 | |
| fungi | heh, it does indeed say "downtown" (ignore me) | 20:09 |
| *** ominakov has quit IRC | 20:10 | |
| jeblair | i'm about to do an extremely ungraceful nodepool restart | 20:10 |
| jeblair | i will delete all of the currently used nodes | 20:10 |
| clarkb | fungi: yup its at broadway and hallish | 20:10 |
| clarkb | jeblair: k | 20:11 |
| *** openstackgerrit has joined #openstack-infra | 20:11 | |
| *** zz_gondoi is now known as gondoi | 20:13 | |
| *** marcoemorais2 has quit IRC | 20:14 | |
| *** marcoemorais has joined #openstack-infra | 20:14 | |
| *** gyee has joined #openstack-infra | 20:15 | |
| jeblair | mordred: there is a server called testmt in the openstackjenkins2 hpcloud account that i will delete | 20:16 |
| fungi | pleia2: yeah, i don't see anything pressing on my schedule for late august, so maybe we should do that. i'll see if my employer will agree to cover my registration and travel | 20:16 |
| pleia2 | fungi: yay, keep me posted :) | 20:17 |
| fungi | pleia2: we should also talk to dds we know who have an interest in our infra tools and who might be attending | 20:17 |
| pleia2 | fungi: if not, I could use your help in preparing something | 20:17 |
| pleia2 | yeah | 20:17 |
| fungi | a few names come to mind | 20:17 |
| pleia2 | great, I should probably get more friendly with them anyway | 20:17 |
| *** pafuent has joined #openstack-infra | 20:18 | |
| *** primemin1sterp has joined #openstack-infra | 20:19 | |
| *** hogepodge has joined #openstack-infra | 20:20 | |
| *** sarob has joined #openstack-infra | 20:20 | |
| reed | can someone help this person here? https://ask.openstack.org/en/question/30423/how-do-i-rebase-my-changes-to-submit-my-first-review/ | 20:20 |
| reed | looks like his git repo is very messed up | 20:20 |
| pafuent | Hi. I'm from Blazar project and I want to add Tempest test to our gate jobs. What should I do? (Of course at Blazar we have some Tempests tests in a contrib directory) | 20:21 |
| *** sarob is now known as sarob_ | 20:22 | |
| *** sarob_ is now known as sarob | 20:22 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: puppetize installation of gerrit third party plugins https://review.openstack.org/91193 | 20:28 |
| andreaf | hi - any infra core around for a tiny review on devstack-gate https://review.openstack.org/#/c/95901/ - it has already a +2 | 20:28 |
| *** trinaths has quit IRC | 20:28 | |
| *** e0ne has joined #openstack-infra | 20:29 | |
| *** melwitt has joined #openstack-infra | 20:30 | |
| fungi | reed: it's the "Could not apply 7304ed8... Initial commit from fork***" which worries me there. i've suggested that they check and make sure they're not trying to submit changes from a clone of a fork of novaclient | 20:33 |
| reed | fungi, thanks | 20:33 |
| *** julim has quit IRC | 20:34 | |
| *** mwagner_lap has quit IRC | 20:36 | |
| fungi | pafuent: it looks like you already have a devstack-based job for blazar at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/blazar.yaml so perhaps you want to consider having it run tempest in that job as well? you may want to ask in #openstack-qa with regards to how to get tempest to run your contrib tests out of tree (from | 20:37 |
| fungi | tempest's perspective). i'm not sure how pluggable it is yet | 20:37 |
| *** Longgeek has quit IRC | 20:37 | |
| *** dims has quit IRC | 20:38 | |
| *** weshay has quit IRC | 20:39 | |
| *** miqui has quit IRC | 20:39 | |
| *** weshay has joined #openstack-infra | 20:39 | |
| *** sarob has quit IRC | 20:40 | |
| *** sarob has joined #openstack-infra | 20:40 | |
| fungi | andreaf: looks fine to me, though if you plan to do anything with grenade, it's probably not deep enough of a change to support the additional overrides you'd need there | 20:40 |
| fungi | specifically where branch selection is concerned | 20:41 |
| *** markmcclain has joined #openstack-infra | 20:41 | |
| fungi | andreaf: oh, i do have one concern actually. your commit message doesn't really mention the git_branch variable you're adding at all, much less the use case for it | 20:42 |
| jeblair | fungi, clarkb: erm, i was overzealous and deleted the long-running slaves. i've started rebuilding those | 20:42 |
| fungi | jeblair: noted. they were long overdue for a refresh anyway. hopefully we didn't miss puppeting something critical on any of them | 20:43 |
| pafuent | fungi: OK, thanks. I'll check that with the Tempest group | 20:43 |
| *** sarob has quit IRC | 20:45 | |
| *** andreykurilin has quit IRC | 20:45 | |
| andreaf | fungi: ok thanks - I will update the comment | 20:46 |
| *** grantbow has quit IRC | 20:46 | |
| fungi | sdague: the reason behind https:// instead of git:// in devstack-gate (following up on your review comment in 95901) is that a lot of people seem to run this behind the corporate iron curtain where anything which isn't a web protocol must be a backdoor channel for criminal activity | 20:47 |
| *** andreykurilin has joined #openstack-infra | 20:47 | |
| *** Sukhdev has joined #openstack-infra | 20:47 | |
| andreaf | fungi: about the use case: I want to be able to run gate in a setup where the "main" branch is not called "master" :) | 20:48 |
| dstufft | fungi: you mean git:// doesn't stand for good internet theieving? | 20:48 |
| fungi | andreaf: i was pretty certain i knew why you wanted it, just prodding you to also mention it in the commit message in case others are less able to guess correctly (also helpful when browsing the commit log) | 20:49 |
| fungi | dstufft: it does now | 20:49 |
| *** yamahata has quit IRC | 20:49 | |
| * fungi retcons it | 20:49 | |
| *** otherwiseguy has joined #openstack-infra | 20:49 | |
| *** yamahata has joined #openstack-infra | 20:50 | |
| fungi | jeblair: do you need help with any of the slave rebuilds? | 20:50 |
| jeblair | it takes a bit longer than i would have expected to delete 600 servers | 20:50 |
| jeblair | puppet agent --environment production --server ci-puppetmaster.openstack.org --no-daemonize --verbose --onetime --pluginsync true --certname mirror27.slave.openstack.org | 20:50 |
| jeblair | info: Retrieving plugin | 20:50 |
| jeblair | err: Could not retrieve plugin: execution expired | 20:50 |
| jeblair | fungi: i'm afraid we may have a problem ^ | 20:51 |
| jeblair | i don't know what that means, but it's happened twice | 20:51 |
| jeblair | 3 times | 20:51 |
| clarkb | I think puppet is downlosding plugin from master and that times out? | 20:51 |
| clarkb | relocating to hack space now | 20:51 |
| jeblair | which plugins and why, i wonder | 20:51 |
| *** kaitlin-farr has quit IRC | 20:52 | |
| *** amcrn has joined #openstack-infra | 20:53 | |
| *** Ajaeger1 has quit IRC | 20:53 | |
| fungi | yeah, that's a strange one... it's not every time then? | 20:53 |
| fungi | seeing if i can glean any details from the master end now | 20:53 |
| jeblair | that's been followed up with: | 20:53 |
| jeblair | err: Could not retrieve catalog from remote server: execution expired | 20:53 |
| jeblair | notice: Using cached catalog | 20:53 |
| jeblair | err: Could not retrieve catalog; skipping run | 20:53 |
| TheJulia | sounds like the puppetmaster is timing out | 20:54 |
| fungi | perhaps this is the "puppetmaster is broken" symptom we've seen a few times, where we end up restarting apache | 20:54 |
| jeblair | fungi: oh yeah | 20:54 |
| jeblair | fungi: i have 1 more run in progress, when it fails out, let's restart apache | 20:55 |
| fungi | sounds like a good next step | 20:55 |
| jeblair | fungi: can you help me make sure all the jenkins masters have 0 slaves and restart them? | 20:56 |
| fungi | jeblair: will do. logging into them now | 20:57 |
| jeblair | fungi: i'll take 1-4, you take 5-7 and jenkins.o.o? | 20:57 |
| fungi | works for me | 20:57 |
| clarkb | oh wait | 20:57 |
| *** amcrn has quit IRC | 20:57 | |
| clarkb | we could upgrade the timeout plugin now if it isn't too much trouble | 20:58 |
| *** amotoki has joined #openstack-infra | 20:58 | |
| jeblair | clarkb: ok | 20:58 |
| jeblair | clarkb: where is it? | 20:58 |
| clarkb | that will allow us to do the timeout variable export | 20:58 |
| clarkb | jeblair: from upstream | 20:58 |
| *** sweston has joined #openstack-infra | 20:58 | |
| fungi | jeblair: also, keep in mind that some of the long-running slaves are in openstackci rather than openstack-jenkins, so no need to rebuild any which survived due to being there | 20:58 |
| fungi | er, rather than in openstackjenkins | 20:58 |
| sdague | jeblair / clarkb: preferences on this being openstack-infra or stackforge? - https://github.com/sdague/gerrit-dash-creator | 20:58 |
| jeblair | sdague: later? | 20:59 |
| sdague | also fungi in that opinion question | 20:59 |
| fungi | also later ;) | 20:59 |
| sdague | jeblair: works for me | 20:59 |
| jeblair | clarkb: mechanically, how do we make that happen? | 20:59 |
| clarkb | jeblair: I would just upgrade via the jenkins gui | 20:59 |
| clarkb | I find that easiest for upstream plugins | 20:59 |
| jeblair | clarkb: puppet doesn't do upgrades, right? | 20:59 |
| clarkb | correct | 21:00 |
| clarkb | because versions aren't a thing on disk just plugin presence | 21:00 |
| jeblair | ok | 21:00 |
| *** jamespage_ has joined #openstack-infra | 21:00 | |
| *** ArxCruz has quit IRC | 21:00 | |
| jeblair | restarting apache on puppetmaster | 21:00 |
| fungi | jeblair: on second look, no we have none of our long-running slaves in the openstackci account. no idea why i thought we still did | 21:01 |
| *** yfried has quit IRC | 21:01 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:01 | |
| *** mburned is now known as mburned_out | 21:01 | |
| *** sarob has joined #openstack-infra | 21:01 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/hacking: Update examples for i18n https://review.openstack.org/97344 | 21:02 |
| clarkb | jeblair: fungi: have you done jenkins masters yet? | 21:02 |
| clarkb | I can help out now if not | 21:02 |
| *** andreykurilin has quit IRC | 21:03 | |
| *** andreykurilin_ has joined #openstack-infra | 21:03 | |
| jeblair | clarkb: yeah, i'm almost done with upgrading the plugin. says avail version is 1.13 | 21:03 |
| clarkb | that sounds right | 21:03 |
| clarkb | zaro: ^ | 21:03 |
| clarkb | I think 1.12 added the feature we want | 21:03 |
| zaro | yep, that's right | 21:03 |
| jeblair | puppet is running on mirror27, will launch others now | 21:03 |
| zaro | no, it was 1.13 i believe | 21:04 |
| *** rfolco has quit IRC | 21:04 | |
| zaro | actually it's whatever is specified in puppet | 21:04 |
| *** dims has joined #openstack-infra | 21:05 | |
| zaro | jenkins.pp says 1.13 | 21:05 |
| clarkb | perfect | 21:05 |
| jeblair | fungi: keep in mind jenkins doesn't always really stop with the init script | 21:05 |
| *** lttrl has quit IRC | 21:05 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/devstack-gate: Allow to configure git base URL https://review.openstack.org/95901 | 21:05 |
| clarkb | ya I usually do stop, ps, then start | 21:06 |
| fungi | jeblair: yeah, i'm almost ready to start stopping them from the cli | 21:06 |
| jeblair | stop; ps; kill -9; ps; edit config.xml to remove slaves; start | 21:06 |
| fungi | they keep logging me out of the webui at odd moments | 21:06 |
| *** e0ne has quit IRC | 21:07 | |
| fungi | were there any plugins i needed to upgrade on these before stopping? | 21:07 |
| andreaf | fungi: so I updated the comment... but I was wondering now what the function of OVERRIDE_ZUUL_BRANCH is... it sets the branch for all projects in non-grenade case, but not for devstack-gate itself | 21:07 |
| jeblair | fungi: just the build-timeout plugin | 21:07 |
| jeblair | fungi: (also, you can leave the slave definitions on jenkins.o.o) | 21:08 |
| *** dkliban is now known as dkliban_afk | 21:08 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add gerrit-dash-creator to stackforge https://review.openstack.org/97346 | 21:08 |
| *** Longgeek has joined #openstack-infra | 21:08 | |
| andreaf | fungi: so perhaps I could just use OVERRIDE_ZUUL_BRANCH and drop the GIT_BASE | 21:08 |
| fungi | jeblair: i had already deleted them, but i guess they won't be too hard to re-enroll | 21:09 |
| *** pdmars has quit IRC | 21:09 | |
| *** dims has quit IRC | 21:09 | |
| dhellmann | clarkb: I've run out of day. What needs to be done to retest those taskflow release jobs tomorrow? | 21:09 |
| clarkb | dhellmann: I (or another infra rooter need to add the jobs back to the gearman queue | 21:10 |
| jeblair | okay, jenkins01-04 done | 21:10 |
| clarkb | dhellmann: which should happen once jenkins + nodepool are happy again | 21:10 |
| clarkb | dhellmann: I don't think you ened to do anything and I will try to do it today if things settle down | 21:10 |
| fungi | jeblair: clarkb jenkins.o.o saw 1.14 of the build timeout plugin available, but jenkins 05-07 only seemed to see 1.13. should i do anything about that? refresh them? | 21:10 |
| dhellmann | clarkb: oh, cool, I really appreciate that | 21:10 |
| jeblair | fungi: erm, maybe best to manually install 1.13 on jenkins master | 21:11 |
| clarkb | jeblair: I agree | 21:11 |
| clarkb | lets do 1.13 as the puppet change sets that value and we have installed it on a bunch of nodes already | 21:11 |
| fungi | i take it 1.14 is newer than we want? | 21:11 |
| fungi | okay, got it | 21:11 |
| fungi | i'll swap it out while the jenkins service isn't running | 21:11 |
| *** atiwari has quit IRC | 21:11 | |
| *** Longgeek has quit IRC | 21:12 | |
| *** arborism has joined #openstack-infra | 21:13 | |
| *** ominakov has joined #openstack-infra | 21:13 | |
| *** pafuent has quit IRC | 21:14 | |
| clarkb | oh gah | 21:14 |
| clarkb | reading the release notes for build timeout plugin we may need 1.14 | 21:14 |
| clarkb | they put a global lock in there | 21:14 |
| clarkb | why do jenkins plugins always do this? | 21:14 |
| fungi | is it safe to just overwrite the build-timeout.jpi with the desired one? | 21:14 |
| jeblair | fungi: when it's stopped, yes | 21:14 |
| jeblair | fungi: it will unpack on next start | 21:14 |
| jeblair | clarkb: did 1.14 remove the global lock? | 21:14 |
| clarkb | https://issues.jenkins-ci.org/browse/JENKINS-23012 | 21:15 |
| clarkb | that claims to have fixed the sync point | 21:15 |
| *** arnaud has joined #openstack-infra | 21:15 | |
| *** mfer has quit IRC | 21:15 | |
| clarkb | https://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin for release notes | 21:15 |
| fungi | okay, well jenkins.o.o:/var/lib/jenkins/plugins/build-timeout.jpi should be 1.14 then since i haven't overwritten it yet, if we want to copy that one to the others | 21:15 |
| clarkb | hrm actually it may be in the log watching which we don't use so possibly we are ok | 21:16 |
| jeblair | clarkb: what version do you recommend we use? | 21:16 |
| clarkb | I think we are ok with 1.13 if I udnerstand the bug properly | 21:17 |
| fungi | okay, then i'll go ahead and put 1.13 on jenkins.o.o | 21:17 |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs https://review.openstack.org/97348 | 21:17 |
| clarkb | but I don't know how you enable captured log outputs | 21:18 |
| clarkb | zaro: ^ do you udnerstand that? | 21:18 |
| clarkb | we don't seem to explicitly enable that feature but I can't figure out where you might do that | 21:18 |
| jeblair | clarkb: tbh, if you and zaro are not fairly certain that this will not cause problems for us, i'd much rather downgrade back to 1.10 and wait for some testing. | 21:20 |
| jeblair | clarkb: i don't think we need the system to blow up again. | 21:20 |
| clarkb | jeblair: that wfm | 21:20 |
| jeblair | fungi: ^? | 21:20 |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs https://review.openstack.org/97348 | 21:21 |
| zaro | sorry, i don't know about that watch log feature. | 21:21 |
| *** esker has quit IRC | 21:21 | |
| fungi | sure. what's the easiest way for me to get the 1.10 jpi file? | 21:21 |
| jeblair | fungi: 1.10 is still installed on jenkins-dev. i think we can get it from there. | 21:21 |
| fungi | got a url? | 21:21 |
| fungi | okay, getting | 21:21 |
| *** esker has joined #openstack-infra | 21:21 | |
| jeblair | /var/lib/jenkins/plugins/build-timeout.jpi | 21:22 |
| clarkb | zaro: after all of this settles down we should put 1.14 on jenkins-dev and test it | 21:23 |
| clarkb | zaro: iirc you tested the plugin but not on -dev | 21:23 |
| jeblair | or, like a private jenkins | 21:23 |
| jeblair | i think we need to test for this behavior | 21:24 |
| *** johnsom has joined #openstack-infra | 21:24 | |
| clarkb | jeblair: yeah zaro's previous testing was on a private jenkins. could do that again | 21:24 |
| *** Guest70585 has joined #openstack-infra | 21:25 | |
| zaro | no, i did test on dev. | 21:25 |
| *** jcoufal has quit IRC | 21:25 | |
| zaro | i mean jenkins-dev | 21:25 |
| *** AJaeger has quit IRC | 21:25 | |
| clarkb | zaro: did you downgrade after testing? the currently instaleld version is 1.10 | 21:26 |
| zaro | but i agree, we should test again with 1.14 | 21:26 |
| zaro | yes, i did | 21:26 |
| fungi | okay, jenkins.o.o and 05-07 are all stopped, configs contain an empty slaves list, and they are all re-downgraded to build-timeout 1.10. anything else before i start them again? | 21:26 |
| jeblair | fungi: i think that's it | 21:26 |
| *** esker has quit IRC | 21:26 | |
| clarkb | fungi: sounds good to me | 21:26 |
| *** doude has quit IRC | 21:26 | |
| johnsom | I am getting "Server Error" and "Cannot store contact information" when I try to set my contact details in gerrit. Known issue? | 21:26 |
| fungi | starting them | 21:26 |
| anteaya | johnsom: have you joined the foundation yet? | 21:27 |
| jeblair | 01-04 are running 1.10 | 21:27 |
| zaro | johnsom: i don't think so. might be a trove disconnect? | 21:27 |
| fungi | johnsom: try to make sure you've followed the recommendation at https://wiki.openstack.org/wiki/CLA-FAQ#When_trying_to_sign_the_new_ICLA_and_include_contact_information.2C_why_am_I.27m_getting_an_error_message_saying_that_my_E-mail_address_doesn.27t_correspond_to_a_Foundation_membership.3F | 21:28 |
| anteaya | johnsom: https://wiki.openstack.org/wiki/Gerrit_Workflow#Account_Setup | 21:28 |
| johnsom | Yes, I have joined. | 21:28 |
| fungi | johnsom: and your primary e-mail address in your foundation profile matches your preferred e-mail address in your gerrit account settings/contact information? | 21:29 |
| clarkb | looks like the build log capturing was something added in 1.13 too which is why I don't see it today. I will keep digging at this so that we can figure out how t otest it | 21:29 |
| *** yfried has joined #openstack-infra | 21:29 | |
| zaro | clarkb, fungi, jeblair : i'll test the ver 1.14 on jenkins-dev now. | 21:30 |
| jeblair | zaro: it has no slaves at the moment | 21:30 |
| clarkb | zaro: ya probably best to let things settle first | 21:30 |
| jeblair | zaro: you can probably test with local executors on the master i guess | 21:30 |
| clarkb | zaro: tere is an example test case in the jenkins bug though. Should be simple to reproduce using that | 21:30 |
| *** lcostantino has quit IRC | 21:30 | |
| jeblair | zaro: you'll just need to make several since we don't normally run in that configuration. | 21:31 |
| jeblair | zaro: though honestly, it might be easier to test locally first | 21:31 |
| fungi | okay, jenkins seems to be up and operable again on jenkins.o.o and 05-07. all report to currently be using build-timeout 1.10 | 21:31 |
| jeblair | fungi, clarkb: i think that means we're read to restart nodepool | 21:31 |
| clarkb | ++ | 21:31 |
| fungi | perfect | 21:31 |
| jeblair | i have dropped the node and subnode tables, installed the new version and its config file | 21:32 |
| fungi | and then i have an appointment to which i need to skedaddle | 21:32 |
| *** arborism is now known as amcrn | 21:32 | |
| jeblair | 2014-06-02 21:32:25,313 DEBUG nodepool.NodePool: <AllocationRequest for 1281.0 of bare-precise> | 21:33 |
| jeblair | 2014-06-02 21:32:25,314 DEBUG nodepool.NodePool: <AllocationRequest for 1084.0 of devstack-precise> | 21:33 |
| fungi | that is plenty lots | 21:33 |
| clarkb | wow | 21:33 |
| fungi | nodepool list reports 936 nodes in a "building" state now | 21:33 |
| jeblair | 2014-06-02 21:32:25,317 DEBUG nodepool.NodePool: <AllocationGrant of 45 of bare-precise from hpcloud-b1> | 21:33 |
| zaro | clarkb, fungi, jeblair : actually looks like i cannot test. i don't have ssh access to jenkins-dev.o.o only the jenkins management ui. the management ui says latest version of build-timeout is 1.13 | 21:33 |
| jeblair | 2014-06-02 21:32:25,318 DEBUG nodepool.NodePool: <AllocationGrant of 38 of devstack-precise from hpcloud-b1> | 21:33 |
| clarkb | zaro: you need to refreash the management ui's plugin list | 21:34 |
| zaro | ohh yeah, that's right. | 21:34 |
| jeblair | jobs are running | 21:34 |
| clarkb | oh nodepool list doesn't show az. is that something I should patch? or meh? | 21:35 |
| jeblair | clarkb: i think so | 21:35 |
| clarkb | I will work on that now as it may help with future debugging | 21:35 |
| jeblair | might also want to log and statsd it | 21:35 |
| fungi | agreed | 21:35 |
| *** smarcet has quit IRC | 21:35 | |
| jeblair | (those could probably be separate changes) | 21:36 |
| clarkb | jeblair: ok | 21:36 |
| clarkb | does nodepool list show subnodes? | 21:36 |
| jeblair | clarkb: i think so | 21:36 |
| jeblair | i have nothing to back that up with atm though :) | 21:37 |
| clarkb | looking at the list() function I am not so sure (mostly curious because subnode azs are set by the parent node | 21:37 |
| clarkb | jeblair: it calls session.getNodes() | 21:37 |
| jeblair | clarkb: if it doesn't handle subnodes, don't worry about that for now | 21:38 |
| clarkb | ok | 21:38 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Add an option to mark uploaded patchsets as WIP https://review.openstack.org/96528 | 21:40 |
| *** e0ne has joined #openstack-infra | 21:40 | |
| jeblair | oh hehe, we got a node counter reset because i dropped the table | 21:40 |
| fungi | okay, i'm heading out for a bit, but back relatively soon (probably not much more than an hour) | 21:40 |
| johnsom | Thanks guys, got it to work. I thought I had setup my foundation account, but I got it confused with another account setup. | 21:40 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Display node AZ in `nodepool list` output https://review.openstack.org/97351 | 21:40 |
| clarkb | someting like that | 21:41 |
| jeblair | "4 million nodes since a lost-time schema change" | 21:41 |
| jeblair | fungi: have fun! | 21:41 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Display node AZ in `nodepool list` output https://review.openstack.org/97351 | 21:42 |
| *** aysyd has quit IRC | 21:42 | |
| jeblair | clarkb: output looks good! | 21:43 |
| *** ominakov has quit IRC | 21:43 | |
| *** johnsom has left #openstack-infra | 21:43 | |
| zaro | clarkb: i tried upgrading build-timeout plugin on jenkins-dev, now it won't start. can you help there? | 21:45 |
| *** signed8bit has quit IRC | 21:45 | |
| Alex_Gaynor | Did something just make all the CI jobs start? | 21:45 |
| jeblair | Alex_Gaynor: me? | 21:46 |
| Alex_Gaynor | jeblair: Thanks :-) | 21:46 |
| jeblair | Alex_Gaynor: clarkb and fungi helped too, but I hit "Enter". | 21:46 |
| Alex_Gaynor | hitting enter is important | 21:46 |
| jeblair | Alex_Gaynor: we made a change to nodepool to accomodate hpcloud1.1 which required basically a full restart of everything (except zuul, which kept its queue) | 21:47 |
| *** xyang1 has quit IRC | 21:47 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add suppressed fingerprint for pep8 failures in gate https://review.openstack.org/95350 | 21:47 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Support suppressing bugs in ES.classify https://review.openstack.org/97354 | 21:47 |
| jeblair | hopefully that change means we'll actually end up with enough nodes to keep up with tests | 21:47 |
| *** miqui has joined #openstack-infra | 21:48 | |
| *** e0ne has quit IRC | 21:48 | |
| jeblair | we now have as many nodes in use as before the restart | 21:49 |
| jeblair | with 600 still building | 21:49 |
| *** gondoi is now known as zz_gondoi | 21:49 | |
| *** UtahDave has quit IRC | 21:49 | |
| Alex_Gaynor | oh god we are using so much cloud | 21:49 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Record provider AZ info in graphite. https://review.openstack.org/97356 | 21:50 |
| *** _nadya_ has quit IRC | 21:50 | |
| JayF | you can't build a cloud without breaking a few clouds | 21:50 |
| clarkb | jeblair: ^ note that doesn't do the aggregate anymore. I figure we can sum on the graphite side btu let me know if you want me to keep the old keys too | 21:50 |
| jeblair | JayF: haha | 21:50 |
| clarkb | I can update the chnage | 21:51 |
| *** mriedem has quit IRC | 21:51 | |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstacklib https://review.openstack.org/97357 | 21:51 |
| *** sileht has quit IRC | 21:52 | |
| zaro | i believe the bug in build-timeout about capturing log is actually "Writing the build description when timeout occurred." none of our jobs use this setting. | 21:52 |
| *** adalbas has quit IRC | 21:53 | |
| *** pcrews has quit IRC | 21:53 | |
| clarkb | zaro: right because the setting doesn't exist until 1.13 | 21:54 |
| zaro | can someone please kick start jenkins-dev? | 21:54 |
| *** anteaya has quit IRC | 21:54 | |
| *** Guest70585 is now known as anteaya | 21:54 | |
| *** david-lyle has quit IRC | 21:55 | |
| *** sileht has joined #openstack-infra | 21:59 | |
| *** bauzas has quit IRC | 21:59 | |
| mordred | jeblair: re: testmt - I'm sure you already did, but deleting is fine | 22:00 |
| *** rlandy__ has quit IRC | 22:01 | |
| mordred | clarkb: the current graphs I've got should still work until your nodepool change above lands, yeah? | 22:01 |
| jeblair | mordred: heh, i deleted that and more. :) | 22:01 |
| clarkb | mordred: I think they will need a new glob | 22:02 |
| mordred | jeblair: awesome | 22:02 |
| clarkb | which si why I ask if people are ok withthat | 22:02 |
| *** andreaf has quit IRC | 22:02 | |
| mordred | clarkb: a new glob once your thing lands? | 22:02 |
| clarkb | mordred: yeah to glob azs | 22:02 |
| *** amotoki has quit IRC | 22:03 | |
| mordred | clarkb: I don't personally need it - but also I'm fine with updating graphs | 22:03 |
| *** dims has joined #openstack-infra | 22:03 | |
| jeblair | okay, mirror26,27,33 and proposal exist again and are attached to jenkins.o.o | 22:03 |
| jeblair | clarkb, fungi: ^ | 22:03 |
| jeblair | i did not recreate the dev slaves or salt-trigger | 22:04 |
| clarkb | ok | 22:04 |
| mordred | ok | 22:04 |
| *** sileht has quit IRC | 22:04 | |
| anteaya | pleia2: changed my vm just in time, I can't log into the old system anymore | 22:04 |
| anteaya | was about to delete my vms, but *shrug* they can do it | 22:04 |
| pleia2 | anteaya: phew :) | 22:05 |
| anteaya | like 40 minutes difference | 22:05 |
| pleia2 | anteaya: so the networking is all set up for good now, all you ever need to do to start new vms is just go to the compute instances page (or use nova commands if you feel like learning those :)) | 22:05 |
| anteaya | I'll learn horizon | 22:05 |
| *** ihrachyshka has quit IRC | 22:05 | |
| pleia2 | I think the hpcloud cli still exists now, but it was always kind of meh | 22:06 |
| pleia2 | s/now/too | 22:06 |
| *** sileht has joined #openstack-infra | 22:06 | |
| anteaya | then I can have stuff to talk to david-lyle about | 22:06 |
| anteaya | kk | 22:06 |
| * pleia2 nods | 22:06 | |
| *** markmcclain has quit IRC | 22:06 | |
| jeblair | clarkb: if i'm reading correctly, everything in hpcloud has been building for 30 minutes | 22:07 |
| clarkb | jeblair: yeah I think its bonkers | 22:07 |
| clarkb | mordred: ^ | 22:07 |
| *** andreaf has joined #openstack-infra | 22:07 | |
| mordred | jeblair: that seems like an exra long time | 22:07 |
| *** chuckC has quit IRC | 22:08 | |
| clarkb | nova availability-zone-list says i have the correct names | 22:08 |
| *** dims has quit IRC | 22:08 | |
| jeblair | tteggel: ^ so we just started spreading our load across az1-az3. we asked for 500 servers and they have all been in the building state for 35 minutes | 22:08 |
| *** Longgeek has joined #openstack-infra | 22:09 | |
| *** masayukig has quit IRC | 22:09 | |
| *** fbo is now known as fbo_away | 22:10 | |
| jeblair | oh, good news, 179 are now in delete! | 22:10 |
| anteaya | well that's a different state | 22:10 |
| clarkb | so hypothesis time. maybe images and networks are not actually cross AZ? | 22:11 |
| jeblair | ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-bafd0526-1bce-411a-b23b-d41ffccd1bc4) | 22:11 |
| clarkb | I am going to do testing with my personal account | 22:11 |
| *** sanjiva has quit IRC | 22:12 | |
| *** andreaf has quit IRC | 22:12 | |
| Kiall | clarkb: images and networks are cross AZ | 22:12 |
| *** jamielennox is now known as jamielennox|away | 22:12 | |
| clarkb | Kiall: that is what we are told :) | 22:12 |
| Kiall | (I assume you're talking about HP Cloud 1.1 / 13.5...) | 22:12 |
| *** dims has joined #openstack-infra | 22:12 | |
| clarkb | Kiall: yes | 22:12 |
| jeblair | so we have 100 that built, 200 that failed, and 200 that are currently building. | 22:13 |
| clarkb | jeblair: and it took 30 minuts for the 100 that built? | 22:13 |
| * clarkb looks at nodepool | 22:13 | |
| *** Longgeek has quit IRC | 22:13 | |
| jeblair | yeah | 22:13 |
| Kiall | jeblair: have an instance ID of one that's building handy? | 22:13 |
| jeblair | | 937 | hpcloud-b5 | az3 | py3k-precise | jenkins02 | py3k-precise-hpcloud-b5-937.slave.openstack.org | py3k-precise-hpcloud-b5-937 | fdaa2e3e-d66e-4cbb-b826-644970d93b7b | None | building | 0.69 | | 22:14 |
| jeblair | Kiall: fdaa2e3e-d66e-4cbb-b826-644970d93b7b | 22:14 |
| *** bauzas has joined #openstack-infra | 22:14 | |
| clarkb | jeblair: looks like they built in all AZs so not a cross AZ issue | 22:14 |
| *** nati_ueno has quit IRC | 22:14 | |
| *** masayukig has joined #openstack-infra | 22:14 | |
| *** nati_ueno has joined #openstack-infra | 22:14 | |
| clarkb | maybe the next thing to check is that they actually built in different AZs | 22:14 |
| Kiall | jeblair: bad choice I think, that one is ACTIVE | 22:15 |
| jeblair | clarkb: oh! | 22:15 |
| Kiall | and it's in az3 | 22:15 |
| mordred | yay! | 22:15 |
| *** e0ne has joined #openstack-infra | 22:15 | |
| *** hashar has quit IRC | 22:16 | |
| clarkb | ya Isee them in different AZs | 22:16 |
| clarkb | so nodepool change worked | 22:16 |
| clarkb | I am using `nova show uuid` to see that | 22:16 |
| *** praneshp has quit IRC | 22:17 | |
| jeblair | Kiall: if that's the case, probably nodepool is just catching up to the status changes on the ones that are building | 22:17 |
| *** _nadya_ has joined #openstack-infra | 22:17 | |
| *** otherwiseguy has quit IRC | 22:17 | |
| jeblair | Kiall: is a failed instance of interest? | 22:17 |
| Kiall | jeblair: Possibly - Depends on the failure, I might be able see more detail than `nova show` | 22:18 |
| *** e0ne has quit IRC | 22:18 | |
| Kiall | (But I don't have root access on the compute nodes either, so, somewhat limited ;)) | 22:18 |
| jeblair | 2014-06-02 22:09:38,770 ERROR nodepool.NodeLauncher: ClientException launching node id: 935 in provider: hpcloud-b5 error: | 22:19 |
| jeblair | return self._create("/os-floating-ips", {'pool': pool}, "floating_ip") | 22:19 |
| jeblair | ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-2eceb7cc-96dc-467f-98ee-32208252872d) | 22:19 |
| jeblair | instance id: 41963302-7eda-4519-84b1-094b1429b933 | 22:19 |
| jeblair | so node #935 got a 500 when creating a floating ip | 22:20 |
| Kiall | Also ACTIVE, az2, but the log looks like a floating IP issue | 22:20 |
| mordred | jeblair: when we restarted things, did we maybe orphan any flaoting ips? | 22:20 |
| *** bknudson has quit IRC | 22:20 | |
| * Alex_Gaynor idly wonders which does more to ensure OpenStack quality, the tests the CI system runs, or the CI system itself | 22:20 | |
| jeblair | mordred: tbh, i don't know how to tell. | 22:21 |
| Kiall | jeblair: could nodepool be masking FIP create/attach failures as instance build failures? | 22:21 |
| mordred | jeblair: I show a bunch of floating ips not attached to servers | 22:21 |
| jeblair | Kiall: it's all a failure to us | 22:21 |
| *** esker has joined #openstack-infra | 22:22 | |
| jeblair | mordred: is that because they just haven't been attached yet, or are they really orphaned? | 22:22 |
| clarkb | jeblair: how did you delete the nodes? | 22:22 |
| clarkb | jeblair: if you just did nova delete that would orphan the IPs iirc | 22:22 |
| jeblair | clarkb: i did that for some of them | 22:22 |
| jeblair | <sigh> | 22:22 |
| clarkb | nodepools has to special delete floating IPs | 22:23 |
| mordred | jeblair: there are a ton missing a fixed ip, and a ton with one - I'd say half and half | 22:23 |
| mordred | I think I should delete the floating ips that don't have fixed ips | 22:23 |
| jeblair | mordred: okay, thanks | 22:23 |
| mordred | and if that race-conditions a node creation, oops | 22:23 |
| clarkb | mordred: are we anywhere near quota though? | 22:23 |
| clarkb | we should have ~600 IPs | 22:23 |
| clarkb | if quota is any good | 22:23 |
| mordred | yes. we are at quota | 22:23 |
| *** andreykurilin_ has quit IRC | 22:24 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/gitdm: Update company affiliation for morganfainberg https://review.openstack.org/95980 | 22:24 |
| mordred | 321 have no server | 22:24 |
| Kiall | I wonder if being at quota is whats causing the 500? OR, if the 500 returned by the FIP create is leaving an orphan behind... | 22:24 |
| mordred | for ip in $(nova floating-ip-list | grep '| -' | awk '{print $2}') ; do echo "Deleting $ip" ; nova floating-ip-delete $ip ; done | 22:25 |
| mordred | ftr | 22:25 |
| *** BadCub_ has quit IRC | 22:26 | |
| *** adalbas has joined #openstack-infra | 22:26 | |
| *** NobodyCam has quit IRC | 22:26 | |
| *** zaro has quit IRC | 22:28 | |
| clarkb | mordred: is that still running? | 22:28 |
| mordred | clarkb: oh yeah. it's going to be running for a while | 22:28 |
| *** jgrimm has quit IRC | 22:28 | |
| *** mwagner_lap has joined #openstack-infra | 22:28 | |
| mordred | I think it's done roughly 30 | 22:28 |
| mordred | so estimate 30 minutes to complete | 22:29 |
| Kiall | mordred: add --os-cache | 22:30 |
| Kiall | It's keep and reuse the same Keystone token, rather than doing an API call to keystone for each | 22:30 |
| Kiall | It'll keep and reuse* | 22:30 |
| clarkb | wow you would think that would be the default | 22:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't gate pecan on barbican for now. https://review.openstack.org/97285 | 22:30 |
| Kiall | clarkb: I know... | 22:31 |
| Kiall | Oh, and install python-keyring, it needs it | 22:32 |
| mordred | oh. right. that's why it's not default | 22:33 |
| clarkb | oh that thing | 22:33 |
| Kiall | It silently continues without caching if that's not installed | 22:33 |
| clarkb | ya ok we can leave it off :) | 22:33 |
| Kiall | So.. I don't count that as a reason to be off | 22:33 |
| Kiall | anyway.. Install, add flag, half the time required to do the deletes ;) | 22:33 |
| Kiall | halve* | 22:33 |
| mordred | mordred@camelot:~$ sudo pip install python-keyring | 22:33 |
| mordred | Downloading/unpacking python-keyring | 22:33 |
| mordred | Could not find any downloads that satisfy the requirement python-keyring | 22:33 |
| clarkb | Kiall: you might have missed it but it was on by default at one point (the keyring) and it brkoe everything | 22:33 |
| clarkb | keyring is terrible | 22:34 |
| Kiall | mordred: ehh.. that's the .deb name I was thinking of | 22:34 |
| clarkb | it makes bad env assumptions iirc | 22:34 |
| Kiall | might just be keyring on pypi | 22:34 |
| clarkb | like no controlling tty? better prompt for a passphrase anyways | 22:34 |
| Kiall | clarkb: oh.. Nice | 22:34 |
| russellb | yeah not so great for automation ... would be nice to have a better solution for that | 22:35 |
| russellb | writing the scripts to the Python API directly is one solution | 22:35 |
| russellb | that's certainly not the only reason the CLI commands aren't well suited to automation | 22:35 |
| Kiall | russellb: yea, the python APIs for OpenStack (bar neutron, sorry neutron folks) are pretty good+intuitive. | 22:36 |
| Kiall | Speaking of which - mordred - You'd be better off going straight to neutron with those FIP deletes ;) | 22:36 |
| clarkb | Kiall: you haven't used the keystone api :) | 22:36 |
| mordred | Kiall: oh. you're right | 22:36 |
| asselin_ | Hi nodepool experts. I'm trying to make sense of the various credentials since I'm getting this error on image update: INFO:paramiko.transport:Authentication (publickey) failed. | 22:36 |
| clarkb | asselin_: that is an ssh publickey auth error | 22:37 |
| asselin_ | my nodepool.yaml proviers has keypair: 'jenkins' | 22:37 |
| asselin_ | which is being used. and images: private-key: '/home/stack/.ssh/jenkins_key' | 22:37 |
| clarkb | asselin_: which may happen when nodepool ssh's into a node to make sure it is ready to go | 22:37 |
| clarkb | asselin_: your image creation scripts need to install that public key | 22:37 |
| clarkb | or you need to configure cloud init or something to put it in place foryou | 22:38 |
| asselin_ | clarkb, I can manually ssh into it...just confused on how to configure nodepool to do so. | 22:38 |
| clarkb | asselin_: using that key? | 22:38 |
| clarkb | is there a passpharse on the private key? | 22:38 |
| asselin_ | clarkb, I manually updated the public key and set the name to keypair: 'jenkins' | 22:38 |
| asselin_ | 22:38 | |
| asselin_ | no | 22:38 |
| clarkb | mordred: down to ~450ish | 22:39 |
| asselin_ | I'm going through the code and it uses the keypair set to create the instance...but nothing at all to ssh into it...?? | 22:39 |
| *** salv-orlando has quit IRC | 22:39 | |
| asselin_ | clarkb, http://paste.openstack.org/show/82498/ both lines 7 and 10 are 'None'. I tried adding line 8...but wanted to ask if I'm missing something before going down that route | 22:41 |
| *** chuckC has joined #openstack-infra | 22:42 | |
| clarkb | asselin_: it creates a key to ssh in as root iirc | 22:43 |
| *** jamielennox|away is now known as jamielennox | 22:43 | |
| clarkb | but it only does that when creating the image | 22:43 |
| *** sweston has quit IRC | 22:43 | |
| clarkb | then the image creation process is expected to put in place the key needed by nodepool to ssh into after a regular node boots | 22:43 |
| asselin_ | clarkb, ok, yes I'm starting to remember from when I did this before...it creates a throw-away key for image update...and uses the export NODEPOOL_SSH_KEY=`cat ~/.ssh/id_rsa.pub` | 22:45 |
| asselin_ | variable.... | 22:45 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 22:45 |
| asselin_ | clarkb, is that what infra is doing? | 22:45 |
| mordred | 404-{u'QuantumError': u'Floating IP b684688c-8893-42f0-9473-df0b4f24af8e could not be found'} | 22:47 |
| jeblair | what's a quantum? | 22:47 |
| mordred | that's so sexy. I love that the error message is still quantum | 22:47 |
| *** camunoz has quit IRC | 22:47 | |
| *** Sukhdev has quit IRC | 22:47 | |
| *** zaro has joined #openstack-infra | 22:47 | |
| clarkb | asselin_: we have the nodepool user setup with the correct key and paramiko uses that by default | 22:47 |
| clarkb | I believe that is how we do it | 22:47 |
| clarkb | mordred: well you know backward compat | 22:47 |
| clarkb | down to ~250 now | 22:48 |
| asselin_ | clarkb, thanks I'll investigate that a bit | 22:48 |
| mordred | clarkb: we are penny-wise and pound-short with our backward compat attempts | 22:48 |
| *** hemna is now known as hemnafk | 22:49 | |
| anteaya | that is the error when you can't associate a floating ip because teh router doesn't ahve an interface | 22:49 |
| anteaya | saw that several times this afternoon | 22:49 |
| *** esker has quit IRC | 22:49 | |
| mordred | anteaya: yah. I was mostly just amused because it wasn't called NeutronError | 22:50 |
| anteaya | yes | 22:50 |
| anteaya | I too found that odd | 22:50 |
| mordred | ok. the script is done | 22:50 |
| mordred | for the record, I ran: | 22:51 |
| mordred | for ip in $(neutron floatingip-list | grep -v '10.0' | awk '{print $2}') ; do echo "Deleting $ip" ; neutron floatingip-delete $ip ; done | 22:51 |
| *** BadCub has joined #openstack-infra | 22:51 | |
| *** NobodyCam has joined #openstack-infra | 22:52 | |
| clarkb | there are still a ton of floating ips with no fixed ip | 22:52 |
| mordred | clarkb: _those_ may be legit in-progress builds | 22:52 |
| *** denis_makogon has quit IRC | 22:52 | |
| *** eharney has quit IRC | 22:53 | |
| clarkb | wow nova commant is floating-ip-list, neutron is floatingip-list | 22:53 |
| clarkb | we should have an openstack uses openstack session at the next summit | 22:54 |
| Kiall | clarkb: and it's `neutron security-group-list`, for consistency ;) | 22:54 |
| jhesketh | Morning | 22:56 |
| clarkb | o/ | 22:57 |
| anteaya | morning jhesketh | 22:58 |
| anteaya | thanks for finding Longeek's https proxy difficulty | 22:58 |
| anteaya | that marked the end of a day and a half of trying to shoe horn in a min.js patch that I never could understand the motivation for | 22:59 |
| *** nati_uen_ has joined #openstack-infra | 22:59 | |
| *** otherwiseguy has joined #openstack-infra | 22:59 | |
| clarkb | mordred: that seems possible. the number of non associated floating IPs is slowly falling but the total number of floating IPs is rising | 22:59 |
| *** camunoz has joined #openstack-infra | 22:59 | |
| *** nati_uen_ has quit IRC | 22:59 | |
| *** nati_uen_ has joined #openstack-infra | 23:00 | |
| *** e0ne has joined #openstack-infra | 23:00 | |
| mattoliverau | Morning | 23:00 |
| anteaya | morning mattoliverau | 23:00 |
| clarkb | fungi: jeblair https://review.openstack.org/#/c/95576/ may be the next step in nodepool fun | 23:01 |
| clarkb | for completeness | 23:01 |
| jeblair | clarkb: lgtm | 23:02 |
| *** nati_ueno has quit IRC | 23:03 | |
| *** harlowja_ has quit IRC | 23:04 | |
| asselin_ | clarkb, ok I got it working by taking out keypair: 'jenkins' | 23:04 |
| asselin_ | from the nodepool.yaml file. This puts the code in the path to use the throw-away key for image creation. | 23:04 |
| *** sarob has quit IRC | 23:04 | |
| *** harlowja has joined #openstack-infra | 23:04 | |
| *** weshay has quit IRC | 23:04 | |
| stevebaker | dstufft: hey, do you have any idea when pip-1.5.7 will be released? I'm waiting for a fix to allow building custom heat images | 23:04 |
| *** sarob has joined #openstack-infra | 23:04 | |
| clarkb | asselin_: good to know and glad you figured it out | 23:05 |
| *** yamahata has quit IRC | 23:05 | |
| *** e0ne has quit IRC | 23:05 | |
| jhesketh | jeblair: ping? | 23:05 |
| *** adalbas has quit IRC | 23:05 | |
| asselin_ | clarkb, I'm curious to know why use the throwaway key? security related? | 23:05 |
| jeblair | jhesketh: pong | 23:06 |
| jhesketh | jeblair: PM okay? | 23:06 |
| clarkb | asselin_: I think for ease of implementation | 23:06 |
| jeblair | jhesketh: yep | 23:06 |
| clarkb | asselin_: the image build is a one shot thing so persistent keys don't help much | 23:06 |
| *** zzelle_ has quit IRC | 23:06 | |
| asselin_ | clarkb, they help when debugging image creation :) | 23:07 |
| clarkb | asselin_: nodepool deletes the node when it is done | 23:08 |
| clarkb | so you would have to change that behavior too. easier to just run the scripts on a node by hand | 23:08 |
| *** sarob has quit IRC | 23:09 | |
| *** Longgeek has joined #openstack-infra | 23:10 | |
| *** mmaglana has quit IRC | 23:10 | |
| asselin_ | clarkb, ok I see | 23:10 |
| *** mburned_out is now known as mburned | 23:10 | |
| clarkb | ok nodepool looks much happier according to mordred's graphs | 23:11 |
| clarkb | we are still seeing some really long build times though | 23:11 |
| clarkb | I need to do a grocery run back soon | 23:11 |
| asselin_ | anyone else like to review/approve these 2 patches: https://review.openstack.org/#/c/96539/ https://review.openstack.org/#/c/96551/ ? | 23:13 |
| openstackgerrit | Nachi Ueno proposed a change to openstack-infra/reviewstats: Add Review Stat WebUI https://review.openstack.org/94288 | 23:14 |
| *** Longgeek has quit IRC | 23:15 | |
| *** pcrews has joined #openstack-infra | 23:17 | |
| *** wenlock has quit IRC | 23:19 | |
| *** doug-fish has left #openstack-infra | 23:22 | |
| *** nati_uen_ has quit IRC | 23:22 | |
| *** nati_ueno has joined #openstack-infra | 23:23 | |
| *** andreaf has joined #openstack-infra | 23:23 | |
| *** otherwiseguy has quit IRC | 23:24 | |
| *** terryw has joined #openstack-infra | 23:24 | |
| *** nati_uen_ has joined #openstack-infra | 23:24 | |
| *** camunoz has quit IRC | 23:26 | |
| *** nati_ueno has quit IRC | 23:27 | |
| *** terryw has quit IRC | 23:28 | |
| *** otherwiseguy has joined #openstack-infra | 23:28 | |
| *** andreaf has quit IRC | 23:29 | |
| *** terryw has joined #openstack-infra | 23:31 | |
| *** sarob has joined #openstack-infra | 23:31 | |
| *** otherwiseguy has quit IRC | 23:33 | |
| mordred | jeblair: ^^ perhaps we should figure out what ACL we need to set so that the REST call to get the list of cores can be done un-authed? | 23:37 |
| mordred | jeblair: it seems like there are a couple of programs out there keeping lists of things or inferring it, and now taht we've got the REST API they could just, you know, query gerrit | 23:37 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Ignore non-voting jobs in gerrit https://review.openstack.org/97369 | 23:37 |
| jeblair | mordred: group listing requires auth? huh, i haven't tried it. | 23:37 |
| mordred | jeblair: I tried it briefly with curl the other day | 23:38 |
| *** yjiang5 has joined #openstack-infra | 23:40 | |
| *** timrc is now known as timrc-afk | 23:40 | |
| *** camunoz has joined #openstack-infra | 23:43 | |
| *** timrc-afk is now known as timrc | 23:48 | |
| *** nati_uen_ has quit IRC | 23:48 | |
| clarkb | ya it does | 23:49 |
| Alex_Gaynor | Are all the py3k-precise nodes in one DC/AZ? | 23:49 |
| *** unicell has joined #openstack-infra | 23:49 | |
| clarkb | Alex_Gaynor: no they are in rax and hpcloud | 23:49 |
| *** ArxCruz has joined #openstack-infra | 23:49 | |
| *** terryw has quit IRC | 23:50 | |
| Alex_Gaynor | Ok, it must be in my head then (for some reason they seem particularly absent) | 23:50 |
| jeblair | clarkb: 518 nodes in use now; much more than earlier | 23:50 |
| clarkb | asselin_: I thought I -1'd 96539 | 23:50 |
| clarkb | asselin_: because the override shoudl just override the entire url prefix not just the protol | 23:50 |
| clarkb | *protocol | 23:50 |
| clarkb | apparnetly that comment never made it to gerrit | 23:51 |
| *** arnaud has quit IRC | 23:51 | |
| clarkb | jeblair: woot | 23:51 |
| jeblair | clarkb: gertty? | 23:51 |
| clarkb | jeblair: ya it must've been gertty I probably closed it wihtout leaving it long enough to sync | 23:51 |
| jeblair | clarkb: it should sync on the next start | 23:52 |
| *** nati_ueno has joined #openstack-infra | 23:52 | |
| jeblair | clarkb: (the first sync task is to push up review comments) | 23:52 |
| clarkb | hrm not sure where that comment went then | 23:52 |
| clarkb | in any case -1 is there now | 23:52 |
| clarkb | asselin_: ^ | 23:52 |
| *** otherwiseguy has joined #openstack-infra | 23:52 | |
| jeblair | (at least, in my branch tip, it's like #3 in master) | 23:52 |
| asselin_ | clarkb, ok will look at that | 23:53 |
| jeblair | clarkb: if you find that you have a draft comment for that change in your gertty, i'm interested in debugging | 23:53 |
| *** nati_uen_ has joined #openstack-infra | 23:54 | |
| clarkb | jeblair: actually I wonder if it was done in the old gertty whose db I deleted for your new changes | 23:54 |
| asselin_ | clarkb, got it: GIT_BASE=${GIT_BASE:-git://git.openstack.org} | 23:54 |
| clarkb | jeblair: I should've backed that DB up instead of deleting it :( | 23:54 |
| clarkb | asselin_: yeah that way you can override the protocol and location if necessary | 23:54 |
| *** sarob has quit IRC | 23:54 | |
| jeblair | clarkb: doh | 23:54 |
| *** jaypipes has quit IRC | 23:55 | |
| *** sarob has joined #openstack-infra | 23:55 | |
| clarkb | asselin_: its a bit more flexible that way | 23:55 |
| asselin_ | clarkb, agreed | 23:55 |
| clarkb | did mordred's unbound change merge which allows 96551 to work? | 23:55 |
| mordred | unsure | 23:56 |
| *** W00dy has joined #openstack-infra | 23:56 | |
| *** W00dy is now known as Guest54700 | 23:56 | |
| mordred | the one thing that gertty does not help me with is tracking the state of my in flight patches | 23:56 |
| clarkb | mordred: yeah its very reviewing all the things oriented | 23:56 |
| mordred | yah | 23:56 |
| mordred | and it's AMAZING | 23:56 |
| mordred | I've been hovering at inbox zero on many repos all day | 23:56 |
| *** nati_ueno has quit IRC | 23:57 | |
| clarkb | jeblair: I am giving the tip of your proposed changes another shot | 23:58 |
| clarkb | sync is happening now | 23:58 |
| *** matsuhashi has joined #openstack-infra | 23:59 | |
| *** sarob has quit IRC | 23:59 | |
| clarkb | woot hyperlinks open | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!