| clarkb | hurray | 00:04 |
|---|---|---|
| *** zul has joined #openstack-infra | 00:10 | |
| *** zul has quit IRC | 00:11 | |
| lifeless | mordred: are you going to follow up on https://code.launchpad.net/~mordred/python-fixtures/agressive-loggers/+merge/150237 ? | 00:11 |
| *** zul has joined #openstack-infra | 00:13 | |
| *** vipul is now known as vipul-away | 00:13 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use nodepool stats for graph https://review.openstack.org/42245 | 00:14 |
| *** ryanpetrello has joined #openstack-infra | 00:16 | |
| clarkb | woot logrotate seems to work as expected. I am applying the changes to etherpad.o.o now | 00:17 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Require a target name when instantiating a node https://review.openstack.org/42246 | 00:17 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch etherpad_lite backups to mysql_backup. https://review.openstack.org/41791 | 00:19 |
| *** colinmcnamara has quit IRC | 00:23 | |
| jeblair | a job has completed, and the node was deleted and removed from jenkins! | 00:26 |
| clarkb | success. jeblair should light a cigar and pour a glass of whiskey | 00:28 |
| jeblair | ooh, and it just deleted a node that didn't come online: Exception: Timeout waiting for ssh access | 00:29 |
| jeblair | clarkb: maybe i'll light that glass of whiskey now! | 00:30 |
| clarkb | jeblair: I am going to leave comments on the first nodepool puppet change if you want to wait on fixing the lint stuff | 00:30 |
| jeblair | clarkb: ok | 00:30 |
| *** dims has quit IRC | 00:30 | |
| *** ryanpetrello has quit IRC | 00:30 | |
| *** UtahDave has quit IRC | 00:31 | |
| *** vipul-away is now known as vipul | 00:32 | |
| *** Ryan_Lane has joined #openstack-infra | 00:33 | |
| *** ryanpetrello has joined #openstack-infra | 00:33 | |
| *** sarob_ has joined #openstack-infra | 00:35 | |
| clarkb | jeblair: done | 00:38 |
| *** sarob has quit IRC | 00:39 | |
| jeblair | lifeless: just so you know, nodepool is basically two year old code that is already in production, but needs to be made into a daemon. i know it has lots of things that could be improved, but i'm not aiming for perfect now. | 00:39 |
| *** sarob_ has quit IRC | 00:40 | |
| jeblair | lifeless: i'm aiming for 'runs in production and can provide the needed test nodes before the rush of changes that will happen in a couple of days for the H3 feature freezes' | 00:40 |
| clarkb | puppet has been started again on etherpad and etherpad-dev | 00:40 |
| lifeless | jeblair: sure; I'm not core and don't know the code yet. | 00:41 |
| lifeless | jeblair: you and other core folk need to assess whether a suggestion I make is important at this stage or not. | 00:41 |
| *** ryanpetrello has quit IRC | 00:41 | |
| lifeless | jeblair: the use of symbolic constants rather than magic numbers for errno for instance, is trivial but makes a big difference | 00:42 |
| jeblair | lifeless: from what i've seen, your suggestions are good, and almost certainly correct; i'm likely to ignore/defer some of them though at the moment. | 00:44 |
| jeblair | lifeless: and i don't want you to be put off by that | 00:44 |
| lifeless | jeblair: oh I won't be; per clarkb and mordreds request, it's in my daily review scan - anything missing a review will get one from me daily :) | 00:44 |
| lifeless | jeblair: thanks for being clear though; much appreciated | 00:44 |
| clarkb | lifeless: and thank you for taking the time to look | 00:44 |
| jeblair | lifeless: (it's also extremely difficult to test; i think the biggest bug is it needs a test suite, which will make cleanup changes a lot more palatable) | 00:44 |
| jeblair | lifeless: indeed | 00:44 |
| clarkb | ++ | 00:44 |
| *** colinmcnamara has joined #openstack-infra | 00:44 | |
| jeblair | lifeless: fwiw, zuul is much more mature and can stand up to that kind of scrutiny | 00:45 |
| *** dims has joined #openstack-infra | 00:45 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Make the local script directory configurable https://review.openstack.org/42233 | 00:49 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Require a target name when instantiating a node https://review.openstack.org/42246 | 00:49 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Use MySQL https://review.openstack.org/42234 | 00:49 |
| *** gyee has quit IRC | 00:50 | |
| fungi | oof! i missed the fun and whiskey lighting | 00:50 |
| clarkb | jeblair: 42234/2 addresses the comment i was going to leave on 42234/1 (import sys) | 00:51 |
| fungi | taking a look at those at least | 00:51 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-pip https://review.openstack.org/39833 | 00:52 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Make the target name required in the schema https://review.openstack.org/42251 | 00:52 |
| *** ^d is now known as ^demon|away | 00:54 | |
| clarkb | jeblair: what is the target name? is that the name used in jenkins? | 00:55 |
| jeblair | clarkb: no, it's jenkins's name; eg jenkins01 or jenkins02 | 00:55 |
| *** sarob has joined #openstack-infra | 00:56 | |
| jeblair | clarkb: http://paste.openstack.org/show/44284/ | 00:56 |
| clarkb | thanks | 00:57 |
| clarkb | jeblair: what do you think about having a default value for script-dir? | 00:58 |
| clarkb | the old code used 'scripts' | 00:58 |
| jeblair | clarkb: that change was a from-production patch because it's basically contextless when installed in production. /etc/nodepool/scripts is about the only thing that makes sense to me as a default; but i'm not sure it's a big imposition to require it in the config file. | 00:59 |
| lifeless | where is the global requirements repo ? | 01:00 |
| clarkb | jeblair: it isn't, I am fine with it as is. lifeless' comment about making target_name non NULL in the DB schema is good | 01:00 |
| jeblair | lifeless: openstack/requirements | 01:01 |
| clarkb | lifeless: $HOST/openstack/requirements where $HOST is one of https://review.o.o https://github.com git://github.com git://git.o.o or https://git.o.o | 01:01 |
| jeblair | clarkb: so good i followed it up with a change to do that | 01:01 |
| jeblair | clarkb: haha | 01:01 |
| lifeless | jeblair: thanks | 01:01 |
| jeblair | clarkb: also https://review.o.o/p | 01:01 |
| jeblair | clarkb: and git://git.o.o/cgit | 01:02 |
| clarkb | https://review.openstack.org/#/c/42251/1..1/nodepool/nodedb.py does not open for me | 01:02 |
| clarkb | oh wait nevermind | 01:02 |
| fungi | heh | 01:02 |
| clarkb | somehow I convicned it to tell me there were diffs between patchsets 1 and 1 | 01:02 |
| fungi | range of 0 | 01:02 |
| clarkb | +2 -2 | 01:02 |
| lifeless | clarkb: it's A FEATURE | 01:02 |
| *** mjfork has quit IRC | 01:06 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add nodepool host https://review.openstack.org/42232 | 01:07 |
| *** sarob has quit IRC | 01:09 | |
| openstackgerrit | lifeless proposed a change to openstack/requirements: Ignore the ending of commit messages. https://review.openstack.org/42255 | 01:09 |
| openstackgerrit | lifeless proposed a change to openstack/requirements: Uncap testscenarios. https://review.openstack.org/42256 | 01:09 |
| openstackgerrit | lifeless proposed a change to openstack/requirements: New fixtures release. https://review.openstack.org/42257 | 01:09 |
| openstackgerrit | lifeless proposed a change to openstack/requirements: Ignore common files that should never be added. https://review.openstack.org/42258 | 01:09 |
| *** sarob has joined #openstack-infra | 01:09 | |
| jeblair | yay more things for nodepool to do. :) | 01:11 |
| clarkb | the number of tests run today has been insane | 01:11 |
| * fungi concurs | 01:11 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Increase nodepool size to 30 (5/provider) https://review.openstack.org/42259 | 01:11 |
| *** Ryan_Lane has quit IRC | 01:12 | |
| jeblair | that's for later ^ | 01:12 |
| jeblair | i'm about to head to dinner | 01:12 |
| jeblair | but if it holds up, i think tomorrow we should be able to merge that and turn off the devstack-gate node launchers | 01:13 |
| *** sarob has quit IRC | 01:14 | |
| jeblair | http://graphite.openstack.org/render/?from=-24hours&height=170&until=now&width=310&bgcolor=ffffff&fgcolor=000000&target=color%28alias%28sumSeries%28stats.gauges.nodepool.target.*.devstack-precise.*.ready%29,%20%27devstack-precise%27%29,%20%27green%27%29&title=Available%20Test%20Nodes&_t=0.8664466904279092#1376615667918 | 01:14 |
| jeblair | there's the new graph, btw ^ | 01:14 |
| lifeless | is there some way to express 'this change in project A depends on other change in project B' ? | 01:16 |
| lifeless | for CI specifically | 01:17 |
| jeblair | lifeless: no; we just use words | 01:17 |
| lifeless | post-CI release version constraints should do it | 01:17 |
| *** sarob has joined #openstack-infra | 01:17 | |
| lifeless | jeblair: ok. I am thinking about how to safe pointless test runs. | 01:17 |
| jeblair | lifeless: https://bugs.launchpad.net/openstack-ci/+bug/1021879 | 01:18 |
| uvirtbot | Launchpad bug 1021879 in openstack-ci "have zuul handle cross-repo-dependencies" [Medium,Triaged] | 01:18 |
| lifeless | jeblair: e.g. 'change in dib which needs a bumped requirement but global requirements hasn't changed yet - don't even /run/ this branch until that lands' | 01:18 |
| lifeless | jeblair: thanks | 01:18 |
| fungi | gerrit is annoying at 640x480. i need to step up my timetable on upgrading these goggles | 01:20 |
| *** jhesketh has quit IRC | 01:22 | |
| *** jhesketh has joined #openstack-infra | 01:23 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove devstack launch nodes and jobs https://review.openstack.org/42263 | 01:31 |
| *** mriedem has joined #openstack-infra | 01:37 | |
| clarkb | that was fun. kernel update seemed to break my nvidia driver + X11 settings | 01:45 |
| fungi | eek. i've been sticking with intel video chipsets because of years of hating nv and ati on 'nix | 01:47 |
| jhesketh | Any zuul devs in here? | 01:48 |
| clarkb | jhesketh: yes | 01:48 |
| clarkb | fungi: I hit it with a sufficiently big hammer and it seems happy now, but for a while there I was a bit confused | 01:48 |
| clarkb | fungi: I was going to fall back on the intel hd 3000 if I had to | 01:49 |
| jhesketh | clarkb: so I'm wondering about getting zuul to do more but I'm concerned about scaling.. For example, perhaps results should be separated from triggers so that we can have a different result action defined in the layout | 01:49 |
| jhesketh | such as emailing a certain list for a particular gate | 01:49 |
| jhesketh | *emailing results | 01:50 |
| jhesketh | so in the layout.yaml we'd define posting results to gerrit and XYZ (or any other plugin etc) | 01:50 |
| jhesketh | what are your thoughts? | 01:50 |
| clarkb | coming from gerrit land (which is what zuul has been focused on) triggers and results are tightly coupled | 01:51 |
| *** anteaya has quit IRC | 01:51 | |
| clarkb | if gerrit causes some work to be done we let gerrit know when it is complete. | 01:52 |
| clarkb | that said I think this view is changing https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L25-L38 | 01:52 |
| clarkb | notice that the gerrit trigger section there only concerns itself with what should cause jobs to run | 01:53 |
| clarkb | then further along we have the resutl reporting details | 01:53 |
| clarkb | all that to say, yes I can see decoupling triggers from reporting results | 01:53 |
| jhesketh | right, valid points though | 01:53 |
| jhesketh | I guess it'd just need to be structured in a way that the result handlers know what to do if triggered from a different source | 01:54 |
| jhesketh | actually, I guess only the gerrit result handler would be coupled to a gerrit trigger | 01:54 |
| jhesketh | clarkb: so I guess the other discussion is whether or not it's good for zuul to be doing so much.. for example, would it be better for workers to have knowledge on how to process results | 01:55 |
| jhesketh | that way we distribute load | 01:56 |
| jhesketh | I'm also concerned with how zuul can scale with gerrit given that it receives all events | 01:56 |
| clarkb | I am not too worried about load, our zuul is fairly heavily loaded right now but it keeps up. | 01:56 |
| clarkb | But pushing more things into gearman is good in general | 01:56 |
| jhesketh | I wonder if there is scope for a master zuul that receives all gerrit events and sends them off to other zuul servers to distribute to workers | 01:57 |
| jhesketh | so in your opinion, if I wanted email reports from a particular worker that I'm designing would it be better for that worker to be emailing the results out or letting zuul collate and distribute? | 01:58 |
| jhesketh | (I eventually also want to get this working as a gate on infra) | 01:58 |
| clarkb | jhesketh: I am not sure we have a need for email in that way with openstack infra | 02:00 |
| clarkb | jhesketh: gerrit emails people when it gets comments and in other cases we have jenkins email people | 02:00 |
| clarkb | jhesketh: and email is cheap, if zuul were to do it it would go into the local spool and exim/postfix/sendmail/etc would handle it | 02:00 |
| clarkb | so I think it is fine if zuul itself handles email | 02:01 |
| jhesketh | sure | 02:01 |
| clarkb | I would run all of this by jeblair before spending too much time on it though | 02:01 |
| jhesketh | okay | 02:01 |
| jhesketh | clarkb: taking a tangent for a bit, what does zuul do with results after it has published the reports? | 02:02 |
| clarkb | jhesketh: after it has left a comment in Gerrit? | 02:03 |
| jhesketh | yeah | 02:03 |
| *** sarob has quit IRC | 02:03 | |
| clarkb | I believe they get garbage collected | 02:03 |
| jhesketh | one of the reasons I think this may be helpful is because if I put my gate into the testing pipeline with no results being pushed back to gerrit I have to visit the worker to know if it was successful or not (in my understanding) | 02:03 |
| *** sarob has joined #openstack-infra | 02:03 | |
| jhesketh | as won't it disappear from zuul as soon as my worker has finished with it | 02:04 |
| clarkb | correct, if the pipeline the job does not run in is not reporting to gerrit you will have to check the worker to see what happened | 02:04 |
| clarkb | we have had jenkins email people when that is the case and we want to distribute results | 02:05 |
| clarkb | jhesketh: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=23&page=2 has several load related graphs if you are interested. and I think today was one of the busiest days zuul has had | 02:05 |
| jhesketh | okay sure | 02:05 |
| jhesketh | shiny :-) | 02:05 |
| jhesketh | so the scenario where we want gerrit to trigger work but don't want the results in the gerrit comment is a good case for having decoupled report modules | 02:06 |
| jhesketh | but given jenkins currently does that at the worker level (by just emailing) perhaps my worker should too | 02:06 |
| *** yaguang has joined #openstack-infra | 02:06 | |
| clarkb | its definitely one way to approach the problem. The biggset problem with jenkins emailing results is it emails on a per job basis | 02:07 |
| clarkb | it is often nice to see all of the jobs related to an event | 02:07 |
| *** sarob has quit IRC | 02:08 | |
| jhesketh | right, and I'm guessing the html_description can't do that if you are hiding tests from gerrit | 02:08 |
| *** sarob has joined #openstack-infra | 02:16 | |
| *** nati_ueno has quit IRC | 02:18 | |
| fungi | also jenkins e-mail is not extremely flexible/extensible/configurable last i looked | 02:19 |
| *** colinmcnamara has quit IRC | 02:24 | |
| *** colinmcnamara has joined #openstack-infra | 02:25 | |
| *** dims has quit IRC | 02:28 | |
| *** melwitt has quit IRC | 02:39 | |
| *** yaguang has quit IRC | 02:40 | |
| *** dguitarbite has joined #openstack-infra | 02:53 | |
| *** xchu has joined #openstack-infra | 02:54 | |
| *** jfriedly has quit IRC | 02:55 | |
| *** yaguang has joined #openstack-infra | 02:58 | |
| *** mriedem has quit IRC | 02:58 | |
| *** jhesketh has quit IRC | 03:10 | |
| *** jhesketh has joined #openstack-infra | 03:12 | |
| openstackgerrit | lifeless proposed a change to openstack/requirements: New fixtures release. https://review.openstack.org/42257 | 03:19 |
| openstackgerrit | lifeless proposed a change to openstack/requirements: Ignore common files that should never be added. https://review.openstack.org/42258 | 03:19 |
| *** colinmcnamara has quit IRC | 03:31 | |
| *** SergeyLukjanov has joined #openstack-infra | 03:32 | |
| *** colinmcnamara has joined #openstack-infra | 03:32 | |
| *** adalbas has quit IRC | 03:36 | |
| *** adalbas has joined #openstack-infra | 03:37 | |
| *** kspear has quit IRC | 03:38 | |
| *** xchu has quit IRC | 03:44 | |
| *** xchu has joined #openstack-infra | 04:01 | |
| *** ^demon|away has quit IRC | 04:02 | |
| yaguang | hi, I find an error occurs in jenkins , No distributions matching the version for netaddr>=0.7.6 (from -r /home/jenkins/workspace/gate-nova-docs/requirements.txt (line 18)) | 04:02 |
| *** ^demon|away has joined #openstack-infra | 04:03 | |
| yaguang | the url is here https://jenkins01.openstack.org/job/gate-nova-docs/924/console | 04:03 |
| *** vogxn has joined #openstack-infra | 04:03 | |
| *** ^demon|away has quit IRC | 04:04 | |
| fungi | yaguang: there was a network connectivity issue around 1600 utc yesterday where we were seeing that error | 04:07 |
| yaguang | fungi, the issue should still exists. I just submit a patch and find the issue | 04:08 |
| fungi | yes, i see your log is from less than an hour ago | 04:10 |
| fungi | i'll reopen the bug report | 04:10 |
| *** UtahDave has joined #openstack-infra | 04:10 | |
| fungi | yaguang: for reference, bug 1212751 | 04:14 |
| uvirtbot | Launchpad bug 1212751 in openstack-ci "netaddr could not be downloaded / installed" [Medium,Triaged] https://launchpad.net/bugs/1212751 | 04:14 |
| *** afazekas_zz is now known as afazekas | 04:23 | |
| *** jhesketh has quit IRC | 04:23 | |
| *** jhesketh has joined #openstack-infra | 04:24 | |
| *** jhesketh_ has joined #openstack-infra | 04:28 | |
| *** jhesketh has quit IRC | 04:29 | |
| *** dguitarbite has quit IRC | 04:29 | |
| *** colinmcnamara has quit IRC | 04:33 | |
| *** nayward has joined #openstack-infra | 04:41 | |
| *** colinmcnamara has joined #openstack-infra | 04:46 | |
| *** sarob has quit IRC | 04:53 | |
| *** sarob has joined #openstack-infra | 04:54 | |
| *** sdake_ has joined #openstack-infra | 04:57 | |
| *** afazekas_ has joined #openstack-infra | 04:58 | |
| *** sarob has quit IRC | 04:58 | |
| *** colinmcnamara has quit IRC | 05:00 | |
| *** SergeyLukjanov has quit IRC | 05:05 | |
| *** xchu has quit IRC | 05:19 | |
| *** jerryz has quit IRC | 05:26 | |
| *** nicedice has quit IRC | 05:26 | |
| *** xchu has joined #openstack-infra | 05:28 | |
| *** kspear has joined #openstack-infra | 05:35 | |
| *** boris-42 has joined #openstack-infra | 05:40 | |
| *** jhesketh_ has quit IRC | 05:45 | |
| *** jhesketh has joined #openstack-infra | 05:46 | |
| *** fifieldt_ has joined #openstack-infra | 06:06 | |
| *** thomasbiege has joined #openstack-infra | 06:07 | |
| *** thomasbiege has quit IRC | 06:11 | |
| *** dkliban has joined #openstack-infra | 06:12 | |
| *** jhesketh has quit IRC | 06:29 | |
| *** jhesketh has joined #openstack-infra | 06:32 | |
| *** odyssey4me has joined #openstack-infra | 06:42 | |
| *** jerryz has joined #openstack-infra | 06:44 | |
| *** yaguang has quit IRC | 06:48 | |
| *** jerryz has quit IRC | 06:57 | |
| *** jerryz has joined #openstack-infra | 06:58 | |
| *** yaguang has joined #openstack-infra | 07:00 | |
| *** afazekas has quit IRC | 07:06 | |
| *** afazekas_ is now known as afazekas | 07:06 | |
| *** yaguang has quit IRC | 07:07 | |
| *** odyssey4me has quit IRC | 07:07 | |
| *** odyssey4me has joined #openstack-infra | 07:08 | |
| *** jerryz has quit IRC | 07:08 | |
| *** odyssey4me2 has joined #openstack-infra | 07:12 | |
| *** odyssey4me has quit IRC | 07:12 | |
| *** llu has joined #openstack-infra | 07:13 | |
| *** llu has left #openstack-infra | 07:13 | |
| *** odyssey4me has joined #openstack-infra | 07:14 | |
| *** odyssey4me2 has quit IRC | 07:16 | |
| *** llu has joined #openstack-infra | 07:24 | |
| llu | hi guys, but I can't access the link https://jenkins01.openstack.org/job/gate-nova-docs/936/ when I saw the gate-nova-doc failure on zuul.openstack.org for my patch(#35764). Anyone know why? | 07:33 |
| *** vogxn has quit IRC | 07:34 | |
| *** Ryan_Lane has joined #openstack-infra | 07:38 | |
| *** yaguang has joined #openstack-infra | 07:39 | |
| *** dina_belova has joined #openstack-infra | 07:42 | |
| *** psedlak has quit IRC | 07:44 | |
| *** dina_belova has quit IRC | 07:46 | |
| *** dina_belova has joined #openstack-infra | 07:47 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:57 | |
| *** jerryz has joined #openstack-infra | 07:59 | |
| *** vkuz has quit IRC | 07:59 | |
| *** Ng_holiday is now known as Ng | 08:00 | |
| *** jpich has joined #openstack-infra | 08:03 | |
| *** boris-42 has quit IRC | 08:05 | |
| *** psedlak has joined #openstack-infra | 08:07 | |
| *** vogxn has joined #openstack-infra | 08:08 | |
| *** DennyZhang has joined #openstack-infra | 08:10 | |
| *** jerryz has quit IRC | 08:11 | |
| *** vogxn has quit IRC | 08:12 | |
| *** BobBall_Away is now known as BobBall | 08:18 | |
| llu | Is gate-nova-docs check test broken? http://logs.openstack.org/64/35764/17/check/gate-nova-docs/4e87ead/console.html, the pip installation keeps failing to find netaddr>=0.7.6 | 08:20 |
| Ryan_Lane | :D | 08:33 |
| Ryan_Lane | the vote window for rating openstack conference talks jumps after the page loads | 08:34 |
| Ryan_Lane | 3 stars becomes a 0 star click | 08:34 |
| morganfainberg | Ryan_Lane: oh thats fun. | 08:34 |
| openstackgerrit | Serg Melikyan proposed a change to openstack-infra/config: Fix ACL for Murano projects https://review.openstack.org/41650 | 08:34 |
| morganfainberg | Ryan_Lane: the next challenge is when the box runs away from the mouse cursor! | 08:35 |
| *** fbo_away is now known as fbo | 08:37 | |
| *** boris-42 has joined #openstack-infra | 08:37 | |
| *** DennyZhang has quit IRC | 08:37 | |
| Ryan_Lane | heh | 08:39 |
| openstackgerrit | Serg Melikyan proposed a change to openstack-infra/config: Fix ACL for Murano projects https://review.openstack.org/41650 | 08:40 |
| openstackgerrit | Serg Melikyan proposed a change to openstack-infra/config: Added murano-common project https://review.openstack.org/41634 | 08:40 |
| jd__ | woh woh | 08:45 |
| jd__ | I sense unstability in the gates | 08:45 |
| *** lucasagomes has joined #openstack-infra | 08:45 | |
| jd__ | an unknown force is disturbing our tests | 08:45 |
| *** yolanda has joined #openstack-infra | 08:46 | |
| yolanda | hi, good morning | 08:46 |
| jd__ | we should call the infra team | 08:46 |
| * jd__ looks up to the sky | 08:46 | |
| jd__ | hi yolanda | 08:46 |
| yolanda | i've been progressing with my gerrit/zuul integration, but now when i submit a change it's stuck on status Submitted, Merge Pending | 08:46 |
| yolanda | the logs don't show any error, any idea if i'm missing something? | 08:46 |
| yolanda | hi jd | 08:46 |
| *** Ajaeger1 has joined #openstack-infra | 08:52 | |
| Ajaeger1 | hi, is jenkins broken? | 08:52 |
| * Ajaeger1 just got twice a failure and the link by Jenkins does not work: https://review.openstack.org/#/c/42300/ | 08:53 | |
| *** ruhe has joined #openstack-infra | 08:53 | |
| BobBall | it appears so | 08:53 |
| BobBall | no jenkins jobs running at all | 08:53 |
| BobBall | and the gate jobs are looking weird | 08:53 |
| BobBall | unfortunately the whole of the infra team are asleep | 08:54 |
| BobBall | seems as thought jenkins01.openstack.org is down for some reason | 08:55 |
| Ajaeger1 | BobBall, thanks | 08:56 |
| Ajaeger1 | BobBall, is there a way to inform everybody about this? | 08:56 |
| BobBall | Apart from sending a mail to openstack-dev? ;) | 08:58 |
| BobBall | but I'm not sure what the problem is | 08:58 |
| BobBall | I'm only guessing | 08:58 |
| BobBall | jobs are referring to jenkins01 but it's not responding to me | 08:58 |
| BobBall | could be something strange though | 08:58 |
| *** odyssey4me2 has joined #openstack-infra | 09:00 | |
| *** odyssey4me3 has joined #openstack-infra | 09:02 | |
| *** odyssey4me has quit IRC | 09:03 | |
| *** odyssey4me2 has quit IRC | 09:05 | |
| Ajaeger1 | BobBall, could you send one to openstack-dev, please? you did the hard work of figuring it out ;) | 09:13 |
| jd__ | I've opened https://bugs.launchpad.net/openstack-ci/+bug/1212990 | 09:17 |
| uvirtbot | Launchpad bug 1212990 in openstack-ci "Disturbance in the force render tests UNSTABLE" [Undecided,New] | 09:17 |
| openstackgerrit | Roman Podolyaka proposed a change to openstack-infra/config: Modify running of sqlalchemy-migrate tests https://review.openstack.org/39304 | 09:18 |
| * jd__ dials 555-JNKNS-BRKN | 09:18 | |
| *** dina_belova has quit IRC | 09:26 | |
| *** DennyZhang has joined #openstack-infra | 09:30 | |
| *** UtahDave has quit IRC | 09:47 | |
| *** thomasbiege has joined #openstack-infra | 09:49 | |
| *** morganfainberg is now known as morganfainberg_a | 09:49 | |
| *** thomasbiege has quit IRC | 09:50 | |
| *** morganfainberg_a is now known as morganfainberg | 09:52 | |
| *** jhesketh has quit IRC | 09:53 | |
| *** morganfainberg is now known as morganfainberg|a | 09:54 | |
| *** morganfainberg|a is now known as morganfainberg | 09:55 | |
| *** morganfainberg is now known as morganfainberg|a | 09:56 | |
| *** rpodolyaka has joined #openstack-infra | 10:03 | |
| *** zaro has quit IRC | 10:06 | |
| *** ruhe has quit IRC | 10:10 | |
| *** zaro has joined #openstack-infra | 10:26 | |
| *** dina_belova has joined #openstack-infra | 10:26 | |
| *** xchu has quit IRC | 10:27 | |
| *** dina_belova has quit IRC | 10:31 | |
| *** pcm_ has joined #openstack-infra | 10:32 | |
| *** pcm_ has quit IRC | 10:32 | |
| *** pcm_ has joined #openstack-infra | 10:33 | |
| *** dina_belova has joined #openstack-infra | 10:34 | |
| *** dina_belova has quit IRC | 10:34 | |
| *** rpodolyaka has left #openstack-infra | 10:38 | |
| *** mjfork has joined #openstack-infra | 10:39 | |
| *** dina_belova has joined #openstack-infra | 10:40 | |
| *** dina_bel_ has joined #openstack-infra | 10:40 | |
| *** dina_belova has quit IRC | 10:40 | |
| *** dina_bel_ has quit IRC | 10:42 | |
| *** yaguang has quit IRC | 10:43 | |
| comstud | who broked jenkins | 10:43 |
| BobBall | not me | 10:44 |
| BobBall | I blame Ajaeger1 | 10:44 |
| * jd__ pleads not guilty | 10:44 | |
| *** DennyZha` has joined #openstack-infra | 10:45 | |
| *** ruhe has joined #openstack-infra | 10:45 | |
| *** DennyZhang has quit IRC | 10:46 | |
| BobBall | how best to raise the alarm comstud ? | 10:47 |
| BobBall | or do we just wait for hours until someone comes back online | 10:47 |
| *** DennyZha` has quit IRC | 10:56 | |
| *** derekh has joined #openstack-infra | 10:57 | |
| fungi | #status alert some sort of gating disruption has been identified--looking into it now | 11:04 |
| fungi | mmm, joy. et tu statusbot? | 11:04 |
| BobBall | hehe | 11:05 |
| BobBall | looks like jenkins01 is down | 11:05 |
| BobBall | that was taking a number of the jobs | 11:05 |
| BobBall | might not be the cause though of course | 11:05 |
| BobBall | bbl | 11:05 |
| *** openstackstatus has joined #openstack-infra | 11:06 | |
| fungi | #status alert some sort of gating disruption has been identified--looking into it now | 11:07 |
| openstackstatus | NOTICE: some sort of gating disruption has been identified--looking into it now | 11:07 |
| *** ChanServ changes topic to "some sort of gating disruption has been identified--looking into it now" | 11:07 | |
| mordred | morning fungi | 11:08 |
| *** odyssey4me3 has quit IRC | 11:08 | |
| *** dims has joined #openstack-infra | 11:08 | |
| mordred | oh god. there's a disturbance in the force? | 11:08 |
| fungi | mordred: for some definitions of "good" | 11:08 |
| fungi | a great disturbance in the force, as if thousands of jenkins jobs cried out... | 11:10 |
| fungi | yeah | 11:10 |
| dims | i see 'em a sea of red | 11:11 |
| fungi | jenkins is in the process list on all three jenkins servers with start times from yesterday | 11:13 |
| dims | can't seem to get to logs.openstack.org either | 11:14 |
| fungi | and yet apache is running on static.o.o where that's served | 11:15 |
| fungi | is something rotten in the state of rackspace? | 11:16 |
| *** odyssey4me3 has joined #openstack-infra | 11:16 | |
| dims | fungi, managed to get url to one jenkins run - https://jenkins02.openstack.org/job/gate-nova-pep8/507/console | 11:17 |
| *** mriedem has joined #openstack-infra | 11:17 | |
| dims | 2013-08-16 10:54:12.503 | [SCP] Connecting to static.openstack.org | 11:18 |
| dims | 2013-08-16 10:54:14.221 | [SCP] Trying to create /srv/static/logs/16/42316 | 11:18 |
| dims | 2013-08-16 10:54:14.227 | ERROR: Failed to upload files | 11:18 |
| fungi | https://status.rackspace.com/ says everything is peachy | 11:18 |
| fungi | crap | 11:19 |
| fungi | /dev/mapper/main-logs 1.5T 1.5T 0 100% /srv/static/logs | 11:19 |
| mordred | fungi: I just arrived from a redeye international flight - so let me know if ... poop | 11:19 |
| dims | 2013-08-16 10:54:14.230 | at be.certipost.hudson.plugin.SCPSite.mkdirs(SCPSite.java:314) | 11:19 |
| mordred | I was going to say "tell me how to help" | 11:19 |
| dims | right, mkdirs is failing that would do it | 11:19 |
| fungi | #status alert the log server has filled up, disrupting job completion--working on it now, ETA 12:30 UTC | 11:21 |
| openstackstatus | NOTICE: the log server has filled up, disrupting job completion--working on it now, ETA 12:30 UTC | 11:21 |
| *** ChanServ changes topic to "the log server has filled up, disrupting job completion--working on it now, ETA 12:30 UTC" | 11:21 | |
| *** nayward has quit IRC | 11:22 | |
| fungi | at least this time all i need to do is tack an additional cinder volume onto that vg and grow the lv/fs | 11:23 |
| *** lucasagomes is now known as lucas-hungry | 11:24 | |
| mordred | fungi: yay! | 11:24 |
| *** lucas-hungry has left #openstack-infra | 11:26 | |
| fungi | it's times like this when i'm glad i make cold brew coffee concentrate and keep it in the fridge | 11:27 |
| fungi | for those following along at home, i'm basically following http://ci.openstack.org/static.html | 11:28 |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Using the jenkins user for tempest run https://review.openstack.org/42101 | 11:30 |
| dims | fungi, nice. thanks | 11:34 |
| *** yamahata has quit IRC | 11:35 | |
| *** thomasbiege has joined #openstack-infra | 11:42 | |
| fungi | oh joy... encountered the same xen page allocation failure trying to attach the block device as jeblair saw | 11:42 |
| fungi | rebooting static.o.o because the second nova nova volume-attach also faulted | 11:47 |
| *** thomasbiege has quit IRC | 11:48 | |
| *** dina_belova has joined #openstack-infra | 11:53 | |
| fungi | i've got /srv/static/logs doing an online resize to 2 terabytes now, so this train has just about sailed back into the garage | 11:56 |
| *** weshay has joined #openstack-infra | 11:57 | |
| *** dina_belova has quit IRC | 11:57 | |
| fungi | "The filesystem on /dev/main/logs is now 538171392 blocks long." | 11:58 |
| fungi | Filesystem Size Used Avail Use% Mounted on | 11:58 |
| fungi | /dev/mapper/main-logs 2.0T 1.5T 545G 74% /srv/static/logs | 11:58 |
| mordred | awesome. nicely done fungi | 12:00 |
| fungi | #status log server has a larger filesystem now--rechecking/reverifying jobs, ETA 12:30 UTC | 12:00 |
| fungi | #status alert log server has a larger filesystem now--rechecking/reverifying jobs, ETA 12:30 UTC | 12:00 |
| openstackstatus | NOTICE: log server has a larger filesystem now--rechecking/reverifying jobs, ETA 12:30 UTC | 12:00 |
| *** ChanServ changes topic to "log server has a larger filesystem now--rechecking/reverifying jobs, ETA 12:30 UTC" | 12:00 | |
| *** radix has joined #openstack-infra | 12:01 | |
| *** markmc has joined #openstack-infra | 12:01 | |
| *** CaptTofu has quit IRC | 12:02 | |
| *** CaptTofu has joined #openstack-infra | 12:02 | |
| *** CaptTofu has quit IRC | 12:04 | |
| *** CaptTofu has joined #openstack-infra | 12:04 | |
| *** ArxCruz has joined #openstack-infra | 12:04 | |
| *** CaptTofu has quit IRC | 12:05 | |
| *** CaptTofu has joined #openstack-infra | 12:05 | |
| BobBall | fungi? Xen page allocation? | 12:08 |
| fungi | BobBall: yeah, when we try to attach a new cinder volume in rackspace we've on occasion seen dmesg report a page allocation failure and then the block device never appears | 12:11 |
| BobBall | Got some logs from that? | 12:12 |
| BobBall | i.e. the actual failure from dmesg :) | 12:12 |
| fungi | i got that, then detached/deleted it, created a new one, tried to attach it, got the same failure again, rebooted the vm and the new volume showed up attached | 12:12 |
| BobBall | it's that whole reboot-in-the-middle step that destroys the logs, right? ;) | 12:12 |
| fungi | BobBall: yeah, though i'd almost guarantee it's the same as the one jeblair opened a trouble ticket on a couple months bach | 12:12 |
| fungi | BobBall: well, the syslog should have it when i get a moment to dig it out | 12:13 |
| fungi | the root fs wasn't full, just the place where we upload build logs | 12:13 |
| BobBall | 1.5TB is a lot of build logs | 12:13 |
| fungi | BobBall: we compress them aggressively, and delete them once they're 6 months old | 12:14 |
| fungi | so yes, that is a *lot* of logs | 12:14 |
| BobBall | Considered a compressed filesystem rather than individual logs? | 12:14 |
| BobBall | I guess the _vast_ majority of the logs are repetitive among groups | 12:14 |
| BobBall | or even multiple logs compressed as one and served from some form of applet would work | 12:15 |
| fungi | it's possible. the current plan is to relocate those into a swift object store instead though | 12:15 |
| BobBall | fair enough | 12:15 |
| fungi | we just need to write a frontend to emulate apache's mod_autoindex | 12:16 |
| fungi | and rework our uploads to go straight to swift and then update the index on the "log server" (which will then only contain a log of the logs) | 12:17 |
| BobBall | btw - diff subject... why does zuul sometimes not have any stylesheet? Seems to be at least 1/4 of the time | 12:18 |
| *** mberwanger has joined #openstack-infra | 12:18 | |
| fungi | sounds like your browser may not be grabbing it either because something's lagging/failing to serve in apache/fs-side or maybe the problem is on your end or packet loss in between | 12:20 |
| *** odyssey4me3 has quit IRC | 12:20 | |
| fungi | i assume the page source still shows the link for it. maybe try reloading the css file a few times and see if you get errors? | 12:20 |
| BobBall | The requested URL /bootstrap/css/bootstrap.min.css was not found on this server | 12:21 |
| BobBall | same problem time and time again :D | 12:21 |
| BobBall | http://zuul.openstack.org/bootstrap/css/bootstrap.min.css doesn't exist, as it suggests | 12:21 |
| BobBall | same with http://zuul.openstack.org/bootstrap/css/bootstrap-responsive.min.css (of course) | 12:22 |
| fungi | BobBall: yeah, you want http://status.openstack.org/zuul | 12:22 |
| BobBall | don't suppose that zuul.openstack.org is loadbalanced and maybe some are set up and not others? ;) | 12:22 |
| BobBall | omg... I'm that stupid... | 12:22 |
| fungi | http://zuul.openstack.org/ is a demo of its built-in example status interface | 12:22 |
| *** ruhe has quit IRC | 12:22 | |
| BobBall | when it works I use the right URL and when it doesn't I use the wrong URL! | 12:23 |
| fungi | but we don't ship bootstrap with zuul, and we don't use bootstrap in openstack's status interface for it | 12:23 |
| BobBall | I am suitably embarassed. | 12:23 |
| BobBall | thanks | 12:23 |
| * fungi spends his life embarrassed | 12:23 | |
| fungi | no worries | 12:24 |
| *** ruhe has joined #openstack-infra | 12:24 | |
| *** odyssey4me3 has joined #openstack-infra | 12:28 | |
| fungi | okay, sifting through build failures, the first sign of trouble i see in job results seems to start around 0830z | 12:30 |
| BobBall | btw, is it worth adding a cron job to check things like disk usage or jenkins job usage and give the infra team a warning when it's at some high percentage? | 12:30 |
| *** SergeyLukjanov has quit IRC | 12:30 | |
| *** woodspa has joined #openstack-infra | 12:31 | |
| fungi | it's been brought up before, and we can discuss it in tuesday's meeting if you like | 12:32 |
| *** mberwanger has quit IRC | 12:33 | |
| *** mberwanger has joined #openstack-infra | 12:34 | |
| fungi | ultimately i think we'd be better off with an extensible monitoring system if we decide to go that route, but many, many years as a data center ops person probably clouds my opinions there | 12:34 |
| *** mberwang_ has joined #openstack-infra | 12:35 | |
| fungi | so far consensus has been that we benefit more from finding ways to proactively make important systems resistant to failure rather than spend time reacting to failure alerts | 12:36 |
| BobBall | yeah | 12:36 |
| BobBall | that's the cloudy way ;) | 12:36 |
| fungi | and i think there may be some concern that starting to rely on a monitoring system will steer us to always reacting to alerts rather than just removing the risks indefinitely | 12:37 |
| mordred | ++ | 12:37 |
| mordred | for instance - the swift engineering stuff | 12:37 |
| *** Ajaeger1 has left #openstack-infra | 12:37 | |
| fungi | yup, precisely | 12:37 |
| mordred | also - our devs are like the best monitoring system ever :) | 12:37 |
| BobBall | haha | 12:37 |
| BobBall | 08:46 < jd__> I sense unstability in the gates | 12:38 |
| fungi | yeah, seriously. i woke up, saw people in irc complaining that i sleep too much (just kidding) and was able to fairly immediately find the issue | 12:38 |
| BobBall | 16 minutes ;) not bad ;) | 12:38 |
| *** mberwanger has quit IRC | 12:38 | |
| fungi | but the down side there is that "devs noticing something is broken" is reaction to a failure rather than warning of an impending failure | 12:39 |
| fungi | #status ok still rechecking/reverifying false negative results on changes, but the gate is moving again | 12:41 |
| openstackstatus | NOTICE: still rechecking/reverifying false negative results on changes, but the gate is moving again | 12:41 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/+milestone/grizzly | https://github.com/openstack-infra/config" | 12:41 | |
| *** thomasbiege has joined #openstack-infra | 12:41 | |
| mordred | fungi: do you have a script for those recheck/reverfys? | 12:43 |
| mordred | fungi: I see you do that from time to time and I always wonder if it's something I can learn how to do sensibly? | 12:43 |
| mordred | or if you just brute force it | 12:44 |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Add the openstackstatus bot to #openstack-dns https://review.openstack.org/42335 | 12:47 |
| *** dims has quit IRC | 12:48 | |
| *** thomasbiege2 has joined #openstack-infra | 12:48 | |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Add the openstack bot to #openstack-dns https://review.openstack.org/42336 | 12:48 |
| *** thomasbiege has quit IRC | 12:49 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Using the jenkins user for tempest run https://review.openstack.org/42101 | 12:50 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Rework run_shell_command https://review.openstack.org/42337 | 12:50 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Ensure that setup_requires are installed https://review.openstack.org/42338 | 12:50 |
| *** sandywalsh has quit IRC | 12:52 | |
| fungi | mordred: i mostly go through and clicky-clicky because i need to look and see if it failed in the right window, if someone else already left a recheck or reverify, if they did it correctly, and i usually also remove the negative check votes from jenkins too | 12:53 |
| *** dina_belova has joined #openstack-infra | 12:53 | |
| fungi | tough to automate, though probably not impossible | 12:53 |
| *** mberwang_ has quit IRC | 12:56 | |
| openstackgerrit | Roman Podolyaka proposed a change to openstack-infra/config: WIP: Run Nova DB API tests on MySQL and PostgreSQL https://review.openstack.org/42142 | 12:56 |
| *** dina_belova has quit IRC | 12:57 | |
| *** ruhe has quit IRC | 12:58 | |
| fungi | i also generally try to remember to unsubscribe myself from the change when i recheck/reverify in situations like this, otherwise my review queue ends up full of mud | 13:00 |
| *** SergeyLukjanov has joined #openstack-infra | 13:00 | |
| *** jjmb has quit IRC | 13:01 | |
| *** anteaya has joined #openstack-infra | 13:01 | |
| mordred | fungi: oh - you can do that? | 13:02 |
| fungi | mordred: after leaving a recheck or reverify comment, i hit the little X button next to my empty vote (and next to jenkins -1 vote if it was a recheck) | 13:03 |
| fungi | that keeps it out of my changes list | 13:03 |
| mordred | ahhh. nice. i should start doing that | 13:03 |
| BobBall | Speaking of which, why can I remove some people and not others? e.g. on https://review.openstack.org/#/c/42144/ I can remove Cristopher but not Mate | 13:04 |
| *** thomasbiege2 has quit IRC | 13:05 | |
| *** dims has joined #openstack-infra | 13:05 | |
| *** sandywalsh has joined #openstack-infra | 13:07 | |
| mordred | https://review.openstack.org/#/c/42335/ <-- we should probably at some point make that a list of things somewhere | 13:09 |
| mordred | BobBall: hrm. interesting. no idea | 13:09 |
| markmc | you know what's really annoying? | 13:10 |
| *** russellb is now known as rustlebee | 13:11 | |
| markmc | well, gerrit's auto-completion for adding people to a review | 13:11 |
| markmc | that's annoying | 13:11 |
| *** comstud is now known as bearhands | 13:11 | |
| * markmc constantly adds the wrong people | 13:11 | |
| markmc | but what's super annoying, is you can't remove them from the review then | 13:11 |
| markmc | only they can remove themselves | 13:11 |
| BobBall | I was going to say when you can't stab a pea with your fork and it falls off the table... but yes, that's also annoying | 13:11 |
| markmc | BobBall, a cheeky two year old half-heartedly trying to eat peas with a fork | 13:12 |
| markmc | and they *all* fall off the table | 13:12 |
| markmc | now that's annoying | 13:12 |
| * markmc gets it all out there | 13:12 | |
| *** mberwanger has joined #openstack-infra | 13:12 | |
| mordred | wow. | 13:12 |
| * mordred knows what really bugs markmc now | 13:12 | |
| BobBall | haha | 13:13 |
| markmc | mordred, you're worse than a two year old | 13:13 |
| mordred | markmc: I'm SO much worse than a two year old | 13:13 |
| markmc | heh | 13:13 |
| mordred | although - funny story. | 13:13 |
| mordred | I just did a thing in Brazil and was flying back home last night | 13:13 |
| *** thomasbiege has joined #openstack-infra | 13:14 | |
| mordred | and the guy sitting next to me heard me talking on the phone to my mom before we took off about what I'd done | 13:14 |
| mordred | when I hung up the phone, he turns to me and says, so, you do OpenStack? | 13:14 |
| mordred | and then told me that he's just started doing openStack at VMware | 13:14 |
| mordred | and is working with Dan | 13:14 |
| mordred | the world is VERY VERY small | 13:14 |
| markmc | heh, that's fun | 13:14 |
| markmc | not some random "I read about that in the NYT" | 13:14 |
| mordred | nope. like, he's actually working on it | 13:15 |
| mordred | told me about things they're running internally that I probably shouldn't know about and whatnot | 13:15 |
| mordred | but actually fun to talk to | 13:15 |
| fungi | i had the same experience on a flight back from the havana summit, some guy next to me was on his way back from a vacation in colorado and worked for netapp in rtp. i talked him into starting to come to the local user group meetings | 13:15 |
| mordred | very interesting perspective - he's been doing vmware since it was an upstart thing that people ran under their desks and didn't tell anyone about - finds it weird to now be the bad guys | 13:16 |
| fungi | though i guess not the exact same experience, since he wasn't working on openstack yet | 13:16 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Fixing override-votes for gerrit trigger https://review.openstack.org/42341 | 13:16 |
| *** dina_belova has joined #openstack-infra | 13:16 | |
| mordred | fungi: nice - the people I talk to on planes who are tech and are not doing openstack already glaze over quickly | 13:16 |
| *** zul has quit IRC | 13:16 | |
| mordred | btw - I should probably tell people | 13:17 |
| mordred | I've unwatched jjb | 13:17 |
| mordred | I do not feel competent to review most of the patches | 13:17 |
| fungi | well, i was just trying to get work done and he saw openstack logos on my netbook screen and started asking all sorts of questions | 13:17 |
| mordred | I will review something if specifically asked, but in general, I think the jjb team is doing a great job | 13:17 |
| fungi | well this is no good. i'm intermittently unable to load the web interface for jenkins01 | 13:21 |
| fungi | proxy timeouts from its web service | 13:21 |
| *** mberwanger has quit IRC | 13:21 | |
| *** dprince has joined #openstack-infra | 13:22 | |
| *** rfolco has joined #openstack-infra | 13:22 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/reviewstats: Clint's name is different on gerrit https://review.openstack.org/42343 | 13:23 |
| *** dansmith is now known as Steely_Dan | 13:24 | |
| *** pentameter has joined #openstack-infra | 13:24 | |
| *** thomasbiege has quit IRC | 13:24 | |
| openstackgerrit | A change was merged to openstack-infra/config: Added murano-common project https://review.openstack.org/41634 | 13:25 |
| *** thomasbiege has joined #openstack-infra | 13:25 | |
| fungi | well, not so much intermittently now | 13:26 |
| *** dina_belova has quit IRC | 13:26 | |
| fungi | yeah, there are a ton of jobs hung at 100% in the zuul status page and they all were running on jenkins01 | 13:27 |
| fungi | the server seems to be basically idle (load average of 0.00 now) but jenkins is still running in the process list | 13:28 |
| fungi | i suspect it's gone to lunch | 13:29 |
| fungi | unfortunately, i suspect even if i force it to restart, zuul is going to think those jobs are still running? | 13:29 |
| *** xchu has joined #openstack-infra | 13:30 | |
| fungi | jeblair: would know, but it's probably still too early for him | 13:30 |
| jd__ | thanks fungi btw | 13:30 |
| fungi | jd__: you're welcome | 13:30 |
| fungi | well, restarting the jenkins process on jenkins01 and seeing what zuul thinks once that's back up and responding | 13:32 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Clint's name is different on gerrit https://review.openstack.org/42343 | 13:32 |
| fungi | #status alert the earlier log server issues seem to have put one of the jenkins servers in a bad state, blocking the gate--working on that, ETA 14:00 UTC | 13:34 |
| openstackstatus | NOTICE: the earlier log server issues seem to have put one of the jenkins servers in a bad state, blocking the gate--working on that, ETA 14:00 UTC | 13:34 |
| *** ChanServ changes topic to "the earlier log server issues seem to have put one of the jenkins servers in a bad state, blocking the gate--working on that, ETA 14:00 UTC" | 13:34 | |
| fungi | jenkins didn't stop cleanly on the server either. giving it a few minutes to react to the sigterms before i give up and send more drastic sigs | 13:35 |
| *** openstackgerrit has quit IRC | 13:36 | |
| *** openstackgerrit has joined #openstack-infra | 13:37 | |
| *** xBsd has joined #openstack-infra | 13:37 | |
| fungi | zomg zuul is awesomesauce | 13:39 |
| mordred | yeah it is | 13:40 |
| fungi | i had to beat jenkins01 senseless to get the old jvm to diaf | 13:40 |
| mordred | fungi: are we holding off on landing nodepool patches for jeblair ? | 13:40 |
| openstackgerrit | A change was merged to openstack-infra/gitdm: add Deutsche Telekom colleagues and update domain-map https://review.openstack.org/42155 | 13:40 |
| *** simonmcc has quit IRC | 13:40 | |
| fungi | mordred: i'm holding off nothing. i think i reviewed them all last night before i passed out, but if memory serves i spotted an issue in the image/provider yaml file which needs fixing | 13:41 |
| mordred | fungi: was just asking before the first one I reviewed had 2x +2 and no aprv | 13:41 |
| mordred | so I wasn'tsure if we were letting jeblair control landing | 13:41 |
| fungi | oh, yeah i didn't approve any of them in case there was some subtle ordering which needed ot be observed between projects there | 13:42 |
| fungi | and also because it was late and i don't think anyone was around in case something broke from me merging a change prematurely | 13:43 |
| *** thomasbiege has quit IRC | 13:43 | |
| *** simonmcc has joined #openstack-infra | 13:45 | |
| mordred | yah | 13:45 |
| *** thomasbiege has joined #openstack-infra | 13:45 | |
| *** SergeyLukjanov has quit IRC | 13:45 | |
| sandywalsh | mordred, hey! Is there a way for me to see all my review submissions in gerrit? (not my code reviews, but my branch submissions). I'm looking for an old patch that got squashed out of my local repo. | 13:46 |
| *** simonmcc has joined #openstack-infra | 13:47 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Fixed timeout wrapper https://review.openstack.org/42348 | 13:48 |
| *** burt has joined #openstack-infra | 13:48 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:51 | |
| fungi | #status ok the gate seems to be properly moving now, but some changes which were in limbo earlier are probably going to come back with negative votes now. rechecking/reverifying those too | 13:53 |
| openstackstatus | NOTICE: the gate seems to be properly moving now, but some changes which were in limbo earlier are probably going to come back with negative votes now. rechecking/reverifying those too | 13:53 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/+milestone/grizzly | https://github.com/openstack-infra/config" | 13:53 | |
| *** beagles is now known as seagulls | 13:54 | |
| *** bnemec is now known as oldben | 13:54 | |
| *** ruhe has joined #openstack-infra | 13:54 | |
| jeblair | fungi: mrmrmmmmm | 13:55 |
| fungi | jeblair: actually, there's still a problem | 13:55 |
| fungi | jeblair: i just noticed that now the only running changes are on 01 and everything zuul still sees running on 02 is stuck at 100% and i can't get to its web interface now | 13:55 |
| fungi | so same thing i was seeing earlier on 01. seems it eventually hit 02 as well, just took longer | 13:56 |
| jeblair | fungi: 1 sec before you restart | 13:59 |
| fungi | jeblair: definitely. i'm holding off so you can grab sufficient diags | 13:59 |
| jeblair | jstack -F 15236 >stack | 14:00 |
| jeblair | for the record | 14:00 |
| *** ryanpetrello has joined #openstack-infra | 14:02 | |
| *** datsun180b has joined #openstack-infra | 14:02 | |
| fungi | i've added that to a more prominent part of my notes. i went looking for it earlier but didn't want to leave things broken long enough for me to track down the jstack utility cli | 14:03 |
| *** datsun180b has quit IRC | 14:03 | |
| *** datsun180b has joined #openstack-infra | 14:03 | |
| fungi | anyway, if you're ready for me to restart the jenkins process, let me know | 14:03 |
| fungi | or feel free to restart it yourself when you're ready | 14:04 |
| *** HenryG has joined #openstack-infra | 14:05 | |
| *** DennyZhang has joined #openstack-infra | 14:05 | |
| jeblair | this is weird; i want more time to look | 14:06 |
| fungi | jeblair: absolutely | 14:06 |
| yolanda | hi, i am trying to use git branches from zuul in jenkins, that syntax should be correct, or am i missing something? http://host/p/path/to/project.git | 14:07 |
| yolanda | i'm receiving an error like Not a git repository, Request not supported | 14:07 |
| *** mjfork has left #openstack-infra | 14:07 | |
| yolanda | if i try like http://host/p/ it also shows errors | 14:07 |
| *** krtaylor has quit IRC | 14:08 | |
| *** changbl has quit IRC | 14:10 | |
| fungi | yolanda: does the apache configuration for your zuul vhost have "ScriptAlias /p/ /usr/lib/git-core/git-http-backend/" and so on to provide access to those git repositories? | 14:11 |
| yolanda | fungi, it has it | 14:11 |
| fungi | yolanda: for reference, https://git.openstack.org/cgit/openstack-infra/config/tree/modules/zuul/templates/zuul.vhost.erb#n17 | 14:12 |
| fungi | is your apache error.log complaining about those requests? | 14:12 |
| yolanda | fungi, is exactly like that | 14:13 |
| yolanda | error.log complains if i try to fetch some repo, but not for the /p/ call | 14:13 |
| yolanda | although it shows error on website | 14:13 |
| yolanda | on browser | 14:13 |
| fungi | does the path your access.log says it served for those requests actually exist on the filesystem? | 14:13 |
| yolanda | fungi, for the /p/ it triggers a notfound | 14:14 |
| yolanda | "GET /p/ HTTP/1.1" 404 281 "-" "Wget/1.14 (linux-gnu)" | 14:14 |
| fungi | yolanda: does "/usr/lib/git-core/git-http-backend" exist on your server? | 14:14 |
| yolanda | yes | 14:14 |
| yolanda | i can call it from command line | 14:15 |
| *** koobs has quit IRC | 14:16 | |
| fungi | toddmorey's not in irc :( | 14:17 |
| fungi | i've got someone asking about correcting a mistake in the talks list | 14:17 |
| *** dprince has quit IRC | 14:18 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Change use of error numbers to errno https://review.openstack.org/42356 | 14:18 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Update style checking to hacking https://review.openstack.org/42357 | 14:18 |
| *** DennyZhang has left #openstack-infra | 14:23 | |
| fungi | yolanda: so when you try to fetch a change and are getting the request not supported error, are there corresponding entries in your apache access.log, and do they correspond to the actual git repositories on your zuul server's filesystem? | 14:23 |
| yolanda | fungi, yes | 14:24 |
| fungi | also, are they readable by the user apache is running as? | 14:24 |
| yolanda | but even the simplest call, /p/ is triggering a 404 | 14:24 |
| yolanda | it has read access for everyhone | 14:24 |
| yolanda | everyone | 14:24 |
| openstackgerrit | A change was merged to openstack-infra/git-review: No longer check for new git-review releases https://review.openstack.org/42214 | 14:25 |
| openstackgerrit | A change was merged to openstack-infra/git-review: Migrate to pbr. https://review.openstack.org/35486 | 14:26 |
| *** dina_belova has joined #openstack-infra | 14:27 | |
| jeblair | fungi: i think the last time i did this we were running jdk6; i think the thread dumps have changed in jdk7, and not for the better | 14:27 |
| fungi | :( | 14:27 |
| *** cthulhup_ has joined #openstack-infra | 14:28 | |
| mordred | YAY | 14:28 |
| mordred | I love it when things change and not for the better | 14:28 |
| yolanda | fungi, i removed the final slash from ScriptAlias /p/ /usr/lib/git-core/git-http-backend | 14:29 |
| yolanda | now i have different error fatal: GIT_PROJECT_ROOT is set but PATH_INFO is not | 14:29 |
| yolanda | but seems that it's doing something, final slash didn't like it | 14:29 |
| fungi | interesting. i wonder if an extra slash is making its way into the request | 14:30 |
| *** zul has joined #openstack-infra | 14:31 | |
| *** dina_belova has quit IRC | 14:32 | |
| mrodden | gfdi... i was just ordered to put access controls on our internal replica of the openstack sources, "becauase of policy and compliance" | 14:32 |
| mrodden | what part of "open source" is difficult to understand... | 14:32 |
| fungi | yolanda: seems to get reported at https://github.com/git/git/blob/master/http-backend.c#L533 for what it's worth | 14:32 |
| *** dprince has joined #openstack-infra | 14:32 | |
| mordred | jeblair: btw - I'm +2ing all of your nodepool changes but leaving them unapproved as I think there are some dependency landing orders going on in there | 14:33 |
| fungi | mrodden: you should agree to restrict read access to that source code to the same set of individuals we do | 14:33 |
| mordred | mrodden: ha. yeah. definitely access controls on a copy of readily available source code are necessary | 14:34 |
| jeblair | mordred: thx | 14:34 |
| mrodden | yeah they want me to put ACLs on it for read access and above | 14:34 |
| jeblair | mordred, fungi: so a lot of threads are waiting to lock the Jenkins object while in the process of performing a delete | 14:34 |
| mrodden | and then have a separate list for each one so its "compartmentalized" | 14:34 |
| mrodden | and approve each request for access to it | 14:34 |
| mrodden | joy | 14:34 |
| jeblair | i can't find a way to determine what thread _has_ that lock | 14:34 |
| mordred | jeblair: AWESOME | 14:35 |
| mordred | jeblair: start deleting threads until the lock goes away - that's which one has it? | 14:35 |
| yolanda | fungi, this shouldn't be empty, just provided from apache, right? | 14:35 |
| jeblair | mordred: heh | 14:35 |
| fungi | jeblair: well, when i restarted jenkins on jenkins01 it seemed that lock got released, so presumably the thread holding that lock is still live | 14:36 |
| mordred | maybe this is a thing to raise in #jenkins - tons of java dudes in there | 14:36 |
| mordred | I mean, I can't imagine that java doesn't have a way to discover what you'relooking for | 14:36 |
| fungi | yolanda: right, anything matching the alias patterns gets served directly from the filesystem and then everything else under /p/ gets sent to the git-http-backend cgi via the scriptalias there | 14:36 |
| *** nayward has joined #openstack-infra | 14:37 | |
| *** dina_belova has joined #openstack-infra | 14:37 | |
| fungi | yolanda: this is no different from how we serve git repositories from the filesystem via http on review.openstack.org or git.openstack.org | 14:38 |
| yolanda | mm, problem i'm facing now is that PATH_INFO var, that is empty | 14:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix intermittent jenkins plugin build failure https://review.openstack.org/42062 | 14:39 |
| *** boris-42 has quit IRC | 14:39 | |
| mordred | btw - I'm going to try to start working through review queue first thing in the morning, because I've been doing way more coding than reviewing lately, and that's not cool | 14:40 |
| fungi | yolanda: i believe $PATH_INFO gets passed to the cgi in the calling environment only if there's something after the cgi alias in the url. so http://host/p/ won't have anything $PATH_INFO but http://host/p/foo will | 14:41 |
| jeblair | mordred: well, that got them quiet. | 14:41 |
| *** dina_belova has quit IRC | 14:41 | |
| fungi | yolanda: so that error might just be artifact of you trying to browse /p/ directly | 14:42 |
| jeblair | mordred: (asking in #jenkins) | 14:42 |
| *** _TheDodd_ has joined #openstack-infra | 14:42 | |
| jeblair | oh, i just found the jbb command 'threadlocks' | 14:43 |
| fungi | yeah, everyone's suddenly afraid to talk in there, for fear you'll ask them directly ;) | 14:43 |
| *** pentameter has quit IRC | 14:43 | |
| fungi | crickets | 14:43 |
| jeblair | Command 'threadlocks' is not supported on the target VM | 14:43 |
| jeblair | maybe that has something to do with why the stacktraces have less lock info. | 14:44 |
| mordred | how do we get them to be supported? | 14:45 |
| jeblair | mordred: i dunno, open a support contract with oracle? | 14:45 |
| fungi | inherit from something besides the notimplemented factory? | 14:46 |
| jeblair | ha | 14:46 |
| *** ruhe has quit IRC | 14:47 | |
| jeblair | Command 'lock' is not supported on a read-only VM connection | 14:47 |
| *** rnirmal has joined #openstack-infra | 14:48 | |
| mordred | wow | 14:48 |
| mordred | what a great design | 14:48 |
| jeblair | i mean, we could read the stacktraces of all 843 threads and figure it out. | 14:50 |
| fungi | i'm going to take this lull as an opportunity to finish my breakfast and grab a quick shower, since i've been glued to irc from the moment i woke up. bbiab | 14:50 |
| jeblair | it's actually _less_ than 10,000 lines of stacktrace | 14:51 |
| *** krtaylor has joined #openstack-infra | 14:51 | |
| jeblair | but not by much | 14:51 |
| mordred | jeblair: I'm writing an email to the internal java dev guys to see if they have any idea | 14:51 |
| *** koobs has joined #openstack-infra | 14:51 | |
| yolanda | fungi, if i have my project for example in /var/lib/zuul/git/sunnyvale/openstack/ceilometer, a git clone like git clone git://91.189.93.35/p/sunnyvale/openstack/ceilometer.git should be ok? | 14:52 |
| *** rcleere has joined #openstack-infra | 14:52 | |
| jeblair | mordred: i have a read-only connection to the vm because i'm using jsadebugd (which i believe i need because we did not have the "foresight" to run all of our production servers with the debug server enabled all the time). | 14:53 |
| yolanda | it triggers a Not a git repository error | 14:53 |
| mordred | jeblair: lovely | 14:53 |
| mordred | yolanda: did you git init the directory? | 14:54 |
| yolanda | yes | 14:54 |
| mordred | yolanda: and/or do you know about manage-projects from jeepyb? | 14:54 |
| mordred | ok | 14:54 |
| yolanda | mordred, directory is a git repo, i can do git calls from it, and i don't know about manage-projects | 14:55 |
| fungi | yolanda: note that using git:// isn't likely going through apache. are you sure you're not using http://91.189.93.35/p/sunnyvale/openstack/ceilometer.git instead? | 14:55 |
| mordred | yolanda: https://git.openstack.org/cgit/openstack-infra/jeepyb is your friend | 14:56 |
| yolanda | fungi, an http just triggers me the not found | 14:56 |
| fungi | or https? | 14:56 |
| mordred | yolanda: it lets you manage the list of projects that are in gerrit from a yaml file | 14:56 |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Using the root user for tempest run https://review.openstack.org/42101 | 14:56 |
| yolanda | mm, from command line http://91.189.93.35/p/sunnyvale/openstack/ceilometer.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server? | 14:56 |
| yolanda | mordred, what i'm trying is to setup zuul refs to work with jenkins | 14:56 |
| mordred | yolanda: it also has provisions/cron-job for following upstream | 14:57 |
| mordred | yup | 14:57 |
| mordred | yolanda: you have to init the repo differently for it to be a server directory... | 14:57 |
| mordred | yolanda: git --bare init | 14:57 |
| *** fifieldt_ has quit IRC | 14:58 | |
| fungi | mordred: zuul needs non-bare repos though, right? at least the ones on zuul.o.o are non-bare in /var/lib/zuul/git | 14:58 |
| *** woodspa_ has joined #openstack-infra | 15:01 | |
| yolanda | mordred, fungi, my repos just were automatically created in zuul, cloning from gerrit | 15:01 |
| BobBall | mordred: not sure we can fix it, but I've been hit a few times by the replacement sudo when using GLOBAL_VENV - e.g. being told to "pip install -U git-review", which when run with sudo installs it in the venv rather than the root, so git review doesn't pick it up | 15:02 |
| fungi | right, it will construct those clones if they don't already exist | 15:02 |
| mordred | oh. these are the zuul repos, not the gerrit mirror repos | 15:03 |
| mordred | gotcha | 15:03 |
| * mordred shuts up | 15:03 | |
| *** yaguang has joined #openstack-infra | 15:03 | |
| mordred | BobBall: ah. interesting | 15:03 |
| mordred | BobBall: I think ... I think we can - but I think we might need to be clever | 15:03 |
| mordred | OR | 15:03 |
| *** ruhe has joined #openstack-infra | 15:03 | |
| *** ^d has joined #openstack-infra | 15:03 | |
| jeblair | i love that the thread list has one kind of thread id, and the stack trace another | 15:03 |
| *** ^d has quit IRC | 15:04 | |
| *** ^d has joined #openstack-infra | 15:04 | |
| mordred | BobBall: we could put a source $dest/.venv/bin/activate into stackrc | 15:04 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gearman-plugin: remove restriction on slave to run single job at a time https://review.openstack.org/42226 | 15:04 |
| mordred | BobBall: would that be a sensible workflow? or should we get cleverer | 15:04 |
| BobBall | stackrc? not sure I understand why that'd help? | 15:04 |
| *** woodspa has quit IRC | 15:04 | |
| mordred | because that's output by devstack, yeah? with keys and stuff | 15:05 |
| mordred | so don't you normally source that before doing things on the box anyway? | 15:05 |
| BobBall | you mean openrc? | 15:05 |
| mordred | yes. that is what Imean | 15:05 |
| BobBall | Ah - yes - that'd help | 15:05 |
| *** colinmcnamara has joined #openstack-infra | 15:06 | |
| BobBall | or have our own sudo alias in devstack that sets it to use the venv? might that work? | 15:06 |
| *** pabelanger is now known as nubbie | 15:10 | |
| *** dina_belova has joined #openstack-infra | 15:10 | |
| *** dina_belova has quit IRC | 15:12 | |
| *** dina_belova has joined #openstack-infra | 15:12 | |
| *** nsaje1 has joined #openstack-infra | 15:12 | |
| *** changbl has joined #openstack-infra | 15:12 | |
| *** yaguang has quit IRC | 15:13 | |
| nsaje1 | hey guys, a question: is it possible to patch devstack to install MongoDB from a 3rd party repo? 10gen repo to be exact, since Ubuntu hasn't backported Mongo 2.2 yet | 15:14 |
| nsaje1 | Ceilometer requires Mongodb >=2.2 so the API doesn't start on devstack gate now | 15:14 |
| *** cthulhup_ has quit IRC | 15:15 | |
| mriedem | nsaje1: ceilometer requires pymongo, not necessarily mongodb, right? | 15:15 |
| mriedem | i mean typically you'd use mongodb as the backing store | 15:16 |
| mriedem | nsaje1: but you can swap out the backend | 15:16 |
| *** nayward has quit IRC | 15:17 | |
| *** mrodden has quit IRC | 15:17 | |
| nsaje1 | mriedem: yes, but mongodb is in use in devstack right now | 15:17 |
| *** markmcclain has joined #openstack-infra | 15:17 | |
| nsaje1 | mriedem: I'd only install from a 3rd party until mongodb is backported | 15:18 |
| nsaje1 | mriedem: well, I'm asking if it's possible :) | 15:18 |
| *** seagulls has quit IRC | 15:20 | |
| *** beagles has joined #openstack-infra | 15:21 | |
| nsaje1 | mriedem: wouldn't want to rewrite the devstack ceilometer script using SQLAlchemy just because the right mongodb version isn't available :/ | 15:21 |
| jeblair | mriedem: i believe jd__ and some folks in here discussed the issue yesterday | 15:21 |
| *** nayward has joined #openstack-infra | 15:22 | |
| mriedem | nsaje1: jeblair: ok, was just thinking out loud more or less, the ceilometer stuff is relatively new to me. | 15:22 |
| jeblair | mriedem: i'm a bit distracted right now with operational issues, and don't recall the outcome... jd__ would probably remember, or you could check eavesdrop | 15:22 |
| mriedem | i only bring it up because we're looking at it in ibm because of the db2 10.5 usage with pymongo | 15:22 |
| mordred | nsaje1: I believe jd__ said that 2.2 isn't actually requirede | 15:23 |
| mordred | nsaje1: and that there is a bug which is making it seem so | 15:23 |
| mriedem | because we can't ship mongodb | 15:23 |
| mordred | nsaje1: but I could be wrong about that | 15:23 |
| mordred | but - in general, we don't install packages from third party sources in devstack, because if we start doing that, we run the risk of becoming our own linux distro | 15:24 |
| *** dprince has quit IRC | 15:24 | |
| ryanpetrello | for stackforge projects that have pypi uploads configured for release | 15:24 |
| mordred | I tihnk we need to get ceilometer gating on devstack, so that if someone tries to add a feature that requires 2.2, that feature would not land | 15:24 |
| ryanpetrello | how does authentication to pypi happen? | 15:24 |
| nsaje1 | mordred: I thought as much | 15:24 |
| ryanpetrello | is this something that stackforge project maintainers actually do by hand? | 15:24 |
| mordred | ryanpetrello: openstackci user | 15:25 |
| mordred | ryanpetrello: you need to add that as an owner or manager to your pypi project by hand (usually also running python setup.py register by hand first) | 15:25 |
| ryanpetrello | okay, figured there had to be some way to give permission | 15:25 |
| mordred | ryanpetrello: and at that point, openstack can upload to your pypi thing | 15:25 |
| ryanpetrello | is this documented somewhere that I just missed it? | 15:25 |
| mordred | possibly not | 15:25 |
| mordred | if it's not on the stackforge document, then no | 15:25 |
| ryanpetrello | okay | 15:26 |
| *** yaguang has joined #openstack-infra | 15:26 | |
| mordred | ryanpetrello: python setup.py register ; log in to pypi ; click the link of your project name on the right ; click "Role" ; then add openstackci | 15:26 |
| mordred | is the tl;dr process | 15:26 |
| ryanpetrello | yup | 15:26 |
| ryanpetrello | thanks :) | 15:26 |
| fungi | i think we originally tried not to get too into the weeds on all the available options within our automation, for fear of scaring projects away from stackforge because it looks complicated | 15:27 |
| mordred | jeblair: I have sent an email to a pile of java devs, but have not heard back yet | 15:27 |
| mordred | fungi: I want to add pypi registration to manage_projects at some point | 15:28 |
| fungi | so i think that howto mentions that we have the ability to upload projects to pypi, but doesn't get into the details | 15:28 |
| *** mrodden has joined #openstack-infra | 15:28 | |
| mordred | fungi: because it would be nice to just have openstackci create the darned thing itself | 15:28 |
| *** colinmcnamara has quit IRC | 15:28 | |
| mordred | fungi: or maybe to upload - if it sees the project doesn't exist, it does a register | 15:28 |
| mordred | fungi: if it does exist and it can't upload to it - that'sa normal error and one that someone would need to correct | 15:29 |
| *** colinmcnamara has joined #openstack-infra | 15:29 | |
| ryanpetrello | fwiw, I didn't find it that complicated | 15:29 |
| ryanpetrello | and I'm super impressed now that we have it up and running | 15:29 |
| mordred | woot! | 15:29 |
| mordred | that's what we like to hear | 15:29 |
| ryanpetrello | the idea of being able to sign and tag a release, run tests, and then upload to pypi is *so nice* | 15:29 |
| ryanpetrello | the pypi thing I think was my only confusion point | 15:29 |
| ryanpetrello | otherwise the stackforge doc was very clear | 15:30 |
| fungi | mordred: would be neat, agreed | 15:30 |
| mordred | ryanpetrello: one of these days, I'm going to convince someone add the ability to review tags in gerrit | 15:30 |
| mordred | ryanpetrello: so that you could propose a tag, have that get tested/reviewed as usual, and when it lands, then trigger the upload | 15:30 |
| mordred | but - that requires java hacking | 15:30 |
| mordred | ew | 15:30 |
| fungi | mordred: along those lines, it would also be cool if we came up with a way to inject detached developer pgp signatures of the tarballs for upload to pypi, but that's going to be human-workflow-complicated i think | 15:31 |
| *** yaguang has quit IRC | 15:31 | |
| fungi | automated signing by contrast is easy to implement, but less meaningful overall | 15:32 |
| *** yolanda has quit IRC | 15:32 | |
| *** krtaylor has quit IRC | 15:32 | |
| *** mrodden has quit IRC | 15:32 | |
| *** spawnofbelliott has quit IRC | 15:33 | |
| *** nsaje1 has quit IRC | 15:33 | |
| *** belliott has joined #openstack-infra | 15:35 | |
| *** cthulhup_ has joined #openstack-infra | 15:35 | |
| mordred | hrm | 15:36 |
| *** mrodden has joined #openstack-infra | 15:37 | |
| mordred | so - we do require that the git sha is signed | 15:37 |
| mordred | that doesn't really help with the trail to the tarball | 15:37 |
| fungi | however there's no way to map that to a signature of the tarball, no | 15:37 |
| jeblair | fungi, mordred: i'm running out of ideas of how to figure out what's wrong with jenkins | 15:37 |
| jeblair | fungi, mordred: i could use some more of them. | 15:38 |
| jd__ | who invoked me about MongoDB? | 15:38 |
| BobBall | mordred: Would you expect to ./run_tests in nova's personal venv or in GLOBAL_VENV? | 15:38 |
| *** zehicle has joined #openstack-infra | 15:38 | |
| fungi | jeblair: well, presumably it was a cascade failure because of something which happened when static.o.o filled up | 15:39 |
| fungi | jeblair: otherwise the timing was too coincidental | 15:39 |
| *** nsaje1 has joined #openstack-infra | 15:39 | |
| fungi | jeblair: also it didn't happen right away, since while stuf was still broken i could get the jenkins01 and 02 interfaces to load | 15:40 |
| fungi | and then after jobs started being able to upload artifacts successfully again, 01 crumbled but 02 was still responsive | 15:40 |
| fungi | then once i got 01 running again, 02 went out to lunch | 15:41 |
| jeblair | fungi: i believe that's because it worked as long as there were available web threads that were not waiting on the Jenkins lock | 15:41 |
| jeblair | fungi: as d-g or nodepool continued to use those threads to modify nodes, they were also dedicated to waiting on that lock and were consumed | 15:41 |
| fungi | so connection avalanche/thundering herd | 15:41 |
| jeblair | i don't think so | 15:42 |
| jeblair | i think it happened one at a time, which is why 01 and 02 stopped responding at different times | 15:42 |
| fungi | ahh, so once the thread pool was 100% consumed by threads waiting for locks, there were no threads to process an unlock? | 15:42 |
| jeblair | there were no threads to handle your http request | 15:43 |
| * fungi grasps at straws | 15:43 | |
| jeblair | something is holding a lock on the jenkins object and i can't figure out what | 15:43 |
| * mordred is out of ideas for tracking it down | 15:43 | |
| jeblair | and it's important, because we wrote some of the code that does the scp plugin, and we wrote the code that's doing all this modifying of jenkins nodes | 15:44 |
| *** markmc has quit IRC | 15:44 | |
| jeblair | so no one else is going to fix it for us | 15:44 |
| *** yaguang has joined #openstack-infra | 15:44 | |
| jeblair | and if we don't figure it out, then we are doomed to a life of restarting jvms at odd hours | 15:44 |
| jeblair | which i am not interested in | 15:44 |
| mordred | do you think we could jam a server into this condition again? | 15:44 |
| jeblair | so i'd really like to try to figure this out | 15:44 |
| fungi | well, we've got one in this condition currently | 15:45 |
| mordred | like, start a new one that's not connected to these and hammer it in a particular way until it jams? (perhaps with a full disk to start with) | 15:45 |
| mordred | I say that | 15:45 |
| mordred | because potentially if we start that server with the debug things that would let us do the lock stuff | 15:45 |
| fungi | ahh | 15:45 |
| mordred | then we might be able to see the thing | 15:45 |
| mordred | but that only works if we think we could reproduce | 15:45 |
| jeblair | mordred: i have no idea which of our complex systems are required for this; or all of them. | 15:46 |
| mordred | jeblair: that is a good point | 15:46 |
| mordred | hrm | 15:46 |
| fungi | well, in this case a potentially contributing factor is the higher-than-normal commit volume, which would be hard to duplicate i suspect | 15:47 |
| mordred | jeblair: you were saying earlier that we could read the tracebacks by hand, all 10k lines - that obviously won't work - but perhaps we could script/parse them? | 15:47 |
| jeblair | i have scanned one of the thread dumps to try to find anomalies; what i discovered is that the two dumps that i have (one from jstack, one from jdb) are different | 15:48 |
| mordred | jeblair: any idea what the performance impact of running with the debugging server enabled is? | 15:48 |
| jeblair | mordred: root@jenkins02:/~jdb/{dump,stack} | 15:48 |
| jeblair | mordred: no | 15:48 |
| mordred | jeblair: because we could call this one a miss, restart them with debugging and wait until the next random time it dies (which is not optimium, but either it never dies again, or when it does we have proper debugging) | 15:49 |
| jeblair | mordred: i don't know if that would help | 15:49 |
| *** xchu has quit IRC | 15:49 | |
| jeblair | mordred: we'd want to test that and make sure that actually gives us what we need | 15:50 |
| mordred | yah | 15:50 |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Skip devstack/exercises by default https://review.openstack.org/42082 | 15:50 |
| jd__ | jeblair, mordred, fungi: I was reading backlog about https://review.openstack.org/#/c/39237/ ; we need MongoDB > 2.2 for that, so it'd be good indeed to be able to use 10gen to get it since Ubuntu's lagging | 15:50 |
| jd__ | cc nsaje1: ^ | 15:50 |
| ryanpetrello | is http://status.openstack.org/zuul/ open source somewhere? | 15:51 |
| ryanpetrello | figured it might be a component of https://github.com/openstack-infra/zuul | 15:51 |
| ryanpetrello | but didn't see it in there | 15:51 |
| mordred | jd__: that would be a big departure from our current support model | 15:51 |
| mordred | ryanpetrello: yeah - it's in our puppet repo - one sec | 15:51 |
| mordred | ryanpetrello: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul | 15:52 |
| jd__ | mordred: I can propose to ./configure && make if you prefer, nah ;) I don't see much better solution :/ | 15:52 |
| *** cwj has joined #openstack-infra | 15:52 | |
| jd__ | mordred: unless we can have a devstack on RHEL? like we do for py26 | 15:52 |
| mordred | jd__: RHEL has mongo 2.2 and ubuntu doesn't? | 15:53 |
| jd__ | mordred: how ironic, isn't it? | 15:53 |
| fungi | jd__: what that would mostly require is getting devstack to work on rhel. you would be a hero to many people if you managed to make that work | 15:53 |
| jd__ | mordred: we run our mongodb test only in py26 because of that | 15:53 |
| pleia2 | fungi: when you have a chance... feedback on https://review.openstack.org/#/c/42168/4/modules/cgit/manifests/init.pp real quick would be appreciated (hopefully my last patch for this) | 15:53 |
| cwj | anyone know if it is possible to configure the global git user.email and user.name settings for jenkins using jenkins-job-builder? | 15:53 |
| jd__ | fungi: no no no, I thought devstack worked on RHEL, forget me! :) | 15:54 |
| *** ^d has quit IRC | 15:54 | |
| ryanpetrello | mordred: is there a process for contributing to this? (the zuul status board) | 15:55 |
| *** zul has quit IRC | 15:55 | |
| fungi | ryanpetrello: yes, submit a review to review.openstack.org for it, same as for any openstack or openstack-infra project | 15:55 |
| ryanpetrello | (duh) | 15:56 |
| ryanpetrello | thanks | 15:56 |
| fungi | ryanpetrello: in this case, it would be a patch to openstack-infra/config for the theming bits hosted/proxied through status.openstack.org, or a review to openstack-infra/zuul for the backend status.json interface it serves | 15:56 |
| ryanpetrello | right | 15:57 |
| fungi | pleia2: i'll have a look in a bit | 15:57 |
| *** jpeeler has quit IRC | 15:58 | |
| *** jpeeler has joined #openstack-infra | 15:59 | |
| *** nayward has quit IRC | 16:04 | |
| *** MarkAtwood has joined #openstack-infra | 16:04 | |
| *** ruhe has quit IRC | 16:06 | |
| *** woodspa__ has joined #openstack-infra | 16:08 | |
| *** cthulhup_ has quit IRC | 16:08 | |
| *** sarob has joined #openstack-infra | 16:11 | |
| xBsd | folks, is jenkins and zulu fully functional now? | 16:11 |
| *** fbo is now known as fbo_away | 16:11 | |
| xBsd | s/zulu/zuul/ | 16:12 |
| *** woodspa_ has quit IRC | 16:12 | |
| *** yaguang has quit IRC | 16:12 | |
| xBsd | I see a bunch of frozen check jobs. | 16:12 |
| *** ftcjeff has joined #openstack-infra | 16:14 | |
| *** yolanda has joined #openstack-infra | 16:16 | |
| clarkb | morning | 16:16 |
| clarkb | xBsd: on a phone so cant check directly | 16:17 |
| clarkb | but best guess is there are not enough resources so things are queuing | 16:17 |
| *** dprince has joined #openstack-infra | 16:17 | |
| xBsd | openstack/nova39920,60 min | 16:18 |
| xBsd | openstack/tempest42325,10 min | 16:18 |
| *** krtaylor has joined #openstack-infra | 16:18 | |
| xBsd | means 0 mins | 16:18 |
| xBsd | they freeze in that state for about hour | 16:19 |
| xBsd | fungi: it's the same issue which was with gate jobs | 16:19 |
| mordred | xBsd: we're still working through some things | 16:20 |
| xBsd | mordred: thanks ) ok, wait for solving ) | 16:20 |
| *** ^d has joined #openstack-infra | 16:20 | |
| fungi | xBsd: yeah, we've left it in that state so we could diagnose it in more detail. it's not holding up gating, but there are a handful of changes who aren't returning checks because of jenkins02 (i think, need to look back at the status page to be sure) | 16:21 |
| *** cthulhup_ has joined #openstack-infra | 16:21 | |
| clarkb | fungi mordred this related to static/logs? | 16:23 |
| fungi | clarkb: yup, deadlock in jenkins. i think it's close to figured out now | 16:25 |
| *** nicedice has joined #openstack-infra | 16:25 | |
| mordred | jd__: honestly, then, I think the best path forward is make ceilometer mongo config profile for centos devstack, and run ceilometer in sqlalchemy mode for ubuntu devstack | 16:26 |
| *** dstufft_ has joined #openstack-infra | 16:26 | |
| *** dstufft has quit IRC | 16:26 | |
| *** dstufft_ is now known as dstufft | 16:26 | |
| anteaya | I'm looking forward to helping load test asterisk, I which I believe is scheduled to happen in 34 minutes | 16:27 |
| jd__ | mordred: ok :( we can't test everything with SQLalchemy but that's better than nothing | 16:27 |
| mordred | jd__: we want to get centos devstack in the gate - so consider it a timing issue | 16:27 |
| *** zul has joined #openstack-infra | 16:30 | |
| *** odyssey4me3 has quit IRC | 16:30 | |
| *** datsun180b_ has joined #openstack-infra | 16:34 | |
| jeblair | clarkb, zaro: with fungi and mordred's help, i believe i tracked down the problem to a race where nodepool deleted a node while gearman-plugin was trying to set it offline, and they deadlocked | 16:37 |
| *** afazekas has quit IRC | 16:37 | |
| jeblair | it's also possible that the timing around finishing jobs was different due to the scp issues, which is why this showed up then... | 16:37 |
| clarkb | jeblair: gotcha | 16:37 |
| *** datsun180b has quit IRC | 16:37 | |
| *** datsun180b_ is now known as datsun180b | 16:37 | |
| clarkb | jeblair: scp'ing the console log may happen after the onFinished | 16:37 |
| fungi | jeblair: want to keep it in this state any longer, or shall i restart jenkins on 02 now? | 16:38 |
| clarkb | s/may/will/ because the console log copy is spun off in order to catch the end of the log | 16:38 |
| jeblair | clarkb: is that 'may' a "definitely is possible" or "i think it might be able to happen but we should test"? | 16:38 |
| jeblair | clarkb: ok | 16:38 |
| clarkb | jeblair: it is a "will happen" | 16:38 |
| *** yolanda has quit IRC | 16:38 | |
| jeblair | clarkb: and we have no indication when that's finished, yeah? | 16:38 |
| jeblair | fungi: i think we can restart it now | 16:39 |
| jeblair | fungi: please do | 16:39 |
| fungi | will do | 16:39 |
| clarkb | we do not. We could potentially add something to the scp plugin that notifies us of that | 16:39 |
| clarkb | the big problem is that in order to catch the end of the log you must keep running after the test finishes | 16:39 |
| *** mriedem1 has joined #openstack-infra | 16:40 | |
| *** mriedem has quit IRC | 16:40 | |
| *** ruhe has joined #openstack-infra | 16:40 | |
| clarkb | jeblair: we could possibly update the Run object so that Jenkins knows internally that it is done. (can't just attach a new member object but a field in the env var set may work) but that is a big hack | 16:42 |
| mordred | clarkb, jeblair: what if the scp plugin emitted a zmq event | 16:42 |
| mordred | when _it_ was done | 16:42 |
| clarkb | mordred: it could do something like that too. It would potentially couple the two plugins fairly tightly | 16:43 |
| mordred | other plugins depend on plugins | 16:43 |
| * mordred not sure it's a great idea | 16:43 | |
| *** radix has left #openstack-infra | 16:43 | |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add the ability to filter Zuul Status on multiple (comma-delimited) terms. https://review.openstack.org/42382 | 16:44 |
| mordred | or - is there a way to make the scp plugin consume the zmq plugin IFF the zmq plugin exists? | 16:44 |
| mordred | so like, if the plugin is there, use it, other wise, don't | 16:44 |
| clarkb | mordred: possibly. Not sure what the plugin registry looks like but if it is anything like entry points yes | 16:44 |
| clarkb | the zmq plugin uses a simple thread safe queue, scp plugin could write directly to that | 16:45 |
| clarkb | jenkins internal event listeners should be based on a queue model rather than for loops at places they think should emit events | 16:46 |
| jeblair | i'm not positive about the degree to which this actually affected us... | 16:46 |
| jeblair | it could be a bit of a red herring | 16:46 |
| *** nsaje1 has quit IRC | 16:46 | |
| clarkb | the netaddr thing is still biting us | 16:47 |
| clarkb | if no one else has had a chance to look at that I will try and sort out what is going on there | 16:48 |
| mordred | clarkb: what is the netaddr thing and how? | 16:48 |
| mordred | clarkb: can you point me at the issue? | 16:48 |
| fungi | clarkb: yeah, someone reported another incident around 0400 utc so i reopened the bug | 16:48 |
| mordred | I'm trying to learn more about what's going on with this | 16:48 |
| clarkb | mordred: https://jenkins01.openstack.org/job/gate-nova-docs/970/console | 16:48 |
| clarkb | its almost like we are bypassing our mirror somehow | 16:50 |
| mordred | clarkb: did we stop using PIP_BUILD_CACHE on jenkins slaves at some point? | 16:50 |
| clarkb | mordred: notice the list of candidates there does not match what we have on our mirror | 16:50 |
| jeblair | i'm going to try to get timing info from logs to determine how much of a factor the scp thing is | 16:50 |
| mordred | jeblair: cool | 16:50 |
| fungi | maybe some subset of slaves is missing the overrides to use our mirror? | 16:50 |
| clarkb | fungi: that particular gate-nova-docs job seems to have run the mirror selection script successfully | 16:51 |
| clarkb | mordred: I am not sure | 16:51 |
| mordred | pip conf file is there on tha tmachien | 16:51 |
| fungi | huh. is there any correlation between particular slaves or particular jobs? i guess you're looking via logstash | 16:51 |
| clarkb | git log -p says we never used PIP_BUILD_CACHE in the infra/config repo | 16:51 |
| clarkb | fungi: no I haven't gotten that far | 16:52 |
| clarkb | fungi: will try that now | 16:52 |
| mordred | it's fine - I mean, that would just let us see easier where it was pulling from due to the way it reports to stdout | 16:52 |
| mordred | while I'm in here - who are the packstack people? | 16:53 |
| mordred | is that harlowja ? | 16:53 |
| clarkb | mordred: its redhat | 16:53 |
| clarkb | dprince may know | 16:53 |
| mordred | dprince: packstack is leaving a bunch of files around in ~jenkins | 16:53 |
| clarkb | mordred: it is always nova tests | 16:53 |
| mordred | dprince: of the form packstack-answers-20130711-124138.txt | 16:53 |
| clarkb | over the last 12 hours anyways | 16:53 |
| clarkb | hmm really need to add more info to the events to figure out which slave, master, and project caused a job to be built... | 16:54 |
| *** jpich has quit IRC | 16:54 | |
| jeblair | fungi, mordred, clarkb: wow, there are full thread dumps (and nice ones too) in the jenkins logs. i wonder how that happened. | 16:56 |
| fungi | eek | 16:56 |
| *** mkerrin has quit IRC | 16:56 | |
| fungi | jeblair: perhaps it adds them when killed ungracefully? | 16:56 |
| clarkb | jeblair: it is a miracle | 16:56 |
| clarkb | mordred: precise7 9 and 10 have all done it | 16:56 |
| clarkb | mordred: which means both masters | 16:56 |
| clarkb | 8 too | 16:57 |
| jeblair | fungi, mordred, clarkb: http://paste.openstack.org/show/44339/ | 16:57 |
| clarkb | mordred: so I don't think this is slave or master specific | 16:57 |
| jeblair | that ^ is the thread dump i wanted -- notice how it says which things are locked, and says right there that it's holding a lock on hudson.model.Hudson. | 16:57 |
| mordred | jeblair: has it succeeded on precise7 ? | 16:57 |
| fungi | ha | 16:57 |
| clarkb | mordred: looking | 16:58 |
| jeblair | fungi: there are several thread dumps, at different times | 16:58 |
| mordred | clarkb: oh - you konw what else? | 16:58 |
| *** sdake_ has quit IRC | 16:58 | |
| clarkb | mordred: yes, looks like gate-nova-pep8 has succeeded on precise7 a few times | 16:58 |
| mordred | clarkb: we should look and see if puppet has fixed /home/jenkins/.pip/pip.conf any time | 16:58 |
| *** sdake_ has joined #openstack-infra | 16:58 | |
| clarkb | pep8, docs, python2X are all affected | 16:58 |
| *** sdake_ has quit IRC | 16:59 | |
| *** sdake_ has joined #openstack-infra | 16:59 | |
| mordred | clarkb: because it's possible that some other job is overwriting ~/.pip/pip.conf erroneously | 16:59 |
| *** nati_ueno has joined #openstack-infra | 16:59 | |
| mordred | clarkb: and we should set its perms in puppet to be non writable to prevent that | 16:59 |
| mordred | because it's writable by the jenkins users | 16:59 |
| mordred | clarkb: any way of knowing which jobs ran before the fail on precise7 ? | 17:00 |
| clarkb | mordred: https://jenkins01.openstack.org/computer/precise7/builds | 17:00 |
| mordred | oh - wait | 17:00 |
| mordred | modules/jenkins/files/slave_scripts/select-mirror.sh | 17:00 |
| clarkb | mordred: several savanna jobs | 17:00 |
| mordred | duh. we don't set that in puppet | 17:00 |
| clarkb | mordred: and yes we do it dynamically as part of the job itself | 17:01 |
| dprince | clarkb/mordred: I believe packstack creates an answers file each time it runs. | 17:01 |
| dprince | clarkb/mordred: So that you can re-run it with the same settings as before... | 17:01 |
| dprince | clarkb/mordred: so it sounds like just a cleanup issue. | 17:01 |
| mordred | dprince: yeah. not a big deal | 17:01 |
| mordred | I just happened to see it | 17:01 |
| anteaya | is anyone in the asterisk conference room #6000? | 17:02 |
| *** Ryan_Lane has quit IRC | 17:02 | |
| clarkb | anteaya: no, thank you for reminding me | 17:02 |
| *** yolanda has joined #openstack-infra | 17:03 | |
| anteaya | clarkb: welcome | 17:04 |
| anteaya | I have dialed into the room #6000 twice now and the call drops | 17:04 |
| *** derekh has quit IRC | 17:04 | |
| clarkb | anteaya: just happened to me too | 17:04 |
| anteaya | I am coming in with Skype using the PSTN number | 17:04 |
| anteaya | okay | 17:04 |
| clarkb | I am hitting the PSTN from verizon | 17:05 |
| anteaya | rustlebee: ping | 17:05 |
| pleia2 | drops for me too | 17:05 |
| jeblair | me too, using sip from my asterisk box | 17:05 |
| anteaya | is the asterisk server up? | 17:05 |
| jeblair | anteaya: you get the prompt, right? | 17:05 |
| anteaya | are we doing the asterisk load testing, rustlebee ? | 17:05 |
| anteaya | jeblair: yes, I get the prompt | 17:05 |
| jeblair | anteaya: "please enter the conference number.." | 17:06 |
| anteaya | I enter 6000# | 17:06 |
| anteaya | yes | 17:06 |
| anteaya | and then the call drops | 17:06 |
| jeblair | did we merge the change that fixes the conf bridge? | 17:06 |
| *** krtaylor has quit IRC | 17:06 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Update style checking to hacking https://review.openstack.org/42357 | 17:08 |
| clarkb | the first occurence of netaddr issues that I see is from 2013-08-15 15:45:00 in a gate-nova-python26 job | 17:09 |
| clarkb | and the occurrences seem clustered together | 17:10 |
| mordred | clarkb: I agree that it looks lke the job is not running with our mirror | 17:10 |
| jeblair | mordred: i don't want to use hacking for nodepool. | 17:10 |
| mordred | jeblair: ok. | 17:10 |
| jeblair | mordred: also, myjenkins should not get style changes | 17:10 |
| jeblair | mordred: it is a set of methods that need to be upstreamed into the jenkins module | 17:10 |
| jeblair | and should follow its conventions | 17:10 |
| *** cthulhup_ has quit IRC | 17:10 | |
| mordred | jeblair: great! can I at least add ignore H into the tox.ini then? | 17:10 |
| anteaya | looks like rustlebee is afk and I don't see pablanger in channel | 17:11 |
| jeblair | mordred: zuul depends on hacking and then ignores H*; i figured why not cut out the middleman. | 17:11 |
| mordred | well, I have hacking installed globally on my system, so it gets really noisy for me for things that don't at least ignore H | 17:11 |
| jeblair | mordred: ok, so add H to tox.ini but don't depend on hacking | 17:11 |
| mordred | k | 17:11 |
| mordred | jeblair: do you have opinions on python 3 support? | 17:11 |
| jeblair | heh, that was meant to be a question? | 17:11 |
| jeblair | gah | 17:11 |
| mordred | :) | 17:11 |
| mordred | because that was the real reason I touched it at all - were the py3 exception and print things | 17:12 |
| jeblair | mordred: i'm not tying right. i meant to ask you "you want to add H to tox.ini but not depend on hacking?" | 17:12 |
| jeblair | mordred: that seems harmless to me, and i'm cool with that | 17:12 |
| mordred | cool | 17:12 |
| harlowja | mordred i'm not packstack | 17:12 |
| harlowja | i'm just anvil :-P | 17:12 |
| jeblair | mordred: i like python3 support | 17:12 |
| mordred | harlowja: k | 17:12 |
| jeblair | anteaya: i'm looking at the pbx host now | 17:13 |
| mordred | jeblair: because we could also depend on hacking but ignore the things that are not checks for python things if that's interesting to you | 17:13 |
| anteaya | jeblair: k | 17:13 |
| Alex_Gaynor | which repository controls which IRC channels a review on a repo goes to? | 17:14 |
| mordred | Alex_Gaynor: it's in openstack-infra/config | 17:14 |
| Alex_Gaynor | mordred: thanks | 17:14 |
| vipul | hi all, question about oslo.cfg release - you guys know when 1.2 goes to pypi? | 17:14 |
| mordred | Alex_Gaynor: look for gerritbot | 17:14 |
| jeblair | mordred: i was sort of thinking that reducing dependency count would be nice. plus, at this point, we have py3 runners | 17:14 |
| mordred | vipul: yup. when we release havana | 17:14 |
| mordred | jeblair: k. I'm fine with that | 17:14 |
| clarkb | mordred: really? thats annoying | 17:15 |
| vipul | mordred: thanks.. what do projects that want to use it in the current cycle do (if there are projects like that) | 17:15 |
| clarkb | 1.1 is not python3 friendly so all of those jobs I added to test py3k with the clients bomb out on oslo.config install | 17:15 |
| mordred | vipul: you can use it directly from tarballs.o.o | 17:16 |
| mordred | vipul: look at nova's requirements.txt | 17:16 |
| jeblair | Alex_Gaynor: E123 and E125 contradict the pep8 specification | 17:16 |
| vipul | mordred: will that totally screw up devstack based tests? | 17:16 |
| Alex_Gaynor | jeblair: I understand, and agree, I can just never remember what they are :) | 17:16 |
| Alex_Gaynor | mordred: does http://bpaste.net/show/123617/ look about right? | 17:16 |
| mordred | Alex_Gaynor: yup | 17:16 |
| mordred | vipul: nope | 17:16 |
| jeblair | Alex_Gaynor: oh, ok, i misparsed that comment then. :) | 17:16 |
| Alex_Gaynor | mordred: okey doke | 17:16 |
| Alex_Gaynor | jeblair: ah sorry if I was unclear | 17:17 |
| clarkb | it is really interesting that only nova is affected by the netaddr thing | 17:17 |
| mordred | vipul: we actually install trunk oslo.config in devstack | 17:17 |
| clarkb | does any other project depend on netaddr? | 17:17 |
| vipul | oh nice | 17:17 |
| mordred | clarkb: yes | 17:17 |
| mordred | clarkb: swift does | 17:17 |
| fungi | clarkb: combination of new pip and site-packages? | 17:17 |
| clarkb | fungi: oh | 17:17 |
| clarkb | damnit site packages | 17:17 |
| clarkb | netaddr==0.7.5 is installed globally on precise7 | 17:18 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Notify the marconi IRC channel on gerrit events https://review.openstack.org/42386 | 17:18 |
| * fungi is trying to think of things which changed in the last couple days and nova special snowflakes | 17:18 | |
| fungi | clarkb: or maybe new tox in the past couple days too | 17:18 |
| clarkb | fungi: yeah possible some combo of site packages being too old forcing the --upgrade to do work in the new tox | 17:18 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Ignore OpenStack hacking rules https://review.openstack.org/42357 | 17:18 |
| clarkb | mordred: did we break the -U possibly? | 17:18 |
| jeblair | anteaya: it's up now | 17:19 |
| mordred | aaaaahhhhhhhhhh | 17:19 |
| jeblair | rustlebee: I restarted asterisk and that caused it to pick up a working config with the confbridge | 17:19 |
| fungi | clarkb: apt says Version: 0.7.5-4build2 is installed on our precise slaves | 17:19 |
| *** jerryz has joined #openstack-infra | 17:19 | |
| pleia2 | woo, no drop | 17:20 |
| *** psedlak has quit IRC | 17:20 | |
| anteaya | jeblair: k, I'm onto something else and will try again once done | 17:20 |
| mordred | the nova require is just netaddr | 17:20 |
| mordred | which means it will always try to upgrade | 17:20 |
| mordred | but that should be fine | 17:20 |
| mordred | the question is - why is it not using .pip/pip.conf | 17:20 |
| *** dina_belova has quit IRC | 17:21 | |
| clarkb | mordred: what if it isn't upgrading | 17:21 |
| clarkb | mordred: and since site packages is enabled we get stuck with the old verison and it bombs out | 17:21 |
| *** SergeyLukjanov has quit IRC | 17:22 | |
| mordred | no - it's loking for candidate versions | 17:22 |
| mordred | and finding ones not on our mirror | 17:22 |
| clarkb | fungi: jog my memory on how to properly run a command across all of our slaves with slat? | 17:22 |
| clarkb | *salt. is it `salt \* test.ping` ? | 17:22 |
| fungi | i'll have to jog my memory | 17:23 |
| clarkb | jesusaurus: ^ | 17:23 |
| fungi | clarkb: looking back at my .bash_history the most recent thing i did with salt was... | 17:23 |
| fungi | sudo salt -E '.*' cmd.run 'dpkg -l *jdk* 2>/dev/null|grep ^i||rpm -qa 2>/dev/null|grep -i jdk||echo NO JDK' >paste | 17:24 |
| fungi | also... | 17:24 |
| fungi | sudo salt '*' test.ping|grep slave|sort|view - | 17:24 |
| jesusaurus | clarkb: `salt \* test.ping` will test all hosts for a connection | 17:25 |
| fungi | however i think salt gets itself into a bad state, and nobody's had time to troubleshoot it | 17:25 |
| clarkb | fungi: ya, it doesn't seem to be working | 17:25 |
| fungi | right now it's timing out for me | 17:25 |
| jesusaurus | `salt -v \* test.ping` will tell you which hosts dont return within $timeout | 17:26 |
| mordred | jesusaurus: only if salt hasn't crapped itself | 17:26 |
| jesusaurus | oh, is it the master thats flailing? | 17:27 |
| mordred | jesusaurus: our salt minions decide to stop talking to the master at some point | 17:27 |
| mordred | but since we dont' really use salt for anything, we don't notice when it happen | 17:27 |
| mordred | happens | 17:27 |
| jeblair | so there are 6 of us on the conference bridge; load average is 0 | 17:27 |
| mordred | only when we try to do something like look at a file on all of the hosts | 17:27 |
| * mordred dials in | 17:27 | |
| jeblair | (well, i'm on there 3 times) | 17:27 |
| mordred | sorry - was destracted by nova gate thing | 17:27 |
| * anteaya is still on hold on another call | 17:28 | |
| *** jerryz has quit IRC | 17:30 | |
| *** jfriedly has joined #openstack-infra | 17:31 | |
| fungi | is it supposed to be 6000? every time i try i get asked again to please enter my conference number followed by the pound key | 17:31 |
| jeblair | fungi: yeah | 17:31 |
| *** sarob has quit IRC | 17:31 | |
| anteaya | fungi how are you dialing in? | 17:32 |
| *** sarob has joined #openstack-infra | 17:32 | |
| fungi | anteaya: pots | 17:32 |
| anteaya | I don't know what that is | 17:32 |
| fungi | hung up and trying again | 17:32 |
| *** nayward has joined #openstack-infra | 17:33 | |
| fungi | pots==plain old telephone system | 17:33 |
| mordred | https://twitter.com/e_monty/status/367393004224409601/photo/1 | 17:33 |
| clarkb | we appear to have tox 1.6.0 just about everywhere | 17:34 |
| clarkb | so I think this is quite possibly the problem | 17:34 |
| clarkb | now to try and replicate locally | 17:34 |
| mordred | clarkb: wait - what is the problem? tox 1.6.0 ? | 17:34 |
| fungi | hung up and dialled in several times, tried multiple times each time to join the conference but the autoattendant pretends that conference doesn't exist | 17:34 |
| clarkb | mordred: we think that is a possibility | 17:35 |
| mordred | o m g | 17:35 |
| clarkb | mordred: because it is new as of not long ago | 17:35 |
| jeblair | fungi: can you try again? | 17:35 |
| fungi | rustlebee: if it helps, i'll be coming from a line with cid ending in 4344 | 17:35 |
| * fungi tries again | 17:35 | |
| *** cthulhup has joined #openstack-infra | 17:36 | |
| *** sarob has quit IRC | 17:36 | |
| *** thomasbiege has quit IRC | 17:36 | |
| *** sarob has joined #openstack-infra | 17:37 | |
| *** melwitt has joined #openstack-infra | 17:37 | |
| *** yolanda has quit IRC | 17:37 | |
| anteaya | fungi ah okay, I was picturing a sturdier version of dixie cups and a string | 17:38 |
| *** UtahDave has joined #openstack-infra | 17:38 | |
| *** sarob has quit IRC | 17:38 | |
| *** sarob has joined #openstack-infra | 17:39 | |
| *** jerryz has joined #openstack-infra | 17:39 | |
| anteaya | mordred: your airplane is beautiful | 17:40 |
| marun | mordred: poing | 17:41 |
| *** morganfainberg|a is now known as morganfainberg | 17:41 | |
| clarkb | mordred: venv installdeps: -U, -r/home/boylancl/tmp/test-tox/nova/requirements.txt, -r/home/boylancl/tmp/test-tox/nova/test-requirements.txt | 17:41 |
| *** yolanda has joined #openstack-infra | 17:41 | |
| *** burt has quit IRC | 17:41 | |
| *** afazekas has joined #openstack-infra | 17:41 | |
| morganfainberg | mordred: pong | 17:42 |
| morganfainberg | mordred: whoopse wrong channel. | 17:42 |
| rustlebee | jeblair: really sorry i wasn't around ... :( | 17:43 |
| jeblair | rustlebee: no prob | 17:43 |
| *** sarob has quit IRC | 17:43 | |
| jeblair | rustlebee: several of us are on the call now | 17:43 |
| rustlebee | ok | 17:43 |
| jeblair | 8 (3 of which are me) | 17:43 |
| rustlebee | k | 17:44 |
| jeblair | all on pstn except for one of mine, which is sip | 17:44 |
| rustlebee | i'll spin up an asterisk server real quick | 17:44 |
| jeblair | rustlebee: the consenses seems to be that we're all hearing a bit of choppyness | 17:45 |
| jeblair | also, this is exciting | 17:45 |
| jeblair | one of my channels gas gone silent | 17:45 |
| *** dkehn_ has joined #openstack-infra | 17:45 | |
| *** dkehn has quit IRC | 17:45 | |
| jeblair | but the other 2 are working | 17:45 |
| rustlebee | if there are gaps in getting CPU time, it would cause that | 17:45 |
| rustlebee | should be roughly a 80 Kbps bidirectional stream per call | 17:46 |
| rustlebee | UDP streams on random ports | 17:46 |
| *** sarob has joined #openstack-infra | 17:49 | |
| Alex_Gaynor | Is the goal of nodepool to be more elastic with the number of CI workers we have? (And is there another place I should be listening to know things like this) | 17:51 |
| *** ruhe has quit IRC | 17:52 | |
| clarkb | Alex_Gaynor: I believe its first goal is to address the issues the old pool management had where it would get out of sync with jenkins | 17:52 |
| SlickNik | jeblair / clarkb: Do you know whom I can contact if there is a mistake with the details of one of the Summit Sessions I proposed? | 17:52 |
| clarkb | because it relied on jenkins jobs which sometimes don't work as expected | 17:52 |
| Alex_Gaynor | Ah | 17:52 |
| clarkb | Alex_Gaynor: by having a long running daemon you can deal with problems more flexibly | 17:52 |
| clarkb | SlickNik: I would check with reed | 17:52 |
| anteaya | anybody still in the asterisk conference call? | 17:53 |
| anteaya | or is that over? | 17:53 |
| pleia2 | anteaya: yep | 17:53 |
| pleia2 | we're on it | 17:53 |
| anteaya | okay | 17:53 |
| SlickNik | clarkb: does he usually hang out in #openstack-infra? Where can I find him? | 17:54 |
| clarkb | ya he is often in here | 17:54 |
| anteaya | yay, I'm in | 17:54 |
| SlickNik | Cool, I'll ping him when he shows up. | 17:55 |
| SlickNik | Thanks! | 17:55 |
| *** cthulhup has quit IRC | 17:56 | |
| anteaya | I just got dropped | 17:58 |
| anteaya | ringing, ringing, ringing | 17:59 |
| anteaya | not connecting | 17:59 |
| *** morganfainberg is now known as morganfainberg|a | 17:59 | |
| anteaya | trying again | 18:00 |
| anteaya | ringing, ringing, ringing | 18:00 |
| anteaya | no connection | 18:00 |
| anteaya | trying again | 18:01 |
| mordred | Alex_Gaynor: yeah - it's mainly currently a refactor of the current CI pool code stuff | 18:01 |
| anteaya | nope | 18:01 |
| anteaya | I can't seem to connect again | 18:01 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add the ability to perist the Zuul Status filter with a cookie. https://review.openstack.org/42393 | 18:02 |
| jeblair | anteaya: :( | 18:02 |
| jeblair | anteaya: i just tried over the pstn and connected | 18:02 |
| anteaya | :( | 18:02 |
| anteaya | okay I will try again | 18:02 |
| *** dkehn_ has quit IRC | 18:02 | |
| *** dkehn has joined #openstack-infra | 18:03 | |
| fungi | for those playing along, pbx system graphs at http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=39 | 18:03 |
| anteaya | seems my issue might be related to my skype connection | 18:03 |
| anteaya | sigh | 18:03 |
| clarkb | mordred: according to the buid log for the job before gate-nova-docs 970 and job after it we do not have overlapping tests | 18:05 |
| mordred | clarkb: what if we do like an inotify type thing to watch those files and log what hapens to them? | 18:05 |
| clarkb | we do have a ton of salt minion processes | 18:06 |
| clarkb | ... | 18:06 |
| clarkb | mordred: ya we can try that | 18:06 |
| clarkb | that seems very heavy weight for debugging this though | 18:07 |
| *** pentameter has joined #openstack-infra | 18:08 | |
| mordred | yeah | 18:09 |
| mordred | it does | 18:09 |
| clarkb | /home/jenkins/workspace/gate-nova-python27/.tox/py27/bin/python2.7 ../bin/pip install --pre -U -r/home/jenkins/workspace/gate-nova-python27/requirements.txt -r/home/jenkins/workspace/gate-nova-python27/test-requirements.txt is the command being run to install the deps | 18:10 |
| clarkb | what is --pre? | 18:11 |
| mordred | --pre Include pre-release and development versions. By | 18:11 |
| mordred | default, pip only finds stable versions. | 18:11 |
| mordred | why is that in there | 18:11 |
| anteaya | am I connected? | 18:11 |
| clarkb | from tox | 18:11 |
| anteaya | I installed skype on this laptop (my newest one) and the sound settings are less that ideal | 18:12 |
| clarkb | mordred: they did close my tox PR to make the -U stuff more formal saying there is some other way of doing it now | 18:12 |
| anteaya | I might be connected though | 18:12 |
| anteaya | I can't hear anything though, which might be my sound settings | 18:12 |
| *** _TheDodd_ has quit IRC | 18:12 | |
| mordred | install non-stable releases. (tox defaults to install with “–pre” everywhere). | 18:13 |
| *** jergerber has quit IRC | 18:13 | |
| mordred | ok. we're going to have to change that | 18:13 |
| mordred | clarkb: add new EXPERIMENTAL “install_command” testenv-option to configure the installation | 18:13 |
| mordred | so, I believe the default installer_command is 'pip --pre' | 18:14 |
| clarkb | could --pre behavior be causing this? | 18:14 |
| clarkb | (I don't expect it to but maybe it tries to be smart and looks for pre packages where it shouldn't?) | 18:14 |
| *** morganfainberg|a is now known as morganfainberg | 18:15 | |
| anteaya | okay, I don't have sound working properly on this laptop so I can't assess my asterisk experience | 18:15 |
| *** dmakogon_ has joined #openstack-infra | 18:18 | |
| dmakogon_ | hey, guys, we seeing bug in devstack, this bug failing gerrit build of Trove dashboard. | 18:19 |
| dmakogon_ | log: http://logs.openstack.org/28/42228/2/check/gate-tempest-devstack-vm-cells-full/12d70e4/console.html | 18:19 |
| mordred | clarkb: https://review.openstack.org/#/c/42178/ | 18:19 |
| mordred | clarkb: a | 18:19 |
| mordred | clarkb: B | 18:19 |
| mordred | clarkb: it IS TOX | 18:19 |
| mordred | during installation of packages HOME is now set to a pseudo | 18:20 |
| mordred | location (envtmpdir/pseudo-home). Also, if an index url was | 18:20 |
| mordred | specified a .pydistutils.cfg file will be written so that index_url | 18:20 |
| mordred | is set if a package contains a ``setup_requires``. | 18:20 |
| dmakogon_ | log of devstack bug - http://logs.openstack.org/28/42228/2/check/gate-tempest-devstack-vm-cells-full/12d70e4/console.html | 18:21 |
| *** dina_belova has joined #openstack-infra | 18:21 | |
| mordred | dmakogon_: hey! do you think it's a bug in how devstack is being run? or devstack itself? if it's devstack itself, #openstack-qa is where dtroyer usually hangs out | 18:21 |
| anteaya | okay back in | 18:22 |
| mordred | clarkb: we can apparently set TOX_INDEX_URL | 18:23 |
| *** sarob has quit IRC | 18:23 | |
| mordred | clarkb: which will set it for pip.conf and .pydistutils.cfg for us | 18:23 |
| *** sarob has joined #openstack-infra | 18:23 | |
| *** dina_belova has quit IRC | 18:25 | |
| *** sarob has quit IRC | 18:28 | |
| marun | mordred: i tried to cc you but my mail client seems to have screwed it up. please see email on os-dev with subject 'Gate breakage process' | 18:30 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Fix join/leave sounds in the conference https://review.openstack.org/42396 | 18:30 |
| clarkb | mordred: o_O catching up. I stepped away to get something to drink | 18:30 |
| mordred | https://bitbucket.org/hpk42/tox/issue/116/new-pypi-override-breaks-people-who | 18:30 |
| mordred | clarkb: ^^ | 18:31 |
| mordred | clarkb: I've got a not-perfect workaround for now | 18:31 |
| *** dina_belova has joined #openstack-infra | 18:31 | |
| clarkb | mordred: can we tell tox to look at the files we write? | 18:32 |
| mordred | no. not right now | 18:32 |
| mordred | that's the bug I just filed | 18:32 |
| mordred | clarkb: can you pass arguments to scripts you source? | 18:33 |
| clarkb | help source says yes | 18:34 |
| clarkb | source: source filename [arguments] | 18:34 |
| clarkb | dmakogon_: the cells jobs are expected to fail. Those tests run non voting until all of the issues get sorted out | 18:34 |
| clarkb | dmakogon_: so failures of that job shouldn't affect your ability to merge code. Then when those tests are reliable we will make them vote | 18:34 |
| Alex_Gaynor | Is there a plan to get more test workers? | 18:35 |
| jeblair | Alex_Gaynor: yes, we're close to being able to | 18:35 |
| Alex_Gaynor | awesome | 18:35 |
| jeblair | Alex_Gaynor: that's what i've been working on for a couple weeks | 18:35 |
| *** MarkAtwood has quit IRC | 18:35 | |
| *** sdake_ has quit IRC | 18:35 | |
| *** dina_belova has quit IRC | 18:36 | |
| fungi | udp: round-trip min/avg/max = 52.3/78.6/297.7 ms | 18:36 |
| fungi | icmp: rtt min/avg/max/mdev = 54.562/55.293/57.363/0.851 ms | 18:36 |
| fungi | from "random broadband provider in nc" | 18:37 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Workaround tox 1.6 pypi workaround https://review.openstack.org/42397 | 18:37 |
| clarkb | fungi: are you just using udp ping to get the udp numbers? | 18:37 |
| fungi | clarkb: yeah, hping3 --udp | 18:37 |
| mordred | clarkb:, jeblair, fungi ^^ | 18:37 |
| mordred | that should address the netaddr problem | 18:38 |
| clarkb | mordred: can you note the bug in your commit message? I am lookin for the number now | 18:38 |
| fungi | mordred: you mean as long as we don't end up needing to workaround the workaround for the workaround that is | 18:38 |
| mordred | fungi: yup | 18:38 |
| mordred | fungi: so thrilled | 18:38 |
| clarkb | mordred: 1212751 | 18:39 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Workaround tox 1.6 pypi workaround https://review.openstack.org/42397 | 18:39 |
| * fungi relocates to $lab with real computers and fan noise... brb | 18:39 | |
| mordred | jeblair: you're going to be so thrilled about the details on that one | 18:39 |
| clarkb | mordred: does the tox thing only affect the pydistutilcsconfigblahmumblemumble file? and .pip/pip.conf is ok? | 18:40 |
| *** SergeyLukjanov has joined #openstack-infra | 18:40 | |
| mordred | yes | 18:40 |
| mordred | well | 18:40 |
| clarkb | mordred: seems like netaddr is being installed by pip though | 18:40 |
| mordred | I mean | 18:40 |
| mordred | yes. it should be | 18:40 |
| mordred | nono | 18:40 |
| mordred | sorry | 18:40 |
| mordred | it's a hack to fix .pydistutils.cfg | 18:40 |
| mordred | which causes the pip commands to be run in a fake hom edir | 18:40 |
| mordred | so anything you have in ~ isn't going to work right | 18:40 |
| mordred | during the pip install command | 18:40 |
| *** nubbie has quit IRC | 18:41 | |
| anteaya | fungi mmmm fan noise | 18:41 |
| mordred | but only while it's installing your requirements | 18:41 |
| jeblair | mordred: it didn't take long for pypi breakage to show up once we started accidentally using it, did it? | 18:42 |
| fungi | quite amusing, that | 18:42 |
| clarkb | jeblair: nope, I have a timestamp in sb | 18:42 |
| clarkb | jeblair: but it was about 8:45am PDT yesterday | 18:43 |
| jeblair | it's nice to have a 'yep, still need to do this' check every now and then. | 18:43 |
| clarkb | and new tox was released around then iirc | 18:43 |
| jeblair | wow | 18:43 |
| * jeblair goes back to thinking about jenkins | 18:43 | |
| clarkb | less than 24 hours at least | 18:44 |
| bodepd | mordred: you guys must be slow this week ;) | 18:44 |
| bodepd | mordred: if you're getting around to puppet refactor patches | 18:44 |
| mordred | bodepd: I have started a new thing | 18:44 |
| bodepd | mordred: what new thing? | 18:44 |
| mordred | bodepd: I'm walking my whole review queue when I wake up before I write patches | 18:45 |
| clarkb | mordred: your workaround lgtm. I will let you decide if you want to address jeblair's comment before approving | 18:45 |
| jeblair | if you do, don't make me vote again. :) | 18:45 |
| clarkb | food is here /me does that | 18:45 |
| bodepd | mordred: I try to do code review from 10-12 every day | 18:45 |
| bodepd | mordred: and not outside of that window ;) | 18:45 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Workaround tox 1.6 pypi workaround https://review.openstack.org/42397 | 18:45 |
| mordred | clarkb: ^^ - feel free to +2+A that | 18:46 |
| bodepd | mordred: what is tuskar? I noticed it started off as ironic | 18:46 |
| bodepd | mordred: and became something else. | 18:46 |
| bodepd | mordred: is that the top level for triple-o? | 18:46 |
| mordred | bodepd: it's a thing the guys at redhat wrote to drive tripleo deployments | 18:46 |
| bodepd | mordred: I'm just asking you b/c I looked through the commit history and say your name | 18:46 |
| mordred | bodepd: lifeless is working through it with them to see how well it fits in with things and whatnot | 18:47 |
| bodepd | mordred: it's really close to what I am working on atm... | 18:47 |
| bodepd | mordred: probably no big surprise :) | 18:47 |
| mordred | bodepd: then you should talk to both them and to lifeless... | 18:47 |
| mordred | bodepd: you know - for the reasons :) | 18:48 |
| * mordred afks for a bit | 18:48 | |
| bodepd | mordred: I had a conversation with lifeless before and his response about triple-o is that is was intended to kill my work | 18:48 |
| bodepd | mordred: I'm not really sure where a conversation goes from there ;) | 18:48 |
| fungi | 90% chance no matter what lifeless says, he's at least somewhat trolling you | 18:49 |
| bodepd | fungi: that is helpful to know :) | 18:49 |
| clarkb | mordred: done | 18:50 |
| afazekas | http://logs.openstack.org/28/41928/4/check/gate-grenade-devstack-vm/e641bfc/console.html.gz how to solve this ? | 18:50 |
| *** wenlock has joined #openstack-infra | 18:51 | |
| fungi | bodepd: my take on tripleo, at least from a somewhat outside perspective, is that it should make your configuration management way easier, by abstracting away most of the nastiness and allowing you to just worry about parameters | 18:51 |
| *** sarob has joined #openstack-infra | 18:51 | |
| bodepd | fungi: we're starting to evaluate it piece-mill | 18:51 |
| bodepd | fungi: right now, we're looking at Heat vs. vagrant for deploying our CI tests against an openstack cluster | 18:52 |
| openstackgerrit | A change was merged to openstack-infra/config: Workaround tox 1.6 pypi workaround https://review.openstack.org/42397 | 18:52 |
| bodepd | fungi: out main issue is that we have a team focusing on how, and one focusing on 3 months from now, but triple-o feels like 6+months from now | 18:53 |
| fungi | it seems to me that heat intends to take care of the "how" and let you worry about the "what" at a higher level | 18:53 |
| wenlock | i was going to be trying to understand heat next week… as well. | 18:53 |
| *** openstackstatus has quit IRC | 18:53 | |
| wenlock | I think i have a good understanding of vagrant | 18:53 |
| fungi | but yes, chances are in the near-term tripleo/heat work and classic configuration management automation are parallel projects with a lot of overlap | 18:54 |
| fungi | so if there are ways for you to steer in a common direction, maybe the future situation will be a less frustrating prospect | 18:55 |
| bodepd | fungi: we're evaluating :) | 18:56 |
| wenlock | is there an infra project that contains work on heat? | 18:56 |
| *** dmakogon_ has left #openstack-infra | 18:56 | |
| bodepd | wenlock: no, I'm off topic :) sorry | 18:56 |
| clarkb | afazekas: I am not sure. At first glance it looks like a legit failure though. The indication is that nova failed to upgrade | 18:57 |
| clarkb | hmm maybe not. set_up_bash_completion command not found | 18:58 |
| fungi | wenlock: we were mostly held up from doing anything with it infra-wise because our donated providers didn't have heat available in production and heat couldn't run stand-alone. now it can, which opens up the possibility in places but we'd need to refactor the things where it fits what we're doing to take advantage of it | 18:58 |
| clarkb | afazekas: looks like your change is adding that to devstack? | 18:59 |
| clarkb | I wonder if it is trying to use the old devstack functions against newer devstack? my knowledge of what grenade actually does under the covers is pretty basic | 18:59 |
| *** thomasbiege has joined #openstack-infra | 19:00 | |
| *** ^d has quit IRC | 19:00 | |
| afazekas | clarkb: may be | 19:00 |
| clarkb | afazekas: but it is breaking on the new line here https://review.openstack.org/#/c/41928/4/lib/neutron | 19:01 |
| afazekas | https://github.com/openstack-dev/grenade/blob/master/functions or it uses this | 19:01 |
| *** ^d has joined #openstack-infra | 19:01 | |
| *** ^d has joined #openstack-infra | 19:01 | |
| *** dina_belova has joined #openstack-infra | 19:01 | |
| clarkb | afazekas: looks like there are function updates from devstack in the grenade history. I think that may be it | 19:01 |
| clarkb | afazekas: so you will probably need to add the function to grenade first then to devstack | 19:01 |
| afazekas | clarkb: A six line simple change will be a little more :) | 19:02 |
| *** thomasbiege has quit IRC | 19:03 | |
| *** ^d has quit IRC | 19:05 | |
| *** yolanda has quit IRC | 19:06 | |
| *** cthulhup has joined #openstack-infra | 19:07 | |
| afazekas | clarkb: looks like it just sources the old functions, am I need to split the patch to two part ? | 19:08 |
| *** mgagne has quit IRC | 19:09 | |
| clarkb | oh ya. define the function then use it | 19:11 |
| *** cthulhup has quit IRC | 19:12 | |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add the ability to perist the Zuul Status filter with a cookie. https://review.openstack.org/42393 | 19:12 |
| *** sandywalsh has quit IRC | 19:14 | |
| *** mriedem1 has quit IRC | 19:22 | |
| *** vipul is now known as vipul-away | 19:24 | |
| *** dprince has quit IRC | 19:25 | |
| *** sandywalsh has joined #openstack-infra | 19:26 | |
| *** vipul-away is now known as vipul | 19:26 | |
| * fungi needs to run some pre-travel errands and get dinner, but will bbl | 19:28 | |
| *** colinmcnamara has quit IRC | 19:28 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add more details to git server documentation https://review.openstack.org/42405 | 19:30 |
| jeblair | mordred, fungi, clarkb, zaro: what if we didn't wait for the thread to join? | 19:33 |
| mordred | jeblair: say more words | 19:33 |
| jeblair | in the stop method in abstractworkerthread | 19:33 |
| mordred | looking | 19:33 |
| mordred | hrm | 19:34 |
| jeblair | from the looks of it, nothing happens after that | 19:34 |
| jeblair | either there or in gearmanproxy | 19:35 |
| mordred | jeblair: I will not pretend to fully understand the implications - but yes, it doesn't seem to be strictly necessary | 19:35 |
| mordred | or even useful | 19:35 |
| jeblair | if it were written a little differently, there might be a pattern where it would prevent a new worker from being started on a node, but i don't think that's the case | 19:35 |
| jeblair | partly because we're calling stop outside of the lock on the worker list | 19:36 |
| clarkb | worker is the thing running in thread? | 19:36 |
| clarkb | looks likt it is a MyGearmanWorkerImpl so it is the thing talking togearman within this thead | 19:37 |
| jeblair | sort of. the proxy has-a list of ExecutorWorkerThreads, each EWT has-a GearmanWorkerImpl; the GWI is the EWT.thread | 19:37 |
| jeblair | clarkb: yeah | 19:37 |
| clarkb | I think it is safe to remove the join. | 19:38 |
| jeblair | so basically, i think the upshot is that the join call does nothing other than just cause the calling thread to wait until its done, and print nice log messages | 19:38 |
| clarkb | at that point either it goes away or it doesn't and we don't really handle either case | 19:38 |
| jeblair | the calling method waiting till it's done of course was the problem today. | 19:38 |
| jeblair | clarkb: yep | 19:39 |
| jeblair | it is likely to increase concurrency of the stopAll call. | 19:39 |
| jeblair | in that stopAll will probably now stop all of the threads at once instead of in rapid succession. | 19:39 |
| *** mriedem has joined #openstack-infra | 19:39 | |
| jeblair | clarkb: can you review https://review.openstack.org/#/c/42226 | 19:41 |
| jeblair | and i'll base my change on that | 19:41 |
| clarkb | yup | 19:42 |
| * mordred agrees with teh scrollback | 19:43 | |
| clarkb | jeblair: done | 19:44 |
| mordred | clarkb: https://jenkins01.openstack.org/job/gate-nova-python26/1003/console | 19:44 |
| mordred | clarkb: worked | 19:44 |
| mordred | clarkb: it is appropriately setting and using the mirror | 19:45 |
| jeblair | mordred: yay! | 19:46 |
| mordred | jeblair: sigh :) | 19:46 |
| *** vipul is now known as vipul-away | 19:46 | |
| mordred | clarkb: also, can you look at this https://review.openstack.org/#/c/42337/ and tell me what I'm doing wrong on py26 and py33 | 19:46 |
| *** zul has quit IRC | 19:47 | |
| clarkb | mordred: ya I will take a look | 19:47 |
| *** shardy is now known as shardy_afk | 19:50 | |
| *** psedlak has joined #openstack-infra | 19:52 | |
| *** ^d has joined #openstack-infra | 19:53 | |
| *** weshay has quit IRC | 19:54 | |
| *** MarkAtwood has joined #openstack-infra | 19:57 | |
| *** vipul-away is now known as vipul | 19:57 | |
| xBsd | folks, could I start recheck job while the error job is not finished? | 19:59 |
| xBsd | does it canceled the running job? | 19:59 |
| clarkb | jeblair: mordred fungi The gearman plugin jobs that should be converted to freestyle projects are still maven projects in jenkins. I am going to manually delete those jobs and see if that is what JJB needs to create them properly | 20:00 |
| clarkb | xBsd: it will not cancel the running job | 20:00 |
| mordred | clarkb: I believe that is the case | 20:00 |
| mordred | clarkb: I believe we have learned before that jjb cannot handle transition from maven to freestyle | 20:01 |
| xBsd | clarkb: thanks | 20:01 |
| clarkb | xBsd: it is effectively a noop | 20:01 |
| xBsd | btw, someone've just restarted the gate? | 20:02 |
| clarkb | xBsd: I think zuul did that automagically. There was a change that failed a test so as soon as it was kicked out everything behind it restarted | 20:04 |
| lifeless | bodepd: I did? Sorry! | 20:04 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gearman-plugin: Don't wait for the worker thread to join https://review.openstack.org/42410 | 20:07 |
| jeblair | okay. that seems to work; i think there may be some bad interactions with having multiple executors and disabling on complete, possibly while also stopping the plugin. | 20:08 |
| jeblair | but it seems to work for our typcal case of one executor | 20:08 |
| *** gyee has joined #openstack-infra | 20:08 | |
| jeblair | i think on balance, it's probably a good idea for us to deploy now, and maybe test a bit further before actually making a release | 20:08 |
| *** psedlak has quit IRC | 20:08 | |
| jeblair | jenkins01 is in shutdown mode | 20:09 |
| clarkb | I am updating jobs on jenkins01. should be done shortly | 20:10 |
| clarkb | (to fix the gearman plugin jobs) | 20:10 |
| clarkb | jeblair: nevermind carry on. Looks like puppet is disabled on that server so it didn't get the new configs | 20:11 |
| jeblair | clarkb: yes, that's true; i did that to apply the new firewall rules | 20:11 |
| clarkb | jeblair: I see. Has that change merged yet? | 20:11 |
| jeblair | i'm about to start merging those changes to restore normalcy | 20:11 |
| clarkb | jeblair: ok, just let me know when you are ready for puppet again and I can update those jobs | 20:12 |
| mordred | jeblair: yay | 20:14 |
| *** xBsd has quit IRC | 20:14 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add nodepool host https://review.openstack.org/42232 | 20:15 |
| jeblair | mordred, clarkb: ^ review/aprv that at will | 20:16 |
| *** sarob has quit IRC | 20:17 | |
| *** nayward has quit IRC | 20:17 | |
| *** sarob has joined #openstack-infra | 20:17 | |
| vipul | clarkb, mordred: why is this failing the requirements check? https://review.openstack.org/#/c/42392/1 | 20:17 |
| vipul | I seem to have the required oslo.config>=1.2.0a3 | 20:18 |
| vipul | Requirement oslo.config>=1.2.0a does not match openstack/requirements value oslo.config>=1.2.0a3 | 20:18 |
| mordred | jeblair: what if I don't wanna? | 20:18 |
| clarkb | vipul: right they are different | 20:18 |
| *** krtaylor has joined #openstack-infra | 20:18 | |
| mordred | vipul: you need a 3 | 20:18 |
| vipul | https://review.openstack.org/#/c/42392/1/requirements.txt | 20:19 |
| mordred | oh! | 20:19 |
| vipul | i do | 20:19 |
| mordred | he has one | 20:19 |
| mordred | wtf | 20:19 |
| mordred | DAMMIT | 20:19 |
| mordred | I hate this particular system | 20:19 |
| mordred | it's so fragile | 20:19 |
| clarkb | mordred: do you know what the problem is? my current hunch is that the test can't split the version off of that url properly | 20:26 |
| clarkb | oh and this is dev-requirements involved | 20:28 |
| mordred | yeah | 20:29 |
| mordred | I'm needing to not think about it for a second, because I'm just going to get angry | 20:29 |
| mordred | meh, that's the wrong word | 20:30 |
| mordred | frustrated | 20:30 |
| mordred | esp because we have a new and better system SO CLOSE | 20:30 |
| jeblair | mordred: to what new and better system are you referring? | 20:30 |
| clarkb | mordred: can you review and possibly approve jeblair's change to add the nodepool node? | 20:30 |
| clarkb | that is something useful that doesn't involve requirements | 20:31 |
| jeblair | mordred: i'll look into the requirements script (i'm waiting for jenkins to get to my changes) | 20:31 |
| *** jpmelos has left #openstack-infra | 20:32 | |
| clarkb | jeblair: I think I see one potential problem | 20:32 |
| mordred | jeblair: thanks! | 20:32 |
| mordred | clarkb: and yes | 20:32 |
| mordred | jeblair: the "upload pre-releases to pypi as wheels only and require pip 1.4" system | 20:33 |
| mordred | jeblair: because, zomg, grokking what's going on with the tarball url is a constant struggle | 20:33 |
| jeblair | mordred: gotcha | 20:33 |
| clarkb | jeblair: I think the version for the tarball is parsed by us, but the version without the tarballs url is processed by pkg_resources | 20:33 |
| *** markmcclain has left #openstack-infra | 20:34 | |
| clarkb | so we may end up with different results. I am trying to find where that actually happens so that I can confrim | 20:34 |
| *** markmcclain has joined #openstack-infra | 20:34 | |
| jeblair | vipul: add a newline to the end of the requirements.txt file | 20:34 |
| *** woodspa__ is now known as woodspa | 20:34 | |
| jeblair | clarkb:, mordred ^ | 20:34 |
| vipul | woah really? | 20:34 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Make the local script directory configurable https://review.openstack.org/42233 | 20:35 |
| mordred | jeblair: thank you | 20:35 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Use MySQL https://review.openstack.org/42234 | 20:35 |
| jeblair | vipul: yep. obviously the error message should be corrected or we should change the parsing, but that's the immediate cause; should get you going. | 20:35 |
| clarkb | jeblair: can you point that out to me in the code? | 20:35 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Require a target name when instantiating a node https://review.openstack.org/42246 | 20:35 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Make the target name required in the schema https://review.openstack.org/42251 | 20:35 |
| *** sdake_ has joined #openstack-infra | 20:36 | |
| *** sdake_ has quit IRC | 20:36 | |
| jeblair | clarkb: line = line[:line.find('#')] | 20:36 |
| vipul | jeblair: thanks for digging that up.. let's see how it goes | 20:36 |
| *** sdake_ has joined #openstack-infra | 20:36 | |
| *** sdake_ has quit IRC | 20:36 | |
| *** sdake_ has joined #openstack-infra | 20:36 | |
| jeblair | that's a very silly error | 20:37 |
| clarkb | jeblair: that should make it egg=oslo.config>=1.2.something right? | 20:37 |
| jeblair | it says "strip off the # and anything after, or if it's not found, just strip off the rightmost char" | 20:37 |
| jeblair | so that means most lines were getting their newline removed there (instead of in the next line which actually calls a strip()) | 20:38 |
| jeblair | so, having said that, should we also enforce newlines at the end of every line? | 20:38 |
| jeblair | with a correct error message? | 20:38 |
| clarkb | I think we should if this is the alternative | 20:39 |
| clarkb | or maybe we can just be smarter about that check | 20:39 |
| jeblair | clarkb: it's not, it's quite a simple fix. i've already written it. | 20:39 |
| *** xBsd has joined #openstack-infra | 20:39 | |
| jeblair | so, consider the problem fixed. but now, as a matter of principle, should we have this script also validate that there's a newline at the end of the file? | 20:40 |
| clarkb | I'm on the fence. My editor does the right thing. But I know many other editors do not | 20:40 |
| clarkb | could become frustrating for people with silly editors | 20:41 |
| *** sarob has quit IRC | 20:42 | |
| mordred | I think that if we were goign to validate newline | 20:44 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix line parsing in requirements check https://review.openstack.org/42415 | 20:44 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add final newline check for requirements https://review.openstack.org/42416 | 20:44 |
| mordred | we shoudl have a message that says that's what we're doing | 20:44 |
| anteaya | mordred I sent you a pm | 20:44 |
| jeblair | mordred: wish granted :) | 20:44 |
| jeblair | clarkb: there are the two changes -- fix (and be loose), followed by be strict | 20:45 |
| jeblair | back to the other thing for me now | 20:45 |
| clarkb | jeblair: ty | 20:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Add nodepool host https://review.openstack.org/42232 | 20:47 |
| *** pabelanger has joined #openstack-infra | 20:49 | |
| jeblair | clarkb: you're clear to start puppet on jenkins* and run jjb | 20:51 |
| jeblair | clarkb: ci-puppetmaster has been updated | 20:51 |
| clarkb | jeblair: ok will do | 20:51 |
| clarkb | jeblair: I noticed that on jenkins.o.o it only updated one of the four jobs. Should I flush all of the cache or just ignore cache for this particular set of jobs? | 20:52 |
| jeblair | clarkb: it takes a loooong time to do all jobs; i'd try to narrow it | 20:52 |
| *** rfolco has quit IRC | 20:53 | |
| jeblair | like 30 mins to an hour | 20:53 |
| clarkb | ok, my only concern is that I think irgnore cache won't remove the offending jobs from the cache | 20:53 |
| clarkb | is it possible to remove them by hand? I haven't gone digging into that cache before | 20:53 |
| jeblair | clarkb: yes, it's just a json file | 20:53 |
| clarkb | I will let puppet run JJB then swing back around and cleanup what got missed | 20:54 |
| jeblair | clarkb, mordred, zaro: can you review this? https://review.openstack.org/#/c/42410/ | 20:55 |
| jeblair | clarkb: when you're done fixing the jobs (since it would be nice for them to work when we aprv that) | 20:56 |
| clarkb | jenkins01 and jenkins02 got the jobs applied just fine | 20:57 |
| clarkb | looking more closely at jenkins.o.o now | 20:57 |
| clarkb | puppet started on 01 and 02 as well | 20:58 |
| pabelanger | jeblair, appologies for missing out on testing, got called away from my computer | 20:59 |
| rustlebee | jeblair: was just thinking about testing ... can spin up other servers, but the DID is only going to work from one of them at a time | 21:00 |
| rustlebee | so probably best to only have one set up at a time | 21:01 |
| jeblair | rustlebee: yeah, we can manually stop/start asterisk on them as needed | 21:01 |
| rustlebee | true | 21:01 |
| rustlebee | that works | 21:01 |
| rustlebee | they'll fight over the DID otherwise :-) | 21:01 |
| jeblair | i'm sure the provider would love that | 21:01 |
| rustlebee | it's *possible* they could allow multiple registrations, but even if they did, the servers would fight over answering the call | 21:02 |
| clarkb | jenkins.o.o should be good now | 21:03 |
| *** lcestari has quit IRC | 21:03 | |
| *** mrodden has quit IRC | 21:04 | |
| clarkb | jeblair: I approved your gearman plugin change | 21:05 |
| clarkb | jeblair: why is jenkins01 ins shutdown only mode? there are no jobs running on it now so you can do what you need there possibly | 21:07 |
| *** sdake has quit IRC | 21:07 | |
| *** sdake has joined #openstack-infra | 21:07 | |
| *** sdake has quit IRC | 21:07 | |
| *** sdake has joined #openstack-infra | 21:07 | |
| jeblair | clarkb: to get the new version of the gearman plugin | 21:08 |
| clarkb | that causes a bit of a chicken and egg... half capacity jenkins is slow to merge things | 21:09 |
| jeblair | clarkb: the irony is terrible | 21:09 |
| mordred | oh no! I just got a message fro mhpcloud | 21:09 |
| mordred | they lost the node that housed Instance Name: devstack-precise-1362719079.template.openstack.org | 21:09 |
| mordred | whatever will we do? | 21:09 |
| *** wenlock has quit IRC | 21:11 | |
| *** changbl has quit IRC | 21:11 | |
| *** adalbas has quit IRC | 21:14 | |
| clarkb | mordred: by the way the source stuff for TOX_THING got applied when I started puppet on jenkins0[12] | 21:15 |
| bodepd | lifeless: no worries. I'm still extremely interested in the work you guys are doing :) | 21:15 |
| *** ryanpetrello has quit IRC | 21:15 | |
| *** pentameter has quit IRC | 21:16 | |
| bodepd | lifeless: the pace of innovation is staggering | 21:16 |
| *** colinmcnamara has joined #openstack-infra | 21:17 | |
| *** mrodden has joined #openstack-infra | 21:18 | |
| *** thomasbiege has joined #openstack-infra | 21:18 | |
| *** thomasbiege has quit IRC | 21:19 | |
| mordred | clarkb: the source stuff for ... oh, that's fine | 21:21 |
| clarkb | woo gate reset again... I think there is flakyness in grenade and neutron devstack tests | 21:21 |
| openstackgerrit | A change was merged to openstack-infra/gearman-plugin: remove restriction on slave to run single job at a time https://review.openstack.org/42226 | 21:22 |
| openstackgerrit | A change was merged to openstack-infra/gearman-plugin: Don't wait for the worker thread to join https://review.openstack.org/42410 | 21:22 |
| clarkb | jgriffith: https://jenkins02.openstack.org/job/gate-grenade-devstack-vm/1931/console | 21:22 |
| clarkb | jeblair: ^ gearman-plugin changes merged | 21:22 |
| clarkb | now we wait for hpi artifact upload | 21:22 |
| jeblair | i'll just build it myself | 21:23 |
| jgriffith | clarkb: looking | 21:23 |
| jgriffith | clarkb: any way to get the cinder logs back? | 21:25 |
| *** dina_belova has quit IRC | 21:25 | |
| clarkb | jgriffith: let me see if I can get them | 21:25 |
| jgriffith | clarkb: and is this a one-off or have you seen multiple failed to creat vol? | 21:25 |
| clarkb | jgriffith: http://logs.openstack.org/31/42331/1/gate/gate-grenade-devstack-vm/3d78ceb/logs/new/ | 21:25 |
| clarkb | jgriffith: not sure yet that was my next thing to check | 21:26 |
| jgriffith | clarkb: cinder-volume never started? | 21:26 |
| jgriffith | clarkb: so the creates fail due to inablity to schedule | 21:27 |
| clarkb | jgriffith: interesting. Looks like it started the first time around http://logs.openstack.org/31/42331/1/gate/gate-grenade-devstack-vm/3d78ceb/logs/old/ | 21:27 |
| jeblair | restarting jenkins01 | 21:27 |
| clarkb | but after the upgrade it didn't? | 21:27 |
| jgriffith | clarkb: don't ahve any insight as to why the startup failed | 21:27 |
| jgriffith | clarkb: indeed, it's not starting up after the upgrade for some reason | 21:28 |
| jeblair | clarkb, zaro: jenkins complained about postbuilders in a freestyle project; i think at least some jobs still need some work | 21:29 |
| jeblair | jenkins01 is running gearman-plugin 0.0.3-7-g8e62012 | 21:29 |
| *** adalbas has joined #openstack-infra | 21:29 | |
| jeblair | let's give the queue some time to recover before shutting jenkins02 down | 21:30 |
| clarkb | jgriffith: I am not seeming more incidences of this over the last 4 hours /me extends search range | 21:31 |
| clarkb | 12 hours is same story. So this may be a fluke | 21:32 |
| jgriffith | clarkb: no changes in grenade recently... /me looking at grenade logs | 21:32 |
| jgriffith | clarkb: concerning because we saw service start issues several months back | 21:35 |
| jgriffith | clarkb: think you were involved, when we reworked the startup scripts | 21:35 |
| zaro | jeblair: looking into build failure. | 21:36 |
| jgriffith | clarkb: Keep me posted if you see more, I'm concerned that the "touch" of the log file apparantly didn't even work, indicating we never really tried to start the service? | 21:37 |
| clarkb | jgriffith: http://paste.openstack.org/show/44352/ | 21:37 |
| *** toddmorey has joined #openstack-infra | 21:37 | |
| *** adalbas has quit IRC | 21:38 | |
| jgriffith | clarkb: interesting, it says it started it | 21:39 |
| clarkb | ya | 21:41 |
| clarkb | jgriffith: + screen -S stack -p c-vol -X stuff 'cd /opt/stack/new/cinder && /opt/stack/new/cinder/bin/cinder-volume --config-file /etc/cinder/cinder.conf || touch "/opt/stack/new/status/stack/c-vol.failure" I wonder if we should be trying to get the .failure file? | 21:42 |
| clarkb | seems like || echo "c-vol failed to start" would be more useful | 21:43 |
| jgriffith | clarkb: +1 | 21:43 |
| clarkb | dtroyer: ^ | 21:43 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: move copy step to builder https://review.openstack.org/42422 | 21:44 |
| dtroyer | jgriffith: that's a good point…we could do both actually... | 21:46 |
| jgriffith | clarkb: deserves credit there :) | 21:46 |
| jgriffith | dtroyer: I like the idea of both | 21:47 |
| dtroyer | jgriffith: a little tee magic would do it I think | 21:47 |
| jgriffith | dtroyer: didn't we go down this road a while back? | 21:47 |
| clarkb | both works. I just find the logs to be easier to deal with than the presence of specific files | 21:47 |
| clarkb | I can write the patch if I can find where it should go | 21:48 |
| openstackgerrit | Petr Blaho proposed a change to openstack-infra/config: Adds Jenkins jobs for python-tuskarclient https://review.openstack.org/41887 | 21:49 |
| dtroyer | it would be in the command line of every screen_it command | 21:49 |
| clarkb | dtroyer: if I update devstack will grenage pick that up | 21:49 |
| clarkb | looks like it is a function so I cna just update that function | 21:49 |
| dtroyer | change "|| touch …." to something like " || echo "message" | tee filename" | 21:50 |
| dtroyer | oh, wait, yeah, it is in the wrapper….duh | 21:50 |
| dtroyer | clarkb: yup, grenade trunk checks out devstack trunk so they stay in sync. | 21:51 |
| clarkb | dtroyer: http://paste.openstack.org/show/44353/ that look correct? | 21:51 |
| clarkb | not sure what the newline $NL thing is for | 21:52 |
| dtroyer | clarkb: close, there's some quote nastiness in there | 21:52 |
| dtroyer | the $NL adds a newline for screen's benefit | 21:52 |
| clarkb | I need to escape the quotes I added | 21:52 |
| dtroyer | you need to escape the " in the echo | 21:52 |
| clarkb | http://paste.openstack.org/show/44354/ better? | 21:53 |
| dtroyer | I think so. what I'm unsure about is the precedence between '||' (or) and '|' (pipe). you may need to wrap the echo | tee bit in a () subshell… | 21:55 |
| clarkb | local testing seems to indicate it works as expected | 21:55 |
| dtroyer | yeah, my simple test worked too | 21:56 |
| clarkb | true || echo "foo" | tee foo.bar then false || echo "foo" | tee foo.bar | 21:57 |
| clarkb | so I think that is fine. /me pushes | 21:57 |
| jgriffith | dtroyer: clarkb worked for me | 21:57 |
| dtroyer | quit copying my shell commands! | 21:57 |
| clarkb | dtroyer: jgriffith https://review.openstack.org/42427 | 21:59 |
| clarkb | it will probably getting tested some time far in the future | 21:59 |
| clarkb | this push to thefeature freeze has been insane | 22:00 |
| dtroyer | the review before it was < 1 hour… there are advantages to working late on Friday | 22:01 |
| * jgriffith is changing his work week to Wed-Sunday | 22:01 | |
| clarkb | oh no the secret is out | 22:01 |
| jgriffith | or just straight friday night to monday am :) | 22:02 |
| clarkb | though us infra people tend to do more of the Sunday -Sunday | 22:02 |
| clarkb | Though I think just about all of us are doing normal people stuff this weekend | 22:02 |
| *** ftcjeff has quit IRC | 22:04 | |
| jgriffith | clarkb: can you clarify *normal-people* | 22:05 |
| fungi | clarkb: dtroyer the rule of thumb is that &&, || and ; don't continue pipelines and are higher order than | is | 22:05 |
| dtroyer | except on odd Thursdays and full moons | 22:06 |
| clarkb | jgriffith: I think of my brother when I think normal people | 22:07 |
| fungi | the test load from this morning is just not catching up | 22:07 |
| clarkb | fungi: yeah it is insane. Turning a jenkins off for an extended period of time didn't help either though | 22:07 |
| clarkb | thankfully hte weekend is around the corner | 22:08 |
| jgriffith | clarkb: interesting... now that you mention it | 22:08 |
| clarkb | jgriffith: he has a job that my mother understands | 22:09 |
| *** mriedem has quit IRC | 22:09 | |
| clarkb | and doesn't work on his days off | 22:09 |
| clarkb | fungi: when will you be joining the not so sunny anymore northwest? | 22:09 |
| anteaya | clarkb: your parents don't understand your work either? | 22:10 |
| clarkb | anteaya: nope. They get really confused when I try to explain that OpenStack is open and free as in beer | 22:10 |
| anteaya | I keep getting advice, and when I try to tell them, they just keep giving me the same advice | 22:10 |
| anteaya | ha ha ha | 22:10 |
| anteaya | my father keeps wanting me to track my hours | 22:10 |
| anteaya | "You have to keep track of your hours." | 22:10 |
| clarkb | "Why would anyone want to pay you to work on something that is free?" | 22:10 |
| anteaya | I have given up trying to explain, so I just nod and try to curb my tongue | 22:11 |
| anteaya | I am bad at the tongue curbing though | 22:11 |
| anteaya | ha ha ha | 22:11 |
| clarkb | jeblair: fungi mordred https://review.openstack.org/#/c/42422/ I will approve that one in a bit if no one beats me to it | 22:11 |
| jeblair | fungi: i don't think it's the test load from this morning; it's the test load from only having one jenkins | 22:11 |
| *** woodspa has quit IRC | 22:12 | |
| clarkb | fungi: and I think that if you are comfortable with potentially flaky tests we should merge https://review.openstack.org/#/c/35104/22 | 22:13 |
| clarkb | fungi: I will update my vote | 22:13 |
| clarkb | (because debugging things will only be harder if we keep hitting that change with a hammer) | 22:13 |
| jeblair | i just got really freaked out because i saw idle devstack nodes | 22:14 |
| jeblair | but i think that's correct -- all the devstack jobs are actually currently being run | 22:14 |
| jeblair | we're waiting on unit test runners | 22:15 |
| clarkb | jeblair: yeah we fell behind on precise workers pretty badly | 22:15 |
| jeblair | we because of the way d-g works, more load shifts to the other jenkins for devstack jobs if we shut one down | 22:15 |
| jeblair | but not unit test runners | 22:16 |
| jeblair | we actually do need to add more, i think; it probably wouldn't hurt to go ahead and add, say 4 precise and 2 centos nodes | 22:16 |
| *** blamar has joined #openstack-infra | 22:16 | |
| clarkb | we basically go into a tighter create, destroy loop in d-g when that happens | 22:16 |
| jeblair | so i think there's a problem with nodepool and mysql connections; i'm going to be spending some time working deeply on that | 22:17 |
| clarkb | http://tinyurl.com/mc2k5oj is pretty impressive | 22:18 |
| clarkb | I should change that to per day | 22:18 |
| jeblair | wow | 22:19 |
| clarkb | it is claiming almost 450 changes merged on 8/12 | 22:19 |
| clarkb | not sure if that is correct but wow | 22:19 |
| fungi | clarkb: we're due in mid-evening tomorrow | 22:25 |
| *** SergeyLukjanov has quit IRC | 22:25 | |
| *** dina_belova has joined #openstack-infra | 22:25 | |
| *** blamar has quit IRC | 22:25 | |
| clarkb | http://tinyurl.com/mz8qgpl there is the daily counter | 22:26 |
| clarkb | which is uhm a little impressive if correct | 22:26 |
| *** sarob has joined #openstack-infra | 22:28 | |
| *** bookmage has quit IRC | 22:28 | |
| *** sarob has quit IRC | 22:28 | |
| *** sarob has joined #openstack-infra | 22:29 | |
| *** dina_belova has quit IRC | 22:30 | |
| fungi | i would believe that | 22:31 |
| NobodyCam | Filter projects <- TY :) | 22:32 |
| *** wu_wenxiang has joined #openstack-infra | 22:33 | |
| *** rcleere has quit IRC | 22:35 | |
| *** wu_wenxiang has quit IRC | 22:35 | |
| *** mriedem has joined #openstack-infra | 22:35 | |
| clarkb | NobodyCam: that was a community contribution. | 22:35 |
| *** dina_belova has joined #openstack-infra | 22:36 | |
| jeblair | clarkb: every contribution is a community contribution | 22:36 |
| openstackgerrit | A change was merged to openstack-infra/config: move copy step to builder https://review.openstack.org/42422 | 22:36 |
| clarkb | jeblair: fair neough. I meant to say came from someone other than the coremudgeons | 22:36 |
| clarkb | it is a handy little feature. I already want it to filter on change number as well :) | 22:37 |
| jeblair | that was from sergey lukjanov | 22:37 |
| jeblair | and ryan petrello has been enhancing it, it looks like | 22:38 |
| *** moted has quit IRC | 22:38 | |
| *** moted has joined #openstack-infra | 22:39 | |
| *** dina_belova has quit IRC | 22:40 | |
| NobodyCam | awesome and I just made use use of it | 22:41 |
| NobodyCam | s/use use/use/ | 22:42 |
| openstackgerrit | A change was merged to openstack-infra/config: Add more details to git server documentation https://review.openstack.org/42405 | 22:47 |
| clarkb | that was fast. I assume that means jenkins is much happier now | 22:47 |
| fungi | clarkb: well, it prioritizes gate jobs | 22:48 |
| fungi | i was waiting for that to come back with check results, but that was probably just lazy of me | 22:48 |
| pleia2 | I do love the spiffy new progress bars zuul status page | 22:48 |
| fungi | especially since i doubt any of our check jobs would have had anything to say about it anyway, being pure documentation updates | 22:49 |
| clarkb | fungi: I approved it after the check jobs reported | 22:49 |
| fungi | you are quick like ninja | 22:49 |
| fungi | oh, and i stand corrected. we actually arrive around 2pm pdt | 22:50 |
| clarkb | jeblair: dtroyer: https://review.openstack.org/#/c/42427/ has passed tests \o/ | 22:50 |
| fungi | the marvels of travelling against the earth's rotation | 22:50 |
| clarkb | fungi: I will be driving to portland saturday then back again on sunday | 22:51 |
| fungi | have fun! | 22:51 |
| clarkb | fungi: I am going to try and bring back a growler or two of the homebrew | 22:51 |
| fungi | sounds like a lot of hassle for an overnight, but beer definitely makes it worthwhile | 22:51 |
| clarkb | it can be a pain, but I prefer the short stays | 22:52 |
| fungi | i am pleased because the bar a block from my townhouse started a brewery in their back room (a sizeable one at that) and started serving their first four varieties a couple weeks ago. good stuff | 22:54 |
| *** xBsd has quit IRC | 22:54 | |
| clarkb | fungi: https://review.openstack.org/#/c/42415/ can I request that that change get reviewed? | 22:54 |
| fungi | taking a look | 22:54 |
| fungi | and then i need to pack two weeks of provisions | 22:54 |
| pleia2 | ooh, is it bug fungi for reviews time? | 22:54 |
| clarkb | pleia2: I guess. Do you have changes I should review but haven't yet? | 22:55 |
| pleia2 | I swear it's quick! https://review.openstack.org/#/c/42168/4/modules/cgit/manifests/init.pp just a question about /var/www directory | 22:55 |
| clarkb | I actually did a reasonable job yesterday of catching up on things | 22:55 |
| fungi | pleia2: sure | 22:55 |
| *** rnirmal has quit IRC | 22:56 | |
| *** nati_uen_ has joined #openstack-infra | 22:56 | |
| fungi | oh, yeah i already looked at 42415 and was waiting on that to come back with a jenkns +1. lgtm. also a very fun python slice mistake ;) | 22:56 |
| fungi | pleia2: oh, and i missed that mordred -1'd 42168 because you were waiting on answers from me! i just saw he had set it wip and figured you were still working on it :/ | 22:58 |
| pleia2 | fungi: hah, yeah | 22:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix line parsing in requirements check https://review.openstack.org/42415 | 22:58 |
| *** nati_ueno has quit IRC | 22:59 | |
| *** ^d has quit IRC | 23:00 | |
| *** ^d has joined #openstack-infra | 23:00 | |
| *** ^d has joined #openstack-infra | 23:00 | |
| vipul | is jenkins slow today or what? | 23:00 |
| clarkb | yes | 23:02 |
| clarkb | vipul: you guys are running a lot of jobs | 23:02 |
| vipul | hardly :P | 23:02 |
| clarkb | and there were some problems earlier today that amplified the affects of running lots of jobs | 23:02 |
| vipul | ok, just checking | 23:03 |
| *** ^d has quit IRC | 23:05 | |
| clarkb | vipul: also, a little while back we made the gate higher priority than everything else which can starve the check queues | 23:05 |
| *** datsun180b has quit IRC | 23:06 | |
| *** zul has joined #openstack-infra | 23:16 | |
| *** gyee has quit IRC | 23:16 | |
| *** colinmcnamara has quit IRC | 23:21 | |
| *** colinmcnamara has joined #openstack-infra | 23:26 | |
| *** pcm_ has quit IRC | 23:30 | |
| *** UtahDave has quit IRC | 23:30 | |
| *** pcm_ has joined #openstack-infra | 23:34 | |
| *** pcm_ has quit IRC | 23:34 | |
| *** dina_belova has joined #openstack-infra | 23:36 | |
| *** vipul is now known as vipul-away | 23:38 | |
| openstackgerrit | A change was merged to openstack/requirements: Allow use of oslo.messaging 1.2.0a5 https://review.openstack.org/42229 | 23:38 |
| *** dina_belova has quit IRC | 23:41 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add static web directory for cgit & initial files https://review.openstack.org/42168 | 23:41 |
| pleia2 | figured I might as well do the final relative path cleanup for the CSS | 23:41 |
| *** vipul-away is now known as vipul | 23:53 | |
| mordred | woot | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!