| *** openstack has joined #openstack-qa | 16:41 | |
| dimsum__ | mriedem: does not ring a bell | 16:42 |
|---|---|---|
| dimsum__ | mriedem: back in a couple of hours and can check then | 16:42 |
| *** flwang has quit IRC | 16:55 | |
| *** dimsum__ has quit IRC | 16:55 | |
| *** flwang has joined #openstack-qa | 17:07 | |
| *** annashen has joined #openstack-qa | 17:14 | |
| *** ylobankov has quit IRC | 17:14 | |
| *** ilyashakhat has quit IRC | 17:14 | |
| *** enikanorov_ has quit IRC | 17:14 | |
| *** vrovachev has quit IRC | 17:14 | |
| *** eezhova has quit IRC | 17:14 | |
| *** enikanorov has joined #openstack-qa | 17:14 | |
| *** ylobankov has joined #openstack-qa | 17:14 | |
| *** ilyashakhat has joined #openstack-qa | 17:14 | |
| *** eezhova has joined #openstack-qa | 17:15 | |
| *** vrovachev has joined #openstack-qa | 17:16 | |
| *** woodster_ has quit IRC | 17:20 | |
| *** woodster_ has joined #openstack-qa | 17:24 | |
| *** e0ne has joined #openstack-qa | 17:35 | |
| *** yamamoto has joined #openstack-qa | 17:36 | |
| *** e0ne has quit IRC | 17:36 | |
| *** yamamoto has quit IRC | 17:40 | |
| *** dimsum__ has joined #openstack-qa | 17:55 | |
| *** dimsum__ has quit IRC | 18:00 | |
| *** andreykurilin_ has joined #openstack-qa | 18:04 | |
| morganfainberg | mriedem, yes. the postgres one | 18:10 |
| morganfainberg | mriedem, it does not use apache | 18:10 |
| morganfainberg | mriedem, i see clarkb beat me to it... i need to read further :P | 18:10 |
| mriedem | morganfainberg: yeah, check this out though http://goo.gl/GeAHEQ | 18:11 |
| morganfainberg | yep | 18:11 |
| morganfainberg | something changed | 18:12 |
| morganfainberg | sooooo | 18:12 |
| morganfainberg | yeah :( | 18:12 |
| mriedem | morganfainberg: i suspect eventlet 0.16 | 18:12 |
| morganfainberg | likely | 18:12 |
| mriedem | because it's also showing up in neutron | 18:12 |
| morganfainberg | my solution:https://review.openstack.org/#/c/157495/ | 18:13 |
| morganfainberg | i mean, we should probably fix the thing as well | 18:13 |
| morganfainberg | but.. thats my *real* solution to this problem | 18:13 |
| mriedem | yeah... | 18:13 |
| morganfainberg | because... eventlet :( | 18:13 |
| mriedem | this doesn't appear fatal which is a plus :) | 18:14 |
| adam_g | mriedem, no idea how/when this integration job got so broken | 18:40 |
| *** andreykurilin_ has quit IRC | 18:40 | |
| mriedem | adam_g: saw the comment in the neutronclient cap review | 18:44 |
| mriedem | not sure | 18:44 |
| adam_g | mriedem, not sure what changed since then, but syncing seems to be happening again now | 18:44 |
| mriedem | i'm gonna head out for a few hours | 18:45 |
| adam_g | but http://logs.openstack.org/22/158022/2/check/check-requirements-integration-dsvm/9414cb6/console.html.gz#_2015-02-21_04_33_29_966 now | 18:45 |
| adam_g | clarkb, ^ | 18:45 |
| clarkb | adam_g: wait syncing works now? | 18:46 |
| adam_g | clarkb, *shrug* it looks like it | 18:46 |
| adam_g | but now its failing trying to sync stable/icehouse requirements into master of pycadf | 18:46 |
| clarkb | and that change is on icehouse | 18:46 |
| clarkb | and all you did was add a few cats | 18:47 |
| clarkb | this is weird | 18:47 |
| adam_g | https://review.openstack.org/#/c/158022/ is where im trying to debug | 18:48 |
| adam_g | looks like its actually updating requirements.txt and committing them locally | 18:49 |
| adam_g | im not sure how the requirement syncs are supposed to work for things listed in projects.txt like pycadf, that have caps in global-requirements.txt | 18:49 |
| adam_g | integration.sh just checks out pycadf master and attempts to sync stable/icehouse and fails, rather than trying to checkout a tag within the GR.txt constraints | 18:50 |
| clarkb | adam_g: it shouldn't work any different, the sync process only looks in requirements files, it doesn't do anything smart to say I am in pycadf an dthis is pinned | 18:50 |
| clarkb | oh right if pycadf doesn't have a stable/icehouse it does the normal fallback which is master | 18:50 |
| *** mriedem1 has joined #openstack-qa | 18:50 | |
| adam_g | clarkb, yea, so i dont know how that'd ever work for pycadf or similar | 18:50 |
| clarkb | right, anything that moves beyond where stable/icehouse is will fail | 18:51 |
| clarkb | * if it odesn't have the icehouse branch | 18:51 |
| adam_g | there are some conditional skips in integration.sh to skip things, but i dont see anything in history that would suggest pycadf was skipped in the past | 18:51 |
| * adam_g is driving through southern oregon | 18:51 | |
| clarkb | adam_g: fishing? | 18:51 |
| adam_g | i wonder if i'll leave service area or get car sick first | 18:52 |
| clarkb | ha | 18:52 |
| adam_g | clarkb, i wish, heading to the bay | 18:52 |
| *** e0ne has joined #openstack-qa | 18:53 | |
| clarkb | adam_g: looking more closely at the change that didn't sync, it failed because of libvirt-python | 18:53 |
| adam_g | clarkb, yeah, thats meant to be fixed by the pyghmi cap | 18:54 |
| clarkb | which should not be in icehouse as a requirement due to precise not having a new enough version of libvirt to link the python against | 18:54 |
| adam_g | but thats not being synced | 18:54 |
| *** mriedem has quit IRC | 18:54 | |
| adam_g | some new pyghmi added a libvirt-python dependecy | 18:54 |
| adam_g | so we're trying to cap that | 18:54 |
| adam_g | but the sync isn't/wasn't working so it'd still blow up with no cap | 18:55 |
| clarkb | right | 18:55 |
| clarkb | so two complementary issues, solving the syncing problem should allow us to fix the libvirt problem | 18:55 |
| * clarkb mumbles something about caps not fixing the world | 18:55 | |
| clarkb | adam_g: and your test patch does sync and as a result fails earlier right? | 18:56 |
| adam_g | clarkb, if fails later, processing pycadf | 18:57 |
| clarkb | gotcha | 18:57 |
| adam_g | the sync worked (forwhatever reason) and fixes the libvirt issue | 18:57 |
| clarkb | adam_g: have we rechecekd the first change to see if the sync problem is consistent there? | 18:57 |
| adam_g | clarkb, yeah, rechecked it a while ago to see if syncing resumed there as well | 18:58 |
| clarkb | (I ended up falling of the internet last night due to major swapping) | 18:58 |
| clarkb | 8GB of ram and chromium + firefox had me 8GB into swap | 18:58 |
| clarkb | was not fun | 18:58 |
| lifeless | clarkb: LOL | 18:58 |
| lifeless | clarkb: ulimit time? | 18:58 |
| clarkb | lifeless: or cgroups | 18:58 |
| lifeless | same same | 18:59 |
| lifeless | interesting current chromium leaks fd's like they are candy | 18:59 |
| lifeless | I have to raise my fd limit to 2K or get sadface errors | 18:59 |
| *** annashen has quit IRC | 18:59 | |
| clarkb | adam_g: looks like your recheck failed the same way with libvirt compile erroring, so theory time. Bug in sync script tickled by that specific update, somehow corrected by your test (this makes no sense because your test change doesn't touch requirements) | 19:01 |
| adam_g | clarkb, im thinking the issue /w pycadf has always been there waiting to crop up, and did once pycadf added oslo.context on 02-04 | 19:02 |
| adam_g | clarkb, re the syncing failing, my only hunch is that the integration script is doing its git work from the wrong directory, maybe the directory stack is being polluted somehow | 19:03 |
| clarkb | adam_g: yup I think that issue has always been there but is only recently a problem with the aggressive version pinning (though we apparently haven't done that to icehouse) | 19:04 |
| clarkb | ^^ is the pycadf issue | 19:04 |
| adam_g | we did cap pycadf sorta recently | 19:04 |
| adam_g | the syncing process: clone repo, sync requirements, commit requirement changes, re-clone updated repo elsewhere and install from there | 19:05 |
| adam_g | when it fails, the commit stage doesnt seem to be picking up any changes | 19:05 |
| clarkb | oh is it not installing from a synced version? | 19:05 |
| clarkb | that would probably explain it, let me dig through logs | 19:05 |
| adam_g | i attempted to add some pwd to my debug patch to find out where its trying to commit changes | 19:06 |
| adam_g | but, of course, that change didnt show the problem | 19:06 |
| adam_g | and it seemed to work | 19:06 |
| clarkb | right | 19:06 |
| adam_g | https://git.openstack.org/cgit/openstack/requirements/tree/tools/integration.sh?h=stable/icehouse#n88 | 19:06 |
| clarkb | that path looks correct REPODIR/SHORTPROJECT would be /opt/stack/new/ironic which is where d-g sets up the repo with the correct branch | 19:09 |
| clarkb | will try to confirm via logs | 19:09 |
| adam_g | yup | 19:10 |
| clarkb | http://logs.openstack.org/06/157606/4/check/check-requirements-integration-dsvm/007603a/console.html#_2015-02-21_00_06_13_790 and http://logs.openstack.org/06/157606/4/check/check-requirements-integration-dsvm/007603a/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-02-20_23_43_42_224 seem to match up properly | 19:11 |
| *** e0ne has quit IRC | 19:12 | |
| clarkb | and i have confirmed syncing works locally | 19:14 |
| clarkb | http://logs.openstack.org/06/157606/4/check/check-requirements-integration-dsvm/007603a/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-02-20_23_40_59_069 the requirements repo is properly updated | 19:16 |
| adam_g | yea | 19:17 |
| adam_g | that all works, it looks like update.py updates the projects requirements.txt | 19:17 |
| adam_g | but resulting changes aren't checked into that git repo | 19:17 |
| adam_g | so the following clone+install clones from an unmodified repo with no requirement change | 19:18 |
| clarkb | ya but why would git diff return 0 there? | 19:18 |
| adam_g | only thing i can think of is its exec'ing git from the wrong place | 19:19 |
| adam_g | hoping this fails https://review.openstack.org/#/c/158073/1/tools/integration.sh | 19:19 |
| clarkb | adam_g: seems like a reasonable check | 19:20 |
| adam_g | http://logs.openstack.org/22/158022/1/check/check-requirements-integration-dsvm/b14c87b/console.html.gz#_2015-02-21_03_43_22_882 | 19:24 |
| adam_g | thats with those cats showing the requiremnts file does get updated | 19:24 |
| adam_g | but nothing gets checked into git | 19:24 |
| clarkb | so git must be doing something funny there | 19:25 |
| clarkb | the git status change will hopefully shed more light | 19:25 |
| adam_g | not sure what to do about the pycadf problem | 19:26 |
| adam_g | we either need to make the script smart enough to hceckout a version within gr's constraints or just have it skip syncing requirements for libraries listed in projects.txt that do not have stable branches | 19:26 |
| adam_g | my connections on dying. will check back when i have service | 19:27 |
| clarkb | adam_g: don't get car sick :) | 19:27 |
| adam_g | :X | 19:28 |
| *** yamamoto has joined #openstack-qa | 19:38 | |
| *** yamamoto has quit IRC | 19:42 | |
| clarkb | adam_g: http://logs.openstack.org/73/158073/1/check/check-requirements-integration-dsvm/ed18956/console.html#_2015-02-21_19_41_35_986 | 19:47 |
| clarkb | so that "worked" this is a seriously weird bug, I am going t o push a new patchset to your change if that is ok | 19:47 |
| *** zz_dimtruck is now known as dimtruck | 19:48 | |
| *** tkammer has quit IRC | 20:04 | |
| *** achanda has joined #openstack-qa | 20:39 | |
| *** achanda has quit IRC | 20:53 | |
| *** achanda has joined #openstack-qa | 20:54 | |
| clarkb | adam_g: http://logs.openstack.org/73/158073/3/check/check-requirements-integration-dsvm/b81cc16/console.html#_2015-02-21_20_47_34_104 it almost looks like git diff is racy, I am going to try adding a sync | 20:59 |
| *** MasterPiece has quit IRC | 21:03 | |
| *** SridharG has quit IRC | 21:15 | |
| *** achanda has quit IRC | 21:45 | |
| *** DinaBelova has quit IRC | 21:47 | |
| *** SergeyLukjanov has quit IRC | 21:47 | |
| *** DinaBelova has joined #openstack-qa | 21:50 | |
| *** e0ne has joined #openstack-qa | 21:50 | |
| *** SergeyLukjanov has joined #openstack-qa | 21:51 | |
| adam_g | clarkb, w. t. f. ?? | 21:53 |
| *** SergeyLukjanov has quit IRC | 21:53 | |
| *** DinaBelova has quit IRC | 21:53 | |
| clarkb | adam_g: indeed :) see latest patch, I am doingthe git status first so we can see what it says prior to the diff | 21:53 |
| *** DinaBelova has joined #openstack-qa | 21:54 | |
| adam_g | clarkb, it certainly seems like a race, it works sometimes | 21:55 |
| *** SergeyLukjanov has joined #openstack-qa | 21:56 | |
| clarkb | adding a sync didn't help | 21:56 |
| adam_g | clarkb, why aren't isn't requirements.txt not staged for commit? | 21:58 |
| clarkb | adam_g: it isn't git added | 21:58 |
| adam_g | oh, right | 21:59 |
| adam_g | https://review.openstack.org/#/c/158022/2/tools/integration.sh | 22:00 |
| adam_g | i wonder if that works because we're touching git before the first 'git diff --quiet' | 22:00 |
| clarkb | maybe, it could be triggering the racy code in a place where we won't fail | 22:00 |
| *** e0ne has quit IRC | 22:08 | |
| clarkb | adam_g: lol running the git status first makes it work | 22:08 |
| clarkb | this is an amazing bug | 22:08 |
| adam_g | clarkb, yeah, thats how i assume it was working on 158022 | 22:09 |
| adam_g | clarkb, where are you testing? | 22:09 |
| clarkb | adam_g: https://jenkins04.openstack.org/job/check-requirements-integration-dsvm/521/console its your change, I have pushed up a few patchsets | 22:09 |
| adam_g | did you try adding a sync after update.py / before 'git diff --quiet' | 22:09 |
| clarkb | adam_g: yes, that didn't fix it | 22:09 |
| adam_g | hmm | 22:10 |
| adam_g | also curious if its specific to a cloud provider | 22:10 |
| *** DinaBelova has quit IRC | 22:17 | |
| *** SergeyLukjanov has quit IRC | 22:17 | |
| *** DinaBelova has joined #openstack-qa | 22:17 | |
| *** SergeyLukjanov has joined #openstack-qa | 22:19 | |
| *** achanda has joined #openstack-qa | 22:19 | |
| *** DinaBelova has quit IRC | 22:22 | |
| *** SergeyLukjanov has quit IRC | 22:22 | |
| *** DinaBelova has joined #openstack-qa | 22:22 | |
| clarkb | could be | 22:23 |
| clarkb | I need to drop off now, but I think we hvae narrowed it down to that first git operation may not work right | 22:23 |
| *** SergeyLukjanov has joined #openstack-qa | 22:24 | |
| *** jester has joined #openstack-qa | 22:31 | |
| jester | anybody use Testcomplete? | 22:33 |
| *** achanda has quit IRC | 22:37 | |
| *** DinaBelova has quit IRC | 22:39 | |
| *** SergeyLukjanov has quit IRC | 22:39 | |
| *** DinaBelova has joined #openstack-qa | 22:40 | |
| *** yamamoto has joined #openstack-qa | 22:40 | |
| *** SergeyLukjanov has joined #openstack-qa | 22:41 | |
| *** DinaBelova has quit IRC | 22:41 | |
| *** SergeyLukjanov has quit IRC | 22:41 | |
| *** DinaBelova has joined #openstack-qa | 22:42 | |
| *** salv-orlando has joined #openstack-qa | 22:43 | |
| *** SergeyLukjanov has joined #openstack-qa | 22:44 | |
| *** yamamoto has quit IRC | 22:45 | |
| *** dmellado has quit IRC | 23:00 | |
| *** andreykurilin_ has joined #openstack-qa | 23:02 | |
| *** achanda has joined #openstack-qa | 23:02 | |
| *** dmellado has joined #openstack-qa | 23:03 | |
| *** adalba has quit IRC | 23:17 | |
| *** achanda has quit IRC | 23:23 | |
| *** achanda has joined #openstack-qa | 23:26 | |
| *** salv-orl_ has joined #openstack-qa | 23:37 | |
| *** salv-orlando has quit IRC | 23:37 | |
| *** yfried has joined #openstack-qa | 23:38 | |
| *** achanda has quit IRC | 23:40 | |
| *** yfried|afk has quit IRC | 23:41 | |
| *** achanda has joined #openstack-qa | 23:43 | |
| *** achanda has quit IRC | 23:43 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!