| harlowja | anyway someone could verify (if possible) | 00:00 |
|---|---|---|
| lifeless | sdague: (overlapping within one worker is hugely unexpected...) | 00:00 |
| *** nati_ueno has quit IRC | 00:00 | |
| sdague | yeh, that was the "heh" | 00:01 |
| *** _david_ has quit IRC | 00:02 | |
| *** nati_ueno has joined #openstack-infra | 00:02 | |
| *** slong_ has joined #openstack-infra | 00:02 | |
| fungi | harlowja: taking a look | 00:02 |
| harlowja | thx fungi | 00:02 |
| zaro | clarkb: looks good | 00:02 |
| *** nati_ueno has quit IRC | 00:02 | |
| pleia2 | fungi: you mentioned you'd look at cache_git_repos.py still referencing review.projects.yaml.erb, I am running into it now, did you look or should I take a stab at fixing? | 00:03 |
| *** nati_ueno has joined #openstack-infra | 00:03 | |
| fungi | pleia2: you need to update jeepyb or config (whichever one carries that script) | 00:03 |
| fungi | pleia2: it's fixed upstream | 00:04 |
| pleia2 | fungi: ah! thanks, just had the presence of mind to actually look upstream now, thanks | 00:04 |
| fungi | you just have an old copy of the script | 00:04 |
| *** mrodden has joined #openstack-infra | 00:04 | |
| * pleia2 nods | 00:04 | |
| fungi | where old means "from yesterday" | 00:04 |
| *** hogepodge has joined #openstack-infra | 00:04 | |
| clarkb | I am going to restart jenkins03 shortly | 00:04 |
| fungi | clarkb: cool | 00:04 |
| pleia2 | fungi: hehe, right :) | 00:04 |
| * fungi has learned not to apologize for fast-moving projects | 00:05 | |
| fungi | soooo oooooold | 00:05 |
| *** slong has quit IRC | 00:05 | |
| *** dims has joined #openstack-infra | 00:07 | |
| fungi | harlowja: clarkb: time to play everyone's favorite game, "spot the initscript!" http://packages.ubuntu.com/precise/all/zookeeper/filelist | 00:08 |
| harlowja | lol | 00:08 |
| fungi | (hint, you lose) | 00:08 |
| harlowja | ya, who makes this stuff, lol | 00:09 |
| harlowja | bad package | 00:09 |
| lifeless | I hate typos | 00:10 |
| harlowja | fungi http://packages.ubuntu.com/precise/zookeeperd | 00:10 |
| lifeless | test/plain;charset=utf8 | 00:10 |
| lifeless | wtf | 00:10 |
| fungi | lifeless: i saw that too... forget which python package that was now | 00:10 |
| fungi | hilarious though | 00:10 |
| lifeless | ~/personal/testrepository/working/testrepository/commands/run.py | 00:10 |
| lifeless | its me, I dids it | 00:10 |
| harlowja | seems like scripts in another package http://packages.ubuntu.com/precise/all/zookeeperd/filelist | 00:10 |
| fungi | lifeless: you haz tests on brane | 00:11 |
| *** andreaf has quit IRC | 00:11 | |
| lifeless | I'll get a release of testr out to fix that shortly | 00:11 |
| fungi | harlowja: yes, seems like. and we don't currently install tat package | 00:11 |
| harlowja | shall i add a review to? | 00:11 |
| fungi | harlowja: go for it. you want to patch the params.pp file in our jenkins module | 00:12 |
| harlowja | k | 00:12 |
| *** hogepodge has joined #openstack-infra | 00:12 | |
| harlowja | damn thats a big file, lol | 00:12 |
| fungi | harlowja: and also the init.pp for the same module i think | 00:12 |
| harlowja | kk | 00:12 |
| fungi | harlowja: just git grep zookeeper | 00:12 |
| harlowja | lets see if i can do it | 00:12 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add new job to build the javamelody gerrit plugin https://review.openstack.org/70014 | 00:12 |
| fungi | harlowja: so not init.pp, slave.pp | 00:13 |
| harlowja | k | 00:13 |
| fungi | (according to the git grep i just vomited into my dev shell) | 00:13 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Pull in the zookeeperd package instead of the zookeeper package https://review.openstack.org/70022 | 00:16 |
| harlowja | not so hard, haha | 00:17 |
| fungi | harlowja: precisely what i had in mind | 00:17 |
| harlowja | ;) | 00:17 |
| harlowja | seems to start and all that on my ubuntu box, so should be fine i think | 00:18 |
| fungi | clarkb: comment lines excepted, one-character patch ^ ...easy way to pad review stats ;) | 00:18 |
| harlowja | *comment lines removed since its not really just for tooz | 00:19 |
| harlowja | for anyone that wants to use zookeeper | 00:19 |
| fungi | yep, taskflow's use of zookeeper makes those comment lines moot, agreed | 00:19 |
| clarkb | gah battery died | 00:19 |
| clarkb | actually doing jenkins03 now | 00:19 |
| harlowja | fungi nova also should use this for its zookeeper testing | 00:20 |
| harlowja | not sure if that is even being tested currently | 00:20 |
| harlowja | *that code-path | 00:20 |
| fungi | not sure i even knew nova had zookeeper tests | 00:20 |
| fungi | actually, i'm sure i didn't know | 00:20 |
| harlowja | http://docs.openstack.org/trunk/config-reference/content//zookeeper-servicegroup-driver.html | 00:21 |
| harlowja | fungi but i'm not sure its being tested (likely not) | 00:21 |
| anteaya | fungi: thank you | 00:21 |
| harlowja | hard to automatically test it without the above review :) | 00:21 |
| harlowja | *to run zookeeper | 00:21 |
| fungi | i'm pretty positive i hadn't even heard of zookeeper until the review a few weeks ago to install it, but i'll admit to being a bit of a luddite | 00:22 |
| clarkb | do we need the zookeeper package too for client lib bits? | 00:22 |
| clarkb | harlowja: why did you remove the comments? | 00:22 |
| harlowja | clarkb not really, i created https://pypi.python.org/pypi/zake/ to get around that problem | 00:22 |
| fungi | clarkb: i thought it was a dependency of the daemon package, but checking | 00:22 |
| harlowja | but zake is pretty new | 00:22 |
| *** CaptTofu has joined #openstack-infra | 00:23 | |
| harlowja | made so that u don't need to depend on a zookeeper server for tests | 00:23 |
| *** branen has joined #openstack-infra | 00:23 | |
| harlowja | clarkb removal of comments is because its really not just for tooz testing | 00:23 |
| fungi | clarkb: alternative would have been to update the comment to say tooz and taskflow (and soon i guess nova) | 00:23 |
| harlowja | nova should be using it, taskflow, nova... | 00:23 |
| clarkb | harlowja: ok, please leave a comment to that effect then :) | 00:23 |
| harlowja | k | 00:23 |
| clarkb | harlowja: we are trying to keep track of why random things are installed on slaves | 00:23 |
| harlowja | np | 00:23 |
| harlowja | sounds fine | 00:23 |
| clarkb | stopping jenkins03 now | 00:23 |
| clarkb | and starting | 00:24 |
| fungi | clarkb: yeah, http://packages.ubuntu.com/precise/zookeeperd says the zookeeper package is a hard dependency, so both will end up installed | 00:24 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Pull in the zookeeperd package instead of the zookeeper package https://review.openstack.org/70022 | 00:24 |
| harlowja | fungi right | 00:24 |
| harlowja | its in a way similar to the sqlite vs mysql testing, zake i made so that it provides a simpler sqlite like capability, but really nice to also use real zookeeper (like using real mysql) | 00:25 |
| *** matsuhashi has joined #openstack-infra | 00:26 | |
| clarkb | 03 is up, not doing anything as it waits for slaves | 00:26 |
| * fungi is drinking nama-zake now | 00:28 | |
| anteaya | not only I am not the only Anita at this conference, I am not the only Anita K at this conference | 00:28 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: Add a Change Owner group https://review.openstack.org/65133 | 00:28 |
| harlowja | fungi interestingly, if u look at the kazoo library, which is the main zookeeper library for python, there unit tests basically only run against a real zookeeper server, i asked them why, they said, thats the best way, so then i created zake to have a fake client to avoid the need to connect to a real zookeeper for tests... so ya, lol | 00:28 |
| fungi | anteaya: sounds like you're in good company | 00:28 |
| fungi | harlowja: and great name ;) | 00:29 |
| clarkb | and jenkins03 has started running a job | 00:29 |
| harlowja | fungi ya jd__ suggested that i think :) | 00:29 |
| clarkb | fungi: how do you feel about putting jenkins02 in shutdown mode so that it can get upgraded next? | 00:29 |
| clarkb | fungi: should I go ahead and do that now? | 00:29 |
| *** lcheng has quit IRC | 00:30 | |
| anteaya | fungi: I am, she is very smart: http://www.saltstack.com/salt-blog/first-ever-saltstack-certified-engineers | 00:30 |
| fungi | clarkb: it doesn't seem too dangerous. if jobs back up waiting for centos6 slaves, they'll complete quickly enough after the restart | 00:30 |
| * fungi favors this | 00:30 | |
| clarkb | fungi: ok putting it in shutdown mode now | 00:30 |
| clarkb | then 01 and jenkins.o.o will happen when I upgrade them. I have a dentist appoitment tomorrow afternoon maybe I an do those before that | 00:31 |
| *** ryanpetrello has joined #openstack-infra | 00:31 | |
| fungi | sounds good | 00:32 |
| fungi | also, dentists like to steal and hoard teeth--take precautions | 00:33 |
| *** hogepodge has quit IRC | 00:34 | |
| *** gokrokve_ has quit IRC | 00:36 | |
| *** gokrokve has joined #openstack-infra | 00:37 | |
| *** sarob has quit IRC | 00:38 | |
| *** sarob has joined #openstack-infra | 00:38 | |
| fungi | hpcloud-az2 has an updated devstack-precise image for over an hour now, but is still trying to build a new bare-precise image | 00:40 |
| clarkb | poor thing | 00:40 |
| openstackgerrit | A change was merged to openstack-infra/config: Pull in the zookeeperd package instead of the zookeeper package https://review.openstack.org/70022 | 00:41 |
| *** gokrokve has quit IRC | 00:41 | |
| *** ryanpetrello has quit IRC | 00:42 | |
| *** thuc has joined #openstack-infra | 00:42 | |
| *** wenlock has quit IRC | 00:43 | |
| *** thuc has quit IRC | 00:43 | |
| *** thuc has joined #openstack-infra | 00:44 | |
| fungi | stevebaker: you should be safe to try again now. all ready devstack-precise nodes are built from updated images at this point | 00:46 |
| *** Sukhdev has joined #openstack-infra | 00:46 | |
| stevebaker | fungi: awesome, I shall give it a crack | 00:47 |
| *** pcrews has quit IRC | 00:49 | |
| *** zehicle_at_dell has joined #openstack-infra | 00:50 | |
| *** nati_uen_ has joined #openstack-infra | 00:51 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add python-pip dependency to Puppet on Fedora https://review.openstack.org/70031 | 00:51 |
| *** olaph has quit IRC | 00:51 | |
| *** mfer has joined #openstack-infra | 00:52 | |
| clarkb | dims: https://review.openstack.org/#/c/69763/ new patch there fyi | 00:52 |
| *** sarob has quit IRC | 00:54 | |
| *** nati_ueno has quit IRC | 00:54 | |
| *** david-lyle has quit IRC | 00:55 | |
| clarkb | fungi: I think we are about half an hour from jenkins02 restart | 00:55 |
| *** flaper87 is now known as flaper87|afk | 01:00 | |
| *** CaptTofu has quit IRC | 01:00 | |
| *** dcramer_ has joined #openstack-infra | 01:01 | |
| *** julim has joined #openstack-infra | 01:04 | |
| *** nati_uen_ has quit IRC | 01:07 | |
| *** nati_ueno has joined #openstack-infra | 01:07 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add tab to only plot gate failures to status.o.o/elastic-recheck/ https://review.openstack.org/69735 | 01:08 |
| *** CaptTofu has joined #openstack-infra | 01:09 | |
| stevebaker | holy crap, heat-slow tests passed! | 01:12 |
| lifeless | stevebaker: \o/ | 01:12 |
| stevebaker | lifeless: ^5 | 01:12 |
| *** vjay has joined #openstack-infra | 01:12 | |
| clarkb | stevebaker: nice | 01:13 |
| stevebaker | I guess the next step is to move it out of experimental and make it non-voting on ... heat only? | 01:13 |
| fungi | stevebaker: many congratulations on your victory | 01:13 |
| openstackgerrit | A change was merged to openstack-infra/config: Update Storyboard jobs https://review.openstack.org/69204 | 01:14 |
| stevebaker | heat tests may well be good at triggering general openstack regressions and races, so long term voting everywhere might be helpful | 01:15 |
| dims | clarkb, looking | 01:15 |
| clarkb | fungi: for the centos slaves that need to be added back in. DO we need to rebuild those or do anything managementy with them before adding them to jenkins? | 01:15 |
| *** mfer has quit IRC | 01:15 | |
| clarkb | I am not sure what state they were in when they were removed | 01:15 |
| clarkb | stevebaker: was the only thing the firewall? | 01:16 |
| stevebaker | clarkb: that was the last thing, there were also a couple of devstack port 8000 changes | 01:17 |
| *** krtaylor has joined #openstack-infra | 01:19 | |
| stevebaker | clarkb: but this was the first time I actually sat down and debugged the issue | 01:19 |
| *** gokrokve has joined #openstack-infra | 01:21 | |
| sdague | stevebaker: if it passes, please put it on the devstack & tempest & nova check queues non-voting as well | 01:21 |
| sdague | and if we can keep it working, we should make it vote | 01:21 |
| stevebaker | sdague: it would be my pleasure | 01:22 |
| sdague | stevebaker: and nice job! | 01:22 |
| jaypipes | clarkb, fungi, jeblair, mordred, Shrews: would love some feedback/criticism on a blog post I wrote that goes into how the OpenStack CI platform works. If you get a chance :) http://www.joinfu.com/2014/01/understanding-the-openstack-ci-system/ | 01:22 |
| clarkb | jaypipes: fwiw jenkins event stream is really gearman worker respones (not sure if you want to get into that level of detail) | 01:24 |
| jaypipes | clarkb: ah, good point. noted :) I might add that in as a footnote. | 01:24 |
| *** krtaylor has quit IRC | 01:26 | |
| clarkb | jaypipes: skimmed it really quick. I like it | 01:26 |
| clarkb | jaypipes: one other thing I might point out is that one reason for archiving logs external to jenkins is we have 8 jenkins now so copying to a central location is handy | 01:26 |
| clarkb | but again details | 01:26 |
| clarkb | two jobs left on jenkins02, I am impatient | 01:27 |
| *** sandywalsh has quit IRC | 01:29 | |
| *** rhsu1 has quit IRC | 01:29 | |
| *** thuc has quit IRC | 01:29 | |
| *** thuc has joined #openstack-infra | 01:30 | |
| fungi | clarkb: the slaves had been offlined automatically by jenkins citing connectivity issues and could probably stand a reboot prior to readding | 01:30 |
| *** ryanpetrello has joined #openstack-infra | 01:31 | |
| clarkb | fungi: ok | 01:32 |
| fungi | i had deleted them from the masters prior to restart so they wouldn't get onlined at start | 01:32 |
| *** gokrokve has quit IRC | 01:33 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Copy libvirt log file after tempest run https://review.openstack.org/61892 | 01:33 |
| jog0 | clarkb: hopefully https://review.openstack.org/#/c/57577/2 should fix keystone | 01:33 |
| *** gokrokve has joined #openstack-infra | 01:33 | |
| *** thuc has quit IRC | 01:33 | |
| clarkb | jog0: I am not convinced it will | 01:35 |
| clarkb | you are basically saying keystone shoulnd't bind to 0.0.0.0 anymore | 01:35 |
| *** CaptTofu has quit IRC | 01:36 | |
| clarkb | but that is how 99% of deployments do it I bet | 01:36 |
| *** ryanpetrello has quit IRC | 01:36 | |
| jog0 | clarkb: baby steps | 01:36 |
| clarkb | hell 0.0.0.0 is the default | 01:36 |
| jog0 | clarkb: so there are two related issues | 01:36 |
| clarkb | jog0: sure, but this doesn't fix the problem | 01:37 |
| clarkb | it ignores it | 01:37 |
| jog0 | 1) we are hitting this in gate, 2) the default port sucks | 01:37 |
| clarkb | (same as changing the port honestly. The proper fix is to move to apache wsgi) | 01:37 |
| fungi | we should fix it in the distros ;) | 01:37 |
| jog0 | I am happy to just fix 1 | 01:37 |
| jog0 | clarkb: people are working on moving devstack to apache wsgi | 01:37 |
| jog0 | which solves part 2 | 01:37 |
| *** gokrokve has quit IRC | 01:38 | |
| jog0 | so this is a short term fix for problem 1, to buy is time to get a good solution for 2 (which will fix 1) as well | 01:38 |
| clarkb | I get cranky when the answer is to not use wildcard bindings | 01:38 |
| clarkb | that is how you deploy stuff... | 01:38 |
| *** nosnos has joined #openstack-infra | 01:38 | |
| clarkb | unless I am deploying a router | 01:39 |
| jog0 | clarkb: I don't blame you -- although nova is saying the short term answer is | 01:39 |
| jog0 | which is why they didn't want to change there own default | 01:39 |
| clarkb | fungi: restarting jenkins02 now | 01:39 |
| fungi | k | 01:39 |
| jog0 | I want them to just change the default port | 01:39 |
| fungi | to not the iana-assigned one? | 01:40 |
| *** yamahata has joined #openstack-infra | 01:40 | |
| clarkb | jenkins starting now | 01:40 |
| clarkb | jog0: I feel very strongly that a thing that has been missed in all this despite my pleading is that we are writing software for people to deploy | 01:41 |
| clarkb | if you defaults do not work then you are broken | 01:41 |
| *** VijayT has quit IRC | 01:41 | |
| clarkb | the keystone defaults do not work | 01:41 |
| clarkb | it binds 0.0.0.0:35357 by default | 01:41 |
| clarkb | and the default way to deploy it as documented by devstack isn't to use wsgi + apache | 01:41 |
| jog0 | so someone is working on changing that re: default documented by devstack is with apache + wsgi | 01:42 |
| clarkb | right morgainfainberg is | 01:42 |
| jog0 | clarkb: right, so until he finishes ... | 01:42 |
| clarkb | until he finishes we are broken | 01:42 |
| clarkb | the devstack change above just masks it | 01:43 |
| clarkb | jenkins02 is running a bunch of jobs. seems happy | 01:43 |
| jog0 | yes, so perhaps we get that bug marked as critical and have this patch be a related-bug not a fix | 01:43 |
| *** mestery_ has joined #openstack-infra | 01:45 | |
| clarkb | jog0: I won't stop that change going through, but I am a proponent of not masking problems :) | 01:45 |
| clarkb | mostly because sqlalchemy | 01:45 |
| jog0 | clarkb: fair enough | 01:45 |
| jog0 | what about sqlalchemy? | 01:46 |
| clarkb | jog0: we are pinned to 0.7 and are two point releases behind | 01:46 |
| jog0 | clarkb: I wouldn't even mind if you -1ed that patch or -0 it | 01:46 |
| jog0 | clarkb: oh didn't even realize | 01:46 |
| clarkb | because instead of fixing problems when they come up we mask them and move on | 01:46 |
| * jog0 isn't touching that one with a 10 foot pole | 01:46 | |
| jog0 | not sure if people we know we are pinned | 01:46 |
| *** markwash has joined #openstack-infra | 01:48 | |
| *** mestery has quit IRC | 01:48 | |
| *** oubiwann_ has quit IRC | 01:51 | |
| *** oubiwann_ has joined #openstack-infra | 01:52 | |
| *** melwitt has quit IRC | 01:52 | |
| clarkb | jog0: people know. it comes up on the ML periodically | 01:53 |
| clarkb | apparently folks like ubuntu have upstream patches to make it work with newer sqlalchemy which just makes the problem worse | 01:53 |
| *** kraman has quit IRC | 01:54 | |
| fungi | the distro packagers make sure nobody forgets (and i applaud their diligence) | 01:54 |
| clarkb | fungi: well it would be in their interest to push their fixes back :) | 01:54 |
| fungi | yeah, i would more loudly applaud more thorough diligence there ;) | 01:55 |
| clarkb | fungi: jenkins02 is happy according to my spot checks | 01:55 |
| fungi | excellent | 01:55 |
| clarkb | fungi: do you have a list of centos slaves that are off handy? | 01:55 |
| *** mrodden has quit IRC | 01:55 | |
| clarkb | I guess I just look for gaps in the lists on 01 and 02 | 01:56 |
| fungi | mmm, i can dig up the list of deletions i mentioned on the channel last time i did them, but yeah whatever's missing from 1-14 | 01:56 |
| clarkb | looks like all of the even nodes are there | 01:56 |
| fungi | i keep wanting to find time to try setting up a bare-centos6 nodepool type | 01:56 |
| dims | anyone see the glance gate fail? "FAIL: glance.tests.unit.v2.test_tasks_resource.TestTasksController.test_index_with_sort_key" | 01:57 |
| clarkb | fungi: I only see centos6-1 as missing | 01:57 |
| *** malini_afk has quit IRC | 01:57 | |
| *** CaptTofu has joined #openstack-infra | 01:57 | |
| fungi | i think the precise slaves had more trouble than the centos6 slaves as stability goes | 01:57 |
| clarkb | 01 has 3,5,7,9,11,13 and 02 has 2,4,6,8,10,12,14 | 01:57 |
| clarkb | rebooting centos6-1 now | 01:58 |
| *** malini_afk has joined #openstack-infra | 01:59 | |
| *** malini_afk is now known as malini | 01:59 | |
| clarkb | dims: that isn't familiar to me | 01:59 |
| clarkb | dims: though I do know that glance has some functional tests in its unittest suite that are prone to silly errors like not being able to bind to sockets because it is already in use | 01:59 |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/config: tempest-dsvm-neutron-heat-slow graduates from experimental https://review.openstack.org/70037 | 01:59 |
| *** sandywalsh has joined #openstack-infra | 02:01 | |
| *** gokrokve has joined #openstack-infra | 02:01 | |
| clarkb | fungi: adding centos6-1 back to 01 | 02:02 |
| fungi | clarkb: thanks! | 02:04 |
| *** ryanpetrello has joined #openstack-infra | 02:04 | |
| *** rhsu has joined #openstack-infra | 02:05 | |
| clarkb | zaro: dims: I updated that zmq event publisher change for hopefully the last time. I cleaned up unneeded imports | 02:06 |
| clarkb | fungi: I am going to walk home now be afk for a bit | 02:06 |
| fungi | clarkb: okay... i'm becoming rapidly afk as the evening progresses | 02:07 |
| fungi | but still more or less around | 02:07 |
| *** pcrews has joined #openstack-infra | 02:07 | |
| jog0 | dims aer you working on a fingerprint for 1274365 | 02:08 |
| jog0 | I was about to push one | 02:08 |
| jog0 | message:"glance/tests/unit/v2/test_tasks_resource.py" AND message:"test_index_with_sort_key" | 02:08 |
| dims | jog0, lgtm | 02:09 |
| jog0 | message:"glance/tests/unit/v2/test_tasks_resource.py" AND message:"test_index_with_sort_key" AND filename:"console.html" AND project:"openstack/glance" | 02:09 |
| anteaya | pleia2: I talked with Ryan_Lane, here is here, he says wikimedia is currently upgrading their puppet to 3 | 02:10 |
| dims | jog0, even better | 02:10 |
| anteaya | he says you can run a linter to evaluate potential issues | 02:10 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1274365 https://review.openstack.org/70039 | 02:10 |
| anteaya | he will talk to you more about it himself | 02:11 |
| anteaya | the tl;dr is he feels the process should be both possible and relatively painless for us | 02:12 |
| *** ryanpetrello has quit IRC | 02:12 | |
| *** gokrokve has quit IRC | 02:13 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1274135 https://review.openstack.org/70041 | 02:14 |
| jog0 | fungi: this ones for you ^ | 02:14 |
| * jog0 is trying to bump our classifcation rate up over 90% | 02:14 | |
| fungi | jog0: lgtm | 02:16 |
| jog0 | fungi: cool | 02:17 |
| jog0 | so WTF: http://git.openstack.org/cgit/openstack/ceilometer/tree/test-requirements.txt#n8 | 02:17 |
| *** VijayT has joined #openstack-infra | 02:17 | |
| jog0 | ceilomer is importing nova when testing in unit tests | 02:17 |
| jog0 | meaning we can randmly break them in gate | 02:18 |
| jog0 | wee | 02:18 |
| fungi | jog0: i also think new pip may have issues handling that | 02:18 |
| fungi | server projects really shouldn't be declaring test dependencies on one another | 02:18 |
| jog0 | fungi: so seeing ceilometer fail in gate http://status.openstack.org/zuul/ | 02:19 |
| jog0 | because they are using non contractual APIs in nova | 02:19 |
| fungi | but fixing that means moving those tests to integration testing, or building mock servers | 02:19 |
| fungi | yes, that also is bad mojo | 02:19 |
| jog0 | very bad | 02:19 |
| *** vkozhukalov has joined #openstack-infra | 02:20 | |
| ewindisch | beautiful, beautiful red-lights everywhere - http://docker-ci.dostack.org/job/docker-tempest-full/ | 02:20 |
| *** ok_delta has joined #openstack-infra | 02:21 | |
| openstackgerrit | A change was merged to openstack/requirements: Bump stevedore to 0.14 https://review.openstack.org/69496 | 02:22 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1274135 https://review.openstack.org/70041 | 02:23 |
| *** CaptTofu has quit IRC | 02:32 | |
| *** yamahata has quit IRC | 02:32 | |
| jog0 | fungi: ML thread started lets see where it goes | 02:34 |
| jog0 | after a quick scan looks like only ceilometer is doing this | 02:36 |
| *** yamahata has joined #openstack-infra | 02:37 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1274365 https://review.openstack.org/70039 | 02:38 |
| *** starmer has joined #openstack-infra | 02:39 | |
| *** markwash has quit IRC | 02:45 | |
| *** nati_uen_ has joined #openstack-infra | 02:45 | |
| clarkb | woot registered for the summit | 02:46 |
| *** Sukhdev has quit IRC | 02:46 | |
| *** salv-orlando has quit IRC | 02:46 | |
| *** jcooley_ has quit IRC | 02:47 | |
| *** nati_ueno has quit IRC | 02:48 | |
| jog0 | anyone see this http://logs.openstack.org/28/69528/2/gate/gate-tempest-python27/b95c3bc/console.html | 02:49 |
| clarkb | jog0: I think that was this mornings tox trouble | 02:50 |
| *** nati_uen_ has quit IRC | 02:50 | |
| jog0 | clarkb: ack | 02:50 |
| clarkb | fungi fixed it by pinning tox | 02:50 |
| jog0 | classification rate of 92% isn't too bad | 02:51 |
| *** CaptTofu has joined #openstack-infra | 02:52 | |
| jog0 | btw are the queue time numbers accurate or was there gate reset | 02:53 |
| jog0 | err zuul restart | 02:53 |
| jog0 | clarkb: does it make sense to snipe https://review.openstack.org/#/c/67850/ | 02:53 |
| jog0 | being ceilometer unit tests are broken | 02:54 |
| *** shivharis has quit IRC | 02:55 | |
| clarkb | jog0: go for it | 02:56 |
| *** sarob has joined #openstack-infra | 02:57 | |
| fungi | jog0: those are accurate at this point afaik | 02:58 |
| jog0 | fungi: what are accurate? by brain is half off | 02:59 |
| fungi | jog0: queue time numbers you just asked about | 02:59 |
| jog0 | ahh right, in that case not too bad | 02:59 |
| fungi | clarkb: did you run manage-projects after approving https://review.openstack.org/62069 (i don't see the project or groups for it in gerrit yet)? | 03:00 |
| * jog0 just sniped his firstpatch | 03:00 | |
| clarkb | fungi: mordred said he would | 03:00 |
| clarkb | I didn't | 03:01 |
| fungi | clarkb: ahh, okay | 03:01 |
| *** gokrokve has joined #openstack-infra | 03:01 | |
| fungi | i'll run it now i guess | 03:01 |
| jog0 | not bad 3 patches an hour over last 6 hours | 03:02 |
| jog0 | in o/o | 03:03 |
| jog0 | anyway time to turn brain completely off | 03:03 |
| *** sarob has quit IRC | 03:04 | |
| *** emagana has quit IRC | 03:05 | |
| *** pcrews has quit IRC | 03:12 | |
| *** krtaylor has joined #openstack-infra | 03:12 | |
| *** amcrn has quit IRC | 03:14 | |
| *** dcramer_ has quit IRC | 03:14 | |
| *** markwash has joined #openstack-infra | 03:19 | |
| *** CaptTofu has quit IRC | 03:19 | |
| *** talluri_ has joined #openstack-infra | 03:20 | |
| *** markmcclain has quit IRC | 03:21 | |
| *** mestery_ is now known as mestery | 03:21 | |
| *** markmcclain has joined #openstack-infra | 03:21 | |
| *** talluri has quit IRC | 03:22 | |
| *** david-lyle has joined #openstack-infra | 03:23 | |
| *** lcheng has joined #openstack-infra | 03:25 | |
| *** coolsvap_away has quit IRC | 03:27 | |
| *** jcooley_ has joined #openstack-infra | 03:27 | |
| *** matsuhashi has quit IRC | 03:29 | |
| *** rcleere has joined #openstack-infra | 03:29 | |
| *** jcooley_ has quit IRC | 03:34 | |
| *** jcooley_ has joined #openstack-infra | 03:35 | |
| *** jp_at_hp has quit IRC | 03:35 | |
| *** packet has quit IRC | 03:36 | |
| *** chandankumar_ has joined #openstack-infra | 03:36 | |
| *** markwash has quit IRC | 03:38 | |
| *** SnowDust has joined #openstack-infra | 03:39 | |
| clarkb | I am a bit confused about the ml2 driver in neutron? isn't that completely open source? why are they doing third party testing for it? what am I missing | 03:39 |
| *** chandankumar_ has quit IRC | 03:40 | |
| *** jcooley_ has quit IRC | 03:40 | |
| clarkb | I feel like by requring third party testing we have gone from one extreme to another. From no tseting to testing all the interesting stuff external to the project | 03:40 |
| *** pcrews has joined #openstack-infra | 03:41 | |
| jesusaurus | clarkb: does the ml2 driver not need "exotic" hardware? | 03:41 |
| *** chandankumar_ has joined #openstack-infra | 03:42 | |
| clarkb | jesusaurus: I should rephrase, I think ml2 is a driver interface and that they want to use it with opendaylight | 03:42 |
| clarkb | aiui opendaylight + ml2 is all open source and doesn't need hardware | 03:42 |
| jesusaurus | oh cool | 03:43 |
| *** leizhang has joined #openstack-infra | 03:44 | |
| *** markwash has joined #openstack-infra | 03:45 | |
| *** chandankumar_ has quit IRC | 03:46 | |
| *** chandankumar__ has joined #openstack-infra | 03:47 | |
| *** praneshp has quit IRC | 03:47 | |
| *** praneshp has joined #openstack-infra | 03:48 | |
| *** chandankumar__ has left #openstack-infra | 03:50 | |
| harlowja | hmmm, having a issue, not sure if its transient, has anyone seen 'Non-zero exit code (2) from test listing.', this causes testr to not have any results, only seems to be happening on python3.3 (but not on my local machine) | 03:51 |
| harlowja | currently seeing @ http://logs.openstack.org/07/69507/13/check/gate-taskflow-python33/2cedd23/console.html | 03:51 |
| harlowja | something up with testr maybe? | 03:51 |
| harlowja | *maybe someone else seen something similar :-/ | 03:51 |
| clarkb | import failures usually | 03:55 |
| harlowja | hmmm | 03:55 |
| openstackgerrit | A change was merged to openstack/requirements: Add futures library to global requirements https://review.openstack.org/66349 | 03:55 |
| harlowja | wonder which import | 03:56 |
| clarkb | it should tell you | 03:57 |
| *** wenlock has joined #openstack-infra | 03:57 | |
| *** olaph has joined #openstack-infra | 03:57 | |
| *** Hefeweizen has joined #openstack-infra | 03:57 | |
| *** dcramer_ has joined #openstack-infra | 03:58 | |
| harlowja | hmmm | 03:59 |
| *** dstanek has quit IRC | 04:00 | |
| *** dstanek has joined #openstack-infra | 04:00 | |
| *** SnowDust has quit IRC | 04:00 | |
| *** sarob has joined #openstack-infra | 04:01 | |
| *** leizhang has quit IRC | 04:01 | |
| *** praneshp has quit IRC | 04:01 | |
| *** markwash has quit IRC | 04:02 | |
| *** matsuhashi has joined #openstack-infra | 04:02 | |
| *** talluri has joined #openstack-infra | 04:04 | |
| *** mfer has joined #openstack-infra | 04:06 | |
| *** sarob has quit IRC | 04:06 | |
| *** talluri_ has quit IRC | 04:06 | |
| *** ArxCruz has quit IRC | 04:10 | |
| *** jcooley_ has joined #openstack-infra | 04:10 | |
| harlowja | ya, totally weird | 04:16 |
| *** jcooley_ has quit IRC | 04:16 | |
| *** jcooley_ has joined #openstack-infra | 04:17 | |
| harlowja | ah, i see | 04:18 |
| harlowja | thx clarkb , darn python 3! | 04:18 |
| harlowja | ImportError: No module named 'thread' | 04:18 |
| harlowja | why they move all this stuff around, lol | 04:18 |
| *** talluri has quit IRC | 04:19 | |
| *** mfer has quit IRC | 04:21 | |
| *** jcooley_ has quit IRC | 04:21 | |
| *** jcooley_ has joined #openstack-infra | 04:22 | |
| *** kraman has joined #openstack-infra | 04:22 | |
| *** lcheng has quit IRC | 04:30 | |
| *** markmcclain has quit IRC | 04:34 | |
| *** lcheng has joined #openstack-infra | 04:35 | |
| zaro | clarkb: would like your opinion on this https://review.openstack.org/#/c/61321 | 04:37 |
| clarkb | zaro if jeblair wants duplication I can live with that | 04:40 |
| zaro | clarkb: well, i was wondering wether you think it's better as is? | 04:46 |
| clarkb | zaro: I like less duplication | 04:48 |
| clarkb | but dont feel strongly about it | 04:48 |
| zaro | clarkb: me too + not having to push another change unless it's necessary. | 04:49 |
| zaro | clarkb: answered your question about buck in https://review.openstack.org/#/c/70014 | 04:50 |
| zaro | the dependent change has been around for a while. are we waiting for something to push that thru? | 04:51 |
| clarkb | no just need more time to review | 04:52 |
| StevenK | clarkb: Tautology? s/ to review// ? | 04:53 |
| clarkb | :) | 04:53 |
| *** praneshp has joined #openstack-infra | 04:58 | |
| *** sarob has joined #openstack-infra | 05:04 | |
| *** sarob has quit IRC | 05:11 | |
| *** matsuhashi has quit IRC | 05:14 | |
| *** talluri has joined #openstack-infra | 05:14 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 05:15 | |
| *** gokrokve has quit IRC | 05:16 | |
| *** gokrokve has joined #openstack-infra | 05:17 | |
| *** matsuhas_ has joined #openstack-infra | 05:17 | |
| *** coolsvap has joined #openstack-infra | 05:19 | |
| *** thuc_ has joined #openstack-infra | 05:21 | |
| *** gokrokve has quit IRC | 05:21 | |
| *** eharney has quit IRC | 05:24 | |
| *** rwsu has quit IRC | 05:27 | |
| *** arborism has joined #openstack-infra | 05:32 | |
| *** rhsu has quit IRC | 05:33 | |
| *** rhsu has joined #openstack-infra | 05:34 | |
| *** arborism is now known as amcrn | 05:34 | |
| *** rhsu has quit IRC | 05:36 | |
| *** gyee has quit IRC | 05:43 | |
| *** nicedice has quit IRC | 05:44 | |
| *** rcleere has quit IRC | 05:45 | |
| *** DinaBelova_ is now known as DinaBelova | 06:01 | |
| *** DinaBelova is now known as DinaBelova_ | 06:04 | |
| *** dstufft has quit IRC | 06:05 | |
| *** sarob has joined #openstack-infra | 06:07 | |
| *** denis_makogon_ has joined #openstack-infra | 06:11 | |
| *** sarob has quit IRC | 06:11 | |
| *** thuc has joined #openstack-infra | 06:13 | |
| *** dstufft has joined #openstack-infra | 06:16 | |
| *** thuc_ has quit IRC | 06:17 | |
| *** thuc has quit IRC | 06:17 | |
| *** vkozhukalov has quit IRC | 06:18 | |
| *** lcheng has quit IRC | 06:25 | |
| *** bookwar has joined #openstack-infra | 06:25 | |
| *** gokrokve has joined #openstack-infra | 06:27 | |
| *** gokrokve has quit IRC | 06:31 | |
| mordred | morning all | 06:35 |
| * Alex_Gaynor checks his clock | 06:36 | |
| mordred | clarkb: interesting question re: ml3 driver | 06:36 |
| mordred | Alex_Gaynor: blame belgium for any clock confusion | 06:37 |
| Alex_Gaynor | mordred: I'll hapilly blame belgium for pretty much anything! | 06:37 |
| *** DinaBelova_ is now known as DinaBelova | 06:40 | |
| *** pblaho has joined #openstack-infra | 06:42 | |
| *** svij has left #openstack-infra | 06:42 | |
| *** pblaho has quit IRC | 06:42 | |
| *** praneshp_ has joined #openstack-infra | 06:42 | |
| *** pblaho has joined #openstack-infra | 06:42 | |
| *** dstanek has quit IRC | 06:43 | |
| *** nati_ueno has joined #openstack-infra | 06:43 | |
| *** praneshp has quit IRC | 06:44 | |
| *** praneshp_ is now known as praneshp | 06:44 | |
| *** kraman has quit IRC | 06:45 | |
| *** Sukhdev has joined #openstack-infra | 06:47 | |
| *** nati_ueno has quit IRC | 06:48 | |
| *** AaronGr is now known as AaronGr_zzz | 06:49 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 06:52 | |
| *** ken1ohmichi has quit IRC | 06:58 | |
| *** jcooley_ has quit IRC | 06:59 | |
| *** DinaBelova is now known as DinaBelova_ | 06:59 | |
| *** jcooley_ has joined #openstack-infra | 07:00 | |
| *** sarob has joined #openstack-infra | 07:08 | |
| *** doude_ has left #openstack-infra | 07:10 | |
| clarkb | mordred: do you know the answer? | 07:13 |
| *** praneshp has quit IRC | 07:13 | |
| *** sarob has quit IRC | 07:15 | |
| *** Sukhdev has quit IRC | 07:15 | |
| *** luisg has quit IRC | 07:15 | |
| *** nati_ueno has joined #openstack-infra | 07:15 | |
| *** jcooley_ has quit IRC | 07:16 | |
| *** luisg has joined #openstack-infra | 07:24 | |
| *** kiall has quit IRC | 07:25 | |
| *** GheRivero has quit IRC | 07:25 | |
| *** nati_ueno has quit IRC | 07:26 | |
| *** Shrews has quit IRC | 07:28 | |
| *** mrda is now known as mrda_away | 07:32 | |
| *** yolanda_ has joined #openstack-infra | 07:32 | |
| openstackgerrit | Peter Liljenberg proposed a change to openstack-infra/jenkins-job-builder: Added support for Jenkins Campfire plugin https://review.openstack.org/69315 | 07:39 |
| *** emagana has joined #openstack-infra | 07:40 | |
| *** wenlock has quit IRC | 07:42 | |
| *** ok_delta has quit IRC | 07:44 | |
| *** e0ne has joined #openstack-infra | 07:49 | |
| *** flaper87|afk is now known as flaper87 | 07:49 | |
| *** jcooley_ has joined #openstack-infra | 07:51 | |
| *** vkozhukalov has joined #openstack-infra | 07:57 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 08:02 | |
| *** jamielennox is now known as jamielennox|away | 08:03 | |
| *** jcooley_ has quit IRC | 08:05 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add storyboard-dev node with krotscheck and ruhe on it https://review.openstack.org/69516 | 08:07 |
| *** vkozhukalov has quit IRC | 08:10 | |
| *** sarob has joined #openstack-infra | 08:11 | |
| *** ttx has quit IRC | 08:12 | |
| *** che-arne has quit IRC | 08:14 | |
| *** hashar has joined #openstack-infra | 08:15 | |
| *** sarob has quit IRC | 08:15 | |
| openstackgerrit | A change was merged to openstack/requirements: Update oauthlib requirement to at least 0.6 https://review.openstack.org/67900 | 08:16 |
| *** odyssey4me has joined #openstack-infra | 08:17 | |
| *** harlowja is now known as harlowja_away | 08:18 | |
| mordred | clarkb: what answer? | 08:19 |
| *** vkozhukalov has joined #openstack-infra | 08:19 | |
| *** jpich has joined #openstack-infra | 08:20 | |
| *** hashar has quit IRC | 08:21 | |
| *** fbo_away is now known as fbo | 08:22 | |
| *** hashar has joined #openstack-infra | 08:23 | |
| *** e0ne_ has joined #openstack-infra | 08:23 | |
| *** emagana has quit IRC | 08:25 | |
| *** e0ne has quit IRC | 08:27 | |
| *** mattoliverau has quit IRC | 08:27 | |
| *** VijayT has quit IRC | 08:28 | |
| *** david-lyle_ has quit IRC | 08:29 | |
| hashar | good morning | 08:29 |
| *** lyle has joined #openstack-infra | 08:29 | |
| *** nosnos has quit IRC | 08:39 | |
| *** nosnos has joined #openstack-infra | 08:39 | |
| *** vkozhukalov has quit IRC | 08:43 | |
| *** markmc has joined #openstack-infra | 08:43 | |
| *** jcooley_ has joined #openstack-infra | 08:46 | |
| *** ttx has joined #openstack-infra | 08:47 | |
| *** ttx has quit IRC | 08:47 | |
| *** ttx has joined #openstack-infra | 08:47 | |
| *** ttx has quit IRC | 08:50 | |
| *** ttx has joined #openstack-infra | 08:50 | |
| *** ttx has quit IRC | 08:50 | |
| *** ttx has joined #openstack-infra | 08:50 | |
| *** ttx has quit IRC | 08:51 | |
| *** ttx has joined #openstack-infra | 08:51 | |
| *** ttx has quit IRC | 08:51 | |
| *** ttx has joined #openstack-infra | 08:51 | |
| *** ttx has quit IRC | 08:56 | |
| *** ttx has joined #openstack-infra | 08:56 | |
| *** ttx has quit IRC | 08:56 | |
| *** ttx has joined #openstack-infra | 08:56 | |
| *** vkozhukalov has joined #openstack-infra | 08:58 | |
| *** denis_makogon_ has quit IRC | 09:09 | |
| *** yassine has joined #openstack-infra | 09:10 | |
| *** johnthetubaguy has joined #openstack-infra | 09:11 | |
| *** AlexF_ has joined #openstack-infra | 09:12 | |
| *** jpich has quit IRC | 09:12 | |
| *** johnthetubaguy has quit IRC | 09:13 | |
| *** GheRiver1 has joined #openstack-infra | 09:14 | |
| *** johnthetubaguy has joined #openstack-infra | 09:14 | |
| *** GheRiver1 is now known as GheRivero | 09:16 | |
| *** jcooley_ has quit IRC | 09:16 | |
| *** derekh has joined #openstack-infra | 09:26 | |
| *** andreaf has joined #openstack-infra | 09:26 | |
| *** rossella_s has joined #openstack-infra | 09:28 | |
| *** matsuhas_ has quit IRC | 09:33 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 09:36 | |
| *** jooools has joined #openstack-infra | 09:38 | |
| *** matsuhashi has joined #openstack-infra | 09:39 | |
| *** DinaBelova_ is now known as DinaBelova | 09:40 | |
| *** yamahata has quit IRC | 09:40 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 09:40 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 09:40 | |
| *** DinaBelova is now known as DinaBelova_ | 09:46 | |
| *** nosnos has quit IRC | 09:46 | |
| *** nosnos has joined #openstack-infra | 09:46 | |
| *** nosnos has quit IRC | 09:47 | |
| *** matsuhashi has quit IRC | 09:50 | |
| *** shardy is now known as shardy_afk | 09:53 | |
| *** che-arne has joined #openstack-infra | 09:56 | |
| *** therve has joined #openstack-infra | 10:02 | |
| *** kiall has joined #openstack-infra | 10:03 | |
| *** e0ne_ has quit IRC | 10:03 | |
| mordred | https://etherpad.openstack.org/p/StoryboardMeetup | 10:04 |
| mordred | btw - for anyone who wants to track what we're doing | 10:04 |
| *** e0ne has joined #openstack-infra | 10:05 | |
| hashar | \O/ | 10:05 |
| hashar | such a pity I am not there :D | 10:06 |
| hashar | will probably send you our bug master Andre Kappler. He should show up at fosdem this weekend | 10:06 |
| hashar | would be nice to list out the reason for starting storyboard, though they are probably obvious to you guys | 10:07 |
| *** jcooley_ has joined #openstack-infra | 10:09 | |
| *** masayukig has joined #openstack-infra | 10:09 | |
| *** andre__ has joined #openstack-infra | 10:09 | |
| *** sarob has joined #openstack-infra | 10:12 | |
| *** jcooley_ has quit IRC | 10:14 | |
| *** jp_at_hp has joined #openstack-infra | 10:14 | |
| *** DinaBelova_ is now known as DinaBelova | 10:15 | |
| *** sarob has quit IRC | 10:16 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 10:19 | |
| *** GheRivero has quit IRC | 10:20 | |
| *** kiall has quit IRC | 10:21 | |
| *** hashar has quit IRC | 10:26 | |
| *** hashar has joined #openstack-infra | 10:26 | |
| *** coolsvap has quit IRC | 10:28 | |
| *** AlexF_ has quit IRC | 10:37 | |
| *** e0ne_ has joined #openstack-infra | 10:39 | |
| hashar | jeblair: would it make sense to integrate code reviewing in Storyboard as well ? | 10:41 |
| hashar | Wikimedia might reconsider http://phabricator.org/ (php based, integrates tasks/trello/bugs/review) | 10:41 |
| *** yamahata has joined #openstack-infra | 10:41 | |
| *** DinaBelova is now known as DinaBelova_ | 10:42 | |
| *** e0ne has quit IRC | 10:42 | |
| *** vkozhukalov has quit IRC | 10:48 | |
| *** slong_ has quit IRC | 10:48 | |
| *** ArxCruz has joined #openstack-infra | 10:48 | |
| *** masayukig has quit IRC | 10:50 | |
| *** yassine has quit IRC | 10:54 | |
| *** boris-42 has quit IRC | 10:54 | |
| *** e0ne_ has quit IRC | 10:54 | |
| *** e0ne has joined #openstack-infra | 10:54 | |
| ProfFalken | hashar: Phapricator still has one of the best websites out there IMHO, the quotes are...inspiring :D | 10:55 |
| chmouel | i think gerrit does a good job for code reviewing already, | 10:55 |
| hashar | at Wikimedia I guess we are more or less in the same position as you | 10:56 |
| hashar | though our main issues is having too many different tools | 10:56 |
| hashar | and lacking a good one for project/product management | 10:56 |
| hashar | (aka wall of cards, such as Trello / Mingle ) | 10:56 |
| hashar | + Bugzilla focus too much on bugs when we would need tasks / blueprints | 10:57 |
| *** dizquierdo has joined #openstack-infra | 10:57 | |
| *** BobBallAway is now known as BobBall | 10:57 | |
| sdague | hashar: so I think there was talk that storyboard *might* in future provide some gerrit dashboards, because with gerrit getting a more complete rest API, things like reviewday could be baked in | 10:58 |
| sdague | but on the actual code review engine, gerrit does a good job | 10:58 |
| *** derekh has quit IRC | 10:58 | |
| *** vkozhukalov has joined #openstack-infra | 11:00 | |
| *** ok_delta has joined #openstack-infra | 11:00 | |
| sdague | where as launchpad has legitimately become too difficult to use. Lots of us are in the habbit of avoiding lp artifacts for cross project things, because if you get too many people subscribed to an artifact you can't update it any more | 11:00 |
| sdague | which is exactly the wrong thing | 11:01 |
| *** jcooley_ has joined #openstack-infra | 11:02 | |
| *** max_lobur_afk is now known as max_lobur | 11:05 | |
| *** dizquierdo has quit IRC | 11:08 | |
| *** sarob has joined #openstack-infra | 11:13 | |
| hashar | sdague: you might want to give that context on some page :-] | 11:15 |
| hashar | maybe in the Problems with existing systems | 11:15 |
| hashar | would give context to help convert people to storyboard | 11:15 |
| *** jcooley_ has quit IRC | 11:16 | |
| *** DinaBelova_ is now known as DinaBelova | 11:17 | |
| *** reed has joined #openstack-infra | 11:18 | |
| *** sarob has quit IRC | 11:20 | |
| *** pblaho has quit IRC | 11:21 | |
| sdague | mordred: I'm throwing a few things in the etherpad, feel free to scrub and clean up | 11:22 |
| *** e0ne_ has joined #openstack-infra | 11:28 | |
| *** e0ne has quit IRC | 11:28 | |
| *** e0ne_ has quit IRC | 11:30 | |
| *** e0ne has joined #openstack-infra | 11:30 | |
| *** coolsvap has joined #openstack-infra | 11:31 | |
| *** dizquierdo has joined #openstack-infra | 11:32 | |
| *** ok_delta has quit IRC | 11:34 | |
| *** e0ne has quit IRC | 11:36 | |
| *** jpich has joined #openstack-infra | 11:38 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/config: XenServer installer script https://review.openstack.org/60581 | 11:41 |
| *** e0ne has joined #openstack-infra | 11:46 | |
| *** HenryG has quit IRC | 11:48 | |
| *** masayukig has joined #openstack-infra | 11:51 | |
| openstackgerrit | A change was merged to openstack/requirements: Ironic requires pyghmi>=0.5.8 https://review.openstack.org/64408 | 11:55 |
| *** masayukig has quit IRC | 11:55 | |
| *** oubiwann_ has quit IRC | 11:56 | |
| *** oubiwann_ has joined #openstack-infra | 11:57 | |
| *** e0ne has quit IRC | 11:58 | |
| *** e0ne has joined #openstack-infra | 11:58 | |
| openstackgerrit | Peter Liljenberg proposed a change to openstack-infra/jenkins-job-builder: Added support for Jenkins Campfire plugin https://review.openstack.org/69315 | 12:01 |
| *** ok_delta has joined #openstack-infra | 12:03 | |
| *** ok_delta has quit IRC | 12:04 | |
| *** yassine has joined #openstack-infra | 12:06 | |
| *** rfolco has joined #openstack-infra | 12:11 | |
| *** sarob has joined #openstack-infra | 12:16 | |
| *** lcheng has joined #openstack-infra | 12:19 | |
| *** coolsvap has quit IRC | 12:20 | |
| *** sarob has quit IRC | 12:22 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 12:22 | |
| *** david-lyle has quit IRC | 12:22 | |
| *** DinaBelova is now known as DinaBelova_ | 12:24 | |
| *** oanufriev has joined #openstack-infra | 12:24 | |
| *** afazekas has joined #openstack-infra | 12:27 | |
| *** masayukig has joined #openstack-infra | 12:31 | |
| *** afrittoli has joined #openstack-infra | 12:33 | |
| *** lcheng has quit IRC | 12:33 | |
| *** andreaf has quit IRC | 12:33 | |
| *** afrittoli has quit IRC | 12:34 | |
| *** andreaf has joined #openstack-infra | 12:34 | |
| *** masayukig has quit IRC | 12:36 | |
| *** smarcet has joined #openstack-infra | 12:39 | |
| *** CaptTofu has joined #openstack-infra | 12:42 | |
| *** che-arne has quit IRC | 12:42 | |
| *** bookwar has quit IRC | 12:44 | |
| *** bookwar has joined #openstack-infra | 12:45 | |
| *** e0ne_ has joined #openstack-infra | 12:49 | |
| *** e0ne has quit IRC | 12:49 | |
| *** jcooley_ has joined #openstack-infra | 12:51 | |
| *** katyafervent_awa is now known as katyafervent | 12:52 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer bug 1274490 https://review.openstack.org/70110 | 12:53 |
| *** mfink has quit IRC | 12:55 | |
| *** masayukig has joined #openstack-infra | 12:55 | |
| *** weshay has joined #openstack-infra | 12:55 | |
| *** HenryG has joined #openstack-infra | 13:03 | |
| *** vkozhukalov has quit IRC | 13:03 | |
| *** eharney has joined #openstack-infra | 13:05 | |
| *** GheRivero has joined #openstack-infra | 13:09 | |
| *** DinaBelova_ is now known as DinaBelova | 13:10 | |
| *** dstanek has joined #openstack-infra | 13:13 | |
| *** e0ne_ has quit IRC | 13:14 | |
| *** e0ne has joined #openstack-infra | 13:15 | |
| *** rlandy has joined #openstack-infra | 13:16 | |
| *** e0ne_ has joined #openstack-infra | 13:16 | |
| *** yamahata has quit IRC | 13:16 | |
| *** vkozhukalov has joined #openstack-infra | 13:18 | |
| *** sarob has joined #openstack-infra | 13:19 | |
| *** e0ne has quit IRC | 13:19 | |
| *** e0ne_ has quit IRC | 13:20 | |
| *** yamahata has joined #openstack-infra | 13:20 | |
| *** e0ne has joined #openstack-infra | 13:20 | |
| *** jcooley_ has quit IRC | 13:25 | |
| *** sarob has quit IRC | 13:25 | |
| *** mestery_ has joined #openstack-infra | 13:35 | |
| *** mestery has quit IRC | 13:37 | |
| *** e0ne has quit IRC | 13:37 | |
| *** e0ne_ has joined #openstack-infra | 13:37 | |
| jeblair | hashar: yeah, we have discussed some things that are close to what sdague said; it will probably be a while until we get there (it is not a high priority) but we think we want to design it so we can eventually get to the point that review workflow can be exposed/facilitated with storyboard | 13:38 |
| *** Shrews has joined #openstack-infra | 13:38 | |
| hashar | jeblair: this way folks only have to use one tool :-] | 13:40 |
| hashar | (more or less) | 13:40 |
| *** CaptTofu has quit IRC | 13:40 | |
| hashar | hacking yaml with aliases :-D | 13:41 |
| hashar | http://paste.grml.org/plain/1741 | 13:41 |
| *** e0ne_ has quit IRC | 13:41 | |
| *** e0ne has joined #openstack-infra | 13:41 | |
| sdague | hashar: well if your goal is one tool only, we should implement all this in elisp as an emacs extension | 13:42 |
| *** e0ne has quit IRC | 13:42 | |
| * sdague ducsk | 13:42 | |
| *** e0ne has joined #openstack-infra | 13:42 | |
| *** temujin has joined #openstack-infra | 13:43 | |
| *** temujin has quit IRC | 13:44 | |
| *** e0ne_ has joined #openstack-infra | 13:44 | |
| *** e0ne has quit IRC | 13:44 | |
| *** oanufriev` has joined #openstack-infra | 13:45 | |
| *** oanufriev has quit IRC | 13:45 | |
| *** e0ne has joined #openstack-infra | 13:46 | |
| *** e0ne_ has quit IRC | 13:46 | |
| *** yamahata has quit IRC | 13:46 | |
| *** yamahata has joined #openstack-infra | 13:47 | |
| * hashar throws a Litron of beer at sdague | 13:47 | |
| *** e0ne_ has joined #openstack-infra | 13:48 | |
| *** e0ne has quit IRC | 13:48 | |
| sdague | honestly, I don't buy the one tool perspective. Multiple tools are fine as long as they all have APIs so you can cross talk them when convenient | 13:48 |
| *** dizquierdo has quit IRC | 13:49 | |
| sdague | the usage of the tools and ui depends on context of what you are doing. Browsing vs. processing, for instance | 13:49 |
| *** thuc has joined #openstack-infra | 13:49 | |
| *** prad has joined #openstack-infra | 13:50 | |
| hashar | if you have API and plugins to glue your tools together, sure. | 13:51 |
| *** prad has quit IRC | 13:51 | |
| sdague | right, so the push should be towards get rid of all tools without APIs | 13:51 |
| sdague | and API here means web service API | 13:51 |
| *** prad has joined #openstack-infra | 13:51 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/storyboard: Be less negative about launchpad https://review.openstack.org/70116 | 13:51 |
| hashar | you still have to maintains the API connector though | 13:51 |
| sdague | hashar: yes, that's fine | 13:52 |
| *** markwash has joined #openstack-infra | 13:55 | |
| *** che-arne has joined #openstack-infra | 13:55 | |
| *** ryanpetrello has joined #openstack-infra | 13:56 | |
| *** lcheng has joined #openstack-infra | 13:57 | |
| * chmouel VOTE FOR EVERYTHING IN EMACS | 13:58 | |
| sdague | :) | 13:59 |
| hashar | as long as I can keep using the emacs-vim plugin, I am fine | 14:00 |
| *** hcc has joined #openstack-infra | 14:00 | |
| *** burt1 has joined #openstack-infra | 14:00 | |
| sdague | though if we do get a solid API on this, I could see writing an emacs mode to do the triage on it. I've been liking org-mode a lot recently | 14:00 |
| *** hcc is now known as hdd_ | 14:00 | |
| *** eharney has quit IRC | 14:01 | |
| *** dcramer_ has quit IRC | 14:02 | |
| hashar | is there anyway to disable pull requests on github ? | 14:03 |
| hashar | some folks proposed change to jjb there: https://github.com/openstack-infra/jenkins-job-builder/pulls | 14:03 |
| fungi | hashar: we have a pull-request closer bot, but it's broken at the moment. maybe today i'll have time to dig into the tracebacks it's throwing | 14:04 |
| *** mriedem has joined #openstack-infra | 14:04 | |
| hashar | \O/ | 14:04 |
| *** beagles is now known as beagles_brb | 14:04 | |
| *** kiall has joined #openstack-infra | 14:05 | |
| *** vkozhukalov has quit IRC | 14:06 | |
| *** yamahata has quit IRC | 14:09 | |
| fungi | http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/close_pull_requests.py http://paste.openstack.org/show/62168/ | 14:09 |
| *** yamahata has joined #openstack-infra | 14:09 | |
| fungi | for the redux | 14:09 |
| *** pblaho has joined #openstack-infra | 14:09 | |
| *** yamahata has quit IRC | 14:10 | |
| *** yamahata has joined #openstack-infra | 14:10 | |
| *** oubiwann_ has quit IRC | 14:14 | |
| *** dims has quit IRC | 14:15 | |
| *** dims has joined #openstack-infra | 14:15 | |
| *** eharney has joined #openstack-infra | 14:16 | |
| *** jcooley_ has joined #openstack-infra | 14:18 | |
| jeblair | fungi: hrm, that suggests that github is not returning openstack-dev in the list of orgs accessible by the user | 14:19 |
| *** vkozhukalov has joined #openstack-infra | 14:19 | |
| *** yamahata has quit IRC | 14:19 | |
| jeblair | possibly it is returning 0 orgs | 14:20 |
| jeblair | openstack-project-creator is an owner of that org | 14:20 |
| *** yamahata has joined #openstack-infra | 14:22 | |
| openstackgerrit | A change was merged to openstack/requirements: Fix testtools version and bump subunit to >=0.0.18 https://review.openstack.org/69956 | 14:22 |
| *** sarob has joined #openstack-infra | 14:22 | |
| *** dkliban has quit IRC | 14:23 | |
| *** mika has quit IRC | 14:23 | |
| *** mika has joined #openstack-infra | 14:24 | |
| *** mika has joined #openstack-infra | 14:24 | |
| *** CaptTofu has joined #openstack-infra | 14:24 | |
| *** CaptTofu has quit IRC | 14:25 | |
| *** max_lobur is now known as max_lobur_afk | 14:26 | |
| *** CaptTofu has joined #openstack-infra | 14:26 | |
| *** jcooley_ has quit IRC | 14:27 | |
| *** sarob has quit IRC | 14:28 | |
| *** dizquierdo has joined #openstack-infra | 14:28 | |
| *** kraman has joined #openstack-infra | 14:29 | |
| *** thuc has quit IRC | 14:29 | |
| *** thuc has joined #openstack-infra | 14:30 | |
| *** CaptTofu has quit IRC | 14:31 | |
| openstackgerrit | Christoph Berg proposed a change to openstack-infra/jenkins-job-builder: Add bzr scm support https://review.openstack.org/70121 | 14:31 |
| fungi | jeblair: yeah, i'm suspecting we're being bitten by another change to the github api/lib, but will need to play around with it to confirm | 14:32 |
| *** homeless has joined #openstack-infra | 14:32 | |
| *** thuc has quit IRC | 14:34 | |
| *** dkliban has joined #openstack-infra | 14:37 | |
| *** yolanda_ has quit IRC | 14:37 | |
| *** starmer has quit IRC | 14:37 | |
| *** yamahata has quit IRC | 14:38 | |
| *** yolanda_ has joined #openstack-infra | 14:38 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: package: read a specific Python version requirement file https://review.openstack.org/63236 | 14:38 |
| *** kraman has quit IRC | 14:39 | |
| *** changbl has quit IRC | 14:39 | |
| *** yamahata has joined #openstack-infra | 14:40 | |
| *** eharney has quit IRC | 14:41 | |
| *** eharney_ has joined #openstack-infra | 14:41 | |
| *** boris-42 has joined #openstack-infra | 14:41 | |
| *** eharney_ is now known as eharney | 14:41 | |
| *** afazekas has quit IRC | 14:43 | |
| *** oubiwann_ has joined #openstack-infra | 14:44 | |
| *** salv-orlando has joined #openstack-infra | 14:46 | |
| *** yolanda_ has quit IRC | 14:46 | |
| *** yolanda_ has joined #openstack-infra | 14:47 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add mistral with subprojects https://review.openstack.org/69683 | 14:48 |
| *** yolanda_ has quit IRC | 14:52 | |
| *** jnoller has joined #openstack-infra | 14:53 | |
| *** lcheng has quit IRC | 14:53 | |
| *** yolanda_ has joined #openstack-infra | 14:56 | |
| *** max_lobur_afk is now known as max_lobur | 14:56 | |
| *** mestery_ is now known as mestery | 14:56 | |
| *** vjay has left #openstack-infra | 14:56 | |
| *** SnowDust has joined #openstack-infra | 14:57 | |
| *** lcheng has joined #openstack-infra | 14:59 | |
| *** hdd_ has quit IRC | 15:00 | |
| *** yolanda_ has quit IRC | 15:01 | |
| *** yolanda_ has joined #openstack-infra | 15:02 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for grenade/swift bug 1274539 https://review.openstack.org/70130 | 15:03 |
| *** yolanda_ has quit IRC | 15:05 | |
| *** yolanda has joined #openstack-infra | 15:05 | |
| *** yolanda has joined #openstack-infra | 15:06 | |
| *** yolanda has quit IRC | 15:06 | |
| *** dstufft has quit IRC | 15:07 | |
| *** dstufft has joined #openstack-infra | 15:07 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/storyboard: Add info about how to run to README https://review.openstack.org/70132 | 15:07 |
| *** dcramer_ has joined #openstack-infra | 15:07 | |
| *** vogxn has joined #openstack-infra | 15:08 | |
| *** mfer has joined #openstack-infra | 15:08 | |
| *** yolanda has joined #openstack-infra | 15:08 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer bug 1274490 https://review.openstack.org/70110 | 15:09 |
| *** talluri has quit IRC | 15:10 | |
| *** talluri has joined #openstack-infra | 15:10 | |
| *** lcheng has quit IRC | 15:14 | |
| *** talluri has quit IRC | 15:15 | |
| *** yolanda has quit IRC | 15:15 | |
| *** gokrokve has joined #openstack-infra | 15:15 | |
| *** yolanda has joined #openstack-infra | 15:17 | |
| *** yolanda has quit IRC | 15:17 | |
| *** yolanda has joined #openstack-infra | 15:17 | |
| *** dims has quit IRC | 15:18 | |
| *** mfer has quit IRC | 15:18 | |
| *** DinaBelova is now known as DinaBelova_ | 15:20 | |
| *** kraman has joined #openstack-infra | 15:21 | |
| *** dims has joined #openstack-infra | 15:21 | |
| *** yolanda has quit IRC | 15:22 | |
| *** yolanda has joined #openstack-infra | 15:23 | |
| *** yolanda has quit IRC | 15:24 | |
| *** yolanda has joined #openstack-infra | 15:24 | |
| *** rwsu has joined #openstack-infra | 15:24 | |
| *** jgrimm has joined #openstack-infra | 15:25 | |
| *** sarob has joined #openstack-infra | 15:25 | |
| *** markmc has quit IRC | 15:26 | |
| *** SnowDust has quit IRC | 15:26 | |
| *** oanufriev` has quit IRC | 15:26 | |
| *** GNithya has joined #openstack-infra | 15:26 | |
| *** lcheng has joined #openstack-infra | 15:27 | |
| *** oanufriev has joined #openstack-infra | 15:27 | |
| *** david-lyle has joined #openstack-infra | 15:27 | |
| *** markwash has quit IRC | 15:29 | |
| *** mfer has joined #openstack-infra | 15:30 | |
| *** lcheng_ has joined #openstack-infra | 15:30 | |
| *** sarob has quit IRC | 15:31 | |
| *** yolanda has quit IRC | 15:31 | |
| *** david-lyle has quit IRC | 15:32 | |
| *** lcheng has quit IRC | 15:32 | |
| *** yolanda has joined #openstack-infra | 15:32 | |
| *** yolanda has quit IRC | 15:32 | |
| *** yolanda has joined #openstack-infra | 15:33 | |
| jeblair | reed: ping | 15:33 |
| reed | pong | 15:34 |
| jeblair | (private msg) | 15:34 |
| *** AaronGr_zzz is now known as AaronGr | 15:34 | |
| *** DinaBelova_ is now known as DinaBelova | 15:35 | |
| *** reed has quit IRC | 15:36 | |
| *** yolanda has quit IRC | 15:38 | |
| *** yolanda has joined #openstack-infra | 15:39 | |
| *** emagana has joined #openstack-infra | 15:39 | |
| *** emagana has quit IRC | 15:39 | |
| *** emagana has joined #openstack-infra | 15:39 | |
| *** dkranz has joined #openstack-infra | 15:40 | |
| *** sandywalsh has quit IRC | 15:40 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add the time at which this page was generated https://review.openstack.org/70143 | 15:41 |
| *** reed has joined #openstack-infra | 15:41 | |
| *** markmcclain has joined #openstack-infra | 15:41 | |
| *** andre__ has quit IRC | 15:43 | |
| *** wenlock has joined #openstack-infra | 15:43 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/storyboard: Add info about how to run to README https://review.openstack.org/70132 | 15:44 |
| fungi | nice... we're about to have 10 changes all merge from the integrated gate queue at once (maybe more) | 15:44 |
| jeblair | yaaay | 15:45 |
| russellb | fungi: i've been watching, there were ahead of these 10, so 12, and looking good to be a lot more | 15:46 |
| fungi | after the past month or so, it's a darned refreshing sight | 15:46 |
| russellb | no kidding | 15:46 |
| *** yolanda has quit IRC | 15:46 | |
| *** yolanda has joined #openstack-infra | 15:47 | |
| russellb | if this passes, it goes to 15 ... https://jenkins05.openstack.org/job/gate-tempest-dsvm-postgres-full/316/console | 15:47 |
| russellb | weeeeee | 15:47 |
| *** ArxCruz has quit IRC | 15:48 | |
| *** AlexF_ has joined #openstack-infra | 15:48 | |
| *** ArxCruz has joined #openstack-infra | 15:48 | |
| * fungi nods | 15:48 | |
| russellb | boo, what broke .. | 15:50 |
| *** rcleere has joined #openstack-infra | 15:51 | |
| *** yolanda has quit IRC | 15:53 | |
| *** beagles_brb is now known as beagles | 15:54 | |
| *** sandywalsh has joined #openstack-infra | 15:54 | |
| *** yolanda has joined #openstack-infra | 15:54 | |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Add attachment pattern expression and matrix trigger mode to email-ext. https://review.openstack.org/70148 | 15:57 |
| *** andre__ has joined #openstack-infra | 15:59 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/storyboard-webclient: Be less negative about launchpad https://review.openstack.org/70151 | 16:00 |
| *** oanufriev has quit IRC | 16:00 | |
| fungi | oh, nice. i think this failed because ceilometer's dependency on nova had it downloading the nova master tarball at the same time a post job was uploading an updated one... https://jenkins06.openstack.org/job/gate-ceilometer-python27/29/console | 16:00 |
| *** matrohon has quit IRC | 16:00 | |
| *** mfer has quit IRC | 16:00 | |
| *** oanufriev has joined #openstack-infra | 16:01 | |
| *** yolanda has quit IRC | 16:01 | |
| fungi | (add to the list, reasons why having our server projects depend on one another for unit tests is a questionable choice) | 16:01 |
| *** mfer has joined #openstack-infra | 16:01 | |
| *** yolanda has joined #openstack-infra | 16:03 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Change mysql package for Fedora >= 19 for bare-precise https://review.openstack.org/70152 | 16:03 |
| *** boris-42 has quit IRC | 16:05 | |
| *** thedodd has joined #openstack-infra | 16:05 | |
| *** CaptTofu has joined #openstack-infra | 16:05 | |
| *** virmitio has joined #openstack-infra | 16:07 | |
| *** oanufriev has quit IRC | 16:07 | |
| *** yolanda has quit IRC | 16:09 | |
| *** yolanda has joined #openstack-infra | 16:09 | |
| *** che-arne has quit IRC | 16:10 | |
| *** VijayT has joined #openstack-infra | 16:11 | |
| *** jcooley_ has joined #openstack-infra | 16:12 | |
| hashar | fungi: it seems gate-tempest-dsvm-postgres-full seems to be the most slowest job you have and might be one of the cause for slow gating. | 16:13 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Be less negative about launchpad https://review.openstack.org/70151 | 16:13 |
| hashar | fungi: maybe it can be made faster somehow? | 16:13 |
| hashar | also it is triggered even when some other fast test is failing .. | 16:14 |
| hashar | that is the case for ceilometer https://review.openstack.org/#/c/68714/ for which the python27 job already failed :/ | 16:15 |
| *** che-arne has joined #openstack-infra | 16:16 | |
| *** yolanda has quit IRC | 16:17 | |
| fungi | hashar: 1) we'd like all jobs to be faster, sure... tempest tests were failing more often due to what seemed to be resource constraints from too much parallelization so they were made less parallel recently, which also made them take longer. i think the tempest team has ideas on how to make those jobs faster, but not sure what the current situation is | 16:17 |
| *** yolanda has joined #openstack-infra | 16:17 | |
| *** reed has quit IRC | 16:17 | |
| hashar | what I mean is that whenever a job fail in the gate, the change should be dequeued | 16:17 |
| hashar | even if other jobs for that change are still running | 16:17 |
| fungi | hashar: but then how do you know whether you have unrelated issues which might cause those other jobs to also fail? | 16:18 |
| *** DinaBelova is now known as DinaBelova_ | 16:18 | |
| hashar | isn't it retrying the jobs behind ? | 16:18 |
| fungi | and anyway, zuul's nnfi algotithm immediately stops testing other changes against the failing change and reparents them to the nearest non-failing change anyway, so those jobs aren't waiting on the failing change to be re-run | 16:19 |
| hashar | A -> B -> C , if one jobs for A fails, doesn't Zuul reenquue B -> C ? | 16:19 |
| hashar | ahh | 16:19 |
| hashar | is that nnfi algorithm kick in whenever a job fails without waiting for other jobs to complete? | 16:20 |
| fungi | which is what the lines connecting those dots is trying to represent | 16:20 |
| fungi | yes | 16:20 |
| hashar | so the second job in the current gate (ceilometer 68714,3) has been detached and all the queue reparented? ;) | 16:21 |
| fungi | correct | 16:21 |
| *** gokrokve has quit IRC | 16:21 | |
| hashar | sorry if I am rephrasing what you said, trying to make sure I understand properly | 16:21 |
| hashar | that is neat | 16:21 |
| *** e0ne_ has quit IRC | 16:21 | |
| fungi | however, the jobs after it will be reparented back onto it if the change ahead of it also fails | 16:21 |
| *** e0ne has joined #openstack-infra | 16:21 | |
| fungi | since it will also get retested in that case | 16:21 |
| hashar | which would cause all jobs to be relaunched again ? | 16:22 |
| *** lcheng_ has quit IRC | 16:22 | |
| *** nicedice has joined #openstack-infra | 16:22 | |
| fungi | correct | 16:22 |
| hashar | (since the parent tree changed) | 16:22 |
| hashar | what a mess | 16:22 |
| hashar | so at least you fail as early as possible | 16:22 |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Add support for TestNG publisher. https://review.openstack.org/70157 | 16:22 |
| hashar | and the failing change will still run everything letting dev the possibility to fix everything before it is gated again | 16:22 |
| hashar | If only the jobs below could be made faster: | 16:23 |
| hashar | gate-tempest-dsvm-full SUCCESS in 49m 59s | 16:23 |
| hashar | gate-tempest-dsvm-postgres-full SUCCESS in 54m 51s | 16:23 |
| hashar | gate-tempest-dsvm-neutron SUCCESS in 36m 52s | 16:23 |
| hashar | you will have less troubles / slowness I guess. | 16:23 |
| fungi | right now 68537 is being tested with 69764, skipping 68714 because it's failing. but if 69764 also fails, jobs for 68714 will be restarted based on the branch tip instead (along with 68537) | 16:24 |
| *** david-lyle has joined #openstack-infra | 16:24 | |
| *** yolanda has quit IRC | 16:24 | |
| fungi | and we don't currently abort and reparent as early as we *could* because right now zuul doesn't know whether a job failed until jenkins finishes running it, even though the script running the job may know very early on the job is going to fail | 16:25 |
| *** yolanda has joined #openstack-infra | 16:25 | |
| *** yolanda has quit IRC | 16:25 | |
| fungi | there are some ideas floating around about how to let zuul know a job is going to fail even before it's done being run, which could speed that determination up considerably in some cases | 16:25 |
| hashar | ahhhh | 16:26 |
| *** weshay has quit IRC | 16:27 | |
| fungi | as for making tempest jobs faster, they're already inherently slow because they build an entire openstack cloud from scratch and then test interacting with it using virtualization inside of more virtualization, which doesn't perform at all well | 16:27 |
| hashar | sounds like Inception | 16:28 |
| fungi | kinda | 16:28 |
| hashar | that must reduce the number of bugs being filled | 16:28 |
| *** zul has quit IRC | 16:29 | |
| *** thuc has joined #openstack-infra | 16:30 | |
| *** smarcet has quit IRC | 16:30 | |
| *** thuc has quit IRC | 16:31 | |
| *** weshay has joined #openstack-infra | 16:31 | |
| *** thuc has joined #openstack-infra | 16:31 | |
| *** malini is now known as malini_afk | 16:31 | |
| *** DinaBelova_ is now known as DinaBelova | 16:31 | |
| *** zul has joined #openstack-infra | 16:32 | |
| *** thuc has quit IRC | 16:32 | |
| *** thuc has joined #openstack-infra | 16:32 | |
| *** lcheng has joined #openstack-infra | 16:33 | |
| *** GNithya has quit IRC | 16:33 | |
| *** vogxn has quit IRC | 16:34 | |
| sdague | fungi: actually, do we have to possibility of kicking rax out of gate again? | 16:34 |
| sdague | because the performance nodes are still slow on the 12.04 image | 16:34 |
| fungi | sdague: it would more than half our capacity, but i guess we could | 16:34 |
| sdague | until we can try to PVM image | 16:34 |
| sdague | fungi: sure, but with gate window we should be fine | 16:35 |
| sdague | because I've yet to see an HP node run over an hour | 16:35 |
| fungi | at the moment, i believe fewer than half of our vms are being supplied by hpcloud, most in an availability zone with is having substantial stabliity/performance issues of its own | 16:35 |
| *** jcooley_ has quit IRC | 16:35 | |
| *** rnirmal has joined #openstack-infra | 16:35 | |
| jeblair | so they used to be faster, what happened? | 16:36 |
| sdague | I guess the other question would be exploring the PVM image | 16:36 |
| sdague | jeblair: they are still wildly variant | 16:36 |
| sdague | back down at concurency of 2 its more obvious | 16:36 |
| jeblair | sdague: and do we know if it's a certain zone? | 16:36 |
| sdague | it's not | 16:36 |
| sdague | let me rebuild my graphs | 16:36 |
| jeblair | sdague: ok; that is something that changed (we used to basically use only 2, now we use all 3) | 16:36 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Change mysql package for Fedora >= 19 for bare-precise https://review.openstack.org/70152 | 16:37 |
| jeblair | sdague: i think dfw had (almost) no slaves, now it has a significant portion | 16:37 |
| sdague | one sec, let me rerun my graphs | 16:37 |
| jeblair | sdague: also, if there's a way to test for this, we can actually have nodepool run that test before marking a node ready (this feature already exists) | 16:38 |
| jeblair | sdague: basically, an acceptance test | 16:38 |
| *** oubiwann_ has quit IRC | 16:39 | |
| sdague | jeblair: honestly, I don't know of any way to do that, I just know the total times on runs | 16:39 |
| fungi | a quick burn-in/benchmark routine, presumably. but knowing which resources to pay attention to relative performance for would be the tricky part | 16:39 |
| sdague | http://dague.net/testing/tempest-full.png | 16:39 |
| sdague | http://dague.net/testing/tempest-pg-full.png | 16:39 |
| sdague | that assumes it's something we could figure out up front | 16:40 |
| jeblair | pvo: ^ you may be interested in that graph; looks like significant performance variation on performance-1 nodes | 16:41 |
| sdague | those should be filtering on success runs only, though I'll need to scrub for the reason the short runs are there | 16:41 |
| jnoller | sdague: You should be able to use the PVHVM images now? | 16:41 |
| fungi | sdague: i'm curious what those numbers are/mean. the lack of labels leaves a lot of questions ;) | 16:41 |
| *** changbl has joined #openstack-infra | 16:41 | |
| sdague | fungi: Y axis is run time | 16:41 |
| sdague | in minutes | 16:41 |
| sdague | x axis is just arbitrary | 16:41 |
| sdague | I'm scraping all the run times off of all the jenkins | 16:41 |
| sdague | this has basically always been how rax looks though | 16:42 |
| jeblair | sdague: er, scraping jenkins? are you filtering out check? | 16:42 |
| sdague | jeblair: yes | 16:42 |
| fungi | so hpcloud-az1 completes some of these jobs in 10 minutes? | 16:42 |
| jeblair | sdague: and aborted jobs? | 16:42 |
| *** e0ne has quit IRC | 16:42 | |
| sdague | jeblair: yeh, I may not be dumping aborted jobs | 16:42 |
| *** malini_afk is now known as malini | 16:42 | |
| sdague | I am filtering out fails | 16:42 |
| *** oubiwann_ has joined #openstack-infra | 16:42 | |
| sdague | I can stick this all in a github somewhere in a sec | 16:43 |
| sdague | and it's only gate queue | 16:43 |
| jnoller | sdague: You're using non-pvhvm performance1-1 VMs? | 16:43 |
| sdague | jnoller: correct, because those images were only available yesterday | 16:43 |
| jeblair | jnoller: 8g performance1 | 16:43 |
| fungi | aside from the (questionably quick) hpcloud-az1 times, the rest look to be a relatively consistent band | 16:43 |
| sdague | and no one has looked into putting those out into production | 16:43 |
| jeblair | so that's, what, performance1-8? | 16:43 |
| jnoller | sdague how can I help getting those pushed into prod (the PVHVM images) | 16:44 |
| sdague | ask jeblair / fungi | 16:44 |
| *** masayukig has quit IRC | 16:44 | |
| sdague | my local sniff testing seemed ok, but they know what we'd need to do to trust them in node pool | 16:44 |
| *** masayukig has joined #openstack-infra | 16:45 | |
| jnoller | tell me how I can help jeblair / fungi | 16:45 |
| jeblair | add them to nodepool as 'devstack-precise-check' images | 16:45 |
| fungi | what we've done with others is to just add a handful of them mixed into the existing check pol | 16:45 |
| fungi | pool | 16:45 |
| *** smarcet has joined #openstack-infra | 16:45 | |
| fungi | what jeblair said | 16:45 |
| jeblair | so we can try them out in the check queue first | 16:45 |
| jeblair | and yeah, a small number | 16:45 |
| jeblair | (which we will drop to 0 if there is a problem) | 16:45 |
| fungi | i'll write up a change to add a few so we can get numbers for them | 16:46 |
| jnoller | ok, how can I do that, pretending I'm very new to infra | 16:46 |
| fungi | jnoller: a patch to the nodepool configuration... i can get you a link. hold on | 16:46 |
| jnoller | ty | 16:46 |
| fungi | jnoller: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 16:46 |
| fungi | that's basically the configuration for our pools of disposable test servers | 16:47 |
| *** hemna has quit IRC | 16:47 | |
| fungi | we'd do something similar to the hpcloud-region-b | 16:47 |
| jeblair | jnoller: mimic what hpcloud-region-b does, except only add it to one jenkins, and only specify 2 servers should be added | 16:48 |
| jnoller | do you store the image IDs or use the "Ubuntu 12.04 LTS (Precise Pangolin)" label | 16:48 |
| fungi | jnoller: we use the image name | 16:48 |
| fungi | rather than the id | 16:48 |
| jnoller | k | 16:48 |
| jnoller | working on it | 16:48 |
| *** talluri has joined #openstack-infra | 16:48 | |
| *** masayukig has quit IRC | 16:49 | |
| fungi | nodepool does the equivalent of a nova image-list to see whether the image it wants is there, which is why we couldn't easily add the hidden images before yesterday without patches to nodepool itself | 16:49 |
| fungi | i suspect it was written with the expectation that there were no such thing as hidden/secret image ids | 16:50 |
| jnoller | yeah | 16:51 |
| jnoller | they're public now | 16:51 |
| *** gyee has joined #openstack-infra | 16:51 | |
| jnoller | so no prob there | 16:51 |
| *** morganfainberg|z is now known as morganfainberg | 16:52 | |
| sdague | https://github.com/sdague/tempest-timings | 16:54 |
| sdague | git clone, run scrape.py | 16:54 |
| *** olaph has quit IRC | 16:54 | |
| sdague | it's kind of hack, but it has worked so far | 16:54 |
| *** markmc has joined #openstack-infra | 16:54 | |
| sdague | the kind of hack is why I didn't have it up on github already | 16:54 |
| jeblair | sdague: the data are in graphite i believe | 16:55 |
| jeblair | sdague: the reason not to publish it is because we are not a multi-cloud benchmark | 16:55 |
| sdague | jeblair: yep, that's fair | 16:55 |
| sdague | do we have the data separated by only success runs? | 16:55 |
| *** dangers_away is now known as dangers | 16:55 | |
| sdague | honestly, graphite and my brain don't work very well together | 16:55 |
| jeblair | we need to have this info, but we shouldn't publish it on a web page to make it easily accessible so people don't take it as such | 16:56 |
| sdague | which is why I did it this other way | 16:56 |
| sdague | jeblair: agreed | 16:56 |
| sdague | again, why this was some random code on my laptop until 60s ago | 16:56 |
| sdague | but we started talking about why jobs take a while, and I figured it was useful to share the snapshot | 16:56 |
| sdague | fungi: now with lables - http://dague.net/testing/tempest-full.png http://dague.net/testing/tempest-pg-full.png | 16:57 |
| jeblair | sdague: we're about to move camp here, so i don't have time to fix this, but here is an old link to a similar graph: http://paste.openstack.org/show/62189/ | 16:57 |
| jeblair | sdague: it doesn't work, i think it needs to be updated to use new metric paths | 16:57 |
| jeblair | sdague: but you should be able to specify gate and success | 16:58 |
| jeblair | sdague: i will fix that later when i have time if someone hasn't beat me to it | 16:58 |
| jnoller | some days, I hate git | 16:58 |
| *** afazekas has joined #openstack-infra | 16:58 | |
| *** hcc has joined #openstack-infra | 17:00 | |
| *** hcc is now known as hdd_ | 17:00 | |
| fungi | is it just me, or does this job log look like something which should not have succeeded? https://jenkins02.openstack.org/job/gate-heat-python27/6121/console | 17:01 |
| Alex_Gaynor | is reverify no bug still a thing? | 17:04 |
| fungi | Alex_Gaynor: nope | 17:04 |
| Alex_Gaynor | fungi: What's the right move for https://review.openstack.org/#/c/68425/ which was hit by a bug that's now fixed and I'm not sure if it ever had a number | 17:04 |
| fungi | Alex_Gaynor: reset your approval vote if there was no bug involved in the failure to merge, or reverify an actual bug number corresponding to the merge failure | 17:05 |
| Alex_Gaynor | fungi: trhanks | 17:05 |
| * fungi looks | 17:05 | |
| fungi | Alex_Gaynor: that was the subunit issue. i'll find their bug number | 17:05 |
| *** blamar has joined #openstack-infra | 17:06 | |
| Alex_Gaynor | fungi: I just re-approved, sorry | 17:06 |
| *** praneshp has joined #openstack-infra | 17:06 | |
| *** oanufriev has joined #openstack-infra | 17:07 | |
| fungi | Alex_Gaynor: 1274056 | 17:07 |
| fungi | eh, no worries | 17:07 |
| *** pblaho has quit IRC | 17:08 | |
| *** cadenzajon has joined #openstack-infra | 17:09 | |
| *** rnirmal has quit IRC | 17:10 | |
| *** _david_ has joined #openstack-infra | 17:12 | |
| *** atiwari has joined #openstack-infra | 17:13 | |
| jeblair | sdague: nope; graphite doesn't have that data; i think it only has the data you are scraping (that is, it doesn't have that by pipeline) | 17:14 |
| *** boris-42 has joined #openstack-infra | 17:14 | |
| jeblair | sdague: it has one metric by pipeline but not provider, and another by provider but not pipeline. :( | 17:14 |
| mika | i've got this jenkins-job-builder config: https://raw.github.com/sipwise/kamailio-deb-jenkins/mika/jjb-unify/jjb/simple.yaml - any ideas how to bypass a list of values from the "project" definition to a "job-template" proberly? | 17:14 |
| *** masayukig has joined #openstack-infra | 17:15 | |
| *** gokrokve has joined #openstack-infra | 17:15 | |
| *** sandywalsh has quit IRC | 17:15 | |
| *** praneshp has quit IRC | 17:16 | |
| *** atiwari has quit IRC | 17:16 | |
| jnoller | fungi: Should I add a new yaml block that defines something like a rac-iad-b? | 17:16 |
| jnoller | er | 17:16 |
| jnoller | rax-iad | 17:16 |
| *** atiwari has joined #openstack-infra | 17:16 | |
| *** masayukig has quit IRC | 17:16 | |
| *** hashar has quit IRC | 17:17 | |
| *** masayukig has joined #openstack-infra | 17:17 | |
| fungi | jnoller: i would add the new image to the existing rax-iad provider | 17:17 |
| fungi | rather than secifying a separate provider entirely | 17:17 |
| *** NithyaG__ has joined #openstack-infra | 17:17 | |
| jnoller | fungi: ok, but can you set the max-servers: 60 on the child? | 17:17 |
| fungi | oh, hrm. good point. let me stare at it a moment | 17:18 |
| jnoller | ok | 17:18 |
| jnoller | I'll brb, but I'll get a patch up as soon as I get back / you let me know | 17:18 |
| jnoller | I have the image ID string and tested it out myself | 17:18 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: project_maven: allow to set private repository https://review.openstack.org/69484 | 17:18 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Add info about how to run to README https://review.openstack.org/70132 | 17:19 |
| fungi | mika: here's a good example of what i think you're asking... https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml#n1833 | 17:20 |
| mika | fungi: hm, this creates new jobs though, while i need them as a *list* like: | 17:21 |
| *** masayukig has quit IRC | 17:21 | |
| *** afazekas has quit IRC | 17:21 | |
| *** yassine has quit IRC | 17:21 | |
| mika | values: | 17:22 |
| mika | - {values} | 17:22 |
| *** julim has quit IRC | 17:23 | |
| clarkb | morning | 17:24 |
| mika | passing one single value is no problem, using a defined list though is :( | 17:24 |
| *** sandywalsh has joined #openstack-infra | 17:24 | |
| *** jcooley_ has joined #openstack-infra | 17:25 | |
| ArxCruz | jeblair: any chance to take a look at https://review.openstack.org/#/c/70152/ ? pretty easy one :) | 17:25 |
| clarkb | mika: list of lists maybe? | 17:25 |
| fungi | mika: hashar had an earlier example of using yaml macros to do something similar... let me dig that back up real quick | 17:25 |
| *** starmer has joined #openstack-infra | 17:26 | |
| anteaya | ArxCruz: he is at a code sprint in Brussells, there might be a pause prior to a response | 17:26 |
| mika | clarkb: list of list actually (i think :)) | 17:26 |
| ArxCruz | oh... | 17:26 |
| mika | fungi: i was the one who came up with problem and worked with hashar on it :) | 17:26 |
| fungi | mika: ohhh | 17:26 |
| ArxCruz | anteaya: not my lucky week I suppose | 17:26 |
| mika | (different channel, for jenkins-debian-glue :)) | 17:26 |
| *** markmc has quit IRC | 17:26 | |
| * fungi nods | 17:26 | |
| anteaya | ArxCruz: join the club | 17:26 |
| anteaya | not a very lucky week for more than a few of us | 17:27 |
| ArxCruz | anteaya: joined, twice | 17:27 |
| ArxCruz | hehehehe | 17:27 |
| anteaya | but at least we have each other | 17:27 |
| mika | any chance i might catch one of you jjb guys as fosdem to figure this out? | 17:27 |
| anteaya | :D | 17:27 |
| fungi | mika: jeblair and mordred are both there | 17:27 |
| ArxCruz | well, I guess I have to hack these scripts/recipes internally from my side :( | 17:28 |
| mika | fungi: cool, any idea where i might find them? :) | 17:28 |
| fungi | mika: i'm pretty certain they would also love to hear about how jjb is enabling jenkins/debian glue | 17:28 |
| *** afazekas has joined #openstack-infra | 17:28 | |
| anteaya | ArxCruz: :( | 17:28 |
| *** jcooley_ has quit IRC | 17:28 | |
| zaro | morning | 17:29 |
| ArxCruz | duplicate the work hehehe | 17:29 |
| mika | fungi: would love to talk about this with them! | 17:29 |
| *** sarob has joined #openstack-infra | 17:29 | |
| fungi | mika: well, they're at a pre-fosdem sprint in brussels currently, but they're also here in the channel and will probably see this in scrollback once they're back near computers | 17:29 |
| anteaya | ArxCruz: testing it locally | 17:29 |
| mika | fungi: do you have any contact information for jeblair and mordred for me, like twitter/mail/jabber/...? | 17:29 |
| ArxCruz | anteaya: I already did | 17:29 |
| mika | fungi: i see, so let's hope i catch them before fosdem is over, thanks :) | 17:30 |
| anteaya | ArxCruz: I'm trying to find a bright side, not being too successful | 17:30 |
| clarkb | mika: I really think values: [[a,b,c], [1,2,3]] will fix it | 17:30 |
| ArxCruz | lol anteaya sorry, I didn't notice that :) | 17:30 |
| anteaya | try as I might, I can't be markmcclain on demand and I haven't tried to be jeblair yet, but I bet I will fail at that too | 17:31 |
| mika | clarkb: will try that, thanks! | 17:31 |
| clarkb | mika: the issue is that JJB builds its own matrixes with lists | 17:31 |
| mika | clarkb: ahhhh! | 17:31 |
| clarkb | so you need your lists to be a level deeper otherwise you just get a matrix job essentially | 17:31 |
| mika | thanks, will give it a shot after babysitting time, need to leave now :) | 17:31 |
| fungi | clarkb: just to confirm, for nodepool we can currently only set max-servers on a provider not on a particular image, right? | 17:33 |
| clarkb | I believe that is correct, the ratio between each image is determined based on the ration of min servers | 17:34 |
| *** jcooley_ has joined #openstack-infra | 17:34 | |
| *** sarob has quit IRC | 17:34 | |
| fungi | yeah, that matches my recollection and what i'm arriving at by comparing the source and docs | 17:35 |
| fungi | thanks | 17:35 |
| fungi | jnoller: so in that case we do actually need a shadow provider name for it as you surmised | 17:36 |
| ArxCruz | anteaya: nah, you're the best ;D | 17:36 |
| *** jpich has quit IRC | 17:37 | |
| clarkb | fungi: why do we want to set max servers on a child though? | 17:37 |
| *** coolsvap has joined #openstack-infra | 17:38 | |
| fungi | clarkb: we have a particular base image we want to test making devstack-precise-check servers from in a rax region but limit the number of them | 17:38 |
| fungi | to collect relative performance numbers | 17:38 |
| clarkb | and we can't rely on the ratio to do that? Does that imply there is a hard upper limit? | 17:39 |
| fungi | there's no hard upper limit, we just want a very small number of devstack-precise-check nodes to use this base image for starters, so that we don't cause broad disruption of it turns out to be problematic for some reason | 17:40 |
| ArxCruz | clarkb: thanks for +2 :) | 17:40 |
| anteaya | ArxCruz: :D | 17:40 |
| clarkb | fungi: yeah so I think you can keep the ratio low and nodepool will do that automagically | 17:41 |
| * ArxCruz hopping that someone else can +2 and merge the change... | 17:41 | |
| *** AlexF_ has quit IRC | 17:42 | |
| *** oanufriev has quit IRC | 17:42 | |
| fungi | clarkb: i'm not seeing a way to explicitly set a ratio between them. or do you mean implicitly using the min-ready on the targets (and setting the min-ready for the others higher than they are now)? | 17:42 |
| *** mrodden has joined #openstack-infra | 17:43 | |
| clarkb | in other news I propose that all new projects we host be named by `cat /dev/urandom | tr -dc '[:alpha:]' | head -c6` | 17:43 |
| clarkb | fungi: yes the ration of min-readies determines the number of nodes created with that image and I believe that number is across all images for a provider so you don't need to bump others | 17:43 |
| *** vkozhukalov has quit IRC | 17:44 | |
| fungi | okay, i that case the only providers with devstack-precise-check images right now are hpcloud-region-a and b, which have max-servers 0 and 5 respectively... and we set min-ready 3 for them on all 7 targets, which is a little weird to begin with | 17:45 |
| clarkb | fungi: if you set min-ready to say 1 on all targets for a rax hvm image I think you will get what you want | 17:46 |
| fungi | clarkb: it won't be a rax hvm image on the targets... it will be devstack-precise-check | 17:46 |
| *** rcleere has quit IRC | 17:47 | |
| clarkb | devstack-precise-check is a label name | 17:47 |
| *** reed has joined #openstack-infra | 17:47 | |
| clarkb | the underlying image will be rax hvm right? | 17:47 |
| fungi | right. we set label names for images on each target | 17:47 |
| fungi | from what i can see in the nodepool.yaml | 17:47 |
| fungi | clarkb: yes, the base image nodepool starts building the template from will be rax hvm | 17:48 |
| clarkb | say I have rax-foo with max servers 10 and two images against one target. image1 has min servers 2 and image2 has min servers 1. I will get twice as many image1s as image2s | 17:48 |
| clarkb | so if you add a rax hvm image with label devstack-precise-check with min servers 1 you will get 1:sumofotherminserversforthatraxprovider ratio servers | 17:48 |
| fungi | but we set min-ready on the image labels, not on specific images, and then we list which providers to source them from | 17:49 |
| clarkb | ah so the problem is that we apply them to labels, I think it will still work well enough | 17:50 |
| fungi | so in aggregate on a given target, we have min-ready (some number) of devstack-precise-check and they can be obtained from one or more providers, one of which is the rax provider where we have the hvm images available with a devstack-precise-check label | 17:50 |
| clarkb | ya so if you want to get more rax hvm nodes you can decrease the max servers for the hpcloud b region | 17:50 |
| *** e0ne has joined #openstack-infra | 17:51 | |
| fungi | and right now we have the max on the existing providers for devstack-precise-check set so low that nodepool can't meet the min-ready on all the targets anyway, so presumably if we don't have a max set on the devstack-precise-check provider for the hvm images we'll end up with potentially far more of those | 17:51 |
| clarkb | fungi: potentially, but remember the ratio applies against hte provider | 17:52 |
| fungi | unless we duplicate, say, rax-iad and put just these new devstack-precise-check images in it so we can set an artificially low max | 17:52 |
| clarkb | I don't think that is necessary because rax-iad supplies images with much larger min-servers | 17:53 |
| *** samuelbercovici has joined #openstack-infra | 17:53 | |
| fungi | oh, for the other image names | 17:53 |
| fungi | s/names/labels/ | 17:53 |
| clarkb | right | 17:53 |
| *** ryanpetrello has quit IRC | 17:54 | |
| clarkb | I agree min servers 3 for devstack-precise-check seems a bit high | 17:54 |
| clarkb | but other than that, I think you can just add hte new image | 17:54 |
| fungi | we have min-ready set to 3*7 across the targets, and a total max across all providers set to 1*5 | 17:55 |
| *** ryanpetrello has joined #openstack-infra | 17:55 | |
| fungi | so currently we're only allowing the providers to build 25% of the minimum we're asking to have | 17:55 |
| clarkb | that heat success with tracebacks seems to have hit problems with eventlet. Those tracebacks went to stdout so happened between tests. Not sure how we can catch that better | 17:55 |
| *** zehicle_at_dell has quit IRC | 17:56 | |
| *** krtaylor has quit IRC | 17:56 | |
| *** BobBall is now known as BobBallAway | 17:56 | |
| fungi | okay, so the job shouldn't have failed then, eventlet was just spewing garbage | 17:56 |
| clarkb | basically, arguably that should be a fail though | 17:56 |
| *** rlandy is now known as rlandy|bbl | 17:56 | |
| clarkb | but hard to detect | 17:56 |
| *** emagana has quit IRC | 17:58 | |
| *** emagana has joined #openstack-infra | 17:58 | |
| clarkb | zaro: I don't see the dependency you merged to install buck | 17:59 |
| clarkb | zaro: it isn't a strict dependency | 17:59 |
| zaro | clarkb: do you mean gerrit dependency? | 18:00 |
| *** NithyaG__ has quit IRC | 18:00 | |
| *** jcooley_ has quit IRC | 18:00 | |
| *** jcooley_ has joined #openstack-infra | 18:01 | |
| clarkb | zaro: you said there was a dependency change to deal with buck. It isn't a strict dependency in gerrit/git | 18:01 |
| *** mrodden has quit IRC | 18:01 | |
| clarkb | fungi: re https://review.openstack.org/#/c/69442/1/manifests/site.pp I interpret that to mean we don't want automagic updates on those servers? | 18:01 |
| fungi | clarkb: that's my assumption of what we want anyway | 18:02 |
| *** praneshp has joined #openstack-infra | 18:02 | |
| clarkb | fungi: actually hmm, I wonder if that is dead code just confusing us | 18:02 |
| clarkb | fungi: the nodepool scripts do a puppet apply and specifically give the class they want, it does not do a puppet agent back to the puppet master | 18:02 |
| fungi | clarkb: that was my other suspicion (thus the question i posed in my comment) | 18:02 |
| fungi | i think that might date back to jclouds use for devstack-gate slaves? | 18:03 |
| fungi | which would have ended up using that mechanism to decide what to install | 18:03 |
| fungi | on bug 1268732 i'm trying to catch a gate-heat-python27 run in progress so i can hold the node and breakdown the filesystem utilization. i caught the end of one but didn't hold it fast enough so didn't log in until it was deleting. got just far enough to see that /var s pretty large (mysql query logging?) | 18:04 |
| zaro | clarkb: yes, you are correct. I didn't think about it when i pushed the patch, the dependency is only in my head. I can make gerrit track it if you like. | 18:04 |
| clarkb | zaro: I think that would help review, I was pretty confused about the buck stuff without the additional context | 18:05 |
| *** markwash has joined #openstack-infra | 18:05 | |
| clarkb | fungi: certainly possible | 18:05 |
| *** weshay has quit IRC | 18:05 | |
| clarkb | fungi: ok yeah, looking at git history on those template host lines, looks like it was used to prime a devstack host | 18:06 |
| clarkb | fungi: I am 95% sure I can just delete those nodes | 18:06 |
| fungi | clarkb: sounds probable | 18:06 |
| fungi | in unrelated news, the inability for hpcloud-az2 to successfully build a bare-precise image is starving us slightly for those i think | 18:07 |
| fungi | i'll see if i can confirm whether it's ssh timeouts | 18:08 |
| *** max_lobur is now known as max_lobur_afk | 18:08 | |
| *** lcheng has quit IRC | 18:08 | |
| zaro | clarkb: will do. actually i have a lot of changes in gerrit that depend on that buck install and gerrit 2.8 build setup :O | 18:08 |
| fungi | okay, 15.185.98.16 is running gate-heat-python27 in hpcloud-az1 and i've held it | 18:09 |
| fungi | that one's also spitting tracebacks during migration and ending early... https://jenkins06.openstack.org/job/gate-heat-python27/57/console | 18:10 |
| *** lcheng has joined #openstack-infra | 18:10 | |
| fungi | and so never built up the filesystem utilization i was seeing on the first one i was almost able to hold | 18:11 |
| clarkb | so that one didn't help? | 18:12 |
| fungi | nope. there's another which was in the check pipeline right after it, also with similar symptoms... https://jenkins04.openstack.org/job/gate-heat-python27/525/console | 18:14 |
| *** mrodden has joined #openstack-infra | 18:15 | |
| *** hogepodge has joined #openstack-infra | 18:16 | |
| fungi | yeah, every heat py27 unit test job i look at now is doing that | 18:16 |
| *** lcheng_ has joined #openstack-infra | 18:17 | |
| *** nati_ueno has joined #openstack-infra | 18:17 | |
| *** rossella_s has quit IRC | 18:17 | |
| clarkb | interesting, it doesn't look like all the tests end early. it claims to run a couple thousand. But maybe those exceptions are a side effect of disk problems? | 18:18 |
| *** masayukig has joined #openstack-infra | 18:18 | |
| *** VijayT has quit IRC | 18:18 | |
| *** lcheng has quit IRC | 18:18 | |
| fungi | py26 unit tests are doing it too | 18:19 |
| fungi | nah, not disk utilization in this case i don't think. they're ending the job with several gigabytes remaining free | 18:19 |
| fungi | also, in unrelated news, nodepool's getting socket timeouts on ci-overcloud.tripleo.org:13000 | 18:21 |
| *** elasticio has quit IRC | 18:21 | |
| *** julim has joined #openstack-infra | 18:22 | |
| *** masayukig has quit IRC | 18:22 | |
| *** smarcet has left #openstack-infra | 18:23 | |
| clarkb | fungi: perhaps we should hop over to #heat and poke them about the unittests? | 18:27 |
| fungi | clarkb: yeah, i'll do that | 18:27 |
| *** elasticio has joined #openstack-infra | 18:28 | |
| *** harlowja_away is now known as harlowja | 18:28 | |
| *** elasticio has quit IRC | 18:28 | |
| *** elasticio has joined #openstack-infra | 18:29 | |
| *** HenryG has quit IRC | 18:30 | |
| *** reed has quit IRC | 18:31 | |
| clarkb | fungi: I don't think I will manage to upgrade jenkins.o.o or jenkins01 before I head out to the dentist. Instead plan to do as much prep as possible and hopefully get back early enough to do them after | 18:31 |
| fungi | clarkb: sounds fin | 18:32 |
| fungi | e | 18:32 |
| *** fbo is now known as fbo_away | 18:33 | |
| *** HenryG has joined #openstack-infra | 18:33 | |
| *** gyee has quit IRC | 18:34 | |
| *** yolanda has joined #openstack-infra | 18:34 | |
| *** weshay has joined #openstack-infra | 18:34 | |
| fungi | i wonder if there's any good reason for us to be keeping >1gib of apt package cache on these bare-precise nodes, since tests running on them don't get root access to install packages anyway | 18:36 |
| fungi | (unlike the devstack-precise nodes) | 18:37 |
| *** e0ne has quit IRC | 18:37 | |
| *** thedodd has quit IRC | 18:37 | |
| *** e0ne has joined #openstack-infra | 18:37 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Redo slave manifests for clarity and correctness. https://review.openstack.org/69442 | 18:38 |
| clarkb | fungi: ^ has been rebased to include new iptables rule and further cleans up manifests/site.pp | 18:38 |
| fungi | excellent | 18:38 |
| clarkb | fungi: good question. I don't see any reason to keep the data | 18:39 |
| fungi | as we've seen, space is tight particularly on the hpcloud nodes, since we only get 10g for / | 18:39 |
| fungi | also, unlike the devstack-precise nodes, /opt is on / but we're still caching all our git repositories into it | 18:40 |
| *** sarob has joined #openstack-infra | 18:40 | |
| fungi | so that's another 1.5gib gone | 18:40 |
| clarkb | fungi: gah | 18:41 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: Support site monitor publisher https://review.openstack.org/69290 | 18:43 |
| *** praneshp has quit IRC | 18:43 | |
| fungi | mmm, more debian developers ^ are taking notice of jjb | 18:45 |
| jnoller | fungi: I'll add a shadow provider right now | 18:49 |
| fungi | jnoller: well, wait | 18:49 |
| fungi | jnoller: clarkb and i discussed it further, and he's suggesting we just add the image to one of the rax providers and not bother with the max setting (relying on our min-ready ratios for that image label to keep the count reasonably low) | 18:50 |
| jnoller | fungi: Yeah, reading the scrollback | 18:51 |
| fungi | since nodepool calculates a ratio of other image labels requested from a given provider, for which the min-ready counts are substantially higher | 18:51 |
| jnoller | I'll add the image to iad | 18:51 |
| fungi | sounds good | 18:51 |
| fungi | jnoller: (for future reference) are those images going to be available in all rax regions? | 18:52 |
| jnoller | Yes | 18:52 |
| jnoller | they should be now | 18:52 |
| fungi | just planning ahead for when we might eventually switch to them | 18:52 |
| jnoller | want me to add them to all? | 18:52 |
| fungi | cool | 18:52 |
| fungi | no, not right now | 18:52 |
| jnoller | fair | 18:52 |
| fungi | just one is fine for testing | 18:52 |
| *** hogepodge has quit IRC | 18:52 | |
| *** jp_at_hp has quit IRC | 18:52 | |
| *** odyssey4me has quit IRC | 18:53 | |
| clarkb | we expect better performance out of those nodes? | 18:54 |
| *** hogepodge has joined #openstack-infra | 18:54 | |
| clarkb | I am trying to remember the problem this hopes to address | 18:54 |
| jnoller | Yes | 18:54 |
| jnoller | clarkb: The performance servers + the PVHVM image are insanely faster than the default images | 18:54 |
| jnoller | clarkb: in many cases | 18:55 |
| clarkb | awesome | 18:55 |
| *** johnthetubaguy has quit IRC | 18:55 | |
| westmaas | you guys have custom images or do you start with the base images on every build? | 18:55 |
| jnoller | clarkb: http://developer.rackspace.com/blog/welcome-to-performance-cloud-servers-have-some-benchmarks.html | 18:55 |
| westmaas | (sorry for jumping in late) | 18:55 |
| jnoller | westmaas: they start with base images AFAICT | 18:55 |
| westmaas | cool. | 18:55 |
| fungi | westmaas: we create templates from a base image, and then snapshot those and reuse them to launch other vms | 18:55 |
| *** VijayT has joined #openstack-infra | 18:55 | |
| clarkb | aiui there is no facility for providing our own image | 18:56 |
| fungi | since there's no glance endpoint for us to upload into | 18:56 |
| westmaas | fungi: soon! | 18:56 |
| fungi | well, except in the tripleo cloud | 18:56 |
| fungi | westmaas: yep. so we've heard. exciting! | 18:56 |
| *** jcooley_ has quit IRC | 18:56 | |
| westmaas | yeah, we do more caching for base images though, even for uploaded images - so you will still experience some download time vs caching on compute | 18:57 |
| westmaas | that wasn't totally clear, but hopefully you see what I mean | 18:57 |
| westmaas | maybe build time is less of a concern than actual run-time performance though | 18:57 |
| fungi | build time is somewhat of a concern, but we build in advance based on speculative demand calculations so runtime performance is still a much bigger impact on our test throughput | 18:58 |
| *** dizquierdo is now known as dizquierdo_afk | 18:58 | |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM) flavor to Rackspace IAD https://review.openstack.org/70192 | 18:59 |
| *** jcooley_ has joined #openstack-infra | 18:59 | |
| fungi | jnoller: thanks! looking at it now | 18:59 |
| jnoller | let me know! | 19:00 |
| *** mrodden1 has joined #openstack-infra | 19:00 | |
| *** nati_uen_ has joined #openstack-infra | 19:00 | |
| *** sarob has quit IRC | 19:00 | |
| *** mrodden has quit IRC | 19:00 | |
| *** sarob has joined #openstack-infra | 19:01 | |
| clarkb | it is all autobalanced based on current and expected load. It is pretty awesome | 19:01 |
| clarkb | also it speaks to 3 different cloud providers | 19:01 |
| clarkb | and something like 8 AZs | 19:01 |
| *** praneshp has joined #openstack-infra | 19:02 | |
| *** david-lyle has quit IRC | 19:03 | |
| *** nati_ueno has quit IRC | 19:03 | |
| fungi | jnoller: one more line we need in it. see my review comment | 19:03 |
| *** nati_uen_ has quit IRC | 19:03 | |
| clarkb | and a couple other things from me :) | 19:03 |
| jnoller | man | 19:04 |
| jnoller | such a tough audience | 19:04 |
| jnoller | ;) | 19:04 |
| *** starmer has quit IRC | 19:04 | |
| *** nati_ueno has joined #openstack-infra | 19:04 | |
| *** lcheng_ has quit IRC | 19:07 | |
| fungi | jnoller: ahh, yep, i agree with clarkb | 19:07 |
| *** nati_ueno has quit IRC | 19:07 | |
| jnoller | fungi: there's a lot of copy/past for the jenkins instances, almost done | 19:07 |
| clarkb | ya, making it more templated is on the todo list somewhere | 19:08 |
| *** burt1 has quit IRC | 19:09 | |
| marun | feature request - allow sorting by bug priority on this page: http://status.openstack.org/reviews | 19:09 |
| fungi | marun: https://git.openstack.org/cgit/openstack-infra/reviewday | 19:10 |
| fungi | (in case you decide you want to take a crack at it) | 19:10 |
| marun | fungi: i'm not sure you want me messing with js ;) | 19:10 |
| marun | fungi: but thank you for the pointer :) | 19:11 |
| fungi | marun: couldn't be any worse than me messing with js ;) | 19:11 |
| fungi | but if you want to file an actual feature request, bug against openstack-ci and tag with reviewday i guess (doesn't look like reviewday has its own separate project on lp) | 19:12 |
| marun | fungi: ok, cool | 19:12 |
| jnoller | blech | 19:12 |
| jnoller | looks like my patch set went into the ether | 19:13 |
| *** rcleere has joined #openstack-infra | 19:13 | |
| *** wenlock has quit IRC | 19:14 | |
| fungi | the git gods are not shining their benevolence down upon you? | 19:14 |
| *** che-arne has quit IRC | 19:15 | |
| jnoller | no | 19:16 |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM) flavor to Rackspace IAD https://review.openstack.org/70192 | 19:17 |
| jnoller | ok | 19:17 |
| jnoller | Learn Squash the hard Way | 19:17 |
| *** masayukig has joined #openstack-infra | 19:18 | |
| *** wenlock has joined #openstack-infra | 19:19 | |
| *** nati_ueno has joined #openstack-infra | 19:19 | |
| *** nati_ueno has quit IRC | 19:19 | |
| *** vkozhukalov has joined #openstack-infra | 19:19 | |
| fungi | i'll babysit it once nodepool picks up the change | 19:19 |
| *** nati_ueno has joined #openstack-infra | 19:20 | |
| jnoller | :fistbump: | 19:21 |
| jnoller | tell me if you have problems | 19:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Add Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM) flavor to Rackspace IAD https://review.openstack.org/70192 | 19:21 |
| fungi | jnoller: your assistance is much appreciated! i'll be sure to keep you apprised | 19:22 |
| jnoller | fungi: I definitely need to know if they're better / worse / same performance wise so I can do some digging with westmaas et al | 19:23 |
| westmaas | yessir | 19:23 |
| fungi | jnoller: we'll hopefully have some relative figures soon | 19:23 |
| *** masayukig has quit IRC | 19:24 | |
| westmaas | fungi: are those generated dynamically on graphite somewhere? | 19:24 |
| westmaas | thought i remembered that in the past | 19:24 |
| *** ryanpetrello has quit IRC | 19:24 | |
| jnoller | sdague just linked to the GH repo he's using to build the graphs he has | 19:24 |
| westmaas | ok cool | 19:24 |
| fungi | westmaas: yes, we have some statsd collection going on for job run times, but it's in need of someone reworking the query. sdague's is built from data he's collecting out of the jenkins masters instead so may differ a little, not sure yet | 19:25 |
| jnoller | westmaas: but my scroll back not go far enough | 19:25 |
| westmaas | jnoller: https://github.com/sdague/tempest-timings | 19:26 |
| westmaas | get more scrollback! | 19:26 |
| westmaas | fungi: cool thanks for the info | 19:26 |
| *** hashar has joined #openstack-infra | 19:26 | |
| clarkb | if there is a drastic improvement or slowdown, we should be able to see it relatively easily just looking at job run times | 19:27 |
| *** ryanpetrello has joined #openstack-infra | 19:27 | |
| *** bknudson has quit IRC | 19:28 | |
| fungi | 2014-01-30 00:17:28,446 ERROR nodepool.image.build.hpcloud-az2.bare-precise: /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill': execution expired (Timeout::Error) | 19:29 |
| westmaas | yeah looks like its the variability thats killer. | 19:29 |
| fungi | i think hpcloud-az2 bare-precise images are failing to build because they're running too slowly and puppet is timing out mid-execution | 19:29 |
| *** bknudson has joined #openstack-infra | 19:29 | |
| clarkb | fungi: wow | 19:30 |
| *** amcrn has quit IRC | 19:30 | |
| fungi | at least that's my best guess from the image.log | 19:30 |
| *** e0ne has quit IRC | 19:30 | |
| fungi | that's the log line immediately following... | 19:30 |
| fungi | sudo puppet apply --modulepath=/root/config/modules:/etc/puppet/modules -e 'class {'\''openstack_project::bare_slave'\'': }' | 19:30 |
| *** e0ne has joined #openstack-infra | 19:31 | |
| fungi | or is that a timeout accessing the file bucket on the master? | 19:32 |
| fungi | https://groups.google.com/forum/#!topic/puppet-users/jhhYR74-DB8 | 19:32 |
| jog0 | sdague: https://review.openstack.org/#/c/70143/ | 19:35 |
| *** e0ne has quit IRC | 19:35 | |
| fungi | jnoller: a devstack-precise-check image is building on rax-iad. once we have nodes launched from that i'll check their run logs to make sure nothing's terribly wrong, but we should be collecting timing data soon for those (in the next hour or so) | 19:36 |
| *** zehicle_at_dell has joined #openstack-infra | 19:38 | |
| pleia2 | oh hey, fungi, clarkb - thoughts on penciling in a bug day for tuesday at 1700? | 19:40 |
| pleia2 | usually would have been this week, but so much traveling of people | 19:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for ceilometer bug 1274490 https://review.openstack.org/70110 | 19:41 |
| *** fbo_away is now known as fbo | 19:42 | |
| *** andreaf has quit IRC | 19:43 | |
| clarkb | pleia2: next week tuesday wfm | 19:43 |
| fungi | pleia2: sounds fine to me (though i try to spread bug triage out and do a little each day anyway, a little catch-up on long-standing bugs never hurts) | 19:43 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for test_server_rescue race bug 1226412 https://review.openstack.org/69530 | 19:44 |
| *** thuc has quit IRC | 19:46 | |
| jnoller | fungi: thank you! | 19:46 |
| pleia2 | fungi: yeah, I have noticed we've been doing a better job at keeping up with bugs thanks to you | 19:46 |
| *** thuc has joined #openstack-infra | 19:46 | |
| *** thedodd has joined #openstack-infra | 19:46 | |
| clarkb | yes fungi has done an awesome job | 19:46 |
| fungi | pleia2: clarkb: it's thanks to all of us | 19:47 |
| clarkb | also with the gerrit service accounts | 19:47 |
| *** hogepodge has quit IRC | 19:47 | |
| *** gyee has joined #openstack-infra | 19:47 | |
| fungi | but yeah, the influx of gate bugs to triage would be insurmountable if we weren't addressing and redirecting them quickly | 19:47 |
| jog0 | wow its amazing to watch zuul when we fix the gate bugs | 19:47 |
| fungi | jog0: it is a rather beautiful sight | 19:47 |
| jog0 | fungi: yup | 19:48 |
| jog0 | we are well on our way to hitting a new record for merges today | 19:48 |
| jog0 | https://github.com/openstack/openstack/graphs/commit-activity | 19:48 |
| *** thuc_ has joined #openstack-infra | 19:48 | |
| *** thuc has quit IRC | 19:48 | |
| jog0 | 140 is the number to beat | 19:48 |
| *** thuc_ has quit IRC | 19:49 | |
| *** thuc has joined #openstack-infra | 19:49 | |
| *** hogepodge has joined #openstack-infra | 19:51 | |
| *** lcheng has joined #openstack-infra | 19:52 | |
| *** jooools has quit IRC | 19:52 | |
| *** fbo is now known as fbo_away | 19:52 | |
| sdague | jog0: so the issue is we actually really need to be able to merge 200 a day by i3 | 19:53 |
| jog0 | sdague: thats it :) | 19:53 |
| jog0 | sdague: I think thats possible | 19:53 |
| sdague | maybe... we've got a lot of resets happening right now | 19:53 |
| clarkb | or, you know we could encourage people to do less bursty work (I know that that will never happen) | 19:54 |
| jog0 | you happned to check in a bad period | 19:54 |
| clarkb | sdague: its one reset, the graph jsut doesn't show it well | 19:54 |
| jog0 | sdague: agreed we have a ways to go, but we are making progress | 19:54 |
| clarkb | as it is eating up the stuff behind | 19:54 |
| sdague | clarkb: it's the cloud... it's all bursts :) | 19:55 |
| sdague | be more elastic! | 19:55 |
| clarkb | sdague: honestly, it would help reviewers a bunch too imo | 19:55 |
| sdague | agreed, but such it is | 19:55 |
| *** eharney has quit IRC | 20:00 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add new job to build the javamelody gerrit plugin https://review.openstack.org/70014 | 20:00 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 20:00 |
| zaro | clarkb: ^ | 20:00 |
| clarkb | thanks | 20:00 |
| *** nati_ueno has quit IRC | 20:02 | |
| *** andre__ has quit IRC | 20:03 | |
| *** praneshp has quit IRC | 20:03 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: replace jeepyb's trivial rebase gerrit hook with copyAllScoresOnTrivialRebase https://review.openstack.org/69800 | 20:04 |
| *** amcrn has joined #openstack-infra | 20:04 | |
| _david_ | clarkb, ^d asked on #gerrit about submit on_behalf_on feature, I wonder if it can be also interesting for us: https://gerrit-review.googlesource.com/#/c/54153 ? | 20:05 |
| *** lcheng has quit IRC | 20:05 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add the time at which this page was generated https://review.openstack.org/70143 | 20:05 |
| _david_ | zaro, the fix for javamelody plugin worked for you? | 20:06 |
| clarkb | _david_: I don't think so because of the CLA | 20:06 |
| zaro | _david_: the build fix? | 20:06 |
| clarkb | ideally every individual pushes their own changes to track CLA stuff properly | 20:06 |
| _david_ | zaro, yes? | 20:07 |
| zaro | _david_: yes, it build ok now. | 20:07 |
| _david_ | zaro, cool, does it also works? | 20:07 |
| *** ArxCruz has quit IRC | 20:07 | |
| zaro | _david_: actually haven't deployed that build. | 20:07 |
| clarkb | _david_: oh I misread that | 20:08 |
| clarkb | _david_: it might be interesting to have, zuul could submit on behalf of the person that approved the change | 20:08 |
| _david_ | clarkb, exactly | 20:08 |
| clarkb | but that would mean giving that person submit access right? | 20:08 |
| zaro | _david_: you just reverted a change so i didn't bother trying. but i probably should :) | 20:08 |
| clarkb | we only give zuul submit access | 20:08 |
| _david_ | clarkb, yeah, that what i did in the first place, but if we have better idea, ...? | 20:09 |
| _david_ | clarkb, then we could skip this test (one if less ;-) | 20:09 |
| *** praneshp has joined #openstack-infra | 20:09 | |
| _david_ | clarkb, anyway, about WIP feature: DRAFT workflow overloading did work! | 20:10 |
| *** vkozhukalov has quit IRC | 20:10 | |
| clarkb | nice | 20:10 |
| _david_ | clarkb, so if this change get merged, then we have WIP workflow in core when we disable DRAFT workflow: https://gerrit-review.googlesource.com/#/c/54088 | 20:10 |
| clarkb | awesome | 20:11 |
| clarkb | _david_: how does it break replication? | 20:11 |
| clarkb | we use replication a lot | 20:11 |
| zaro | _david_:w00t! can you still disable both? wip & draft? | 20:11 |
| _david_ | clarkb, as i said yesterday, i went further and implemented WIP workflow in addition to DRAFT: https://gerrit-review.googlesource.com/#/c/50254 | 20:11 |
| *** yolanda has quit IRC | 20:12 | |
| clarkb | oh and hmm you can't go back to WIP again? that will be annoying | 20:12 |
| *** zehicle_at_dell is now known as zehicle | 20:12 | |
| *** zehicle is now known as zehicle_ | 20:12 | |
| _david_ | zaro, with first change not: either wip or draft, with second change yes, you can disable draft and you must not enable WIP, | 20:12 |
| _david_ | but it would be a trivial change to the first concept: instead of boolean, have enum with DRAFT, WIP, DISABLE, done | 20:13 |
| zaro | _david_: cool. other's might like the WIP vote option. i know the gerrit guys prefer that. | 20:13 |
| _david_ | clarkb, if you have draft changes, you can remove them | 20:13 |
| *** lcestari has quit IRC | 20:13 | |
| clarkb | fungi: are we ready to give sdage d-g core? | 20:14 |
| clarkb | the d-g core group add change merged so I am filling in the list of members | 20:14 |
| _david_ | zaro, yeah, and they don't want to loose DRAT | 20:14 |
| fungi | clarkb: yes, let's do that (i can add him now). were we still waiting on mordred to cast a vote on the ml, or are we taking his silence as assent? | 20:15 |
| clarkb | fungi: I just did it | 20:15 |
| *** nati_ueno has joined #openstack-infra | 20:15 | |
| fungi | perfect | 20:15 |
| clarkb | fungi: he did respond so it was unanimous | 20:16 |
| * zaro will be afk, back in a few hrs. | 20:16 | |
| fungi | excellent | 20:16 |
| *** mrodden1 has quit IRC | 20:17 | |
| *** mrodden has joined #openstack-infra | 20:17 | |
| *** virmitio has quit IRC | 20:18 | |
| clarkb | sdague: you have d-g core rights now, have fun | 20:19 |
| *** melwitt has joined #openstack-infra | 20:19 | |
| *** masayukig has joined #openstack-infra | 20:19 | |
| sdague | clarkb: oh, nice, thansk :) | 20:20 |
| *** sparkycollier has joined #openstack-infra | 20:22 | |
| *** masayukig has quit IRC | 20:23 | |
| *** rlandy|bbl is now known as rlandy | 20:24 | |
| *** markmc has joined #openstack-infra | 20:25 | |
| jog0 | clarkb: so the e-r failing thread is back | 20:25 |
| jog0 | not sure why its failing to comment on this guys patch | 20:25 |
| *** talluri has quit IRC | 20:26 | |
| mtreinish | jog0: I did like that instead of running recheck he abandons it and brings it back though | 20:27 |
| *** praneshp has quit IRC | 20:27 | |
| sdague | jog0: we still doing diagnostics when we lose logs? | 20:27 |
| jog0 | sdague: we only log if its console log right now | 20:27 |
| sdague | jog0: not on irc? | 20:28 |
| jog0 | double checking what we have | 20:28 |
| jog0 | mtreinish: heh yeah | 20:28 |
| jog0 | sigh | 20:28 |
| clarkb | jog0: first thing to check would be if the logs for those tests got indexed | 20:28 |
| jog0 | build_change:"57524" AND build_uuid:"c50ce3b6d3d34e268e1eba7d6660d99b" | 20:28 |
| *** DinaBelova is now known as DinaBelova_ | 20:30 | |
| clarkb | I am seeing a lot of log files including console.html | 20:30 |
| clarkb | and they seem to have been indexed about 2 minutes after job competion | 20:30 |
| *** jcooley_ has quit IRC | 20:31 | |
| *** jcooley_ has joined #openstack-infra | 20:32 | |
| jog0 | yeah it looks we log if any file is missing | 20:32 |
| jog0 | in irc | 20:32 |
| jog0 | so they appear to all be there | 20:32 |
| *** VijayT has quit IRC | 20:33 | |
| clarkb | so I am guessing bug in e-r | 20:33 |
| clarkb | are we using uuids to get the exact job yet? | 20:33 |
| jog0 | can you search the e-r logs for anything about build 57524 | 20:33 |
| clarkb | if possible we are still getting data for other builds that don't match then failing to report | 20:33 |
| jog0 | clarkb: short uuids yeah | 20:33 |
| clarkb | jog0: that should be good neough | 20:33 |
| clarkb | yeah I can grep the logs | 20:33 |
| jog0 | we do patch set, revision, and short build uuid | 20:33 |
| jog0 | thanks | 20:33 |
| *** praneshp has joined #openstack-infra | 20:33 | |
| clarkb | the logging ends a week ago | 20:37 |
| clarkb | am I looking at the right thing? | 20:37 |
| jog0 | o_0 | 20:37 |
| jog0 | sdague: you touched the logs format a little right | 20:38 |
| sdague | ? | 20:38 |
| clarkb | 2014-01-23 21:24:30,941 is the last timestamp in the e-r log | 20:39 |
| clarkb | the debug log is working fine | 20:39 |
| jog0 | sdague: e-r stopped logging | 20:40 |
| jog0 | or at least has appeared to | 20:40 |
| clarkb | fungi: I have put the version of the SCP plugin we want and grabbed jenkins 1.543 deb on jenkins.o.o and jenkins01 | 20:40 |
| clarkb | fungi: so should be able to do the restarts quickly when not about to afk | 20:40 |
| sdague | jog0: I don't know, look into how the config deploy works | 20:41 |
| fungi | great--thanks clarkb! | 20:41 |
| sdague | because we pick up logging config from there IIRC | 20:41 |
| sdague | unless my logging cleanups stopped processing that | 20:41 |
| sdague | though I though I just added defaults | 20:41 |
| sdague | jog0: at any rate, I won't really be able to get into this any time soon | 20:42 |
| clarkb | I don't see any changes to the logging.config in openstack-infra/config | 20:42 |
| clarkb | did you change somethign in e-r that expected a change? | 20:43 |
| sdague | fungi: so I have no idea where the short runs came from before in my timing scrape scripts, as I just cleaned up a couple of things and regenerated, and now they aren't there | 20:46 |
| sdague | https://dague.net/testing/tempest-full.png | 20:46 |
| *** CaptTofu has quit IRC | 20:47 | |
| clarkb | e-r is running on status.o.o now right? that is where I am looking and there is a running process | 20:48 |
| *** denis_makogon_ has joined #openstack-infra | 20:50 | |
| clarkb | yup looks like status | 20:50 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add new CLI Reference document https://review.openstack.org/70213 | 20:50 |
| *** eharney has joined #openstack-infra | 20:51 | |
| clarkb | right so, I think that the logging change sdague made may be the problem. You are getting loggers before setting config on loggers, does that update them appropriately? | 20:52 |
| sdague | clarkb: that's a good question | 20:52 |
| clarkb | I think you might want to do the per module logging stuff after reading the logging config in | 20:52 |
| sdague | clarkb: ok, just change the order? | 20:52 |
| clarkb | sdague: I think so | 20:52 |
| clarkb | sdague: see the warning text at http://docs.python.org/2/howto/logging.html#configuring-logging | 20:55 |
| clarkb | so that may be part of the problem as anything not called out in the config explicitly will be disabled | 20:55 |
| *** FallenPegasus has joined #openstack-infra | 21:01 | |
| *** _david_ has quit IRC | 21:01 | |
| * fungi sighs loudly... | 21:01 | |
| fungi | 2014-01-30 19:39:14,824 ERROR nodepool.image.build.rax-iad.devstack-precise-check: + curl -O https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py | 21:02 |
| fungi | 2014-01-30 19:39:14,824 ERROR nodepool.image.build.rax-iad.devstack-precise-check: install_puppet.sh: line 23: curl: command not found | 21:02 |
| *** denis_makogon_ is now known as denis_makogon | 21:02 | |
| fungi | jnoller: westmaas: ^ do you have different preinstalled packages in the new ubuntu precise image? | 21:03 |
| westmaas | :/ | 21:04 |
| fungi | the other precise images we were using did have curl available by default | 21:04 |
| westmaas | not intentionally that I know of. | 21:04 |
| *** bgresham has joined #openstack-infra | 21:04 | |
| westmaas | will ask around | 21:04 |
| fungi | we can certainly adjust our scripts to make sure it gets installed or to fall back on wget or something, but it would be nice if the list of initial packages between them were consistent | 21:05 |
| westmaas | makes sense | 21:05 |
| westmaas | the other precise image on RS had it installed, right? | 21:05 |
| fungi | yep | 21:06 |
| *** HenryG has quit IRC | 21:06 | |
| fungi | same script trying to run it here works fine on the other precise base image | 21:06 |
| fungi | it's basically the first real step in the bootstrap... we log in and retrieve the more complex bootstrapping scripts which do the real work | 21:07 |
| *** changbl has quit IRC | 21:07 | |
| westmaas | fungi: question sent out, will let you know | 21:07 |
| fungi | westmaas: thanks | 21:08 |
| *** coolsvap is now known as coolsvap_away | 21:10 | |
| marun | so… when git_clone gets called by devstack, what is supposed to happen on failure? | 21:10 |
| fungi | marun: normally that function looks for a local cached copy of the repo and uses that, and if it doesn't find one then it falls back on the baked in git urls i believe | 21:11 |
| marun | fungi: fair enough, but what if there is an error during the clone? | 21:11 |
| marun | fungi: i.e. network failure or bad repo uri or whatever | 21:11 |
| fungi | marun: it just fails out | 21:12 |
| clarkb | and fwiw we set error on clone to true in the gate so may mask bugs there | 21:12 |
| fungi | marun: for our test slaves, we pre-cache all our repositories on disk so that it's less likely to be of issue there | 21:12 |
| marun | fungi: do you mean 'continues running like nothing happened' or...? | 21:12 |
| marun | ah, error on clone... | 21:12 |
| marun | so that's not set by default because...? | 21:12 |
| marun | i'm not sure I understand the case of that not being a reason to stop devstack | 21:12 |
| fungi | marun: it's a good question... might be worth bringing up with the devstack developers (dtroyer et al in #openstack-qa) | 21:13 |
| clarkb | not default because then you would need to clone everything | 21:13 |
| *** samuelbercovici has quit IRC | 21:13 | |
| marun | clarkb: wait, error on clone triggers an error if the destination does not exist... | 21:13 |
| clarkb | it means fail devstack if devstack clones something | 21:13 |
| marun | clarkb: if an error occurs during clone , things just keep running from what i can see | 21:13 |
| jgriffith | fungi: clarkb jenkins doesn't seem to be detecting the recheck on this: https://review.openstack.org/#/c/69787/ | 21:14 |
| jgriffith | fungi: is there some way for me to kick it, other than upload new version? | 21:14 |
| fungi | marun: clarkb: right, we intentionally cause devstack to fail if it tries to clone from the network, so that we find out about and correct any missing repository cache on our test slaves | 21:14 |
| marun | fungi: ah, sorry, wrong channel | 21:14 |
| jgriffith | or what it "thinks" is a new version | 21:14 |
| clarkb | marun: right I am saying that is possible because the code path isnt tested | 21:14 |
| marun | fungi: makes sense, the problem i'm seeing is unrelated | 21:14 |
| fungi | jgriffith: looking | 21:15 |
| jnoller | fungi: yeah, just saw this; westmaas and I are digging | 21:15 |
| fungi | jgriffith: drafts strike again | 21:15 |
| fungi | jgriffith: there's a child depending on that change, and it's a draft | 21:16 |
| jgriffith | fungi: oh? | 21:16 |
| jgriffith | fungi: crap | 21:16 |
| jgriffith | fungi: and it's MINE | 21:16 |
| jgriffith | fungi: I didn't know that was an issue... silly moi | 21:16 |
| jgriffith | fungi: so eitehr abandon or release the draft and it should go? | 21:16 |
| jgriffith | "either" even | 21:17 |
| fungi | jgriffith: low-priority bug in zuul, when it inspects a change it needs to look at the relationships between all depending and dependant changes for a given change to determine whether they should be enqueued as well. when it tries to insect the draft gerrit refuses to grant it access, and that causes an exception in the enqueuing call | 21:17 |
| jgriffith | fungi: nice | 21:17 |
| jgriffith | fungi: ok... I'll take care of that | 21:17 |
| jgriffith | fungi: thanks for pointing that out to me | 21:18 |
| fungi | jgriffith: if you get the authr to un-draft the draft change and set it work-in-progress instead, that would work | 21:18 |
| jgriffith | suppose I'll go back to using myown remote | 21:18 |
| jgriffith | it was me so no problem :) | 21:18 |
| fungi | drafts are terrible. we have a work-in-progress status to serve the useful purposes draft might hope to offer | 21:19 |
| jgriffith | fair enough | 21:19 |
| jgriffith | fungi: thanks, I'll stop using them :) | 21:19 |
| fungi | recommended ;) | 21:20 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add option to elastic-recheck-graph to pick a single queue https://review.openstack.org/70218 | 21:21 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add tab to only plot gate failures to status.o.o/elastic-recheck/ https://review.openstack.org/69735 | 21:21 |
| jgriffith | fungi: deleted, neverusing draft again :) | 21:21 |
| jgriffith | although a year from now I'll forget and do it again most likely | 21:22 |
| *** rlandy has quit IRC | 21:22 | |
| jog0 | sdague: really dumb question | 21:22 |
| jog0 | or clarkb | 21:23 |
| jog0 | in modules/openstack_project/manifests/static.pp I want to add a second call to elastic-recheck-graph | 21:23 |
| jog0 | do I just make that string twice as long? | 21:23 |
| *** virmitio has joined #openstack-infra | 21:23 | |
| *** markwash has quit IRC | 21:24 | |
| *** julim has quit IRC | 21:26 | |
| *** rnirmal has joined #openstack-infra | 21:28 | |
| jnoller | fungi: as a stop-gap, which script can I tweak to apt-get install curl? | 21:29 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: If curl is missing try wget instead https://review.openstack.org/70221 | 21:29 |
| fungi | jnoller: that one ^ | 21:29 |
| jnoller | hahahahaha | 21:29 |
| fungi | jnoller: do those images have wget? just curious | 21:30 |
| jnoller | let me confirm | 21:30 |
| openstackgerrit | A change was merged to openstack/requirements: Ignore egg-info directory https://review.openstack.org/68425 | 21:30 |
| openstackgerrit | A change was merged to openstack/requirements: psutil >= 1.1.0 is actually on PyPI https://review.openstack.org/68946 | 21:30 |
| jnoller | yes | 21:31 |
| jnoller | they do | 21:31 |
| fungi | i think we defaulted to curl because it used to be distributed as part of the perl interpreter binary packages, but these days it's evolved out into its own separate package anyway | 21:31 |
| *** jcooley_ has quit IRC | 21:32 | |
| pleia2 | heh, in one of my patches I had to install wget on fedora (it comes with curl, not wget, and we use both in devstack scripts) | 21:32 |
| *** jcooley_ has joined #openstack-infra | 21:32 | |
| pleia2 | s/devstack/devstack-gate | 21:32 |
| fungi | pleia2: our bootstrap script should probably support using whichever is available and then our puppetry should probably make sure both get installed | 21:33 |
| pleia2 | ah, that would be nice | 21:34 |
| openstackgerrit | DennyZhang proposed a change to openstack-infra/gitdm: update personal profile https://review.openstack.org/68772 | 21:34 |
| fungi | some of our puppet manifests specifically call out to install curl, so i think it's mostly covered (but with that and wget being preinstalled in some places it's hard to know when we're missing it in places where we use it) | 21:37 |
| fungi | clarkb: if you're around still and think https://review.openstack.org/70221 is safe, that would allow us to keep some momentum on the new performance images | 21:39 |
| *** nati_uen_ has joined #openstack-infra | 21:39 | |
| *** praneshp has quit IRC | 21:39 | |
| clarkb | not really around but lgtm | 21:41 |
| fungi | k | 21:41 |
| jnoller | fungi clarkb: I'm going to stay on top of this until we get everything good to go :] | 21:41 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add tab to only plot gate failures to status.o.o/elastic-recheck/ https://review.openstack.org/69735 | 21:41 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add option to elastic-recheck-graph to pick a single queue https://review.openstack.org/70218 | 21:41 |
| jnoller | though I should be writing blueprints | 21:41 |
| *** nati_ueno has quit IRC | 21:42 | |
| *** hashar has quit IRC | 21:42 | |
| openstackgerrit | A change was merged to openstack-infra/config: If curl is missing try wget instead https://review.openstack.org/70221 | 21:47 |
| *** jnoller has quit IRC | 21:50 | |
| *** markwash has joined #openstack-infra | 21:50 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Generate two elastic-recheck-graph json files all, and gate only https://review.openstack.org/70223 | 21:52 |
| *** rcleere has quit IRC | 21:52 | |
| jog0 | sdague: I think I have all the patches up https://review.openstack.org/#/q/status:open++branch:master+topic:graph-gate,n,z | 21:52 |
| *** emagana has quit IRC | 21:53 | |
| *** burt1 has joined #openstack-infra | 21:53 | |
| *** Sukhdev has joined #openstack-infra | 21:53 | |
| jog0 | I am least sure about the config change | 21:54 |
| *** praneshp has joined #openstack-infra | 21:54 | |
| *** emagana has joined #openstack-infra | 21:54 | |
| *** lcheng has joined #openstack-infra | 21:55 | |
| *** sparkycollier has quit IRC | 21:55 | |
| *** mfer has quit IRC | 21:55 | |
| *** sparkycollier has joined #openstack-infra | 21:55 | |
| sdague | ok, I'm +2 on patch 1 | 21:55 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make IRC messages provide more context https://review.openstack.org/68811 | 21:56 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add the time at which this page was generated https://review.openstack.org/70143 | 21:56 |
| *** lcheng_ has joined #openstack-infra | 21:56 | |
| *** krtaylor has joined #openstack-infra | 21:56 | |
| jog0 | sdague: thanks | 21:56 |
| jog0 | mtreinish jeblair:^ | 21:57 |
| sdague | once the config bits get mergable, ping me for patch 2 | 21:57 |
| jog0 | sdague: will do, figure it may talk infra a few days to get to it being they are so busy | 21:58 |
| sdague | yep | 21:58 |
| sdague | that's fine | 21:58 |
| jog0 | yup | 21:58 |
| mtreinish | jog0: this one? https://review.openstack.org/#/q/status:open++branch:master+topic:graph-gate,n,z | 21:58 |
| jog0 | mtreinish: yeah | 21:58 |
| *** reed has joined #openstack-infra | 21:58 | |
| *** jcooley_ has quit IRC | 21:58 | |
| jog0 | sdague: ahh rebase | 21:58 |
| sdague | heh | 21:59 |
| jog0 | oh wait that was a network timeout | 21:59 |
| jog0 | http://logs.openstack.org/18/70218/2/gate/gate-elastic-recheck-pep8/7f31eee/console.html | 21:59 |
| jog0 | thats new | 21:59 |
| jog0 | launchpad | 21:59 |
| mtreinish | jog0: its a 503... | 21:59 |
| *** lcheng has quit IRC | 22:00 | |
| jog0 | mtreinish: I am gonna just rekick | 22:00 |
| *** rcleere has joined #openstack-infra | 22:01 | |
| sdague | jog0: I'm not convinced it's new | 22:01 |
| sdague | I saw that yesterday | 22:01 |
| jog0 | damn you launchpad | 22:03 |
| jog0 | message:"HTTP error 503 while getting" AND message:"launchpadlib" | 22:04 |
| sdague | fungi: did something change in the mirror today? | 22:04 |
| jog0 | 4 hits in 24 hours | 22:04 |
| jog0 | sdague: strange why are we not pulling from pypi.openstack.org | 22:04 |
| sdague | because right now all the nova jobs are failing because psutils > 1.1 can't be found | 22:04 |
| mtreinish | jog0: it looked like it was going right to launchpad not the mirror | 22:04 |
| sdague | jog0: is it on pypi? | 22:04 |
| fungi | someone merged that requirements change i guess | 22:04 |
| mtreinish | jog0: I should read first... | 22:04 |
| jog0 | sdague: no | 22:05 |
| sdague | fungi: explain? | 22:05 |
| jog0 | err the binary isn't hosted on pypi | 22:05 |
| fungi | sdague: i saw a requirements change just go through for psutils>=1.1 is on pypi | 22:05 |
| fungi | i didn't look too closely. digging it up now | 22:05 |
| sdague | fungi: ok | 22:05 |
| *** rfolco has quit IRC | 22:05 | |
| sdague | well, we can't seem to find it | 22:06 |
| fungi | https://git.openstack.org/cgit/openstack/requirements/commit/?id=ee07adf62d874266be433bf1c4c2073f4460f661 | 22:06 |
| fungi | oh, psutil, not psutils | 22:06 |
| jog0 | https://pypi.python.org/pypi/psutil | 22:07 |
| fungi | do we use both? | 22:07 |
| sdague | http://logs.openstack.org/31/69631/2/gate/gate-tempest-dsvm-neutron/93e36ee/console.html#_2014-01-30_20_31_31_295 | 22:07 |
| fungi | http://pypi.openstack.org/openstack/psutil/ | 22:07 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Generate two elastic-recheck-graph json files all, and gate only https://review.openstack.org/70223 | 22:08 |
| sdague | fungi: so they removed 1.1.1? | 22:08 |
| sdague | and we didn't have 1.2.1 yet? | 22:09 |
| jog0 | sdague: lolz | 22:09 |
| fungi | the previous requirements were <1.0 so we weren't mirroring anything newer than that | 22:09 |
| jog0 | ahh | 22:09 |
| jog0 | fungi: any ideas about http://logs.openstack.org/18/70218/2/gate/gate-elastic-recheck-pep8/7f31eee/console.html | 22:09 |
| fungi | then it switched to >1.1.0 so the mirror run started | 22:09 |
| jog0 | why we are not suing pypi.openstack.org | 22:10 |
| jog0 | using | 22:10 |
| sdague | fungi: ok, and it just didn't complete yet? | 22:10 |
| fungi | and we mirrored the latest version at 21:47 according to the filesystem timestamps | 22:10 |
| sdague | so the problem was an non overlaping requirements change | 22:10 |
| fungi | but prior to that, some tests were run with the new requirements against the not-yet-updated mirror | 22:10 |
| sdague | ok | 22:10 |
| sdague | yeh, it emptied the gate rather nicely | 22:11 |
| jog0 | sdague: haha | 22:11 |
| sdague | I think we failed out about 30 jobs on that | 22:11 |
| fungi | because mirror updating includes a bit of a lag | 22:11 |
| fungi | so probably we shouldn't make non-overlapping requirements changes | 22:11 |
| jog0 | sdague: want to add the e-r fingerprint | 22:11 |
| *** nati_uen_ has quit IRC | 22:11 | |
| sdague | jog0: if you file the bug | 22:11 |
| fungi | otherwise there will be some period where our mirror does not include the stated requirements | 22:12 |
| fungi | and then, *boom* | 22:12 |
| *** nati_ueno has joined #openstack-infra | 22:12 | |
| sdague | yeh, seems like an issue actually, as it means a non overlapping requirements job enters the gate, and there is a known window where it might go terribly wrong | 22:13 |
| sdague | because it's telling everyone behind it to do a thing | 22:13 |
| sdague | which they can't do | 22:13 |
| sdague | until it merges | 22:13 |
| sdague | this actually explains today's carnage a lot | 22:14 |
| sdague | jog0: yuo got a bug #? | 22:14 |
| fungi | yeah, maybe we can add a check to openstack/requirements changes to confirm that they overlap when changing an existing requirement | 22:15 |
| fungi | we'd need to special-case addition and removal of requirements since those don't necessarily pose the same issues | 22:16 |
| jog0 | 1274731 | 22:16 |
| jog0 | https://bugs.launchpad.net/openstack-ci/+bug/1274731 | 22:16 |
| jog0 | that bug need a little more explanation | 22:16 |
| jog0 | bbiab food time | 22:16 |
| *** mrda_away is now known as mrda | 22:16 | |
| sdague | I think launchpad is just falling over in general today | 22:18 |
| jog0 | sdague: I blame mordred and his hiring habbits | 22:18 |
| fungi | he has single-handedly gutted them | 22:19 |
| openstackgerrit | A change was merged to openstack/requirements: Use new hplefthandclient https://review.openstack.org/65179 | 22:19 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add in bug for the psutil upgrade https://review.openstack.org/70231 | 22:22 |
| sdague | jog0: I updated the bug with more words | 22:23 |
| anteaya | is mordred hiring hobbits now? | 22:23 |
| lifeless | fungi: is 'timeout: x' in jjb seconds or minutes? | 22:25 |
| fungi | lifeless: minutes | 22:27 |
| *** sparkycollier has quit IRC | 22:27 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add an undercloud test for tripleo. https://review.openstack.org/70233 | 22:29 |
| lifeless | fungi: ^ There's a little too much redundancy in the tripleo.yaml definition for my taste, but I don't know jjb well enough to avoid it. | 22:29 |
| lifeless | fungi: clarkb: ^ I'd like to land that, subjection to any changes you need, asap, so that we can work on our next phase | 22:29 |
| *** cadenzajon has quit IRC | 22:32 | |
| *** rcleere has quit IRC | 22:35 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: turn on config drive for all jobs https://review.openstack.org/70235 | 22:37 |
| dansmith | sdague: I guess I don't see which bug is being discussed here that looks like what I'm hitting | 22:37 |
| sdague | dansmith: https://bugs.launchpad.net/openstack-ci/+bug/1274731 | 22:37 |
| dansmith | sdague: your e-r change looks for a log message that I don't have, so I figured that was different | 22:38 |
| dansmith | unless I'm missing something | 22:39 |
| sdague | oh, what's your log message? | 22:39 |
| dansmith | lpsyutogs.openstack.org/31/69631/2/gate/gate-tempest-dsvm-neutron/93e36ee/console.html | 22:39 |
| dansmith | failure to find neutron-db-manage | 22:39 |
| sdague | dansmith: right, but yuo need to go up to root cause | 22:39 |
| sdague | neutron-db-manage wasn't there because neutron didn't install | 22:39 |
| sdague | and it didn't install because of the psutil issue | 22:40 |
| *** dims has quit IRC | 22:40 | |
| dansmith | sdague: oh, I see it, I must've typoed my first search | 22:41 |
| dansmith | sdague: so, no point in reverifying until something lands? | 22:41 |
| sdague | the mirror is updated now | 22:41 |
| sdague | you just got caught in the wake | 22:41 |
| dansmith | okay | 22:41 |
| sdague | it's a whole new kind of fail we didn't know we had :) | 22:42 |
| sdague | *sigh* | 22:42 |
| openstackgerrit | David Boucha proposed a change to openstack-infra/config: Create a seafile instance. https://review.openstack.org/54030 | 22:42 |
| sdague | clarkb / fungi: https://review.openstack.org/#/c/70235/ - that should let neutron work again | 22:43 |
| *** jhesketh has joined #openstack-infra | 22:43 | |
| *** mriedem has quit IRC | 22:43 | |
| lifeless | sdague: is that turning off metadata server for some jobs? | 22:44 |
| lifeless | sdague: huh, metadata doesn't need key injection | 22:45 |
| fungi | lifeless: didn't you say something about configdrive not being workable for ironic? | 22:45 |
| fungi | we need to leave it =0 on the ironic tests, right? | 22:45 |
| sdague | lifeless: well it's been triggering it | 22:45 |
| sdague | so... salv-orlando has some root cause on this. I need to drop out | 22:46 |
| sdague | I won't be able to be the middle man on this one | 22:46 |
| lifeless | sdague: I've put an altertive up in the review | 22:46 |
| lifeless | sdague: ack | 22:46 |
| lifeless | salv-orlando: ping ^ | 22:46 |
| *** melwitt has quit IRC | 22:46 | |
| lifeless | fungi: *baremetal don't support config-drive, thats right | 22:46 |
| *** lcheng_ has quit IRC | 22:46 | |
| *** dangers is now known as dangers_away | 22:47 | |
| fungi | okay, glad to know i haven't beered away all my brain cells yet | 22:47 |
| sdague | lifeless: well baremetal solved this by turning off neutron | 22:47 |
| sdague | lifeless: get in -neutron | 22:48 |
| lifeless | I'm in there :) | 22:48 |
| salv-orlando | lifeless: I'm everywhere. choose your favourite room | 22:48 |
| salv-orlando | lifeless: if that's about the pmtud thing, I forgot and I'll do that soon | 22:49 |
| lifeless | salv-orlando: no, its about https://review.openstack.org/#/c/70235/ | 22:49 |
| *** markmc has quit IRC | 22:50 | |
| salv-orlando | lifeless: we don't have config files from the gate unfortunately | 22:50 |
| salv-orlando | but if you look at the logs you can definely see nbd devices being mounted | 22:50 |
| *** dizquierdo_afk is now known as dizquierdo | 22:51 | |
| *** jooools has joined #openstack-infra | 22:52 | |
| lifeless | salv-orlando: yes, I know | 22:52 |
| *** slong has joined #openstack-infra | 22:52 | |
| lifeless | salv-orlando: the default is on | 22:52 |
| lifeless | salv-orlando: the config setting I have put in the review, shows how to disable | 22:52 |
| salv-orlando | lifeless: agreed. It's up to the infra team now to decide whether they want to disable injection now or later, I think | 22:52 |
| salv-orlando | whatever prevents nbd device mount will work for me | 22:53 |
| salv-orlando | neutron was unable to merge anything for 11 days now, and the backlog is terrible | 22:53 |
| dhellmann | mordred: http://lists.openstack.org/pipermail/openstack-dev/2014-January/026014.html in case you didn't see it | 22:55 |
| *** dims has joined #openstack-infra | 22:56 | |
| lifeless | salv-orlando: https://review.openstack.org/70239 should fix it too | 22:56 |
| harlowja | hey guys, testr question, how do u view the tests which were skipped, --list-skipped? | 22:57 |
| salv-orlando | lifeless: thanks. I am happy with that too, I just have a question regarding impact on existing deployment. Added a comment to the patch. | 22:59 |
| lifeless | harlowja: good q! there isn't a trivial answer for that - sorry. THis will get it testr last --subunit | subunit-filter -e -f | subunit-ls | 22:59 |
| *** prad has quit IRC | 22:59 | |
| harlowja | thats fine lifeless :) | 22:59 |
| *** nati_uen_ has joined #openstack-infra | 23:00 | |
| *** nati_uen_ has quit IRC | 23:01 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Remove reference to Developer Setup in docs https://review.openstack.org/69494 | 23:01 |
| *** nati_uen_ has joined #openstack-infra | 23:01 | |
| *** lcheng has joined #openstack-infra | 23:03 | |
| *** nati_ueno has quit IRC | 23:03 | |
| *** jamielennox|away is now known as jamielennox | 23:04 | |
| *** emagana has quit IRC | 23:06 | |
| *** emagana has joined #openstack-infra | 23:07 | |
| *** emagana has quit IRC | 23:07 | |
| *** jooools has quit IRC | 23:08 | |
| lifeless | clarkb: fungi: sorry to be a nag - https://review.openstack.org/#/c/70233/ | 23:09 |
| *** jhesketh has quit IRC | 23:09 | |
| fungi | lifeless: sorry, was in meetings and clarkb's afk | 23:09 |
| *** dcramer_ has quit IRC | 23:10 | |
| fungi | looking | 23:10 |
| fungi | lifeless: you only want to collect the console and pip.log from check-tripleo-seed-precise? | 23:11 |
| fungi | oh, i guess that's all gate-tripleo-deploy collected anyway | 23:12 |
| lifeless | fungi: there's a separate patch to change whats collected | 23:13 |
| lifeless | maybe I should rollon top of that | 23:13 |
| lifeless | lets see | 23:14 |
| *** emagana has joined #openstack-infra | 23:14 | |
| fungi | yeah, those are going to logically merge-conflict even if not necessarily literally | 23:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add in bug for the psutil upgrade https://review.openstack.org/70231 | 23:14 |
| lifeless | we need | 23:14 |
| lifeless | https://review.openstack.org/#/c/67685/ | 23:14 |
| lifeless | https://review.openstack.org/#/c/69824/ | 23:14 |
| jog0 | sdague: thanks | 23:14 |
| lifeless | https://review.openstack.org/#/c/67958/ | 23:15 |
| lifeless | I'll make mine depend on 69824 in a sec | 23:15 |
| *** burt1 has quit IRC | 23:16 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add an undercloud test for tripleo. https://review.openstack.org/70233 | 23:17 |
| lifeless | fungi: there, updated mine | 23:18 |
| lifeless | fungi: if we can have all four, we'll be super happy | 23:18 |
| *** jgrimm has quit IRC | 23:18 | |
| stevebaker | I was wondering, is there anyone on the infra team who works for rackspace? | 23:19 |
| lifeless | they used to be :P | 23:19 |
| fungi | stevebaker: not in core/root, but we do have heavy contributors from rackspace | 23:19 |
| fungi | and yes, the more venerable half of our core/root were rackspace who later went to hpcloud (one of whom still works there) | 23:20 |
| stevebaker | ok. I ask because they run a CD heat against rackspace cloud, which just broke because identity service != keystone | 23:21 |
| *** david-lyle has joined #openstack-infra | 23:21 | |
| fungi | heh | 23:21 |
| stevebaker | it might be nice to get a gate job which tests heat against rackspace cloud | 23:21 |
| mrda | hey stevebaker, we also have jhesketh and mattoliverau from Rackspace who hack on -infra | 23:21 |
| *** hogepodge has quit IRC | 23:22 | |
| *** sandywalsh has quit IRC | 23:22 | |
| jog0 | sdague: looks like we are very for off your magical 200 number for the day | 23:22 |
| fungi | i don't think we're going to get a gate job testing the internal functionality of any particular cloud provider (though that's a good candidate for third-party testing) | 23:22 |
| jog0 | https://github.com/openstack/openstack/graphs/commit-activity | 23:22 |
| sdague | jog0: yeh, well that requirements issue | 23:23 |
| *** mattoliverau has joined #openstack-infra | 23:23 | |
| jog0 | sdague: that put us back an hour? | 23:23 |
| clarkb | fungi: i am headed home want to put 01 in shutdown mode now? | 23:23 |
| *** sarob has quit IRC | 23:23 | |
| jog0 | sdague: but yeah a more work to go | 23:23 |
| stevebaker | jog0: so third-party testing would have to be separate from the infra infra? | 23:23 |
| jog0 | sdague: want a fun review https://review.openstack.org/#/c/57577/ | 23:24 |
| fungi | clarkb: can do | 23:24 |
| lifeless | stevebaker: lol, perhaps they should use keystone? | 23:24 |
| jog0 | stevebaker: I am not the best person to answer that -- but the short is infra is overloaded as is | 23:24 |
| jog0 | sdague: I am temped to -1 my own patch ^ | 23:25 |
| dolphm | lifeless: that's the obvious solution ;) | 23:25 |
| jog0 | dolphm: https://review.openstack.org/#/c/57577/ your vote would be greatly appreciated | 23:25 |
| lifeless | stevebaker: tests that can't be reproduced by a developer aren't suitable for gating tests | 23:25 |
| lifeless | stevebaker: because developers can't fix them | 23:25 |
| jog0 | dolphm: re keystone bind host | 23:25 |
| stevebaker | lifeless: they most definitely should use keystone! | 23:26 |
| lifeless | stevebaker: thats why baremetal has had such a long ramp up - we've had to make sure /all/ our stack is entirely reproducible by anyone | 23:26 |
| dolphm | jog0: ooh interesting, i was thinking about something like this yeseterday | 23:26 |
| stevebaker | lifeless: In theory all a developer would need is a rackspace account | 23:26 |
| fungi | to phrase differently, the openstack project infrastructure is not going to run tests to confirm that we don't break implementations of particular cloud providers. instead we test reference implementations and encourage those providers to use similar service models to the reference | 23:26 |
| fungi | (or evolve the reference if reasonable to do so) | 23:26 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 23:27 |
| stevebaker | btw this was caused by us ditching keystone v2 for v3 | 23:27 |
| jog0 | dolphm: yeah I am just trying to get this issue out of the gate -- one less bug to deal with in gate. | 23:27 |
| mattoliverau | stevebaker: I'll ping the rackspace devs and see what's going on. | 23:28 |
| dolphm | jog0: i don't think that's a workaround, i think that's a perfectly viable solution for the gate | 23:28 |
| dolphm | (but i actually didn't think it would work for some reason...) | 23:28 |
| dolphm | happy to see a +1 from jenkins! | 23:28 |
| stevebaker | mattoliverau: Thanks. I'm taking the position that if it is not tested it is already broken | 23:29 |
| jog0 | dolphm: solution for gate, not all production cases | 23:29 |
| dolphm | jog0: no, but in production, you have more options to choose from, and we can address that via docs | 23:29 |
| jog0 | dolphm: right | 23:29 |
| *** hogepodge has joined #openstack-infra | 23:30 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add some dependencies required by toci https://review.openstack.org/67685 | 23:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable precise-backports on tripleo test nodes https://review.openstack.org/67958 | 23:30 |
| sdague | +A on the keystone port devstack patch | 23:31 |
| jog0 | sdague: woot | 23:31 |
| sdague | thanks clarkb | 23:31 |
| dolphm | jog0: sdague: clarkb: danke! | 23:31 |
| lifeless | stevebaker: to run a heat? I thought you needed admin creds ? | 23:31 |
| *** david-lyle has quit IRC | 23:31 | |
| lifeless | fungi: I disagree with your different phrasing :) | 23:31 |
| *** dkranz has quit IRC | 23:32 | |
| dolphm | lifeless: heat is using a v3-only api to take advantage of delegation to produce ec2 creds | 23:32 |
| lifeless | fungi: since you are *now* running a cloud provider (tripleo)'s tests. | 23:32 |
| *** nati_uen_ has quit IRC | 23:32 | |
| lifeless | fungi: we're not a commercial provider, sure :> | 23:32 |
| stevebaker | lifeless: yes, I must assume wait conditions don't work on rax | 23:33 |
| *** sandywalsh has joined #openstack-infra | 23:35 | |
| fungi | lifeless: i was considering the tripleo cloud to be part of the reference implementation, but agreed i was ambiguous | 23:35 |
| lifeless | fungi: ah, fair enough | 23:35 |
| lifeless | fungi: so https://review.openstack.org/#/c/70233/ - I rebased for the dependency, it needs +A again ;) | 23:35 |
| *** nati_ueno has joined #openstack-infra | 23:36 | |
| openstackgerrit | A change was merged to openstack-infra/config: Updated the logs directory for tripleo jobs https://review.openstack.org/69824 | 23:36 |
| fungi | lifeless: yes, i'm working my way through them | 23:36 |
| * fungi is failing at multitasking too many things at once | 23:36 | |
| lifeless | fungi: ack - thank you! | 23:36 |
| lifeless | fungi: I'll pop out for an errand, give you some space | 23:36 |
| *** lcheng has quit IRC | 23:37 | |
| *** denis_makogon has quit IRC | 23:37 | |
| clarkb | ok back, the right side of my face is still numb but that shouldn't affect my typing | 23:38 |
| clarkb | looks like about a half an hour to 01 upgrade | 23:38 |
| fungi | clarkb: stop trying to type with your face and you should be fine | 23:38 |
| clarkb | fungi: my plan is stop jenkins, put scp plugin in place, install deb | 23:39 |
| fungi | sounds great | 23:39 |
| openstackgerrit | A change was merged to openstack-infra/config: Add an undercloud test for tripleo. https://review.openstack.org/70233 | 23:39 |
| *** CaptTofu has joined #openstack-infra | 23:40 | |
| clarkb | fungi: fwiw I have decided to ignore the zmq plugin updates for now. Want reviews on my change to fix it and I think there is less of a rush for that | 23:40 |
| *** lcheng has joined #openstack-infra | 23:41 | |
| fungi | clarkb: okay | 23:43 |
| fungi | in other news, we have a ton of the new rackspace performance nodes in use... i'll check up on the jobs running on them shortly | 23:43 |
| clarkb | nice | 23:43 |
| *** lcheng has quit IRC | 23:43 | |
| clarkb | so the curl thing was the only hiccup? | 23:43 |
| fungi | we've had nodes from it for about an hour since adding wget as an option in the install_puppet.sh script, yeah | 23:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: only print unrecognized errors in gate https://review.openstack.org/70258 | 23:45 |
| *** zzelle has joined #openstack-infra | 23:45 | |
| zzelle | hi, everyone | 23:45 |
| clarkb | zzelle: hello | 23:46 |
| fungi | zzelle: hi there | 23:46 |
| zzelle | does someone know why we can request https://review/openstack.org/projects/ ? | 23:46 |
| *** FallenPegasus has quit IRC | 23:46 | |
| *** cody-somerville has joined #openstack-infra | 23:47 | |
| clarkb | fungi: https://jenkins04.openstack.org/job/check-tempest-dsvm-full/2663/console seems to have run relatively quickly. should be done in just over 50 minutes | 23:47 |
| clarkb | though we did remove the xml v3 tests so I don't know how quick that is relative to new runtimes /me checks other nodes | 23:47 |
| fungi | zzelle: i don't understand your question | 23:48 |
| clarkb | https://jenkins03.openstack.org/job/gate-tempest-dsvm-full/3892/ is one running on iad without pvhvm | 23:48 |
| zzelle | oups, my mistake | 23:48 |
| zzelle | does someone know why we can NOT request https://review/openstack.org/projects/ ? | 23:48 |
| fungi | zzelle: i still don't understand your questions. is that meant to be a url? | 23:49 |
| zzelle | it provides the same result as gerrit ls-projects but through http | 23:49 |
| clarkb | zzelle: is that part of the new rest API? | 23:49 |
| clarkb | our gerrit is old enough to not have the REST api | 23:49 |
| fungi | zzelle: we're running gerrit 2.4.4 with some local patches | 23:49 |
| zzelle | arg | 23:49 |
| clarkb | zzelle: feel free to help zaro in the upgrade process | 23:49 |
| zzelle | i am no more java compliant :) | 23:50 |
| *** reed has quit IRC | 23:50 | |
| zzelle | my aim was to allow to use git-review with http:// remote | 23:51 |
| *** flaper87 is now known as flaper87|afk | 23:51 | |
| clarkb | zzelle: you should be able to do that without listing the projects | 23:51 |
| clarkb | I would continue to use a .gitreview file and handle the http case too | 23:51 |
| zzelle | to get minimal support for http, you are right | 23:52 |
| *** oubiwann_ has quit IRC | 23:52 | |
| fungi | clarkb: i was interested to discover that you can in fact push changes for review to our gerrit version via http, as long as it goes to the gerrit web service rather than hitting the apache shadow url pattern | 23:52 |
| zzelle | for the full package, it's not possible | 23:52 |
| clarkb | fungi: nice | 23:52 |
| clarkb | zzelle: why? what does listing the projects give git review? | 23:53 |
| * clarkb checks if ssh does ls-projects in git-review | 23:53 | |
| clarkb | oh hey it does, hrm what is that for? | 23:53 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 23:53 | |
| zzelle | ls-projects is for git review -s | 23:53 |
| clarkb | oh it tests the remote | 23:53 |
| *** Sukhdev has quit IRC | 23:53 | |
| zzelle | (ok, it's a little bit dumm) | 23:53 |
| clarkb | I am sure there is a way to do that sanely without the /projects path. Maybe HEAD the path to the project | 23:54 |
| zzelle | IMHO, there are others ways to do check it | 23:54 |
| zzelle | exactly | 23:54 |
| clarkb | eg curl -XHEAD https://review.openstack.org/openstack/nova | 23:54 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Capture disk space available at the start of a run https://review.openstack.org/69156 | 23:54 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 23:55 | |
| zzelle | currently the only really needed modif is to allow to get commit hook through http | 23:55 |
| zzelle | i am currently working on it | 23:55 |
| clarkb | zzelle: at one point I was pushing for git-review to bundle the commit hook | 23:55 |
| clarkb | it would reduce a few round trips that git-review needs to make to install that for each repo | 23:56 |
| clarkb | westmaas: fungi: my first unscientific data collection shows an improvement with pvhvm | 23:57 |
| *** pcrews has quit IRC | 23:58 | |
| *** pcrews has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
