*** nati_ueno has quit IRC | 00:00 | |
clarkb | http://cloud-images.ubuntu.com/releases/14.04/release/ is where I got the image btw | 00:01 |
---|---|---|
*** praneshp_ has joined #openstack-infra | 00:02 | |
*** praneshp has quit IRC | 00:04 | |
*** praneshp_ is now known as praneshp | 00:04 | |
*** thuc has joined #openstack-infra | 00:05 | |
jogo | clarkb: for now I am just wondering what a default devstack dsvm full run looks like | 00:06 |
*** thuc has quit IRC | 00:06 | |
clarkb | jogo: oh I think we are a long way away from that | 00:06 |
*** thuc has joined #openstack-infra | 00:07 | |
clarkb | first the puppet/nodepool stuff must be made to work | 00:07 |
clarkb | then we need jobs | 00:07 |
clarkb | then we can try that | 00:07 |
clarkb | jogo: you can always run it manually though | 00:07 |
*** thuc_ has joined #openstack-infra | 00:08 | |
jogo | clarkb: true on both counts. but its nice to have all the logs and whatnot in our fancy system. | 00:09 |
*** thuc__ has joined #openstack-infra | 00:09 | |
jogo | anyway its getting cold out here time to go inside for a bit | 00:09 |
*** sarob has joined #openstack-infra | 00:10 | |
*** zns has quit IRC | 00:12 | |
*** thuc has quit IRC | 00:12 | |
mordred | clarkb: might be time to start accelerating our parallel puppet3 master | 00:12 |
*** SumitNaiksatam has quit IRC | 00:12 | |
mordred | clarkb: nibalizer was going to work on that some wasn't he? | 00:12 |
*** thuc_ has quit IRC | 00:12 | |
*** ilives has joined #openstack-infra | 00:13 | |
clarkb | mordred: well our slave stuff should be puppet3 clean coutesy of pleia2 and fedora 20 | 00:13 |
mordred | oh right | 00:13 |
clarkb | mordred: I think its more likely that we need to modify puppet to handle new and fun special caes | 00:13 |
mordred | so we might not be too bad puppet-wise on trusty | 00:13 |
*** thuc__ has quit IRC | 00:13 | |
mordred | we also probably want to stop adding a ppa in the launch script | 00:13 |
*** thuc has joined #openstack-infra | 00:14 | |
*** sarob has quit IRC | 00:14 | |
*** ilives has quit IRC | 00:17 | |
*** ilives has joined #openstack-infra | 00:17 | |
*** praneshp has quit IRC | 00:17 | |
*** nati_ueno has joined #openstack-infra | 00:17 | |
*** melwitt has quit IRC | 00:18 | |
*** praneshp has joined #openstack-infra | 00:19 | |
*** thuc has quit IRC | 00:19 | |
*** reed has quit IRC | 00:19 | |
*** gokrokve has joined #openstack-infra | 00:21 | |
*** matsuhashi has joined #openstack-infra | 00:23 | |
clarkb | fyi https://bugs.launchpad.net/python-glanceclient/+bug/1309272 | 00:26 |
uvirtbot | Launchpad bug 1309272 in python-glanceclient "Required options should be listed as required and error when not provided." [Undecided,New] | 00:26 |
clarkb | only one more devstack precise image to build with the get pip and vm image caching change | 00:30 |
clarkb | it is looking good | 00:30 |
*** derekh has quit IRC | 00:31 | |
sdague | salv-orlando: awesome great | 00:33 |
*** bhuvan has quit IRC | 00:33 | |
nibalizer | there is a puppet 3 master change proposed | 00:34 |
clarkb | sdague: ok all devstack-precise* images have been rebuilt | 00:34 |
nibalizer | i think its failing puppet parser test right now | 00:34 |
nibalizer | i can look at that later on tonight | 00:34 |
clarkb | sdague: is the pip thing being tracked in e-r? we should see that fall to zero over the next few hours as nodes are cycled out | 00:34 |
sdague | clarkb: great | 00:34 |
sdague | yes, we've got it in er | 00:35 |
sdague | Bug 1299176 - Rackspace network periodically blips and won't let us download get-pip from github | 00:35 |
mnaser | how many vm's does the testing infra usually use up to? | 00:35 |
uvirtbot | Launchpad bug 1299176 in openstack-ci "Rackspace network periodically blips and won't let us download get-pip from github" [High,Fix released] https://launchpad.net/bugs/1299176 | 00:35 |
clarkb | mnaser: the graph at http://status.openstack.org/zuul/ shows you. I think we peak at about 1k due to quotas | 00:35 |
anteaya | clarkb are we up to 1k now? | 00:36 |
anteaya | cool | 00:36 |
clarkb | mnaser: you can always copy that image url and edit it to see other data | 00:36 |
mnaser | and 8gb vms each? | 00:36 |
clarkb | mnaser: most are 8GB, some are 30GB but enforced to 8GB because we only get enough cpu time from that provider on the large 30GB images | 00:36 |
clarkb | anteaya: no we appear to be at ~400 right now | 00:37 |
mnaser | so up to 8tb of memory all just testing openstack | 00:37 |
mnaser | pretty cool | 00:37 |
anteaya | sorry I meant our upper limit | 00:39 |
mordred | nibalizer: hey - you know things ... | 00:39 |
anteaya | didn't know we could go as high as 1k | 00:39 |
*** atiwari has quit IRC | 00:39 | |
anteaya | yay to our providers | 00:39 |
mordred | nibalizer: how do I set postgresql::globals::version so that it's set when I call postgresql::server | 00:40 |
*** markmcclain has quit IRC | 00:41 | |
*** mrodden has quit IRC | 00:41 | |
*** dstanek has quit IRC | 00:43 | |
*** changbl has joined #openstack-infra | 00:44 | |
clarkb | mordred: I think you class { 'postgresql::globals': version => 'someversion', } class {'postgresql::server': require => Class['postgresql::globals'], } | 00:44 |
*** dstanek has joined #openstack-infra | 00:44 | |
clarkb | mordred: that should ensure that the globals version value is set before postgresql::server does anything with it | 00:45 |
* clarkb & | 00:45 | |
sdague | https://bugs.launchpad.net/openstack-ci/+bug/1290562 - adrian left at comment at the bottom of that that new setuptools fixes it | 00:46 |
uvirtbot | Launchpad bug 1290562 in openstack-ci "failing pypy jobs with "option --single-version-externally-managed not recognized"" [High,In progress] | 00:46 |
*** SumitNaiksatam has joined #openstack-infra | 00:48 | |
openstackgerrit | Sean Dague proposed a change to openstack/requirements: require new setuptools https://review.openstack.org/88474 | 00:51 |
sdague | mordred: you feel like pushing +A on this - https://review.openstack.org/#/c/88302/ - jeblair and SergeyLukjanov were +2 previously | 00:52 |
sdague | will prevent us from landing a patch in tempest that breaks icehouse | 00:52 |
mordred | nibalizer, clarkb: looks like very few patches get us to working trusty :) | 00:52 |
mordred | sdague: looking | 00:52 |
openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Ruby evaluates empty strings as true https://review.openstack.org/88475 | 00:53 |
*** gokrokve_ has joined #openstack-infra | 00:53 | |
nibalizer | wooot | 00:55 |
*** gokrokve has quit IRC | 00:56 | |
openstackgerrit | A change was merged to openstack-infra/config: add stable icehouse jobs for tempest master https://review.openstack.org/88302 | 00:57 |
*** dcramer_ has joined #openstack-infra | 00:57 | |
*** dcramer_ has joined #openstack-infra | 00:57 | |
*** timrc is now known as timrc-afk | 00:58 | |
nibalizer | mordred: re: postgres, do you have an example where the code is? | 00:59 |
*** marcoemorais has quit IRC | 01:00 | |
sdague | mordred: thanks | 01:01 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Changes to get nodepool working with trusty https://review.openstack.org/88477 | 01:01 |
mordred | nibalizer: ^^ look at that - what I did, I think, is ugly | 01:01 |
mordred | but seems to physically work | 01:01 |
adam_g | how often do puppet changes get applied to the slave images ? | 01:02 |
nibalizer | clarkb steered you right according to this https://github.com/puppetlabs/puppetlabs-postgresql#class-postgresqlglobals | 01:02 |
nibalizer | i've never worked with this pattern before | 01:02 |
mordred | adam_g: when they are created | 01:04 |
mordred | nibalizer: ossum | 01:05 |
nibalizer | mordred: since this merged, the postgres module should be dealing with quotes correctly now | 01:05 |
nibalizer | https://github.com/puppetlabs/puppetlabs-postgresql/commit/e26036c176870539c26e568c986de6c68be17fec | 01:05 |
mordred | adam_g: we run puppet once when we are creating the base image from which we make the slave nodes | 01:05 |
adam_g | mordred, right.. is that a daily/nightly affair? | 01:05 |
mordred | adam_g: daily-ish | 01:05 |
adam_g | ah, cool | 01:05 |
clarkb | I just did the ones ironic uses | 01:06 |
clarkb | you should be good | 01:06 |
clarkb | assuming I didnt derp | 01:07 |
adam_g | clarkb, oh, party | 01:07 |
jeblair | clarkb: i don't want us to use manual image uploads for anything | 01:07 |
clarkb | jeblair ok | 01:07 |
*** dims has quit IRC | 01:08 | |
clarkb | even for using current nodepool image building process? | 01:08 |
nibalizer | mordred: also your changes for trusty and my puppet3 branch look similarish | 01:08 |
jeblair | i think we should get dib working and use that | 01:08 |
clarkb | k | 01:08 |
clarkb | jogo: there's your answer | 01:09 |
openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 01:11 |
jeblair | i mean, it shouldn't take long to add dib and glance support | 01:11 |
nibalizer | jeblair: clarkb mordred ^ | 01:11 |
nibalizer | will need to rebase against mordreds trusty changes if those land first | 01:11 |
nibalizer | but for now it should be green | 01:11 |
jeblair | clarkb: also, figuring out, eg, how networking works using a bare cloud image in hp and rax is a separate problem | 01:12 |
jeblair | clarkb: so i think manually uploading an image and understanding that is time well spent | 01:12 |
jeblair | but i don't want to incorporate manual image uploads into the current process (as soon as anything about it needs tweaking or is otherwise unusual, we either need to start documenting that process or building our own images, and that would end up being duplicated effort) | 01:13 |
*** wchrisj has joined #openstack-infra | 01:15 | |
clarkb | jeblair: ok, so we can test and work stuff out using the image I uploaded but no nodepool or jenkins/zuul testing on that | 01:15 |
clarkb | seems sane | 01:15 |
jeblair | clarkb: ++ and that happening in parallel to mordred's dib will get us there faster | 01:15 |
jeblair | nibalizer: were you going to respond to the ps7 questions? | 01:16 |
*** asselin has quit IRC | 01:16 | |
jeblair | clarkb, mordred: i do think we should both try to get gerrit 2.8 on cloud db as well as on trusty | 01:17 |
mordred | jeblair: ++ | 01:17 |
*** asselin has joined #openstack-infra | 01:17 | |
mordred | jeblair: I've got 80% patch for nodepool + dib | 01:17 |
mordred | maybe I should publish the work in progress... | 01:17 |
jeblair | mordred: prolly a good idea | 01:18 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Changes to get nodepool working with trusty https://review.openstack.org/88477 | 01:18 |
mordred | jeblair, nibalizer: some more fixes ^^ | 01:18 |
mordred | jeblair: so far, running our stuff on a node based off of the bare trusty image has not been to yucky | 01:18 |
jeblair | mordred: cool | 01:19 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 01:20 |
mordred | jeblair, clarkb ^^ | 01:20 |
mordred | that does NOT WORK | 01:20 |
mordred | I got as far as realizing I needed to figure out how to get a glance client | 01:20 |
mordred | and then I haven't been able to work on it any more | 01:20 |
jeblair | mordred: do you want to continue working on the dib part and i can start on glance? | 01:20 |
*** dims has joined #openstack-infra | 01:20 | |
mordred | jeblair: yah - sure enough | 01:20 |
nibalizer | jeblair: okay now im getting it | 01:21 |
anteaya | mordred 46482 has a whitespace error | 01:21 |
*** wchrisj has quit IRC | 01:21 | |
anteaya | the rest I am trying to follow along on | 01:21 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support for disk-image-builder in nodepool https://review.openstack.org/88479 | 01:22 |
mordred | anteaya: oh - 46482 is terrible | 01:22 |
mordred | I mean, it's quite literally half-done | 01:22 |
anteaya | oh | 01:22 |
mordred | so is 88479 | 01:22 |
jeblair | mordred: is there more dib work to do? | 01:22 |
anteaya | k | 01:22 |
mordred | I'm just pushing them up for communication | 01:22 |
mordred | jeblair: lemme look real quick - I think the dib bits are largely done | 01:22 |
jeblair | (i intended to volunteer for parallel work, not to try to take it over if you're done with the dib) | 01:23 |
mordred | yah | 01:23 |
mordred | yes. step 1 of dib work should be servicable. it's very non-optimal, as it does not contain the desired de-coupling of image and provider | 01:24 |
*** nosnos has joined #openstack-infra | 01:24 | |
mordred | but so we build/upload 4 times - who cares | 01:24 |
mordred | the next set of DIB work is the patch to config above | 01:24 |
mordred | which needs to be updated as well | 01:24 |
mordred | I should maybe make an etherpad... | 01:24 |
*** thuc has joined #openstack-infra | 01:24 | |
openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 01:25 |
jeblair | mordred: ok; well, i'm not going to work on it right now, so we can regroup later; happy to pitch in but don't want to step on toes | 01:25 |
mordred | jeblair: pitching in is totally welcome - even making sure what's there that's mostly completeish other than upload isn't crazy | 01:26 |
mordred | and I'm about to stop hacking for the night as well :) | 01:26 |
*** nati_ueno has quit IRC | 01:26 | |
*** UtahDave has quit IRC | 01:27 | |
*** mriedem has joined #openstack-infra | 01:28 | |
*** thuc has quit IRC | 01:29 | |
*** dstanek has quit IRC | 01:34 | |
*** Sukhdev has quit IRC | 01:34 | |
*** dcramer_ has quit IRC | 01:34 | |
*** zns has joined #openstack-infra | 01:34 | |
*** saschpe has quit IRC | 01:36 | |
*** saschpe has joined #openstack-infra | 01:38 | |
*** yaguang has joined #openstack-infra | 01:43 | |
*** dstanek has joined #openstack-infra | 01:43 | |
mordred | jeblair, clarkb: I just had a possibly terrible idea | 01:44 |
mordred | what if run_remote_puppet.sh was actually a Makefile (I know we were carping about this a little bit earlier) - | 01:45 |
mordred | but one with a phony target that ran the equiv of the run_ssh function | 01:45 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add a fingerprint for bug 1309252 https://review.openstack.org/88462 | 01:47 |
uvirtbot | Launchpad bug 1309252 in tempest "HTTP errors observed in test_load_balancer_basic" [High,Triaged] https://launchpad.net/bugs/1309252 | 01:47 |
mordred | something like this: http://paste.openstack.org/show/76197/ | 01:49 |
* mordred was thinking about how to try to generalize the single depenedncy list | 01:49 | |
*** khyati_ has quit IRC | 01:49 | |
mordred | ok. NOW I'm going away | 01:49 |
mordred | and will eat salmon | 01:49 |
anteaya | enjoy salmon | 01:53 |
greghaynes | Its funny, beause there was a puppet slogan for a while about "because ssh and a for loop doesnt cur it" | 01:56 |
clarkb | printed on tshirts | 01:58 |
* nibalizer has this shirt | 01:58 | |
greghaynes | I think I somehow have that shirt, even | 01:58 |
nibalizer | mordred: i've thought that you do want to have a more general solution to who goes first, but yea dependency resolution is hard | 01:59 |
nibalizer | makefiles dont seem like a horrible idea | 01:59 |
nibalizer | maybe fabric, since it is actually for orchestration | 02:00 |
nibalizer | actually it looks like fabric doesn't do dependency resolution | 02:02 |
*** rlandy has quit IRC | 02:06 | |
*** nati_ueno has joined #openstack-infra | 02:11 | |
*** aysyd has quit IRC | 02:13 | |
*** Longgeek has joined #openstack-infra | 02:14 | |
*** zns has quit IRC | 02:16 | |
*** rwsu has quit IRC | 02:22 | |
*** amcrn has quit IRC | 02:32 | |
*** pcrews has quit IRC | 02:35 | |
*** nati_ueno has quit IRC | 02:41 | |
*** nati_ueno has joined #openstack-infra | 02:43 | |
*** Ryan_Lane has quit IRC | 02:47 | |
mnaser | check-grenade-dsvm-partial-ncpu hasn't passed for the past ~5 hours | 02:48 |
mnaser | what's usually the thing to do in these cases | 02:48 |
*** CaptTofu has quit IRC | 02:48 | |
*** harlowja is now known as harlowja_away | 02:49 | |
clarkb | mnaser: usually figure out why, then fix it if possible | 02:49 |
*** CaptTofu has joined #openstack-infra | 02:49 | |
clarkb | if a fix is not practical we can make the test non voting or stop running it completely | 02:49 |
mnaser | alrighty, i'll try to dig out the root cause, would like my patch to go in but keeps failing there (but it doesn't touch any part of that) | 02:49 |
mnaser | https://bugs.launchpad.net/nova/+bug/1254890 -- i'll dig in | 02:50 |
uvirtbot | Launchpad bug 1254890 in nova ""Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures" [High,Confirmed] | 02:50 |
*** alexpilotti has quit IRC | 02:50 | |
mnaser | thanks btw, clarkb :) | 02:52 |
*** jhesketh has joined #openstack-infra | 02:52 | |
*** nati_ueno has quit IRC | 02:53 | |
jhesketh | Afternoon | 02:53 |
*** CaptTofu has quit IRC | 02:53 | |
mikal | jhesketh: heya | 02:53 |
clarkb | mnaser: oh you know what | 02:53 |
*** Longgeek has quit IRC | 02:54 | |
clarkb | so we released icehouse today and started updating the upgrade tests | 02:54 |
*** Longgeek has joined #openstack-infra | 02:54 | |
mnaser | aaaah | 02:54 |
*** Longgeek has quit IRC | 02:54 | |
clarkb | I bet the partial ncpu isn't doing that right | 02:54 |
mnaser | should we look into moving it to non voting | 02:54 |
*** Longgeek has joined #openstack-infra | 02:54 | |
clarkb | or possibly we have actually broken the ability to upgrade from icehouse to master | 02:54 |
clarkb | mnaser: ideally no, since this is likely to be a real bug in novas ability to do upgrades in thsi way | 02:55 |
mnaser | clarkb: let me do a tiny bit of initial digging | 02:55 |
mnaser | and i'll report back, cause it might be a minor thing.. or not :p | 02:55 |
clarkb | mnaser: http://logs.openstack.org/07/83207/21/check/check-grenade-dsvm-partial-ncpu/983dbdd/logs/old/screen-n-cpu.txt.gz?level=ERROR maybe? | 02:56 |
mnaser | eek | 02:57 |
clarkb | all of those errors start after the upgrade | 02:57 |
clarkb | so ya I think this may be a legit fail | 02:57 |
clarkb | those milestone branches should be killed >_> | 02:57 |
*** coolsvap|afk is now known as coolsvap | 02:58 | |
*** gokrokve_ has quit IRC | 02:58 | |
*** gyee has quit IRC | 02:59 | |
*** gokrokve has joined #openstack-infra | 03:05 | |
*** matsuhashi has quit IRC | 03:09 | |
*** matsuhashi has joined #openstack-infra | 03:09 | |
*** matsuhashi has quit IRC | 03:09 | |
*** jhesketh has quit IRC | 03:09 | |
*** gokrokve has quit IRC | 03:09 | |
mnaser | clarkb: it looks like n-cpu is not starting at all post-upgrade | 03:09 |
clarkb | mnaser: it keeps running | 03:11 |
clarkb | mnaser: n-cpu is the one service not restarted during that test | 03:11 |
mnaser | clarkb: i think that might be the issue, because it gets rpc calls that are newer than the one it supports... let me verify | 03:12 |
clarkb | so you need to look at timestamps in the old/screen-n-cpu.txt file to correlate when that happens | 03:12 |
clarkb | and it looks like upgrade happens around 01:54ish | 03:12 |
clarkb | errors show up 02:00 ish | 03:12 |
mnaser | i'm curious if any other service does rpc calls towards n-cpu | 03:13 |
clarkb | I think cond and sch? | 03:15 |
clarkb | I am not super familiar with nova's rpc setup though | 03:15 |
mnaser | clarkb: i think n-sch is making calls to n-cpu which it does not support | 03:15 |
mnaser | more specifically, n-cpu is getting calls of version 2.37 which it doesn't support | 03:16 |
mordred | greghaynes: yah. it would be great if it were better tahn ssh and a for loop :) | 03:16 |
clarkb | right, but that test exists to specifically make sure it supports it | 03:16 |
clarkb | mnaser: so I think this is valid, nova messed up somewhere | 03:16 |
mordred | also - for the record - they did not have any pacific salmon :( | 03:16 |
mordred | we'll be having beef instead | 03:17 |
mnaser | so basically, you need to restart your n-cpu for upgrades.. or this has to be addressed (latter obviously) | 03:17 |
clarkb | mordred: now I am going to eat my lox | 03:17 |
clarkb | mnaser: right, this test exists to make sure that you don't have to restart your ncpu | 03:17 |
mnaser | hmm | 03:17 |
clarkb | the only reason we have this test is ^ | 03:17 |
clarkb | I think icehouse being off on milestone proposed and master being juno with havana being the stable branch allowed something to slip through | 03:18 |
clarkb | mordred: can we kill milestone proposed | 03:18 |
clarkb | mordred: seems like it is good for breaking things | 03:18 |
*** nosnos has quit IRC | 03:19 | |
mnaser | um | 03:20 |
mnaser | i think there is an oops here | 03:20 |
clarkb | yup I agree | 03:20 |
*** wenlock has joined #openstack-infra | 03:20 | |
*** chandan_kumar has joined #openstack-infra | 03:20 | |
mordred | clarkb: I think so - although I think I need to retrigger the merge propose jobs | 03:20 |
mnaser | clarkb: i think if onud the reason | 03:20 |
clarkb | I think us only testing havana to master allowed the switch to icehouse to master to fubar | 03:21 |
clarkb | mordred: no no I mean completely | 03:21 |
clarkb | mordred: like don't use them ever | 03:21 |
clarkb | mordred: we should cut and icehouse branch after FF | 03:21 |
mordred | I'm down with that | 03:21 |
mnaser | clarkb: https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L248 | 03:21 |
mordred | it's ttx you have to convince | 03:21 |
clarkb | then release off of something on that branch 6 weeks later | 03:21 |
mordred | I've never thought they were a good idea | 03:21 |
mnaser | i think icehouse should be set and cut off there | 03:21 |
mnaser | it was never cutoff.. | 03:21 |
clarkb | but that way we can test this without crazyness | 03:21 |
mordred | but - until ttx says otherwise, I will support his special brand of crazy | 03:22 |
clarkb | mnaser: ugh | 03:22 |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: add run-tests.sh https://review.openstack.org/88430 | 03:22 |
clarkb | mordred: the problem with the crazy is it breaks CI | 03:22 |
mordred | yah | 03:22 |
clarkb | mordred: we stop testing stuff | 03:22 |
*** matsuhashi has joined #openstack-infra | 03:25 | |
*** dstanek has quit IRC | 03:26 | |
mnaser | clarkb: https://github.com/openstack/nova/commit/13cb4e7d3fb7b119f7ee01d8aa3f265d4be20090 | 03:26 |
mnaser | is compute being set to icehouse-compat? | 03:27 |
*** matsuhashi has quit IRC | 03:27 | |
*** matsuhashi has joined #openstack-infra | 03:28 | |
clarkb | I think so | 03:29 |
mnaser | clarkb: http://logs.openstack.org/24/88324/6/check/check-grenade-dsvm-partial-ncpu/2e18c53/logs/new/screen-n-sch.txt.gz#_2014-04-18_00_44_53_075 -- that should be icehouse-compat .. where can i go to put the fix in? | 03:29 |
*** gokrokve has joined #openstack-infra | 03:29 | |
*** timrc-afk is now known as timrc | 03:30 | |
clarkb | grenade manages that test | 03:30 |
clarkb | fix probably belongs there | 03:30 |
clarkb | but this is icehouse to juno that is breaking | 03:30 |
clarkb | not havana to icehouse iirc | 03:30 |
mnaser | oh? | 03:31 |
mnaser | https://github.com/openstack-dev/grenade/blob/master/from-havana/upgrade-nova -- shouldn't this be icehouse-compat | 03:31 |
*** matsuhashi has quit IRC | 03:31 | |
*** gokrokve_ has joined #openstack-infra | 03:31 | |
clarkb | no, because from havana is correct | 03:31 |
*** matsuhashi has joined #openstack-infra | 03:31 | |
mnaser | oh | 03:31 |
mnaser | so | 03:31 |
clarkb | what isn't correct is from icehouse | 03:31 |
mnaser | shouldn't there be from-icehouse in there somewehre | 03:31 |
clarkb | yeah I am guessing that is what is missing | 03:32 |
clarkb | its kind of a bummer you have to set any config at all | 03:32 |
mnaser | so i guess grenade needs to be updated | 03:32 |
clarkb | yup, its possible the changes have been proposed already | 03:32 |
clarkb | sdague/dtroyer would know | 03:32 |
*** gokrokv__ has joined #openstack-infra | 03:33 | |
*** nati_ueno has joined #openstack-infra | 03:33 | |
*** matsuhas_ has joined #openstack-infra | 03:34 | |
*** unicell has joined #openstack-infra | 03:34 | |
*** gokrokve has quit IRC | 03:34 | |
mnaser | i see.. i guess tomorrow they can address this, until then, no code can be merged | 03:35 |
*** dcramer_ has joined #openstack-infra | 03:35 | |
mnaser | or will be able to | 03:35 |
*** matsuhashi has quit IRC | 03:35 | |
*** gokrokve_ has quit IRC | 03:35 | |
openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for grenade icehouse upgrade bug 1309324 https://review.openstack.org/88487 | 03:37 |
mriedem | clarkb: mnaser: ^ | 03:37 |
uvirtbot | Launchpad bug 1309324 in grenade "check-grenade-dsvm-partial-ncpu* failing since icehouse release due to compat issue" [Undecided,New] https://launchpad.net/bugs/1309324 | 03:37 |
*** lnxnut has quit IRC | 03:37 | |
*** Longgeek has quit IRC | 03:37 | |
*** gokrokv__ has quit IRC | 03:37 | |
*** Longgeek has joined #openstack-infra | 03:38 | |
*** dcramer_ has quit IRC | 03:39 | |
*** lnxnut has joined #openstack-infra | 03:39 | |
*** wchrisj has joined #openstack-infra | 03:41 | |
*** Longgeek has quit IRC | 03:41 | |
mriedem | don't see any grenade patches up for that specifically https://review.openstack.org/#/q/status:open+project:openstack-dev/grenade,n,z | 03:42 |
mriedem | just one to open stable/icehouse | 03:42 |
mnaser | yeah.. looks like nova is going to be blocked until that's fixed | 03:43 |
clarkb | I wonder if we can trick it into doing a havana to master again | 03:44 |
clarkb | if that works icehouse to master should be possible too | 03:44 |
*** nati_ueno has quit IRC | 03:44 | |
*** masayukig has quit IRC | 03:45 | |
dtroyer | I'm trying to catch up with what is happening here… what I don't know is what changed since yesterday? | 03:46 |
clarkb | also I think part of the issue here is we don't get d-g on the partial n-cpu test | 03:46 |
clarkb | dtroyer: we started testing icehouse to master | 03:46 |
clarkb | dtroyer: instead of havana to master | 03:46 |
dtroyer | too early for that | 03:46 |
*** Longgeek has joined #openstack-infra | 03:46 | |
clarkb | dtroyer: If46d1624e0b35a37d5dfb043edb513fc04b6ea68 is the change | 03:46 |
clarkb | dtroyer: why is that tool early? | 03:46 |
dtroyer | grenade has nothing to build an 'old' devstack that is icehouse | 03:47 |
dtroyer | grenaderc has to have a couple of settings changed and the entire from-icehouse stuff needs to be created | 03:47 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for grenade icehouse upgrade bug 1309324 https://review.openstack.org/88487 | 03:47 |
uvirtbot | Launchpad bug 1309324 in grenade "check-grenade-dsvm-partial-ncpu* failing since icehouse release due to compat issue" [Undecided,New] https://launchpad.net/bugs/1309324 | 03:47 |
clarkb | dtroyer: ok I will propose a revert of If46d1624e0b35a37d5dfb043edb513fc04b6ea68 | 03:48 |
dtroyer | https://review.openstack.org/#/c/88421/ is the first step, to nail down the final havana -> icehouse, rhather than master | 03:48 |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Revert "update grenade branches" https://review.openstack.org/88488 | 03:49 |
clarkb | there we go | 03:49 |
clarkb | mordred: jeblair sdague ^ | 03:49 |
clarkb | SergeyLukjanov: ^ | 03:49 |
clarkb | dtroyer: any idea why only that test fails? | 03:51 |
clarkb | dtroyer: does grenade have fallback behavior that is reasonable for the other grenade tests? | 03:51 |
dtroyer | no. I'm not sure what is different between the tests | 03:51 |
clarkb | dtroyer: the partial ncpu test does not upgrade n-cpu | 03:52 |
clarkb | dtroyer: and apparently needs a config option set in nova to make that possible | 03:53 |
clarkb | the others do a full upgrade from old to new | 03:53 |
mnaser | i suspect what's happening is that n-cpu is receiving RPC requests for a version that it doesn't support/understand | 03:53 |
clarkb | which apparently passed | 03:53 |
dtroyer | ah, than that's why…that is what is set in from-havana/upgrade-nova. we'll need the equivalent for from-icehouse | 03:53 |
mnaser | so when a new VM gets created, the RPC call sent from n-sch to n-cpu fails because the version is too new | 03:54 |
*** matsuhas_ has quit IRC | 03:54 | |
*** thuc has joined #openstack-infra | 03:54 | |
dtroyer | so with 'upgrade_levels compute icehouse' in nova.conf it shoudl work, right? | 03:55 |
mnaser | dtroyer: unfortuantely no, i think | 03:55 |
mnaser | because that version level was never defined in icehouse.. which is probably a mistake | 03:55 |
clarkb | dtroyer: but why do the normal grenade tests work? | 03:55 |
clarkb | dtroyer: I think my confusion si related to grenade otherwise being happy | 03:55 |
*** khyati has joined #openstack-infra | 03:55 | |
dtroyer | what I understand ATM is the RPC versions all match | 03:56 |
mnaser | clarkb: beacuse when you restart everything, everything is running the same rpcapi, so they all understand each other | 03:56 |
clarkb | mnaser: right, and that is why I think the milestone proposed branches are bad | 03:56 |
mnaser | dtroyer: this is the issue https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L248 | 03:56 |
mriedem | clarkb: can you put bug 1309324 on that commit message? | 03:56 |
mnaser | there was no RPC API version set in stone for icehouse | 03:56 |
clarkb | mnaser: if we had cut an icehouse branch 6/7 weeks ago this would've been apparent before we released | 03:56 |
uvirtbot | Launchpad bug 1309324 in grenade "check-grenade-dsvm-partial-ncpu* failing since icehouse release due to compat issue" [Undecided,New] https://launchpad.net/bugs/1309324 | 03:56 |
clarkb | mriedem: sure | 03:56 |
mriedem | thanks | 03:56 |
*** dcramer_ has joined #openstack-infra | 03:57 | |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Revert "update grenade branches" https://review.openstack.org/88488 | 03:57 |
*** matsuhashi has joined #openstack-infra | 03:57 | |
mnaser | the cutoff for icehouse should have happened and a new juno-compat should be been introduced i guess | 03:57 |
*** thuc_ has joined #openstack-infra | 03:58 | |
dtroyer | so is it backward? ie, for the ncpu run the upgrade_levels should not be set (yet)? | 03:59 |
dtroyer | for icehouse -> master | 03:59 |
clarkb | dtroyer: no it should be | 03:59 |
clarkb | but it may not fix the issue | 03:59 |
clarkb | but it will need to be set either way I think | 04:00 |
dtroyer | ok. It's late here. hopefully it'll make sense in the morning. | 04:00 |
mnaser | rpc versioning isn't fun | 04:00 |
clarkb | more importantly I think we should have that conversation with ttx. Milestone proposed shoves things in a corner where it is harder for things to test it | 04:01 |
clarkb | must better to be upfront and start the icehouse testing before we haev an icehouse release | 04:01 |
dtroyer | I do think though that getting the 'from-icehouse' bits into grenade needs to happen first/next | 04:01 |
clarkb | dtroyer: agreed | 04:01 |
*** dcramer_ has quit IRC | 04:01 | |
clarkb | dtroyer: but you shouldn't need much right? | 04:01 |
clarkb | I mean it works other than this partial ncpu thing | 04:01 |
*** thuc has quit IRC | 04:01 | |
*** Longgeek has quit IRC | 04:01 | |
mriedem | mnaser: i've gotta drop, you're in good hands though :) | 04:02 |
dtroyer | I'm not familiar offhand with what is in from-havana, but it shouldn't be much. | 04:02 |
mriedem | thanks for bringing that up! | 04:02 |
mnaser | heh, take care, mriedem :) no problem | 04:02 |
*** mriedem has quit IRC | 04:03 | |
mnaser | i think this should be brought up with russellb and ask him about https://github.com/openstack/nova/commit/13cb4e7d3fb7b119f7ee01d8aa3f265d4be20090 | 04:03 |
*** zhiyan_ is now known as zhiyan | 04:04 | |
*** praneshp has quit IRC | 04:05 | |
*** zns has joined #openstack-infra | 04:06 | |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Test d-g against partial n-cpu job. https://review.openstack.org/88492 | 04:06 |
clarkb | and ^ should make sure we actually test this before merging | 04:06 |
clarkb | ya we should definitely follow up with nova folks. jogo too | 04:06 |
*** zns has quit IRC | 04:07 | |
*** wchrisj has quit IRC | 04:08 | |
*** david-lyle has joined #openstack-infra | 04:08 | |
*** praneshp has joined #openstack-infra | 04:08 | |
*** zns has joined #openstack-infra | 04:10 | |
*** nosnos has joined #openstack-infra | 04:10 | |
*** harlowja_at_home has joined #openstack-infra | 04:14 | |
*** timrc is now known as timrc-afk | 04:16 | |
zaro | clarkb: why am i not seeing any gearman packets with tcpdump? | 04:19 |
clarkb | zaro: make sure you are listening to the correct interface | 04:19 |
clarkb | and that your filter is correct | 04:19 |
zaro | my vm only has one interface, 'eth0' | 04:20 |
*** harlowja_at_home has quit IRC | 04:21 | |
clarkb | zaro: it should have an lo as well | 04:22 |
zaro | clarkb: listening for everyhting on default gearman port 4730 but getting nothing. command is 'tcpdump -i eth0 port 4730' | 04:22 |
clarkb | and if you are connecting to localhost it should go over the loopback | 04:22 |
*** lnxnut has quit IRC | 04:22 | |
*** harlowja_at_home has joined #openstack-infra | 04:23 | |
*** zns has quit IRC | 04:23 | |
*** zns has joined #openstack-infra | 04:24 | |
zaro | clarkb: ok getting something now. had to set to 'lo' instead of 'eth0' | 04:25 |
*** dstanek has joined #openstack-infra | 04:28 | |
*** sweston has quit IRC | 04:30 | |
*** sweston has joined #openstack-infra | 04:30 | |
*** gokrokve has joined #openstack-infra | 04:32 | |
*** dstanek has quit IRC | 04:33 | |
*** harlowja_at_home has quit IRC | 04:35 | |
*** _nadya_ has joined #openstack-infra | 04:36 | |
*** gokrokve has quit IRC | 04:37 | |
*** david-lyle has quit IRC | 04:37 | |
*** coolsvap is now known as coolsvap|afk | 04:39 | |
zaro | clarkb: gear doc says that you need all 3 files, ssl_key, ssl_cert, and ssl_ca. any pointers on how to generate that ssl_ca file? | 04:41 |
*** zhiyan is now known as zhiyan_ | 04:41 | |
*** thuc has joined #openstack-infra | 04:42 | |
clarkb | zaro: yeah use your cert file as the ca cert (this isn't recommended iirc) then you generate a third cert to use as the cert iirc | 04:42 |
clarkb | I don't know off the top of my head the process for that though | 04:43 |
*** blamar has joined #openstack-infra | 04:43 | |
*** thomasem has quit IRC | 04:43 | |
*** thuc_ has quit IRC | 04:45 | |
*** thuc has quit IRC | 04:46 | |
clarkb | zaro: you might be able to use the cert as the ca cert too | 04:48 |
mgagne | zaro: is it for test purpose? | 04:49 |
*** jgrimm has joined #openstack-infra | 04:50 | |
zaro | mgagne: yes | 04:52 |
*** ildikov has quit IRC | 04:52 | |
mgagne | zaro: would this procedure work? http://gearman.info/gearmand/ssl.html | 04:52 |
mgagne | zaro: otherwise I think you could use a snakeoil cert | 04:53 |
mgagne | zaro: mine is located here: /etc/ssl/certs/ssl-cert-snakeoil.pem | 04:53 |
*** _nadya_ has quit IRC | 04:53 | |
*** wenlock has quit IRC | 04:54 | |
*** praneshp has quit IRC | 04:54 | |
zaro | ughh, it's all so confusing. will try those out. thanks. | 04:55 |
*** ildikov has joined #openstack-infra | 04:57 | |
*** coolsvap|afk is now known as coolsvap | 04:58 | |
*** Longgeek has joined #openstack-infra | 05:02 | |
mgagne | clarkb: with gearman, when using SSL, is the server expecting the client to have a client certificate for authentication? | 05:02 |
clarkb | mgagne: yeah | 05:03 |
clarkb | pretty sure client auth is performed | 05:03 |
clarkb | workers too | 05:03 |
mgagne | clarkb: ok, so zaro has to find a CA and sign a bunch of certs for his clients unless he can reuse the same cert as CA, server cert and client cert right? | 05:04 |
clarkb | ya | 05:04 |
*** coolsvap is now known as coolsvap|afk | 05:05 | |
*** ildikov has quit IRC | 05:06 | |
*** Longgeek has quit IRC | 05:07 | |
*** wenlock has joined #openstack-infra | 05:12 | |
*** nati_ueno has joined #openstack-infra | 05:17 | |
*** Longgeek has joined #openstack-infra | 05:18 | |
*** khyati has quit IRC | 05:20 | |
*** basha_ has joined #openstack-infra | 05:21 | |
* zaro very ignorant about keys | 05:22 | |
zaro | this has been the most useful so far http://forum.synology.com/wiki/index.php/How_to_generate_custom_SSL_certificates | 05:24 |
*** nati_ueno has quit IRC | 05:27 | |
mgagne | =) | 05:27 |
*** nati_ueno has joined #openstack-infra | 05:29 | |
zaro | clarkb: your suggestion about using sert file as ca cert didn't work. but it could also be that i'm not doing right. | 05:29 |
zaro | i've hit the end for tonight. nighty-night | 05:30 |
*** gokrokve has joined #openstack-infra | 05:32 | |
*** jamespage_ has joined #openstack-infra | 05:36 | |
*** jamespage_ has quit IRC | 05:37 | |
*** gokrokve has quit IRC | 05:37 | |
*** zhiyan_ is now known as zhiyan | 05:42 | |
SergeyLukjanov | http://alexnisnevich.github.io/untrusted/ very nice game :) | 05:42 |
*** Ajaeger has joined #openstack-infra | 05:45 | |
*** thuc has joined #openstack-infra | 05:53 | |
*** nati_ueno has quit IRC | 05:55 | |
*** thuc has quit IRC | 05:57 | |
*** dcramer_ has joined #openstack-infra | 05:57 | |
*** basha_ has quit IRC | 06:00 | |
*** dcramer_ has quit IRC | 06:02 | |
*** denis_makogon has joined #openstack-infra | 06:05 | |
*** Ajaeger has quit IRC | 06:12 | |
*** coolsvap|afk is now known as coolsvap | 06:18 | |
*** e0ne has joined #openstack-infra | 06:21 | |
*** nati_ueno has joined #openstack-infra | 06:32 | |
*** gokrokve has joined #openstack-infra | 06:32 | |
*** ildikov has joined #openstack-infra | 06:35 | |
*** zns has quit IRC | 06:35 | |
*** gokrokve has quit IRC | 06:36 | |
*** rcarrillocruz has joined #openstack-infra | 06:40 | |
*** adalbas has quit IRC | 06:46 | |
*** adalbas has joined #openstack-infra | 06:50 | |
openstackgerrit | Florian Preinstorfer proposed a change to openstack-infra/jenkins-job-builder: Add support for the Jenkins Doxygen publisher plugin. https://review.openstack.org/86604 | 06:54 |
*** chandan_kumar has quit IRC | 06:57 | |
*** flaper87|afk is now known as flaper87 | 07:01 | |
*** wenlock has quit IRC | 07:04 | |
*** chandan_kumar has joined #openstack-infra | 07:05 | |
*** matsuhashi has quit IRC | 07:06 | |
*** matsuhashi has joined #openstack-infra | 07:08 | |
SergeyLukjanov | sdague, ping | 07:09 |
SergeyLukjanov | sdague, could you please take a look at https://review.openstack.org/#/c/88422/ and https://review.openstack.org/#/c/88423/ | 07:09 |
SergeyLukjanov | sdague, due to the sahara enabled in d-g by default, we should update the job list for sahara | 07:09 |
*** jgallard has joined #openstack-infra | 07:10 | |
ttx | mordred, clarkb: we have a session to disucss my special brand of crazy: http://summit.openstack.org/cfp/details/293 | 07:11 |
SergeyLukjanov | ttx, morning, very interesting opic | 07:15 |
SergeyLukjanov | topic* | 07:16 |
SergeyLukjanov | ttx, is proposed/icehouse a part of it? | 07:16 |
SergeyLukjanov | (replacement for m-p branch I mean, don't actually remember the name pattern you proposed) | 07:16 |
ttx | SergeyLukjanov: yes | 07:16 |
SergeyLukjanov | ttx, sahara was enabled by default in gate tonight :) | 07:17 |
ttx | win! | 07:17 |
openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 07:17 |
openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools for Documentation repos https://review.openstack.org/88048 | 07:18 |
*** praneshp has joined #openstack-infra | 07:18 | |
*** jcoufal has joined #openstack-infra | 07:20 | |
*** andreykurilin_ has joined #openstack-infra | 07:20 | |
therve | SergeyLukjanov, ^^^updated the heat gate job | 07:26 |
SergeyLukjanov | ttx, here are the last steps - replace custom jobs with common - https://review.openstack.org/#/c/88422/ https://review.openstack.org/#/c/88423/ | 07:27 |
SergeyLukjanov | therve, lgtm | 07:28 |
*** dstanek has joined #openstack-infra | 07:31 | |
therve | Thanks! | 07:31 |
*** gokrokve has joined #openstack-infra | 07:32 | |
*** morganfainberg is now known as morganfainberg_Z | 07:32 | |
*** afazekas has joined #openstack-infra | 07:33 | |
*** e0ne_ has joined #openstack-infra | 07:34 | |
*** dstanek has quit IRC | 07:35 | |
*** e0ne has quit IRC | 07:37 | |
*** gokrokve has quit IRC | 07:37 | |
*** hashar has joined #openstack-infra | 07:37 | |
*** jp_at_hp has joined #openstack-infra | 07:38 | |
*** nati_ueno has quit IRC | 07:45 | |
*** andreykurilin_ has quit IRC | 07:52 | |
*** jpich has joined #openstack-infra | 07:54 | |
*** dcramer_ has joined #openstack-infra | 07:59 | |
*** ildikov_ has joined #openstack-infra | 08:01 | |
*** dcramer_ has quit IRC | 08:03 | |
*** jcoufal has quit IRC | 08:03 | |
*** ildikov has quit IRC | 08:04 | |
*** Ajaeger has joined #openstack-infra | 08:09 | |
Ajaeger | good morning, anybody around to review a requirements patch, please? https://review.openstack.org/#/c/88048/ | 08:10 |
Ajaeger | ah, SergeyLukjanov you did already - thanks! | 08:10 |
SergeyLukjanov | Ajaeger, morning, np :) | 08:11 |
*** pblaho has joined #openstack-infra | 08:11 | |
openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 08:19 |
*** achuprin has quit IRC | 08:19 | |
*** oomichi has quit IRC | 08:19 | |
*** achuprin has joined #openstack-infra | 08:22 | |
*** denis_makogon has quit IRC | 08:22 | |
*** ihrachyshka has joined #openstack-infra | 08:25 | |
*** lukego has joined #openstack-infra | 08:27 | |
*** lukego has quit IRC | 08:27 | |
*** lukego has joined #openstack-infra | 08:27 | |
*** jhesketh_ has joined #openstack-infra | 08:29 | |
*** jhesketh_ has quit IRC | 08:30 | |
*** jhesketh has joined #openstack-infra | 08:31 | |
*** dstanek has joined #openstack-infra | 08:31 | |
openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 08:31 |
*** gokrokve has joined #openstack-infra | 08:32 | |
lukego | Hoi. re: Tail-f jenkins comments, sorry about that, I had a misconfigured trigger for some hours yesterday while setting up jenkins. afaik the problem stopped around 24h ago. if there’s more trouble or actions needed please shoot me an email using the contact info on the wiki page. | 08:35 |
*** rcarrillocruz has quit IRC | 08:36 | |
*** rcarrillocruz has joined #openstack-infra | 08:36 | |
*** dstanek has quit IRC | 08:36 | |
*** gokrokve has quit IRC | 08:36 | |
*** jlibosva has joined #openstack-infra | 08:38 | |
*** unicell has quit IRC | 08:42 | |
Ajaeger | lukego: haven't seen it again, thanks | 08:46 |
*** reed has joined #openstack-infra | 08:56 | |
*** alexpilotti has joined #openstack-infra | 09:03 | |
*** ihrachyshka has quit IRC | 09:11 | |
*** rcarrillocruz has quit IRC | 09:21 | |
*** jcoufal has joined #openstack-infra | 09:23 | |
*** ildikov has joined #openstack-infra | 09:27 | |
SergeyLukjanov | lukego, yup, I've not seen it for last ~24h | 09:27 |
*** ildikov_ has quit IRC | 09:31 | |
*** gokrokve has joined #openstack-infra | 09:32 | |
*** Ajaeger has quit IRC | 09:35 | |
*** hogepodge has quit IRC | 09:36 | |
*** gokrokve has quit IRC | 09:37 | |
*** hogepodge has joined #openstack-infra | 09:37 | |
*** bookwar has quit IRC | 09:43 | |
*** ChanServ sets mode: +o SergeyLukjanov | 09:44 | |
*** ChanServ sets mode: -o SergeyLukjanov | 09:44 | |
*** bookwar has joined #openstack-infra | 09:44 | |
*** ihrachyshka has joined #openstack-infra | 09:55 | |
*** lukego has quit IRC | 10:01 | |
*** e0ne_ has quit IRC | 10:02 | |
*** e0ne has joined #openstack-infra | 10:02 | |
*** jgallard has quit IRC | 10:02 | |
*** praneshp has quit IRC | 10:03 | |
*** e0ne has quit IRC | 10:06 | |
*** dkehn__ has joined #openstack-infra | 10:11 | |
*** dkehn_ has quit IRC | 10:14 | |
*** dkehn has quit IRC | 10:14 | |
*** dkehn has joined #openstack-infra | 10:16 | |
*** om has joined #openstack-infra | 10:18 | |
*** blamar has quit IRC | 10:21 | |
*** rcarrillocruz has joined #openstack-infra | 10:27 | |
*** rcarrillocruz has quit IRC | 10:29 | |
*** rcarrillocruz has joined #openstack-infra | 10:29 | |
*** gokrokve has joined #openstack-infra | 10:32 | |
*** gokrokve has quit IRC | 10:37 | |
*** unicell has joined #openstack-infra | 10:42 | |
*** e0ne has joined #openstack-infra | 10:43 | |
sdague | SergeyLukjanov: | 10:50 |
openstackgerrit | Sean Dague proposed a change to openstack-infra/config: don't vote on icehouse jobs we don't on master https://review.openstack.org/88543 | 10:50 |
sdague | any chance you can push that change in? | 10:50 |
*** coolsvap is now known as coolsvap|afk | 10:50 | |
sdague | turns out we started voting on some neutron jobs that don't pass in tempest with the icehouse gating | 10:51 |
sdague | because our regex are strict :) | 10:51 |
*** om has quit IRC | 10:54 | |
*** e0ne has quit IRC | 11:00 | |
*** e0ne has joined #openstack-infra | 11:00 | |
*** nosnos has quit IRC | 11:01 | |
*** CaptTofu has joined #openstack-infra | 11:07 | |
*** om has joined #openstack-infra | 11:09 | |
*** matsuhashi has quit IRC | 11:12 | |
*** matsuhashi has joined #openstack-infra | 11:14 | |
*** jgallard has joined #openstack-infra | 11:19 | |
SergeyLukjanov | sdague, looking | 11:20 |
sdague | also, I should hopefully have the n-cpu partial bit fixed | 11:21 |
SergeyLukjanov | sdague, approved | 11:21 |
sdague | thanks | 11:22 |
SergeyLukjanov | sdague, I want to push https://review.openstack.org/#/c/88422/ and https://review.openstack.org/#/c/88423/ to add sahara to the integrated queue in gate pipeline and replace custom jobs | 11:22 |
sdague | SergeyLukjanov: so why isn't heat-slow in that list? | 11:23 |
dims | SergeyLukjanov, sdague, libvirt 1.2.x etc from UCA/precise/icehouse is behaving well. Where should i look for scripts that create/update images for each provider? | 11:23 |
*** _nadya_ has joined #openstack-infra | 11:23 | |
SergeyLukjanov | sdague, did you mean check-tempest-dsvm-neutron-heat-slow? | 11:23 |
sdague | SergeyLukjanov: yes | 11:23 |
sdague | dims: that's a good question, and one that I don't quite know :) | 11:24 |
SergeyLukjanov | sdague, hm, it's non-voting in check pipeline and doesn't affect sahara | 11:25 |
SergeyLukjanov | sdague, do you think we should add it? | 11:25 |
sdague | I thought sahara was building a heat based provisioning tool | 11:25 |
dims | sdague, :) np. still looking. i really don't like the fact that we are running on a patched libvirt | 11:25 |
openstackgerrit | A change was merged to openstack-infra/config: don't vote on icehouse jobs we don't on master https://review.openstack.org/88543 | 11:25 |
sdague | so it seems like working heat is an important part of the story | 11:25 |
SergeyLukjanov | sdague, yup, sounds fair, I'll add it in separated commit | 11:26 |
*** nosnos has joined #openstack-infra | 11:26 | |
sdague | cool | 11:27 |
*** nosnos has quit IRC | 11:29 | |
*** om has quit IRC | 11:29 | |
sdague | btw, I think sahara wins for fastest project to be put into the integrated gate after release | 11:30 |
sdague | within 12 hrs of release sahara was in :) | 11:30 |
sdague | good work on that | 11:30 |
*** rcarrill` has joined #openstack-infra | 11:30 | |
SergeyLukjanov | :) | 11:31 |
*** matsuhashi has quit IRC | 11:32 | |
*** gokrokve has joined #openstack-infra | 11:32 | |
*** matsuhashi has joined #openstack-infra | 11:32 | |
*** rcarrillocruz has quit IRC | 11:33 | |
*** aglarendil|nb has joined #openstack-infra | 11:34 | |
openstackgerrit | A change was merged to openstack-infra/config: Add integrated-gate-sahara jobs https://review.openstack.org/88422 | 11:34 |
SergeyLukjanov | I hope that we'll have provisioning testing in gate | 11:35 |
*** CaptTofu has quit IRC | 11:35 | |
SergeyLukjanov | in summit time | 11:35 |
SergeyLukjanov | sdague ^^ | 11:35 |
*** CaptTofu has joined #openstack-infra | 11:35 | |
*** gokrokve has quit IRC | 11:37 | |
*** e0ne has quit IRC | 11:37 | |
*** matsuhashi has quit IRC | 11:38 | |
*** rediskin has joined #openstack-infra | 11:38 | |
openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add tempest-dsvm-neutron-heat-slow to sahara gate https://review.openstack.org/88552 | 11:38 |
openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove obsolete custom Sahara dsvm jobs https://review.openstack.org/88423 | 11:38 |
rediskin | sdague SergeyLukjanov could you pls take a look? i have a lot of work depends on it https://review.openstack.org/#/c/88272/ | 11:39 |
openstackgerrit | Vladislav Kuzmin proposed a change to openstack-infra/nodepool: Add layer for work with KVM/QEMU hosts https://review.openstack.org/88554 | 11:39 |
openstackgerrit | Vladislav Kuzmin proposed a change to openstack-infra/nodepool: Correction logic for work with new KVM/QEMU provider manager https://review.openstack.org/88555 | 11:39 |
*** CaptTofu has quit IRC | 11:40 | |
*** aglarendil|nb has quit IRC | 11:40 | |
sdague | SergeyLukjanov: cool | 11:41 |
SergeyLukjanov | sdague, we're working on fake plugin that'll make us able to test the whole provisioning layer w/o running hadoop on top of it and it'll work good in gate, in fact it's test of sahara-heat integration | 11:41 |
sdague | SergeyLukjanov: nice | 11:42 |
sdague | that sounds very cool | 11:42 |
SergeyLukjanov | sdague, could you please +1 https://review.openstack.org/#/c/88552/& | 11:42 |
SergeyLukjanov | rediskin, looking | 11:42 |
rediskin | sdague SergeyLukjanov thanks | 11:43 |
*** e0ne has joined #openstack-infra | 11:43 | |
sdague | SergeyLukjanov: +1 | 11:44 |
SergeyLukjanov | sdague, thx | 11:44 |
SergeyLukjanov | rediskin, nice job name ...-rally-rally | 11:44 |
openstackgerrit | Vladislav Kuzmin proposed a change to openstack-infra/nodepool: Correction logic for work with new KVM/QEMU provider manager https://review.openstack.org/88555 | 11:44 |
openstackgerrit | A change was merged to openstack-infra/config: Remove obsolete custom Sahara dsvm jobs https://review.openstack.org/88423 | 11:46 |
openstackgerrit | Victor Sergeyev proposed a change to openstack-infra/config: Import oslo.db repository https://review.openstack.org/78955 | 11:46 |
openstackgerrit | A change was merged to openstack-infra/config: Add tempest-dsvm-neutron-heat-slow to sahara gate https://review.openstack.org/88552 | 11:52 |
openstackgerrit | Vladislav Kuzmin proposed a change to openstack-infra/nodepool: Correction logic for work with new KVM/QEMU provider manager https://review.openstack.org/88555 | 11:54 |
openstackgerrit | Vladislav Kuzmin proposed a change to openstack-infra/nodepool: Add layer for work with KVM/QEMU hosts https://review.openstack.org/88554 | 11:54 |
*** dprince has joined #openstack-infra | 11:56 | |
*** _nadya_ has quit IRC | 12:00 | |
openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-jobs job group https://review.openstack.org/88272 | 12:00 |
*** ccorrigan has joined #openstack-infra | 12:02 | |
ccorrigan | anyone working on gate-tripleo-image-elements? its currently breaking on docs and python26 | 12:04 |
*** lnxnut has joined #openstack-infra | 12:08 | |
openstackgerrit | Victor Sergeyev proposed a change to openstack-infra/config: Import oslo.db repository https://review.openstack.org/78955 | 12:08 |
*** dstanek has joined #openstack-infra | 12:11 | |
*** pdmars has joined #openstack-infra | 12:23 | |
*** dstanek has quit IRC | 12:24 | |
*** pdmars has quit IRC | 12:24 | |
*** pdmars has joined #openstack-infra | 12:24 | |
*** gokrokve has joined #openstack-infra | 12:32 | |
*** CaptTofu has joined #openstack-infra | 12:34 | |
*** mriedem has joined #openstack-infra | 12:35 | |
openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add hacking to requirements https://review.openstack.org/86679 | 12:36 |
*** gokrokve has quit IRC | 12:37 | |
*** dstanek has joined #openstack-infra | 12:39 | |
*** jhesketh has quit IRC | 12:42 | |
*** flaper87 is now known as flaper87|afk | 12:44 | |
*** dcramer_ has joined #openstack-infra | 12:45 | |
*** dkehn__ is now known as dkehnx | 12:46 | |
*** rlandy has joined #openstack-infra | 12:46 | |
*** andreykurilin_ has joined #openstack-infra | 12:48 | |
*** dcramer_ has quit IRC | 12:49 | |
*** dcramer_ has joined #openstack-infra | 12:50 | |
*** aglarendil|nb has joined #openstack-infra | 12:59 | |
*** mfer has joined #openstack-infra | 12:59 | |
*** andreykurilin_ has quit IRC | 13:00 | |
openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Make neutron full job voting (neutron gate only) https://review.openstack.org/88289 | 13:02 |
*** zz_gondoi is now known as gondoi | 13:03 | |
openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Add require-approvals to EventFilter representation https://review.openstack.org/88570 | 13:04 |
rediskin | sdague: could you pls take a look? https://review.openstack.org/#/c/88272/ | 13:06 |
YorikSar | jeblair: Can you take another look at https://review.openstack.org/87872 ? It looks like everybody's finally happy with it. | 13:07 |
*** miqui has joined #openstack-infra | 13:08 | |
*** dims has quit IRC | 13:09 | |
*** gondoi is now known as zz_gondoi | 13:11 | |
*** dcramer_ has quit IRC | 13:13 | |
*** yaguang has quit IRC | 13:14 | |
*** aglarendil|nb has quit IRC | 13:18 | |
*** dims has joined #openstack-infra | 13:22 | |
*** aglarendil has quit IRC | 13:23 | |
openstackgerrit | A change was merged to openstack/requirements: Update openstack-doc-tools for Documentation repos https://review.openstack.org/88048 | 13:25 |
*** dcramer_ has joined #openstack-infra | 13:25 | |
*** dcramer_1 has joined #openstack-infra | 13:25 | |
*** dcramer_ has quit IRC | 13:26 | |
openstackgerrit | A change was merged to openstack/requirements: Remove old pep8, pyflakes and flake8 versions https://review.openstack.org/86759 | 13:26 |
*** timrc-afk is now known as timrc | 13:26 | |
*** andreykurilin_ has joined #openstack-infra | 13:28 | |
*** gokrokve has joined #openstack-infra | 13:32 | |
salv-orlando | sdague: good morning. I read a comment from anteaya regarding https://review.openstack.org/#/c/88289/ and a conflict with https://review.openstack.org/#/c/88302 | 13:33 |
salv-orlando | she mentioned a conflict but can't find any - maybe is that about ensuring 4 jobs are executed on stable/icehouse as well? | 13:34 |
openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Upgrade six requirement to 1.6.0 https://review.openstack.org/88575 | 13:36 |
*** gokrokve has quit IRC | 13:37 | |
*** bknudson has joined #openstack-infra | 13:38 | |
*** zns has joined #openstack-infra | 13:38 | |
*** prad_ has joined #openstack-infra | 13:38 | |
*** lcostantino has joined #openstack-infra | 13:39 | |
*** chandan_kumar has quit IRC | 13:42 | |
*** lukego has joined #openstack-infra | 13:43 | |
*** sweston has quit IRC | 13:44 | |
*** thuc_ has joined #openstack-infra | 13:44 | |
*** signed8bit has joined #openstack-infra | 13:46 | |
*** wchrisj has joined #openstack-infra | 13:47 | |
*** maxbit has joined #openstack-infra | 13:48 | |
sdague | salv-orlando: yeh, I think in the strictest sense there would be a merge conflict | 13:50 |
sdague | because they were changing related chunks | 13:50 |
sdague | so I think if you just rebase now that the tempest change landed, life should be good | 13:51 |
mnaser | sdague: jenkins failed for the revert review, probably needs a recheck due to a neutron thing. | 13:52 |
salv-orlando | sdague: I did a rebase and had no conflict. However, I'm thinking that the patch as it is will put us in a situation where we have only non-voting jobs on stable branches. | 13:52 |
salv-orlando | because I'm keeping the full job non voting on stable branches. Especially for havana it will fail most of the times. | 13:53 |
sdague | salv-orlando: so you could leave the smoke job on stable | 13:55 |
salv-orlando | that's what I want to do. I think I can edit project.yaml and add something there | 13:56 |
salv-orlando | with branch-override set to stable/icehouse | 13:56 |
sdague | basically, don't delete it from the neutron gate, and instead use a branch stanza on name: ^(gate|check)-tempest-dsvm-neutron$ | 13:56 |
salv-orlando | ok makes sense | 13:57 |
sdague | salv-orlando: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L381 | 13:58 |
sdague | an example of doing that | 13:58 |
sdague | running only on stable | 13:58 |
salv-orlando | sdague: thanks | 14:00 |
*** vhoward has joined #openstack-infra | 14:02 | |
*** blamar has joined #openstack-infra | 14:02 | |
*** mriedem1 has joined #openstack-infra | 14:03 | |
*** mriedem has quit IRC | 14:04 | |
*** achuprin has quit IRC | 14:04 | |
*** achuprin has joined #openstack-infra | 14:07 | |
*** andreykurilin_ has quit IRC | 14:08 | |
*** sw_away is now known as sandywalsh | 14:10 | |
*** lukego has quit IRC | 14:18 | |
*** markmcclain has joined #openstack-infra | 14:20 | |
*** msabramo has quit IRC | 14:21 | |
* anteaya is awake | 14:22 | |
anteaya | checking email | 14:22 |
anteaya | deva seems to be out last candidate | 14:25 |
anteaya | our | 14:25 |
*** andreykurilin_ has joined #openstack-infra | 14:27 | |
SergeyLukjanov | anteaya, morning | 14:27 |
*** dcramer_1 has quit IRC | 14:28 | |
*** dcramer_ has joined #openstack-infra | 14:28 | |
*** sweston has joined #openstack-infra | 14:29 | |
*** gokrokve has joined #openstack-infra | 14:32 | |
*** dkliban is now known as dkliban_afk | 14:33 | |
*** msabramo has joined #openstack-infra | 14:33 | |
*** gokrokve has quit IRC | 14:37 | |
*** ilives has quit IRC | 14:40 | |
*** atiwari has joined #openstack-infra | 14:42 | |
anteaya | SergeyLukjanov: morning | 14:44 |
sdague | anteaya: any chance there will be a single wiki page with links to everyone's post for TC? | 14:45 |
*** thedodd has joined #openstack-infra | 14:46 | |
sdague | to make it easy for voters to see all the platforms without hunting the list themselves | 14:46 |
*** thuc has joined #openstack-infra | 14:46 | |
*** prad_ has quit IRC | 14:47 | |
YorikSar | Let's create a Git repo for that. It's so trending today :) | 14:47 |
anteaya | sdague: https://wiki.openstack.org/wiki/TC_Elections_April_2014#Candidates | 14:48 |
sdague | see, you are just awesome :) | 14:49 |
anteaya | sdague: awww, thanks | 14:49 |
*** bookwar has quit IRC | 14:49 | |
anteaya | YorikSar: campagin however you like, just make sure it is public for all to access | 14:49 |
*** thuc_ has quit IRC | 14:50 | |
*** bookwar has joined #openstack-infra | 14:50 | |
*** tjones has joined #openstack-infra | 14:50 | |
YorikSar | anteaya: I'm just kidding :) | 14:50 |
*** zns has quit IRC | 14:51 | |
reed | ttx, any clue about this ? https://ask.openstack.org/en/question/27463/authors-file-missing-in-the-cinder-repository-to-allow-merging-in-trunk/ | 14:52 |
ttx | reed: looking... | 14:53 |
*** jcoufal has quit IRC | 14:53 | |
*** matrohon has quit IRC | 14:53 | |
*** wenlock has joined #openstack-infra | 14:54 | |
*** gokrokve has joined #openstack-infra | 14:54 | |
ttx | reed: so nowadays AUTHORS file is autogenerated | 14:55 |
ttx | that said... the one in Cinder 2014.1 tarball looks hopelessly out of date. | 14:55 |
ttx | mordred: pbr autogenerates it, right ? | 14:56 |
reed | I guess there is no mandate to do anything like the cinder dev documentation is saying, right? | 14:56 |
*** prad_ has joined #openstack-infra | 14:57 | |
ttx | reed: probably a bug in doc. Will answer that question and followup | 14:57 |
reed | ttx, thank you | 14:57 |
reed | ttx, jgriffith told me on #openstack-cinder https://bugs.launchpad.net/sahara/+bug/1261774 | 14:59 |
uvirtbot | Launchpad bug 1261774 in sahara "Remove AUTHORS file from repos" [Medium,Fix released] | 14:59 |
ttx | reed: sure, but in the cinder case AUTHORS is already removed and that doesn't make it cirrectly autogenerated either | 15:00 |
reed | the docs is old, too | 15:01 |
openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: TimeLine events support https://review.openstack.org/88405 | 15:01 |
openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added service comments https://review.openstack.org/87959 | 15:02 |
*** pcrews has joined #openstack-infra | 15:02 | |
*** mriedem has joined #openstack-infra | 15:04 | |
*** zns has joined #openstack-infra | 15:05 | |
openstackgerrit | Antoine Musso proposed a change to openstack-infra/git-review: Skip invalid unicode in commit messages https://review.openstack.org/82826 | 15:05 |
*** mriedem1 has quit IRC | 15:08 | |
ttx | mordred, jeblair: https://bugs.launchpad.net/pbr/+bug/1309585 | 15:09 |
uvirtbot | Launchpad bug 1309585 in pbr "AUTHORS file generation is borken" [Undecided,New] | 15:09 |
ttx | reed: answered | 15:09 |
reed | ttx, thanks | 15:10 |
*** ihrachyshka has quit IRC | 15:11 | |
*** thuc_ has joined #openstack-infra | 15:11 | |
*** thomasem has joined #openstack-infra | 15:13 | |
*** thomasem has quit IRC | 15:14 | |
*** thomasem has joined #openstack-infra | 15:15 | |
*** thuc has quit IRC | 15:15 | |
*** lcostantino has quit IRC | 15:15 | |
*** ildikov has quit IRC | 15:22 | |
*** thuc_ has quit IRC | 15:22 | |
openstackgerrit | A change was merged to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 15:23 |
anteaya | go: http://lists.openstack.org/pipermail/openstack-dev/2014-April/033173.html | 15:23 |
*** thuc has joined #openstack-infra | 15:23 | |
ttx | anteaya: yay! let the games begin | 15:24 |
anteaya | yay | 15:24 |
ttx | anteaya: note that there is a typo for Favio's name on the wiki page | 15:25 |
anteaya | argh | 15:25 |
ttx | it's Percoco | 15:25 |
anteaya | flaper87|afk: sorry about the typo in your name in the wikipage | 15:26 |
ttx | he is also misnamed on the ballot, but then there is only one Fkavio so damage is limited | 15:26 |
anteaya | flaper87|afk: I did a straight copy/paste from the wikipage for the poll | 15:26 |
anteaya | yeah, except it is embarassing | 15:26 |
*** jgallard has quit IRC | 15:26 | |
anteaya | getting the names spelled properly is important | 15:26 |
*** thuc has quit IRC | 15:27 | |
*** nati_ueno has joined #openstack-infra | 15:29 | |
YorikSar | jeblair: Thanks for +2 | 15:29 |
*** nati_ueno has quit IRC | 15:29 | |
YorikSar | jeblair: How about +A? :) | 15:30 |
anteaya | http://lists.openstack.org/pipermail/openstack-dev/2014-April/033176.html | 15:30 |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command https://review.openstack.org/85580 | 15:30 |
jeblair | YorikSar: going to give the latest version a little more time first | 15:31 |
YorikSar | jeblair: Ok. Thanks :) | 15:31 |
YorikSar | jeblair: I'm thinking about adding a filter to further limit the size of JSON to allow users to look at their CR progress only. | 15:34 |
*** david-lyle has joined #openstack-infra | 15:35 | |
msabramo | jenkins-job-builder folks: I could use another review for https://review.openstack.org/84778 — I'd love to get this merged soon as it will make it easier to test other changes I hope to work on | 15:36 |
openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Make neutron full job voting (neutron gate only) https://review.openstack.org/88289 | 15:36 |
msabramo | hashar already approved it; need 1 more for merge | 15:36 |
msabramo | (fingers crossed :-)) | 15:36 |
hashar | msabramo: lot of folks where attending US Pycon and european folks (at least) have Easter on monday. | 15:37 |
hashar | msabramo: so there might be some delay :-D | 15:37 |
msabramo | hashar: yeah, I was at Pycon as well | 15:37 |
hashar | msabramo: I wish I had a million dollars to relocate to US and attend all those conferences hehe | 15:37 |
msabramo | hashar: but yeah delay is a fact of life; no big deal | 15:38 |
hashar | msabramo: at least I am sure the patch will land eventually | 15:38 |
msabramo | hashar: you don't need a million dollars, you just need a job at an open source company such as Rackspace or Yahoo :-) | 15:39 |
*** thuc has joined #openstack-infra | 15:39 | |
hashar | I am probably just finding good excuses to just stay at home so | 15:40 |
msabramo | hashar: all these patches are all intertwined | 15:40 |
*** thuc has quit IRC | 15:40 | |
msabramo | hashar: it's too much for my small brain to handle :-) | 15:40 |
*** pblaho has quit IRC | 15:40 | |
*** thuc has joined #openstack-infra | 15:40 | |
*** msabramo1 has joined #openstack-infra | 15:43 | |
msabramo1 | hashar: Maybe you can get your company to pay for you to go to Pycon in Montreal next year — good times and you will get to hang out with lots of francophones. I met mgagne and jeblair but not you unfortunately. | 15:44 |
ttx | jeblair: does the pbr bug minked above ring a bell ? Do you agree that those Authors files are suspiciously empty ? | 15:45 |
ttx | linked* | 15:45 |
msabramo1 | I met a lot of francophones - my colleage, Alex Conrad, is French and knows a lot of French Pythonistas from AfPy | 15:45 |
*** msabramo has quit IRC | 15:45 | |
ttx | damn French | 15:45 |
anteaya | they are all over the place | 15:45 |
hashar | msabramo1: I am a contractor for Wikimedia Foundation and we are on a tight budget. So we only travel when it is absolutely necessary. | 15:46 |
msabramo1 | the French seem to love Python | 15:46 |
* hashar gives another La Lubie to ttx | 15:46 | |
* ttx thinks we love Python because we don't have any wandering in our streets | 15:46 | |
msabramo1 | hashar: maybe you can get a grant from OpenStack Foundation or PSF | 15:46 |
anteaya | there are travel grants to attend summits | 15:48 |
*** jlibosva has quit IRC | 15:49 | |
*** e0ne has quit IRC | 15:50 | |
*** zhiyan is now known as zhiyan_ | 15:50 | |
*** msabramo1 has quit IRC | 15:50 | |
*** chandan_kumar has joined #openstack-infra | 15:51 | |
hashar | I can most probably show up at the Paris summit beginning of November since that is closely and i got accommodations there hehe | 15:52 |
sdague | jeblair: I'm about to pumpkin for the day, however 88488,2 is really important to land. We were missing the partial test on d-g. Until it lands, devstack, tempest, grenade, and nova are blocked from landing code. | 15:52 |
sdague | it's top of gate right now | 15:52 |
sdague | however it was this morning as well, and bounced off a neutron race | 15:52 |
sdague | so hopefully in 20 minutes everything is just fine. But if it bounces again, it might be worth a straight ninja merge | 15:53 |
zaro | morning | 15:53 |
*** nati_ueno has joined #openstack-infra | 15:54 | |
sdague | https://review.openstack.org/#/c/88492/ - will close the failing to test parial bug | 15:54 |
SpamapS | https://review.openstack.org/#/c/69918/12 ... failing on a docs check.. but.. I'm not sure why tripleo-image-elements has a docs check.. as.. it has.. no docs. | 15:55 |
*** jpich has quit IRC | 15:55 | |
*** msabramo has joined #openstack-infra | 15:57 | |
mordred | ttx: I just tried locally and I get the AUTHORS file I would expect | 15:58 |
ttx | mordred: you agree that "what you expect" is not "what is in the release tarballs", though ? | 15:59 |
jeblair | http://logs.openstack.org/d1/d135dde44eb80d15cf6ec8f9077d5fe410223eff/release/glance-tarball/3586bb0/console.html | 16:00 |
jeblair | ttx, mordred: ^ i think that is the release build, yeah? | 16:00 |
anteaya | morning zaro | 16:00 |
mordred | jeblair: did we start doing shallow clones at some point? | 16:01 |
ttx | jeblair: let me check md5 | 16:01 |
jeblair | mordred: i don't see a shallow clone in there | 16:01 |
ttx | jeblair: yes that's the job that generated the glance tarball | 16:01 |
ttx | mordred: fwiw the nova AUTHORS is slightly longer, but still nowhere as long as it should | 16:02 |
mordred | jeblair: me either - I just had a moment of remembering doing shallow clones | 16:02 |
ttx | so it's not just taking the first 4 names | 16:02 |
ttx | maybe some kind of race ? | 16:02 |
mordred | what is in the tarball is not what I get when I run things locally | 16:03 |
* mordred poking deeper | 16:05 | |
mordred | this is extremely strange | 16:05 |
*** ildikov has joined #openstack-infra | 16:06 | |
*** thuc has quit IRC | 16:07 | |
*** thuc has joined #openstack-infra | 16:07 | |
jeblair | mordred: i reproduced it locally | 16:09 |
sdague | SpamapS: https://github.com/openstack-infra/config/commit/2774501523a3bee388e52ba2ba119d1770e74850 - that was added by bnemec | 16:10 |
zaro | jeblair: i've rebased my change on yours https://review.openstack.org/#/c/88026 | 16:10 |
jeblair | mordred: ttx: tried to replicate what the actual build did as closely as possible: http://paste.openstack.org/show/76289/ | 16:11 |
sdague | ok, later folks, have a good weekend | 16:11 |
SpamapS | hrm | 16:11 |
SpamapS | ok so I need a way to make docs a noop | 16:11 |
jeblair | sdague: you too (ack on that change btw) | 16:11 |
*** yfried__ has quit IRC | 16:12 | |
*** thuc has quit IRC | 16:12 | |
SpamapS | ooo | 16:12 |
SpamapS | my git has colors in trusty | 16:12 |
*** gokrokve has quit IRC | 16:13 | |
mordred | jeblair: ack. I did something similar, and I got a regular authors file | 16:14 |
*** Sukhdev has joined #openstack-infra | 16:15 | |
openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Support email lists in require-approval https://review.openstack.org/88616 | 16:15 |
openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add support for the Jenkins Doxygen publisher plugin. https://review.openstack.org/86604 | 16:16 |
jeblair | mordred: btw, i'm confused by "--use-mailmap" which dosen't seem to work with my 'git log' | 16:16 |
zigo | mordred: & sdague: If you missed it, tgt with init script finally got approved in debian backports. | 16:17 |
pleia2 | good morning | 16:18 |
*** shakamunyi has joined #openstack-infra | 16:18 | |
openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Enabled functional tests https://review.openstack.org/88617 | 16:19 |
*** nati_ueno has quit IRC | 16:19 | |
mordred | jeblair: it works in mine - perhaps there are git versions that don't have it and we're not properly catching that | 16:19 |
* ttx signoffs | 16:20 | |
ttx | mordred, jeblair: ping me when you understand what's happening :) | 16:20 |
reed | good friday | 16:20 |
openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Support email lists in require-approval https://review.openstack.org/88616 | 16:20 |
reed | I'm taking off too, | 16:20 |
reed | it's Easter time here | 16:20 |
*** nati_uen_ has joined #openstack-infra | 16:20 | |
*** reed has quit IRC | 16:20 | |
mordred | jeblair: but you still get an AUTHORS list yeah? | 16:21 |
mordred | jeblair: or do you get what the release gets? | 16:21 |
jeblair | mordred: running the commands in that paste, i get an abbreviated authors list. running "git log --use-mailmap '--format=%aN <%aE>'" i get fatal: unrecognized argument: --use-mailmap | 16:22 |
*** thuc has joined #openstack-infra | 16:22 | |
SpamapS | https://review.openstack.org/#/c/88618/ <-- is this the best way to solve tripleo-image-elements failing gate because the docs job fails, or is there a simple way to exclude that job from that gate? | 16:22 |
mordred | jeblair: aha! | 16:22 |
mordred | then you've found the bug. I'm confused why there is anybody in the list at all in your case | 16:22 |
jeblair | mordred: me too, which is why i hadn't assumed i had found the bug yet :) | 16:22 |
mordred | well, you've _reproduced_ the bug - which is more than I can do | 16:23 |
mordred | so at least to some degree, precise git vs. trusty git is failing us | 16:23 |
jeblair | mordred: i've reproduced it on a fresh clone | 16:24 |
*** andreykurilin_ has quit IRC | 16:24 | |
*** hashar has quit IRC | 16:24 | |
jeblair | eg: git clone glance && cd glance && tox -evenv python setup.py sdist | 16:24 |
jeblair | so yeah, signs are pointing to "this doesn't work on precise" | 16:24 |
*** marun_afk has quit IRC | 16:25 | |
jeblair | mordred: were you using trusty in october? | 16:26 |
*** dcramer_1 has joined #openstack-infra | 16:26 | |
jeblair | http://git.openstack.org/cgit/openstack-dev/pbr/commit/?id=745d615c678b412c89be1cc3615c19d9bd9b9837 | 16:27 |
*** dcramer_ has quit IRC | 16:29 | |
SpamapS | http://logs.openstack.org/18/88618/1/check/gate-tripleo-image-elements-docs/8442350/console.html ... failing even though 'tox -evenv -- python setup.py build_sphinx' works locally for me.... | 16:31 |
*** dcramer_1 has quit IRC | 16:31 | |
mordred | jeblair: I was using saucy | 16:32 |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Revert "update grenade branches" https://review.openstack.org/88488 | 16:32 |
jeblair | mordred: so _run_git_command _never_ raises an exception on error | 16:32 |
*** bknudson has quit IRC | 16:32 | |
openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Revert "Let git do mailmap mapping for us" https://review.openstack.org/88625 | 16:33 |
*** gyee has joined #openstack-infra | 16:33 | |
jeblair | so pbr pretty much ignores all git errors | 16:33 |
mordred | jeblair: I feel like there was a specific reason we had it ignore git errors but at the moemnt i can't think of a good reason for that | 16:34 |
*** sweston has quit IRC | 16:34 | |
mnaser | now that the above problem has been fixed, can I force a reverify mid-jenkins job to restart, or do I have to wait for it to finish completely first? | 16:34 |
jeblair | mordred: perhaps the default should be flipped | 16:34 |
mordred | jeblair: yah | 16:35 |
mnaser | more specifically, trying to get this in to avoid the unit test race conditions - https://review.openstack.org/#/c/88324/ | 16:35 |
jeblair | mnaser: either wait, or push a new patchset (which will cancel/restart jobs) | 16:35 |
mnaser | jeblair: thanks, probably won't push again because then i'd have to get approved, etc. again.. guess i'll wait! | 16:35 |
*** dangers_away is now known as dangers | 16:36 | |
*** Longgeek has quit IRC | 16:36 | |
*** nati_uen_ has quit IRC | 16:37 | |
SpamapS | n/m .. something wonk in there | 16:37 |
jeblair | ttx, mordred: AUTHORS should actually be called "CO-AUTHORS" | 16:38 |
*** Ryan_Lane has joined #openstack-infra | 16:38 | |
openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Make logger available during tests https://review.openstack.org/86825 | 16:38 |
*** thuc_ has joined #openstack-infra | 16:38 | |
jeblair | as we found above, the actual authors list wasn't being generated because pbr was using a newer git command line syntax than what is available in precise. but the part that scans for co-authors didn't use that syntax and so worked. so the abbreviated list is just the list of co-authors. | 16:39 |
jeblair | ttx, mordred: ^ | 16:39 |
openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Allow to specify multiple branches per project in Gerrit trigger https://review.openstack.org/85648 | 16:39 |
*** comstud is now known as bearhands | 16:39 | |
openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Revert "Let git do mailmap mapping for us" https://review.openstack.org/88627 | 16:40 |
mordred | jeblair: hah. nice. that explains it | 16:40 |
mordred | jeblair: I updated the commit message on the above commit | 16:40 |
jeblair | mordred: i +2d | 16:40 |
SpamapS | wtf.. I have sphinx pinned to <1.2, but Sphinx-1.2b3-py2.7.egg-info is in my venv | 16:41 |
*** Ajaeger has joined #openstack-infra | 16:41 | |
Ajaeger | Hi everybody. The translation proposal jobs are working now - do you want to abandon the old ones that were run with user Jenkins? | 16:42 |
openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Support Fitnesse results publisher https://review.openstack.org/87711 | 16:42 |
mordred | SpamapS: that's <1.2 | 16:42 |
mordred | SpamapS: what version of pip are you using? | 16:42 |
SpamapS | mordred: the feeling of dread I have when you ask that is hard to describe ;) | 16:43 |
*** bhuvan has joined #openstack-infra | 16:43 | |
*** thuc has quit IRC | 16:43 | |
jeblair | "the wrong one" | 16:43 |
SpamapS | mordred: pip --version shows 1.5.4 | 16:43 |
SpamapS | tox 1.6.1 | 16:44 |
SpamapS | also have we tried to push the source_dir thing upstream to sphinx? | 16:44 |
mordred | SpamapS: that version of pip should appropriately not download pre-releases unless you ask for them | 16:45 |
SpamapS | mordred: is tox for sure using that pip? | 16:45 |
SpamapS | venv runtests: commands[0] | pip --version | 16:47 |
SpamapS | pip 1.5.4 from /home/clint/src/tripleo-image-elements/.tox/venv/local/lib/python2.7/site-packages (python 2.7) | 16:47 |
*** jogo is now known as flashgordon | 16:47 | |
* SpamapS is perplexed | 16:47 | |
*** khyati has joined #openstack-infra | 16:47 | |
mordred | SpamapS: oh - well... possibly not | 16:48 |
mordred | yeah - t-i-e's tox.ini looks good | 16:49 |
Ajaeger | mordred, jeblair, clarkb can either of you abandon the old translation patches that were sent by Jenkins? | 16:51 |
*** BadCub01 has joined #openstack-infra | 16:51 | |
jeblair | Ajaeger: will do | 16:51 |
Ajaeger | jeblair: use this query: https://review.openstack.org/#/q/status:open++branch:master+topic:transifex/translations,n,z | 16:51 |
Ajaeger | jeblair: thanks! | 16:51 |
jeblair | mordred: btw, i see a good deal of hard-coding jenkins addresses, etc in pbr. i think the change to use proposal bot will affect that | 16:51 |
Ajaeger | jeblair: and only take the one by Jenkins ;) | 16:51 |
mordred | jeblair: yup | 16:53 |
Ajaeger | jeblair: that was quick - thanks! | 16:54 |
*** branen has joined #openstack-infra | 16:54 | |
jeblair | mordred: i think it might match "infra@lists" right now, but clarkb was thinking of changing that | 16:55 |
SpamapS | so... | 16:55 |
mordred | jeblair: I will wait for clarkb to tell me things | 16:55 |
SpamapS | t-i-e's gate is totally broken now. | 16:55 |
mordred | oh. well then I should care more | 16:56 |
SpamapS | I ask again, is there a simple way to simply not run the docs part of the unit tests? | 16:56 |
mordred | you have docs things in your unittests? | 16:56 |
jeblair | SpamapS: so it looks like they have released sphinx versions >1.2 that fix the bug | 16:56 |
SpamapS | we're running, and failing them | 16:56 |
jeblair | according to mriedem http://bugs.python.org/issue19570 | 16:57 |
jeblair | 1.2.1 works | 16:57 |
mordred | jeblair: should we try to change our sphinx pin to >1.2.1 ? | 16:57 |
SpamapS | lovely that 1.2b is there, and breaks stuff | 16:57 |
jeblair | and the merge request for the workaround landed: https://bitbucket.org/birkenfeld/sphinx/pull-request/193/builddoc-shouldnt-fail-on-unicode-paths/diff | 16:57 |
jeblair | and they are up to 1.2.2 now (unknown if that works) | 16:57 |
mordred | SpamapS: well, I'd like to dig in to that - because you should not be getting 1.2b | 16:57 |
SpamapS | yes I can confirm, >=1.2.1 works for me | 16:57 |
SpamapS | mordred: happy to oblige. | 16:57 |
*** alaski is now known as lascii | 16:58 | |
*** harlowja_away is now known as harlowja | 16:58 | |
*** gokrokve has joined #openstack-infra | 16:58 | |
SpamapS | mordred: https://review.openstack.org/#/c/88618/ | 16:58 |
openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Containers now left-align https://review.openstack.org/88630 | 16:59 |
*** Ryan_Lane has quit IRC | 16:59 | |
*** markmcclain has quit IRC | 16:59 | |
flashgordon | clarkb: https://review.openstack.org/#/c/88492/ what broke when moving partial ncpu over? | 17:00 |
jeblair | install_command is missing | 17:00 |
mordred | SpamapS: oh - eek - ok, I want to edit that patch - why are you gating on docs already if you don't have docs? | 17:00 |
openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Increase sphinx pin now that 1.2.1 is out https://review.openstack.org/88631 | 17:00 |
*** vponomaryov has left #openstack-infra | 17:01 | |
boris-42 | hi all | 17:01 |
boris-42 | could somebody take a look at one more attempt to make rally gates works https://review.openstack.org/#/c/88272/ | 17:01 |
jeblair | mordred: ^ see comment about install_command? | 17:01 |
*** nati_ueno has joined #openstack-infra | 17:01 | |
SpamapS | mordred: I don't know. If there is a way to turn off just gating on docs, let's do that. | 17:01 |
*** thuc_ has quit IRC | 17:02 | |
SpamapS | mordred: recent commit enabled unit tests https://github.com/openstack-infra/config/commit/2774501523a3bee388e52ba2ba119d1770e74850 | 17:02 |
mordred | jeblair: when you say install_command is missing ... | 17:02 |
*** thuc has joined #openstack-infra | 17:02 | |
mordred | oh - in their tox.ini | 17:02 |
jeblair | yup | 17:02 |
mordred | *facepalm* thank you | 17:02 |
*** gokrokve has quit IRC | 17:03 | |
openstackgerrit | Mike Perez proposed a change to openstack-dev/hacking: Import Guideline Exception for gettextutils https://review.openstack.org/88632 | 17:04 |
*** jaypipes is now known as leakypipes | 17:04 | |
SpamapS | mordred: I don't mind having a stub and conforming. This shouldn't be hard. | 17:04 |
mordred | SpamapS: I just pushed an update | 17:05 |
openstackgerrit | Mike Perez proposed a change to openstack-dev/hacking: Import Guideline Exception for gettextutils https://review.openstack.org/88632 | 17:05 |
SpamapS | mordred: ugh | 17:06 |
SpamapS | mordred: so install_command has to be set or you get "evil pip" ? | 17:06 |
mordred | I think so - it just worked for me locally and I got 1.1.3 in my venv | 17:06 |
SpamapS | mordred: I too | 17:07 |
mordred | SpamapS: you should run ci.md through a converter and make it rst - but that's less important here | 17:07 |
*** thuc has quit IRC | 17:07 | |
mordred | actually - let me send one update... | 17:07 |
*** dripton has quit IRC | 17:08 | |
*** dripton has joined #openstack-infra | 17:09 | |
*** ociuhandu has joined #openstack-infra | 17:11 | |
clarkb | aroo? | 17:17 |
clarkb | I have 4 highlights on one page of sb | 17:17 |
*** gokrokve has joined #openstack-infra | 17:17 | |
*** lakshmisampath has joined #openstack-infra | 17:18 | |
mordred | clarkb: FIX ALL THE THINGS!!! | 17:18 |
clarkb | mordred: I pushed a few patches last night to do that | 17:18 |
mordred | to fix them all? | 17:18 |
mordred | neat | 17:18 |
* mordred goes back to bed - clarkb is here | 17:18 | |
clarkb | well to fix ncpu and d-g's asymetrical gating of that test | 17:18 |
clarkb | mordred: what thing were you looking for answers about pbr and infra@lists? | 17:19 |
lakshmisampath | I have pending review on config project. (https://review.openstack.org/#/c/88139/). can someone review it? | 17:19 |
*** Ryan_Lane has joined #openstack-infra | 17:19 | |
krotscheck | Hey, can I get someone to poke into the storyboard.o.o logs? I’m getting 500 errors on everything | 17:20 |
clarkb | wow everything really is broken | 17:20 |
krotscheck | Oh, is this a known thing? | 17:20 |
clarkb | no, probably not | 17:20 |
clarkb | just that mordred was asking if I fixed everything | 17:20 |
clarkb | oh ya install command because tox has a really silly default | 17:21 |
clarkb | jeblair: mordred https://review.openstack.org/#/c/88492/ would be a good thing to get in | 17:21 |
clarkb | then we can be sure that we test the grenade branch switches properly before merging | 17:22 |
*** nati_ueno has quit IRC | 17:24 | |
openstackgerrit | A change was merged to openstack-infra/config: Test d-g against partial n-cpu job. https://review.openstack.org/88492 | 17:26 |
SpamapS | mordred: oh good.. we _also_ have never run tests on python 2.6 | 17:27 |
*** nati_uen_ has joined #openstack-infra | 17:28 | |
flashgordon | clarkb: can you point me to a failed grenade partial npcu job | 17:28 |
clarkb | flashgordon: any that ran between about 2pm and 6pm PDT yesterday | 17:28 |
*** bknudson has joined #openstack-infra | 17:28 | |
flashgordon | such as? | 17:29 |
flashgordon | clarkb: was hoping you had one already open | 17:29 |
* clarkb queries logstash | 17:29 | |
clarkb | I don't have one open | 17:29 |
clarkb | actually may have one in my sb | 17:29 |
mnaser | there is an elastic recheck query for that issue, flashgordon | 17:29 |
mnaser | mriedem wrote it yesterday, me thinks | 17:30 |
mnaser | https://github.com/openstack-infra/elastic-recheck/commit/d779b5293674d3174852a609c9cf09ac53072cf7 | 17:30 |
clarkb | flashgordon: http://logs.openstack.org/24/88324/6/check/check-grenade-dsvm-partial-ncpu/2e18c53/logs/new/screen-n-sch.txt.gz#_2014-04-18_00_44_53_075 was the portion of config pointed out in a failing job that may be wrong | 17:30 |
clarkb | flashgordon: you can see the errors in old/screen-n-cpu.txt | 17:30 |
flashgordon | ohh that makes sense | 17:31 |
boris-42 | clarkb hi | 17:31 |
boris-42 | clarkb could you pls take a look at this patch https://review.openstack.org/#/c/88272/ ? | 17:31 |
mnaser | flashgordon: i think it's related to this which i'll be reverifying shortly after another race condition makes itway in - https://review.openstack.org/#/c/88494/ | 17:31 |
mnaser | that fails unit tests | 17:31 |
boris-42 | clarkb seems without it our rally gates doesn't work =( | 17:32 |
boris-42 | dont* | 17:32 |
boris-42 | clarkb thanks | 17:32 |
*** sabari has left #openstack-infra | 17:32 | |
clarkb | boris-42: why does that change completely redo the job setup? | 17:33 |
*** dstanek has quit IRC | 17:33 | |
flashgordon | mnaser: yup that sounds right, along with https://review.openstack.org/#/c/88549 | 17:33 |
clarkb | boris-42: it is hard to understand what that change is fixing because it is refactoring the jobs too | 17:33 |
boris-42 | clarkb okay we have some strange issues (that environments variable are not stetted) | 17:34 |
mnaser | flashgordon: right now i'm trying to get https://review.openstack.org/#/c/88324/ (in jenkins atm) to avoid the race condition, then https://review.openstack.org/#/c/88494/ to address that, then that review to test and see if it works after | 17:34 |
openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack-infra/config: Revert "Run unit tests in tripleo-image-elements" https://review.openstack.org/88637 | 17:34 |
SpamapS | mordred: ^^ reverting turning on the unit tests until we fix them | 17:34 |
clarkb | boris-42: right, but that change doesn't just set those variables | 17:34 |
clarkb | boris-42: it completely rewrites the way the jobs are done | 17:34 |
boris-42 | clarkb yep it's the another goal, to be able to run in other projects rally gates | 17:34 |
* clarkb is just trying to understand what this achieves | 17:34 | |
boris-42 | clarkb e.g. in cinder we would like to run performance gates | 17:35 |
boris-42 | clarkb with specific rally task | 17:35 |
clarkb | boris-42: you can;t really have performance gates | 17:35 |
boris-42 | clarkb why? | 17:35 |
clarkb | bceause we get nodes from ~7 different AZs and they all act differently | 17:35 |
boris-42 | clarkb it's not the big deal | 17:35 |
clarkb | you can certainly do performance testing | 17:35 |
clarkb | but gating requires stability | 17:35 |
boris-42 | clarkb yep that our goal for juno | 17:36 |
boris-42 | clarkb to normalize results in case of different nodes | 17:36 |
boris-42 | clarkb but for now we would like to get any results | 17:36 |
jeblair | mordred: are you going to try out the ssh manage-projects thing, or should we do manual new-project-friday? | 17:36 |
boris-42 | clarkb e.g. http://pavlovic.me/rally/glance_list.html | 17:36 |
boris-42 | clarkb even without absolute numbers it's clear that something wired is here | 17:37 |
*** aconrad has joined #openstack-infra | 17:37 | |
*** lakshmi has joined #openstack-infra | 17:38 | |
clarkb | ok, I think I understand that change now | 17:38 |
*** nati_uen_ has quit IRC | 17:38 | |
boris-42 | clarkb so for example that graph with glance I got for 5 mins | 17:38 |
clarkb | it allows you to also have check-rally-dsvm-cinder or check-rally-dsvm-glance | 17:38 |
*** lakshmisampath has quit IRC | 17:39 | |
boris-42 | clarkb yep it should at least =) | 17:39 |
mordred | jeblair: I want to do it with the new set of patches | 17:40 |
clarkb | mordred: I will look at those patches next | 17:40 |
jeblair | mordred: cool | 17:40 |
openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: `test` command read/write stdin/stdout by default https://review.openstack.org/84778 | 17:41 |
clarkb | mordred: https://review.openstack.org/#/c/88461/ can haz review? | 17:42 |
*** amcrn has joined #openstack-infra | 17:42 | |
boris-42 | clarkb thanks | 17:42 |
boris-42 | clarkb I hope this will run finally properly our jobs in rally | 17:42 |
clarkb | mordred: why are there two pbr revert git mailmap changes? | 17:42 |
mordred | clarkb: we decided to stay with infra@lists ? | 17:42 |
mordred | jeblair: did you see my terrible idea from last night: http://paste.openstack.org/show/76197/ ? | 17:43 |
clarkb | mordred: I think I am missing context here that I didn't pick up on in sb | 17:43 |
mordred | clarkb: there was a conversation yesterday or the day before about whether openstack-infra@lists.openstack.org was the right email address - I didn't follow the conclusion - was just asking about it | 17:43 |
clarkb | mordred: ah, I left the address in the external account ids table but removed the preferred email address to disable email | 17:44 |
clarkb | mordred: that way if people have questions about things they can send them to the list which I think is a good thing | 17:44 |
clarkb | but we won't get bot spammed | 17:44 |
openstackgerrit | A change was merged to openstack-infra/config: Add rally-jobs job group https://review.openstack.org/88272 | 17:45 |
mordred | kk | 17:45 |
clarkb | pleia2: assuming that the jobs last night didn't fill up the infra@lists queue again | 17:45 |
jeblair | mordred: heh, cool. :) also, i want to look at clarkb's makefile replacement for tox sometime. :) | 17:45 |
clarkb | pleia2: any chance you can double check the queue? | 17:45 |
pleia2 | clarkb: thumbs up | 17:45 |
clarkb | woot | 17:45 |
pleia2 | yeah, I checked this morning, all is well :) | 17:45 |
mordred | jeblair: I looked at that with him - we ran in to a hitch with it | 17:46 |
mordred | which is argument passing | 17:46 |
clarkb | mordred: no arg passing works fine | 17:46 |
clarkb | you just do make py27 ARGS='testname' | 17:46 |
mordred | you can't do "make py27 only_run_this_test" | 17:46 |
mordred | right. you have to do ARGS= | 17:46 |
jeblair | gotta run for a bit | 17:47 |
SpamapS | Would appreciate this getting approved ASAP: https://review.openstack.org/#/c/88637/ as I suspect it will require a restart or something to actually enact the change and unbreak tripleo-image-elements' gate. | 17:47 |
clarkb | mordred: did your fixes for the ssh stuff merge? | 17:48 |
clarkb | mordred: so next change is driving puppet with it change? | 17:48 |
mordred | clarkb: yup | 17:48 |
clarkb | SpamapS: why not just fix the tests? | 17:48 |
clarkb | The gate for tripleo-image-elements is broken due to this commit. isn't quite true | 17:49 |
clarkb | the gate is broken because the tests are broken | 17:49 |
openstackgerrit | A change was merged to openstack-infra/config: Set config prior to git review -s. https://review.openstack.org/88461 | 17:49 |
*** EmilienM has quit IRC | 17:50 | |
*** EmilienM has joined #openstack-infra | 17:50 | |
*** Hefeweizen has joined #openstack-infra | 17:50 | |
mordred | clarkb: https://review.openstack.org/#/c/80976/ is the next up - then https://review.openstack.org/#/c/80976 | 17:50 |
clarkb | I love that while fungi is on vacation we are just going to completely change how we run puppet | 17:52 |
clarkb | :) | 17:52 |
anteaya | he'll love that too | 17:52 |
mordred | clarkb: :) | 17:52 |
mordred | clarkb: just wait until I rewrite run_all.sh to be a gearman client | 17:52 |
anteaya | so what is going to happen with new project friday today? | 17:52 |
mordred | anteaya: we're going to land the fix to the thing and try it out | 17:53 |
anteaya | is the mordred sshing in place? | 17:53 |
mordred | yup | 17:53 |
anteaya | woohoo | 17:53 |
mordred | but we have two more patches to land to finish hooking it up | 17:53 |
clarkb | mordred: ok +2'd the two changes. But haven't approved. I will let you babysit that? | 17:53 |
anteaya | 80976 and ? | 17:53 |
mordred | clarkb: yup. I'll do them in just a sec | 17:53 |
clarkb | 87343 | 17:53 |
anteaya | k | 17:53 |
mordred | when we land this - it means we'll be ceasing to run puppet agent completely | 17:54 |
mordred | well - the daemon | 17:54 |
mordred | I'm excited about that | 17:54 |
SpamapS | clarkb: We've _never_ run those tests. | 17:54 |
SpamapS | clarkb: docs, and python26, are both broken | 17:54 |
SpamapS | clarkb: so we'll have to ask for a manual merge of at least one of those anyway. | 17:55 |
clarkb | SpamapS: no manual merge should be necessary | 17:55 |
SpamapS | clarkb: and this is blocking everyone while we do that. | 17:55 |
clarkb | just push a change that fixes it | 17:55 |
SpamapS | fixes both | 17:55 |
clarkb | yes | 17:55 |
SpamapS | neither are small changes | 17:55 |
mordred | clarkb: I think he's saying "landing the change to enforce two gates that they weren't prepared to meet as a mistake" | 17:55 |
mordred | was | 17:55 |
clarkb | mordred: sure | 17:55 |
SpamapS | I for one value a commit history that consists of single logical changes. | 17:55 |
clarkb | but the fix isn't on infras plate imo | 17:55 |
clarkb | if a projects reqwuests tests then removes them because they don't pass... | 17:56 |
SpamapS | bnemec should not have requested them | 17:56 |
SpamapS | I don't know why he did. I have not received word back. I don't think he thought it through. | 17:56 |
SpamapS | Probably just ran py27 and figured that was enough. | 17:56 |
mriedem | flashgordon: yeah https://review.openstack.org/#/c/88487/ | 17:57 |
SpamapS | just now have a lucid box to test py26 on | 17:57 |
*** lukego has joined #openstack-infra | 17:57 | |
anteaya | 80976 used to have run_all.sh and ensure_puppet.sh which are no longer part of the patch, where did they go? | 17:57 |
anteaya | are they merged already? | 17:57 |
mordred | anteaya: I split them out into a patch that's already merged | 17:57 |
SpamapS | clarkb: btw, I did try to fix the tests.. :-/ | 17:58 |
flashgordon | mriedem: so I guess the good news is we know why things started failinga nd have fixes for many of the issues | 17:58 |
clarkb | SpamapS: does py27 pass? should we maybe explicitly add that test to keep it from regressing? | 17:58 |
*** wchrisj_ has joined #openstack-infra | 17:58 | |
mriedem | flashgordon: yeah, i sent out a note to the dev list before leaving for lunch | 17:58 |
SpamapS | clarkb: yes. I think we can risk it for a few more days seeing as it has never been gated before. | 17:59 |
clarkb | flashgordon: so about that (I bet ttx is afk at this point deep in a bottle of fine wine) but I think we should consider dropping milestone proposed branches completely | 17:59 |
*** aconrad has quit IRC | 17:59 | |
mriedem | flashgordon: http://lists.openstack.org/pipermail/openstack-dev/2014-April/033182.html | 17:59 |
clarkb | flashgordon: then we can do a much more coordinated switch to release branches prior to release to ensure that all of the test keep working even after that branching | 17:59 |
SpamapS | clarkb: simplest thing for everyone: revert the unit test running.. we'll fix.. then un-revert. | 17:59 |
clarkb | as it is milestone proposed is a temporary hackish way around doing that | 17:59 |
flashgordon | mriedem: yeah I saw it | 17:59 |
anteaya | zaro: if you feel like chatting about your -1 here, let me know: https://review.openstack.org/#/c/77376/1 | 17:59 |
mordred | anteaya: https://review.openstack.org/#/q/status:merged+owner:2+project:openstack-infra/config,n,z the top 5 - except not the Use virtualenv_version patch | 18:00 |
mordred | anteaya: if you want to look | 18:00 |
flashgordon | clarkb: areed on both counts | 18:00 |
flashgordon | agreed on * | 18:00 |
anteaya | mordred: thanks yes was looking for it | 18:00 |
clarkb | something like, update grenade with known gotchas, update d-g with fallback to previous stable branch if current stable branch does not exist yet, create stable branches, fix issues that show up | 18:00 |
mordred | ok guys. I just pulled the trigger | 18:00 |
clarkb | then we can sort all of this out well before release | 18:00 |
clarkb | then 6 weeks later we tag commits on stable/icehouse and everything has been working for 6 weeks | 18:01 |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command https://review.openstack.org/85580 | 18:01 |
mordred | when it lands, I'm going to trigger a puppet run everywhere | 18:01 |
*** sabari has joined #openstack-infra | 18:01 | |
flashgordon | clarkb: are you documenting all the issues we are hitting? so we can avoid them next time? | 18:01 |
mordred | and then verify that puppet is no longer running on any of the nodes | 18:01 |
clarkb | flashgordon: no, there is only one issue | 18:01 |
clarkb | flashgordon: we have no coordinated way of making the switch to a stable branch ebtween infra, release, and qa | 18:02 |
flashgordon | right | 18:02 |
clarkb | and a big part of that is it happens all at once rather than as part of the gradual process that milestone proposed does today | 18:02 |
*** wchrisj has quit IRC | 18:02 | |
flashgordon | I just counted that as sub items | 18:02 |
flashgordon | anyway its agood thing reviews die down around this time anyway | 18:03 |
openstackgerrit | A change was merged to openstack-infra/config: Revert "Run unit tests in tripleo-image-elements" https://review.openstack.org/88637 | 18:03 |
*** UtahDave has joined #openstack-infra | 18:03 | |
clarkb | flashgordon: my biggest concern is I think this makes it possible to release broken code | 18:04 |
clarkb | I don't necessarily think that happened here (I could be wrong, not super familiar with nova rpc) but it points out a gap in testing | 18:04 |
anteaya | mordred where did the run_ssh.sh script end up? I am not seeing it in here: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files | 18:04 |
mordred | anteaya: it's templated | 18:05 |
mordred | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/run_remote_puppet.sh.erb | 18:05 |
clarkb | mordred: so 2 pbr reverts. what is the difference there? | 18:05 |
mordred | clarkb: the first was the auto-generated one from gerrit | 18:05 |
clarkb | can it be abandoned? | 18:05 |
mordred | yes | 18:05 |
mordred | I thought I had - sorry | 18:06 |
flashgordon | clarkb: how do you think we could have shiped broken code? | 18:06 |
clarkb | flashgordon: because icehouse was on milestone proposed for the last ~6 weeks. Which meant all of the upgrade testing effectively ignored it | 18:06 |
flashgordon | clarkb: oh crap | 18:07 |
clarkb | similar to how swift milestone proposed derped tempest | 18:07 |
flashgordon | clarkb: well I guess we will learn soon | 18:07 |
clarkb | milestone proposed is a corner it isn't well dealt with by the CI stuff | 18:07 |
clarkb | flashgordon: well havana -> icehouse should've been tested | 18:08 |
*** dcramer_ has joined #openstack-infra | 18:08 | |
clarkb | flashgordon: but icehouse -> master would not be | 18:08 |
anteaya | mordred: I was wrong, run_ssh isn't a script it is a function within run_remote_puppet.sh.erb, that was where I was confused | 18:08 |
clarkb | so for CD deployments it could mean hurt | 18:08 |
notmyname | shouldn't m-p branches be deleted once they are merged back to master? | 18:08 |
clarkb | notmyname: yes but that is a 6 week period after feature freeze | 18:08 |
notmyname | (also, have those jobs been retriggered yet?) | 18:08 |
clarkb | notmyname: yes that is how we found the fail | 18:08 |
clarkb | oh those other jobs | 18:09 |
flashgordon | clarkb: during milestone proposed we tseted havana -< icehouse (Which was master at the time) but not milestone right? | 18:09 |
clarkb | I don't know about those. mordred was going to do that iirc | 18:09 |
clarkb | flashgordon: any changes to m-p should be tested as havana -> m-p | 18:09 |
SpamapS | thanks for the revert | 18:09 |
SpamapS | now I just realized we could just make py26 a noop couldn't we? | 18:09 |
clarkb | flashgordon: but any master changes would have tested havana -> master not m-p to master | 18:09 |
mordred | notmyname: retriggering is on my plate for today | 18:10 |
clarkb | flashgordon: so there is a gap there for CD folks | 18:10 |
flashgordon | clarkb: yeah. but we could have derped a backport to break milestone proposed too I think | 18:10 |
* mordred thinks instead of m-p we shoudl just cut the stable/* branch early | 18:10 | |
clarkb | mordred: yes | 18:10 |
notmyname | mordred: ok, thanks. in the meanwhile, all patches that land will have the wrong version number calculated by pbr | 18:10 |
mordred | but not tag a release on it until it's time to tag a release | 18:10 |
mordred | notmyname: yah. sorry about the dep there | 18:11 |
*** mrodden has joined #openstack-infra | 18:11 | |
mordred | derp | 18:11 |
clarkb | mordred: and before we do that, we make sure d-g and grenade are ready for it and test stable/* from commit 1 | 18:11 |
mordred | man | 18:11 |
mordred | I derped typing derp | 18:11 |
mordred | clarkb: ++ | 18:11 |
notmyname | :-) | 18:11 |
clarkb | mordred: completely agree. it simplifies testing so much more | 18:11 |
notmyname | mordred: just a side-effect of the way openstack does versioning. it has upsides, but period after every m-p tag is the downside | 18:11 |
mordred | notmyname: yeah - I want to sort out making that period and transition not suck | 18:12 |
clarkb | mordred: the branching model above cuold help fix it | 18:12 |
clarkb | branch stable/* and make a tag that convinces master to do the right thing | 18:12 |
clarkb | and do that at the beginning of the process not end | 18:12 |
mordred | I do not think it will help | 18:12 |
*** thuc has joined #openstack-infra | 18:13 | |
SpamapS | clarkb: heh.. so.. I may have pushed a bit aggressively there. I think we can just noop the py26 tests and then undo that revert. Sorry for the churn. :-/ | 18:13 |
mordred | I thnk it will just mush the fail aroudn like a plate of unwanted mashed potatoes | 18:13 |
clarkb | SpamapS: meh | 18:14 |
clarkb | SpamapS: when the tests pass, push an unrevert | 18:14 |
SpamapS | clarkb: but we can at least land those things as individual commits and _then_ turn on the tests. ;) | 18:14 |
clarkb | mordred: nice | 18:14 |
*** Ajaeger has quit IRC | 18:14 | |
*** Sukhdev has quit IRC | 18:15 | |
SpamapS | clarkb: do I need to wait for a zuul/jenkins restart before I start to recheck/reverify things? | 18:16 |
clarkb | SpamapS: you need to wait for puppet to apply the change to zuul, once that happens zuul is ready (it reloads config on the fly). At the bottom of the zuul status page should be a timestamp for when the last reload happened | 18:16 |
anteaya | zaro: thanks | 18:16 |
clarkb | SpamapS: looks like it just happened so you should be good | 18:17 |
anteaya | before this gets buried again, can I request a +A? https://review.openstack.org/#/c/77376/ | 18:17 |
anteaya | a lineup of +1s and +2s | 18:17 |
*** thuc has quit IRC | 18:17 | |
clarkb | mordred: https://review.openstack.org/#/c/88627/ reviewed | 18:19 |
clarkb | anteaya: I think we are all for it, but trying to avoid doing policy decisions in the config repo | 18:19 |
clarkb | anteaya: this is one of the reason for my proposed summit session. We should be able to talk about items like that and more | 18:20 |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Refactor delete_jobs https://review.openstack.org/88646 | 18:20 |
*** denis_makogon_ has joined #openstack-infra | 18:21 | |
anteaya | clarkb: okay I'll add the link to the review in teh comments section of your summit proposal so it doesn't get lost | 18:21 |
anteaya | is that okay? | 18:21 |
clarkb | sure | 18:21 |
anteaya | k | 18:21 |
mordred | btw - I've found a new and glorious way in which my laptop can go mental on me | 18:22 |
clarkb | story time | 18:22 |
mordred | last night - google stopped working. just google. I can get anywhere else, but not google websites | 18:23 |
mordred | nothing shows in logs, nothing seems unhappy | 18:23 |
mordred | basically, google won't resolve | 18:23 |
mordred | restart networking? no fixy | 18:23 |
openstackgerrit | A change was merged to openstack-infra/config: Drive puppet from the master over ssh https://review.openstack.org/80976 | 18:23 |
mordred | reboot computer? yup. google is back | 18:23 |
mordred | W T F | 18:23 |
clarkb | this reminds me I should update from saucy to trusty soon so that I can have working vim again | 18:24 |
mordred | trusty mostly works | 18:24 |
clarkb | I updated my htpc last night. There was at least one segfault during the upgrade | 18:24 |
mordred | running puppet everywhere to roll out new change | 18:25 |
nibalizer | clarkb: i like that the only thing you copied to keep safe was /etc/pulse | 18:25 |
nibalizer | that is exactly how I feel | 18:25 |
*** dstanek has joined #openstack-infra | 18:25 | |
anteaya | mordred: so did you find out what the problem was? | 18:26 |
anteaya | or have we reached the end of this tale? | 18:26 |
mordred | that was the end | 18:26 |
mordred | it might have been absurdist comedy | 18:26 |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Refactor delete_jobs https://review.openstack.org/88646 | 18:26 |
clarkb | nibalizer: all it does is play videos from the internet. Sound is pretty important | 18:27 |
clarkb | it was my hbo go box for a long time. Then I ran out of hbo go | 18:27 |
clarkb | there was a time when I had diablo running on it | 18:30 |
clarkb | manually installed wtihout devstack | 18:30 |
clarkb | do not recommend | 18:30 |
*** andreykurilin_ has joined #openstack-infra | 18:30 | |
anteaya | the hat passing routine | 18:30 |
nibalizer | also i hate fighting pulse | 18:30 |
anteaya | from godot | 18:31 |
clarkb | I have to say openstack has come a long way since diablo | 18:31 |
nibalizer | clarkb: i parsed that as blizzards diablo at first, was confused | 18:31 |
clarkb | adam_g: how are those firewall rules working? | 18:34 |
*** marun has joined #openstack-infra | 18:34 | |
adam_g | clarkb, they are working perfect. thanks. i have a patch up to make devstack do that itself, so we can drop them from the puppet config later if you'd prefer it | 18:35 |
*** sweston has joined #openstack-infra | 18:35 | |
openstackgerrit | A change was merged to openstack-dev/hacking: Import Guideline Exception for gettextutils https://review.openstack.org/88632 | 18:35 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Stop trying to start gerrit https://review.openstack.org/88650 | 18:36 |
clarkb | adam_g: sounds good. It may actually be a good idea to attempt to make devstack do a completely locked down iptables and only punch necessary holes documenting the purpose of each hole | 18:37 |
*** lukego has quit IRC | 18:37 | |
clarkb | adam_g: might make lives easier for people deploying in datacenters | 18:37 |
notmyname | is the world falling? https://review.openstack.org/#/c/88359/ | 18:37 |
clarkb | but probably also a lot of work | 18:37 |
*** rcarrill` has quit IRC | 18:37 | |
clarkb | notmyname: ERROR: unknown branch name feature/ec | 18:37 |
notmyname | clarkb: this hasn't happened before | 18:38 |
clarkb | oh, this is probably a consequence of sdague's config generator? | 18:38 |
clarkb | sdague: ^ | 18:38 |
mordred | he's afk | 18:38 |
clarkb | notmyname: there is a new thing to formalize what branches support which features and so on | 18:38 |
adam_g | clarkb, yeah, i was thinking the same thing yesterday. a devstack-INPUT /w similar defaults to openstack-INPUT. that would have saved me a bit of time trying pinpoint why i wasns't able reproduce issues i was seeing in the gate | 18:38 |
clarkb | notmyname: I'm guessing it didn't accomodate random branches... | 18:38 |
* clarkb reads some code | 18:38 | |
mordred | clarkb: or feature branches - yeah - I bet that' it | 18:38 |
clarkb | yurp that was an oversight | 18:39 |
clarkb | https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/test-matrix.py#n24 | 18:40 |
clarkb | so how do we want to handle this. treat the feature/* prefix as equivalent to master? | 18:41 |
mordred | yah. I think that makes the most sense | 18:41 |
* clarkb writes a thing | 18:41 | |
mordred | by definition feature branches don't chase stable | 18:41 |
clarkb | any other prefixes we should care about or can I hard code feature like stable? | 18:43 |
*** ociuhandu has quit IRC | 18:43 | |
mordred | I think you can hardcode feature like stable | 18:43 |
pleia2 | running out to lunch | 18:43 |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Handle feature branches in test-matrix.py https://review.openstack.org/88652 | 18:45 |
clarkb | how does that look? | 18:45 |
mordred | clarkb: +2 | 18:46 |
*** hashar has joined #openstack-infra | 18:50 | |
zaro | mordred: how else would gerrit start? | 18:51 |
*** denis_makogon_ has quit IRC | 18:52 | |
mordred | zaro: by hand - we tend to want to restart gerrit when we choose | 18:56 |
notmyname | clarkb: can you retrigger the patch (or let me know) when zuul is ready? | 18:56 |
clarkb | notmyname: sure | 18:58 |
clarkb | that apparently broke something though :( | 18:58 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Properly don't fail on success https://review.openstack.org/88656 | 18:58 |
clarkb | gah I can't python | 18:58 |
*** dims is now known as dims-openstack | 18:58 | |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Handle feature branches in test-matrix.py https://review.openstack.org/88652 | 18:58 |
mordred | clarkb: a fix I'd applied locally yesterday got missed in being submited ^^ | 18:58 |
clarkb | mordred: k | 18:58 |
clarkb | I'm going to make lunch while tests for 88652 are in flight | 18:59 |
*** atiwari has quit IRC | 18:59 | |
mordred | clarkb: ci-backup-rs-ord.openstack.org | 18:59 |
mordred | clarkb: things can't ssh in to that- do we expect that? | 19:00 |
krotscheck | Sorry, my log got screwed up. Has someone bounced out to storyboard.o.o yet and taken a look at why our API went belly up? | 19:00 |
mordred | krotscheck: looking now | 19:01 |
mordred | [Fri Apr 18 19:00:30 2014] [error] [client 74.202.214.170] ConfigFileParseError: Failed to parse /etc/storyboard/storyboard.conf: at /etc/storyboard/storyboard.conf:3, Section must be started before assignment: None | 19:01 |
krotscheck | mordred: What the… did something in the puppet run after the token expiration screw up the config file? | 19:02 |
*** dcramer_ has quit IRC | 19:03 | |
mordred | nope. the puppet has correctly installed a very evil config file | 19:03 |
mordred | krotscheck: check out ./modules/storyboard/templates/storyboard.conf.erb | 19:03 |
mordred | you'll find it less exciting than you'd want | 19:03 |
krotscheck | mordred: Looking now. I’m expecting demons and succubi | 19:04 |
*** jamespage_ has joined #openstack-infra | 19:04 | |
mordred | krotscheck: dude. if you find succubi, let me know... | 19:05 |
*** dcramer_ has joined #openstack-infra | 19:05 | |
clarkb | mordred: no we don't expect that | 19:05 |
clarkb | mordred: we should figure that out | 19:06 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Put a default section header above the TTL https://review.openstack.org/88660 | 19:06 |
mordred | clarkb: my personal user accoutn also does not log in there | 19:06 |
mordred | clarkb: so I'm thinking that puppet has not run there ina very long time | 19:06 |
mordred | krotscheck: ^^ | 19:06 |
clarkb | oh ya its manual puppet | 19:06 |
clarkb | I will give it a shot after this sandwich | 19:06 |
mordred | clarkb: why is it manual puppet? should we exclude it from the ssh puppet? | 19:07 |
anteaya | so who has any thoughts on how to improve the search feature in the wiki | 19:07 |
mordred | krotscheck: (the puppet patch, not the other thing) | 19:07 |
*** ihrachyshka has joined #openstack-infra | 19:07 | |
anteaya | so that like, it actually returns results that are useful | 19:07 |
clarkb | mordred: because it is the backup host | 19:07 |
clarkb | mordred: idea is it won't automatically foobar on a puppet run | 19:07 |
clarkb | did puppet run against it? we may not have re certed it yet | 19:08 |
mordred | anteaya: https://www.google.com/#q=useful+things+site:wiki.openstack.org works well? | 19:08 |
krotscheck | mordred: Noticed, thanks. | 19:08 |
mordred | clarkb: it gets picked up in the puppet cert list output | 19:08 |
clarkb | mordred: huh | 19:08 |
mordred | clarkb: so, we should run puppet on it by hand, but then also run "puppet agent --disable" | 19:08 |
clarkb | ya | 19:09 |
mordred | clarkb: OR - we should make an "excluded_hosts" array to block running on hosts that puppet knows about but which we know we don't want to run thigns on | 19:09 |
mordred | or both | 19:09 |
clarkb | I like both | 19:10 |
*** jamespage_ has quit IRC | 19:10 | |
anteaya | mordred: yes it does | 19:10 |
*** jamespage_ has joined #openstack-infra | 19:10 | |
clarkb | mordred: when you say things can't ssh to it, what in particular can't? | 19:11 |
clarkb | mordred: eg are backups failing or is it just your ssh script thing? | 19:11 |
mordred | the ssh script thing - and my personal user accoutn | 19:11 |
openstackgerrit | A change was merged to openstack-infra/config: Properly don't fail on success https://review.openstack.org/88656 | 19:13 |
openstackgerrit | A change was merged to openstack-infra/config: Put a default section header above the TTL https://review.openstack.org/88660 | 19:14 |
clarkb | I just realized I should've written a test for the feature test matrix thing... I have written on locally which I will push as a second patch so that we can get the fix through | 19:17 |
clarkb | not ready for all this bash stuff to actually have tests | 19:18 |
mordred | ++ | 19:18 |
*** jamespage_ has quit IRC | 19:19 | |
*** dims-openstack is now known as dims | 19:24 | |
*** shakamunyi has quit IRC | 19:27 | |
*** shakayumi has joined #openstack-infra | 19:27 | |
*** BadCub01 has quit IRC | 19:32 | |
*** zns has quit IRC | 19:32 | |
jgriffith | clarkb: Questions on creating cinder-specs | 19:32 |
jgriffith | clarkb: clarkb cookie-cutter seems a bit overkill | 19:33 |
*** mark0n has joined #openstack-infra | 19:33 | |
jgriffith | clarkb: regardless, still need to figure out "creating" the repo on github/openstack to push to | 19:33 |
jgriffith | jeblair: ^^ Not sure who I should be harassing on this :) | 19:34 |
anteaya | jgriffith: so far I get you want to create cinder-specs repo | 19:36 |
anteaya | and the cookie-cutter doesn't answer all your questions | 19:36 |
dims | jgriffith, there's a specs cookit cutter too - http://git.openstack.org/cgit/openstack-dev/specs-cookiecutter/ | 19:36 |
jgriffith | anteaya: well... yeah, sort of | 19:36 |
jgriffith | anteaya: so cookie cutter seems overkill | 19:36 |
jgriffith | anteaya: ie tests etc etc | 19:36 |
jgriffith | anteaya: I can create the repo that way no problem | 19:37 |
dims | jgriffith, see the specs cookie-cutter ^^ it's stripped down | 19:37 |
jgriffith | anteaya: but creating a remote to push it too is the question :) | 19:37 |
jgriffith | dims: k | 19:37 |
jgriffith | dims: OHHH | 19:37 |
jgriffith | dims: awesome | 19:37 |
jgriffith | dims: thanks | 19:37 |
dhellmann | jgriffith: I usually create the remote on my own github account and then import it from there | 19:37 |
dhellmann | then I just delete that repo when I'm done | 19:38 |
jgriffith | dhellmann: makes sense | 19:38 |
dims | jgriffith, not my work, better minds than me - https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg20245.html | 19:38 |
*** lukego has joined #openstack-infra | 19:38 | |
dhellmann | the "openstack" project on github is essentially read-only | 19:38 |
*** marun_ has joined #openstack-infra | 19:38 | |
jgriffith | dhellmann: that's the part I ws trying to figure out... assumign I could just bug folks here again after I have it in my github :) | 19:39 |
*** marun has quit IRC | 19:39 | |
*** marun_ is now known as marun | 19:39 | |
dhellmann | jgriffith: this might help: https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary#Importing_the_Repository_into_the_CI_System | 19:39 |
anteaya | jgriffith: this might also help: https://review.openstack.org/#/c/79363/ | 19:39 |
anteaya | and you don't _have_ to create a repo anywhere | 19:39 |
anteaya | if the sole purpose is to create a new openstack repo | 19:40 |
jgriffith | Yay... all kinds of useful info | 19:40 |
jgriffith | thanks dims dhellmann and anteaya | 19:40 |
anteaya | we can start with nothing | 19:40 |
anteaya | np | 19:40 |
anteaya | if you use the topic:new-project for your patch, that helps us | 19:40 |
*** zns has joined #openstack-infra | 19:40 | |
anteaya | though since new project creation is going back to automated it isn't strictly necessary | 19:40 |
anteaya | though I do like reviewing all the new-project/new-repo patches at once | 19:41 |
anteaya | helps me spot missing items | 19:41 |
*** amcrn has quit IRC | 19:44 | |
*** lukego has quit IRC | 19:45 | |
*** dstanek has quit IRC | 19:46 | |
*** jamespage_ has joined #openstack-infra | 19:46 | |
*** jamespage_ has quit IRC | 19:48 | |
*** jp_at_hp has quit IRC | 19:50 | |
clarkb | I just self approved the test matrix fix | 19:55 |
clarkb | check testing looked good | 19:55 |
*** e0ne has joined #openstack-infra | 19:56 | |
mordred | clarkb: ++ | 19:57 |
mordred | clarkb: I'm going to land the revert revert revert patch | 19:58 |
clarkb | ok | 19:58 |
*** amcrn has joined #openstack-infra | 20:02 | |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Test matrix generation for feature branches. https://review.openstack.org/88671 | 20:02 |
clarkb | mordred: for your pbr revert I think you need to revert 9375a768100f16b5493062449d1274899b900cfd at the same time | 20:04 |
clarkb | mordred: to get the parse mailmap function back | 20:04 |
openstackgerrit | A change was merged to openstack-infra/config: Revert "Revert "Revert "Stop running manage-projects""" https://review.openstack.org/87343 | 20:04 |
*** thuc has joined #openstack-infra | 20:09 | |
*** thuc_ has joined #openstack-infra | 20:13 | |
*** thuc has quit IRC | 20:16 | |
*** e0ne has quit IRC | 20:16 | |
openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Expose assignee_id to API https://review.openstack.org/88676 | 20:19 |
*** chandan_kumar has quit IRC | 20:21 | |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Refactor the `matches` function https://review.openstack.org/88678 | 20:25 |
*** marun has quit IRC | 20:26 | |
openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed text display on Task Status dropdown. https://review.openstack.org/88679 | 20:26 |
*** e0ne has joined #openstack-infra | 20:27 | |
*** sweston has quit IRC | 20:28 | |
*** sweston has joined #openstack-infra | 20:29 | |
*** e0ne has quit IRC | 20:30 | |
openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Support the Store SCM plugin https://review.openstack.org/85729 | 20:30 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: WIP: Use ansible instead of direct ssh calls https://review.openstack.org/87821 | 20:32 |
mordred | jeblair, clarkb, anteaya: ^^ I forgot to publish the POC script with teh last version of the patch | 20:33 |
mordred | again - as I mention in the commit - I don't think that will actually work as is right now - merely a point of comparison | 20:33 |
*** imcsk8_lap has quit IRC | 20:34 | |
*** sweston has quit IRC | 20:34 | |
*** imcsk8 has quit IRC | 20:34 | |
*** marun has joined #openstack-infra | 20:35 | |
*** imcsk8 has joined #openstack-infra | 20:35 | |
openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Fix envvars in devstack-rally-gate builder https://review.openstack.org/88682 | 20:36 |
openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/devstack-gate: Add RUN_EXERCISES to generated localrc https://review.openstack.org/88683 | 20:36 |
*** miqui has quit IRC | 20:38 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: WIP: Use ansible instead of direct ssh calls https://review.openstack.org/87821 | 20:38 |
rediskin | could someone tell why envvars are cleaned here? https://review.openstack.org/88682 | 20:38 |
rediskin | SergeyLukjanov sdague fungi ^^ | 20:38 |
*** imcsk8 has quit IRC | 20:39 | |
rediskin | in this log you can see how RUN_EXERCISES is exported, but later it dissapeared http://logs.openstack.org/45/86945/3/check/check-rally-dsvm-rally/08f8be0/console.html | 20:39 |
*** david_lyle_ has joined #openstack-infra | 20:40 | |
rediskin | here it exported http://logs.openstack.org/45/86945/3/check/check-rally-dsvm-rally/08f8be0/console.html#_2014-04-18_19_32_47_300 | 20:40 |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command https://review.openstack.org/85580 | 20:40 |
rediskin | and here is gone http://logs.openstack.org/45/86945/3/check/check-rally-dsvm-rally/08f8be0/console.html#_2014-04-18_19_46_01_330 | 20:40 |
*** imcsk8 has joined #openstack-infra | 20:40 | |
* anteaya clicks to read mordred's latest ramblings | 20:41 | |
*** nati_ueno has joined #openstack-infra | 20:41 | |
*** dklyle has joined #openstack-infra | 20:41 | |
clarkb | rediskin: they shouldn't be | 20:42 |
*** david-lyle has quit IRC | 20:43 | |
rediskin | clarkb: they shouldn't be dissapeared? xD | 20:44 |
clarkb | correct | 20:44 |
clarkb | is devstack doing something silly? | 20:44 |
boris-42 | clarkb "is devstack doing something silly?" lol=) | 20:45 |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command https://review.openstack.org/85580 | 20:45 |
rediskin | clarkb: i thought it was devstack-gate, but i dont found anything | 20:45 |
clarkb | rediskin: I don't think devstack gate does that | 20:45 |
*** david_lyle_ has quit IRC | 20:45 | |
clarkb | as this methods works for lots of other projects | 20:45 |
clarkb | and there are no cases of unset in devstack gate | 20:45 |
*** imcsk8 has quit IRC | 20:45 | |
rediskin | clarkb: but as you can see at that log, it is all gone | 20:45 |
clarkb | rediskin: yes, I agree it is gone | 20:46 |
asselin | Hi, I have check-tempest-dsvm-full basically working in my nodepool infrastructure. I'm looking to add support for cinder certification tests on real HW. There's quite a few files that need to be updated across multiple projects. Since the scripts pull many of these files from git, is there any recommended approach to develop and test local changes? | 20:46 |
rediskin | poltergeist | 20:46 |
clarkb | I suppose it may start a subshell without inheriting the env somewhere | 20:46 |
clarkb | asselin: yeah, we use the zuul provided ref. devstack gate will check that out for each project | 20:47 |
clarkb | asselin: so you can create a ref with the same name in each project with the changes you need then feed that as teh zuul ref to devstack gate | 20:47 |
clarkb | rediskin: I don't think that change will fix anything though | 20:48 |
clarkb | exporting in the pre test hook vs exporting before d-g starts shouldn't make a difference | 20:48 |
clarkb | rediskin: has that actually been tested? | 20:48 |
rediskin | clarkb: it will. it was working in old version of this job | 20:48 |
*** hashar is now known as FoobAR300 | 20:48 | |
*** thuc_ has quit IRC | 20:49 | |
rediskin | clarkb: looks like evvars dropping happened before pre_test_hook is called | 20:49 |
asselin | clarkb, so I make the change locally, and do the git review for each project to push into openstack gerrit? does that create the zuul ref? how to make it the same ref across projects? | 20:49 |
*** thuc has joined #openstack-infra | 20:50 | |
*** harlowja is now known as harlowja_away | 20:50 | |
clarkb | asselin: well zuul creates it for us based on the order or approved jobs | 20:50 |
clarkb | you are a bit on your own if trying to do it without zuul | 20:51 |
asselin | clarkb, so do I use a local version of zuul? or openstack zuul? http://status.openstack.org/zuul/ | 20:52 |
clarkb | asselin: you would need a local instance of zuul | 20:52 |
*** e0ne has joined #openstack-infra | 20:52 | |
*** FoobAR300 is now known as hashar | 20:53 | |
*** nati_ueno has quit IRC | 20:53 | |
*** imcsk8_lap has joined #openstack-infra | 20:53 | |
clarkb | rediskin: so if we read d-g, ENABLED_SERVICES isn't used until after the pre_test hook | 20:53 |
clarkb | I think | 20:53 |
*** jamespage_ has joined #openstack-infra | 20:53 | |
*** hashar is now known as insecte | 20:54 | |
clarkb | rediskin: yup, ENABLED_SERVICES is used in gate_hook which happens after pre_test_hook | 20:54 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add some system logging to run_all https://review.openstack.org/88685 | 20:54 |
clarkb | so the entire env isn't being dropped | 20:54 |
mordred | clarkb: ^^ | 20:54 |
clarkb | otehrwise enabled services wouldn't work | 20:54 |
clarkb | but we have tempest telling us it does work | 20:54 |
clarkb | rediskin: so I don't think that is the issue. Would be good to understand why this happens | 20:54 |
*** thuc has quit IRC | 20:55 | |
*** insecte is now known as wmf-insecte | 20:55 | |
*** wmf-insecte is now known as hashar | 20:56 | |
clarkb | rediskin: and gate_hook is what runs the exercises | 20:56 |
asselin | clarkb, suppose I wanted to make the changes using openstack's infrastructure, what would be the best approach? i.e. add a new environment to tempext tox, and entry in devstack-vm-gate, etc... | 20:57 |
*** dklyle has quit IRC | 20:57 | |
clarkb | asselin: I'm not sure there is a way to do that if you need specialized hardware | 20:58 |
*** imcsk8_lap has quit IRC | 20:58 | |
clarkb | asselin: we do everything with open source tools on donated cloud resources | 20:58 |
*** wchrisj_ has quit IRC | 20:58 | |
clarkb | asselin: anything we can test in that environment we can definitely do what you describe | 20:58 |
asselin | clarkb, no, I would run the LVM version. then when that works, set it up for read hardware | 20:58 |
*** jamespage_ has quit IRC | 20:59 | |
clarkb | asselin: right so anything you can do on our donated cloud resources with open tools you can just edit the existing JJB jobs | 21:01 |
clarkb | asselin: in some cases that might mean extending existing tests or adding new ones | 21:01 |
rediskin | clarkb: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L245 | 21:03 |
rediskin | sudo -H drops envvars | 21:03 |
*** david-lyle has joined #openstack-infra | 21:04 | |
rediskin | clarkb: so this probably should partially fix my problem https://review.openstack.org/#/c/88683/ | 21:04 |
clarkb | cool so your fix shouldn't fix it | 21:04 |
clarkb | also I don't see RALLY_SCENARIOS used by anything in the rally devstack contrib stuff | 21:04 |
clarkb | is that for future thigns? | 21:04 |
asselin | clarkb, this is the part I'm getting stuck on conceptually. I need a new jenkins job (config project) that needs updates to tempest project (new test) and devstack-gate project (invoke new test).... | 21:05 |
rediskin | clarkb: yes, this is for future things | 21:05 |
asselin | clarkb, so that's 3 separate commits for everything to work | 21:05 |
rediskin | clarkb: i thought all envvars are available, so now i dont know what to do. i gonna think about it. thanks for help | 21:06 |
clarkb | rediskin: you can provide your own gate_hook | 21:06 |
clarkb | rediskin: and when you do that ensure that the variables you need are passed throguh | 21:07 |
clarkb | that might require some wheel reinvention though | 21:07 |
rediskin | clarkb: yea. thanks | 21:07 |
clarkb | asselin: we try to do these things in a self tested manner. Usually that means get a job in place that is in the experimental pipeline for the projects that you need to touch | 21:08 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: php5-json is provided by php5-common https://review.openstack.org/88692 | 21:08 |
*** dcramer_ has quit IRC | 21:08 | |
clarkb | asselin: then you can push your changes to those projects and comment with `check experimental` to run that test. Eventually you should be able to get it to pass at which point it can be made a normal test | 21:08 |
*** mfer has quit IRC | 21:09 | |
asselin | clarkb, ok I'll take a look at that thanks | 21:09 |
clarkb | rediskin: my initial thought is we should move setup_localrc to the functions file so that it can be sourced and called without all that extra stuff | 21:09 |
clarkb | rediskin: then your custom gate_hook can set up env vars, call that function, then run devstack exercises directly | 21:10 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add some system logging to run_all https://review.openstack.org/88685 | 21:11 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: php5-json is provided by php5-common https://review.openstack.org/88692 | 21:12 |
mordred | clarkb: would you take an eyeball at https://review.openstack.org/88685 ? I'd like to land that so that it's easier to watch the project stuff happen when we land a new project change | 21:12 |
mordred | and then I'd like to try landing new project changes | 21:12 |
clarkb | mordred: can you see my comment on patchset 1 | 21:13 |
mordred | oh - looking | 21:13 |
clarkb | mordred: it covers a thing you fixed in ps2 but also one other thing | 21:13 |
clarkb | I just managed to apply the comment, sorry for the delay | 21:13 |
mordred | no worries - race condition in commenting :) | 21:14 |
clarkb | mordred: I think the etherpad logrotate does the thing I describe | 21:15 |
clarkb | possibly also the logstash worker log rotates | 21:16 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add some system logging to run_all https://review.openstack.org/88685 | 21:16 |
mordred | clarkb: there yah go | 21:16 |
mordred | copytruncate seems to be the truck | 21:16 |
mordred | trick | 21:16 |
clarkb | yup | 21:16 |
anteaya | I stuck my +1 on it too | 21:18 |
anteaya | though I only understood about 1/2 of it | 21:19 |
*** dprince has quit IRC | 21:19 | |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add `browse` command https://review.openstack.org/85580 | 21:20 |
*** imcsk8 has joined #openstack-infra | 21:20 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Stop trying to start gerrit https://review.openstack.org/88650 | 21:21 |
mordred | clarkb: any opposition from me going ahead and landing that - I'd like to see some logs roll by | 21:21 |
clarkb | mordred: not from me | 21:21 |
rediskin | clarkb: what about just ask sudo to preserve envvars? | 21:22 |
*** _nadya_ has joined #openstack-infra | 21:22 | |
clarkb | rediskin: I don't think we want to do that in the general case | 21:22 |
clarkb | rediskin: the idea is that devstack gate will configure devstack via localrc | 21:22 |
clarkb | leakage around that could lead to subtle brokeness in devsatck | 21:22 |
*** prad_ has quit IRC | 21:22 | |
rediskin | sad | 21:22 |
clarkb | mordred: ^ thoughts? | 21:22 |
mordred | yah. I agree. can the hook write the values into a file? | 21:23 |
openstackgerrit | A change was merged to openstack-infra/config: Add some system logging to run_all https://review.openstack.org/88685 | 21:24 |
mordred | hehe | 21:25 |
mordred | root 5880 5879 0 21:19 ? 00:00:00 bash -c brazenly_run_ssh cacti.openstack.org | 21:25 |
mordred | that's just fun to see in the ps output | 21:26 |
*** _nadya_ has quit IRC | 21:26 | |
clarkb | brazenly_run_ssh is a thing I missed | 21:26 |
*** imcsk8 has quit IRC | 21:26 | |
clarkb | now I wish I had managed to review those fixes | 21:26 |
clarkb | because that sounds fun | 21:26 |
mordred | clarkb: it's "run ssh commands and categorically ignore antyhing that goes wrong" | 21:27 |
anteaya | it explictly returns 0 | 21:27 |
clarkb | mordred: is that good for the make sure thinsg run in order and don't fail stuff? | 21:27 |
mordred | clarkb: there are two functions | 21:27 |
clarkb | ah | 21:27 |
mordred | clarkb: one of them cares very much about return codes | 21:27 |
mordred | one does not | 21:28 |
clarkb | btw did manage projects do its thing? | 21:28 |
mordred | haven't asked it to yet | 21:28 |
mordred | well - although, yeah - it should have run recently on review | 21:28 |
anteaya | does manage projects have anything to do? | 21:28 |
*** jamespage_ has joined #openstack-infra | 21:28 | |
clarkb | I think there was at least one ACL update pending | 21:28 |
anteaya | ok | 21:28 |
mordred | oh - neat - it should have run by now | 21:28 |
clarkb | we just merge those because ordering doesn't matter on an ACL update | 21:28 |
anteaya | these 3 are still sitting here: https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:new-project,n,z | 21:29 |
mordred | clarkb: we could also look and see if gerrit has new refs | 21:29 |
anteaya | 2 are good to go, solum-dashboard needs an edit | 21:29 |
anteaya | clarkb: right | 21:29 |
*** imcsk8 has joined #openstack-infra | 21:30 | |
mordred | oh - manage_projects will not have run | 21:31 |
mordred | because puppet will have applied the changes to the acls files ages ago | 21:31 |
mordred | so there will be nothign to trigger it | 21:31 |
clarkb | oh right | 21:31 |
clarkb | mordred: so I was thinking about that too | 21:32 |
clarkb | mordred: makefiles | 21:32 |
clarkb | oh ya I need to dig up my makefile for jeblair | 21:32 |
mordred | yup | 21:32 |
* clarkb boots other machine | 21:32 | |
mordred | you know krow managed all of his servers with make files for the last several years | 21:32 |
mordred | and thought we were crazy for using puppet at all | 21:33 |
clarkb | well I don't think puppet is completely crazy, but I do think make has been ignored as a useable tool | 21:33 |
clarkb | probably because of what sendmail did with it | 21:33 |
*** dcramer_ has joined #openstack-infra | 21:33 | |
mordred | nod | 21:33 |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add a "list" command https://review.openstack.org/87090 | 21:34 |
*** imcsk8 has quit IRC | 21:34 | |
mordred | clarkb: also - the virtualenv_version facter change helped | 21:34 |
mordred | so we're down to a smaller number of thigns that report activity all the time | 21:35 |
mordred | I think it might be close to time to consider fixing those | 21:35 |
*** sweston has joined #openstack-infra | 21:35 | |
rediskin | clarkb: btw ./exercise.sh is run after ./stack.sh is done and stack user removed from sudoers | 21:36 |
rediskin | clarkb: it cant broke too much if we preserve env for ./exercise.sh | 21:36 |
mordred | root 5880 5879 0 21:19 ? 00:00:00 bash -c brazenly_run_ssh cacti.openstack.org | 21:37 |
mordred | gah | 21:37 |
mordred | notice: /File[/etc/resolv.conf]/seltype: seltype changed 'file_t' to 'net_conf_t' | 21:37 |
clarkb | rediskin: well it could mean that devstack itself isn't able to run its exercises | 21:37 |
clarkb | rediskin: it woudl be possible for devstack to become dependent on devstack gate | 21:37 |
mordred | clarkb: ^^ on git03 (And ony on git03) we get that every time puppet is run | 21:37 |
mordred | pleia2: ^^ | 21:37 |
clarkb | mordred: probably related to jeblairs dns cache changes | 21:38 |
mordred | clarkb: but why only on one server? | 21:38 |
clarkb | no idea | 21:38 |
*** e0ne has quit IRC | 21:40 | |
*** harlowja_away is now known as harlowja | 21:41 | |
pleia2 | yeah, that's a bit weird | 21:41 |
pleia2 | might compare the selinux attributes on /etc/resolv.conf between servers, ie: ls -lZ /etc/resolv.conf | 21:42 |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a simple tox replacement Makefile. https://review.openstack.org/88698 | 21:42 |
clarkb | jeblair: mordred ^ there is the makefile I have been fiddling with. I have been using it in zuul hence the push to zuul | 21:43 |
clarkb | it is a bit repetitive and I am sure a make expert can clean it up a little. But make variable evaluation and dependency evaluation happens in such a way that having venvs like tox with make is hairy | 21:43 |
clarkb | we might want to consider a different structure but I wanted to keep it somewhat consistent | 21:44 |
anteaya | I see tabs | 21:44 |
anteaya | do makefiles need tabs? | 21:44 |
mordred | yup | 21:45 |
anteaya | okay then | 21:45 |
anteaya | good work on the tabs | 21:45 |
mordred | clarkb: figured it out - it was a sequencing thign - you have to have chattr +i set to change the selinux context - but puppet sets it to -i - so something originally got done in the wrong sequence | 21:45 |
mordred | s/+i/-i/ | 21:45 |
mordred | I have fixed it by hand - subseuqnet runs should be claen | 21:46 |
pleia2 | oh puppet | 21:46 |
jeblair | well, immutable files aren't really ideal for use with puppet | 21:46 |
*** marun_ has joined #openstack-infra | 21:46 | |
jeblair | it's hard to blame it for that :/ | 21:46 |
clarkb | jeblair: mordred: another neat trick with that makefile is now we can control pip/virtualenv ourselves | 21:47 |
clarkb | which may or may not be a good thing :) | 21:47 |
jeblair | i'd prefer to shift the blame to the crazy ways our providers manage to find to configure networking :) | 21:47 |
*** imcsk8 has joined #openstack-infra | 21:47 | |
*** imcsk8_lap has joined #openstack-infra | 21:47 | |
rediskin | clarkb: or maybe i can run ./exercises.sh in post_gate_hook without sudo -H | 21:48 |
clarkb | rediskin: oh, thats not a bad idea. set the vars such that gate_hook ends up being a noop | 21:48 |
clarkb | rediskin: then rerun in post_gate_hook | 21:48 |
clarkb | well not a complete noop you still want devstack to start, but no exercises | 21:48 |
mordred | clarkb: what do we expect check_elasticsearch_sha1 to do? | 21:49 |
clarkb | mordred: it ensures that the sha1 of the downloaded package matches the sha1 we expect it to be | 21:49 |
mordred | right - but to trigger what thing? | 21:49 |
clarkb | mordred: to make sure we isntall a good package | 21:49 |
clarkb | so it should only fire when we are reinstalling ES | 21:49 |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Handle feature branches in test-matrix.py https://review.openstack.org/88652 | 21:49 |
*** marun has quit IRC | 21:49 | |
*** marun_ is now known as marun | 21:49 | |
mordred | gotcha. it fires every time | 21:49 |
clarkb | notmyname: ^ I am going to kick your change now | 21:49 |
notmyname | clarkb: thanks. just saw the email | 21:51 |
*** dcramer_ has quit IRC | 21:54 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Silence successful elasticsearch runs https://review.openstack.org/88700 | 21:54 |
clarkb | jeblair: let me know what you think about that makefile | 21:55 |
clarkb | jeblair: but you should be able to git review -d and use make instead of tox locally | 21:55 |
*** imcsk8_l1p has joined #openstack-infra | 21:56 | |
mordred | jeblair http://puppetdb.openstack.org/report/review.openstack.org/f95f51a6a8bade4871256a1ac92264705a391cff gerrit puppet is unhappy and it's causing other thigns to not fire | 21:56 |
*** pdmars has quit IRC | 21:56 | |
*** imcsk8_lap has quit IRC | 21:57 | |
*** imcsk8 has quit IRC | 21:58 | |
*** marun has quit IRC | 21:58 | |
*** gokrokve has quit IRC | 21:58 | |
mordred | jeblair: oh - well, just running it a second time seems to have done the trick ::( | 21:59 |
*** marun has joined #openstack-infra | 21:59 | |
*** andreykurilin_ has quit IRC | 22:00 | |
*** nati_ueno has joined #openstack-infra | 22:01 | |
*** ihrachyshka has quit IRC | 22:02 | |
openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: `test` command read/write stdin/stdout by default https://review.openstack.org/84778 | 22:03 |
*** lukego has joined #openstack-infra | 22:05 | |
clarkb | oh! I can have make call make to work around the weird variable expansion and dep resolution | 22:06 |
mordred | jeblair: when you get a sec - I'd love a quick look at https://review.openstack.org/#/c/88650/ but also at https://review.openstack.org/#/c/88692/ and https://review.openstack.org/#/c/88700/ | 22:06 |
jeblair | mordred: heh, that's teh definition of insanity, but it also appears to be standard operating procedure with puppet. ;) | 22:06 |
mordred | jeblair: tell me about it | 22:06 |
mordred | jeblair: (I'm trying to clean up a few things before I try landing a new project so that there's less stuff to wade through to track success) | 22:07 |
openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Run rally ./exercise.sh inside post_gate_hook https://review.openstack.org/88682 | 22:08 |
openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Run rally ./exercise.sh inside post_test_hook https://review.openstack.org/88682 | 22:09 |
rediskin | clarkb: ^^ i hope it gonna work :) | 22:10 |
jeblair | mordred: so for the first thing, i think it's worth trying to understand why that doesn't work | 22:10 |
jeblair | the theory is that gerrit will auto restart when it's upgraded | 22:10 |
jeblair | and that's been the practice; so what broke it? | 22:10 |
*** thedodd has quit IRC | 22:11 | |
* rediskin went to sleep | 22:13 | |
*** rediskin has left #openstack-infra | 22:13 | |
clarkb | hrm nope submakes don't really fix the issue of one .testrepository and lots of venvs | 22:16 |
clarkb | oh I know | 22:16 |
jeblair | mordred: in order, -1, +2, -1 on those changes | 22:16 |
mordred | jeblair: awesome | 22:16 |
jeblair | mordred: maybe i missed something, but i think the 3rd one won't run anymore (clarkb you may want to look at https://review.openstack.org/#/c/88700/ again) | 22:17 |
mordred | jeblair: it's required by the install exec | 22:17 |
jeblair | mordred: does a require trigger an exec? i thought only notify or subscribe do | 22:17 |
jeblair | (if it's refreshonly, that is) | 22:18 |
mordred | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/elasticsearch/manifests/init.pp#n72 | 22:18 |
mordred | hrm. let me check | 22:18 |
clarkb | jeblair: oh good point. should s/require/subscribe/ | 22:19 |
*** rlandy has quit IRC | 22:19 | |
*** lascii is now known as alaski | 22:22 | |
mordred | jeblair: gerrit-initial-init is run every time | 22:22 |
*** hashar has quit IRC | 22:22 | |
*** imcsk8_l1p is now known as imcsk8 | 22:22 | |
mordred | it subscribes to /home/gerrit2/review_site/bin/gerrit.war | 22:22 |
mordred | but doesn't hav e refreshonly on it | 22:23 |
jeblair | mordred: yep, i have just come to that same conclusion | 22:23 |
jeblair | mordred: i don't know why it's necessary if gerrit-init is there... | 22:23 |
jeblair | mordred: i mean, i read the comment, but gerrit-init has a ';' in it, so shouldn't it ignore init script stop errors? | 22:24 |
mordred | yeah | 22:24 |
jeblair | mordred: iow, shouldn't gerrit-init be enough, and maybe we can drop gerrit-initial-init? | 22:24 |
*** dangers is now known as dangers_away | 22:24 | |
mordred | jeblair: ++ let's try that | 22:25 |
mordred | jeblair: you know - I think we had it in there originally and didn't care because the java command was a no0op - but that was before we needed to install the plugins too | 22:26 |
jeblair | mordred: you added initial-init in f774c793ba6d271b8b64430a9925aaafb65668fb | 22:26 |
*** nati_ueno has quit IRC | 22:26 | |
jeblair | mordred: oh! | 22:27 |
jeblair | onlyif and unless are important | 22:27 |
jeblair | mordred: it's supposed to do one or the other depending on whether the init script exists | 22:27 |
mordred | ah | 22:27 |
jeblair | mordred: so maybe the right thing is to add refreshonly to initial init | 22:27 |
mordred | ++ | 22:27 |
jeblair | oh wait, those lines disappeared | 22:28 |
jeblair | you added them when you added initial init, but they're gone now. i need git blame for when lines go away :) | 22:29 |
mordred | :) | 22:29 |
*** sweston has quit IRC | 22:30 | |
jeblair | 75cee29cd6be6e044488088b50af76be0c38526e | 22:31 |
mordred | jeblair: nod | 22:31 |
clarkb | jeblair: git log -p | 22:33 |
jeblair | how about this: add refreshonly to initial-init, add the onlyif and unless lines back to both inits, remove the unless from install-core-plugins, add refreshonly to install-core-plugins | 22:33 |
clarkb | jeblair: then I grep through that. its a bit of a sledghammer on thumb tack approach but works | 22:33 |
mordred | yup | 22:34 |
jeblair | clarkb: i used "git log -Sonlyif .../init.pp" | 22:34 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Stop trying to start gerrit every time https://review.openstack.org/88650 | 22:35 |
*** thomasem has quit IRC | 22:35 | |
jeblair | mordred: +2 | 22:36 |
*** mark0n has quit IRC | 22:36 | |
mordred | while I've got your attention ... | 22:37 |
*** jeblair is now known as corvus | 22:37 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Silence successful elasticsearch runs https://review.openstack.org/88700 | 22:38 |
mordred | how does that one look? | 22:38 |
mordred | clarkb: I can get second +2 on https://review.openstack.org/88650 ? | 22:38 |
mordred | corvus: if we don't watch out, we're going to have a mostly clean puppet run that runs at predictable times | 22:39 |
*** corvus is now known as jeblair | 22:40 | |
*** dcramer_ has joined #openstack-infra | 22:40 | |
jeblair | mordred: then we'll be ready to move to saltible, or whatever the cool new way to run something over ssh is! | 22:41 |
mordred | hehe | 22:41 |
mordred | jeblair: I uploaded the missing python script from the ansible patch | 22:41 |
clarkb | mordred: done | 22:42 |
*** alexpilotti has quit IRC | 22:42 | |
mordred | which might make it more betterer to actually, yo u know, read | 22:42 |
mordred | clarkb: ok. last one to bug you on: https://review.openstack.org/#/c/88700/ <-- does that look better to you now? | 22:43 |
openstackgerrit | A change was merged to openstack-infra/config: Stop trying to start gerrit every time https://review.openstack.org/88650 | 22:46 |
openstackgerrit | A change was merged to openstack-infra/config: php5-json is provided by php5-common https://review.openstack.org/88692 | 22:46 |
openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Story detail view cleanup https://review.openstack.org/88707 | 22:47 |
openstackgerrit | A change was merged to openstack-infra/config: Silence successful elasticsearch runs https://review.openstack.org/88700 | 22:50 |
*** msabramo has quit IRC | 22:51 | |
mordred | jeblair: woot! the gerrit change worked | 22:55 |
*** david-lyle has quit IRC | 22:55 | |
openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Story detail view cleanup https://review.openstack.org/88707 | 22:55 |
*** zns has quit IRC | 22:56 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix typo in resource name https://review.openstack.org/88708 | 22:59 |
*** dcramer_ has quit IRC | 23:01 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix two typos in updated elasticsearch module https://review.openstack.org/88708 | 23:03 |
mordred | self approving that ^^ there's a flat-out typo - the patch above is tested and fixes it | 23:04 |
jeblair | mordred: woo! | 23:04 |
*** msabramo has joined #openstack-infra | 23:04 | |
mordred | jeblair: yah man | 23:04 |
mordred | jeblair: once that bad boy lands, I'm a gonna land a new project and see if everything works like magic like it's supposed to | 23:04 |
*** dcramer_ has joined #openstack-infra | 23:05 | |
clarkb | ++ | 23:07 |
*** msabramo1 has joined #openstack-infra | 23:08 | |
*** msabramo has quit IRC | 23:08 | |
clarkb | I got nerd sniped by make again | 23:08 |
mordred | clarkb: it can suck your soul | 23:08 |
clarkb | I should have a second patchset for a more terse version of the makefile | 23:08 |
mordred | haha | 23:08 |
mordred | clarkb: so, be wary the arms race here | 23:09 |
mordred | because if you do that too much, I'm going to replace all of our puppet infrastructure with a pile of makefiles | 23:09 |
mordred | and THEN where will we be? | 23:09 |
clarkb | the 80s | 23:09 |
mordred | the only way forward then will be to install AFS | 23:09 |
*** CaptTofu has quit IRC | 23:10 | |
asselin | clarkb, thanks for the tips earlier...through that investigation I ended up finding some existing hooks I can tap into to run cinder cert tests on LVM "backend" which pass in my setup. | 23:10 |
mordred | and then re-write all of openstack as a set of Makefiles that call a series of small C programs to do thigns | 23:10 |
*** CaptTofu has joined #openstack-infra | 23:10 | |
clarkb | asselin: cool | 23:10 |
clarkb | mordred: oh man | 23:11 |
clarkb | mordred: openstack will be like NIS | 23:11 |
mordred | clarkb: except better, because we'd use hesiod instead | 23:11 |
asselin | given that my setup is proving to work, I'd like to get some reviews in my patch. https://review.openstack.org/#/c/87384/ | 23:12 |
clarkb | you know you are smoking the make crack when you break into the functions to generate python versions from target names | 23:12 |
*** wchrisj has joined #openstack-infra | 23:13 | |
mordred | clarkb: that's when it's getting good | 23:14 |
* nibalizer actually does want to install kerberos and AFS | 23:14 | |
nibalizer | which seems to come up a lot on fridays | 23:14 |
mordred | nibalizer: I do too. I'm REALLY not kidding when I say that | 23:14 |
*** CaptTofu has quit IRC | 23:14 | |
*** dims has quit IRC | 23:15 | |
nibalizer | mordred: do we just set it up, or do we try to find a purpose first? | 23:15 |
*** dcramer_ has quit IRC | 23:15 | |
*** nati_ueno has joined #openstack-infra | 23:17 | |
mordred | nibalizer: oh, we have so many purposes | 23:17 |
*** lcostantino has joined #openstack-infra | 23:20 | |
*** dcramer_ has joined #openstack-infra | 23:28 | |
*** UtahDave has quit IRC | 23:28 | |
mordred | WOW - why are there so many changes in the nova queue | 23:29 |
mordred | flashgordon: I blame you personally for this ^^ | 23:30 |
*** sweston has joined #openstack-infra | 23:31 | |
flashgordon | mordred: Sreeram Chiropractor | 23:34 |
flashgordon | https://review.openstack.org/#/c/84455/ | 23:34 |
flashgordon | he is running recheck-vmware on all thethings | 23:35 |
*** sweston has quit IRC | 23:35 | |
tjones | minesweeper is back up after a very long time.... | 23:35 |
tjones | "Sreeram Chiropractor" hee hee | 23:36 |
mordred | flashgordon: I hate everyone | 23:37 |
mordred | flashgordon: I just want to land an infra patch | 23:37 |
clarkb | mordred: I think this is a good illustration for why it is insane that tempest needs 31 nodes to test one patch | 23:38 |
flashgordon | clarkb: haha 31 nodes wow | 23:39 |
*** dcramer_ has quit IRC | 23:42 | |
*** dims has joined #openstack-infra | 23:42 | |
jeblair | mordred: the zuul job graph does not look favorable | 23:43 |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a simple tox replacement Makefile. https://review.openstack.org/88698 | 23:44 |
clarkb | mordred: jeblair ^ that patchset makes the makefile less copy pastay | 23:44 |
mordred | jeblair: it looks like hipsters found a new foodtruck | 23:45 |
clarkb | ok I need to get ready for portland trip | 23:46 |
* clarkb afks | 23:46 | |
jeblair | anyone know who Sreeram Yerrapragada is? i mean, that's a lot of rechecking. | 23:46 |
clarkb | before I go getting https://review.openstack.org/#/c/88671/ in would be good | 23:47 |
*** bhuvan has quit IRC | 23:47 | |
clarkb | jeblair: no clue | 23:47 |
*** lcostantino has quit IRC | 23:47 | |
jeblair | clarkb: nothings going in for hours :/ | 23:47 |
clarkb | jeblair: gate does have precedence | 23:47 |
mordred | gate requires check | 23:47 |
clarkb | so I think stuff should merge if it managed to check prior to that rush | 23:47 |
clarkb | mordred: ya 88671 has check | 23:47 |
mordred | +A | 23:48 |
jesusaurus | im looking at all the various puppet modules around the jenkins slaves, and its a bit of spaghetti. as far as i can tell, anything with "bare_node" in it's name sets bare=false | 23:48 |
mnaser | lol. | 23:49 |
mnaser | dat check queue. | 23:50 |
flashgordon | I like that spike | 23:50 |
mordred | jesusaurus: the two words are a bit overused | 23:50 |
clarkb | yup and I fixed some of the spaghetti recently | 23:51 |
clarkb | its a lot better than it was | 23:51 |
clarkb | jesusaurus: the issue there is we have bare == no services because devsatck and bare == service but no crazy devstack stuff | 23:51 |
jesusaurus | clarkb: wow. i want to refactor some of this, but its hard to tell where to start | 23:51 |
mordred | jesusaurus: in devstack-gate- "bare" means "not a devstack node" - in puppet, bare means "doesn't have infra creds" | 23:51 |
clarkb | it got overloaded in a bad way | 23:51 |
jesusaurus | oh, ugh | 23:51 |
clarkb | jesusaurus: there are only two entry points. single_use_slave and slave.pp | 23:51 |
mordred | jesusaurus: AaronGr also wants to refactor some of it - I think because he similarly wanted to figure something out and got confused | 23:52 |
clarkb | which is better than it was before I hacked on it | 23:52 |
jesusaurus | clarkb: what about puppet apply in prepare_node? | 23:52 |
clarkb | jesusaurus: that should be single_use_slave | 23:52 |
jeblair | we have no plain precise nodes now, so we can actually change the 'bare-precise' node label to 'precise' | 23:52 |
mordred | jeblair: ++ | 23:52 |
clarkb | jesusaurus: you may be looking at mordreds fork which is probably speghetti | 23:53 |
clarkb | but if you look upstream it is much better than befre (but bare is still overloaded terribly) | 23:53 |
jesusaurus | im looking at origin/master (where my origin is the github mirror) | 23:53 |
clarkb | jesusaurus: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/nodepool/scripts/prepare_node.sh#n45 single_use_slave | 23:54 |
jesusaurus | ah, yep | 23:54 |
jesusaurus | im really just trying to wrap my head around everywhere that bare gets used in the puppet modules | 23:55 |
clarkb | jesusaurus: see fc249c341fbc8c3ae7195965316769ac8ee2b78e | 23:56 |
jesusaurus | jenkins::slave seems to have a good bit of logic around whether thats set, and i want to move a lot of the openstack-infra specific stuff out of ::jenkins and into ::openstack_project | 23:56 |
clarkb | jesusaurus: and 0f4b0fae644d1c1156e87466d3988d86d6c39a18 | 23:57 |
jesusaurus | ah, fc249c341 is enlightening | 23:57 |
*** khyati has quit IRC | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!