| clarkb | sdague: jogo 2014-03-10 23:56:33,409 Pushing 732795 log lines. is the problem | 00:00 |
|---|---|---|
| clarkb | that is from the libvirtd logs | 00:00 |
| *** esker has joined #openstack-infra | 00:01 | |
| clarkb | I think we must have a lot of those http://logs.openstack.org/89/74889/6/check/check-tempest-dsvm-savanna-postgres-nv/1b275b7/logs/libvirtd.txt.gz | 00:01 |
| jeblair | clarkb: that is several lines. | 00:01 |
| clarkb | jeblair: yeah its several orders of magnitude more than what I expect | 00:01 |
| * jogo hopes he doesn't crash his browser | 00:02 | |
| clarkb | http://logs.openstack.org/14/77814/1/check/check-tempest-dsvm-full/b1beec5/logs/libvirtd.txt.gz is a nicer file | 00:02 |
| clarkb | much smaller | 00:02 |
| jeblair | clarkb: maybe we should have a "wc -l" check on log lines in the gate. | 00:02 |
| SergeyLukjanov | Oh, my phone is warming up trying to open this | 00:02 |
| clarkb | SergeyLukjanov: hahahaha | 00:02 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: dequeue abandoned changes https://review.openstack.org/65467 | 00:02 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Cancel a build even if not found https://review.openstack.org/79391 | 00:02 |
| clarkb | SergeyLukjanov: careful | 00:02 |
| * StevenK waits for "Now my phone is glowing red" | 00:03 | |
| clarkb | things that are silly, running watch on a pidfile, but it has freed me up to do other things | 00:03 |
| clarkb | jogo: though that looks like dims change so maybe not the root of all evil | 00:04 |
| clarkb | I just noticed that it has a worker way bogged down | 00:05 |
| jogo | lolz 60MB uncompressed | 00:06 |
| *** rfolco has joined #openstack-infra | 00:07 | |
| fungi | jeblair: so you're saying <1300 seems like a better option for minimizing impact? i can do 1200 on wednesday instead in that case | 00:08 |
| fungi | apparently the project "wakes up" at ~9am usa east coast time | 00:09 |
| fungi | interesting metric | 00:09 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Be more careful with locks https://review.openstack.org/79469 | 00:10 |
| fungi | i'm very curious why the graphite graphs seem to be in edt even though i have *all* my systems set with utc as local time. is firefox trying to be way, way, way too smart? | 00:11 |
| clarkb | fungi: yes Ithink so | 00:11 |
| clarkb | fungi: it was showing me MST at one point | 00:11 |
| fungi | like, callout to some external service which geocodes your source ip address based on estimated timezone of your service provider too smart? | 00:12 |
| clarkb | no I don't think that smart | 00:12 |
| clarkb | but maybe? | 00:12 |
| clarkb | also DST | 00:12 |
| clarkb | I have been awake since what feels like 5am | 00:12 |
| jeblair | fungi: i would prefer earlier; i feel like people are already plenty busy by the time i wake up. but i don't want to push you into a time you aren't comfortable with. if that's the case, then i fully support just doing this on friday | 00:12 |
| fungi | i'm perfectly comfortable whenever. SergeyLukjanov: let's do 12:00 utc on wednesday. i'll send an announcement | 00:13 |
| * fungi spent decades doing 3am maintenance on sundays and wednesdays. this is a cake walk | 00:13 | |
| jeblair | TIME_ZONE | 00:16 |
| jeblair | Default: America/Chicago | 00:16 |
| jeblair | graphite ^ | 00:16 |
| *** weshay has joined #openstack-infra | 00:16 | |
| jeblair | so it's on CST because it hasn't restarted since the switch | 00:16 |
| fungi | weird | 00:17 |
| clarkb | ah | 00:17 |
| *** blamar has quit IRC | 00:17 | |
| fungi | where does it get "TIME_ZONE" from, and can we correct that to utc somewhere? | 00:17 |
| jeblair | i think they made that a lot better in the latest version (using the browser more), but that requires django 1.4 | 00:17 |
| *** atiwari has quit IRC | 00:17 | |
| jeblair | fungi: yeah, a local settings file, i'm working up a patch | 00:17 |
| fungi | jeblair: oh--cool, thanks@ | 00:17 |
| jeblair | http://graphite.readthedocs.org/en/latest/config-local-settings.html and http://en.wikipedia.org/wiki/Zoneinfo#Names_of_time_zones | 00:18 |
| jeblair | suggest that it should probably be "Etc/UTC" ? | 00:18 |
| fungi | answering the question of "why aren't these graphs correct for my local time zone?" is a lot easier if the answer is "utc, suck it up" rather than "chicago, _of course_" | 00:18 |
| * jeblair cancels his move to chicago | 00:19 | |
| fungi | Etc/UTC yeah | 00:19 |
| fungi | though i think posix may also just include a bare "UTC" on top of that | 00:20 |
| jeblair | yeah, this is the var they got rid of on master, so this will hopefully be a short lived fix (once we move to 14.04) | 00:20 |
| fungi | i know on some fairly ancient posix-compliant systems i used to just feed in "UTC" (with no prefix) and they would dtrt | 00:20 |
| jeblair | okay, i just tested 'Etc/UTC' in production | 00:21 |
| jeblair | looks like that worked | 00:21 |
| *** weshay has quit IRC | 00:21 | |
| jeblair | now i just have to figure out how to get it to stop using the nonsensical "12AM" and "12PM" | 00:21 |
| fungi | best place to test. also, shoot from the hip | 00:21 |
| *** matsuhashi has joined #openstack-infra | 00:21 | |
| fungi | 12am and 12pm are ancient earth terms for 00:00 and 12:00 | 00:22 |
| fungi | i learnt that in my stasis pod instructional programs | 00:22 |
| clarkb | jogo: fwiw looking at indexing rates and so on, I think something changed recently in the logs we are indexing | 00:23 |
| jeblair | it's totally 12 hours before noon! no it's definitely 12 hours after noon! is it ever actually noon? | 00:23 |
| clarkb | jogo: we are actually doing a good job from a rate perspective which makes me think we bumped the amount of data to index somewhere | 00:24 |
| jogo | the libvirt file? | 00:24 |
| clarkb | jogo: it might be libvirt or something else | 00:24 |
| *** yamahata__ has joined #openstack-infra | 00:24 | |
| fungi | jeblair: noon is irrelevant when we all move subsurface to avoid being bathed in deadly solar radiation | 00:24 |
| clarkb | libvirt stood out to me because we spent a bunch of time indexing that one file but that is dims test change | 00:24 |
| clarkb | jogo: so its impact may be small overall | 00:24 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Have graphite use the One True Timezone. https://review.openstack.org/79495 | 00:24 |
| jeblair | fungi: The sooner the better says this reporter! | 00:25 |
| jogo | clarkb: ahh Hmm | 00:25 |
| devananda | for the python ironic agent project, what's the deciding factor whether to use a separate launchpad group/bugs/etc vs. share with ironic? | 00:26 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a script to manage IRC perms https://review.openstack.org/78483 | 00:26 |
| fungi | devananda: laziness | 00:27 |
| fungi | devananda: or, to use a synonym, convenience | 00:27 |
| fungi | (whichever works better for your community) | 00:27 |
| devananda | :) | 00:27 |
| devananda | if there's no policy on it, we'll be convenient | 00:28 |
| *** sarob_ has quit IRC | 00:28 | |
| fungi | please be convenient. it leaves you more time to help us fix things! ;) | 00:28 |
| *** bhuvan_ has quit IRC | 00:28 | |
| *** bhuvan has quit IRC | 00:28 | |
| clarkb | jogo: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all logs.o.o seems like it may be pushing more bits recently | 00:28 |
| *** sarob_ has joined #openstack-infra | 00:28 | |
| fungi | clarkb: ~1kjph was the tipping point we saw before between "logstash keeps up" and "logstash falls behind" | 00:30 |
| clarkb | fungi: that sounds about right | 00:30 |
| fungi | and we did hit that around when the steady climb began, but i figured it would have recovered by now | 00:30 |
| *** ianw has quit IRC | 00:31 | |
| *** ianw has joined #openstack-infra | 00:32 | |
| jeblair | yeah, we've been << 1kjph for >2h now | 00:33 |
| *** sarob_ has quit IRC | 00:33 | |
| *** blamar has joined #openstack-infra | 00:34 | |
| *** nati_uen_ has quit IRC | 00:35 | |
| jeblair | btw, i find it very interesting that there are now _several_ openstack manual jobs stuck on queued jobs, and they all seem to be either openstack-config-reference or openstack-user-guide-admin. | 00:37 |
| *** sarob_ has joined #openstack-infra | 00:37 | |
| *** thomasem has joined #openstack-infra | 00:38 | |
| *** wchrisj has quit IRC | 00:39 | |
| jeblair | clarkb: what's your latest thinking on logstash? anything i can help with? | 00:39 |
| *** esker has quit IRC | 00:39 | |
| clarkb | jeblair: if folks want to grep through log-worker-* logs to see which files are taking a long time or having lots of lines we might be able to pinpoint why things have changed | 00:39 |
| *** CaptTofu has quit IRC | 00:39 | |
| clarkb | the backend throughput seems to be fine so I think it is something further upstream like files grew | 00:39 |
| *** esker has joined #openstack-infra | 00:39 | |
| StevenK | Heh. Zuul has two gate jobs marked as waiting for the window that are running jobs | 00:40 |
| jeblair | StevenK, clarkb: ^ that seems bug-like. | 00:40 |
| fungi | StevenK: that can happen if they were already running before it shrank the window? | 00:40 |
| clarkb | fungi: yes that | 00:41 |
| *** jungleboyj has joined #openstack-infra | 00:41 | |
| clarkb | fungi: we won't kill jobs that already started because we may as well try to use those resources | 00:41 |
| jeblair | so more of a ui bug. | 00:41 |
| *** wchrisj has joined #openstack-infra | 00:41 | |
| StevenK | I'd agree with that. The windowing stuff doesn't seem to "fit" well in the UI | 00:42 |
| jeblair | clarkb: logstash worker 03 is dead | 00:42 |
| jeblair | according to graphite | 00:42 |
| *** sarob_ has quit IRC | 00:42 | |
| jeblair | s/graphite/cacti/ | 00:42 |
| *** yamahata has joined #openstack-infra | 00:42 | |
| clarkb | jeblair: huh it is connected, but I believe it (watchdog won't catch it if it is still connected) | 00:43 |
| *** wchrisj has quit IRC | 00:43 | |
| jeblair | clarkb: i just mean the host is doing no work | 00:43 |
| clarkb | jeblair: ya | 00:43 |
| *** thomasem has quit IRC | 00:43 | |
| clarkb | jeblair: 11 and 12 too | 00:43 |
| *** esker has quit IRC | 00:44 | |
| clarkb | jeblair: 14 and 16 recently flapped, but I don't see anything in the logs about 3,11,12 | 00:45 |
| jeblair | clarkb: yeah. also, some hosts are moving a lot more data than normal, but maybe those hosts are just dealing with that very large job you found? | 00:45 |
| clarkb | jeblair: want to dump the jvm for the logstash process there? | 00:45 |
| clarkb | jeblair: or picking up the slack | 00:46 |
| jeblair | clarkb: will do | 00:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Have graphite use the One True Timezone. https://review.openstack.org/79495 | 00:46 |
| clarkb | jeblair: then I think we should restart the logstash-indexer service on 3,11,12 | 00:46 |
| portante | jogo: you asked about devstack/swift changes? | 00:46 |
| clarkb | jeblair: also possible that the spike in traffic there is related to a spike in the size of the logs | 00:46 |
| jeblair | clarkb: so i have to install the jdk on 03 | 00:47 |
| jeblair | it's going to install lots of packages | 00:47 |
| clarkb | :/ I think that should be ok | 00:47 |
| clarkb | its not great but not terrible either | 00:47 |
| jeblair | yeah, those hosts are easy to rebuild | 00:47 |
| jeblair | oh yeah it's jruby | 00:49 |
| jeblair | clarkb: ~root/stack on 03 | 00:49 |
| portante | jogo: see commit f19ccb6 in devstack | 00:49 |
| fungi | jruby? is that some mutant hybrid offspring of the two languages i'm secretly thinking of? | 00:50 |
| clarkb | jeblair: thanks, I will dig through that though probably not today and submit a big upstream if necessary | 00:50 |
| jeblair | fungi: heh, it's a pretty common way to run ruby, actually, esp. in 'enterprise' environments | 00:50 |
| clarkb | jeblair: I do see that no deadlocks were detected | 00:50 |
| jeblair | fungi: it's ruby running in the jvm | 00:50 |
| fungi | interesting | 00:50 |
| fungi | so, like, i already have this java application running in a jvm, and now i think it might be hipper if it grew a little ruby too? | 00:51 |
| jeblair | fungi: it's much easier to drop a ruby app in a tomcat container as a war than to figure out how to actually run a ruby server. :) | 00:51 |
| jhesketh | clarkb: okay, so just (re)reviewing your patch... I forget now, how would the override location get its refs? can gerrit push them to other locations? | 00:51 |
| fungi | jeblair: without being run out of town on rails. got it | 00:51 |
| clarkb | jhesketh: yup, you need to setup gerrit replication in your gerrit serve rto replicate to the other location | 00:52 |
| *** Mithrandir has quit IRC | 00:52 | |
| jeblair | clarkb: yeah, the stacktrace isn't looking terribly helpful since it's mostly just ruby interpreter stuff | 00:52 |
| jhesketh | okay | 00:53 |
| jeblair | clarkb: there's a little bit like we can see one thread is waiting on an epoll, but there's no context for what that thread might be doing | 00:53 |
| fungi | jhesketh: bonus points for spotting the race condition in that design | 00:53 |
| *** Mithrandir has joined #openstack-infra | 00:53 | |
| jogo | portante: thanks, not sure how that would break things though | 00:53 |
| jeblair | rather, several threads waiting on epoll | 00:53 |
| jhesketh | fungi: which now? | 00:53 |
| fungi | jhesketh: sometimes gerrit falls behind on replication | 00:54 |
| jogo | portante: its a red herring ahh | 00:54 |
| jogo | STDOUT: 2014-03-10 22:40:50.423 26911 DEBUG root [-] HTTP PERF: 0.00207 seconds to GET 127.0.0.1:6012 /sdb1/503/AUTH_64dd76f9ce8f4c15b2c81d3cc8ec3da3?format=json) getresponse /opt/stack/new/swift/swift/common/bufferedhttp.py:128 (txn: tx1219402b41d040dbbebc2-00531e3f72) | 00:54 |
| jeblair | clarkb: oh, wait some of those actually have elasticsearch methods.... | 00:54 |
| jhesketh | oh yeah, that's what the poll would be for | 00:54 |
| fungi | right, polling would help. i should look at the patch | 00:54 |
| jeblair | fungi: it's not as funny if you do that | 00:55 |
| jhesketh | it's already there :-) | 00:55 |
| fungi | 'tis what i figured after you said it ;) | 00:55 |
| *** wchrisj has joined #openstack-infra | 00:55 | |
| fungi | clarkb gets the bonus points in that case | 00:56 |
| jhesketh | indeed :-) | 00:56 |
| jeblair | clarkb: the two threads that are doing this look mildly interesting: | 00:57 |
| jeblair | - org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) | 00:57 |
| clarkb | jeblair: and there are a bunch of threads polling | 00:58 |
| clarkb | jeblair: this UTC is messing with me. I am going to head home now. I think you can safely `restart logstash-indexer` on those nodes whenever you are ready | 00:59 |
| jeblair | clarkb: ok | 01:00 |
| clarkb | jeblair: I have a hunch that the es client lib is buggy | 01:00 |
| clarkb | resulting in this bevhaior and hte behavior that needs a watchdog | 01:00 |
| jogo | clarkb: silly elasticSearch question | 01:01 |
| *** wenlock has quit IRC | 01:01 | |
| clarkb | jogo: sure | 01:01 |
| jogo | how do I search for the strong "503 " | 01:01 |
| clarkb | strong "503 "? | 01:01 |
| jogo | when I try the spaces seem to be stripped | 01:01 |
| jogo | search for the string* | 01:01 |
| clarkb | oh, you mean verbatim | 01:01 |
| clarkb | quoting it should work | 01:01 |
| clarkb | message:"503 " | 01:01 |
| *** SumitNaiksatam has joined #openstack-infra | 01:02 | |
| clarkb | since you have to unquote if you just want terms | 01:02 |
| jogo | message:"503 " AND filename:"logs/syslog.txt" AND syslog_program:"proxy-server" | 01:02 |
| jhesketh | FYI, turbo-hipsters mirror problems are solved by pointing to openstacks pypi | 01:02 |
| jogo | doesn'tw ork | 01:02 |
| jeblair | clarkb: maybe there was nothing wrong with the indexer | 01:03 |
| jeblair | clarkb: the workers are not running on 03 because the oom killer hit them | 01:03 |
| jeblair | clarkb: what's your current target workers/node? | 01:03 |
| fungi | jhesketh: awesome! now is the time when i make the obligatory statement about pypi.openstack.org being on a system shared with hosting a lot of other stuff, a single point of failure, an incomplete mirror, not behind a cdn like pypi.python.org not providing https with a server certificate pip can validate, and so on | 01:04 |
| jhesketh | yes, but it's significantly more useful than a mirror behind a broken CDN | 01:04 |
| fungi | jhesketh: no comment ;) | 01:05 |
| dstufft | Broken CDN? | 01:05 |
| clarkb | jeblair: oh gah. 2 is safe, 3 helps keep queue down | 01:05 |
| jeblair | clarkb: there's a flaw in that. :) | 01:05 |
| clarkb | jeblair we should probably just go with 2 for now | 01:05 |
| clarkb | jeblair yes I know | 01:06 |
| fungi | jhesketh: definitely provide dstufft with your symptoms so it does not stay broken | 01:06 |
| *** fcarpenter has joined #openstack-infra | 01:06 | |
| jeblair | ok | 01:06 |
| dstufft | is jhesketh mikal on twitter | 01:06 |
| dstufft | oh | 01:06 |
| dstufft | there's a person in here named that too | 01:06 |
| fungi | dstufft: they're able to throw crumpled wads of paper at one another, i believbe | 01:07 |
| jhesketh | dstufft: we work together but are not the same people | 01:07 |
| jeblair | which is actually how twitter works too | 01:07 |
| dstufft | I was talking to mikal on twitter but then got distracted by RL stuff :/ | 01:07 |
| fungi | jeblair: i thought twitter was limited to those little slips of paper they used in fortune cookies? | 01:07 |
| jhesketh | fungi: actually we'd have throw the paper a very long way as we live in different states | 01:07 |
| dstufft | jhesketh: so I'm like a PyPI administrator and python-infra person | 01:08 |
| dstufft | thing | 01:08 |
| dstufft | creature | 01:08 |
| jhesketh | ah cool | 01:08 |
| jhesketh | yeah mikal was poking at our pypi mirror | 01:08 |
| fungi | jhesketh: oh, you and mikal share edges of a large desert then | 01:08 |
| clarkb | jeblair can you check 11 12 and 16 too? | 01:08 |
| dstufft | jhesketh: can you give me full output of the blowy uppy | 01:08 |
| clarkb | I am leaving office now | 01:08 |
| jeblair | clarkb: i will check all of them | 01:08 |
| clarkb | danke | 01:08 |
| jhesketh | dstufft: I'll put something useful together | 01:09 |
| jeblair | clarkb: i will cause all of the hosts to be running 2 workers | 01:09 |
| dstufft | if this is using openstack's mirror script then it's even more useful because i'm also a pip developer and IIRC the mirror script uses pip :V | 01:09 |
| jogo | clarkb: any ideas? | 01:09 |
| jeblair | jogo: about what? | 01:09 |
| fungi | dstufft: the openstack pypi-mirror script totally uses pip | 01:09 |
| fungi | a ton | 01:09 |
| *** rfolco has quit IRC | 01:09 | |
| fungi | no idea if that's what jhesketh and mikal are using though | 01:10 |
| *** Daisy_ has joined #openstack-infra | 01:11 | |
| jogo | jeblair: how to get message:"503 " AND filename:"logs/syslog.txt" AND syslog_program:"proxy-server | 01:12 |
| jogo | to use those spaces around the 503 | 01:13 |
| dstufft | periodic reminder that openstack should get rid of it's external urls and then just use bandersnatch :D | 01:13 |
| fungi | dstufft: we have fewer and fewer every day | 01:13 |
| fungi | soon we'll be at the point of forking/reimplementing the remainder, i suspect | 01:14 |
| clarkb | bandersnatch doesnt solve wheels thoygh | 01:14 |
| fungi | does bandersnatch not mirror wheels? | 01:14 |
| fungi | or does it just not generate platform-specific wheels for you on the fly while it mirrors? | 01:15 |
| dstufft | it doesn't generate | 01:15 |
| *** bada_ has quit IRC | 01:15 | |
| fungi | maybe that's a feature we want to contribute to it when the time comes | 01:15 |
| *** emagana has joined #openstack-infra | 01:15 | |
| *** CaptTofu has joined #openstack-infra | 01:15 | |
| fungi | rather than reinventint the... | 01:16 |
| fungi | round things | 01:16 |
| *** dkliban has joined #openstack-infra | 01:19 | |
| jeblair | fungi, clarkb: i restarted all the workers but i don't think they're getting jobs | 01:19 |
| jeblair | maybe they aren't restarted... | 01:20 |
| jeblair | oh, i think they don't like to be restarted right after the indexer | 01:20 |
| jeblair | will do another pass | 01:21 |
| fungi | i don't think i've restarted the logstash workers before, bit i'll pull up these useful javascript-based utilities clarkb recommended and see whether they say anything helpful | 01:21 |
| clarkb | jeblair ya if tcp 9999 isnt open they get cranky | 01:21 |
| clarkb | and jruby does that slowly | 01:22 |
| *** wchrisj has quit IRC | 01:22 | |
| jeblair | okay gearman rerports 32 online | 01:22 |
| *** rpodolyaka has quit IRC | 01:23 | |
| *** blamar has quit IRC | 01:23 | |
| fungi | both bigdesk and elasticsearch-head show pictures which incorporate workers 1-16 | 01:23 |
| jeblair | yeah, they are all running | 01:24 |
| jeblair | i'm seeing a lot of really big libvirt files | 01:24 |
| jeblair | 938219 lines | 01:24 |
| jeblair | i think that's a big problem | 01:24 |
| jeblair | but i have to go to dinner now | 01:24 |
| fungi | go eat | 01:24 |
| fungi | i wonder whether something changed in our libvirt logging | 01:25 |
| clarkb | its possible | 01:25 |
| fungi | million-line logfiles? | 01:25 |
| clarkb | or nova made it cranky | 01:25 |
| fungi | sheesh | 01:25 |
| *** rpodolyaka has joined #openstack-infra | 01:25 | |
| *** wenlock has joined #openstack-infra | 01:28 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove false positives from query 1240256 https://review.openstack.org/79503 | 01:28 |
| *** thuc has joined #openstack-infra | 01:30 | |
| *** thuc_ has joined #openstack-infra | 01:30 | |
| *** thuc has quit IRC | 01:34 | |
| jogo | jeblair: there has bee na libvirt failure spike recently http://status.openstack.org/elastic-recheck/ | 01:37 |
| *** rfolco has joined #openstack-infra | 01:37 | |
| fungi | which could explain an uptick in libvirt log sizes, if they're full of failspam | 01:38 |
| jogo | fungi: so the version of libvirt appears to not have changed | 01:39 |
| jogo | I wonder if the kernel has | 01:39 |
| fungi | jogo: oh, maybe... rackspace supposedly redid their base images last week? | 01:39 |
| jogo | fungi: how do we confirm | 01:39 |
| *** nosnos has joined #openstack-infra | 01:40 | |
| jogo | kernel_version=3.2.0-60-virtual | 01:40 |
| *** alexpilotti has quit IRC | 01:40 | |
| fungi | Linux py3k-precise-hpcloud-az1-2518397 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | 01:40 |
| fungi | Linux py3k-precise-rax-dfw-2518514.slave.openstack.org 3.2.0-60-generic #91-Ubuntu SMP Wed Feb 19 03:54:44 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | 01:40 |
| fungi | well, wrong image variants, but the precise kernels we use on them should be consistent at each provider anyway | 01:41 |
| fungi | built remarkably close together | 01:41 |
| jogo | this may be nothing, the spike may just be a spike in development | 01:42 |
| fungi | or spacetime compression due to a passing black hole | 01:42 |
| jogo | oh wait | 01:43 |
| jogo | confirmed | 01:43 |
| clarkb | no libvirt thing is anamolous | 01:43 |
| jogo | unless kibana derped out on me | 01:43 |
| clarkb | ? | 01:43 |
| jogo | kernel_version=3.2.0-60-virtual | 01:43 |
| jogo | message:"kernel_version=3.2.0-60-virtual | 01:43 |
| jogo | " | 01:43 |
| jogo | either my kibana died or that started on 3/7 | 01:44 |
| clarkb | ah | 01:44 |
| jogo | right around the spike | 01:44 |
| jogo | my kibana did die | 01:44 |
| jogo | but still | 01:44 |
| jogo | can someone else confirm | 01:46 |
| *** emagana has quit IRC | 01:49 | |
| zns | Hi folks - may I get input on pep257 testing tools in requirements? https://review.openstack.org/#/c/79054/ | 01:49 |
| jogo | fungi: yup new kernel on 3/7 | 01:51 |
| jogo | or at least no no hits for message:"kernel_version=3.2.0-60-virtual" AND filename:"logs/devstacklog.txt" before then | 01:51 |
| *** rpodolyaka has quit IRC | 01:52 | |
| *** rpodolyaka has joined #openstack-infra | 01:52 | |
| *** dkliban has quit IRC | 01:52 | |
| jogo | ahh 3.2.0-59-virtual worked better | 01:52 |
| jogo | clarkb: ^ I think we found the issue | 01:53 |
| *** wchrisj has joined #openstack-infra | 01:53 | |
| *** mgagne has quit IRC | 01:53 | |
| fungi | zul will want to hear about it when he's awake (or maybe he'd rather not) | 01:53 |
| jogo | fungi: about to head home and AFK for a bit, I'll file a bug and can you follow up | 01:55 |
| *** thuc has joined #openstack-infra | 01:55 | |
| jogo | not even sure where to file the bug | 01:55 |
| *** thuc_ has quit IRC | 01:55 | |
| jogo | https://bugs.launchpad.net/openstack-ci/+bug/1290637 | 01:57 |
| jogo | fungi: ^ I didn't know where to file it | 01:57 |
| fungi | and i, similarly, have no idea what to do with it now that you have ;) | 01:58 |
| jogo | wendar: ^ any ideas | 01:58 |
| fungi | we'll find a home for it, after some shopping around, have no doubt | 01:58 |
| jogo | fungi: so this is really bad for nova | 01:58 |
| jogo | nova essentially has a massive bug we have to release with | 01:58 |
| jogo | unless dims can pull some magic | 01:59 |
| * jogo hopes dims is magical | 01:59 | |
| fungi | go go gadget 3.2.0-61? | 01:59 |
| jogo | lolz | 01:59 |
| jogo | fungi: in other news W screen-ceilometer-collector.txt.gz10-Mar-2014 W I'M | 02:02 |
| jogo | 9.7M WAT | 02:02 |
| fungi | ceilo-WAT | 02:02 |
| jogo | haha | 02:02 |
| fungi | 9.6m is complaining about too-old mongodb probably | 02:02 |
| jogo | /opt/stack/new/ceilometer/ceilometer/openstack/common/db/sqlalchemy/session.py:526: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 | 02:02 |
| jogo | yeah not cool | 02:02 |
| fungi | (unless you're running python 2.5) | 02:02 |
| *** antonym has quit IRC | 02:03 | |
| *** antonym has joined #openstack-infra | 02:03 | |
| *** khyati has quit IRC | 02:06 | |
| jogo | libvirtd.txt.gz11-Mar-2014 W I'M | 02:07 |
| jogo | besides the paste fail | 02:07 |
| jogo | 32M! | 02:07 |
| jogo | haha | 02:07 |
| jogo | damn | 02:08 |
| clarkb | compressed? | 02:08 |
| jogo | yup | 02:08 |
| * clarkb eats dinner and cries | 02:08 | |
| StevenK | Saving to: 'screen-ceilometer-collector.txt.gz’ | 02:08 |
| StevenK | [ <=> ] 118,647,609 491KB/s | 02:08 |
| jogo | http://logs.openstack.org/28/75728/5/check/check-tempest-dsvm-full/30d9cb5/logs/?C=S;O=D | 02:08 |
| jogo | StevenK: try libvirtd! | 02:08 |
| StevenK | I'd prefer Apache to send me the gz | 02:09 |
| StevenK | Rather than now 140MiB of text | 02:09 |
| *** Ryan_Lane has quit IRC | 02:10 | |
| *** Ryan_Lane has joined #openstack-infra | 02:10 | |
| *** rpodolyaka has quit IRC | 02:10 | |
| jogo | StevenK: does firefox work with those files? | 02:10 |
| clarkb | StevenK it will if you ask for it | 02:11 |
| clarkb | wget and curl dont by default | 02:11 |
| StevenK | Silly things | 02:11 |
| *** yamahata has quit IRC | 02:11 | |
| *** yamahata__ has quit IRC | 02:12 | |
| *** Ryan_Lane has quit IRC | 02:14 | |
| *** reed has quit IRC | 02:16 | |
| jogo | fungi: during this rename can we sort out the future of gantt? | 02:18 |
| jogo | if not what is blocking that | 02:18 |
| fungi | jogo: have we determined that there's something preventing nova cores who don't care about gantt to ignore/not subscribe to reviews for it? | 02:18 |
| fungi | put it on the agenda for tomorrow's infra meeting https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting and we can hash it out if it's urgent to get added | 02:19 |
| *** fcarpenter has quit IRC | 02:21 | |
| fungi | jogo: though the nova program probably should unfriend it officially first regardless... https://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n14 | 02:21 |
| fungi | that's a lot more official than some prefix on a url | 02:21 |
| lifeless | jogo: why you no like gantt :( | 02:22 |
| lifeless | jogo: (teasing, mostly) | 02:22 |
| *** rfolco has quit IRC | 02:24 | |
| *** xchu_ has joined #openstack-infra | 02:27 | |
| * fungi is sort of caught up on some of the things he promised people he would do today, so sort of disappearing for a while to go nap | 02:27 | |
| *** zns has quit IRC | 02:28 | |
| *** mgagne has joined #openstack-infra | 02:28 | |
| *** dstufft|laptop has joined #openstack-infra | 02:29 | |
| lifeless | fungi: https://review.openstack.org/#/c/76588/ - any chance of a review of this, and a release perhaps ? | 02:29 |
| lifeless | we're hit by it every week or so and have to restart geard... | 02:29 |
| *** rpodolyaka has joined #openstack-infra | 02:30 | |
| *** zhiwei has joined #openstack-infra | 02:31 | |
| *** derekh has quit IRC | 02:33 | |
| *** thuc has quit IRC | 02:33 | |
| *** thuc has joined #openstack-infra | 02:33 | |
| *** zhiwei has quit IRC | 02:36 | |
| *** thuc has quit IRC | 02:38 | |
| *** mgagne has quit IRC | 02:38 | |
| *** zhiwei has joined #openstack-infra | 02:38 | |
| *** zhiyan_ is now known as zhiyan | 02:51 | |
| *** malini is now known as malini_afk | 02:52 | |
| *** mgagne has joined #openstack-infra | 02:52 | |
| *** harlowja is now known as harlowja_away | 02:54 | |
| *** Sukhdev has joined #openstack-infra | 02:57 | |
| *** mgagne has quit IRC | 02:58 | |
| *** rpodolyaka has quit IRC | 02:58 | |
| *** dkliban has joined #openstack-infra | 03:04 | |
| *** harlowja_away is now known as harlowja | 03:06 | |
| *** vkozhukalov has quit IRC | 03:09 | |
| *** fifieldt has joined #openstack-infra | 03:09 | |
| *** dkliban has quit IRC | 03:13 | |
| *** CaptTofu has quit IRC | 03:14 | |
| *** khyati has joined #openstack-infra | 03:16 | |
| *** dkliban has joined #openstack-infra | 03:16 | |
| *** krotscheck has quit IRC | 03:19 | |
| *** luisg has quit IRC | 03:22 | |
| *** dcramer__ has joined #openstack-infra | 03:22 | |
| *** luis_ has joined #openstack-infra | 03:22 | |
| *** zhiwei has quit IRC | 03:22 | |
| *** gokrokve has quit IRC | 03:22 | |
| *** zhiwei has joined #openstack-infra | 03:22 | |
| *** dripton has quit IRC | 03:23 | |
| *** dripton has joined #openstack-infra | 03:24 | |
| harlowja | interesting, has anyone seen jenkins do 'sudo: 3 incorrect password attempts ERROR: gate_hook failed' , this then causes the job to fail | 03:25 |
| * harlowja not trying to guess the password (i swear) | 03:26 | |
| harlowja | lol | 03:26 |
| StevenK | harlowja: Haha | 03:26 |
| StevenK | harlowja: That seems a little wrong for a jenkins job | 03:26 |
| harlowja | https://jenkins07.openstack.org/job/check-tempest-dsvm-full/3399/consoleText | 03:26 |
| harlowja | def seems odd, maybe already open bug | 03:26 |
| *** caleb_ has joined #openstack-infra | 03:27 | |
| StevenK | I would have expected passwordless sudo, TBH | 03:27 |
| harlowja | ya, pretty weird | 03:27 |
| *** gokrokve has joined #openstack-infra | 03:27 | |
| harlowja | maybe someone was changing the password or something recently | 03:28 |
| *** lttrl has joined #openstack-infra | 03:29 | |
| harlowja | anyways, https://bugs.launchpad.net/openstack-ci/+bug/1290669 i guess | 03:30 |
| harlowja | my guess, mr.jenkins forgot the password | 03:31 |
| harlowja | *mr jenkins is pretty old afterall | 03:32 |
| caleb_ | Does our instance of Gerrit expose the REST API? I get a 404 at https://review.openstack.org/projects. | 03:35 |
| *** Daisy_ has quit IRC | 03:35 | |
| harlowja | caleb_ not afaik | 03:35 |
| harlowja | caleb_ although u can do it via ssh, https://github.com/gd-imelnikov/gerda/blob/master/bin/gerda#L193 (is an example that lists projects) | 03:35 |
| pleia2 | caleb_: we're still running 2.4, which doesn't have it (upgrade to 2.8 is in the works) | 03:35 |
| *** enikanorov_ has joined #openstack-infra | 03:36 | |
| caleb_ | harlowja, pleia2: Thanks. | 03:37 |
| caleb_ | pleia2: won't quote you later but any idea on when the 2.8 upgrade will land? | 03:37 |
| clarkb | harlowja: thats an interesting fail. there are a bunch of sudos prior to that one that succeed | 03:37 |
| clarkb | harlowja: not sure how that happened | 03:37 |
| *** enikanorov has quit IRC | 03:38 | |
| harlowja | clarkb mr.jenkins fogot i think | 03:38 |
| harlowja | *forgot | 03:38 |
| clarkb | harlowja: http://logs.openstack.org/14/70914/6/check/check-tempest-dsvm-full/56f2985/logs/devstack-gate-setup-host.txt.gz no I think it may be related to the users being su'd to | 03:39 |
| clarkb | tempest and stack both want a tty for some reason but root is happy without it | 03:39 |
| clarkb | which must be another image thing | 03:39 |
| pleia2 | caleb_: we're pretty close, zaro would probably know more estimate-wise | 03:40 |
| harlowja | clarkb there is a previous command for 'sudo useradd -U -s /bin/bash -m tempest' | 03:40 |
| *** matsuhashi has quit IRC | 03:41 | |
| harlowja | and corresponding tempest ALL=(root) stuffs | 03:42 |
| clarkb | harlowja: those are rules for tempest to sudo | 03:42 |
| clarkb | not to sudo to the tempest user | 03:42 |
| *** thuc has joined #openstack-infra | 03:44 | |
| *** thuc has quit IRC | 03:48 | |
| harlowja | clarkb hmmm | 03:49 |
| harlowja | wonder why its doing that then | 03:49 |
| *** amcrn has quit IRC | 03:50 | |
| clarkb | harlowja: best guess is we have require tty in our sudoers file now for not root | 03:51 |
| * clarkb logs into some machines | 03:52 | |
| jogo | so the gate seems to moving pretty slow these days | 03:53 |
| jogo | https://github.com/openstack/openstack/graphs/commit-activity | 03:53 |
| harlowja | mr.jenkins take a nap jogo | 03:53 |
| harlowja | lol | 03:53 |
| harlowja | *taking | 03:53 |
| jogo | and 9 patches in post-gate | 03:53 |
| jogo | ahh good ol' resource starvation | 03:54 |
| zaro | caleb_: patches for gerrit 2.8 upgarde are up on gerrit. just need approvals. | 03:54 |
| jogo | I wonder what happens if we drop the recheck rate | 03:54 |
| jogo | sdague: ^ | 03:54 |
| jogo | we just don't have enough capacity it seems like | 03:55 |
| clarkb | jogo: tests are failing | 03:55 |
| clarkb | because sudo | 03:56 |
| jogo | clarkb: ? | 03:56 |
| harlowja | jogo https://bugs.launchpad.net/openstack-ci/+bug/1290669 | 03:56 |
| clarkb | we have an hour old dfw image I bet its related to that | 03:56 |
| jogo | clarkb: ahh well we have been this way fora few days I think | 03:57 |
| jogo | err thats a lie | 03:57 |
| jogo | there was the weekend | 03:57 |
| clarkb | we merged the second most commits ever last week iirc | 03:57 |
| jogo | true but things were slowish | 03:57 |
| jogo | so things aren't so bad we don't clear out the queue | 03:57 |
| jogo | so delay shouldn't hurt the flow | 03:58 |
| clarkb | ? | 03:58 |
| jogo | clarkb: long as gate empties out every night we don't get a buildup | 03:58 |
| clarkb | right but if every test fails that doesn't happen | 03:59 |
| jogo | clarkb: also http://jogo.github.io/gate/' | 03:59 |
| jogo | jobs be less stable | 03:59 |
| jogo | which I think is the damn kernel | 03:59 |
| clarkb | no its the sudo thing... | 04:00 |
| *** markmcclain has quit IRC | 04:00 | |
| *** pcrews has quit IRC | 04:00 | |
| jogo | clarkb: that is the most recent part of the spike | 04:00 |
| jogo | see http://status.openstack.org/elastic-recheck/ | 04:00 |
| clarkb | I mean libvirt mgiht be related | 04:00 |
| jogo | libvirt march 7yh | 04:00 |
| jogo | 7th | 04:00 |
| jogo | clarkb: oh so anyway logstash | 04:00 |
| jogo | halp | 04:00 |
| clarkb | jogo: what do you want me to do? | 04:01 |
| jogo | that line keeps rising | 04:01 |
| jogo | clarkb: a miracle? | 04:01 |
| clarkb | make nova stop hurting libvirt | 04:01 |
| clarkb | or stop indexing libvirt. those are your options | 04:01 |
| jogo | clarkb: so that rise is newer then the kernel bump | 04:01 |
| *** jepoy has joined #openstack-infra | 04:01 | |
| * jogo goes through git logs | 04:01 | |
| clarkb | I don't think kernel bump is to blame honestly | 04:01 |
| clarkb | its more likely a nova change | 04:01 |
| clarkb | or some combination of the two | 04:01 |
| clarkb | ubuntu does a pretty good job with their lts kernels | 04:02 |
| clarkb | and that is an lts kernel too so upstream halp | 04:02 |
| jogo | added logs maybe a combination | 04:02 |
| jogo | rise in bug 1254872 lines up with kernel change exactly | 04:02 |
| jogo | clarkb: are we 100 sure the issue is the libvirt logs? | 04:05 |
| clarkb | jogo: yes its several orders of mangitude more data we must filter and index | 04:05 |
| clarkb | I mean 99.99% sure | 04:05 |
| clarkb | jogo: when I looked at the es systems they were indexing at record rates | 04:05 |
| clarkb | becuase we were shoving a ton more data through it | 04:05 |
| clarkb | basically the index pipe is only so big | 04:05 |
| clarkb | when you suddenly go from tens of thousands of log lines per job to million range stuff grinds to a halt | 04:05 |
| jogo | so timeline loks like this started last night? | 04:05 |
| jogo | oh wait | 04:05 |
| jogo | it ahppend this afternoon | 04:05 |
| jogo | https://review.openstack.org/#/c/79346/ | 04:08 |
| jogo | I wonder ... | 04:08 |
| *** david-lyle has joined #openstack-infra | 04:08 | |
| clarkb | jogo: that wouldn't surprise me if it is the cause | 04:10 |
| jogo | yeah | 04:10 |
| *** CaptTofu has joined #openstack-infra | 04:10 | |
| *** yamahata__ has joined #openstack-infra | 04:11 | |
| jogo | clarkb: in that case my vote is pull from ES | 04:11 |
| clarkb | wfm | 04:11 |
| clarkb | just delete the lines in the client yaml config | 04:11 |
| jogo | I can do the patch so you can review | 04:11 |
| clarkb | but first I need to finish debugging this sudo problem | 04:11 |
| clarkb | because nothing is happening until that gets fixed | 04:11 |
| *** yamahata has joined #openstack-infra | 04:12 | |
| clarkb | I think jeblair's sudo change broke it | 04:12 |
| clarkb | https://review.openstack.org/#/c/72777/ is the change | 04:12 |
| *** pcrews has joined #openstack-infra | 04:13 | |
| *** markmcclain has joined #openstack-infra | 04:14 | |
| clarkb | https://review.openstack.org/#/c/72777/2/modules/jenkins/files/jenkins-sudo.sudo I don't see anything wrong with that, jenkins user can run ALL commands as root with no password | 04:14 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Revert "Add logstash rules to parse libvirtd.txt" https://review.openstack.org/79535 | 04:14 |
| clarkb | where does requiretty come into affect | 04:14 |
| *** krotscheck has joined #openstack-infra | 04:15 | |
| jogo | clarkb: ^ | 04:15 |
| *** nati_ueno has joined #openstack-infra | 04:15 | |
| *** CaptTofu has quit IRC | 04:15 | |
| clarkb | jogo: honestly a 30MB libvirt log file compressed doesn't really help anyone | 04:15 |
| clarkb | jogo: I am good with that change,but maybe we should consider not logging so verbosely | 04:15 |
| jogo | clarkb: we are debugging libvirt failing | 04:16 |
| jogo | it helps libvirt developers and we have one | 04:16 |
| jogo | so its for him | 04:16 |
| clarkb | but he doesn't need it 15k times a day | 04:16 |
| clarkb | flip that switch when you need it | 04:16 |
| jogo | disable client side libvirt debug logging | 04:16 |
| jogo | 04:16 | |
| jogo | and tune server side libvirt logging to the values that danpb | 04:16 |
| jogo | suggested would be useful on the openstack-dev mailing list. | 04:16 |
| jogo | clarkb: I want to give danpb all the data he needs, at least for a day or two | 04:17 |
| jogo | this really should be an experimental job | 04:17 |
| clarkb | but its not useable data | 04:18 |
| clarkb | you need hadepp | 04:18 |
| *** pcrews has quit IRC | 04:18 | |
| clarkb | *hadoop | 04:18 |
| jogo | clarkb: perhaps but I am not comfortable making that judgement call without danpb ... I think stop indexing is the quickest solution, unless we want to add an experimental job with this instead | 04:20 |
| jogo | clarkb: also unless dtroyer is up this is easier to merge at this hour | 04:20 |
| clarkb | I don't plan on merging ^ tonight | 04:21 |
| clarkb | its late and the sudo thing is more important | 04:21 |
| *** sweston has quit IRC | 04:21 | |
| jogo | ahh in that case | 04:23 |
| *** Daisy_ has joined #openstack-infra | 04:23 | |
| *** talluri has joined #openstack-infra | 04:24 | |
| *** wchrisj has quit IRC | 04:24 | |
| jogo | so the sudo issue only pops up when you try becoem a different user bsides root | 04:25 |
| jogo | 2014-03-11 03:18:08.207 | + sudo -u stack mkdir -p /opt/stack/new/.pip | 04:25 |
| clarkb | I have held 2551489 | 04:26 |
| clarkb | jogo: yes, jenkins sudo fails | 04:26 |
| clarkb | oh duh I see it now | 04:26 |
| clarkb | woo | 04:26 |
| jogo | from http://logs.openstack.org/14/70914/6/check/check-tempest-dsvm-full/56f2985/logs/devstack-gate-setup-host.txt.gz | 04:27 |
| *** sdake has quit IRC | 04:27 | |
| clarkb | ya I should have a fix proposed shortly | 04:27 |
| clarkb | gah the node I held doesn't have the derpage yet | 04:28 |
| SlickNik | I came here to report sudo issues with tempest tests, but you guys are already on it. :) | 04:30 |
| SlickNik | Much thanks! | 04:30 |
| morganfainberg | clarkb, ++ woot and infra is on it already! | 04:30 |
| jogo | clarkb: so I can't write a ER query for it yet, becuase the long aren't in ;) | 04:31 |
| clarkb | jogo: you don't need one | 04:32 |
| jogo | for http://status.openstack.org/elastic-recheck/data/uncategorized.html | 04:33 |
| jogo | anyway I don't want to distract you from fixing it | 04:33 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Allow jenkins to sudo as any user. https://review.openstack.org/79536 | 04:34 |
| clarkb | mordred: jeblair fungi SergeyLukjanov ^ so that is a bit important | 04:34 |
| clarkb | I don't see any immediate issues with allowing it to sudo as any user | 04:34 |
| mordred | clarkb: looking | 04:34 |
| clarkb | since the one user we intended on allowing before was root | 04:34 |
| mordred | clarkb: ok | 04:35 |
| clarkb | mordred: do you think we should just push that through and trigger some image rebuilds? | 04:37 |
| clarkb | I have a hunch fungi and jeblair aren't going to be around soon | 04:37 |
| clarkb | SergeyLukjanov might but he doesn't sleep | 04:37 |
| clarkb | oh darn I didn't put the bug in the commit message /me fixes | 04:38 |
| mordred | clarkb: yah | 04:38 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Allow jenkins to sudo as any user. https://review.openstack.org/79536 | 04:39 |
| clarkb | I will give it long enough to check test then approve | 04:39 |
| mordred | clarkb: I support you +Aing it in | 04:39 |
| *** matsuhashi has joined #openstack-infra | 04:40 | |
| *** sarob_ has joined #openstack-infra | 04:42 | |
| *** esker has joined #openstack-infra | 04:43 | |
| *** zehicle_at_dell has joined #openstack-infra | 04:45 | |
| *** zns has joined #openstack-infra | 04:45 | |
| *** sweston has joined #openstack-infra | 04:46 | |
| *** sarob_ has quit IRC | 04:46 | |
| *** lttrl has quit IRC | 04:47 | |
| *** esker has quit IRC | 04:47 | |
| clarkb | gah the gate is eating all the nodes | 04:52 |
| clarkb | mordred: should I god mode submit it? | 04:52 |
| *** krotscheck has quit IRC | 04:52 | |
| clarkb | I really hate doing that but given the situation... | 04:52 |
| mordred | clarkb: yes. you want me to? | 04:52 |
| clarkb | mordred: sure | 04:53 |
| clarkb | then I can trigger a bunch of nodepool image udpates | 04:53 |
| *** wchrisj has joined #openstack-infra | 04:54 | |
| *** Sukhdev has quit IRC | 04:57 | |
| *** esker has joined #openstack-infra | 04:57 | |
| *** krotscheck has joined #openstack-infra | 04:57 | |
| clarkb | mordred: were you going to do it? | 04:57 |
| mordred | clarkb: yah | 04:58 |
| clarkb | ok it hasn't merged yet so was wondering | 04:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow jenkins to sudo as any user. https://review.openstack.org/79536 | 04:59 |
| mordred | clarkb: done | 04:59 |
| clarkb | danke | 04:59 |
| *** arosen-home has joined #openstack-infra | 05:01 | |
| arosen-home | Looks like there was some interesting CI event within the hour :/ | 05:01 |
| arosen-home | Guess you guys already know all about it though. | 05:02 |
| StevenK | arosen-home: The fix was merged 4 minutes ago | 05:03 |
| StevenK | Now we have to wait for image builds | 05:03 |
| *** wchrisj has quit IRC | 05:04 | |
| *** oda-g_ has joined #openstack-infra | 05:05 | |
| *** jepoy_ has joined #openstack-infra | 05:05 | |
| StevenK | clarkb: What's the rough ETA for less explosions? | 05:06 |
| clarkb | StevenK: huh half hour ish for new images then nodes will need to cycle out | 05:06 |
| clarkb | if images come up qucik enough I can manually delete things | 05:06 |
| clarkb | but its getting late here and i had a really early morning | 05:06 |
| mordred | clarkb had to do an HP thing for me | 05:07 |
| * mordred hangs head | 05:07 | |
| clarkb | mordred: it wouldn't have been so bad if DST didn't happen | 05:07 |
| *** jepoy has quit IRC | 05:08 | |
| *** caleb_ has quit IRC | 05:09 | |
| oda-g_ | It seems the gate become very bad. All check and gate jobs are going to fail. | 05:12 |
| clarkb | oda-g_: yes see scrollback, basically a bad sudo rule for jenkins merged. fix is inflight | 05:13 |
| jeblair | clarkb: need help? | 05:13 |
| *** gokrokve has quit IRC | 05:13 | |
| oda-g_ | clarkb: Thanks for your infomation. | 05:14 |
| *** caleb_ has joined #openstack-infra | 05:14 | |
| *** oda-g_ has quit IRC | 05:14 | |
| clarkb | jeblair: I think I have it udner control. mordred pushed the fix through and I have image updates going | 05:15 |
| clarkb | jeblair: only devstack-* nodes seem to be affected so focusing on those | 05:15 |
| jeblair | clarkb: what's the deal with logstash? | 05:15 |
| clarkb | jeblair: sdague made a change to devstack that made libvirt logs blow up | 05:15 |
| clarkb | jeblair: I told jogo either we stop indexing libvirt or fix the logs being massive | 05:16 |
| jeblair | clarkb: we need to make them not be massive because they also take up space on logs.o.o | 05:16 |
| mordred | jeblair: ++ | 05:16 |
| clarkb | jeblair: image updates are running in screen on nodepool.o.o now | 05:16 |
| clarkb | jeblair: yup | 05:16 |
| jeblair | clarkb: or we need to not copy them at all | 05:16 |
| clarkb | this vmware image makes me cranky | 05:17 |
| jeblair | clarkb: have a link to the devstack change? | 05:17 |
| clarkb | mordred: we should kill that with fire imo | 05:17 |
| clarkb | jeblair: ya one sec | 05:17 |
| jeblair | clarkb: no kidding | 05:17 |
| clarkb | jeblair: https://review.openstack.org/#/c/79346/ | 05:17 |
| mordred | yeah. I want to kill the vmware image | 05:17 |
| mordred | what about a filter for *.vmdk? we're never going to consume those in the gate itself | 05:18 |
| clarkb | wfm | 05:18 |
| jeblair | mordred: that would work if you don't want to make the argument it shouldn't be in devstack anyway | 05:18 |
| clarkb | I can make that argument :) | 05:18 |
| mordred | jeblair: well, I do - but I have +2 on d-g | 05:18 |
| mordred | s/d-g/config/ | 05:19 |
| clarkb | rax-iad was cranky I finally have it building the second image now | 05:19 |
| clarkb | all devstack-precise* images are building across all azs now | 05:19 |
| jeblair | that logic makes me sad | 05:19 |
| clarkb | jeblair: did you see the sudo fix? | 05:20 |
| jeblair | clarkb: do you recall the procedure for "we have a broken image in nodepool?" | 05:20 |
| clarkb | jeblair: ? delete the old images? | 05:20 |
| jeblair | clarkb: delete the broken image | 05:21 |
| clarkb | oh right because it will fall back gah | 05:21 |
| jeblair | yeah. pretty much an immediate fix to such a problem | 05:21 |
| jeblair | clarkb: i think we should still do that, and soon. | 05:21 |
| clarkb | jeblair: ok, I will do that now | 05:21 |
| *** caleb_ has quit IRC | 05:21 | |
| *** nicedice has quit IRC | 05:21 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Skip downloading giant VMware images https://review.openstack.org/79537 | 05:22 |
| jeblair | clarkb: because as soon as your new images build, nodepool will think it's okay to delete the good ones (because they would be N-2) | 05:22 |
| SergeyLukjanov | clarkb, /me agreed with https://review.openstack.org/#/c/79536/ | 05:22 |
| mordred | SergeyLukjanov: are you STILL awake? or waking up? | 05:22 |
| clarkb | jeblair: yup | 05:22 |
| SergeyLukjanov | mordred, already wake up :) | 05:22 |
| mordred | wow | 05:22 |
| * mordred hands SergeyLukjanov a sandwich | 05:23 | |
| SergeyLukjanov | mordred, and coffee pls ;) | 05:23 |
| SergeyLukjanov | mordred, /me need to order kitchen furniture today heh | 05:24 |
| *** esker has quit IRC | 05:24 | |
| jeblair | mordred: i think we should see if the rest of the devstack core would be okay with removing that from devstack first, then go with your approach if it fails | 05:24 |
| clarkb | jeblair: done, all images are either more than a day old or less than an hour old | 05:24 |
| jeblair | clarkb: cool | 05:25 |
| clarkb | for devstack-precise* | 05:25 |
| SergeyLukjanov | clarkb, yay! | 05:25 |
| clarkb | the other images don't need sudo so should be fine and i am not rebuilding there either | 05:25 |
| SergeyLukjanov | jeblair, IMO the more safe option is just have a list of needed images in d-g | 05:25 |
| jeblair | SergeyLukjanov: I don't want to reimplement devstack in devstack-gate; it should be minimal, and devstack-gate should run it as close to the default as possible | 05:26 |
| jeblair | SergeyLukjanov: part of the reason for that is the intention is to test what developers and users actually get when they run devstack | 05:26 |
| jeblair | SergeyLukjanov: basically, aside from saying things like "i want to run (or don't want to run) sahara" devstack-gate shouldn't do much configuration of devstack | 05:28 |
| mordred | jeblair: works for me | 05:28 |
| mordred | jeblair: should I submit a patch to devstack to kickstart the conversation? | 05:29 |
| SergeyLukjanov | My point is that devstack could install more images than we need for d-g | 05:29 |
| jeblair | mordred: sure; i added dtroyer to the config review too so he'll see that background. and maybe this ping. :) | 05:29 |
| *** sweston has quit IRC | 05:30 | |
| jeblair | SergeyLukjanov: perhaps, and that's okay -- but the question is whether every possible image needs to be in devstack | 05:30 |
| SergeyLukjanov | For ex. In savanna we already have more than 10 useful images | 05:30 |
| SergeyLukjanov | Yup, agreed | 05:30 |
| SergeyLukjanov | Bah, sahara | 05:31 |
| jeblair | SergeyLukjanov: i want _at least_ every image we might possibly use in the gate to be in devstack, because that's how they get cached on our test nodes | 05:31 |
| *** dkliban has quit IRC | 05:31 | |
| mordred | jeblair: main objection I could see is that vmware has stepped up to the plate with third party testing - and it's possible they're also using devstack to do that | 05:32 |
| jeblair | SergeyLukjanov: and of course, since they are in the gate, that means they are basic configurations that developers should be able to test easily | 05:32 |
| SergeyLukjanov | And dib built images too? | 05:32 |
| jeblair | mordred: yeah, but maybe there's an easy way for them to get that information into devstack without it actually being in the tree | 05:32 |
| jeblair | mordred: it's just that i don't see most developers using vmware images | 05:32 |
| jeblair | mordred: because it's proprietary software, and we're writing free software | 05:33 |
| jeblair | among other reasons | 05:33 |
| mordred | jeblair: I say this mainly because of following the fact that there are vsphere logic branches | 05:33 |
| mordred | jeblair: I mean, I agree with you ... | 05:33 |
| mordred | just bringing up the possible pushback | 05:33 |
| jeblair | mordred: i'm aware of possible pushback; that's why i'm not saying no to your change, just that we ought to take a look at the idea. | 05:34 |
| jeblair | mordred: if the current state of the art is the best way for developers to test the vsphere nova driver, that's fine | 05:35 |
| *** dkliban has joined #openstack-infra | 05:35 | |
| jeblair | we can do the cat-and-mouse thing | 05:35 |
| mordred | https://review.openstack.org/79539 | 05:35 |
| mordred | it's also REALLY easy for the vsphere folks to drop an IMAGE_URL into their localrc | 05:36 |
| *** zehicle_at_dell has quit IRC | 05:36 | |
| *** kashyap has left #openstack-infra | 05:38 | |
| *** kashyap has joined #openstack-infra | 05:38 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Temporarily stop copying libvirt log https://review.openstack.org/79540 | 05:42 |
| jeblair | clarkb: ^ | 05:42 |
| clarkb | jeblair: lgtm | 05:42 |
| *** sarob_ has joined #openstack-infra | 05:43 | |
| jeblair | mordred, SergeyLukjanov: ^ want to aprv that? | 05:43 |
| jeblair | i approved it | 05:48 |
| *** sarob_ has quit IRC | 05:49 | |
| *** yjiang51 has joined #openstack-infra | 05:49 | |
| *** yjiang51 has left #openstack-infra | 05:49 | |
| jeblair | there are only 3 jenkins log workers | 05:50 |
| jeblair | one on 13, 2 on 16 | 05:50 |
| clarkb | wow | 05:50 |
| clarkb | jeblair: I wonder if they are ooming due to the size of that file | 05:51 |
| clarkb | it does load them into memory to unzip | 05:51 |
| StevenK | clarkb: Are the image builds all done? | 05:51 |
| clarkb | StevenK: no, but I killed broken images so things should be happier nowish | 05:51 |
| clarkb | jeblair: it could be better about that, unzipping incrementally and writing to disk, but this has never been a problem before | 05:52 |
| *** amcrn has joined #openstack-infra | 05:52 | |
| *** sweston has joined #openstack-infra | 05:53 | |
| *** yaguang has joined #openstack-infra | 05:53 | |
| StevenK | clarkb: It was a fairly handy way to clear out gate and check :-P | 05:53 |
| jeblair | clarkb: it's streaming; it should just do it 1 line at a time.. | 05:53 |
| *** nati_ueno has quit IRC | 05:54 | |
| jeblair | clarkb: (i don't mean to say i think that's what it is doing, but rather i think what it probably should do if it isn't) | 05:54 |
| StevenK | Haha, so much stuff has failed that Jenkins thinks that check-tempest-dsvm-postgres-full now takes 15 minutes. | 05:54 |
| *** rpodolyaka has joined #openstack-infra | 05:54 | |
| jeblair | clarkb: just spot checking 12: | 05:54 |
| jeblair | 2014-03-11 01:25:02,021 Retrieving: http://logs.openstack.org/55/74955/2/check/check-tempest-dsvm-full/48e7357/logs/libvirtd.txt.gz?level=INFO | 05:54 |
| jeblair | 2014-03-11 01:40:10,452 Decompressing gzipped source file. | 05:54 |
| clarkb | jeblair: yeah it could do that too, maybe not one line at a time since gzipped but some chink at a time | 05:55 |
| clarkb | *chunk | 05:55 |
| jeblair | that's the tail of one of the worker logs | 05:55 |
| *** nati_ueno has joined #openstack-infra | 05:55 | |
| jeblair | i don't see the oom killer being activated though | 05:55 |
| jeblair | clarkb: so i think we lost information on why that might have died | 05:55 |
| clarkb | jeblair: I would've expected tracebacks if not oomkiller, but I would agree with your statement given the log | 05:56 |
| jeblair | yeah me too | 05:56 |
| StevenK | Rargh, nodes haven't cycled out | 05:57 |
| jeblair | clarkb: oh wait, it did oom, i just missed the log entries | 05:57 |
| jeblair | clarkb: yeah, both workers on 12 hit the oom killer | 05:58 |
| *** CaptTofu has joined #openstack-infra | 05:58 | |
| *** ildikov_ has quit IRC | 05:59 | |
| jeblair | StevenK: jobs started after 05:30 should be ok | 05:59 |
| jeblair | and yet that's clearly not the case since https://jenkins04.openstack.org/job/check-tempest-dsvm-full/6131/ failed | 06:00 |
| *** krotscheck has quit IRC | 06:00 | |
| clarkb | jeblair: I think there were a few nodes out there on the old image | 06:00 |
| SergeyLukjanov | jeblair, https://review.openstack.org/79540 lgtm | 06:00 |
| StevenK | jeblair: I retried my nova patch about 6 minutes ago | 06:01 |
| jeblair | clarkb: there must have been a few building; the one i just looked at was building for 43 minutes | 06:01 |
| jeblair | clarkb: i forgot rax can take so long | 06:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Temporarily stop copying libvirt log https://review.openstack.org/79540 | 06:02 |
| *** CaptTofu has quit IRC | 06:02 | |
| StevenK | https://jenkins05.openstack.org/job/check-tempest-dsvm-neutron-heat-slow/1288/ for example | 06:03 |
| jeblair | (restarting tests ^) | 06:03 |
| clarkb | I should probably rewrite the log workers to use co routines and deal with a line at a time (ish) | 06:03 |
| jeblair | StevenK: yeah, it might take a bit longer than i thought due to some nodes taking a long time to build | 06:04 |
| clarkb | I need to crash shortly | 06:04 |
| clarkb | was a long day and I find myself unable to think properly | 06:04 |
| *** matsuhashi has quit IRC | 06:05 | |
| jeblair | clarkb: me too | 06:06 |
| jeblair | i think we need to wait until after https://review.openstack.org/79540 merges, then dump the logstash queue and restart the workers | 06:06 |
| clarkb | ++ | 06:07 |
| jeblair | clarkb: i mean, someone does, not necessarily us. :) | 06:07 |
| clarkb | ya I am going to go read red mars now so that the brain can shutdown proeprly | 06:07 |
| clarkb | I am running windows 95 | 06:07 |
| jeblair | clarkb: i don't understand what you are saying except i'm pretty sure it all just means it's your bedtime. good night. :) | 06:08 |
| clarkb | jeblair: you don't remember windows 95 needing to be shut down properly? | 06:08 |
| SergeyLukjanov | clarkb, good night :) | 06:08 |
| *** matsuhas_ has joined #openstack-infra | 06:08 | |
| clarkb | it would be quite cranky if you dropped power out from under it | 06:08 |
| SergeyLukjanov | clarkb, I remember | 06:08 |
| *** harlowja is now known as harlowja_away | 06:08 | |
| clarkb | "It is now safe to shutdown the computer" or some such | 06:09 |
| jeblair | clarkb: no, actually, but i remember that ext2 behaved similarly. :) | 06:09 |
| SergeyLukjanov | clarkb, I was about 8 years old and it was painful to not forget to do it | 06:09 |
| jeblair | fungi, mordred: (probably fungi unless mordred stays up late). Can you please do the following after https://review.openstack.org/#/c/79540/ merges? http://paste.openstack.org/show/73108/ | 06:14 |
| jeblair | (that's about 1.5 from now probably) | 06:15 |
| mordred | jeblair: I'll do it if I'm up | 06:17 |
| SergeyLukjanov | jeblair, if you have instructions than I can do such stuff when everyone else sleeping | 06:24 |
| jeblair | SergeyLukjanov: unfortunately it's a root-access task. we should work toward getting you root access. :) | 06:24 |
| SergeyLukjanov | jeblair, :) | 06:25 |
| SergeyLukjanov | heh, my phone is updated to iOS 7.1 | 06:25 |
| SergeyLukjanov | and now call screen contains only circles... wat? | 06:26 |
| openstackgerrit | Zhiwei Chen proposed a change to openstack-infra/config: Rename a StackForge project https://review.openstack.org/64788 | 06:26 |
| *** vkozhukalov has joined #openstack-infra | 06:27 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Rename savanna group to sahara https://review.openstack.org/79543 | 06:31 |
| *** dstufft|laptop has quit IRC | 06:32 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove logging of sahara projects to #savanna https://review.openstack.org/79545 | 06:32 |
| jeblair | some of the new images that clarkb kicked off seem to be in use now and also seem to be working | 06:33 |
| *** nati_ueno has quit IRC | 06:34 | |
| SergeyLukjanov | jeblair, that's awesome | 06:34 |
| *** skraynev has quit IRC | 06:34 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Document bad-image procedure for nodepool https://review.openstack.org/79546 | 06:34 |
| SergeyLukjanov | jeblair, oh, that's much easier than I thought | 06:37 |
| SergeyLukjanov | jeblair, re 79546 | 06:37 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove savanna from logstash/jenkins-log-client.yaml https://review.openstack.org/79548 | 06:38 |
| jeblair | SergeyLukjanov: yeah, we keep forgetting that it's that easy. but nodepool is designed for this; hopefully if we all review those docs we'll remember. :) | 06:38 |
| *** afazekas has quit IRC | 06:39 | |
| *** denis_makogon has joined #openstack-infra | 06:39 | |
| *** fifieldt has quit IRC | 06:41 | |
| *** wenlock has quit IRC | 06:43 | |
| *** gyee has quit IRC | 06:43 | |
| *** arosen-home has quit IRC | 06:44 | |
| *** skraynev has joined #openstack-infra | 06:48 | |
| *** khyati has quit IRC | 06:51 | |
| mordred | jeblair: bah | 06:58 |
| *** rpodolyaka has quit IRC | 07:01 | |
| *** rpodolyaka has joined #openstack-infra | 07:01 | |
| *** rpodolyaka has quit IRC | 07:01 | |
| *** rpodolyaka has joined #openstack-infra | 07:01 | |
| *** rpodolyaka has quit IRC | 07:02 | |
| AJaeger | Good morning, we still have many manual changes queued at http://status.openstack.org/zuul/ - since 40h | 07:03 |
| AJaeger | Those could be easily killed, every job does the same... | 07:03 |
| StevenK | Do we think the gate has recovered now? | 07:07 |
| *** yolanda has joined #openstack-infra | 07:07 | |
| *** mkoderer has joined #openstack-infra | 07:17 | |
| *** arosen has joined #openstack-infra | 07:20 | |
| *** afazekas has joined #openstack-infra | 07:28 | |
| *** zns has quit IRC | 07:37 | |
| *** davidhadas has quit IRC | 07:39 | |
| *** Daisy_ has quit IRC | 07:42 | |
| *** jamielennox is now known as jamielennox|away | 07:44 | |
| *** CaptTofu has joined #openstack-infra | 07:46 | |
| *** flaper87|afk is now known as flaper87 | 07:50 | |
| *** arosen has quit IRC | 07:50 | |
| *** vkozhukalov has quit IRC | 07:50 | |
| *** CaptTofu has quit IRC | 07:51 | |
| *** markmcclain has quit IRC | 07:53 | |
| *** rcarrillocruz1 has joined #openstack-infra | 07:57 | |
| *** mrodden1 has joined #openstack-infra | 07:58 | |
| *** rcarrillocruz has quit IRC | 07:59 | |
| *** mrodden has quit IRC | 08:00 | |
| *** mrda is now known as mrda_away | 08:01 | |
| *** jlibosva has joined #openstack-infra | 08:03 | |
| *** pblaho has joined #openstack-infra | 08:05 | |
| *** eharney has joined #openstack-infra | 08:06 | |
| *** rcarrillocruz has joined #openstack-infra | 08:07 | |
| *** rcarrillocruz1 has quit IRC | 08:08 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| *** dizquierdo has joined #openstack-infra | 08:13 | |
| *** yamahata has quit IRC | 08:14 | |
| *** yamahata__ has quit IRC | 08:15 | |
| *** jgallard has joined #openstack-infra | 08:16 | |
| *** Daisy_ has joined #openstack-infra | 08:20 | |
| GheRivero | did git.openstack.org go down last night? | 08:21 |
| *** jepoy_ has quit IRC | 08:23 | |
| *** jepoy has joined #openstack-infra | 08:23 | |
| *** jcoufal has joined #openstack-infra | 08:24 | |
| *** jepoy_ has joined #openstack-infra | 08:26 | |
| *** morganfainberg is now known as morganfainberg_Z | 08:27 | |
| *** jepoy has quit IRC | 08:29 | |
| *** yamahata__ has joined #openstack-infra | 08:30 | |
| *** yamahata has joined #openstack-infra | 08:30 | |
| *** jepoy has joined #openstack-infra | 08:33 | |
| *** jepoy_ has quit IRC | 08:37 | |
| *** rlandy has joined #openstack-infra | 08:41 | |
| *** bogdando has quit IRC | 08:42 | |
| *** jepoy has quit IRC | 08:43 | |
| *** rcarrillocruz1 has joined #openstack-infra | 08:46 | |
| *** rcarrillocruz has quit IRC | 08:46 | |
| *** andreaf has joined #openstack-infra | 08:48 | |
| *** rossella_s has joined #openstack-infra | 08:50 | |
| *** spredzy has quit IRC | 08:51 | |
| *** spredzy has joined #openstack-infra | 08:51 | |
| *** denis_makogon_ has joined #openstack-infra | 08:55 | |
| *** vponomaryov has joined #openstack-infra | 08:58 | |
| *** Pradnya has joined #openstack-infra | 08:59 | |
| *** Pradnya has left #openstack-infra | 08:59 | |
| *** denis_makogon has quit IRC | 08:59 | |
| *** Pradnya has joined #openstack-infra | 08:59 | |
| Pradnya | Hi, I am contributing to Openstack first time. I am trying to submit the test cases in openstack Tempest. I am facing issue with SSH key authentication. I am getting "Permission denied" error when I am issuing the command "git review -s". I am following this link: "https://wiki.openstack.org/wiki/GerritWorkflow". Please help in submission in tempest. | 09:00 |
| *** jcoufal has quit IRC | 09:00 | |
| *** jcoufal_ has joined #openstack-infra | 09:00 | |
| *** derekh has joined #openstack-infra | 09:01 | |
| *** jcoufal_ has quit IRC | 09:02 | |
| *** yamahata has quit IRC | 09:04 | |
| *** yamahata__ has quit IRC | 09:04 | |
| ruhe | Pradnya: hi. did you register your email and public key in gerrit? | 09:05 |
| *** pbelanyi has joined #openstack-infra | 09:05 | |
| *** jpich has joined #openstack-infra | 09:06 | |
| Pradnya | Yes | 09:06 |
| Pradnya | I registered my email id and SSh key | 09:07 |
| ruhe | can you try this command: | 09:07 |
| ruhe | ssh -p 29418 review.openstack.org | 09:07 |
| Pradnya | i am getting the error "Permission denied" when i am giving "git review -s" | 09:07 |
| *** denis_makogon_ has quit IRC | 09:08 | |
| *** matsuhas_ has quit IRC | 09:08 | |
| Pradnya | cloud@idmlocal:~/tempest$ ssh -p 29418 review.openstack.org Permission denied (publickey). | 09:08 |
| Pradnya | Error is pasted in paste.openstack.org | 09:08 |
| Pradnya | http://paste.openstack.org/show/73112/ | 09:08 |
| *** hashar has joined #openstack-infra | 09:09 | |
| *** matsuhashi has joined #openstack-infra | 09:09 | |
| ruhe | Pradnya: is your username "cloud" registered in gerrit settings at https://review.openstack.org/#/settings/ ? | 09:09 |
| Pradnya | no | 09:10 |
| Pradnya | my username is Hemanth on gerrit | 09:10 |
| *** jooools has joined #openstack-infra | 09:10 | |
| ruhe | so, either need to update your username in gerrit settings | 09:10 |
| ruhe | or | 09:11 |
| ruhe | update your gitreview settings | 09:11 |
| ruhe | username should be the same on your local devbox and in gerrit | 09:11 |
| Pradnya | okay... | 09:11 |
| Pradnya | let me try | 09:11 |
| *** zhiwei has quit IRC | 09:13 | |
| *** Ryan_Lane1 has quit IRC | 09:13 | |
| *** jcoufal has joined #openstack-infra | 09:15 | |
| Pradnya | Hi | 09:17 |
| Pradnya | I have craeted the user with my user name. "Hemanth@idmlocal" will this do? | 09:18 |
| *** johnthetubaguy has joined #openstack-infra | 09:19 | |
| *** Daisy_ has quit IRC | 09:19 | |
| *** lahoucine has left #openstack-infra | 09:21 | |
| *** unicell has quit IRC | 09:22 | |
| *** ildikov_ has joined #openstack-infra | 09:24 | |
| *** julienvey has joined #openstack-infra | 09:24 | |
| *** alff has joined #openstack-infra | 09:25 | |
| ruhe | Pradnya: it shouldn't contain hostname part | 09:26 |
| Pradnya | okay | 09:26 |
| Pradnya | Will try that | 09:26 |
| *** unicell has joined #openstack-infra | 09:28 | |
| *** CaptTofu has joined #openstack-infra | 09:35 | |
| *** alff has quit IRC | 09:35 | |
| *** zhiwei has joined #openstack-infra | 09:37 | |
| *** CaptTofu has quit IRC | 09:40 | |
| *** johnthetubaguy has quit IRC | 09:44 | |
| *** johnthetubaguy has joined #openstack-infra | 09:44 | |
| *** sarob_ has joined #openstack-infra | 09:48 | |
| *** jp_at_hp has joined #openstack-infra | 09:49 | |
| *** Pradnya has quit IRC | 09:49 | |
| *** sarob_ has quit IRC | 09:54 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:57 | |
| *** johnthetubaguy has quit IRC | 10:01 | |
| *** saju_m has joined #openstack-infra | 10:05 | |
| *** yamahata__ has joined #openstack-infra | 10:08 | |
| *** yassine has joined #openstack-infra | 10:09 | |
| *** yamahata has joined #openstack-infra | 10:09 | |
| *** xchu_ has quit IRC | 10:15 | |
| *** saju_m has quit IRC | 10:18 | |
| *** rcarrillocruz has joined #openstack-infra | 10:18 | |
| rcarrillocruz | hmm, just cloned zuul git repo | 10:20 |
| rcarrillocruz | ran tox | 10:20 |
| rcarrillocruz | and got 'tox.ConfigError: ConfigError: substitution key 'posargs' not found' | 10:20 |
| rcarrillocruz | is this an upstream bug? | 10:20 |
| *** rcarrillocruz1 has quit IRC | 10:20 | |
| *** eharney has quit IRC | 10:23 | |
| ihrachys | rcarrillocruz: downgrade tox to 1.6 | 10:26 |
| ihrachys | it's an issue with tox 1.7 | 10:26 |
| *** matsuhashi has quit IRC | 10:29 | |
| *** rcarrillocruz1 has joined #openstack-infra | 10:30 | |
| *** saju_m has joined #openstack-infra | 10:30 | |
| *** rcarrillocruz has quit IRC | 10:31 | |
| *** andreaf has quit IRC | 10:35 | |
| rcarrillocruz1 | k, thx ihrachys | 10:37 |
| sdague | jeblair / clarkb: we made the libvirt change so that we could finally debug the top bug in the gate | 10:44 |
| *** yaguang has quit IRC | 10:47 | |
| *** markmc has joined #openstack-infra | 10:48 | |
| *** dcramer__ has quit IRC | 10:48 | |
| *** sarob_ has joined #openstack-infra | 10:51 | |
| *** saju_m has quit IRC | 10:54 | |
| *** sarob_ has quit IRC | 10:55 | |
| *** bobba has joined #openstack-infra | 11:00 | |
| *** bobba is now known as GnomeKing | 11:00 | |
| *** vogxn has joined #openstack-infra | 11:01 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: ZMQ Event Publisher plugin support added https://review.openstack.org/77561 | 11:01 |
| *** vogxn has quit IRC | 11:02 | |
| *** CaptTofu has joined #openstack-infra | 11:05 | |
| *** johnthetubaguy1 has quit IRC | 11:09 | |
| *** dkliban has quit IRC | 11:13 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin https://review.openstack.org/73601 | 11:14 |
| *** mbacchi has joined #openstack-infra | 11:14 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:16 | |
| *** dkliban has joined #openstack-infra | 11:16 | |
| *** hashar has quit IRC | 11:17 | |
| *** rcarrillocruz has joined #openstack-infra | 11:17 | |
| *** hashar has joined #openstack-infra | 11:19 | |
| *** rcarrillocruz1 has quit IRC | 11:20 | |
| *** talluri has quit IRC | 11:26 | |
| *** jgallard has quit IRC | 11:26 | |
| *** talluri has joined #openstack-infra | 11:26 | |
| *** eharney has joined #openstack-infra | 11:27 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Ruby metrics plugin support added https://review.openstack.org/73528 | 11:28 |
| *** rcarrillocruz1 has joined #openstack-infra | 11:28 | |
| *** rcarrillocruz has quit IRC | 11:29 | |
| *** talluri has quit IRC | 11:30 | |
| *** rcarrillocruz has joined #openstack-infra | 11:32 | |
| *** andreaf has joined #openstack-infra | 11:35 | |
| *** rcarrillocruz1 has quit IRC | 11:35 | |
| *** malini_afk is now known as malini | 11:37 | |
| *** ociuhandu has quit IRC | 11:41 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: stop indexing libvirt logs https://review.openstack.org/79586 | 11:42 |
| sdague | SergeyLukjanov / fungi ^^^^^^^^^ | 11:42 |
| *** dkliban has quit IRC | 11:47 | |
| *** pradnya has joined #openstack-infra | 11:49 | |
| SergeyLukjanov | sdague, hey | 11:50 |
| *** sarob_ has joined #openstack-infra | 11:52 | |
| kashyap | sdague, Hm, that bug 1254872 has been around for quite a while, I noticed some discussion flying by b/n you and danpb on #openstack-nova, any root-cause analysis on it yet? | 11:53 |
| sdague | kashyap: nope | 11:53 |
| sdague | danpb said he needed more logs | 11:54 |
| SergeyLukjanov | sdague, https://review.openstack.org/#/c/79586/ approved | 11:54 |
| pradnya | Hi, I am contributing to Openstack first time. I am trying to submit the test cases in openstack Tempest. I am facing issue with SSH key authentication. I am getting "Permission denied" error when I am issuing the command "git review -s". I am following this link: "https://wiki.openstack.org/wiki/GerritWorkflow". Please help in submission in tempest. | 11:55 |
| pradnya | please help | 11:55 |
| kashyap | sdague, Okay, I'll check with him if I can help debug it or setup a minimal env to reproduce it, looks like a pesky case | 11:55 |
| sdague | kashyap: thanks! | 11:56 |
| pradnya | Please help in tempest submission | 11:56 |
| sdague | kashyap: the logs should be on in the gate now | 11:56 |
| kashyap | sdague, I attempted in the past couldn't just narrow it down | 11:57 |
| kashyap | sdague, If you have a pointer that'd be appreciated, my look ups have been darn slow :( | 11:57 |
| sdague | pradnya: did you upload your public key to gerrit | 11:57 |
| *** hashar has quit IRC | 11:57 | |
| *** lcostantino has joined #openstack-infra | 11:57 | |
| pradnya | sdague: yes | 11:57 |
| *** sarob_ has quit IRC | 11:58 | |
| *** ArxCruz has joined #openstack-infra | 12:00 | |
| pradnya | sdague: http://paste.openstack.org/show/73132/ | 12:00 |
| sdague | are you using the right username? | 12:01 |
| pradnya | user name in gerrit is : Hemanth | 12:02 |
| pradnya | Hemanth@localhost:~$ | 12:02 |
| pradnya | on my local system | 12:02 |
| *** changbl has quit IRC | 12:02 | |
| *** pradnya has left #openstack-infra | 12:03 | |
| sdague | what is it in gerrit? | 12:03 |
| sdague | oh, sorry, I see you answered that | 12:03 |
| *** pradnya has joined #openstack-infra | 12:03 | |
| *** weshay has joined #openstack-infra | 12:03 | |
| pradnya | got discoonnected | 12:03 |
| sdague | https://review.openstack.org/#/settings/ssh-keys - that lists your key? | 12:03 |
| pradnya | yes | 12:04 |
| pradnya | it shows correctly | 12:04 |
| sdague | ok, then I'm out of helpfulness. I don't actually have enough access to look at your account | 12:04 |
| sdague | need to wait for fungi to wake up | 12:04 |
| pradnya | :-) | 12:04 |
| *** yamahata__ has quit IRC | 12:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: stop indexing libvirt logs https://review.openstack.org/79586 | 12:06 |
| pradnya | can you look at logs http://paste.openstack.org/show/73132/ | 12:06 |
| *** aysyd has joined #openstack-infra | 12:07 | |
| *** yamahata has quit IRC | 12:09 | |
| pradnya | need help for tempest submission | 12:09 |
| pradnya | debug1: Authentications that can continue: publickey debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey) | 12:10 |
| pradnya | getting error while git "review -s" : Permission denied | 12:11 |
| pradnya | successfuly generated ssh key and uploaded to launchpad gerrit | 12:11 |
| *** dims_ has quit IRC | 12:12 | |
| *** JuanManuelOlle has joined #openstack-infra | 12:14 | |
| SergeyLukjanov | sdague, looks like we have a problem with reverting sqla 0.9 | 12:14 |
| SergeyLukjanov | https://review.openstack.org/79454 | 12:14 |
| fungi | why do we need fungi? | 12:14 |
| SergeyLukjanov | fungi, morning! | 12:14 |
| *** dims_ has joined #openstack-infra | 12:14 | |
| fungi | ahh, stop indexing libvirt logs | 12:14 |
| pradnya | Hi Fungi | 12:16 |
| pradnya | need help in submiting test case in tempest | 12:16 |
| pradnya | debug1: Authentications that can continue: publickey debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey) | 12:16 |
| fungi | pradnya: most likely cause is that your gerrit username is not what you're trying to authenticate as. that seems to trip up a lot of people | 12:17 |
| *** adalbas has joined #openstack-infra | 12:17 | |
| fungi | pradnya: oh, i see you already said it was the same further up | 12:18 |
| fungi | pradnya: second most common problem is that people add their ssh keys with linebreaks in them. that will also cause problems | 12:18 |
| fungi | the entire key need to be one long, unbroken line | 12:18 |
| pradnya | right | 12:19 |
| pradnya | debug logs: http://paste.openstack.org/show/73132/ | 12:19 |
| fungi | sdague: should https://review.openstack.org/79586 go in before or after https://review.openstack.org/79540 ? | 12:19 |
| sdague | fungi: if we stop copying the libvirt log, then we give up on the bug in question | 12:20 |
| fungi | sdague: so you're suggesting 79586 instead of 79540 | 12:21 |
| sdague | I get that the logs are big, but danpb basically said - Bug 1254872 is not debugable without them | 12:21 |
| sdague | yeh, just drop indexing them | 12:21 |
| *** david-lyle has quit IRC | 12:21 | |
| fungi | okay, fair enough | 12:21 |
| SergeyLukjanov | fungi, sdague, but what's about free space on logs.o.o? | 12:23 |
| fungi | SergeyLukjanov: well, once i add one more 1tb volume, that server is tapped out. rackspace won't go above 1tb on cinder volumes and their xen deployment prevents more than 14 additional virtual block devices besides the two which the server gets by default | 12:24 |
| fungi | pradnya: [2014-03-11 10:57:43,691 +0000] 9e8fa436 Hemanth - AUTH FAILURE FROM 125.22.63.138 no-matching-key | 12:24 |
| fungi | pradnya: that's from the server | 12:24 |
| fungi | pradnya: i'l look around in teh database to make sure things on this end match what you're seeing in the web interface for your account | 12:25 |
| sdague | SergeyLukjanov: my hope is we can get some resolution quicker and turn them off | 12:25 |
| SergeyLukjanov | sdague, cool, just to clarify | 12:25 |
| pradnya | fungi : ok.any work around. We are stuck with this for 4 days. It would be of great help if I get any pointers | 12:26 |
| sdague | fungi: is the logstash drop real? | 12:29 |
| fungi | pradnya: when you go to https://review.openstack.org/#/settings/ it says Account ID 10366 right? | 12:29 |
| *** wchrisj has joined #openstack-infra | 12:29 | |
| sdague | or is that synthetic because something restarted? | 12:29 |
| pradnya | fungi: yes | 12:29 |
| fungi | sdague: no, the workers are dropping like flies because they're oom'ing when trying to index libvirt logs | 12:29 |
| *** miqui has quit IRC | 12:29 | |
| *** wchrisj has quit IRC | 12:29 | |
| sdague | fungi: ok | 12:30 |
| sdague | fungi: no, I mean the queue dropped right now | 12:30 |
| sdague | a lot | 12:30 |
| *** mburned_out is now known as mburned | 12:30 | |
| fungi | sdague: oh, that... maybe a bad snmp sample, or that might be where they restarted for the config change? | 12:30 |
| *** rfolco has joined #openstack-infra | 12:31 | |
| openstackgerrit | shihanzhang proposed a change to openstack/requirements: Upgrade requirements to alembic https://review.openstack.org/79593 | 12:32 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update to not use libvirt log https://review.openstack.org/79594 | 12:34 |
| fungi | pradnya: does the output of 'ssh-keygen -y -f /home/Hemanth/.ssh/id_rsa' match what was added to https://review.openstack.org/#/settings/ssh-keys (except for the comment field at the end)? | 12:35 |
| sdague | fungi: so the config change restart means we just end up dumping all the intermediate results | 12:35 |
| fungi | sdague: i'm not sure | 12:36 |
| sdague | fungi: honestly, it looks like this bug was the only one that ever needed libvirt logs anyway | 12:36 |
| sdague | and the fingerprint worked fine on screen-n-cpu | 12:36 |
| fungi | sdague: yeah, we started archiving those logs specifically because of that bug | 12:37 |
| sdague | yeh | 12:37 |
| sdague | I just want to get our requirements reverted so we can actually test with global requirements again | 12:38 |
| sdague | fungi: if we see a reset, can you promote - 79454,1 | 12:38 |
| fungi | sdague: is that the requirements backoff? | 12:38 |
| pradnya | fungi: id_rsa is having public and private. We have uploaded public key in gerrit. | 12:38 |
| sdague | fungi: yeh, it's a revert of the allow for 0.9 | 12:39 |
| *** e0ne_ has joined #openstack-infra | 12:39 | |
| sdague | because it breaks ceilometer's db migrations | 12:39 |
| sdague | which I think I've actually got the fix for | 12:39 |
| fungi | pradnya: right, but 'ssh-keygen -y -f /home/Hemanth/.ssh/id_rsa' will tell you what the private key corresponding to /home/Hemanth/.ssh/id_rsa should be, in case the one in /home/Hemanth/.ssh/id_rsa.pub is actually incorrect for some reason | 12:39 |
| sdague | however, we should back that off, fix devstack so it actually enforces g-r (which we stopped doing), land the ceilo fix, then try the g-r update again (so it can be tested for real this time) | 12:40 |
| *** heyongli has joined #openstack-infra | 12:40 | |
| sdague | so I've got the first 3 patches all in flight | 12:40 |
| fungi | sdague: awesome | 12:40 |
| sdague | if I had cross project dependencies I could even tell you if #4 would work :) | 12:40 |
| *** e0ne has quit IRC | 12:41 | |
| sdague | fungi: we just reset (all but the top item), want to do a promote? | 12:42 |
| sdague | we can lose the time on the top item | 12:42 |
| fungi | sdague: well, if i do that cinder change is also going to get retested | 12:42 |
| sdague | oh, right | 12:43 |
| sdague | and they failed on unit tests | 12:43 |
| sdague | never mind | 12:43 |
| pradnya | fungi: I have pasted both keys : http://paste.openstack.org/show/73139/ | 12:43 |
| *** nosnos has quit IRC | 12:44 | |
| *** CaptTofu has quit IRC | 12:44 | |
| fungi | pradnya: is that first line the output from 'ssh-keygen -y -f /home/Hemanth/.ssh/id_rsa' or just the contents of /home/Hemanth/.ssh/id_rsa.pub | 12:44 |
| sdague | fungi: any idea why the logstash workers are still queuing up - http://graphite.openstack.org/render/?from=-24hours&height=1024&until=now&width=1600&bgcolor=ffffff&fgcolor=000000&target=color(alias(stats.gauges.logstash.geard.queue.running,%20%27Running%27),%20%27blue%27)&target=color(alias(stats.gauges.logstash.geard.queue.waiting,%20%27Waiting%27),%20%27red%27)&target=color(alias(stats.gauges.logstash.geard.queue.total,%20%27Total%20Jobs%27),%20%2788 | 12:45 |
| sdague | 8888%27)&target=color(alias(stats.gauges.logstash.geard.workers,%20%27Workers%27),%20%27green%27)&title=Logstash%20Job%20Queue&_t=0.00997619959525764#1394541771491 | 12:45 |
| sdague | I would have expected after dropping the libvirt logs we'd be ok | 12:45 |
| fungi | sdague: i wouldn't be surprised if there's a brief increase for some backlog which built up while they were restarting, but it's possible they're just climbing again | 12:46 |
| sdague | ok | 12:46 |
| GheRivero | is there any plan to release a new version of pypi-mirror? | 12:47 |
| fungi | GheRivero: if you want one, i'll stick it on the agenda for today's meeting | 12:47 |
| fungi | GheRivero: looks like there are a couple non-cosmetic patches since 1.0.0 so it may be time for 1.1.0 | 12:48 |
| GheRivero | fungi: I could be worth it. The one actually released doesn-t support --allow-external sources, so it's useless :) | 12:49 |
| fungi | GheRivero: well, the d24f19b commit just hard-codes a whitelist for netifaces, psutil, pysendfile and pytidylib but as long as those are all you need, then i guess that helps | 12:50 |
| pradnya | fungi: Thanks we have corrected ssh key in gerrit manually and now its going forward | 12:50 |
| fungi | GheRivero: we hoped it would be temporary, but there do seem to be some holdouts still | 12:51 |
| fungi | pradnya: you're not getting the authentication error any longer? excellent. i'm glad that exercise was worth trying then | 12:51 |
| *** jhesketh has quit IRC | 12:51 | |
| *** mugsie has joined #openstack-infra | 12:52 | |
| fungi | sometimes people do, for various reasons, end up with public and private keys which aren't from the same set and don't realize it | 12:52 |
| pradnya | fungi: yes we are not getting authentication error any more. We were stuck here for 4 days. thanks a ton | 12:52 |
| fungi | pradnya: you're welcome. that's a weird issue to diagnose and a lot of people don't think to check it | 12:52 |
| *** dkliban has joined #openstack-infra | 12:53 | |
| *** rcarrillocruz1 has joined #openstack-infra | 12:53 | |
| fungi | sdague: it looks like that queue bump may be cresting now | 12:53 |
| sdague | fungi: great | 12:53 |
| pradnya | fungi: true | 12:53 |
| *** rcarrillocruz has quit IRC | 12:54 | |
| *** dkranz has joined #openstack-infra | 12:55 | |
| *** sarob_ has joined #openstack-infra | 12:55 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: update to not use libvirt log https://review.openstack.org/79594 | 12:57 |
| *** mwagner_lap has quit IRC | 13:00 | |
| *** sarob_ has quit IRC | 13:00 | |
| *** afazekas has quit IRC | 13:04 | |
| *** alexpilotti has joined #openstack-infra | 13:05 | |
| *** dprince has joined #openstack-infra | 13:05 | |
| *** CaptTofu has joined #openstack-infra | 13:07 | |
| *** yamahata__ has joined #openstack-infra | 13:11 | |
| *** CaptTofu has quit IRC | 13:11 | |
| *** jgallard has joined #openstack-infra | 13:13 | |
| *** hashar has joined #openstack-infra | 13:14 | |
| *** yamahata__ has quit IRC | 13:15 | |
| *** mriedem has joined #openstack-infra | 13:15 | |
| *** yamahata__ has joined #openstack-infra | 13:16 | |
| *** afazekas has joined #openstack-infra | 13:17 | |
| *** thuc has joined #openstack-infra | 13:18 | |
| *** julim has joined #openstack-infra | 13:18 | |
| sdague | fungi: the worker queue still looks like linear growth | 13:20 |
| sdague | any chance you could go look. It would be good to know if we need to do something other than drop the libvirt indexing here | 13:20 |
| sdague | especially if there were queued events that are just killing things | 13:21 |
| *** yamahata__ has quit IRC | 13:21 | |
| *** yamahata__ has joined #openstack-infra | 13:22 | |
| fungi | i'll see if i can figure out what jeblair and clarkb were looking at last night | 13:22 |
| sdague | cool | 13:23 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Only public user fields in unauthorized requests https://review.openstack.org/78586 | 13:24 |
| *** plaurin_ has joined #openstack-infra | 13:25 | |
| *** lukego has joined #openstack-infra | 13:25 | |
| lukego | Howdy! I'm testing 3rd party CI with the "Tail-f Jenkins" account on the openstack-dev/sandbox project using Jay Pipes' scripst. I don't see any votes coming from my side. Is that because the account is non-voting even for the sandbox repo? | 13:26 |
| *** yamahata has joined #openstack-infra | 13:26 | |
| *** yamahata__ has quit IRC | 13:27 | |
| *** yamahata__ has joined #openstack-infra | 13:28 | |
| *** johnthetubaguy has joined #openstack-infra | 13:28 | |
| *** wchrisj has joined #openstack-infra | 13:28 | |
| *** yamahata__ has quit IRC | 13:30 | |
| *** yamahata__ has joined #openstack-infra | 13:31 | |
| fungi | lukego: the "Tail-f NCS Jenkins" account is listed in https://review.openstack.org/#/admin/groups/270,members so it should be capable of leaving +/-1 vrif votes on changes for the openstack-dev/sandbox project | 13:32 |
| lukego | OK. Could be something on my side then. Thanks! | 13:33 |
| fungi | lukego: have you tried (or observed) uploading a new change or a new patchset for an existing change on that project? | 13:33 |
| lukego | Do you have any tips for how to check? I was expecting votes. have looked in every log I can think of. I do see the tests running and just not the votes going up | 13:33 |
| lukego | yep | 13:33 |
| lukego | I see the "Starting jobs..." update from my CI but nothing more | 13:33 |
| lukego | Now might be a good time to setup the public logging :-) | 13:34 |
| fungi | lukego: hmm... i'm not sure where to look for error logs from the gerrit-jenkins trigger plugin since i don't use it myself | 13:34 |
| *** yamahata__ has quit IRC | 13:34 | |
| fungi | lukego: jaypipes or others who are using it might have better suggestions on what to look at | 13:34 |
| plaurin_ | Hello everyone, I want to implement Openstack CI with Atlassian Bamboo. After some searching, I did not found a lot of information.. I found about the run_test.sh files though. Any guidelines? Is that script for 3rd party CI? | 13:35 |
| lukego | I'll see how far I can get before the sun reaches their timezone | 13:35 |
| *** yamahata__ has joined #openstack-infra | 13:35 | |
| *** afazekas has quit IRC | 13:36 | |
| *** mfer has joined #openstack-infra | 13:37 | |
| fungi | plaurin_: i have no idea what an atlassian bamboo is... we do have some basic documentation on doing third-party integration testing, but without knowing what you're trying to test or which run_test.sh files you're talking about (tempest? nova? what?) it's hard to know where to send you | 13:38 |
| plaurin_ | I'm working with Horizon for now | 13:40 |
| *** mrodden1 has quit IRC | 13:40 | |
| *** thuc has quit IRC | 13:41 | |
| *** yamahata has quit IRC | 13:41 | |
| *** yamahata has joined #openstack-infra | 13:41 | |
| fungi | plaurin_: so are you trying to run horizon unit/functional tests or are you trying to perform integration testing between horizon and other openstack software? | 13:42 |
| *** bogdando has joined #openstack-infra | 13:42 | |
| plaurin_ | Unit tests/Func. | 13:43 |
| *** zns has joined #openstack-infra | 13:43 | |
| *** yamahata__ has quit IRC | 13:43 | |
| sdague | plaurin_: honestly, you are going to need to do a ton of work to get bamboo integrated here | 13:46 |
| sdague | is there really a reason for that decision? | 13:46 |
| *** yamahata__ has joined #openstack-infra | 13:47 | |
| pradnya | Hello Fungi | 13:48 |
| pradnya | I am getting error while submitting the code. for "git review", i am getting error pasted at http://paste.openstack.org/show/73140/ | 13:48 |
| plaurin_ | A ton of work! Oh :P We use Bamboo for most of our work but maybe I will use Jenkins for openstack.. | 13:49 |
| pradnya | Error is for email address that is not matching with the registered email id in gerrit. eg. remote: ERROR: In commit 7b3c9c46a3406ad0c0b1886a2a550ba956118b81 remote: ERROR: committer email address Hemanth@localhost.(none) remote: ERROR: does not match your user account. | 13:49 |
| fungi | pradnya: yes, you need to configure git to know what e-mail address it should associate with your commits | 13:50 |
| *** jcoufal has quit IRC | 13:50 | |
| pradnya | I have set hemanth.n@tcs.com as email-address | 13:50 |
| pradnya | both in gerrit and launchpad | 13:50 |
| *** yamahata__ has quit IRC | 13:50 | |
| fungi | pradnya: you need to set it in your global git config on your workstation | 13:50 |
| sdague | plaurin_: so I haven't used bamboo in 7 years, however, realize that *everyone* doing openstack work, all 30 3rd party ci systems, are using jenkins | 13:51 |
| *** bada has joined #openstack-infra | 13:51 | |
| sdague | so you'd be out on your own figuring out things from scratch | 13:51 |
| sdague | including the reporting pieces | 13:51 |
| fungi | sdague: (well, except for turbo-hipster) | 13:51 |
| pradnya | fungi: I have run " git config --global user.name "Hemanth" | 13:51 |
| *** yamahata__ has joined #openstack-infra | 13:51 | |
| *** malini is now known as malini_afk | 13:52 | |
| pradnya | also git config --global user.email hemanth.n@tcs.com | 13:52 |
| plaurin_ | okay, I am surprised, thanks for the info | 13:52 |
| fungi | pradnya: you probably set that after you committed the change you're trying to submit for review. if so, git commit --amend first | 13:52 |
| pradnya | after that also i am getting the above error | 13:53 |
| *** smarcet has joined #openstack-infra | 13:53 | |
| *** wenlock has joined #openstack-infra | 13:53 | |
| pradnya | okay | 13:53 |
| fungi | pradnya: what gerrit's complaining about there is that a commit you're trying to submit for review was made by "Hemanth@localhost" | 13:54 |
| fungi | pradnya: if you run 'git log' you'll probably see the same thing | 13:54 |
| sdague | fungi: sure, it's running completely different kinds of tests though. So was ground up built for our system. | 13:54 |
| fungi | sdague: agreed | 13:54 |
| pradnya | okay. next step would be "git commit --amend" and then again running above command for config and review. Is that fine? | 13:55 |
| fungi | pradnya: you probably shouldn't have to re-run the git config commands | 13:55 |
| sdague | pradnya: you need --reset-author | 13:55 |
| pradnya | okay | 13:56 |
| fungi | pradnya: sdague is right, actually. just a --amend won't change the author | 13:56 |
| * fungi forgot about that | 13:56 | |
| fungi | usually if you're running git commit for the first time on a system it will provide output telling you these things | 13:56 |
| fungi | and i tend to just cut and paste what it says | 13:56 |
| *** mrodden has joined #openstack-infra | 13:56 | |
| pradnya | I am giving the command "git commit --amend --reset-author "hemanth.n@tcs.com"... | 13:57 |
| *** zns has quit IRC | 13:57 | |
| pradnya | is that fine? | 13:57 |
| fungi | pradnya: you don't have to specify your e-mail address at the end of it. it should figure out which one to use from your global config | 13:58 |
| fungi | automatically | 13:59 |
| pradnya | okay | 13:59 |
| *** yamahata__ has quit IRC | 13:59 | |
| fungi | this is just fixing the commit which you made prior to setting up your git configuration correctly | 13:59 |
| *** esker has joined #openstack-infra | 14:00 | |
| fungi | since the commit predates your configuration | 14:00 |
| *** zns has joined #openstack-infra | 14:00 | |
| *** heyongli has quit IRC | 14:01 | |
| *** pdmars has joined #openstack-infra | 14:01 | |
| *** fcarpenter has joined #openstack-infra | 14:02 | |
| pradnya | Fungi and sdague: Thanka a lot :) | 14:03 |
| pradnya | I have submitted the test case | 14:04 |
| pradnya | :) | 14:04 |
| *** jgrimm has joined #openstack-infra | 14:04 | |
| fungi | pradnya: congratulations! your first commit is always the hardest. it gets a lot easier from this point on | 14:04 |
| russellb | fungi: can we move gantt and python-ganttclient to stackforge in the upcoming rename window? | 14:05 |
| *** yassine has quit IRC | 14:05 | |
| *** yamahata__ has joined #openstack-infra | 14:05 | |
| *** sarob_ has joined #openstack-infra | 14:05 | |
| fungi | russellb: do you have a change up for openstack/governance already? | 14:06 |
| fungi | we should probably get that proposed too | 14:06 |
| russellb | https://review.openstack.org/#/c/79519/ | 14:06 |
| *** afazekas has joined #openstack-infra | 14:08 | |
| fungi | russellb: okay, fair enough. i'm a little worried about moving a project out of the openstack namespace while it's still officially considered part of an official program, but as long as both happen in close proximity i guess that's fine. i'll bring it up on the agenda for today's team meeting | 14:08 |
| *** pblaho has quit IRC | 14:08 | |
| russellb | oh, you'd rather merge the governance change first? | 14:08 |
| russellb | that's fine | 14:08 |
| *** gokrokve has joined #openstack-infra | 14:09 | |
| fungi | russellb: keep in mind that melange and python-melangeclient are still in the openstack namespace | 14:09 |
| fungi | russellb: i don't see that causing anyone to think they have to review changes for them | 14:09 |
| *** yamahata__ has quit IRC | 14:09 | |
| openstackgerrit | Ziad Sawalha proposed a change to openstack/requirements: Add PEP-257 Testing Tools https://review.openstack.org/79054 | 14:10 |
| *** sarob_ has quit IRC | 14:10 | |
| *** alexpilotti has quit IRC | 14:10 | |
| fungi | what makes them officially part of your program or not is your decision to include or not include it in the list for the governance repo | 14:10 |
| russellb | fungi: maybe i'm being too paranoid about what being under openstack/ says to people | 14:10 |
| russellb | and true | 14:11 |
| *** fcarpenter has quit IRC | 14:11 | |
| *** dkranz has quit IRC | 14:11 | |
| *** resker has joined #openstack-infra | 14:12 | |
| russellb | fungi: i had gone neutral on the rename, and then the thread went this way: http://lists.openstack.org/pipermail/openstack-infra/2014-February/000879.html | 14:12 |
| * fungi is woefully behind on ml reading | 14:13 | |
| russellb | we all are | 14:13 |
| SergeyLukjanov | yeah... | 14:13 |
| sdague | russellb: we still have openstack/melange so I wouldn't be too concerned about what being under openstack/ means | 14:14 |
| *** esker has quit IRC | 14:14 | |
| russellb | i don't think we should have that either, heh | 14:14 |
| russellb | anyway, i don't feel *that* strongly about it | 14:14 |
| sdague | and, honestly, if gantt's going to be a doover anyway, it probably doesn't need all the gerrit history preserved | 14:15 |
| fungi | i think these cosmetic namespace groupings are causing more confusion than they solve, personally | 14:15 |
| *** pradnya has quit IRC | 14:15 | |
| sdague | and could just reimport as stackforge instead of the outage rename | 14:15 |
| *** changbl has joined #openstack-infra | 14:15 | |
| fungi | though they would need a new project name since we have a lot of systems which rely on just the part afterthe slash | 14:16 |
| sdague | I think that's the real question in my mind. it sounds like gantts going to end up as a do-over. | 14:16 |
| sdague | so I'm not sure why we're preserving anything | 14:16 |
| *** ociuhandu has joined #openstack-infra | 14:16 | |
| russellb | personally i'm not sure why it's being kept | 14:16 |
| russellb | but those working on it want to play with it some more | 14:16 |
| russellb | so *shrug* doesn't hurt anything | 14:16 |
| sdague | well, except it generates more work for the infra team, because someone wants to play :) | 14:17 |
| *** atiwari has joined #openstack-infra | 14:17 | |
| sdague | I feel very similar to the kite discussion | 14:17 |
| sdague | it has not purpose, delete it. | 14:17 |
| fungi | well, deleting things from gerrit doesn't really work | 14:17 |
| *** pcrews has joined #openstack-infra | 14:18 | |
| *** resker has quit IRC | 14:18 | |
| sdague | fungi: ok, that's news to me | 14:18 |
| *** wenlock has quit IRC | 14:18 | |
| *** yassine has joined #openstack-infra | 14:18 | |
| SergeyLukjanov | what's the melange project? | 14:19 |
| *** alexpilotti has joined #openstack-infra | 14:19 | |
| sdague | something that predates me | 14:19 |
| fungi | it can be basically abandoned and left there once nobody cares about it any longer, and if we really want someone can merge a commit to master which removes all the other files and puts a deprecation notice in the readme or something | 14:19 |
| fungi | SergeyLukjanov: sdague: description='IPAM mangement service for Openstack' http://git.openstack.org/cgit/openstack/melange/tree/setup.py#n78 | 14:21 |
| fungi | even the name was redundant! ;) | 14:21 |
| fungi | er, the description | 14:21 |
| fungi | "ipam management" | 14:21 |
| fungi | isn't management what the m in ipam stands for? | 14:21 |
| *** malini_afk is now known as malini | 14:22 | |
| *** zhiwei has quit IRC | 14:23 | |
| fungi | sdague: the logstash queue is caught back up now. none of the jenkins log workers were running and had stale pidfiles left behind | 14:23 |
| fungi | [Tue Mar 11 09:45:06 2014] Out of memory: Kill process 8594 (log-gearman-wor) score 688 or sacrifice child | 14:24 |
| fungi | from dmesg on one of them. but they all did that looks like | 14:24 |
| *** dkranz has joined #openstack-infra | 14:24 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews https://review.openstack.org/79280 | 14:25 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 14:25 |
| *** david-lyle has joined #openstack-infra | 14:26 | |
| lukego | I think my dsvm-tempest-full is stuck. any ideas for troubleshooting? The last message in Jenkins console was 30 minutes ago: Running devstack ... this takes 5 - 8 minutes (logs in logs/devstacklog.txt.gz) | 14:28 |
| *** krotscheck has joined #openstack-infra | 14:29 | |
| fungi | lukego: you could log into that slave and see what's in the devstack log | 14:29 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews https://review.openstack.org/79280 | 14:29 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 14:29 |
| fungi | lukego: and connect to the screen session and see what the various services are doing | 14:30 |
| *** JuanManuelOlle has quit IRC | 14:30 | |
| *** afazekas has quit IRC | 14:32 | |
| lukego | devstack log says: | 14:32 |
| lukego | 2014-03-11 15:15:15.437 | + bash | 14:32 |
| lukego | 2014-03-11 15:15:15.438 | ++ sudo virsh list --all | 14:32 |
| lukego | 2014-03-11 15:15:15.439 | ++ grep instance- | 14:32 |
| lukego | 2014-03-11 15:15:15.440 | ++ sed 's/.*\(instance-[0-9a-fA-F]*\).*/\1/g' | 14:32 |
| lukego | (sorry - is that too much to paste at once here?) | 14:32 |
| fungi | lukego: we have a paste server at http://paste.openstack.org/ if you want to add more detail | 14:33 |
| fungi | lukego: also, for devstack troubleshooting, you might have more luck asking in #openstack-qa | 14:33 |
| *** markmcclain has joined #openstack-infra | 14:33 | |
| lukego | Thanks | 14:33 |
| *** JuanManuelOlle has joined #openstack-infra | 14:35 | |
| *** wchrisj has quit IRC | 14:38 | |
| *** jcoufal has joined #openstack-infra | 14:40 | |
| *** GnomeKing has quit IRC | 14:40 | |
| *** lcheng has joined #openstack-infra | 14:42 | |
| *** rcarrillocruz1 has quit IRC | 14:42 | |
| *** reed has joined #openstack-infra | 14:42 | |
| *** mgagne has joined #openstack-infra | 14:42 | |
| *** esker has joined #openstack-infra | 14:42 | |
| *** rcarrillocruz has joined #openstack-infra | 14:43 | |
| *** zns has quit IRC | 14:44 | |
| jeblair | fungi: good morning | 14:46 |
| fungi | morning jeblair | 14:46 |
| *** mgagne has quit IRC | 14:46 | |
| *** esker has quit IRC | 14:47 | |
| *** wenlock has joined #openstack-infra | 14:47 | |
| *** afazekas has joined #openstack-infra | 14:47 | |
| fungi | your change last night ran into an integration test failure and didn't merge, but sdague had an alternate change to just stop indexing the file but keep collecting it for now so they can continue to troubleshoot libvirt upgrade testing | 14:47 |
| *** plaurin_ has quit IRC | 14:48 | |
| fungi | also all the logstash worker processes oom'd overnight, so i've restarted them following your paste this morning after sdague's change landed | 14:48 |
| sdague | fungi: it's not even libvirt upgrade testing | 14:48 |
| sdague | it's actually existing libvirt bugs | 14:48 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews https://review.openstack.org/79280 | 14:48 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 14:48 |
| fungi | jeblair: oh, and i had a (completely unsupportable) theory about why it's only openstack-manuals jobs we see hanging... as of a few weeks ago we now try to start 17 jobs for them simultaneously in the post pipeline, but frequently only end up with 16 which run... is there maybe a gearman protocol limit coming into play there? | 14:51 |
| jeblair | fungi, sdague: yes, we considered that, but we also need to not run out of space on logs.o.o | 14:52 |
| fungi | 16 just seems like a suspiciously non-arbitrary number | 14:52 |
| jeblair | fungi: i think devstack-gate starts more | 14:52 |
| fungi | jeblair: okay, fair enough | 14:52 |
| jeblair | sdague: how long do you want to have the system copying the libvirt logs | 14:52 |
| sdague | jeblair: sure. however I was told by danpb that we didn't have sufficient logs to actually debug the libvirt issue which is our top ER bug | 14:52 |
| jeblair | sdague: because i want to help get it solved, but it's also unsustainable to be archiving 1M line log files | 14:53 |
| *** Daisy_ has joined #openstack-infra | 14:53 | |
| sdague | jeblair: hopefully only a few days, I'm hoping that we can get enough data by then to get a fix | 14:53 |
| jeblair | fungi: did you get the message i left for you? | 14:53 |
| SergeyLukjanov | jeblair, morning | 14:53 |
| jeblair | fungi: here? http://paste.openstack.org/show/73108/ | 14:54 |
| sdague | jeblair: the biggest thing now is for es / er to recover so we can actually detect one of these fails again | 14:54 |
| jeblair | sdague: that was the intent | 14:54 |
| sdague | then point the libvirt folks at them | 14:54 |
| sdague | jeblair: yep | 14:54 |
| fungi | jeblair: that's what i meant when i said i restarted the workers following your paste | 14:55 |
| jeblair | fungi: ah cool | 14:55 |
| fungi | jeblair: sdague made a compelling argument for not dumping the libvirt logs completely, and SergeyLukjanov had already emergency-approved it, so i didn't reverify the change | 14:55 |
| fungi | but did otherwise follow the restart steps | 14:55 |
| *** thedodd has joined #openstack-infra | 14:56 | |
| jeblair | sdague: so, shortly, when we no longer need the logs, how should we reduce the verbosity? | 14:56 |
| jeblair | sdague: is it just remove all the lines in your devstack change? | 14:56 |
| sdague | jeblair: yes | 14:57 |
| jeblair | russellb: is there ever going to be a gantt? | 14:57 |
| *** dizquierdo_afk is now known as dizquierdo | 14:57 | |
| *** pdmars has quit IRC | 14:57 | |
| sdague | probably will turn if off by making it a conditional that defaults false, so in the future we could turn it back on with d-g for some window easily if we felt we needed it | 14:58 |
| jeblair | russellb: because i think it's silly to move it to stackforge and then move it back. | 15:00 |
| *** caleb_ has joined #openstack-infra | 15:01 | |
| *** blamar has joined #openstack-infra | 15:02 | |
| *** wchrisj has joined #openstack-infra | 15:02 | |
| *** sparkycollier has joined #openstack-infra | 15:03 | |
| *** pdmars has joined #openstack-infra | 15:04 | |
| *** jcoufal has quit IRC | 15:06 | |
| *** jcoufal has joined #openstack-infra | 15:07 | |
| *** rlandy has quit IRC | 15:07 | |
| *** johnthetubaguy1 has joined #openstack-infra | 15:07 | |
| *** fungi has quit IRC | 15:07 | |
| *** rlandy has joined #openstack-infra | 15:08 | |
| russellb | jeblair: maybe | 15:09 |
| russellb | jeblair: it's fine, if it's a pain for you guys, let's just leave it alone, and we'll remove it in the governance repo | 15:09 |
| russellb | jeblair: if there is, it will be a do-over from the current repo, if that matters to you | 15:09 |
| *** johnthetubaguy has quit IRC | 15:10 | |
| jeblair | russellb: so i think the restrictions we have to work with are (1) repos have indepedent names even across org boundaries (so no openstack/gantt _and_ stackforge/gantt at the same time) | 15:10 |
| jeblair | russellb: and (2) nothing is ever deleted | 15:10 |
| *** mkoderer has quit IRC | 15:11 | |
| *** mgagne has joined #openstack-infra | 15:11 | |
| jeblair | russellb: so that probably means either a bit of git-work on gantt to effect the do-over, or a rename to something not named gantt for the current repo at some point | 15:12 |
| jeblair | russellb: (we did completely replace keystone with some git magic, so it's entirely possible) | 15:12 |
| russellb | ok | 15:12 |
| *** jcoufal has quit IRC | 15:12 | |
| russellb | well i don't expect that to be a near term thing | 15:12 |
| russellb | like, not juno timeframe | 15:12 |
| *** jcoufal has joined #openstack-infra | 15:13 | |
| jeblair | russellb: okay, that sounds far enough off and vague enough that renaming it isn't unreasonable | 15:14 |
| russellb | heh | 15:14 |
| *** rcarrillocruz1 has joined #openstack-infra | 15:19 | |
| *** dcramer__ has joined #openstack-infra | 15:21 | |
| *** rcarrillocruz has quit IRC | 15:21 | |
| *** jcoufal has quit IRC | 15:25 | |
| *** zns has joined #openstack-infra | 15:26 | |
| *** thuc has joined #openstack-infra | 15:27 | |
| *** sparkycollier has quit IRC | 15:30 | |
| *** fcarpenter has joined #openstack-infra | 15:30 | |
| *** fcarpenter has left #openstack-infra | 15:30 | |
| *** gokrokve has quit IRC | 15:33 | |
| *** miqui has joined #openstack-infra | 15:33 | |
| *** gokrokve has joined #openstack-infra | 15:33 | |
| *** gokrokve has quit IRC | 15:37 | |
| *** harlowja_away is now known as harlowja | 15:38 | |
| *** Daisy_ has quit IRC | 15:38 | |
| *** johnthetubaguy1 has quit IRC | 15:40 | |
| *** krotscheck has quit IRC | 15:41 | |
| *** johnthetubaguy has joined #openstack-infra | 15:42 | |
| *** harlowja has quit IRC | 15:43 | |
| openstackgerrit | A change was merged to openstack/requirements: Revert "Switch to SQLAlchemy 0.9" https://review.openstack.org/79454 | 15:43 |
| *** jcoufal-mobile has joined #openstack-infra | 15:45 | |
| *** fungi has joined #openstack-infra | 15:46 | |
| fungi | fun. my connection to the server was dead a good 45 minutes without the client attempting to reconnect | 15:47 |
| fungi | (and yet i never lost my connection to oftc) | 15:48 |
| *** harlowja has joined #openstack-infra | 15:48 | |
| SergeyLukjanov | heh | 15:48 |
| *** sarob_ has joined #openstack-infra | 15:49 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Creating a new story now requires authentication. https://review.openstack.org/79390 | 15:50 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Fixed padding on modals. https://review.openstack.org/79393 | 15:50 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Clean-up the address-bar after auth https://review.openstack.org/78953 | 15:50 |
| *** gyee has joined #openstack-infra | 15:50 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added UI switches for authentication state. https://review.openstack.org/79478 | 15:50 |
| *** zhiyan is now known as zhiyan_ | 15:51 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Only public user fields in unauthorized requests https://review.openstack.org/78586 | 15:51 |
| *** khyati has joined #openstack-infra | 15:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revoke sudo permissions before infra jobs https://review.openstack.org/72778 | 15:54 |
| david-lyle | I would like to add an additional Horizon gate job to check python 2.7 combined with django 1.5, what repo do I need to be looking at? | 15:54 |
| sdague | jeblair / fungi: is there an easy way to put a block on requirements changes until we actually fix the testing of them? | 15:54 |
| *** bogdando has quit IRC | 15:55 | |
| fungi | sdague: depends on what you mean by easy. we can revoke approval rights on the acl for the requirements project... | 15:55 |
| jeblair | sdague: link to change that fixes testing? | 15:56 |
| fungi | david-lyle: is it in the global requirements list yet? | 15:56 |
| david-lyle | django 1.6 is, want to make sure we maintain support for 1.5 | 15:56 |
| sdague | jeblair: I believe this does - https://review.openstack.org/#/c/77815/ | 15:57 |
| david-lyle | I added a tox env to Horizon to test py 2.7 and django 1.5 combo | 15:57 |
| fungi | david-lyle: got it... the tox.ini in horizon i believe already has a section for 1.4 so you'll need to add | 15:57 |
| fungi | you beat me to it | 15:57 |
| *** gokrokve has joined #openstack-infra | 15:57 | |
| david-lyle | but somewhere I need to tell the gate to run that tox env, no? | 15:57 |
| AJaeger | fungi, the openstack-manuals changes are still in the post gate at http://status.openstack.org/zuul/ - since 49 hours now. Can't we just remove them from the queue? | 15:58 |
| sdague | fungi: revoke approval is probably good | 15:58 |
| jeblair | AJaeger: no way to do that | 15:58 |
| sdague | I'm just concerned we're chasing to get things testing again | 15:58 |
| david-lyle | Or does that happen in patches post tox.ini merge? | 15:58 |
| sdague | the approvers on requirements are often not very plugged into the state of the world | 15:58 |
| fungi | david-lyle: in that case, in the openstack-infra/config repo you need to add the job definition in modules/openstack_project/files/jenkins_job_builder/config/horizon.yaml | 15:58 |
| sdague | for better or worse | 15:58 |
| fungi | david-lyle: and then entries in modules/openstack_project/files/zuul/layout.yaml | 15:59 |
| AJaeger | jeblair, is there a way to fix these? | 15:59 |
| fungi | AJaeger: we would have to restart zuul | 15:59 |
| AJaeger | jeblair, note that the post gate just publishes all guides, so each of the jobs does the same, and removing all of them except the latest would not cause any problems on the endresult... | 15:59 |
| fungi | AJaeger: we don't have a feature implemented to remove pending jobs directly right now | 15:59 |
| *** jcoufal has joined #openstack-infra | 16:00 | |
| jeblair | AJaeger: i know. the ones that have run have already published, so the only effect of this is that the status screen looks messy. | 16:00 |
| david-lyle | fungi: thank you | 16:00 |
| AJaeger | jeblair, the config-reference is not getting published, everything else is. | 16:01 |
| sdague | jeblair: so I'm about to go run outside for a bit. I'm hoping the change moves through check -> gate fine at this point. Then I'll check in later on that one. | 16:01 |
| AJaeger | oh, content looks updated today - now I'm confused ;) | 16:01 |
| jeblair | AJaeger: the config reference ran for 6294e8b | 16:02 |
| sdague | then we can actually be testing global requirements again, which would be goodness | 16:02 |
| AJaeger | jeblair, ok, I'll live with the mess for now ;) | 16:02 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Cancel a build even if not found https://review.openstack.org/79391 | 16:02 |
| jeblair | https://jenkins07.openstack.org/job/openstack-config-reference/ | 16:03 |
| jeblair | https://jenkins05.openstack.org/job/openstack-user-guide-admin/ | 16:03 |
| jeblair | fungi, AJaeger: ^ that's why they are stuck | 16:03 |
| *** SumitNaiksatam has quit IRC | 16:03 | |
| *** johnthetubaguy1 has joined #openstack-infra | 16:04 | |
| fungi | jeblair: weird. if those were converted to non-maven jobs in the change that i approved, then i'm sure i deleted them with jjb on all the masters | 16:05 |
| fungi | checking my shell history | 16:05 |
| *** johnthetubaguy has quit IRC | 16:05 | |
| *** weiwei has quit IRC | 16:06 | |
| jeblair | fungi: i think it's 2 things -- those jobs are set to concurrent=false and then probably a jenkins/nodepool/gearman-plugin thing happened and one got scheduled on a dead node | 16:06 |
| jeblair | so the others are waiting | 16:06 |
| fungi | ohhh... so they are supposed to still be maven jobs? | 16:06 |
| jeblair | i think at this point with multiple jenkins masters, we can't make any guarantees about nonconcurrency, so we should just remove that | 16:07 |
| jeblair | fungi: no, maven/not-maven doesn't matter for this | 16:07 |
| fungi | good point. i'll propose a patch to strip concurrency=1 from anything which runs on an untrusted slave | 16:07 |
| fungi | presumably we still want it for the jobs which run on jenkins.o.o though | 16:08 |
| AJaeger | fungi, yes, they are still maven jobs. Haven't converted them yet. | 16:08 |
| fungi | but yeah, i see them sitting in the build queue on those masters as well | 16:08 |
| jeblair | fungi: oh, i just wrote that | 16:09 |
| fungi | all the better | 16:09 |
| AJaeger | fungi, I plan to convert them later today to non-maven and have one question: Do you prefer one large job or smaller ones? | 16:09 |
| jeblair | fungi: well, i removed it from all jobs | 16:09 |
| jeblair | fungi: you think we should keep them on the trusted slaves for now? | 16:09 |
| jeblair | fungi: actually, it's the same anyway | 16:09 |
| AJaeger | I would do one publishing job that all maven jobs | 16:09 |
| *** bogdando has joined #openstack-infra | 16:09 | |
| jeblair | fungi: there's only one instance of each trusted slave so they can't run concurrently anyway | 16:10 |
| jeblair | fungi: so i'll remove them everywhere | 16:10 |
| fungi | AJaeger: whichever is more convenient for you | 16:10 |
| fungi | jeblair: great point. we only have one executor on each of those trusted slaves anyway so they'll still queue up | 16:11 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove concurrency restriction from all jobs https://review.openstack.org/79659 | 16:11 |
| AJaeger | fungi, ok, will try going with one as I expect that to be the easiest | 16:11 |
| jeblair | there is a small chance that when that change is approved, some or all of the jobs zuul is waiting on will run. | 16:12 |
| fungi | once jjb reruns on those masters anyway | 16:12 |
| AJaeger | jeblair, what does the concurrency say? Does it allow running the same post job in parallel for two changes - or only different post jobs for the same change? | 16:13 |
| fungi | AJaeger: it's supposed to prevent two copies of the same job from running at the same time, but only applies locally for each jenkins master (we don't have any cross-master coordination which could enforce it currently) | 16:14 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add experimental keystonev3 pipeline https://review.openstack.org/79314 | 16:14 |
| AJaeger | fungi, those jobs run on different VMs? In that case there's no problem for the manual jobs with jeblair's change | 16:15 |
| fungi | AJaeger: so while jenkins01 wouldn't run two of that job simultaneously, there's nothing stopping jenkins02 from running the same job for a different change at the same time | 16:15 |
| fungi | AJaeger: yes, all those jobs run on different vms. except for a handful of jobs which we have pinned to specific trusted vms and those have only one executor apiece so can't run more than one job of any kind at the same time anyway | 16:16 |
| *** vkozhukalov has joined #openstack-infra | 16:16 | |
| *** eharney has quit IRC | 16:16 | |
| *** wchrisj has quit IRC | 16:17 | |
| *** afazekas has quit IRC | 16:17 | |
| *** wchrisj has joined #openstack-infra | 16:17 | |
| AJaeger | ok, then the change is fine from the manuals side. Thanks for the explanation. | 16:18 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: pin nose to 1.3.0 https://review.openstack.org/79661 | 16:18 |
| *** weiwei has joined #openstack-infra | 16:19 | |
| AJaeger | it only means I need to rebase https://review.openstack.org/#/c/77446/ ;) will do later... | 16:19 |
| jeblair | AJaeger: sorry :( it's for a good cause though. :) | 16:19 |
| fungi | AJaeger: the biggest risk right now from a manuals concurrency standpoint which i can see is that it's possible for a two changes to merge at roughly the same time for the same document and then finish in a different order than they merged, which leaves the manual published with slightly outdated content. but that's already the case today | 16:20 |
| AJaeger | jeblair, no problems. Such things happen.... | 16:20 |
| jeblair | fungi, AJaeger: yes, to fix this correctly, we'd need some changes to zuul | 16:20 |
| jeblair | but most of the time it's going to work right. and when it is wrong, it won't be wrong for long. | 16:21 |
| AJaeger | fungi, I hope that we publish often enough ;) | 16:21 |
| AJaeger | jeblair, exactly | 16:21 |
| fungi | yep, i'm sure it's happened, but i've never heard anyone mention that they've noticed it | 16:21 |
| AJaeger | Ok, will need to cycle home now and send patches later tonight. Thanks for working on the good cause ;) | 16:21 |
| *** e0ne has joined #openstack-infra | 16:27 | |
| *** zigo has joined #openstack-infra | 16:29 | |
| *** e0ne_ has quit IRC | 16:30 | |
| fungi | clarkb: jeblair: was https://launchpad.net/bugs/1290669 fallout from the sudoers changes? | 16:30 |
| *** vhoward has joined #openstack-infra | 16:31 | |
| jeblair | fungi: yes | 16:31 |
| *** SumitNaiksatam has joined #openstack-infra | 16:31 | |
| *** markwash has joined #openstack-infra | 16:33 | |
| *** talluri has joined #openstack-infra | 16:34 | |
| *** dcramer__ has quit IRC | 16:34 | |
| *** jcoufal-mobile has quit IRC | 16:36 | |
| *** jcoufal-mobile has joined #openstack-infra | 16:36 | |
| *** jcoufal-mobile has quit IRC | 16:37 | |
| *** rcarrillocruz has joined #openstack-infra | 16:38 | |
| *** rcarrillocruz1 has quit IRC | 16:38 | |
| *** achuprin has quit IRC | 16:39 | |
| pabelanger | jeblair, recommendations for zuul talks? I know you have a few floating around | 16:41 |
| jeblair | pabelanger: http://docs.openstack.org/infra/publications/ are the slides... | 16:41 |
| *** krotscheck has joined #openstack-infra | 16:41 | |
| jeblair | pabelanger: i hear there are recordings on youtube | 16:42 |
| jeblair | pabelanger: i think this is the most recent i've done: http://www.youtube.com/watch?v=xTPILCPRGeg | 16:43 |
| *** CaptTofu has joined #openstack-infra | 16:43 | |
| *** CaptTofu has quit IRC | 16:43 | |
| *** markmcclain has quit IRC | 16:43 | |
| *** thuc has quit IRC | 16:44 | |
| *** thuc has joined #openstack-infra | 16:44 | |
| *** Ryan_Lane has joined #openstack-infra | 16:44 | |
| *** nicedice has joined #openstack-infra | 16:44 | |
| *** CaptTofu has joined #openstack-infra | 16:45 | |
| pabelanger | jeblair, great, thank you | 16:45 |
| jeblair | pabelanger: ttx did a good talk too that was a little broader (not as deep into zuul, but put it into context of the whole system): https://fosdem.org/2014/schedule/event/openstack_testing_automation/ | 16:46 |
| dims_ | clarkb, where can i hang out to bug libvirt/uca folks? | 16:46 |
| jeblair | i can't find the video for that | 16:46 |
| *** aysyd has quit IRC | 16:47 | |
| jeblair | apparently they haven't transcoded it yet | 16:47 |
| fungi | well, you did say that the concept of recording and publishing sessions was sort of new to the fosdem disorganizers | 16:51 |
| jeblair | saturday is available! | 16:52 |
| andreaf | sdague: ping | 16:52 |
| openstackgerrit | A change was merged to openstack-infra/config: OpenstackID Documentation https://review.openstack.org/69620 | 16:56 |
| *** afazekas has joined #openstack-infra | 16:56 | |
| *** bhuvan has joined #openstack-infra | 16:56 | |
| *** bhuvan_ has joined #openstack-infra | 16:56 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Shut down the gearman server last in tests https://review.openstack.org/79472 | 16:58 |
| pleia2 | bug day time! https://etherpad.openstack.org/p/cibugreview-march2014 | 17:00 |
| *** david-lyle has quit IRC | 17:01 | |
| *** jcoufal has quit IRC | 17:04 | |
| *** sabari2 has joined #openstack-infra | 17:04 | |
| *** lcheng has quit IRC | 17:04 | |
| *** caleb_ has quit IRC | 17:04 | |
| *** ociuhandu has quit IRC | 17:04 | |
| *** esker has joined #openstack-infra | 17:05 | |
| *** zns has quit IRC | 17:06 | |
| fungi | i'm working on triaging the ones in new status starting from the top | 17:06 |
| pleia2 | thanks fungi | 17:07 |
| andreaf | Hi - I'm setting up an experimental check for a keystone v3 based test run - I've never done changes to layout / jobs before, so I'd very much like feedback reviews - https://review.openstack.org/#/c/79212/, https://review.openstack.org/#/c/79213/ | 17:08 |
| *** yassine has quit IRC | 17:09 | |
| jeblair | pleia2: i'm in! | 17:09 |
| pleia2 | \o/ | 17:10 |
| clarkb | morning | 17:10 |
| jeblair | clarkb: just in time for bug day! | 17:10 |
| clarkb | jeblair: yup | 17:10 |
| clarkb | to be fair this is slightly early for me on a UTC clock >_> | 17:10 |
| pleia2 | jeblair: thanks for the fedora review, I think there was a reason we didn't just pull in devstack script but looking at it now that might be gone | 17:10 |
| *** khyati has quit IRC | 17:10 | |
| mordred | dhellmann: ping | 17:11 |
| jeblair | pleia2: cool; the -1 was mostly for the script name and confusion there; i wouldn't block it on the devstack reuse, but would love it if things can converge | 17:11 |
| *** e0ne has quit IRC | 17:11 | |
| jeblair | mordred: oh good you're here in time for bug day too! :) | 17:11 |
| *** sdake has joined #openstack-infra | 17:11 | |
| mordred | jeblair: I'm only partially here- my dad has decided to make a last-minute visit so I'm scrambling a bit | 17:12 |
| clarkb | dims_: zul and jamespage hang out in here usually | 17:12 |
| jeblair | fungi: i'll work from the bottom up on that list | 17:12 |
| fungi | jeblair: great, thanks | 17:13 |
| openstackgerrit | Paul Michali proposed a change to openstack/requirements: Add httmock to test-requirements. https://review.openstack.org/79684 | 17:13 |
| jeblair | sdague: can you please tag gate failure bugs with 'gate-failure'? | 17:14 |
| *** lcheng has joined #openstack-infra | 17:15 | |
| clarkb | sdague: jeblair: I honeslty don't see how having all of that extra logging helps. If ES can't keep up you'd need a hadoop laying around | 17:16 |
| openstackgerrit | Ricardo Carrillo Cruz proposed a change to openstack-infra/gerritbot: Add support for notification of releases on IRC Closes-Bug: #1227985 Change-Id: I9b80b465d9d6b0e2793286cc8229b82b4389a08d https://review.openstack.org/79205 | 17:16 |
| *** dcramer__ has joined #openstack-infra | 17:16 | |
| *** sarob_ has quit IRC | 17:17 | |
| *** sarob_ has joined #openstack-infra | 17:18 | |
| SergeyLukjanov | clarkb, IIRC ES isn't too bad, probably, it'll be hard to replace it with other tools to have better performance | 17:18 |
| SergeyLukjanov | jeblair, clarkb, fungi, pleia2, how can I help with bugs triage? | 17:18 |
| *** rms_13 has joined #openstack-infra | 17:18 | |
| jeblair | SergeyLukjanov: go through the list here: https://etherpad.openstack.org/p/cibugreview-march2014 | 17:19 |
| rms_13 | sdague: ping | 17:19 |
| pleia2 | SergeyLukjanov: today tends to be a good reminder to update any of your own bugs, and review the bug list to see if there is anything you should fix yourself - particularly the higher priority ones | 17:19 |
| clarkb | SergeyLukjanov: hop on the etherpad (which I am about to do), and start plugging away at the list. usually we go through the old bugs to make sure they are up to date and also go through the new list to categorize and prioritize | 17:19 |
| SergeyLukjanov | I can start from the end of list assuming that I'll be able to evaluate low prio issues :) | 17:19 |
| jeblair | SergeyLukjanov: if you examine a bug listed in the etherpad, make a note in the etherpad | 17:20 |
| clarkb | by the way, holger responded to my pull request and bug for tox | 17:20 |
| rms_13 | Can a devstack-core member take a look at https://review.openstack.org/#/c/78042/ please? | 17:21 |
| jeblair | SergeyLukjanov: i updated the top 2 bugs as examples | 17:21 |
| *** nati_ueno has joined #openstack-infra | 17:21 | |
| *** miqui has quit IRC | 17:21 | |
| *** SumitNaiksatam has quit IRC | 17:21 | |
| SergeyLukjanov | jeblair, are we using some mark to mark that issue is now under review? | 17:21 |
| jeblair | SergeyLukjanov: you might not want to worry about the low priority ones. we won't get to everything, so it'd be better to skip those. | 17:21 |
| fungi | rms_13: devstack development tends to mostly get discussed in the #openstack-qa channel i think | 17:22 |
| fungi | you might have better luck there | 17:22 |
| SergeyLukjanov | jeblair, ok | 17:22 |
| rms_13 | fungi: ok. Saw few names up here so took a chance :) | 17:23 |
| *** SumitNaiksatam has joined #openstack-infra | 17:25 | |
| *** caleb_ has joined #openstack-infra | 17:25 | |
| *** pbelanyi has quit IRC | 17:26 | |
| *** lcheng has quit IRC | 17:26 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed a change to openstack-infra/gerritbot: Add support for notification of releases on IRC https://review.openstack.org/79205 | 17:26 |
| *** JuanManuelOlle has quit IRC | 17:26 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove false positives from query 1240256 https://review.openstack.org/79503 | 17:27 |
| openstackgerrit | A change was merged to openstack/requirements: Correct iso8601 parseing invalid year num https://review.openstack.org/77033 | 17:27 |
| *** JuanManuelOlle has joined #openstack-infra | 17:28 | |
| *** lcheng has joined #openstack-infra | 17:28 | |
| jeblair | fungi: can you triage https://bugs.launchpad.net/openstack-ci/+bug/1287975 ? it seems to have something to do with py3k; you might have more info handy (and possibly a dup bug) | 17:28 |
| fungi | jeblair: sure | 17:29 |
| *** marun has quit IRC | 17:30 | |
| *** marun has joined #openstack-infra | 17:31 | |
| *** markmc has quit IRC | 17:31 | |
| fungi | jeblair: one of the bugs i triaged was about the zuul unit test issues i think you just solved, so i corrected the lp project and assigned it to you for confirmation | 17:32 |
| jeblair | fungi: cool, thx | 17:32 |
| clarkb | pleia2: mordred https://bugs.launchpad.net/openstack-ci/+bug/1082795 is that something we need to track under openstack-ci? | 17:32 |
| SergeyLukjanov | clarkb, re jjb @ py26 https://review.openstack.org/#/c/71424/ looks ok for me, could I approve it or should we wait for someone else from jjb team? | 17:33 |
| clarkb | SergeyLukjanov: you can approve | 17:34 |
| SergeyLukjanov | clarkb, should I remove it from the etherpad when it'll be landed? | 17:35 |
| *** zns has joined #openstack-infra | 17:35 | |
| *** sarob_ has quit IRC | 17:36 | |
| SergeyLukjanov | oh, I see all savanna jobs failed with http://logs.openstack.org/47/79347/2/gate/gate-tempest-dsvm-savanna-full/e9f3dff/console.html#_2014-03-11_17_23_07_852 | 17:36 |
| clarkb | SergeyLukjanov: you can mark it as fix released on the etherpad then next bug day it won't show up | 17:36 |
| *** sarob_ has joined #openstack-infra | 17:36 | |
| *** markmcclain has joined #openstack-infra | 17:38 | |
| bknudson | jenkins is posting updates to the requirements faster than they can be checked, reviewed, validated, merged. | 17:39 |
| *** rcarrillocruz1 has joined #openstack-infra | 17:39 | |
| *** khyati_ has joined #openstack-infra | 17:39 | |
| *** rcarrillocruz has quit IRC | 17:39 | |
| SergeyLukjanov | clarkb, ok, thx, got it | 17:39 |
| clarkb | bknudson: that should be ok, it is updated the existing changes right? | 17:40 |
| jeblair | clarkb: it's kind of annoying when it does that while it's in the gate. :( | 17:40 |
| bknudson | clarkb: yes, it's updating it's existing change, so it loses the vote | 17:40 |
| *** rcarrillocruz has joined #openstack-infra | 17:40 | |
| clarkb | jeblair: yeah but that is a different problem | 17:41 |
| jeblair | well, bknudson said "validated, merged" so i assume it was at least a component | 17:41 |
| *** melwitt has joined #openstack-infra | 17:42 | |
| fungi | bknudson: sounds like a sort of review which should be prioritized if a project is having trouble keeping up with global changes to their requirements. also participating in reviewing changes proposed to the openstack/requirements project's global-requirements.txt which have impact on a project of interest may allow for getting ahead of potentially detrimental proposals | 17:43 |
| dims_ | clarkb, thanks | 17:43 |
| *** rcarrillocruz1 has quit IRC | 17:43 | |
| clarkb | https://bugs.launchpad.net/openstack-ci/+bug/1280008 is a requirements bug (asserting that packages are missing) is that something we want in openstack-ci's bug list? | 17:44 |
| bknudson | the sqlachemy 0.9 change caused keystone docs to fail to build, but that was reverted... haven't had a chance to look at it. | 17:44 |
| clarkb | I don't know that there is a better location for that | 17:44 |
| *** esker has quit IRC | 17:45 | |
| pleia2 | clarkb: so I think mordred agreed that we'd make testing happen in -infra once nova baremetal went in, but now that things are moving to ironic which isn't graduated yet I'm not really sure where this all landed | 17:45 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Logging in with a pre-existing user saves details https://review.openstack.org/79705 | 17:45 |
| jeblair | clarkb: that looks like a taskflow bug | 17:45 |
| clarkb | jeblair: wfm | 17:45 |
| jogo | fungi: as per youre comments last night https://review.openstack.org/#/c/79519/ | 17:45 |
| fungi | bknudson: yes, there were some recent changes to devstack which broke requirements change vetting and allowed that through when it shouldn't have gotten. i think sdague is very closed to having that solved | 17:45 |
| jeblair | clarkb: i'm not sure requirements can have bugs. :) | 17:45 |
| *** esker has joined #openstack-infra | 17:45 | |
| clarkb | jeblair: nice | 17:45 |
| clarkb | this is the way it will be no bugs about it | 17:46 |
| clarkb | pleia2: I think more importantly there isn't much we can do to facilitate that | 17:46 |
| *** sabari2 has quit IRC | 17:46 | |
| jeblair | (i mean, mirror or requirements enforcement scripts are infra bugs) | 17:46 |
| *** zns has quit IRC | 17:46 | |
| clarkb | either they write tests to run against tripleo baremetal or they do devstack stuff to have fake baremetal | 17:46 |
| fungi | jogo: okay. i'll add it to the agenda for today's 19:00 infra meeting | 17:46 |
| jogo | I added something last night, but it was very vague | 17:47 |
| sdague | fungi: so the fix is in the gate | 17:47 |
| sdague | 77815,2 | 17:47 |
| sdague | if we get a reset we should bump it | 17:47 |
| pleia2 | clarkb: yeah | 17:47 |
| fungi | sdague: okay. if we're lucky it'll just merge in an hour | 17:47 |
| sdague | yep | 17:48 |
| sdague | then I'll re propose the sqla thing, and we'll see how it goes | 17:48 |
| *** hashar has quit IRC | 17:48 | |
| *** freyes has joined #openstack-infra | 17:48 | |
| sdague | then I need to come up with a clever way of actually testing this | 17:49 |
| sdague | so we never get in this situation again | 17:49 |
| jeblair | clarkb: https://bugs.launchpad.net/openstack-ci/+bug/1285593 | 17:49 |
| jeblair | clarkb: seems similar to horizon issue | 17:49 |
| *** markwash has quit IRC | 17:49 | |
| jeblair | clarkb: is there a bug or review for that? | 17:49 |
| *** mfer has quit IRC | 17:50 | |
| *** johnthetubaguy1 has quit IRC | 17:50 | |
| clarkb | I am not familiar with the horizon issue. Do you mean keystone port 35357 issue? | 17:50 |
| *** esker has quit IRC | 17:50 | |
| jeblair | clarkb: yes! | 17:51 |
| jeblair | clarkb: oh, what's the lower bound for the ephemeral port range? | 17:52 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: make command parsing compatible with python 2.6 https://review.openstack.org/71424 | 17:52 |
| nibalizer | so daylight savings time is confusing me, is the meeting in 10 minutes ? | 17:52 |
| clarkb | jeblair: there is a bug for that against devstack and keystone because it is related to the use of a port in that port range | 17:52 |
| clarkb | jeblair: 9000 is way outside of that range so should be fine | 17:52 |
| nibalizer | or an hour and 10 minutes? | 17:52 |
| clarkb | different bug | 17:52 |
| clarkb | nibalizer: an hour | 17:52 |
| *** johnthetubaguy has joined #openstack-infra | 17:52 | |
| nibalizer | ah k | 17:52 |
| fungi | nibalizer: one hour and 10 minutes (19)) utc regardless of dst) | 17:52 |
| nibalizer | okay cool | 17:52 |
| fungi | s/19))/1900/ | 17:52 |
| clarkb | devananda: areyou around? https://bugs.launchpad.net/openstack-ci/+bug/1281385 | 17:52 |
| clarkb | devananda: pyghmi is a stackforge project | 17:53 |
| devananda | clarkb: looking | 17:53 |
| clarkb | devananda: pyghmi is more than welcome to do the things you have proposed -infra do for it | 17:53 |
| clarkb | but that isn't an infra bug | 17:53 |
| fungi | clarkb: i think what makes it an infra bug is that it would be closed with the approval of https://review.openstack.org/74499 (which changes infra config) | 17:54 |
| devananda | clarkb: I agree that its not a bug in infra, but ^ | 17:55 |
| devananda | clarkb: also a related question came up last week at the sprint | 17:55 |
| clarkb | fungi: devananda right but we try to have a separation of concerns there | 17:55 |
| jeblair | and we don't require bugs for patches | 17:55 |
| clarkb | we are not responsible for stackforge project testing | 17:55 |
| jamespage | clarkb, yes | 17:55 |
| *** talluri has quit IRC | 17:55 | |
| clarkb | jamespage: oh hi, I think dims_ has questions for you | 17:56 |
| devananda | clarkb: should we (ironic and other projects) add things to requirements.txt if they are required for third-party drivers? | 17:56 |
| fungi | clarkb: yeah, not an infra bug, but people filing bugs because they want to see them closed by patches. i agree it's unnecessarily pedantic | 17:56 |
| *** sweston has quit IRC | 17:56 | |
| clarkb | jamespage: dims_ has a devstack change that enables UCA so that he can test it in a gate environment | 17:56 |
| *** talluri has joined #openstack-infra | 17:56 | |
| *** david-lyle has joined #openstack-infra | 17:56 | |
| clarkb | devananda: if you are asserting you need them then yes | 17:56 |
| jamespage | clarkb, ack | 17:56 |
| jogo | fungi: https://bugs.launchpad.net/bugs/1290637 | 17:57 |
| devananda | clarkb: where "need" is "support for this optional driver" - yes. where "need" is "core functionality requires it" - no. | 17:57 |
| jogo | that wasn't meant to be a duplicate | 17:57 |
| fungi | jogo: so what is the actual bug there? | 17:57 |
| jogo | new kernel doesn't like us | 17:57 |
| devananda | clarkb: so i'm seekign some guidance on whether external libs for third-party drivers should go into requirements or not | 17:57 |
| jogo | fun notsure how to resolve that though | 17:58 |
| jogo | fungi: * | 17:58 |
| *** esker has joined #openstack-infra | 17:58 | |
| fungi | jogo: um... open a bug against ubuntu's kernel package or against libvirt? | 17:58 |
| jogo | but a noticable spike in failures | 17:58 |
| jogo | fungi: I think that would be the right thing | 17:58 |
| jogo | not even sure how to do that | 17:58 |
| fungi | jogo: you already have a bug open for the failures. and increase in those failures doesn't necessarily require a new bug | 17:58 |
| clarkb | devananda: we use requiremetns for two things today. Building the pypi mirror and enforcing the requirements you list in your project. If you need the thing to be in your requirements list then you need it on the mirror which means it shoudl go in global requirements | 17:59 |
| jogo | fungi: well this one maps identically to a change in the kernel we use | 17:59 |
| *** weiwei has quit IRC | 17:59 | |
| jogo | anyway how do I add this to ubuntu kernel? | 17:59 |
| jogo | or libvirt | 17:59 |
| fungi | jogo: it's possible other things upgraded at the same time that the kernel did, so that's conjecture. but "here's something we noticed which increases the incidence of this bug" is potentially useful information to the bug which is happening more often now | 18:00 |
| *** amcrn has quit IRC | 18:00 | |
| clarkb | dtroyer: https://bugs.launchpad.net/openstack-ci/+bug/1282772 is that only a grenade bug or do we need to do something from an infra side? I will mark it incomplete as it is missing info about what infra can do to help | 18:00 |
| fungi | jogo: ubuntu and libvirt both use launchpad for their bugs, i believe | 18:00 |
| *** talluri has quit IRC | 18:00 | |
| *** jnoller has joined #openstack-infra | 18:00 | |
| *** sweston has joined #openstack-infra | 18:00 | |
| *** derekh has quit IRC | 18:00 | |
| *** esker has quit IRC | 18:01 | |
| dims_ | jamespage, i still see 1.2.1-0ubuntu5~cloud0 which was built on Feb 12 on precise-proposed/icehouse - wanted to find out when we could see a fresh update? | 18:01 |
| jogo | libvirt doesn't | 18:01 |
| jogo | what is the name of ubuntu's kernel group? | 18:01 |
| *** esker has joined #openstack-infra | 18:01 | |
| jogo | wendar or other ex Canonical folks? | 18:01 |
| jamespage | dims_, prob later this week as I pushed b3 out to proposed/updates | 18:01 |
| fungi | jogo: i guess the question i have is whether you think fixing the bug from the libvirt or nova side is going to cause it to not be fixed because it happens more often under a specific kernel version | 18:01 |
| jogo | fungi: not really sure, I do know that the timing is suspicious and the spike in in failures is bad | 18:02 |
| dims_ | jamespage, does b3 have this? https://bugs.launchpad.net/nova/+bug/1228977/comments/40 | 18:02 |
| jogo | dims_: ^^ | 18:02 |
| clarkb | jeblair: fungi: devananda: it wasn't clear to me from the above if I shouldn't mark that pyhgmi test thing as invalid in openstack-ci | 18:02 |
| clarkb | fungi: I think you were arguing we should keep it and jeblair was on board with marking it invalid | 18:03 |
| jogo | fungi: and want to get one of the ubuntu kernel folks to glance at this ideally | 18:03 |
| *** ryanpetrello has joined #openstack-infra | 18:03 | |
| jamespage | dims_, can you check the version in trusty development - that's what I'll land this week | 18:03 |
| fungi | clarkb: if we had a "pedandic and unnecessary, track your infra config changes in your own project not ours" status for bugs i'd say that would fit. we could just on policy start closing bugs people open against openstack-ci purely because they're planning to make a config change | 18:03 |
| dims_ | jamespage, unfortunately i can test only precise | 18:04 |
| jamespage | dims_, it should be 1.2.2 | 18:04 |
| *** johnthetubaguy has quit IRC | 18:04 | |
| *** johnthetubaguy has joined #openstack-infra | 18:04 | |
| *** zns has joined #openstack-infra | 18:05 | |
| clarkb | fungi: I want to say https://bugs.launchpad.net/openstack-ci/+bug/1282795 was something you dug into but couldn't find anything out of the ordinary? | 18:05 |
| fungi | jogo: probably best to get 1254872 brought to the attention of the ubuntu kernel packagers then and let them know that we're seeing it more often under the new kernel, but without knowing the underlying cause i'm still unconvinced that an increase in occurrence of a bug is a separate bug | 18:05 |
| jogo | jamespage: you work for cannonical right? ^ | 18:06 |
| *** esker has quit IRC | 18:06 | |
| *** mfer has joined #openstack-infra | 18:06 | |
| *** jamielennox|away is now known as jamielennox | 18:06 | |
| jamespage | jogo, yes | 18:07 |
| jogo | https://bugs.launchpad.net/bugs/1254872 is hurting us a lot more under the new .60 kernel | 18:07 |
| jogo | not sure who to pass that info onto perhaps you would know | 18:07 |
| jogo | jamespage: http://status.openstack.org/elastic-recheck/ | 18:07 |
| *** jgallard has quit IRC | 18:08 | |
| dims_ | jamespage, thanks for the info. please do ping me when you push 1.2.2 to precise. | 18:08 |
| jeblair | clarkb: i think you should just remove openstackci from the list of affected projects | 18:09 |
| jamespage | dims_, will do - just doing a vUDS session right now | 18:09 |
| fungi | clarkb: i simply couldn't fathom how 1282795 could happen since tox is a python program, it's complaining "InterpreterNotFound: python2.6" well into execution on centos, and that's the only python interpreter centos has (so tox should never have been able to run). i don't think i was able to catch one where the failure happened and log into it to look for the interpreter existing though | 18:09 |
| clarkb | jeblair: wfm | 18:10 |
| clarkb | fungi: :/ that is a fun one | 18:10 |
| clarkb | fungi: I will let you update the bug then as you are most familiar with it | 18:10 |
| fungi | clarkb: can do | 18:11 |
| *** weiwei has joined #openstack-infra | 18:11 | |
| *** jlibosva has quit IRC | 18:12 | |
| *** sabari2 has joined #openstack-infra | 18:13 | |
| clarkb | great thanks | 18:13 |
| jogo | jamespage: looks like 0.9.8 is buggy we should at least use 0.9.9 | 18:13 |
| jeblair | I'm typing " | 18:13 |
| jeblair | This is a real test failure, not an infrastructure bug. | 18:13 |
| jeblair | " a lot. | 18:13 |
| reed | jeblair, fungi: can either of you open port 5901 on activity.openstack.org? we need it for profiling | 18:13 |
| dims_ | jogo, even with jamespage 's 1.2.2 - we'll need at least one more fix from Daniel :( | 18:13 |
| clarkb | jeblair: :/ | 18:13 |
| jogo | dims_: so I see this as a two prong effort | 18:14 |
| jogo | get 1.2.2 working | 18:14 |
| jogo | and get 0.9x working a little better | 18:14 |
| dims_ | jogo, ack | 18:14 |
| jogo | I want to move to 1.2.2 in gate | 18:14 |
| jogo | but I also want 0.9.x working in the meantime | 18:14 |
| clarkb | jeblair: gah we are racing | 18:14 |
| jogo | sounds like danpb made headway on https://bugs.launchpad.net/nova/+bug/1254872 and things it just needs 0.9.9 | 18:15 |
| fungi | reed: i don't believe any of is have a login to activity.o.o? at least i don't seem to | 18:15 |
| jeblair | clarkb: oh, are you on new bugs too? | 18:15 |
| jogo | dims_: so not sure if you know who to ask to make that happen | 18:15 |
| clarkb | jeblair: ya but I think I mostly managed to avoid you :) | 18:15 |
| reed | fungi, can i do it from iptables? I thought it was a config panel thing on rackspace | 18:15 |
| clarkb | jeblair: I will let you finish | 18:16 |
| jeblair | ok | 18:16 |
| * clarkb goes back to older bugs | 18:16 | |
| *** mriedem has quit IRC | 18:16 | |
| fungi | reed: i don't think we use security zone configuration on rackspace, but maybe someone did on that one since it's not controlled by infra. i'll check | 18:16 |
| *** sabari3 has joined #openstack-infra | 18:17 | |
| clarkb | it should be via iptables | 18:17 |
| *** sabari2 has quit IRC | 18:17 | |
| *** andreaf has quit IRC | 18:17 | |
| clarkb | we manage firewalls that way rather than using security groups | 18:17 |
| reed | clarkb, thanks, I'll go check iptables | 18:18 |
| *** ildikov_ has quit IRC | 18:19 | |
| jeblair | clarkb, fungi: does this ring a bell? https://bugs.launchpad.net/openstack-ci/+bug/1282262 | 18:20 |
| reed | uhm ... I thought I had access to that machine | 18:21 |
| clarkb | jeblair: no, but I am reading logs for it now | 18:21 |
| jamespage | dims_, jogo - give me 20 mins and we can talk better | 18:21 |
| clarkb | which have purple links | 18:21 |
| clarkb | so I have read them before | 18:21 |
| dims_ | jamespage, thx | 18:21 |
| fungi | clarkb: jeblair: was this maybe around the time we did the tempest account split and had to do several iterations of sudo -u/chown patches to get things to be writeable by the right accounts? | 18:23 |
| clarkb | fungi: I bet that is it | 18:23 |
| jeblair | okay, cool, i'll mark it fixed released | 18:23 |
| *** Ajaeger1 has joined #openstack-infra | 18:23 | |
| jogo | jamespage: I need to go AFK for a a little but everything should be in bug 1254872 | 18:24 |
| *** dizquierdo has quit IRC | 18:25 | |
| fungi | reed: i'm not seeing anything obvious in the rackspace dashboard to imply that we've configured them to do any network filtering of that server | 18:26 |
| *** SumitNaiksatam_ has joined #openstack-infra | 18:27 | |
| reed | fungi, thanks, i'll have sanjiva check iptables himself then | 18:27 |
| *** SumitNaiksatam has quit IRC | 18:27 | |
| *** rcarrillocruz1 has joined #openstack-infra | 18:28 | |
| *** vhoward has left #openstack-infra | 18:29 | |
| *** rcarrillocruz has quit IRC | 18:29 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 18:29 | |
| *** jooools has quit IRC | 18:29 | |
| *** VijayTripathi has joined #openstack-infra | 18:30 | |
| *** jpich has quit IRC | 18:31 | |
| *** arborism has joined #openstack-infra | 18:33 | |
| *** dcramer__ has quit IRC | 18:33 | |
| *** mriedem has joined #openstack-infra | 18:34 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Replaced first_name and last_name with full_name. https://review.openstack.org/79720 | 18:34 |
| *** arborism is now known as amcrn | 18:34 | |
| *** khyati_ has quit IRC | 18:34 | |
| jeblair | fungi: we're done! | 18:37 |
| *** thuc has quit IRC | 18:37 | |
| fungi | well, you're done. i'm still triaging the last one, but yay! | 18:37 |
| *** thuc has joined #openstack-infra | 18:37 | |
| jeblair | fungi: i might have already done it | 18:38 |
| *** esker has joined #openstack-infra | 18:38 | |
| fungi | oh, so you did | 18:38 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Consume full_name rather than first_name and last_name https://review.openstack.org/79723 | 18:38 |
| *** johnthetubaguy has quit IRC | 18:39 | |
| fungi | i tried to obtain a lock on it by setting to triaged before i started working on it. i think that one's a duplicate, just trying to hunt down the original | 18:39 |
| jeblair | fungi: oops, sorry | 18:40 |
| *** thuc_ has joined #openstack-infra | 18:40 | |
| *** thuc_ has quit IRC | 18:40 | |
| fungi | no worries! | 18:41 |
| *** thuc has quit IRC | 18:41 | |
| jamespage | jogo, looking now | 18:41 |
| *** thuc has joined #openstack-infra | 18:41 | |
| clarkb | jeblair: https://bugs.launchpad.net/openstack-ci/+bug/1278210 may interest you | 18:42 |
| clarkb | and is https://bugs.launchpad.net/openstack-ci/+bug/1281320 the thing affecting doc jobs? | 18:42 |
| jeblair | clarkb: a fix to 320 would help the doc job situation | 18:44 |
| jeblair | clarkb: (though the root cause of that is now understood to be related to concurrent builds and jenkins+nodepool+gearman-plugin) | 18:44 |
| jeblair | clarkb: so fixing 320 would help mask the problem. | 18:45 |
| clarkb | interesting | 18:45 |
| jeblair | clarkb: fix for docs situation: https://review.openstack.org/#/c/79659/ | 18:45 |
| jeblair | clarkb: the actual problem: https://jenkins05.openstack.org/job/openstack-user-guide-admin/ | 18:45 |
| clarkb | jeblair: fungi SergeyLukjanov mordred https://bugs.launchpad.net/openstack-ci/+bug/1273283 is that fixable now? we are using single use slaves for all of the major testing | 18:45 |
| clarkb | jeblair: looking | 18:45 |
| *** freyes has quit IRC | 18:46 | |
| SergeyLukjanov | clarkb, but we still have proposal node | 18:46 |
| jeblair | SergeyLukjanov: we don't cancel any jobs that run on the long-running nodes | 18:46 |
| jamespage | dims_, which version of libvirt are you running in gate? the 1.1.1 from the Havana CA or the 0.9.8 from 12.04 base? | 18:46 |
| jeblair | SergeyLukjanov: and canceling jobs was what triggered that error | 18:47 |
| dims_ | james 0.9.8 | 18:47 |
| SergeyLukjanov | jeblair, oh, /me forget that it was because of cancelation... | 18:47 |
| dims_ | jamespage, 0.9.8 | 18:47 |
| *** dcramer__ has joined #openstack-infra | 18:47 | |
| jamespage | dims_, OK | 18:48 |
| dims_ | jamespage, 1.1.1 from Havana was bad because it had locking issues which Daniel fixed. i was able to validate his changes worked. | 18:48 |
| jeblair | clarkb: logstash graph is interesting. 32 workers are still online. | 18:48 |
| dims_ | jamespage, by building his changes straight from libvirt git | 18:49 |
| *** mkoderer has joined #openstack-infra | 18:49 | |
| jeblair | clarkb: (i haven't checked to see if they are doing anything) | 18:49 |
| jamespage | dims_, we can fix this stuff btw; but need to know :-) | 18:49 |
| *** JuanManuelOlle has left #openstack-infra | 18:49 | |
| jamespage | dims_, if there are commits that we can take and SRU that's great | 18:49 |
| jeblair | http://graphite.openstack.org/render/?from=-5hours&height=180&until=now&width=334&bgcolor=ffffff&fgcolor=000000&target=color%28alias%28stats.gauges.logstash.geard.queue.running,%20%27Running%27%29,%20%27blue%27%29&target=color%28alias%28stats.gauges.logstash.geard.queue.waiting,%20%27Waiting%27%29,%20%27red%27%29&target=color%28alias%28stats.gauges.logstash.geard.queue.total,%20%27Total%20Jobs%27%29,%20%27888888%27%29&target=color%28alias%2 | 18:49 |
| clarkb | jeblair: they appear to be doing stuff, I just refreshed a kibana saerch and have data | 18:49 |
| jamespage | dims_, so bug 1254872 is related to the 0.9.8 issue right? | 18:49 |
| sdague | jamespage: danpb has a pretty deep analysis, unfortunately I don't think he put it out to the list yet | 18:50 |
| * clarkb looks at cacti | 18:50 | |
| jeblair | clarkb: http://paste.openstack.org/show/73175/ | 18:50 |
| dims_ | jamespage, yes, so we tried to upgrade to precise-proposed and saw the stuff mentioned in here - https://bugs.launchpad.net/nova/+bug/1228977 | 18:51 |
| clarkb | jeblair: I have a graph similar to that, I use the one on zuul status but change the time to -2hours | 18:51 |
| dhellmann | mordred: just saw your ping, if you're still around let me know | 18:51 |
| jeblair | k | 18:51 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove concurrency restriction from all jobs https://review.openstack.org/79659 | 18:52 |
| dims_ | jamespage, so at this point we need 1.2.2 + Daniel's fix mentioned here (https://bugs.launchpad.net/nova/+bug/1228977/comments/40) | 18:52 |
| *** SumitNaiksatam has quit IRC | 18:52 | |
| clarkb | jeblair: I am going to look at numbers and see if moving to a third worker on each node is feasible | 18:53 |
| jamespage | dims_, OK - so hallyn is working on fixing up the 1.1.1 in proposed; once its in there I can push it out to havana-proposed | 18:53 |
| jeblair | clarkb: all 32 are active | 18:53 |
| jeblair | clarkb: so i think we're just right at the inflection point | 18:53 |
| dims_ | jamespage, you want to leave icehouse-proposed for 1.2.x? | 18:54 |
| sdague | jeblair / fungi: we keep not getting a nice reset point in the gate, however we get a lot of resets today. So I'm wondering if we could do a promote of 77815,2 anyway | 18:54 |
| jamespage | dims_, icehouse-proposed will get the 1.2.2 that's in trusty | 18:54 |
| jamespage | if that needs a patch then we'll get that sorted | 18:54 |
| jamespage | lemme check with hallyn | 18:54 |
| clarkb | jeblair: I agree. free shows that most nodes have between 1.8GB and 2GB of cached + free memory. I think that is plenty of room to run a third worker | 18:55 |
| clarkb | jeblair: what do you think? should I start up -C on all of those nodes? | 18:55 |
| dims_ | jamespage, that will then need a fix for this as well https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1283179 in addition to Daniels fix https://www.redhat.com/archives/libvir-list/2014-March/msg00501.html which was post 1.2.2 | 18:55 |
| fungi | sdague: i've seen the same cinder unit test mismatch error in the gate a couple times now... i think maybe a racy change must have slipped in on them | 18:56 |
| jeblair | clarkb: wfm; i assume it was libvirt that killed them all before | 18:56 |
| fungi | sdague: is that known already? | 18:56 |
| SergeyLukjanov | folks, have you seen the d-g failures with "ImageKilledException: Image 45a63b5f-9c6d-48e2-b203-de4bf2d2a3cf 'killed' while waiting for 'active'"? | 18:56 |
| clarkb | jeblair: yup I think so | 18:56 |
| dims_ | jamespage, sounds like a good plan 1.1.1 on precise/havana and 1.2.x on precise/icehouse | 18:56 |
| *** dcramer__ has quit IRC | 18:56 | |
| jamespage | dims_, we can't bump havana - that drives of saucy which is released so SRU only - we can cherry pick the fix tho | 18:57 |
| clarkb | jamespage: done | 18:57 |
| clarkb | gah jeblair ^ | 18:57 |
| *** hallyn has joined #openstack-infra | 18:57 | |
| jamespage | hey hallyn | 18:57 |
| clarkb | jamespage: dims_ 1.1.1 is horribly broken | 18:57 |
| sdague | fungi: I don't know | 18:57 |
| sdague | I was basically spending all my attention on getting gr back to sanity | 18:58 |
| sdague | because I consider this a giant hole in our system right now | 18:58 |
| jamespage | clarkb, I'm sure hallyn is doing his best to unbreak it | 18:58 |
| fungi | sdague: the main reason to promote 77815,2 would be if there were requirements changes ahead of it though, right? | 18:58 |
| clarkb | jamespage: I think danpb's patch for 1.2.2 is the fix | 18:58 |
| *** khyati_ has joined #openstack-infra | 18:58 | |
| clarkb | but /me defers to people that actually grok libvirt | 18:59 |
| sdague | fungi: the main reason is so we can actually test any of the requirements outstanding | 18:59 |
| sdague | which I'd like to be able to do in this work day to start getting results back | 18:59 |
| jamespage | hallyn, I was just discussing 1.2.2 in the context of bug 1228977 | 18:59 |
| sdague | especially the sqla one | 18:59 |
| jamespage | with dims_ | 18:59 |
| hallyn | jamespage: when i have a free moment i'll push the patch to 1.2.2 | 18:59 |
| jamespage | hallyn, thanks - much appreciated | 19:00 |
| clarkb | meeting time | 19:00 |
| jeblair | fungi, sdague: i think since it fixes a gating hole, it's okay to promote | 19:00 |
| jamespage | dims_, pending a few MIR's for Ubuntu trusty I hope to have b3 plus updated dependencies in the CA by the end of this week - I'll keep you updated | 19:00 |
| hallyn | for the precise version, i will push a package with the patch dberrange suggested; if that doesn't suffice i can look deeper | 19:00 |
| fungi | jeblair: okay, will do (even though it's not a hole for any of the changes ahead of it in the gate) | 19:00 |
| *** jolyonbrown has joined #openstack-infra | 19:00 | |
| jamespage | hallyn, +1 | 19:01 |
| dims_ | hallyn, yep, that will do | 19:01 |
| jamespage | ta | 19:01 |
| dims_ | hallyn, jamespage thanks! | 19:02 |
| jamespage | dims_, np | 19:02 |
| jamespage | dims_, please keep slinging stuff like this at us - need to know :-) | 19:02 |
| dims_ | jamespage, yep will do | 19:03 |
| *** rcarrillocruz has joined #openstack-infra | 19:04 | |
| *** gyee has quit IRC | 19:04 | |
| *** harlowja has quit IRC | 19:04 | |
| jamespage | dims_, you might find these reports useful btw | 19:05 |
| jamespage | http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/ | 19:05 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Replaced first_name and last_name with full_name. https://review.openstack.org/79720 | 19:05 |
| *** gyee has joined #openstack-infra | 19:05 | |
| dims_ | jamespage, very cool. thx | 19:06 |
| *** rcarrillocruz1 has quit IRC | 19:06 | |
| jogo | dims_ jamespage hallyn: thanks for looking into the libvirt stuff | 19:07 |
| *** yolanda has quit IRC | 19:08 | |
| *** harlowja has joined #openstack-infra | 19:08 | |
| jogo | any change of replacing 0.9.8 with 0.9.11 as well? | 19:08 |
| *** cosmogon has joined #openstack-infra | 19:08 | |
| adam_g | while we are discussing libvirt issues WRT the gate, i've hit a wall trying to use newer libvirt 1.1.1 (cloud-havana or saucy) on RAX: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1248025 | 19:09 |
| openstackgerrit | Paul Michali proposed a change to openstack/requirements: Add httmock to test-requirements. https://review.openstack.org/75296 | 19:09 |
| *** cosmogon has quit IRC | 19:09 | |
| adam_g | AFAIK a newer libvirt will eventually be required to get ironic tested there | 19:09 |
| clarkb | fungi: and for my last bug that I started before the meeting https://bugs.launchpad.net/openstack-ci/+bug/1266616 what do you think about my comments there? | 19:09 |
| fungi | clarkb: it's possible--do you have a link to a git bug about trying ipv6 directly rather than leaving it up to the tcp/ip stack on the system? | 19:11 |
| clarkb | fungi: I don't but I want to say I have seen that when I ran precise | 19:11 |
| dims_ | adam_g, yes i have a variant of that bug open as well https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1283179 | 19:11 |
| clarkb | fungi: I can certainly dig through the git change log | 19:11 |
| *** skraynev is now known as skraynev_afk | 19:12 | |
| dims_ | adam_g, yes, i had to nuke the offending shared objects to get the ones i needed working | 19:12 |
| *** waluta has joined #openstack-infra | 19:16 | |
| dims_ | jamespage, is the source tree for say 1.1.1-0ubuntu8.5~cloud0 is publicly available? | 19:16 |
| jamespage | dims_, always yes | 19:16 |
| *** gyee has quit IRC | 19:16 | |
| jamespage | dims_, apt-get source libvirt | 19:16 |
| jamespage | (of course you will have to be running on precise with the CA enabled to get that version) | 19:17 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add oslotest jobs for oslo.config https://review.openstack.org/76654 | 19:17 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add oslo.test integration test https://review.openstack.org/76381 | 19:17 |
| dims_ | jamespage, ah. was thinking of a git repo somewhere where i could switch around versions and view commit history etc. | 19:18 |
| jamespage | dims_, lp:ubuntu/saucy/libvirt might be up to date - but our package importer always seems to have challenges with libvirt for some reason | 19:18 |
| dims_ | jamespage, thanks again | 19:19 |
| jamespage | dims_, you can always retrieve specific versions - LP keeps a full history for the main Ubuntu archive | 19:19 |
| jamespage | and CA is just a backport of that | 19:19 |
| jamespage | dims_, pull-lp-source in ubuntu-dev-tools is your friend | 19:20 |
| jamespage | dims_, pull-lp-source libvirt saucy | 19:20 |
| jamespage | for example | 19:20 |
| jamespage | but you can also pull specific versions | 19:20 |
| jogo | jamespage: ping | 19:21 |
| jamespage | jogo, hello again | 19:21 |
| jogo | back, so libvirt 0.9.8 | 19:21 |
| jamespage | jogo | 19:21 |
| dhellmann | clarkb: I think ^^ addresses the comments you left on the review for the oslo.test/oslo.config cross-test job | 19:21 |
| jamespage | ok | 19:21 |
| jogo | although I am more interested in getting libvirt 1.x working it looks like libvirt 0.9.8 is broken for us and a fix was added in 0.9.8 | 19:22 |
| jamespage | jogo, hallyn is going to cherry pick the commit and SRU it | 19:22 |
| jogo | jamespage: oh sweet! | 19:22 |
| jamespage | jogo, we don't have a micro release exception for libvirt (not sure why) so we can't just bump it | 19:23 |
| jamespage | but hopefully that will fix it up | 19:23 |
| hallyn | would someone care to put a detailed test case in that bug? | 19:23 |
| jogo | that should help we have seen 381 failures in the past 14 days on this one | 19:23 |
| hallyn | i'm waiting on my test build to finish atm... | 19:23 |
| jogo | hallyn: its a race condition, and I am not sure how to make it fail 100% of the time | 19:23 |
| jogo | but our gate system does good job of hitting it | 19:23 |
| fungi | do we know how to consistently trigger that failure mode? | 19:23 |
| fungi | that's what i thoughth | 19:23 |
| *** waluta has quit IRC | 19:24 | |
| jogo | fungi: I don't, perhaps danpb does but he said its race | 19:24 |
| jamespage | jogo, I know its cheeky but could you point some gate nodes at the proposed packages and see if the failure rate drops? | 19:24 |
| jogo | which makessense | 19:24 |
| hallyn | jogo: 50% of the time would be sufficient :) | 19:24 |
| *** sarob_ has quit IRC | 19:24 | |
| jogo | jamespage I like that idea | 19:24 |
| jogo | fungi: ^ thoughts | 19:24 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Change operations-guide and openstack-manuals jobs to not use maven https://review.openstack.org/77446 | 19:24 |
| sdague | jogo: we'd have to build a whole new nodepool image | 19:24 |
| hallyn | ok i'll post debs as soon as they're built | 19:24 |
| sdague | and class | 19:25 |
| *** sarob_ has joined #openstack-infra | 19:25 | |
| fungi | jogo: i don't really know anything about the bug, was just asking because i got the impression we didn't know how to consistently reproduce it | 19:25 |
| jogo | fungi: correct we don't know how to | 19:25 |
| jogo | except we saw it 55 times in last 24 hours | 19:25 |
| Ajaeger1 | fungi, clarkb: So, for stable/havana and master the openstack projects do not use maven anymore with the above patch. | 19:26 |
| fungi | Ajaeger1: awesome! | 19:26 |
| jogo | sdague: this may be useful in the future for other race conditions like this | 19:26 |
| dims_ | hallyn, jamespage - "precise-proposed"? | 19:26 |
| *** pblaho has joined #openstack-infra | 19:26 | |
| *** SpamapS has quit IRC | 19:26 | |
| sdague | hallyn / jogo: so here is an excerpt from what danpb sent me just before he went to bed - http://paste.openstack.org/show/73178/ | 19:27 |
| sdague | I was hoping to get him to just stick the whole thing on the list, but given he's asleep, and you guys are chasing this, there you go | 19:27 |
| jogo | sdague: he put that in the bug | 19:27 |
| sdague | ok | 19:27 |
| jamespage | dims_, yes | 19:27 |
| *** bhuvan_ has quit IRC | 19:28 | |
| *** bhuvan has quit IRC | 19:28 | |
| Ajaeger1 | fungi: stable/grizzly still has maven but they will soon be obsolete. If you want me to change those as well, tell me and I work on getting our tools working with grizzly. I prefer to spend my time elsewhere ;) | 19:28 |
| Ajaeger1 | but if these grizzly jobs block you, tell me. | 19:29 |
| fungi | Ajaeger1: no, technically we were supposed to eol grizzly last thursday | 19:29 |
| *** SpamapS_ has joined #openstack-infra | 19:29 | |
| fungi | we're just behind on pulling the trigger there | 19:29 |
| *** sarob_ has quit IRC | 19:29 | |
| *** rcarrillocruz1 has joined #openstack-infra | 19:30 | |
| *** SpamapS_ is now known as SpamapS | 19:30 | |
| *** jp_at_hp has quit IRC | 19:30 | |
| hallyn | dims_: well i can just post them someplace for you to wget for a first test | 19:31 |
| Ajaeger1 | cool... | 19:31 |
| *** e0ne has joined #openstack-infra | 19:31 | |
| *** rcarrillocruz has quit IRC | 19:31 | |
| jogo | jamespage: so as for using OpenStack CI as a test for the patch, you would have to convince fungi and sdague that its worth it | 19:33 |
| dims_ | hallyn, y let me know where i can wget from as precise-updates seem to have 0.9.8-2ubuntu17.17 (looking at http://us.archive.ubuntu.com/ubuntu/dists/precise-updates/main/binary-i386/) | 19:33 |
| jogo | I like the idea, but not sure how much overhead it will involve | 19:33 |
| dims_ | jogo, no need, i can adapt my UCA patch to wget/dpkg | 19:33 |
| jamespage | jogo, that's ok - I appreciate it was a bit cheeky :-) | 19:33 |
| jamespage | jogo, we do need some sort of test case for the SRU tho | 19:33 |
| *** markmcclain has quit IRC | 19:34 | |
| *** jgrimm has quit IRC | 19:34 | |
| *** markmcclain has joined #openstack-infra | 19:34 | |
| jogo | dims_: that may work | 19:34 |
| clarkb | jogo: teh test wouldn't have to merge | 19:34 |
| clarkb | you can just push it | 19:34 |
| clarkb | let it run and done | 19:34 |
| hallyn | sdague: yes, taht's the commit i'm pushing | 19:34 |
| sdague | hallyn: great, thanks | 19:34 |
| jogo | clarkb: well it depends on how many runs we need ... but we can just babysit it | 19:34 |
| dims_ | jamespage, hallyn just drop me a note or ping me when i can wget 0.9.9 from a public location | 19:35 |
| *** malini is now known as malini_afk | 19:36 | |
| *** bhuvan has joined #openstack-infra | 19:37 | |
| *** bhuvan_ has joined #openstack-infra | 19:37 | |
| jogo | dims_: thanks again for doing all this | 19:37 |
| *** khyati_ has quit IRC | 19:37 | |
| *** amcrn has quit IRC | 19:38 | |
| sdague | jamespage: part of the problem is the repeat scenario includes our volume | 19:39 |
| sdague | so set up tests that create 50k guests in a day, you'll get 12 fails | 19:39 |
| jogo | clarkb: what was the fix for ES being backed up | 19:40 |
| *** VijayTripathi1 has joined #openstack-infra | 19:40 | |
| clarkb | jogo: we stopped archiving libvirt logs | 19:40 |
| *** sarob_ has joined #openstack-infra | 19:40 | |
| *** amcrn has joined #openstack-infra | 19:40 | |
| *** moted has quit IRC | 19:40 | |
| *** sarob_ has quit IRC | 19:41 | |
| fungi | and soon need to stop collecting/producing libvirt logs so the logserver doesn't fall over too | 19:41 |
| clarkb | fungi: that was the fix | 19:41 |
| *** moted has joined #openstack-infra | 19:41 | |
| clarkb | devstack-gate stopped copying them for us | 19:41 |
| clarkb | because 30MB * 10k tests a day is a lot of gigabytes | 19:41 |
| jogo | ouch | 19:42 |
| jogo | clarkb: so I will abandon my patch https://review.openstack.org/#/c/79535/ | 19:42 |
| fungi | clarkb: no, actually that fix failed to merge, and sdague proposed an alternative to stop indexing them instead | 19:42 |
| clarkb | fungi: oh so sdague did jogo's patch? | 19:42 |
| clarkb | fungi: we should've pushed the d-g change through for the reason above | 19:42 |
| *** bhuvan_ has quit IRC | 19:43 | |
| *** bhuvan has quit IRC | 19:43 | |
| sdague | clarkb: yes | 19:43 |
| *** lottosport has joined #openstack-infra | 19:43 | |
| *** VijayTripathi has quit IRC | 19:43 | |
| sdague | clarkb: so if you pushed that through, we never would have gotten any progress on this bug | 19:43 |
| sdague | we needed the enhanced logs for a reason | 19:43 |
| clarkb | sdague: right see my comment before | 19:43 |
| clarkb | there is far too much data to do anything useful with it | 19:43 |
| clarkb | we need a more targetted approach imo | 19:44 |
| sdague | in ES, for sure | 19:44 |
| clarkb | I don't think danpb needs 30MB compressed libvirt logs for every tempest run | 19:44 |
| sdague | clarkb: what do you propose for more targetted approach? | 19:44 |
| clarkb | sdague: prop up a change that gives you more log data, run it experimentally or recheck it | 19:45 |
| *** lukego has quit IRC | 19:45 | |
| clarkb | that also allows you to tweak things on the fly | 19:45 |
| sdague | clarkb: if we had a way to "recheck 50x" on a change, that would be fine | 19:45 |
| hallyn | precise compile seems to be taking longer than the trustyone did | 19:45 |
| *** dprince has quit IRC | 19:46 | |
| clarkb | sdague: right but instead of 50x we are doing 10kx | 19:46 |
| *** lottosport has quit IRC | 19:46 | |
| sdague | clarkb: yeh, because some of the failures turned out not to be useful in getting to the bottom of things | 19:46 |
| sdague | we turned this on for 24hrs, got enough data to analyze something, and now I can go and put a conditional around this | 19:47 |
| *** e0ne has quit IRC | 19:47 | |
| *** morganfainberg_Z is now known as morganfainberg | 19:48 | |
| clarkb | sdague: it just occurred to me that we could do recheck 50x talking direclty to gearman | 19:49 |
| *** rcarrillocruz has joined #openstack-infra | 19:49 | |
| clarkb | sdague: its not something anyone could do so not easy, but doable | 19:49 |
| sdague | we have definitely been in need of that in the past in patches we suspected would show or fix races | 19:50 |
| sdague | and the solution today has been have a developer go back an "recheck no bug" 10 times | 19:50 |
| sdague | when they remembered | 19:50 |
| jogo | sdague: we are ready to turn this off libvirt debug in gate? | 19:50 |
| sdague | over 4 days | 19:50 |
| clarkb | sdague: you only need about 5 rechecks | 19:50 |
| sdague | jogo: I think so, I'm going to work on a devstack fix | 19:50 |
| sdague | clarkb: depends on the bug | 19:50 |
| *** rcarrillocruz1 has quit IRC | 19:50 | |
| jogo | sdague: cool | 19:51 |
| *** bhuvan has joined #openstack-infra | 19:52 | |
| *** bhuvan_ has joined #openstack-infra | 19:52 | |
| *** zns has quit IRC | 19:53 | |
| jogo | sdague: and we want to turn libvirt indexing back on? | 19:53 |
| *** yassine has joined #openstack-infra | 19:53 | |
| sdague | jogo: honestly, I don't think so | 19:54 |
| sdague | this was the only bug that used it | 19:54 |
| jogo | true | 19:54 |
| sdague | and it didn't need to | 19:54 |
| sdague | the same line was in n-cpu | 19:54 |
| jogo | I added it originally thinking it would help give us abetter fingprint | 19:55 |
| jogo | but it didn't | 19:55 |
| *** sweston has quit IRC | 19:57 | |
| SergeyLukjanov | btw, http://gabrielecirulli.github.io/2048/ funny game | 19:57 |
| *** bhuvan_ has quit IRC | 19:59 | |
| *** bhuvan has quit IRC | 19:59 | |
| *** markmc has joined #openstack-infra | 19:59 | |
| *** pblaho has quit IRC | 20:01 | |
| pleia2 | morganfainberg: lmk when you have some time to chat about monitoring, I'm around all afternoon | 20:02 |
| *** jnoller has quit IRC | 20:02 | |
| pleia2 | speaking of being around, I'm visiting family next week in Maine, will still be working but will be on east coast time and will probably pop out for "lunch with mom" and "buy nephew more toys" here and there | 20:02 |
| *** mriedem1 has joined #openstack-infra | 20:02 | |
| SlickNik | hey guys | 20:03 |
| *** pdmars has quit IRC | 20:03 | |
| *** mriedem has quit IRC | 20:03 | |
| clarkb | pleia2: have fun | 20:03 |
| clarkb | SlickNik: hi | 20:03 |
| SlickNik | I had a question regarding openstack/requirements and trove. | 20:03 |
| *** mwagner_lap has joined #openstack-infra | 20:03 | |
| fungi | pleia2: which part of maine? portland area? | 20:03 |
| clarkb | SlickNik: go for it | 20:03 |
| fungi | pleia2: i do love to go there, though it's been a few years since my last visit | 20:04 |
| morganfainberg | pleia2, hi | 20:04 |
| pleia2 | fungi: the charming town of wilton (more central Maine than Portland, Portland is where I grew up though) | 20:04 |
| SlickNik | Today, trove's got 3 packages that exist as part of its requirements that are not in global-requirements. | 20:04 |
| pleia2 | morganfainberg: hey | 20:04 |
| morganfainberg | pleia2, i;m just getting to work, bad sleep schedule due to issues with a maintenance last week :P | 20:04 |
| fungi | pleia2: oh, i vaguely know where wilton is. you can't get there from here though | 20:04 |
| morganfainberg | pleia2, so give me a bit to get settled and i'm up for chatting | 20:04 |
| SlickNik | pexpect in requirements.txt; and wsgi_intercept and mockito in test_requirements.txt | 20:04 |
| morganfainberg | :P | 20:04 |
| pleia2 | morganfainberg: no worries, I'm going to grab lunch now anyway | 20:04 |
| morganfainberg | er, :) | 20:04 |
| morganfainberg | pleia2, good idea! | 20:05 |
| pleia2 | fungi: hehe, ayup | 20:05 |
| *** bhuvan has joined #openstack-infra | 20:05 | |
| *** bhuvan_ has joined #openstack-infra | 20:05 | |
| clarkb | SlickNik: you will need to get them into global requirements if appropriate or remove them as dependencies if inappropriate | 20:05 |
| clarkb | SlickNik: guessing that adding them isfine | 20:05 |
| *** wenlock has quit IRC | 20:05 | |
| clarkb | I need to grab lunch | 20:06 |
| * clarkb afks | 20:06 | |
| *** denis_makogon_ has joined #openstack-infra | 20:06 | |
| SlickNik | clarkb: Thanks! That's what I thought. I'll push up a patchset to global requirements to address this. Just wanted to clarify if there's some other "process" involved (or special incantation that I need to recite). | 20:07 |
| SlickNik | Cheers. | 20:07 |
| *** lukego has joined #openstack-infra | 20:09 | |
| *** smarcet has left #openstack-infra | 20:10 | |
| *** bhuvan_ has quit IRC | 20:11 | |
| *** bhuvan has quit IRC | 20:11 | |
| *** denis_makogon_ is now known as denis_makogon | 20:11 | |
| *** dstanek is now known as dstanek_afk | 20:12 | |
| *** vkozhukalov has quit IRC | 20:13 | |
| *** rcarrillocruz has quit IRC | 20:13 | |
| *** jolyonbrown has quit IRC | 20:14 | |
| *** zns has joined #openstack-infra | 20:15 | |
| *** zzelle has joined #openstack-infra | 20:16 | |
| *** rcarrillocruz has joined #openstack-infra | 20:16 | |
| *** hashar has joined #openstack-infra | 20:17 | |
| *** aysyd has joined #openstack-infra | 20:17 | |
| *** bhuvan has joined #openstack-infra | 20:18 | |
| *** bhuvan_ has joined #openstack-infra | 20:18 | |
| *** gyee has joined #openstack-infra | 20:18 | |
| *** mbacchi has quit IRC | 20:19 | |
| *** arosen has joined #openstack-infra | 20:19 | |
| *** mbacchi has joined #openstack-infra | 20:20 | |
| *** andreaf has joined #openstack-infra | 20:20 | |
| *** andreaf has quit IRC | 20:21 | |
| *** sweston has joined #openstack-infra | 20:22 | |
| openstackgerrit | Ziad Sawalha proposed a change to openstack/requirements: Add PEP-257 Testing Tools https://review.openstack.org/79054 | 20:22 |
| *** bhuvan__ has joined #openstack-infra | 20:23 | |
| *** bhuvan___ has joined #openstack-infra | 20:23 | |
| *** bhuvan_ has quit IRC | 20:24 | |
| *** bhuvan has quit IRC | 20:24 | |
| *** weshay has quit IRC | 20:26 | |
| *** rlandy has quit IRC | 20:27 | |
| fungi | pleia2: any objections to resolving https://launchpad.net/bugs/995592 ? at this point i've deleted the last of the untrusted static jenkins slaves, so i think the original need in that bug is solved (even if you're still working separately on being able to introduce fedora nodepool nodes, that seems like a separate project/bug at this point) | 20:28 |
| *** sweston has quit IRC | 20:30 | |
| *** rcarrillocruz has quit IRC | 20:31 | |
| openstackgerrit | SlickNik proposed a change to openstack/requirements: Updating global_requirements for trove https://review.openstack.org/79756 | 20:33 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 20:37 |
| *** khyati_ has joined #openstack-infra | 20:39 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 20:39 |
| *** julim has quit IRC | 20:40 | |
| *** mrda_away is now known as mrda | 20:41 | |
| *** lcestari has quit IRC | 20:42 | |
| *** jgrimm has joined #openstack-infra | 20:42 | |
| fungi | clarkb: jeblair: the combined work on graphite and logstash for job statistics means https://launchpad.net/bugs/1172391 is done, yeah? | 20:43 |
| *** adalbas has quit IRC | 20:44 | |
| jeblair | i think the idea there was to see like "is nova pegging the cpu" or something... | 20:44 |
| fungi | ahh, got it | 20:44 |
| jeblair | though we have something in devstack-gate doing that now... | 20:44 |
| fungi | so actually something like getting the sar data we currently archive into a more queryable system? | 20:45 |
| jeblair | sar, that's it | 20:45 |
| jeblair | sdague was looking at that recently too | 20:45 |
| fungi | or is the fact that we can look at the sar report files individually enough for this? | 20:45 |
| sdague | jeblair: so we dumped sar for dstat in devstack | 20:45 |
| fungi | righth that | 20:45 |
| fungi | much more better sar | 20:45 |
| jeblair | sdague: is there more work needing to be done to make that useful? | 20:45 |
| sdague | jeblair: it's useful on it's own now | 20:46 |
| sdague | I have some ideas to be able to get us a little more data on the state of the world with dstat plugins | 20:46 |
| sdague | which you can write in python | 20:46 |
| fungi | just wondering whether that bug should be closed if we've got what we need | 20:46 |
| sdague | what bug is this? | 20:46 |
| fungi | or if we really need fancier queryable aggregate data for devstack performance | 20:46 |
| fungi | sdague: https://launchpad.net/bugs/1172391 | 20:47 |
| *** mwagner_lap has quit IRC | 20:47 | |
| sdague | yeh, I think dstat is probably sufficient at this point | 20:47 |
| sdague | that seemed to get us what we needed to make the # of testr workers decision | 20:48 |
| fungi | sdague: thanks. i'll consider you a primary stakeholder for that bug | 20:48 |
| fungi | fix released then | 20:48 |
| sdague | so I'd call that closed | 20:48 |
| sdague | yep | 20:48 |
| *** mark0n has joined #openstack-infra | 20:49 | |
| fungi | jeblair: any reason we might still want to move forward with https://launchpad.net/bugs/1172401 or is the automated rechecking plus whatever fun comes in via new gerrit enough? | 20:50 |
| jeblair | fungi: i lean toward the latter | 20:50 |
| *** SumitNaiksatam has joined #openstack-infra | 20:51 | |
| *** amcrn has quit IRC | 20:51 | |
| fungi | jeblair: me too. i'll mark it fix released | 20:51 |
| *** sarob_ has joined #openstack-infra | 20:51 | |
| mark0n | hi, anybody familiar with Jenkins Job Builder? I'd like to clone two upstream Git repos into two branches of the same repo in the working directory. This is easy with the Jenkins web front-end but I didn't find a way to do this with JJB. Any ideas? | 20:52 |
| *** freyes has joined #openstack-infra | 20:52 | |
| *** arborism has joined #openstack-infra | 20:53 | |
| mark0n | from the web front-end I don't need to use MultiSCM but just the normal Git plugin to accomplish that | 20:54 |
| *** lcestari has joined #openstack-infra | 20:55 | |
| *** bhuvan has joined #openstack-infra | 20:58 | |
| *** bhuvan_ has joined #openstack-infra | 20:58 | |
| *** rfolco has quit IRC | 20:59 | |
| *** melwitt has quit IRC | 21:00 | |
| *** melwitt has joined #openstack-infra | 21:00 | |
| fungi | pleia2: do you want to pass the torch on https://launchpad.net/bugs/1218631 to nibalizer and let him stick a status update in there when he has time? | 21:00 |
| *** SumitNaiksatam has quit IRC | 21:00 | |
| *** e0ne has joined #openstack-infra | 21:00 | |
| *** markwash has joined #openstack-infra | 21:01 | |
| fungi | mark0n: we don't really use much in the way of jenkins plugins, and don't use the git plugin at all, so not sure what to suggest there (we just run a shell builder which does a git clone/clean/reset as needed before running the job) | 21:01 |
| *** bhuvan___ has quit IRC | 21:02 | |
| *** bhuvan__ has quit IRC | 21:02 | |
| *** bhuvan has quit IRC | 21:03 | |
| *** bhuvan has joined #openstack-infra | 21:03 | |
| *** bhuvan__ has joined #openstack-infra | 21:03 | |
| *** bhuvan_ has quit IRC | 21:03 | |
| mark0n | fungi: do you think it makes sense to try the openstack-infra mailing list? | 21:04 |
| afazekas | mark0n: I wonder isn't enough to create two - git: entry with different 'branches:' and 'basedir:' ? | 21:04 |
| dolphm | jeblair: before we play tennis with this bug, it does look like an -infra bug to me, but very low impact https://bugs.launchpad.net/keystone/+bug/1284579 | 21:05 |
| *** sabari3 has left #openstack-infra | 21:05 | |
| jeblair | dolphm: oh, maybe a devstack bug then | 21:06 |
| jeblair | dolphm: oh, no, because of a network error between the host and pypi? | 21:07 |
| jeblair | ok | 21:07 |
| jeblair | dolphm: i'll take it back and mark it wontfix, thanks. | 21:08 |
| dolphm | jeblair: that's what it looks like to me -- dogpile looks to be the first python dep not in /var/cache | 21:08 |
| fungi | mark0n: possibly. there are some jjb users who subscribe to that but may not be around in irc right this moment who might use that plugin in other environments | 21:08 |
| *** jeckersb is now known as jeckersb_gone | 21:08 | |
| mark0n | afazekas: I tried that, but as soon as there are two "git:" entries, JJB generates XML code that requires MultiSCM and I didn't manage to clone into two branches of the _same_ repo with MultiSCM | 21:08 |
| fungi | dolphm: jeblair: i want to say we already have at least one bug open on "Cannot fetch index base URL http://pypi.openstack.org/openstack/" so probably better as a duplicate | 21:09 |
| * fungi needs to skedaddle to dinner... back later this evening | 21:10 | |
| clarkb | mark0n: it would probably help to grab the xml jenkins generates and compare to the xml jjb generates | 21:10 |
| clarkb | mark0n: then we can convince jjb to do the right thing | 21:11 |
| *** jeckersb_gone is now known as jeckersb | 21:13 | |
| nibalizer | fungi: updated | 21:16 |
| *** e0ne has quit IRC | 21:16 | |
| mark0n | the ultimate goal is to copy a branch of a Github repo to a branch in our company repo. With the web front-end I just configured two remotes and pushed to one of them (emty build section). | 21:16 |
| *** e0ne has joined #openstack-infra | 21:16 | |
| *** jgriffith has quit IRC | 21:19 | |
| *** mailforjob has joined #openstack-infra | 21:20 | |
| *** jgriffith has joined #openstack-infra | 21:23 | |
| *** mailforjob has quit IRC | 21:24 | |
| *** StevenK_ has joined #openstack-infra | 21:25 | |
| *** HenryG has quit IRC | 21:28 | |
| *** markwash has quit IRC | 21:28 | |
| *** harlowja has quit IRC | 21:28 | |
| *** zzelle has quit IRC | 21:28 | |
| *** weiwei has quit IRC | 21:28 | |
| *** nati_ueno has quit IRC | 21:28 | |
| *** alexpilotti has quit IRC | 21:28 | |
| *** jungleboyj has quit IRC | 21:28 | |
| *** mkerrin has quit IRC | 21:28 | |
| *** sdake_ has quit IRC | 21:28 | |
| *** sandywalsh has quit IRC | 21:28 | |
| *** matrohon has quit IRC | 21:28 | |
| *** rwsu has quit IRC | 21:28 | |
| *** salv-orlando has quit IRC | 21:28 | |
| *** melwitt has quit IRC | 21:28 | |
| *** lcestari has quit IRC | 21:28 | |
| *** sarob_ has quit IRC | 21:28 | |
| *** mbacchi has quit IRC | 21:28 | |
| *** aysyd has quit IRC | 21:28 | |
| *** yassine has quit IRC | 21:28 | |
| *** moted has quit IRC | 21:28 | |
| *** mkoderer has quit IRC | 21:28 | |
| *** lcheng has quit IRC | 21:28 | |
| *** mgagne has quit IRC | 21:28 | |
| *** atiwari has quit IRC | 21:28 | |
| *** dims_ has quit IRC | 21:28 | |
| *** spredzy has quit IRC | 21:28 | |
| *** skraynev_afk has quit IRC | 21:28 | |
| *** luis_ has quit IRC | 21:28 | |
| *** Mithrandir has quit IRC | 21:28 | |
| *** ianw has quit IRC | 21:28 | |
| *** dstufft has quit IRC | 21:28 | |
| *** homeless has quit IRC | 21:28 | |
| *** pfallenop has quit IRC | 21:28 | |
| *** dkehn_ has quit IRC | 21:28 | |
| *** amrith has quit IRC | 21:28 | |
| *** jd__ has quit IRC | 21:28 | |
| *** jamespage has quit IRC | 21:28 | |
| *** tteggel has quit IRC | 21:28 | |
| *** EntropyWorks has quit IRC | 21:28 | |
| *** annegentle has quit IRC | 21:28 | |
| *** dansmith has quit IRC | 21:28 | |
| *** rainya has quit IRC | 21:28 | |
| *** stevebaker has quit IRC | 21:28 | |
| *** dragondm has quit IRC | 21:28 | |
| *** tristanC has quit IRC | 21:28 | |
| *** bookwar has quit IRC | 21:28 | |
| *** zhiyan_ has quit IRC | 21:28 | |
| *** wayneeseguin has quit IRC | 21:28 | |
| *** tnurlygayanov has quit IRC | 21:28 | |
| *** bhuvan__ has quit IRC | 21:28 | |
| *** bhuvan has quit IRC | 21:28 | |
| *** jgrimm has quit IRC | 21:28 | |
| *** gyee has quit IRC | 21:28 | |
| *** zns has quit IRC | 21:28 | |
| *** mriedem1 has quit IRC | 21:28 | |
| *** nicedice has quit IRC | 21:28 | |
| *** wchrisj has quit IRC | 21:28 | |
| *** mrodden has quit IRC | 21:28 | |
| *** ArxCruz has quit IRC | 21:28 | |
| *** julienvey has quit IRC | 21:28 | |
| *** antonym has quit IRC | 21:28 | |
| *** bknudson has quit IRC | 21:28 | |
| *** mrda has quit IRC | 21:28 | |
| *** JoshNang has quit IRC | 21:28 | |
| *** simonmcc has quit IRC | 21:28 | |
| *** dangers_away has quit IRC | 21:28 | |
| *** loquacities has quit IRC | 21:28 | |
| *** nibalizer has quit IRC | 21:28 | |
| *** sdague has quit IRC | 21:28 | |
| *** Alex_Gaynor has quit IRC | 21:28 | |
| *** asadoughi has quit IRC | 21:28 | |
| *** cyeoh has quit IRC | 21:28 | |
| *** shortstop has quit IRC | 21:28 | |
| *** denis_makogon has quit IRC | 21:28 | |
| *** markmc has quit IRC | 21:28 | |
| *** VijayTripathi1 has quit IRC | 21:28 | |
| *** mfer has quit IRC | 21:28 | |
| *** david-lyle has quit IRC | 21:28 | |
| *** caleb_ has quit IRC | 21:28 | |
| *** CaptTofu has quit IRC | 21:28 | |
| *** reed has quit IRC | 21:28 | |
| *** dkranz has quit IRC | 21:28 | |
| *** pcrews has quit IRC | 21:28 | |
| *** yamahata has quit IRC | 21:28 | |
| *** lcostantino has quit IRC | 21:28 | |
| *** dripton has quit IRC | 21:28 | |
| *** cody-somerville has quit IRC | 21:28 | |
| *** AJaeger has quit IRC | 21:28 | |
| *** chandan_kumar has quit IRC | 21:28 | |
| *** mtreinish has quit IRC | 21:28 | |
| *** esmute has quit IRC | 21:28 | |
| *** Kiall has quit IRC | 21:28 | |
| *** ilyashakhat has quit IRC | 21:28 | |
| *** vipul has quit IRC | 21:28 | |
| *** saschpe has quit IRC | 21:28 | |
| *** sbadia has quit IRC | 21:28 | |
| *** dkehn has quit IRC | 21:28 | |
| *** dhellmann has quit IRC | 21:28 | |
| *** dachary has quit IRC | 21:28 | |
| *** saper has quit IRC | 21:28 | |
| *** morgabra has quit IRC | 21:28 | |
| *** gaelL_ has quit IRC | 21:28 | |
| *** maurosr has quit IRC | 21:28 | |
| *** Shrews has quit IRC | 21:28 | |
| *** sahumada has quit IRC | 21:28 | |
| *** clarkb has quit IRC | 21:28 | |
| *** kevinbenton has quit IRC | 21:28 | |
| *** kmartin has quit IRC | 21:28 | |
| *** e0ne has quit IRC | 21:28 | |
| *** arborism has quit IRC | 21:28 | |
| *** hashar has quit IRC | 21:28 | |
| *** lukego has quit IRC | 21:28 | |
| *** ryanpetrello has quit IRC | 21:28 | |
| *** marun has quit IRC | 21:28 | |
| *** afazekas has quit IRC | 21:28 | |
| *** Ryan_Lane has quit IRC | 21:28 | |
| *** blamar has quit IRC | 21:28 | |
| *** changbl has quit IRC | 21:28 | |
| *** unicell has quit IRC | 21:28 | |
| *** vponomaryov has quit IRC | 21:28 | |
| *** pete5 has quit IRC | 21:28 | |
| *** gmoro has quit IRC | 21:28 | |
| *** juice has quit IRC | 21:28 | |
| *** davidlenwell has quit IRC | 21:28 | |
| *** AaronGreen has quit IRC | 21:28 | |
| *** mattoliverau has quit IRC | 21:28 | |
| *** jaypipes has quit IRC | 21:28 | |
| *** kashyap has quit IRC | 21:28 | |
| *** medieval1 has quit IRC | 21:28 | |
| *** yamahata_ has quit IRC | 21:28 | |
| *** jamielennox has quit IRC | 21:28 | |
| *** flaper87 has quit IRC | 21:28 | |
| *** bradm has quit IRC | 21:28 | |
| *** koolhead17 has quit IRC | 21:28 | |
| *** cp16net has quit IRC | 21:28 | |
| *** locke105 has quit IRC | 21:28 | |
| *** freyes has quit IRC | 21:28 | |
| *** khyati_ has quit IRC | 21:28 | |
| *** arosen has quit IRC | 21:28 | |
| *** Ajaeger1 has quit IRC | 21:28 | |
| *** thedodd has quit IRC | 21:28 | |
| *** bada has quit IRC | 21:28 | |
| *** rossella_s has quit IRC | 21:28 | |
| *** hogepodge has quit IRC | 21:28 | |
| *** Hefeweizen has quit IRC | 21:28 | |
| *** dstanek_afk has quit IRC | 21:28 | |
| *** jeblair has quit IRC | 21:28 | |
| *** openstackgerrit has quit IRC | 21:28 | |
| *** jpeeler has quit IRC | 21:28 | |
| *** jeremyb has quit IRC | 21:28 | |
| *** soren has quit IRC | 21:28 | |
| *** gema has quit IRC | 21:28 | |
| *** alaski has quit IRC | 21:28 | |
| *** NikitaKonovalov has quit IRC | 21:28 | |
| *** michchap has quit IRC | 21:28 | |
| *** westmaas has quit IRC | 21:28 | |
| *** DuncanT- has quit IRC | 21:28 | |
| *** SergeyLukjanov has quit IRC | 21:28 | |
| *** ruhe has quit IRC | 21:28 | |
| *** DinaBelova has quit IRC | 21:28 | |
| *** ogelbukh has quit IRC | 21:28 | |
| *** jasond` has quit IRC | 21:28 | |
| *** gothicmindfood has quit IRC | 21:28 | |
| *** notmyname has quit IRC | 21:28 | |
| *** pleia2 has quit IRC | 21:28 | |
| *** comstud has quit IRC | 21:28 | |
| *** Vivek has quit IRC | 21:28 | |
| *** niska has quit IRC | 21:28 | |
| *** samalba has quit IRC | 21:28 | |
| *** geekinutah has quit IRC | 21:28 | |
| *** rahmu has quit IRC | 21:28 | |
| *** plomakin_ has quit IRC | 21:28 | |
| *** rdwrer has quit IRC | 21:28 | |
| *** dosaboy has quit IRC | 21:28 | |
| *** vladan has quit IRC | 21:28 | |
| *** spiffxp_ has quit IRC | 21:28 | |
| *** jcooley has quit IRC | 21:28 | |
| *** belliott has quit IRC | 21:28 | |
| *** adarazs_ has quit IRC | 21:28 | |
| *** jogo has quit IRC | 21:28 | |
| *** guitarzan has quit IRC | 21:28 | |
| *** mburned has quit IRC | 21:28 | |
| *** aude has quit IRC | 21:28 | |
| *** Steap has quit IRC | 21:28 | |
| *** gilliard has quit IRC | 21:28 | |
| *** lifeless has quit IRC | 21:28 | |
| *** mika has quit IRC | 21:28 | |
| *** jesusaurus has quit IRC | 21:28 | |
| *** primeministerp has quit IRC | 21:28 | |
| *** gpocentek has quit IRC | 21:28 | |
| *** mordred has quit IRC | 21:28 | |
| *** ihrachys has quit IRC | 21:28 | |
| *** dtroyer has quit IRC | 21:28 | |
| *** Daviey has quit IRC | 21:28 | |
| *** markmcclain has quit IRC | 21:28 | |
| *** dolphm has quit IRC | 21:28 | |
| *** greghaynes has quit IRC | 21:28 | |
| *** pabelanger has quit IRC | 21:28 | |
| *** sileht has quit IRC | 21:28 | |
| *** boris-42 has quit IRC | 21:28 | |
| *** Adri2000 has quit IRC | 21:28 | |
| *** StevenK has quit IRC | 21:28 | |
| *** JayF has quit IRC | 21:28 | |
| *** EmilienM has quit IRC | 21:28 | |
| *** chmouel has quit IRC | 21:29 | |
| *** jroll has quit IRC | 21:29 | |
| *** pvo has quit IRC | 21:29 | |
| *** Guest34377 has quit IRC | 21:29 | |
| *** wendar has quit IRC | 21:29 | |
| *** jbryce has quit IRC | 21:29 | |
| *** russellb has quit IRC | 21:29 | |
| *** jlk has quit IRC | 21:29 | |
| *** adam_g has quit IRC | 21:29 | |
| *** ttx has quit IRC | 21:29 | |
| *** gondoi has quit IRC | 21:29 | |
| *** msuriar has quit IRC | 21:29 | |
| *** mikal has quit IRC | 21:29 | |
| *** jeckersb has quit IRC | 21:29 | |
| *** hughsaunders has quit IRC | 21:29 | |
| *** isviridov has quit IRC | 21:29 | |
| *** fbo_away has quit IRC | 21:29 | |
| *** ChanServ has quit IRC | 21:29 | |
| *** morganfainberg has quit IRC | 21:29 | |
| *** mark0n has quit IRC | 21:29 | |
| *** rms_13 has quit IRC | 21:29 | |
| *** psedlak has quit IRC | 21:29 | |
| *** Ng has quit IRC | 21:29 | |
| *** ekarlso has quit IRC | 21:29 | |
| *** Hunner has quit IRC | 21:29 | |
| *** bodepd has quit IRC | 21:29 | |
| *** zaro has quit IRC | 21:29 | |
| *** malini_afk has quit IRC | 21:29 | |
| *** bnemec has quit IRC | 21:29 | |
| *** therve has quit IRC | 21:29 | |
| *** anteaya has quit IRC | 21:29 | |
| *** tru_tru has quit IRC | 21:29 | |
| *** tchaypo has quit IRC | 21:29 | |
| *** jomara has quit IRC | 21:29 | |
| *** portante has quit IRC | 21:29 | |
| *** hdd has quit IRC | 21:29 | |
| *** hub_cap has quit IRC | 21:29 | |
| *** phschwartz has quit IRC | 21:29 | |
| *** med_ has quit IRC | 21:29 | |
| *** shardy has quit IRC | 21:29 | |
| *** NobodyCam has quit IRC | 21:29 | |
| *** devananda has quit IRC | 21:29 | |
| *** SlickNik has quit IRC | 21:29 | |
| *** gondoi_ has joined #openstack-infra | 21:29 | |
| *** gondoi_ is now known as gondoi | 21:29 | |
| *** pabelanger_ has joined #openstack-infra | 21:29 | |
| *** jeckersb has joined #openstack-infra | 21:29 | |
| *** krotscheck has quit IRC | 21:30 | |
| *** sileht has joined #openstack-infra | 21:30 | |
| *** Adri2000 has joined #openstack-infra | 21:30 | |
| *** krotscheck has joined #openstack-infra | 21:32 | |
| *** pabelanger_ is now known as pabelanger | 21:34 | |
| *** krotscheck has quit IRC | 21:35 | |
| *** zhiyan_ has joined #openstack-infra | 21:38 | |
| *** yassine has joined #openstack-infra | 21:38 | |
| *** lcheng has joined #openstack-infra | 21:38 | |
| *** atiwari has joined #openstack-infra | 21:38 | |
| *** jd__ has joined #openstack-infra | 21:38 | |
| *** jd__ is now known as 23LAAONZW | 21:41 | |
| *** chmouel has joined #openstack-infra | 21:41 | |
| *** wayneeseguin has joined #openstack-infra | 21:41 | |
| *** melwitt has joined #openstack-infra | 21:41 | |
| *** homeless has joined #openstack-infra | 21:41 | |
| *** EntropyWorks has joined #openstack-infra | 21:41 | |
| *** wendar has joined #openstack-infra | 21:41 | |
| *** jbryce has joined #openstack-infra | 21:41 | |
| *** bookwar has joined #openstack-infra | 21:41 | |
| *** dansmith has joined #openstack-infra | 21:41 | |
| *** russellb has joined #openstack-infra | 21:41 | |
| *** pvo has joined #openstack-infra | 21:41 | |
| *** tteggel has joined #openstack-infra | 21:41 | |
| *** skraynev_afk has joined #openstack-infra | 21:41 | |
| *** dims_ has joined #openstack-infra | 21:41 | |
| *** sarob has joined #openstack-infra | 21:41 | |
| *** jroll has joined #openstack-infra | 21:41 | |
| *** luis_ has joined #openstack-infra | 21:41 | |
| *** dkehn_ has joined #openstack-infra | 21:41 | |
| *** amrith has joined #openstack-infra | 21:41 | |
| *** ianw has joined #openstack-infra | 21:41 | |
| *** mbacchi has joined #openstack-infra | 21:41 | |
| *** aysyd has joined #openstack-infra | 21:41 | |
| *** moted has joined #openstack-infra | 21:41 | |
| *** mgagne has joined #openstack-infra | 21:41 | |
| *** mkoderer has joined #openstack-infra | 21:41 | |
| *** tristanC has joined #openstack-infra | 21:41 | |
| *** Guest96737 has joined #openstack-infra | 21:41 | |
| *** jlk has joined #openstack-infra | 21:41 | |
| *** spredzy has joined #openstack-infra | 21:41 | |
| *** annegentle has joined #openstack-infra | 21:41 | |
| *** adam_g has joined #openstack-infra | 21:41 | |
| *** stevebaker has joined #openstack-infra | 21:41 | |
| *** ttx has joined #openstack-infra | 21:41 | |
| *** pfallenop has joined #openstack-infra | 21:41 | |
| *** ociuhandu has joined #openstack-infra | 21:41 | |
| *** krotscheck_ has joined #openstack-infra | 21:41 | |
| *** isviridov has joined #openstack-infra | 21:41 | |
| *** greghaynes has joined #openstack-infra | 21:41 | |
| *** sandywalsh_ has joined #openstack-infra | 21:41 | |
| *** hughsaunders has joined #openstack-infra | 21:41 | |
| *** HenryG_ has joined #openstack-infra | 21:41 | |
| *** dolphm has joined #openstack-infra | 21:41 | |
| *** fbo has joined #openstack-infra | 21:41 | |
| *** EmilienM has joined #openstack-infra | 21:41 | |
| *** jooools has joined #openstack-infra | 21:41 | |
| *** derekh has joined #openstack-infra | 21:41 | |
| *** boris-42 has joined #openstack-infra | 21:41 | |
| *** mikal has joined #openstack-infra | 21:41 | |
| *** JayF has joined #openstack-infra | 21:41 | |
| *** msuriar_ has joined #openstack-infra | 21:41 | |
| *** markmcclain1 has joined #openstack-infra | 21:41 | |
| *** rpodolyaka has joined #openstack-infra | 21:41 | |
| *** weshay has joined #openstack-infra | 21:41 | |
| *** ildikov_ has joined #openstack-infra | 21:41 | |
| *** mriedem has joined #openstack-infra | 21:41 | |
| *** e0ne has joined #openstack-infra | 21:41 | |
| *** arborism has joined #openstack-infra | 21:41 | |
| *** freyes has joined #openstack-infra | 21:41 | |
| *** jgrimm has joined #openstack-infra | 21:41 | |
| *** khyati_ has joined #openstack-infra | 21:41 | |
| *** arosen has joined #openstack-infra | 21:41 | |
| *** gyee has joined #openstack-infra | 21:41 | |
| *** hashar has joined #openstack-infra | 21:41 | |
| *** zns has joined #openstack-infra | 21:41 | |
| *** lukego has joined #openstack-infra | 21:41 | |
| *** denis_makogon has joined #openstack-infra | 21:41 | |
| *** markmc has joined #openstack-infra | 21:41 | |
| *** VijayTripathi1 has joined #openstack-infra | 21:41 | |
| *** Ajaeger1 has joined #openstack-infra | 21:41 | |
| *** ryanpetrello has joined #openstack-infra | 21:41 | |
| *** david-lyle has joined #openstack-infra | 21:41 | |
| *** marun has joined #openstack-infra | 21:41 | |
| *** caleb_ has joined #openstack-infra | 21:41 | |
| *** rms_13 has joined #openstack-infra | 21:41 | |
| *** afazekas has joined #openstack-infra | 21:41 | |
| *** CaptTofu has joined #openstack-infra | 21:41 | |
| *** nicedice has joined #openstack-infra | 21:41 | |
| *** Ryan_Lane has joined #openstack-infra | 21:41 | |
| *** wchrisj has joined #openstack-infra | 21:41 | |
| *** blamar has joined #openstack-infra | 21:41 | |
| *** thedodd has joined #openstack-infra | 21:41 | |
| *** reed has joined #openstack-infra | 21:41 | |
| *** pcrews has joined #openstack-infra | 21:41 | |
| *** changbl has joined #openstack-infra | 21:41 | |
| *** mrodden has joined #openstack-infra | 21:41 | |
| *** bada has joined #openstack-infra | 21:41 | |
| *** yamahata has joined #openstack-infra | 21:41 | |
| *** ArxCruz has joined #openstack-infra | 21:41 | |
| *** unicell has joined #openstack-infra | 21:41 | |
| *** julienvey has joined #openstack-infra | 21:41 | |
| *** vponomaryov has joined #openstack-infra | 21:41 | |
| *** rossella_s has joined #openstack-infra | 21:41 | |
| *** kashyap has joined #openstack-infra | 21:41 | |
| *** dripton has joined #openstack-infra | 21:41 | |
| *** antonym has joined #openstack-infra | 21:41 | |
| *** bknudson has joined #openstack-infra | 21:41 | |
| *** pete5 has joined #openstack-infra | 21:41 | |
| *** hogepodge has joined #openstack-infra | 21:41 | |
| *** gmoro has joined #openstack-infra | 21:41 | |
| *** Hefeweizen has joined #openstack-infra | 21:41 | |
| *** cody-somerville has joined #openstack-infra | 21:41 | |
| *** juice has joined #openstack-infra | 21:41 | |
| *** davidlenwell has joined #openstack-infra | 21:41 | |
| *** AaronGreen has joined #openstack-infra | 21:41 | |
| *** AJaeger has joined #openstack-infra | 21:41 | |
| *** mattoliverau has joined #openstack-infra | 21:41 | |
| *** chandan_kumar has joined #openstack-infra | 21:41 | |
| *** mtreinish has joined #openstack-infra | 21:41 | |
| *** jaypipes has joined #openstack-infra | 21:41 | |
| *** mrda has joined #openstack-infra | 21:41 | |
| *** JoshNang has joined #openstack-infra | 21:41 | |
| *** simonmcc has joined #openstack-infra | 21:41 | |
| *** dangers_away has joined #openstack-infra | 21:41 | |
| *** dstanek_afk has joined #openstack-infra | 21:41 | |
| *** psedlak has joined #openstack-infra | 21:41 | |
| *** loquacities has joined #openstack-infra | 21:41 | |
| *** jeblair has joined #openstack-infra | 21:41 | |
| *** nibalizer has joined #openstack-infra | 21:41 | |
| *** esmute has joined #openstack-infra | 21:41 | |
| *** openstackgerrit has joined #openstack-infra | 21:41 | |
| *** jpeeler has joined #openstack-infra | 21:41 | |
| *** medieval1 has joined #openstack-infra | 21:41 | |
| *** sdague has joined #openstack-infra | 21:41 | |
| *** Alex_Gaynor has joined #openstack-infra | 21:41 | |
| *** shortstop has joined #openstack-infra | 21:41 | |
| *** cyeoh has joined #openstack-infra | 21:41 | |
| *** asadoughi has joined #openstack-infra | 21:41 | |
| *** jeremyb has joined #openstack-infra | 21:41 | |
| *** ilyashakhat has joined #openstack-infra | 21:41 | |
| *** vipul has joined #openstack-infra | 21:41 | |
| *** soren has joined #openstack-infra | 21:41 | |
| *** saschpe has joined #openstack-infra | 21:41 | |
| *** yamahata_ has joined #openstack-infra | 21:41 | |
| *** sbadia has joined #openstack-infra | 21:41 | |
| *** dkehn has joined #openstack-infra | 21:41 | |
| *** dhellmann has joined #openstack-infra | 21:41 | |
| *** jamielennox has joined #openstack-infra | 21:41 | |
| *** gema has joined #openstack-infra | 21:41 | |
| *** flaper87 has joined #openstack-infra | 21:41 | |
| *** dachary has joined #openstack-infra | 21:41 | |
| *** saper has joined #openstack-infra | 21:41 | |
| *** Ng has joined #openstack-infra | 21:41 | |
| *** morgabra has joined #openstack-infra | 21:41 | |
| *** malini_afk has joined #openstack-infra | 21:41 | |
| *** jomara has joined #openstack-infra | 21:41 | |
| *** bradm has joined #openstack-infra | 21:41 | |
| *** gaelL_ has joined #openstack-infra | 21:41 | |
| *** maurosr has joined #openstack-infra | 21:41 | |
| *** alaski has joined #openstack-infra | 21:41 | |
| *** Shrews has joined #openstack-infra | 21:41 | |
| *** NikitaKonovalov has joined #openstack-infra | 21:41 | |
| *** ekarlso has joined #openstack-infra | 21:41 | |
| *** gilliard has joined #openstack-infra | 21:41 | |
| *** michchap has joined #openstack-infra | 21:41 | |
| *** sahumada has joined #openstack-infra | 21:41 | |
| *** clarkb has joined #openstack-infra | 21:41 | |
| *** koolhead17 has joined #openstack-infra | 21:41 | |
| *** westmaas has joined #openstack-infra | 21:41 | |
| *** kevinbenton has joined #openstack-infra | 21:41 | |
| *** DuncanT- has joined #openstack-infra | 21:41 | |
| *** kmartin has joined #openstack-infra | 21:41 | |
| *** SergeyLukjanov has joined #openstack-infra | 21:41 | |
| *** ruhe has joined #openstack-infra | 21:41 | |
| *** DinaBelova has joined #openstack-infra | 21:41 | |
| *** Hunner has joined #openstack-infra | 21:41 | |
| *** ogelbukh has joined #openstack-infra | 21:41 | |
| *** bodepd has joined #openstack-infra | 21:41 | |
| *** zaro has joined #openstack-infra | 21:41 | |
| *** jasond` has joined #openstack-infra | 21:41 | |
| *** gothicmindfood has joined #openstack-infra | 21:41 | |
| *** notmyname has joined #openstack-infra | 21:41 | |
| *** pleia2 has joined #openstack-infra | 21:41 | |
| *** comstud has joined #openstack-infra | 21:41 | |
| *** Vivek has joined #openstack-infra | 21:41 | |
| *** niska has joined #openstack-infra | 21:41 | |
| *** samalba has joined #openstack-infra | 21:41 | |
| *** geekinutah has joined #openstack-infra | 21:41 | |
| *** bnemec has joined #openstack-infra | 21:41 | |
| *** therve has joined #openstack-infra | 21:41 | |
| *** rdwrer has joined #openstack-infra | 21:41 | |
| *** cp16net has joined #openstack-infra | 21:41 | |
| *** phschwartz has joined #openstack-infra | 21:41 | |
| *** locke105 has joined #openstack-infra | 21:41 | |
| *** Steap has joined #openstack-infra | 21:41 | |
| *** rahmu has joined #openstack-infra | 21:41 | |
| *** plomakin_ has joined #openstack-infra | 21:41 | |
| *** dosaboy has joined #openstack-infra | 21:41 | |
| *** tchaypo has joined #openstack-infra | 21:41 | |
| *** vladan has joined #openstack-infra | 21:41 | |
| *** spiffxp_ has joined #openstack-infra | 21:41 | |
| *** belliott has joined #openstack-infra | 21:41 | |
| *** jcooley has joined #openstack-infra | 21:41 | |
| *** adarazs_ has joined #openstack-infra | 21:41 | |
| *** jogo has joined #openstack-infra | 21:41 | |
| *** guitarzan has joined #openstack-infra | 21:41 | |
| *** mburned_out has joined #openstack-infra | 21:41 | |
| *** anteaya has joined #openstack-infra | 21:41 | |
| *** tru_tru has joined #openstack-infra | 21:41 | |
| *** aude has joined #openstack-infra | 21:41 | |
| *** portante has joined #openstack-infra | 21:41 | |
| *** lifeless has joined #openstack-infra | 21:41 | |
| *** mika has joined #openstack-infra | 21:41 | |
| *** jesusaurus has joined #openstack-infra | 21:41 | |
| *** primeministerp has joined #openstack-infra | 21:41 | |
| *** gpocentek has joined #openstack-infra | 21:41 | |
| *** mordred has joined #openstack-infra | 21:41 | |
| *** ihrachys has joined #openstack-infra | 21:41 | |
| *** Daviey has joined #openstack-infra | 21:41 | |
| *** dtroyer has joined #openstack-infra | 21:41 | |
| *** hdd has joined #openstack-infra | 21:41 | |
| *** hub_cap has joined #openstack-infra | 21:41 | |
| *** med_ has joined #openstack-infra | 21:41 | |
| *** morganfainberg has joined #openstack-infra | 21:41 | |
| *** ChanServ has joined #openstack-infra | 21:41 | |
| *** shardy has joined #openstack-infra | 21:41 | |
| *** NobodyCam has joined #openstack-infra | 21:41 | |
| *** devananda has joined #openstack-infra | 21:41 | |
| *** SlickNik has joined #openstack-infra | 21:41 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 21:41 | |
| *** rainya has joined #openstack-infra | 21:41 | |
| *** jeckersb has quit IRC | 21:41 | |
| *** jeckersb has joined #openstack-infra | 21:41 | |
| *** atiwari has quit IRC | 21:41 | |
| *** atiwari has joined #openstack-infra | 21:41 | |
| clarkb | afazekas: does `ssh -p29418 review.openstack.org gerrit ls-projects` work for you? that is a cheap way to test ssh connectivity | 21:41 |
| *** bhuvan has joined #openstack-infra | 21:41 | |
| *** bhuvan has quit IRC | 21:41 | |
| *** bhuvan has joined #openstack-infra | 21:41 | |
| clarkb | that works for me and the web UI seems to work too | 21:41 |
| *** pabelanger has quit IRC | 21:41 | |
| *** pabelanger has joined #openstack-infra | 21:41 | |
| *** 23LAAONZW is now known as jd__` | 21:41 | |
| *** jhesketh_ has joined #openstack-infra | 21:41 | |
| *** kiall has joined #openstack-infra | 21:42 | |
| *** Mithrandir has joined #openstack-infra | 21:42 | |
| *** Adri2000 is now known as Guest68283 | 21:42 | |
| *** jd__` is now known as jd__ | 21:42 | |
| *** krotscheck_ is now known as krotscheck | 21:42 | |
| *** dragondm has joined #openstack-infra | 21:42 | |
| *** dkliban has quit IRC | 21:42 | |
| jhesketh_ | Morning | 21:42 |
| *** nicedice_ has joined #openstack-infra | 21:44 | |
| *** mriedem has quit IRC | 21:44 | |
| *** mbacchi has quit IRC | 21:45 | |
| *** harlowja has joined #openstack-infra | 21:45 | |
| *** matrohon has joined #openstack-infra | 21:45 | |
| afazekas | clarkb: the same issue | 21:45 |
| *** sdake_ has joined #openstack-infra | 21:46 | |
| clarkb | afazekas: I think it must be a connectivity thing. what does `mtr review.openstack.org` show | 21:46 |
| *** bhuvan_ has joined #openstack-infra | 21:46 | |
| *** rwsu has joined #openstack-infra | 21:46 | |
| pleia2 | clarkb: I seem to recall you being involved with the translations stuff, it's on our priorities list, what on our end do we need to move this forward? | 21:46 |
| clarkb | pleia2: context? I think translation stuff is moving | 21:47 |
| afazekas | clarkb: I have connection, I got data in both direction, than it stops working in select(2) | 21:47 |
| clarkb | afazekas: huh | 21:47 |
| *** nati_ueno has joined #openstack-infra | 21:47 | |
| pleia2 | clarkb: looking at our https://etherpad.openstack.org/p/infrastructure-priorities and trying to find something to work on (tripleo stuff is winding down for me now that things are coming together) | 21:47 |
| *** jerryz has joined #openstack-infra | 21:48 | |
| clarkb | fungi: https://review.openstack.org/#/c/75213/ do we want to try pushing that through to help rax pypy nodes? | 21:48 |
| *** nicedice has quit IRC | 21:48 | |
| clarkb | pleia2: ah right those translation tools. I think we need to swing around with the translation folks again and see if there is some consensus on a tool that is open | 21:48 |
| *** dstufft has joined #openstack-infra | 21:49 | |
| clarkb | pleia2: that was a havana session with a few strong opinions :) then once we know what we need to be running spin up a server for that | 21:49 |
| clarkb | with puppeting and things | 21:49 |
| clarkb | pleia2: jeblair already has a dev pootle server but ryan_lane had people convinced we should look at the mediawiki related thing too | 21:49 |
| pleia2 | clarkb: yeah, just saw it on our prioirties list and wondered if there was something I could do there | 21:50 |
| pleia2 | like, look at mediawiki things or something | 21:50 |
| *** freyes has quit IRC | 21:50 | |
| clarkb | pleia2: ya that probably wouldn't be terrible also checking with daisy to see if they did any evaluation on their side whcih I thought they said they would during the summit | 21:50 |
| *** markmcclain1 has quit IRC | 21:51 | |
| pleia2 | the translations folks seem happy to continue to use tranxifex, in spite of proprietaryness, so I think we'll need to nudge them | 21:51 |
| pleia2 | clarkb: ok, I can follow up with daisy then | 21:51 |
| *** ryanpetrello has quit IRC | 21:51 | |
| clarkb | right, nudging is probably necessary | 21:51 |
| *** mrodden has quit IRC | 21:51 | |
| *** tnurlygayanov has joined #openstack-infra | 21:52 | |
| clarkb | did we lose fungi and jeblair in the splits? | 21:52 |
| *** Ajaeger1 has quit IRC | 21:53 | |
| mattoliverau | phew reading scroll back takes a while when you join at the end of the US day :P So bug day.. is there anything I can help with, now that I'm back from Rackspace international trips? | 21:53 |
| clarkb | mattoliverau: we are currently working through https://etherpad.openstack.org/p/cibugreview-march2014 updating bugs as appropriate | 21:54 |
| clarkb | mattoliverau: always good to update bugs assigned to you, and if you have questions comments about other bugs feel free to bring them up | 21:54 |
| mattoliverau | Cools, time to start reading then :) | 21:55 |
| sdague | clarkb: https://review.openstack.org/79784 | 21:55 |
| sdague | sorry, it was an irc meeting afternoon | 21:55 |
| sdague | that will turn the logs back to sanity by default | 21:56 |
| clarkb | sdague: whats with trueorfalse instead of using the variable with default value as on the previous line? | 21:57 |
| sdague | because it coalleses everything into true or false in a nice way | 21:57 |
| *** e0ne has quit IRC | 21:57 | |
| sdague | regardless of the value provided | 21:57 |
| *** markmc has quit IRC | 21:57 | |
| clarkb | gotcha so 0 1 True false yes no get normalized? | 21:58 |
| sdague | yeh | 21:58 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews https://review.openstack.org/79280 | 21:59 |
| sdague | anyway, going to dinner, but back on from hackerspace later. So I'll make sure that moves through the gate tonight | 21:59 |
| clarkb | sdague: awesome thank you, +1 from me | 21:59 |
| *** alexandra has joined #openstack-infra | 21:59 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 22:00 |
| *** ryanpetrello has joined #openstack-infra | 22:00 | |
| jeblair | clarkb, pleia2: yeah, i think checking with daisy is the next step | 22:00 |
| jeblair | pleia2: thanks for picking that up, that would be really helpful | 22:01 |
| *** hashar has quit IRC | 22:01 | |
| clarkb | jeblair: https://bugs.launchpad.net/openstack-ci/+bug/1240065 is the bug similar to what I saw with zuul mergers right? I think it might be beneficial to add the sigusr2 handler to statusbot to help debug. What do you think? | 22:02 |
| pleia2 | jeblair: sure thing, translation tools are interesting to me anyway so it seemed like a good fit | 22:03 |
| *** weiwei has joined #openstack-infra | 22:03 | |
| jeblair | clarkb: afaik all known statusbot issues should be resolved; that is, once we merge some changes and get it running again | 22:03 |
| *** sarob has quit IRC | 22:03 | |
| clarkb | jeblair: gotcha | 22:03 |
| jeblair | clarkb: maybe there will be more once we run it | 22:04 |
| jeblair | speaking of which, having it running tomorrow morning would be really nice | 22:04 |
| clarkb | :) I will leave that bug as is for now until we get it running again | 22:04 |
| *** sarob has joined #openstack-infra | 22:04 | |
| *** bhuvan_ has quit IRC | 22:04 | |
| * jeblair merges some statusbot changes | 22:04 | |
| jerryz | clarkb: is there a delay for the gerrit refs to be synced to git mirrors? now my gerrit triggered jobs can not get the refs from git.openstack.org | 22:05 |
| *** bhuvan has quit IRC | 22:05 | |
| clarkb | jerryz: yes there is a delay | 22:05 |
| clarkb | gerrit has to push changes out to the replicas after they happen | 22:05 |
| *** ryanpetrello has quit IRC | 22:05 | |
| *** bhuvan has joined #openstack-infra | 22:06 | |
| *** bhuvan_ has joined #openstack-infra | 22:06 | |
| jeblair | jerryz: best solutions would be to either use zuul, or sleep and retry from git.o.o | 22:06 |
| jerryz | clarkb: if the delay is so bad, i guess it is the same for zuul refs. how openstack jenkins jobs can get zuul refs from git.openstack.org instantly? | 22:06 |
| jeblair | jerryz: (those are better than having a lot of jobs hit gerrit directly) | 22:06 |
| jeblair | jerryz: they get refs from zuul mergers, and the mergers get them from gerrit | 22:07 |
| *** jooools has quit IRC | 22:07 | |
| jeblair | jerryz: so it's one fetch to get it to the merger, then hundreds of fetches from the workers hit the mergers instead of gerrit | 22:07 |
| jeblair | jerryz: so most workers (a) clone from a local cache or git.o.o; (b) update the repo to current state from git.o.o; (c) fetch changes to test from zuul merger | 22:08 |
| *** arosen has quit IRC | 22:08 | |
| clarkb | pleia2: jeblair https://bugs.launchpad.net/openstack-ci/+bug/1257964 that was completed around LCA right? | 22:08 |
| clarkb | iirc you guys sat in an airport and did that | 22:09 |
| *** sarob has quit IRC | 22:09 | |
| *** arosen has joined #openstack-infra | 22:09 | |
| jeblair | clarkb: i think so; if pleia2 confirms, we can delete the openstack-ci/publications repo | 22:09 |
| *** ociuhandu has quit IRC | 22:09 | |
| jeblair | then close the bug | 22:10 |
| clarkb | woot | 22:10 |
| *** StevenK_ is now known as StevenK | 22:10 | |
| pleia2 | clarkb: ah, yes! I can close | 22:10 |
| clarkb | pleia2: I think jeblair wants the old repo deleted first. I will let him do that as he is most familiar with it at this point | 22:11 |
| jeblair | i'm doing it right now | 22:11 |
| clarkb | \o/ | 22:11 |
| pleia2 | ok, closing | 22:11 |
| pleia2 | thanks jeblair | 22:11 |
| *** lukego has quit IRC | 22:11 | |
| jeblair | poof | 22:12 |
| clarkb | man github is slow today | 22:13 |
| jeblair | i just got a tcp rst from it | 22:13 |
| pleia2 | heh | 22:13 |
| clarkb | jeblair: I would like to close https://bugs.launchpad.net/openstack-ci/+bug/1260983 but there is an upstream scp plugin patch that needs merging in github | 22:13 |
| clarkb | jeblair: do you want to keep out bug open as a reminder to fix upstream? | 22:13 |
| jeblair | clarkb: are we running that patch in prod? | 22:14 |
| clarkb | jeblair: yes | 22:14 |
| jeblair | jeblair merged commit 7ce951d into jenkinsci:master from zaro0508:fix-FileNotFound a month ago | 22:14 |
| jeblair | James E. Blair jeblair closed this a month ago | 22:14 |
| clarkb | though it may not be necessary anymore with the downgrade | 22:14 |
| jeblair | clarkb: ^ | 22:15 |
| clarkb | jeblair: great, I was going to check that but github... | 22:15 |
| clarkb | jeblair: I am closing our bug now thanks | 22:15 |
| jeblair | cool | 22:15 |
| clarkb | releasestatus is CD right? I think we can close https://bugs.launchpad.net/openstack-ci/+bug/1262891 | 22:17 |
| *** salv-orlando has joined #openstack-infra | 22:18 | |
| jeblair | clarkb: i think so | 22:18 |
| pleia2 | clarkb: I think ttx still manages that himself, so.. probably | 22:18 |
| * clarkb investigates | 22:18 | |
| pleia2 | it's on our list to move to us | 22:18 |
| jerryz | jeblair: merger is in zuul, right? workers get the code base from git.o.o and changes from zuul mergers. but i can't find where devstack code references zuul merger. i can only see a GIT_BASE as remote | 22:18 |
| *** alexandra is now known as alex-backinbed | 22:19 | |
| jeblair | pleia2: there's a whole module for it in puppet | 22:19 |
| pleia2 | jeblair: oh, maybe I'm thinking of something else then :\ | 22:19 |
| clarkb | yup its on status.o.o running via puppte and I have confirmed that the commit to fix it is there | 22:19 |
| *** flaper87 is now known as flaper87|afk | 22:19 | |
| jeblair | jerryz: yes, merger is in zuul; devstack-gate handles the repo setup for us, and it doesn't reference zuul directly, it uses the value of ZUUL_URL to get the merger to fetch from | 22:19 |
| clarkb | I have marked it fix released | 22:20 |
| clarkb | well when I don't get a timeout from lp I will do that | 22:20 |
| *** mrodden has joined #openstack-infra | 22:22 | |
| openstackgerrit | A change was merged to openstack-infra/statusbot: Handle lots of channels better https://review.openstack.org/79034 | 22:24 |
| *** enikanorov_ has quit IRC | 22:24 | |
| openstackgerrit | A change was merged to openstack-infra/statusbot: Support SSL https://review.openstack.org/79047 | 22:24 |
| *** yassine has quit IRC | 22:24 | |
| *** mrodden1 has joined #openstack-infra | 22:24 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Make zuul-merger default pidfile match init script https://review.openstack.org/79109 | 22:24 |
| clarkb | oh I had intended on reading 79047 since I suggested it... | 22:24 |
| * clarkb does that now | 22:24 | |
| clarkb | nice and simple | 22:25 |
| *** dizquierdo has joined #openstack-infra | 22:25 | |
| *** afazekas has quit IRC | 22:26 | |
| *** denis_makogon has quit IRC | 22:26 | |
| harlowja | arg, stop breaking github people, lol | 22:27 |
| *** mrodden has quit IRC | 22:27 | |
| clarkb | harlowja: we don't use github | 22:27 |
| clarkb | so someone else broke it | 22:27 |
| harlowja | i know ;) | 22:27 |
| harlowja | ha | 22:27 |
| *** rossella_s has quit IRC | 22:27 | |
| harlowja | to much pushing off openstack code to github | 22:27 |
| harlowja | overflowed the tubes | 22:27 |
| jogo | clarkb: where is the code that generates '[SCP] Trying to create /srv/static/logs/61/79661/1/gate/gate-tempest-dsvm-postgres-full/b7c0133' | 22:29 |
| jogo | that log msg | 22:29 |
| jogo | I was thinking of adding a link to the logs.o.o location in there | 22:29 |
| jogo | to make my life easier | 22:29 |
| clarkb | jogo: in the jenkins scp plugin | 22:29 |
| *** dims_ has quit IRC | 22:30 | |
| jogo | clarkb: ohh thats not our code directly? | 22:30 |
| jeblair | jogo: there's supposed to be a link there already due to jjb | 22:30 |
| *** andreaf has joined #openstack-infra | 22:31 | |
| jeblair | jogo: 2014-03-11 21:25:35.915 | Detailed logs: http://logs.openstack.org/49/78749/4/check/check-tempest-dsvm-full/c72a6ab/ | 22:31 |
| hallyn | libvirt build keeps hanging on make check... trying on another box. | 22:31 |
| jeblair | jogo: from https://jenkins01.openstack.org/job/check-tempest-dsvm-full/10481/console | 22:31 |
| clarkb | jeblair: I marked https://bugs.launchpad.net/openstack-ci/+bug/1270288 fix released and assigned it to you | 22:31 |
| jogo | jeblair: I didn't see that link in https://jenkins01.openstack.org/job/gate-tempest-dsvm-postgres-full/8289/console | 22:32 |
| clarkb | because it failed and jenkins short circuits | 22:32 |
| clarkb | we could put that shell blob first | 22:32 |
| clarkb | and link at the beginning rather than the end | 22:32 |
| jeblair | clarkb: not a bad idea | 22:33 |
| jeblair | jogo: we'll have more control over that kind of output when we move logs to swift (because of the way the log upload will work) | 22:34 |
| jogo | jeblair: woot to swift, but in the meantime I like clarkbs idea | 22:34 |
| jeblair | ++ | 22:34 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add support for Fedora 20 to nodepool https://review.openstack.org/79789 | 22:35 |
| pleia2 | dooh | 22:35 |
| jogo | clarkb: do I just need to move the link-logs builder before the shell one? | 22:35 |
| * pleia2 lost change-id | 22:35 | |
| clarkb | jeblair: fungi https://bugs.launchpad.net/openstack-ci/+bug/1270321 I seem to briefly remember what happened there but zuul supports modern statsd now so whatever | 22:35 |
| clarkb | jogo: yup | 22:35 |
| clarkb | JJB creates them in order | 22:35 |
| jogo | clarkb: patch coming soon | 22:35 |
| clarkb | and jenkins respects that | 22:35 |
| * jogo likes how things in ifra do what you think they do | 22:35 | |
| jeblair | clarkb: wfm | 22:36 |
| *** Sukhdev has joined #openstack-infra | 22:36 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add support for Fedora 20 to nodepool https://review.openstack.org/69510 | 22:36 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Ensure there's always an http server object https://review.openstack.org/79792 | 22:37 |
| jeblair | clarkb: that fixes the nondeterministic zuul test error seen in your pidfile change | 22:38 |
| clarkb | jeblair: woot /me reviews | 22:38 |
| *** thuc has quit IRC | 22:39 | |
| jogo | clarkb: should link-logs go before or after devstack-checkout | 22:39 |
| jogo | I was going with after, but didn't put much thought into it | 22:39 |
| *** thuc has joined #openstack-infra | 22:40 | |
| clarkb | jeblair: https://bugs.launchpad.net/openstack-ci/+bug/1289432 is that related to the gear patch you had questions about? | 22:40 |
| jeblair | jogo: maybe before -- make it the first thing in every job? | 22:40 |
| clarkb | jogo: jeblair ya first thing will make it easier to find | 22:40 |
| jeblair | clarkb: re gear: i don't _think_ so... | 22:42 |
| clarkb | fungi: https://review.openstack.org/#/c/71323/ was tested with manage projects on Friday | 22:43 |
| clarkb | fungi: it seems to have worked well, but we didn't test any of the other jeepyb commands. Could you review that an possibly approve if you think it is safe? | 22:43 |
| *** yamahata has quit IRC | 22:43 | |
| clarkb | fungi: mostly looking for an opinion on whether or not we should do more testing before approval happens | 22:43 |
| clarkb | mordred: https://review.openstack.org/#/c/77311/ needs some love | 22:44 |
| *** openstackstatus has joined #openstack-infra | 22:44 | |
| jeblair | statusbot is joining... | 22:44 |
| *** thuc has quit IRC | 22:44 | |
| *** ociuhandu has joined #openstack-infra | 22:45 | |
| *** dims_ has joined #openstack-infra | 22:45 | |
| jeblair | so watching the statusbot debug log is like reading all of the openstack channels simultaneously... | 22:46 |
| jeblair | i mean, not like it... it is it. | 22:46 |
| clarkb | jeblair: :) | 22:46 |
| jeblair | it's actually much less of a firehose right now than i expected. | 22:47 |
| *** wchrisj has quit IRC | 22:48 | |
| reed | if you don't mind, can you please move your comments into another column or into comments so I can do the sums? | 22:48 |
| *** jhesketh__ has joined #openstack-infra | 22:48 | |
| reed | doh :) | 22:48 |
| reed | previous message is a decoy, to confuse spies | 22:49 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: run link-logs builder first https://review.openstack.org/79794 | 22:49 |
| * jeblair moves his comments | 22:49 | |
| jogo | clarkb jeblair: I think that should do it | 22:49 |
| *** CaptTofu has quit IRC | 22:51 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Update the keyring dependency https://review.openstack.org/79796 | 22:51 |
| *** arborism is now known as amcrn | 22:55 | |
| hallyn | jamespage: http://people.canonical.com/~serge/libvirt-0.9.8-2ubuntu18 | 22:56 |
| jeblair | fungi: should we work on prep for tomorrow? | 22:56 |
| hallyn | dims_: sdague: ^ | 22:56 |
| *** melwitt has quit IRC | 22:58 | |
| *** jeckersb is now known as jeckersb_gone | 22:58 | |
| *** che-arne has joined #openstack-infra | 22:58 | |
| *** changbl has quit IRC | 22:59 | |
| *** alex-backinbed is now known as Alexandra | 22:59 | |
| *** thuc has joined #openstack-infra | 23:01 | |
| *** thuc has quit IRC | 23:01 | |
| *** thuc has joined #openstack-infra | 23:02 | |
| *** mgagne has quit IRC | 23:02 | |
| *** ArxCruz has quit IRC | 23:03 | |
| *** jungleboyj has joined #openstack-infra | 23:05 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews https://review.openstack.org/79280 | 23:06 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 23:06 |
| clarkb | dtroyer: Alex_Gaynor jnoller etc https://bugs.launchpad.net/openstack-ci/+bug/1272598 who should be in the openstacksdk-ptl group? Would be great to get that bug closed | 23:07 |
| clarkb | because yay bug day | 23:07 |
| clarkb | pleia2: what do you do to get bug numbers? would be curious what they look like now compared to the numbers on the ehterpad | 23:08 |
| *** david-lyle has quit IRC | 23:08 | |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Add the kite (KDS) project and kiteclient project to gerrit/jenkins https://review.openstack.org/73074 | 23:08 |
| *** rpodolyaka has quit IRC | 23:09 | |
| pleia2 | clarkb: I wrote this script to gather everything: https://github.com/pleia2/openstack-infra-scripts/blob/master/infra_bugday.py | 23:10 |
| dims_ | hallyn, thanks will give it a whirl | 23:10 |
| pleia2 | clarkb: grabbing the whole bug link, which includes the number | 23:10 |
| *** caleb_ has quit IRC | 23:10 | |
| pleia2 | it's probably something like task.number though | 23:10 |
| pleia2 | clarkb: ooh wait, I know what you're saying | 23:11 |
| pleia2 | clarkb: right hand side of https://bugs.launchpad.net/openstack-ci/ | 23:11 |
| clarkb | pleia2: I am confused :) I just want to see post bug day stats. Since we seem to be winding down | 23:11 |
| clarkb | pleia2: thanks | 23:11 |
| pleia2 | I will post them to the list too, and on my blog to the planet | 23:11 |
| *** rpodolyaka has joined #openstack-infra | 23:12 | |
| *** thuc has quit IRC | 23:12 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Add tox.ini, setup.py and do syntax cleanup https://review.openstack.org/79800 | 23:13 |
| pleia2 | clarkb: we did very well today :) | 23:14 |
| jeblair | hopefully the next bug day will involve storyboard | 23:14 |
| *** thedodd has quit IRC | 23:15 | |
| SlickNik | Here's the requirements patch that I was mentioning earlier: https://review.openstack.org/#/c/79756/ | 23:16 |
| SlickNik | clarkb / other infra folks, please take a look when you get a chance.Thanks! | 23:16 |
| clarkb | SlickNik: you actually want requirements core to look at it :) | 23:16 |
| clarkb | pleia2: we did? yay | 23:16 |
| SlickNik | clarkb: Oh, I didn't realize there was a separate requirements-core. :) | 23:17 |
| jogo | SergeyLukjanov: it loosk like savanna patches are all failing | 23:17 |
| jogo | Bug 1257885 - horizon does not work with django 1.6 | 23:18 |
| jogo | all hits are in savanna jobs | 23:18 |
| *** che-arne has quit IRC | 23:20 | |
| *** andreaf has quit IRC | 23:21 | |
| * fungi is back now for the evening | 23:28 | |
| fungi | clarkb: we can push 75213 through, sure, the patch itself seems like a fine idea. but i already tried it on the current round of problem and it didn't make any difference. seems actually not at all dependent on the locally-installed bits of the system and entirely happening within the virtualenv anyway | 23:28 |
| pleia2 | I think this is my favorite bug day post photo yet http://princessleia.com/journal/?p=9277 | 23:29 |
| pleia2 | all bugs wrapped up nicely! | 23:29 |
| jeblair | pleia2: thanks! | 23:29 |
| jeblair | clarkb: what's the state of the idea of a create-db-per-test fixture for independent parallel tests? | 23:30 |
| jeblair | clarkb: specifically for mysql | 23:30 |
| *** mriedem has joined #openstack-infra | 23:31 | |
| clarkb | jeblair: its basically done. On single use slaves we give the openstack_citest users ALL privs | 23:31 |
| jeblair | clarkb: is there code to actually use it? | 23:31 |
| clarkb | jeblair: so tests can set up dbs per whatever | 23:31 |
| *** lcheng has quit IRC | 23:31 | |
| clarkb | jeblair: that is project specific. I know taskflow and storyboard take advantage of the permissions, but sure of any other projects | 23:31 |
| jeblair | clarkb: pretty sure storyboard doesn't since it's set up to log in as "openstack_cifail" | 23:32 |
| clarkb | jeblair: well storyboard devs yelled at me for the feature :) | 23:32 |
| clarkb | I thought that was because they were running into problems with existing tests | 23:32 |
| jeblair | clarkb: anyway, i'm trying to figure out the curernt state and help storyboard fix that | 23:33 |
| jeblair | clarkb: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/migration/test_migrations_base.py#n140 | 23:33 |
| jeblair | there's where the login is set... | 23:33 |
| clarkb | jeblair: gotcha, so any project logging in at openstack_citest can create and use however many schemas they want | 23:33 |
| fungi | jeblair: clarkb: the only reason the log link was at the end of jobs originally was to support jenkins smtp reporter for periodic jobs since it only quoted the end of the log in the e-mail. completely obsolete reason now | 23:33 |
| clarkb | jeblair: I would probably do schema per tests because its easier to reason about but you could do schema per test process | 23:33 |
| jeblair | clarkb: can you walk me through how schema-per-anything works in the context of that file? | 23:34 |
| jeblair | clarkb: i agree it should be schema per test | 23:34 |
| jeblair | though it seems like it should be in this file http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/db_fixture.py | 23:35 |
| clarkb | jeblair: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/migration/test_migrations_base.py#n269 in methods like that you would drop and create a DB particluar to that test, probably based on test name | 23:35 |
| clarkb | jeblair: yes, using a proper fixture is probbly better /me looks at db_fixture | 23:35 |
| clarkb | jeblair: I would add an argument to that fixture called schema_name or similar. Then in setUp do an if present DROP finally create schema_name | 23:36 |
| jeblair | clarkb: okay, i'm trying to reconcile what i'm reading with the idea that this feature is supposed to already exist in storyboard... but that create line looks like it would just "create databasy storyboard" | 23:36 |
| clarkb | jeblair: then that schema will be available to whichver test is using that fixture | 23:36 |
| fungi | clarkb: i think i'm okay with 71323, though close-pull-requests is still broken on a completely separate issue i've not been able to track down yet (either a change in the github api or a change to our openstack-gerrit account is making its orgs list empty) | 23:36 |
| clarkb | jeblair: it may be that storyboard wanted the feature not for DB per test but so that it could use DB with different name than provided | 23:37 |
| clarkb | fungi: ya I was looking at that. I can't figure out how it would be a pygithub problem | 23:37 |
| fungi | clarkb: so 71323 says "Closes-Bug: 1275824" but it really doesn't at all | 23:37 |
| clarkb | fungi: oh because the errors changed... | 23:38 |
| clarkb | gah | 23:38 |
| fungi | jeblair: i'm getting started on prep for tomorrow once i finish catching up on dinnertime scrollback in here. then hopefully finally getting started on my delinquent track chair duties before i pass out for the night | 23:39 |
| clarkb | fungi: http://developer.github.com/v3/orgs/ is the documented api for that. https://github.com/jacquev6/PyGithub/blob/master/github/AuthenticatedUser.py#L824 is the code we should be running | 23:39 |
| clarkb | fungi: one of us may need to run it under pdb to figure out what is happening | 23:39 |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Add the kite (KDS) project and kiteclient project to gerrit/jenkins https://review.openstack.org/73074 | 23:39 |
| jeblair | clarkb: yeah, this file: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/migration/test_migrations.py#n48 | 23:40 |
| jeblair | clarkb: is going to cause it to just "create database openstack_citest". | 23:40 |
| jeblair | which i'm betting isn't going to work. | 23:40 |
| jeblair | and also is not schema-per-anything | 23:40 |
| nibalizer | so i would like to give a talk to PuppetConf about what we do here, and my part in it, which to date hasn't been much, but should be lots of cool stuff by september | 23:41 |
| clarkb | jeblair: that should actually work | 23:41 |
| nibalizer | and the CFP closes this week so its kinda now or never | 23:41 |
| clarkb | jeblair: as long as no other tests try to use that schema | 23:41 |
| nibalizer | how would you all feel about that? | 23:41 |
| jeblair | nibalizer: i feel great about it | 23:41 |
| clarkb | jeblair: so much more robust to do schema per test using the db fixture and stop worrying | 23:41 |
| *** mgagne has joined #openstack-infra | 23:41 | |
| nibalizer | its possible i could turn the larger puppet community on to this project and get more drive-bys and maybe a consistent contributor or two | 23:42 |
| jeblair | nibalizer: feel free to look at slides we've done from previous talks: http://docs.openstack.org/infra/publications/ | 23:42 |
| jeblair | nibalizer: in particular "Code Review for Systems Administrators" may be of interest... | 23:42 |
| nibalizer | okay great! | 23:42 |
| jeblair | nibalizer: also " PuppetConf, 2012: Collaboratively managing the OpenStack project with Puppet" | 23:42 |
| jeblair | nibalizer: and you're gonna like this.... | 23:42 |
| nibalizer | oh yea! | 23:42 |
| nibalizer | oh well if this has already been covered maybe i dont need to give this talk | 23:42 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Ensure there's always an http server object https://review.openstack.org/79792 | 23:42 |
| jeblair | nibalizer: that was 2 years ago! | 23:43 |
| clarkb | and much has changed | 23:43 |
| jeblair | nibalizer: so much has (and will) have change by then! | 23:43 |
| jeblair | nibalizer: http://git.openstack.org/cgit/openstack-infra/publications/ | 23:43 |
| sdague | cool, looks like the libvirt debug change is in the gate | 23:43 |
| jeblair | nibalizer: feel free to _use_ those slides, and submit changes, or if you make a new set, push them as well. :) | 23:44 |
| jeblair | nibalizer: they all go through code review in gerrit. :) | 23:44 |
| jeblair | nibalizer: http://git.openstack.org/cgit/openstack-infra/publications/tree/README.rst | 23:44 |
| *** thuc has joined #openstack-infra | 23:44 | |
| *** thuc has quit IRC | 23:44 | |
| nibalizer | haha oh man thats awesome | 23:44 |
| *** thuc has joined #openstack-infra | 23:45 | |
| jeblair | nibalizer: the repo gets automatically published in the usual way :) | 23:45 |
| fungi | clarkb: i've been running close-pull-requests as the github user on review-dev with various iterations of debugging in modified copies of the script over past weeks as i get a spare moment here or there. the get_groups() api call literally gives us an empty list for that account these days. i even logged into the github webui with that account and used the restful api calls for it and get the same | 23:45 |
| jeblair | http://git.openstack.org/cgit/openstack-infra/publications/tree/README.rst | 23:45 |
| fungi | behavior. i think maybe they stopped returning orgs for groups to which your account belongs? we might be better off iterating over projects.yaml to decide what projects to act on instead anyway | 23:45 |
| jeblair | fungi: ++ | 23:45 |
| *** mwagner_lap has joined #openstack-infra | 23:45 | |
| nibalizer | oh yes this is going to be sweet | 23:46 |
| jeblair | clarkb: so does a db fixture that does what we actually want exist? | 23:46 |
| clarkb | jeblair: I don't think so. Openstack went the route of global locking isntead of fixing this properly | 23:46 |
| *** mgagne has quit IRC | 23:46 | |
| nibalizer | jeblair: clarkb so do I have general approval to do this then? how many of you need to approve before I can fire off my proposal? | 23:46 |
| clarkb | jeblair: so we didn't realize it was an issue until taskflow and storyboard complained after we thought we fixed it | 23:46 |
| clarkb | nibalizer: go for it. We are pretty laid back I don't think you need much approval | 23:47 |
| fungi | clarkb: er, i just re-read what i wrote, and meant to say i've been running close-pull-requests as the github user on review.o.o (not review-dev) | 23:47 |
| clarkb | fungi: interesting, so github changed thair api | 23:47 |
| nibalizer | okay thanks! | 23:47 |
| clarkb | fungi: so you think we should just assume the org exists if it is in projects.yaml? I think that is fair | 23:47 |
| jeblair | krotscheck, mordred: we just spent like 100 lines talking about storyboard and databases ^ | 23:47 |
| fungi | er, and it's not get_groups() (i was misremembering) it's get_orgs() | 23:47 |
| fungi | clarkb: yeah, that's what we probably need to have it do | 23:48 |
| clarkb | fungi: ya get_orgs(). Using projects.yaml that way sounds fine to me | 23:48 |
| jeblair | krotscheck, mordred, clarkb: i think the short version is that what's in storyboard should work if we just unbreak it, but it's not what we really want (which is a schema-per-test fixture) | 23:48 |
| clarkb | jeblair: agreed | 23:48 |
| fungi | because github is too much of a black box. i poured over their current api documentation and honestly can't figure out why it's not still working the way we have it written | 23:48 |
| clarkb | fungi: agreed, reading their docs implies it should work as is | 23:49 |
| fungi | and it did! up until a few months ago when it didn't any longer | 23:49 |
| clarkb | fungi: does GET /users/:user/orgs form work any better? | 23:50 |
| clarkb | fungi: we could ghub.get_user("nameofgithubuser").get_orgs() to make that work I think | 23:50 |
| clarkb | but maybe not | 23:50 |
| fungi | clarkb: nope. it was giving me a webpage back with [] | 23:50 |
| clarkb | :( | 23:50 |
| fungi | even when authenticated | 23:50 |
| *** jhesketh__ has quit IRC | 23:51 | |
| fungi | i didn't get as far as stuffing that form into the python, but i'll give it a shot just in case | 23:51 |
| fungi | but the web api was giving me back an empty array | 23:51 |
| *** jhesketh__ has joined #openstack-infra | 23:51 | |
| *** thuc has quit IRC | 23:52 | |
| *** zul has joined #openstack-infra | 23:52 | |
| *** zul has quit IRC | 23:52 | |
| clarkb | jeblair: fungi https://bugs.launchpad.net/openstack-ci/+bug/1082759 any news on that front? | 23:52 |
| clarkb | it is a bit saddening that we talk about it at every summit I have been to but haven't been able to do much more than retarget to the next release | 23:52 |
| *** bhuvan_ has quit IRC | 23:52 | |
| *** bhuvan has quit IRC | 23:52 | |
| fungi | clarkb: supposedly the openstackid project is a prerequisite for that (though i've yet to hear an adequate explanation as to why) so i've been trying to prioritize reviews for it when they show up | 23:53 |
| fungi | in hopes that once it's running, the www site code will just magically fall from the sky and plop down into a public git repo or something | 23:54 |
| clarkb | fungi: I don't see anything on http://developer.github.com/changes/ indicated the /user/orgs api changed at all | 23:54 |
| fungi | clarkb: nor i. i checked there in hopes that it did | 23:55 |
| *** dizquierdo has quit IRC | 23:55 | |
| jogo | clarkb: https://review.openstack.org/#/c/78612/1/modules/openstack_project/files/zuul/layout.yaml | 23:55 |
| jogo | I am confused | 23:55 |
| jogo | line 938, why 2x check-tempest-dsvm-neutron-pg | 23:56 |
| fungi | clarkb: yeah, i just tried list(ghub.get_user("openstack-gerrit").get_orgs()) and it too returns [] | 23:56 |
| clarkb | fungi: :( I say screw it use projects.yaml | 23:56 |
| fungi | jogo: so that it runs twice | 23:57 |
| fungi | clarkb: yeah, it's not worth my time to keep guessing further | 23:57 |
| *** caleb_ has joined #openstack-infra | 23:57 | |
| clarkb | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1253774 we never fixed that did we? I can probably whip up a nodepool change that does that quickly | 23:57 |
| *** bhuvan has joined #openstack-infra | 23:57 | |
| *** bhuvan_ has joined #openstack-infra | 23:57 | |
| jogo | why? | 23:57 |
| clarkb | jogo: because sdague wanted more numbers | 23:57 |
| jogo | why 2x? salv-orlando was wondering as was I | 23:57 |
| jogo | ahh | 23:57 |
| fungi | and an increased chance that nondeterminism in those jobs would block changes from the projects most likely to cause them | 23:58 |
| sdague | jogo: because having an integrated job be able to pass code running < 4 devstack runs is bad | 23:58 |
| jogo | sdague: ahhh | 23:58 |
| sdague | and leads to problems because it's too easy to land a coin flip race bug | 23:58 |
| jogo | now it makes sense | 23:58 |
| sdague | which neutron used to do a lot | 23:58 |
| salv-orlando | I suspected it was by design. I just wanted to be sure I was not seeing double because I was drunk ;) | 23:58 |
| *** malini_afk is now known as malini | 23:59 | |
| fungi | salv-orlando: i just always divide by two on basic principle | 23:59 |
| sdague | we invented other jobs at one point, to help there, but then zuul let us define doubles | 23:59 |
| sdague | so that was just simpler | 23:59 |
| sdague | also, I agree, it should be explained in the config somewhere, because it's completely tribal knowledge | 23:59 |
| salv-orlando | I mean from the neutron perspective I think it's pretty much the same. We already executed 4 devstack runs per patch. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!