| *** sarob has joined #openstack-infra | 00:02 | |
| fungi | okay, so here's a weird one, though maybe just for those of us who don't deploy openstack... https://launchpad.net/bugs/1281200 | 00:03 |
|---|---|---|
| *** dstanek has joined #openstack-infra | 00:03 | |
| fungi | obviously it's filed against the wrong project, but wondering which project contains the "openstack-status" utility, or whether that's just something the sles packagers worked up? | 00:04 |
| *** sarob has quit IRC | 00:08 | |
| *** lcheng has joined #openstack-infra | 00:13 | |
| *** jhesketh__ has quit IRC | 00:14 | |
| nibalizer | AaronGr: are you online? want to review https://review.openstack.org/#/c/73840 | 00:14 |
| *** pcrews has joined #openstack-infra | 00:14 | |
| fungi | jerryz: what were you trying to get up with me about earlier? | 00:16 |
| AaronGr | hi nibalizer: i was peeking at that yesterday, i'll look it over now | 00:16 |
| *** yamahata has joined #openstack-infra | 00:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Revert "Temporarily stop running tripleo seed/undercloud"" https://review.openstack.org/74186 | 00:18 |
| jeblair | i'm going to restart zuul now, with the new mergers | 00:19 |
| jeblair | it will be a hard stop/start with re-enqueue | 00:19 |
| fungi | awesome | 00:19 |
| mordred | ooh! | 00:19 |
| fungi | well, not so awesome. hung jobs do suck | 00:19 |
| mordred | jeblair, fungi: nibalizer's new puppetdb patch looks really good - and is also quite small and easy to grok | 00:19 |
| nibalizer | AaronGr: awesome thanks! | 00:19 |
| *** dstanek has quit IRC | 00:20 | |
| nibalizer | mordred: especially compared to my 25Kline patch :) | 00:20 |
| AaronGr | nibalizer: good stuff, very clean. | 00:20 |
| jeblair | nibalizer: great! i will review it tomorrow. thank you very much! | 00:20 |
| nibalizer | no prob, that one should come up pretty easily, the more scary one is when we apply the puppetdb::master class to the puppet master | 00:21 |
| *** salv-orlando has quit IRC | 00:22 | |
| nibalizer | also architecturaly it puts the postgres server for puppetdb on the puppetdb server, if we wanted to we could run those on separate nodes, or setup some kind of postgres replication for the puppetdb data and so on | 00:22 |
| nibalizer | AaronGr: jeblair thanks for the compliments | 00:22 |
| *** pcm_ has joined #openstack-infra | 00:22 | |
| pcm_ | Folks, I got a Jenkins failure that appears to be in Nicira plugin, foreign key contstraint database error. Is this known? | 00:23 |
| *** zigo has quit IRC | 00:25 | |
| fungi | pcm_: you'd probably want to ask the neutron folks | 00:25 |
| pcm_ | fungi: OK. Thanks. | 00:25 |
| fungi | maybe check open bugs against neutron and open one if you don't see anything relevant | 00:25 |
| pcm_ | see two similar, but not sure it's the same. This is failing Jenkins run and is unrelated to my fix AFAICT | 00:26 |
| *** zigo has joined #openstack-infra | 00:26 | |
| anteaya | pcm_: if it was in the unit tests, yes markmcclain knows about it | 00:26 |
| fungi | pcm_: then it's probably some sort of nondeterministic failure either in neutron or possibly in neutron's tests | 00:26 |
| pcm_ | fungi: anteaya: Thanks. I'm asking over in Neutron... | 00:27 |
| anteaya | pcm_: ping someone who works with nicira, armax, salvatore, arosen | 00:27 |
| pcm_ | roger that | 00:28 |
| anteaya | k | 00:28 |
| anteaya | I don't have a bug number for it | 00:28 |
| *** matsuhashi has joined #openstack-infra | 00:28 | |
| anteaya | I'm getting a timeout error for launchpad.net | 00:29 |
| anteaya | anyone else? | 00:29 |
| anteaya | connected now | 00:29 |
| mordred | nibalizer: what happens to the system overall if there is a postgres issue? | 00:30 |
| mordred | nibalizer: like, does puppetmaster stop working now? or just reporting? | 00:30 |
| *** mrodden1 has joined #openstack-infra | 00:32 | |
| *** mrodden has quit IRC | 00:33 | |
| nibalizer | i think if puppetdb is up, puppetdb queues things to be written for a time, eventually giving up on writing them to postgres and writes them to the dead letter office(16 retries i think) | 00:34 |
| nibalizer | not sure though, i have a small personal installation I can do some tesitng on | 00:34 |
| nibalizer | or #puppet would know | 00:34 |
| mordred | nibalizer: it's not _that_ big of a deal - the gate here does not stop if puppetmaster goes down | 00:35 |
| mordred | nibalizer: was more just curious | 00:35 |
| mordred | and none of us are really postgres people, so that's my biggest worry in any of that | 00:35 |
| mordred | nibalizer: I think we can just co-locate puppetdb and its postgres for now too | 00:36 |
| nibalizer | from what i've been told, puppetdb uses features that require postgres, can't just slide in anything else | 00:36 |
| nibalizer | how does alerting/paging work for this team? i.e. if puppetdb/postgres did fall over who would be responding? | 00:37 |
| *** pcrews has quit IRC | 00:37 | |
| *** zigo has quit IRC | 00:42 | |
| *** lcheng has quit IRC | 00:42 | |
| *** jhesketh__ has joined #openstack-infra | 00:43 | |
| jeblair | nibalizer: no one; if someone notices, ideally tools like puppetboard should help anyone diagnose the problem. if that doesn't work, an infra-root would have to log in and fix | 00:43 |
| morganfainberg | jeblair, do we need to requeue our checks/gate jobs? or will they be reloaded? | 00:44 |
| morganfainberg | jeblair, just saw you guys did a stop on zuul, figured i'd ask | 00:44 |
| jeblair | morganfainberg: they should be reloaded, i'm weiting to see if the first jobs work | 00:44 |
| morganfainberg | jeblair, awesome :) | 00:44 |
| jeblair | and they don't. :) | 00:44 |
| morganfainberg | jeblair, aww well at least you know! | 00:45 |
| nibalizer | jeblair: puppetboard will give you an 'i cant connect to puppetdb', which will start someone on their way to debug | 00:45 |
| *** hdd_ has joined #openstack-infra | 00:46 | |
| *** lcheng has joined #openstack-infra | 00:47 | |
| jeblair | huh, it's not a complete failure... https://jenkins01.openstack.org/job/gate-solum-pep8/722/console | 00:47 |
| jeblair | solum just fetched a ref from zm01 | 00:47 |
| fungi | ooh! | 00:48 |
| *** pcrews has joined #openstack-infra | 00:48 | |
| fungi | stirrings of life from the first of zuul's newly minted hordes of minions | 00:48 |
| pcm_ | Folks. I see that Grenade is failing, something about upgrade-swift. Looks like 1280464 (already reported by someone) Do I need to do anything for my commit? | 00:50 |
| jeblair | i think i fixed the problem that the first jobs hit; i'll re-enqueue the saved jobs now | 00:52 |
| fungi | pcm_: probably just leave a review comment in your change of "recheck bug 1280464" and maybe also help the neutron folks track down what's causing that issue in the neutron grenade upgrade tests | 00:53 |
| pcm_ | looks like it's non-voting. | 00:53 |
| fungi | pcm_: oh, then i wouldn't worry about it | 00:53 |
| pcm_ | Will try that. | 00:53 |
| pcm_ | On the other bug (in gate), thought it may be a commit (to skip test) committed this morning, but I have that commit. | 00:54 |
| mordred | nibalizer: so - because I'm dumb... | 00:55 |
| mordred | nibalizer: the idea is that this patch runs puppetdb, then the next patch tells puppetmaster to talk to it | 00:55 |
| mordred | nibalizer: and then there will be a third patch that installs puppetboard? or installing puppetdb also means installing puppetboard? | 00:55 |
| *** pcrews has quit IRC | 00:56 | |
| *** banix has joined #openstack-infra | 00:56 | |
| *** lcheng has quit IRC | 00:57 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Register merge jobs before starting the worker https://review.openstack.org/74211 | 00:57 |
| mordred | pcm_, fungi: there is currentlya known issue with swift and grenade and grizzly | 00:57 |
| jeblair | mordred, clarkb, fungi: ^ that is the first in a series of changes that I have hand-applied in production; quick review/approval would be appreciated | 00:58 |
| jeblair | more on the way | 00:58 |
| fungi | jeblair: yep, lgtm | 00:58 |
| *** lcheng has joined #openstack-infra | 00:59 | |
| jeblair | clarkb: ^ you'll be interested in that one since you questioned whether that was necessary in review. I'm not positive but I'm starting to suspect it is. | 00:59 |
| *** zigo has joined #openstack-infra | 00:59 | |
| pcm_ | mordred: I did a recheck, partly to incr count of that bug seen, but because of another failure in neutron gate. | 01:00 |
| mordred | pcm_: cool | 01:00 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Fix exception in status admin command https://review.openstack.org/74212 | 01:00 |
| jeblair | mordred, fungi: ^ #2 | 01:00 |
| jeblair | clarkb: ^ and that's also the bug that we hit sometimes with logstash i think | 01:01 |
| pcm_ | mordred: The gate issue is a nicira DB integrity failure in some router binding or such (totally unrelated to my change) | 01:01 |
| *** flaper87 is now known as flaper87|afk | 01:02 | |
| nibalizer | mordred: yes, this patch will bring up a puppetdb, but it will be lonely and kinda useless | 01:03 |
| nibalizer | a second patch will configure the puppetmaster to talk to it through installing puppetdb-terminus package and configuring puppetdb.yaml and maybe a line or two in puppet.conf | 01:04 |
| nibalizer | and a third patch will bring puppetboard online, puppetboard is a flask app | 01:04 |
| nibalizer | im not sure where we want to locate puppetboard, maybe on the puppetdb host or maybe its own host | 01:04 |
| *** zigo has quit IRC | 01:08 | |
| mordred | nibalizer: I betcha on the puppetdb host will be fine for a start | 01:08 |
| mordred | I don't expect huge amounts of people hammering it for stuff | 01:08 |
| *** dstanek has joined #openstack-infra | 01:09 | |
| nibalizer | that also vastly simplifies puppetdb <-> puppetboard communication since then they can speak over an unecrypted localhost port | 01:13 |
| *** zigo has joined #openstack-infra | 01:13 | |
| nibalizer | otherwise we'd have to set up http certs and stuff which can be a pain | 01:13 |
| clarkb | jeblair: thanks I am heades to computer soon. will review | 01:14 |
| *** lcheng has quit IRC | 01:15 | |
| *** zigo has quit IRC | 01:20 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix zuul installation https://review.openstack.org/74215 | 01:21 |
| jeblair | clarkb, fungi, mordred: and that's the last of the manually applied fixes ^ | 01:21 |
| *** zigo has joined #openstack-infra | 01:22 | |
| fungi | jeblair: puppet-lint is going to complain about mismatched spaces+tabs in at least manifests/site.pp | 01:22 |
| jeblair | fungi: thx | 01:23 |
| fungi | manifests/zuul_dev.pp too | 01:23 |
| fungi | i'm betting most of them | 01:23 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix zuul installation https://review.openstack.org/74215 | 01:25 |
| mordred | nibalizer: yeah. then let's definitely start there | 01:25 |
| jeblair | i've only started the merger on zm01 | 01:26 |
| jeblair | mordred, fungi, clarkb: once all those changes are in, we can test that they are good on zm02, then shoud be able to start puppet everywhere | 01:26 |
| mordred | jeblair: awesome | 01:27 |
| *** zigo has quit IRC | 01:27 | |
| notmyname | mordred: I haven't been online today. I just sat down briefly (ie can't stay now) and saw you mentioned some "known problem with swift and grenade". something I need to take a look at later? (crosspost from -swift) | 01:27 |
| notmyname | mordred: ah, I see you in -swift | 01:27 |
| jeblair | mordred, fungi, clarkb: zuul depends on gear releases, so i think we'll need to cut one of those and bump zuul's dep too. | 01:28 |
| *** zigo has joined #openstack-infra | 01:28 | |
| fungi | jeblair: i think the known_hosts files need the .ssh directory created first, though i might be wrong about that. comments inline for all 3 occurrences | 01:30 |
| *** nosnos has joined #openstack-infra | 01:33 | |
| anteaya | fungi: by the way, did you ever get anywhere digging into why manage-projects failed to build repos on gerrit, from these error messages: http://paste.openstack.org/show/66599/ | 01:35 |
| *** zhiyan_ is now known as zhiyan | 01:35 | |
| fungi | anteaya: i'm still trying to collect all the most recent details into an update in bug 1242569 (tracebacks, symptoms, et cetera) | 01:36 |
| anteaya | k thanks | 01:36 |
| anteaya | I'll look for the bug update once it happens | 01:37 |
| *** jroovers|afk has quit IRC | 01:37 | |
| *** emagana has quit IRC | 01:37 | |
| anteaya | who can change the text on https://launchpad.net/openstack-ci | 01:38 |
| anteaya | we need the repos to point at git.o.o, it still points to github | 01:39 |
| fungi | anteaya: fixed | 01:41 |
| anteaya | thanks | 01:41 |
| *** jroovers has joined #openstack-infra | 01:41 | |
| openstackgerrit | A change was merged to openstack-infra/gear: Fix exception in status admin command https://review.openstack.org/74212 | 01:46 |
| *** zigo has quit IRC | 01:51 | |
| lifeless | fungi: btw you know that for the ci-overcloud a) we don't get alerted if its down yet and b) there are 8 other admins :) | 01:51 |
| lifeless | fungi: http://git.openstack.org/cgit/openstack/tripleo-incubator/tree/tripleo-cloud/tripleo-cd-admins | 01:51 |
| fungi | lifeless: oh, cool | 01:52 |
| fungi | lifeless: this time, as soon as i noticed i popped into #tripleo and discussed it with derekh. he was already looking into it at that point | 01:52 |
| lifeless | fungi: so -please- don't sit and wait for someone to fix it if it glitches. run around screaming and tell us all | 01:53 |
| lifeless | everyone in that list should consider themselvse on call to address issues | 01:53 |
| *** zigo has joined #openstack-infra | 01:53 | |
| lifeless | since they are the only ones that can | 01:53 |
| lifeless | and its a production system | 01:53 |
| lifeless | if folk don't want to be on the hook, they should remove themselves | 01:53 |
| lifeless | I know I can be rung to sort out issues in the cloud | 01:53 |
| jerryz | fungi: do you know whether nested virtualization is enabled on hp cloud? | 01:53 |
| lifeless | though I'd hope folk would ring someone tz compatible in the first instance | 01:54 |
| lifeless | jerryz: its not. | 01:54 |
| mordred | jerryz: it is not | 01:54 |
| fungi | the previous outage i asked around in #tripleo when you weren't around but nobody except you was answering me for a couple days, until you passed it off to derekh | 01:54 |
| lifeless | fungi: yes, that one was bad for one simple reason. I had deleted the cloud. | 01:54 |
| lifeless | fungi: and we didn't have automation to deploy it again yet. | 01:54 |
| jerryz | lifeless, mordred: thank you for the fast response. | 01:54 |
| fungi | heh. we all delete a cloud every now and then ;) | 01:54 |
| lifeless | fungi: its a 'should never ever happen' situation. | 01:54 |
| * mordred deletes them all the time | 01:54 | |
| lifeless | fungi: yeah, production ones though ? :) | 01:55 |
| lifeless | fungi: anyhow, we took our time bringing it back to fix the automation and deploy it at medium scale | 01:55 |
| *** jroovers has joined #openstack-infra | 01:55 | |
| clarkb | jeblair: the zuul and geard changes lgtm | 01:55 |
| fungi | lifeless: yes, the increased capacity is pretty awesome | 01:55 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Register merge jobs before starting the worker https://review.openstack.org/74211 | 01:56 |
| lifeless | fungi: any normal fault should be much more like the one today where a bug crops up and it should be straight forward to fix | 01:56 |
| fungi | lifeless: out of curiosity, what was the issue this time? derekh was still looking into it last i sync'd up with him, but that's been quite a few hours now | 01:56 |
| lifeless | fungi: eth2 and br-untagged both had ip addresses. | 01:56 |
| lifeless | fungi: https://bugs.launchpad.net/tripleo/+bug/1272969 | 01:57 |
| fungi | ahh, yep, on the same broadcast domain in that case | 01:57 |
| fungi | assuming eth2 is in your br | 01:57 |
| lifeless | same ip address | 01:57 |
| fungi | ooh, painful :/ | 01:57 |
| lifeless | this works poorly | 01:57 |
| lifeless | since the result is that no traffic flows | 01:58 |
| fungi | yes, arp overwrites out the wazoo | 01:58 |
| lifeless | and you're boned | 01:58 |
| lifeless | simple to fix, derekh hadn't seen it before | 01:58 |
| lifeless | fungi: where should one document how to reach a given cloud provider for assistance | 01:59 |
| lifeless | its not documented atm AFAICT, and I really don't like the idea that you were left hanging | 01:59 |
| clarkb | and config change reviewed now as well | 01:59 |
| lifeless | since we went to a bunch of effort setting up a team to handle problems :( | 01:59 |
| fungi | poor example, but we usually scream in irc at rackspace people who don't deserve it, or hpcloud people who sometimes do ;) | 02:00 |
| fungi | and open tickets in their support system for good measure | 02:00 |
| fungi | in the case of tripleo i've just gone and pouted pitifully in #tripleo hoping someone would know what was going on, but this time there was at least someone already looking into it before i noticed | 02:01 |
| fungi | since this is our first example of a non-commercial service provider, our usual ad-hoc contact solutions may need to be fine-tuned a bit to suit | 02:02 |
| fungi | but really, i didn't encounter any issues finding someone to look into it, so it seems to be going okay in that regard | 02:03 |
| *** zigo has quit IRC | 02:04 | |
| dstufft | yelling at people in IRC is the best way to get support for literally anything | 02:05 |
| *** vkozhukalov has joined #openstack-infra | 02:05 | |
| fungi | dstufft: can you help me with my homework? i'm supposed to implement a bubble sort in javasharp | 02:05 |
| dstufft | fungi: i'm bad at this proramming thing, you don't want my help | 02:06 |
| *** sarob has joined #openstack-infra | 02:06 | |
| *** morganfainberg is now known as morganfainberg_Z | 02:06 | |
| *** zigo has joined #openstack-infra | 02:06 | |
| *** yaguang has joined #openstack-infra | 02:09 | |
| *** sarob has quit IRC | 02:13 | |
| *** vkozhukalov has quit IRC | 02:19 | |
| *** morganfainberg_Z is now known as morganfainberg | 02:25 | |
| *** jaypipes has quit IRC | 02:33 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Fix zuul installation https://review.openstack.org/74215 | 02:33 |
| fungi | clarkb: moar correctful ^ | 02:33 |
| fungi | testing that with --noop on zm02 using a dev env | 02:34 |
| lifeless | fungi: since we're not in zuul now and jeblair has concerns about our being back in zuul, I don't think its going okay :) | 02:34 |
| lifeless | fungi: ok is when you say 'its all fine and business as usual' | 02:35 |
| *** zigo has quit IRC | 02:35 | |
| fungi | lifeless: well, clearly there's is still process being worked out within the tripleo admins group as far as them each knowing they can reach out to others in their group to help them diagnose failures, but that just seems like natural growing pains | 02:36 |
| *** morganfainberg is now known as morganfainberg_Z | 02:38 | |
| *** jeckersb is now known as jeckersb_gone | 02:38 | |
| fungi | clarkb: jeblair: i've got the tip of the gear master branch tagged as 0.5.1 locally and am ready to push it if desired | 02:48 |
| *** zigo has joined #openstack-infra | 02:48 | |
| fungi | 3fcb8e9 "Fix exception in status admin command" | 02:48 |
| fungi | since it was mostly just that plus a requirements tweak and a tox config change, it didn't feel like a 0.6.0 | 02:49 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/zuul: Require gear 0.5.1 https://review.openstack.org/74237 | 02:54 |
| fungi | and that ^ would make use of it | 02:54 |
| lifeless | fungi: well, I didn't think that was the case :) | 02:56 |
| lifeless | fungi: so,there is a disconnect somewhere. | 02:56 |
| mordred | fungi: gerrit_ssh_host_key => hiera('gerrit_ssh_rsa_pubkey_contents'), | 02:57 |
| mordred | fungi: i BELIEVE gerrit_ssh_rsa_pubkey_contents is named poorly | 02:57 |
| mordred | gah | 02:57 |
| mordred | ECAPSLOCK | 02:57 |
| fungi | mordred: perhaps so. i'll have a double-check in the hiera file to make sure that's the right one | 02:58 |
| mordred | fungi: I'd expand that to say that all of the gerrit-related keys are named poorly, but I chalk that up to hysterical raisins | 02:58 |
| fungi | mordred: is that youre way of saying you were the one who chose those names? ;) | 02:59 |
| fungi | your | 02:59 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Fix log Footer README's for 'check-tempest-dsvm' https://review.openstack.org/74238 | 02:59 |
| mordred | fungi: yes | 02:59 |
| mordred | fungi: I believe in all cases where the names are unclear, I take the blame | 03:00 |
| mordred | fungi: just be glad none of them are called gerrit_host_ssh_rsa_assassass | 03:00 |
| mordred | that means I'm growing as a person | 03:00 |
| *** markwash has quit IRC | 03:00 | |
| mordred | fungi: 74215 looks good to me - any reason not to merge it? | 03:01 |
| fungi | mordred: confirmed, gerrit_ssh_rsa_pubkey_contents in hiera matches the contents of ~review_site/etc/ssh_host_rsa_key.pub | 03:01 |
| fungi | er, ~gerrit2/review_site/etc/ssh_host_rsa_key.pub | 03:01 |
| mordred | fungi: we should maybe at some point rename that to gerrit_ssh_host_rsa_key_pub | 03:02 |
| fungi | mordred: sounds like a smashing idea | 03:02 |
| mordred | I mean, not right now | 03:02 |
| mordred | dear god | 03:02 |
| mordred | fungi: anywho - any reason to not merge the zuul config change? | 03:02 |
| fungi | and sure, i expect 74215 is safe enough to merge. i'll make sure it takes effect successfully on zm02 where puppet agent is still running | 03:03 |
| mordred | +A | 03:03 |
| mordred | fungi: what if we made gerritbot allow us to vote on changes from here in channel? | 03:03 |
| fungi | then it's just a question of whether we're cool with me pushing the gear 0.5.1 tag, at which point 74237 is probably also warranted | 03:03 |
| mordred | fungi: what could go wrong with that? | 03:03 |
| mordred | fungi: I +2'd 74237 already | 03:04 |
| mordred | and I'm good with you pushing the tag - but I'll defer to jeblair on that | 03:04 |
| fungi | mordred: yeah, he said it was warranted, but not what version number he wanted for it | 03:04 |
| *** CaptTofu has joined #openstack-infra | 03:05 | |
| fungi | so i'll wait for some direction on that | 03:05 |
| fungi | and go back to seeing what i can do to get the zmq plugin release jobs running again | 03:05 |
| HenryG | Hello. I have a small gerrit nitpick. I *think* it is a setting/hook on review.openstack.org that is not quite right. | 03:06 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix zuul installation https://review.openstack.org/74215 | 03:06 |
| *** banix has quit IRC | 03:06 | |
| *** esker has joined #openstack-infra | 03:07 | |
| HenryG | When someone posts a comment to a review, and puts a link to a launchpad bug in the comment, in the web view of the review the last digit of the bug link is dropped. | 03:07 |
| HenryG | For example, see line 114 here: https://review.openstack.org/#/c/73372/8/neutron/plugins/ml2/drivers/cisco/apic/mechanism_apic.py | 03:08 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Fix log Footer README's for 'check-tempest-dsvm' https://review.openstack.org/74238 | 03:08 |
| HenryG | That should be a link to https://bugs.launchpad.net/neutron/+bug/1276391 <-- but the last '1' is missing. | 03:09 |
| *** tteggel has quit IRC | 03:09 | |
| *** sarob has joined #openstack-infra | 03:09 | |
| *** tteggel has joined #openstack-infra | 03:10 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Remove depreciated htmlify-screen-log https://review.openstack.org/74241 | 03:11 |
| fungi | HenryG: yes, something's still not quite right with the gerrit commentlink configuration. the previous iterations are now causing it to remove the final digit if the bug is pasted in as a url (maybe having to do with it being at the end of a line of text). i think there's a change proposed to fix it. needle buried somewhere in our current review haystack | 03:11 |
| *** esker has quit IRC | 03:11 | |
| HenryG | fungi: thanks, glad the problem is known | 03:12 |
| *** sarob has quit IRC | 03:14 | |
| fungi | zaro: clarkb: strangely, i retriggered zmq-event-publisher-hpi-artifact for 0.0.3 and it worked this time, so we've got some sort of nondeterministic behavior in that job | 03:14 |
| mordred | fungi: AWESOME | 03:14 |
| *** dkranz has joined #openstack-infra | 03:15 | |
| *** afazekas has quit IRC | 03:16 | |
| fungi | triggered zmq-event-publisher-jenkinsci-upload for it after that, and it worked fine too | 03:17 |
| *** afazekas has joined #openstack-infra | 03:19 | |
| *** lcheng has joined #openstack-infra | 03:20 | |
| *** matsuhashi has quit IRC | 03:23 | |
| jeblair | fungi: ERISOTTO here; pushing tag is fine with me | 03:23 |
| HenryG | fungi: yup, https://review.openstack.org/71743 | 03:24 |
| fungi | jeblair: done | 03:25 |
| notmyname | mordred: mtreinish: I'm around for a while if there are swift issues that need discussing | 03:25 |
| mfisch | Hey infra guys, there's a bug filed against openstack-manuals where the script claims not to know where openstack/glance bugs should go | 03:25 |
| mfisch | Is that correct? Should it be filing against glance directly? https://bugs.launchpad.net/openstack-manuals/+bug/1279866 | 03:26 |
| *** Sukhdev has joined #openstack-infra | 03:26 | |
| *** pcm_ has quit IRC | 03:28 | |
| fungi | mfisch: i believe that's the result of having a docimpact-group set for it in http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/review.projects.yaml | 03:28 |
| mfisch | so it looks like it's going to openstack-manuals on purpose | 03:29 |
| *** lifeless has quit IRC | 03:29 | |
| fungi | mfisch: i think so. though openstack-manuals is the fallback bug target for docimpact tags, so it might not have worked as designed... | 03:29 |
| *** pcrews has joined #openstack-infra | 03:29 | |
| fungi | mika: jhesketh_: ^ you worked on that feature, right? | 03:30 |
| fungi | er, mikal (sorry mika!) | 03:30 |
| jhesketh__ | fungi: yeah mikal implemented | 03:31 |
| mfisch | fungi: nova, neutron, etc also do the same but the message from the script indicates confusion. Regardless there's no action here for infra, so excuse the noise | 03:31 |
| *** CaptTofu has quit IRC | 03:31 | |
| *** talluri has joined #openstack-infra | 03:31 | |
| *** lifeless has joined #openstack-infra | 03:31 | |
| fungi | mfisch: well, i think it's only supposed to add that boilerplate when it's unable to find the mapping in projects.yaml, so it may actually indicate a bug in our docimpact hook script | 03:32 |
| fungi | though luckily the default behavior was chosen sanely so that it ended up in the right place anyway | 03:33 |
| mfisch | yep | 03:33 |
| jhesketh__ | yeah I'm taking a glance | 03:33 |
| jhesketh__ | (no pun intended) | 03:33 |
| mfisch | it was absolutely intended | 03:34 |
| fungi | heh | 03:34 |
| fungi | thanks! | 03:34 |
| mfisch | fixing this will be a keystone to a solid release | 03:34 |
| jhesketh__ | lol | 03:34 |
| mfisch | hopefully a fix is on the horizon | 03:34 |
| mfisch | okay I'll stop now | 03:34 |
| openstackgerrit | Eric Windisch proposed a change to openstack/requirements: Blacklist pyghmi version 0.5.9.1 https://review.openstack.org/74246 | 03:38 |
| *** dnavale has joined #openstack-infra | 03:39 | |
| *** dnavale has left #openstack-infra | 03:39 | |
| *** lcheng has quit IRC | 03:42 | |
| *** talluri has quit IRC | 03:48 | |
| *** talluri has joined #openstack-infra | 03:48 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Require gear 0.5.1 https://review.openstack.org/74237 | 03:49 |
| *** talluri has quit IRC | 03:51 | |
| jhesketh__ | I think I've found the problem, so a fix should be on the horizon - thanks mfisch ;-) | 03:51 |
| fungi | thanks again jhesketh__! | 03:51 |
| mfisch | glad I brought the heat on this one | 03:51 |
| *** talluri has joined #openstack-infra | 03:51 | |
| jhesketh__ | don't make me go all super-nova on you | 03:51 |
| fungi | oh no you didn't | 03:52 |
| fungi | job activity for the past 180 days... http://graphite.openstack.org/render/?from=-180days&target=alias(summarize(sumSeries(stats_counts.zuul.pipeline.*.all_jobs),%271d%27),%27All%20Jobs%27)&title=Zuul%20Jobs%20Launched%20(per%20Day)&_t=0.3401787835629061 | 03:52 |
| mfisch | like a neutron, you're not getting a reaction from me jhesketh_ | 03:52 |
| jhesketh__ | now that's just ironic | 03:53 |
| *** mgagne has quit IRC | 03:53 | |
| *** pcrews has quit IRC | 03:54 | |
| fungi | i think you burned him to a cinder with that one | 03:55 |
| jhesketh__ | well I am a turbo-hipster when it comes to irony | 03:55 |
| *** talluri has quit IRC | 03:56 | |
| mfisch | this discussion has been a trove of puns | 03:58 |
| jhesketh_ | there's just too much fuel in the projects | 03:59 |
| *** david-lyle has quit IRC | 04:00 | |
| fungi | clarkb: zaro: jenkins-dev is running zmq plugin 0.0.3 now | 04:01 |
| *** gokrokve has quit IRC | 04:01 | |
| *** gokrokve has joined #openstack-infra | 04:01 | |
| clarkb | fungi woot thanks | 04:01 |
| *** gokrokve has quit IRC | 04:01 | |
| clarkb | I tested that commit so it should be good | 04:01 |
| *** gokrokve has joined #openstack-infra | 04:02 | |
| fungi | awesome | 04:02 |
| *** gokrokve has joined #openstack-infra | 04:02 | |
| fungi | clarkb: i note that the plugin metadaya still points to your github account... you probably want to change that at some point | 04:02 |
| fungi | er, metadata | 04:03 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/jeepyb: Fix docimpact_target project selection https://review.openstack.org/74253 | 04:03 |
| jhesketh__ | fungi, mfisch: all these swift jokes aside, that should be your fix ^ | 04:03 |
| jhesketh__ | The bug was indeed reported to the correct place by default anyway | 04:03 |
| mfisch | thanks box | 04:03 |
| mfisch | err both | 04:03 |
| *** david-lyle has joined #openstack-infra | 04:04 | |
| fungi | jhesketh__: was that the only script calling ProjectsYamlRegistry.get()? | 04:04 |
| jhesketh__ | yep, took a hunt around | 04:04 |
| jhesketh__ | might be worth double checking though | 04:04 |
| fungi | jhesketh__: cool. since that changes the number of parameters it's worth a double-check, agreed | 04:04 |
| jhesketh__ | fungi: also, oddly, there is __getitem__ just above it.. perhaps we should rename get to 'get_project_item'? | 04:05 |
| jhesketh__ | that way pep will find anything we've potentially broken | 04:05 |
| jhesketh__ | (actually it might not because it's an object, but still) | 04:06 |
| fungi | probably a wise idea... with the explicit get() method shadow the one from getter/setter? | 04:06 |
| *** gokrokve has quit IRC | 04:07 | |
| jhesketh__ | as in leave the get method? | 04:07 |
| fungi | i'm wondering which ends up getting called | 04:07 |
| *** Sukhdev has quit IRC | 04:07 | |
| fungi | or maybe we're missing a decorator i was thinking was there. nevermind | 04:08 |
| jhesketh__ | oh right | 04:08 |
| * fungi needs to get some rest anyway. brain no thinky good | 04:09 | |
| jhesketh__ | not sure what decorator you're referring to, but taking a look at http://docs.python.org/2/reference/datamodel.html#object.__getitem__ implies that the __getitem__ method would only be used if we did yamlconfig['item'] | 04:09 |
| fungi | ahh, yeah, so an explicit yamlconfig.get('item') would be handled separately anyway | 04:10 |
| *** sarob has joined #openstack-infra | 04:10 | |
| fungi | which is all sorts of counterintuitive. fun | 04:10 |
| jhesketh__ | yeah | 04:10 |
| jhesketh__ | so what do you think about renaming the current 'def get' method to get_project_item? | 04:10 |
| fungi | i think that's a stellar idea | 04:11 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/jeepyb: Fix docimpact_target project selection https://review.openstack.org/74253 | 04:11 |
| jhesketh__ | done ^ | 04:11 |
| fungi | awesome | 04:12 |
| *** sarob has quit IRC | 04:16 | |
| *** lcheng has joined #openstack-infra | 04:16 | |
| *** Ryan_Lane has joined #openstack-infra | 04:19 | |
| jhesketh__ | quick question, does logstash actually serve any logs, or only its indexed knowledge (ie via kabana) | 04:29 |
| jhesketh__ | *kibana | 04:29 |
| clarkb | just indexed knowledge | 04:30 |
| clarkb | the size difference between compressed log and indexed log is massive | 04:31 |
| clarkb | better to serve compressed logs elsewhere | 04:31 |
| jhesketh__ | yep | 04:32 |
| jhesketh__ | thanks | 04:32 |
| *** gokrokve has joined #openstack-infra | 04:33 | |
| openstackgerrit | Bill Maxwell proposed a change to openstack-infra/jenkins-job-builder: Implements: Refactor of YamlParser.parse method https://review.openstack.org/70563 | 04:33 |
| *** gokrokve_ has joined #openstack-infra | 04:35 | |
| mordred | fugi, clarkb: you guys seen a nodepool bug where it'll leak nova keypairs if it hits quota issues? | 04:36 |
| notmyname | I'm seeing a very opaque error condition from jenkins. console log is very short: http://logs.openstack.org/63/71163/3/gate/gate-grenade-dsvm/6933a25/console.html | 04:37 |
| notmyname | on patch https://review.openstack.org/#/c/71163/ | 04:37 |
| *** gokrokve has quit IRC | 04:37 | |
| *** mgagne has joined #openstack-infra | 04:38 | |
| openstackgerrit | Bill Maxwell proposed a change to openstack-infra/jenkins-job-builder: Implements: Refactor of YamlParser.parse method https://review.openstack.org/70563 | 04:38 |
| *** gokrokve_ has quit IRC | 04:40 | |
| *** matsuhashi has joined #openstack-infra | 04:43 | |
| zaro | fungi: what should we do about that jenkins plugin job flakiness? | 04:44 |
| *** zhiyan has quit IRC | 04:46 | |
| *** banix has joined #openstack-infra | 04:47 | |
| *** skraynev_afk is now known as skraynev | 04:51 | |
| *** gokrokve has joined #openstack-infra | 04:52 | |
| *** zhiyan has joined #openstack-infra | 04:52 | |
| *** michchap has joined #openstack-infra | 04:55 | |
| *** gokrokve has quit IRC | 04:56 | |
| *** matsuhashi has quit IRC | 05:09 | |
| *** matsuhashi has joined #openstack-infra | 05:09 | |
| *** matsuhashi has quit IRC | 05:10 | |
| *** matsuhashi has joined #openstack-infra | 05:10 | |
| *** sarob has joined #openstack-infra | 05:13 | |
| *** sarob has quit IRC | 05:18 | |
| *** chandan_kumar has joined #openstack-infra | 05:20 | |
| *** dkliban has quit IRC | 05:23 | |
| *** gokrokve has joined #openstack-infra | 05:27 | |
| *** CaptTofu has joined #openstack-infra | 05:32 | |
| *** luis_ has quit IRC | 05:33 | |
| *** nosnos has quit IRC | 05:33 | |
| *** nosnos has joined #openstack-infra | 05:34 | |
| *** sarob has joined #openstack-infra | 05:35 | |
| *** matsuhashi has quit IRC | 05:36 | |
| *** CaptTofu has quit IRC | 05:37 | |
| *** matsuhashi has joined #openstack-infra | 05:37 | |
| *** matsuhas_ has joined #openstack-infra | 05:38 | |
| *** dkliban has joined #openstack-infra | 05:39 | |
| *** matsuhashi has quit IRC | 05:39 | |
| *** nicedice has quit IRC | 05:39 | |
| *** gokrokve has quit IRC | 05:41 | |
| *** gokrokve has joined #openstack-infra | 05:41 | |
| *** gokrokve_ has joined #openstack-infra | 05:42 | |
| *** dkliban has quit IRC | 05:44 | |
| *** gokrokve has quit IRC | 05:46 | |
| *** ArxCruz has quit IRC | 05:53 | |
| *** hdd_ has quit IRC | 05:59 | |
| *** mrda is now known as mrda_away | 06:02 | |
| *** banix has quit IRC | 06:06 | |
| *** nosnos_ has joined #openstack-infra | 06:10 | |
| *** nosnos has quit IRC | 06:10 | |
| *** dhellmann has quit IRC | 06:10 | |
| *** dhellmann has joined #openstack-infra | 06:12 | |
| *** jhesketh_ has quit IRC | 06:16 | |
| *** jhesketh__ has quit IRC | 06:17 | |
| *** markwash has joined #openstack-infra | 06:21 | |
| *** coolsvap has joined #openstack-infra | 06:26 | |
| *** jhesketh_ has joined #openstack-infra | 06:29 | |
| *** jhesketh__ has joined #openstack-infra | 06:29 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin https://review.openstack.org/73601 | 06:36 |
| *** sarob has quit IRC | 06:36 | |
| *** sarob has joined #openstack-infra | 06:36 | |
| *** DinaBelova_ is now known as DinaBelova | 06:39 | |
| *** sarob has quit IRC | 06:41 | |
| *** saju_m has joined #openstack-infra | 06:48 | |
| *** jcooley_ has quit IRC | 06:50 | |
| *** lcheng has quit IRC | 06:52 | |
| *** yolanda has quit IRC | 06:55 | |
| *** gokrokve_ has quit IRC | 06:56 | |
| *** gokrokve has joined #openstack-infra | 06:57 | |
| *** dstanek has quit IRC | 06:59 | |
| *** DinaBelova is now known as DinaBelova_ | 07:00 | |
| *** gokrokve has quit IRC | 07:01 | |
| *** jpeeler has quit IRC | 07:03 | |
| *** AaronGr is now known as AaronGr_Zzz | 07:05 | |
| *** sarob has joined #openstack-infra | 07:07 | |
| *** dpyzhov has joined #openstack-infra | 07:10 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack-dev/pbr: Remove unused _parse_mailmap() https://review.openstack.org/74279 | 07:14 |
| *** rlandy has joined #openstack-infra | 07:16 | |
| *** jpeeler has joined #openstack-infra | 07:17 | |
| *** jpeeler has joined #openstack-infra | 07:17 | |
| *** nosnos_ has quit IRC | 07:19 | |
| *** nosnos has joined #openstack-infra | 07:19 | |
| *** marun has quit IRC | 07:21 | |
| *** dpyzhov has quit IRC | 07:21 | |
| *** ociuhandu has joined #openstack-infra | 07:22 | |
| *** yolanda has joined #openstack-infra | 07:23 | |
| *** gokrokve has joined #openstack-infra | 07:27 | |
| *** e0ne has joined #openstack-infra | 07:27 | |
| *** jcooley_ has joined #openstack-infra | 07:29 | |
| *** dkliban has joined #openstack-infra | 07:29 | |
| *** gokrokve_ has joined #openstack-infra | 07:29 | |
| *** e0ne has quit IRC | 07:30 | |
| *** gokrokve has quit IRC | 07:32 | |
| *** CaptTofu has joined #openstack-infra | 07:33 | |
| *** gokrokve_ has quit IRC | 07:34 | |
| *** gokrokve has joined #openstack-infra | 07:35 | |
| *** CaptTofu has quit IRC | 07:37 | |
| *** gokrokve has quit IRC | 07:40 | |
| *** sarob has quit IRC | 07:40 | |
| *** markwash has quit IRC | 07:43 | |
| *** afazekas has quit IRC | 07:46 | |
| *** flaper87|afk is now known as flaper87 | 07:47 | |
| *** dkliban has quit IRC | 07:48 | |
| *** pblaho has joined #openstack-infra | 07:48 | |
| *** jcooley_ has quit IRC | 07:52 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 08:02 | |
| *** adam_g has quit IRC | 08:06 | |
| *** adam_g has joined #openstack-infra | 08:07 | |
| *** adam_g has quit IRC | 08:07 | |
| *** adam_g has joined #openstack-infra | 08:07 | |
| *** DinaBelova_ is now known as DinaBelova | 08:10 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 08:18 | |
| *** afazekas has joined #openstack-infra | 08:25 | |
| *** jgallard has joined #openstack-infra | 08:27 | |
| *** jcoufal has joined #openstack-infra | 08:27 | |
| *** mrmartin has joined #openstack-infra | 08:28 | |
| *** vkozhukalov has joined #openstack-infra | 08:30 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/devstack-gate: Archive config files along with logs https://review.openstack.org/69344 | 08:33 |
| *** gokrokve has joined #openstack-infra | 08:35 | |
| *** afazekas has quit IRC | 08:37 | |
| *** sarob has joined #openstack-infra | 08:37 | |
| *** gokrokve has quit IRC | 08:40 | |
| *** jhesketh_ has quit IRC | 08:41 | |
| *** jhesketh__ has quit IRC | 08:41 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 08:44 | |
| *** amotoki has joined #openstack-infra | 08:51 | |
| *** afazekas has joined #openstack-infra | 08:52 | |
| *** rossella-s has joined #openstack-infra | 08:52 | |
| *** vkozhukalov has quit IRC | 09:00 | |
| *** chandankumar_ has joined #openstack-infra | 09:02 | |
| *** chandan_kumar has quit IRC | 09:03 | |
| *** derekh has joined #openstack-infra | 09:04 | |
| *** yassine has joined #openstack-infra | 09:05 | |
| *** e0ne has joined #openstack-infra | 09:07 | |
| *** sarob_ has joined #openstack-infra | 09:08 | |
| *** sarob has quit IRC | 09:11 | |
| *** unicell has quit IRC | 09:14 | |
| *** jcooley_ has joined #openstack-infra | 09:18 | |
| *** jcooley_ has quit IRC | 09:23 | |
| *** fbo_away is now known as fbo | 09:23 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Fix docimpact_target project selection https://review.openstack.org/74253 | 09:23 |
| *** dpyzhov has joined #openstack-infra | 09:25 | |
| *** dizquierdo has joined #openstack-infra | 09:30 | |
| *** DinaBelova is now known as DinaBelova_ | 09:33 | |
| *** DinaBelova_ is now known as DinaBelova | 09:33 | |
| *** CaptTofu has joined #openstack-infra | 09:33 | |
| *** gokrokve has joined #openstack-infra | 09:36 | |
| *** saju_m has quit IRC | 09:37 | |
| *** saju_m has joined #openstack-infra | 09:38 | |
| *** CaptTofu has quit IRC | 09:38 | |
| *** e0ne_ has joined #openstack-infra | 09:38 | |
| *** jp_at_hp has joined #openstack-infra | 09:38 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove hardcoded direct-release project list https://review.openstack.org/74309 | 09:38 |
| *** e0ne__ has joined #openstack-infra | 09:40 | |
| *** e0ne_ has quit IRC | 09:40 | |
| *** e0ne_ has joined #openstack-infra | 09:40 | |
| *** sarob_ has quit IRC | 09:41 | |
| *** nosnos has quit IRC | 09:41 | |
| *** nosnos_ has joined #openstack-infra | 09:41 | |
| *** gokrokve has quit IRC | 09:41 | |
| *** e0ne has quit IRC | 09:42 | |
| *** johnthetubaguy has joined #openstack-infra | 09:44 | |
| *** andre__ has quit IRC | 09:44 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add d-g jobs for python-savannaclient https://review.openstack.org/74310 | 09:44 |
| *** andre__ has joined #openstack-infra | 09:44 | |
| *** e0ne__ has quit IRC | 09:45 | |
| *** saju_m has quit IRC | 09:46 | |
| *** talluri has joined #openstack-infra | 09:46 | |
| *** dpyzhov has left #openstack-infra | 09:51 | |
| *** markmc has joined #openstack-infra | 09:54 | |
| *** jcoufal has quit IRC | 10:00 | |
| *** saju_m has joined #openstack-infra | 10:05 | |
| *** sarob has joined #openstack-infra | 10:07 | |
| *** thomasbiege has joined #openstack-infra | 10:08 | |
| *** thomasbiege1 has joined #openstack-infra | 10:09 | |
| *** dpyzhov has joined #openstack-infra | 10:10 | |
| *** thomasbiege has quit IRC | 10:13 | |
| *** Xurong has quit IRC | 10:14 | |
| *** ociuhandu has quit IRC | 10:14 | |
| *** hashar has joined #openstack-infra | 10:17 | |
| *** david-lyle has quit IRC | 10:17 | |
| *** jcooley_ has joined #openstack-infra | 10:17 | |
| *** jp_at_hp has quit IRC | 10:17 | |
| *** jcoufal has joined #openstack-infra | 10:20 | |
| *** jcooley_ has quit IRC | 10:23 | |
| *** coolsvap has quit IRC | 10:28 | |
| *** chandan_kumar has joined #openstack-infra | 10:30 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 10:32 | |
| *** chandankumar_ has quit IRC | 10:34 | |
| *** gokrokve has joined #openstack-infra | 10:37 | |
| *** DinaBelova is now known as DinaBelova_ | 10:37 | |
| *** amcrn has quit IRC | 10:38 | |
| *** sarob has quit IRC | 10:41 | |
| *** Xurong has joined #openstack-infra | 10:42 | |
| *** gokrokve has quit IRC | 10:42 | |
| *** yolanda has quit IRC | 10:44 | |
| *** yolanda has joined #openstack-infra | 10:45 | |
| *** ociuhandu has joined #openstack-infra | 10:54 | |
| *** jp_at_hp has joined #openstack-infra | 10:55 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Add support for python 3-<3.3 https://review.openstack.org/73946 | 11:02 |
| *** mrmartin has quit IRC | 11:04 | |
| *** yamahata has quit IRC | 11:05 | |
| *** sarob has joined #openstack-infra | 11:07 | |
| *** talluri has quit IRC | 11:10 | |
| *** jcoufal has quit IRC | 11:10 | |
| *** talluri has joined #openstack-infra | 11:10 | |
| *** talluri has quit IRC | 11:15 | |
| *** jcoufal has joined #openstack-infra | 11:15 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: stop putting setup_* into separate log files https://review.openstack.org/74331 | 11:16 |
| *** jcooley_ has joined #openstack-infra | 11:16 | |
| *** unicell has joined #openstack-infra | 11:19 | |
| *** jgallard has quit IRC | 11:21 | |
| *** jcooley_ has quit IRC | 11:23 | |
| *** hashar has quit IRC | 11:24 | |
| *** yaguang has quit IRC | 11:29 | |
| *** CaptTofu has joined #openstack-infra | 11:34 | |
| *** lcestari has joined #openstack-infra | 11:36 | |
| *** CaptTofu has quit IRC | 11:36 | |
| *** CaptTofu has joined #openstack-infra | 11:37 | |
| *** gokrokve has joined #openstack-infra | 11:38 | |
| *** CaptTofu has quit IRC | 11:41 | |
| *** sarob has quit IRC | 11:41 | |
| *** gokrokve has quit IRC | 11:42 | |
| *** boris-42_ has quit IRC | 11:43 | |
| *** nosnos_ has quit IRC | 11:43 | |
| *** nosnos has joined #openstack-infra | 11:44 | |
| openstackgerrit | Justin Shepherd proposed a change to openstack-infra/config: Restricting chef-cookbook-chefspec job to spec dir https://review.openstack.org/74339 | 11:49 |
| *** Ryan_Lane has quit IRC | 12:00 | |
| *** Ryan_Lane has joined #openstack-infra | 12:01 | |
| *** hashar has joined #openstack-infra | 12:03 | |
| *** jcoufal has quit IRC | 12:05 | |
| *** sarob has joined #openstack-infra | 12:07 | |
| *** CaptTofu has joined #openstack-infra | 12:08 | |
| openstackgerrit | Vadim Rovachev proposed a change to openstack-infra/devstack-gate: Add change in README file according to changes in code https://review.openstack.org/74342 | 12:08 |
| *** DinaBelova_ is now known as DinaBelova | 12:10 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 12:10 | |
| *** ArxCruz has joined #openstack-infra | 12:14 | |
| *** talluri has joined #openstack-infra | 12:15 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove hardcoded direct-release project list https://review.openstack.org/74309 | 12:15 |
| *** thomasbiege1 has quit IRC | 12:16 | |
| *** che-arne has joined #openstack-infra | 12:20 | |
| *** dpyzhov has quit IRC | 12:20 | |
| *** dpyzhov has joined #openstack-infra | 12:20 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Remove hardcoded direct-release project list https://review.openstack.org/74309 | 12:23 |
| *** talluri has quit IRC | 12:27 | |
| *** lcostantino has joined #openstack-infra | 12:28 | |
| *** talluri has joined #openstack-infra | 12:28 | |
| *** alexpilotti has joined #openstack-infra | 12:31 | |
| *** talluri has quit IRC | 12:32 | |
| *** mrmartin has joined #openstack-infra | 12:35 | |
| *** e0ne has joined #openstack-infra | 12:36 | |
| *** e0ne_ has quit IRC | 12:36 | |
| *** gokrokve has joined #openstack-infra | 12:39 | |
| *** sarob has quit IRC | 12:39 | |
| *** smarcet has joined #openstack-infra | 12:40 | |
| *** jcoufal has joined #openstack-infra | 12:42 | |
| *** nosnos_ has joined #openstack-infra | 12:43 | |
| *** nosnos has quit IRC | 12:43 | |
| *** gokrokve has quit IRC | 12:43 | |
| *** rfolco has joined #openstack-infra | 12:44 | |
| mrmartin | SergeyLukjanov hi, may I ask you to review this: https://review.openstack.org/#/c/73549/ ? | 12:46 |
| SergeyLukjanov | mrmartin, hey, it's already in my backlog | 12:46 |
| mrmartin | ok, thanks | 12:46 |
| mrmartin | sorry for nagging you, but it is an important step required for a working openstackid deployment | 12:47 |
| *** lcostantino has quit IRC | 12:49 | |
| *** thomasbiege has joined #openstack-infra | 12:51 | |
| *** yamahata has joined #openstack-infra | 12:54 | |
| dhellmann | sdague: ping? | 12:54 |
| sdague | pong | 12:54 |
| dhellmann | good morning | 12:55 |
| dhellmann | I need to bounce some ideas off you about testing oslo.test, if you have a few minutes | 12:55 |
| sdague | sure | 12:56 |
| sdague | fire away | 12:56 |
| dhellmann | I *think* we really only need to run tests for gating oslo.test to ensure changes there don't break the projects' unit tests -- changes going the other way seems extremely unlikely | 12:57 |
| *** mflobo has quit IRC | 12:58 | |
| dhellmann | however, it is possible that a project could make changes in their unit tests that would be incompatible with an unreleased version of oslo.test | 12:58 |
| dhellmann | so if we can do symmetric gating, that would be best | 12:58 |
| dhellmann | I looked, but didn't see an existing gate job that seemed to be running the unit tests -- they all seem to just run tempest | 12:59 |
| dhellmann | did I miss one? | 12:59 |
| sdague | no, we don't cross gate on unit tests | 12:59 |
| dhellmann | right | 12:59 |
| sdague | so, unit test runs mostly come from the template | 12:59 |
| dhellmann | in this one case, that's what needs to happen | 12:59 |
| dhellmann | at least, I think so | 12:59 |
| sdague | so... honestly, I'm actually trying to get unit tests out of the gate | 13:00 |
| dhellmann | yeah, that makes sense for real code, but this is the unit test base class and fixture library | 13:00 |
| dhellmann | "real" == "production" | 13:00 |
| sdague | that being said, the way you'd do this is you'd add all the unit tests for all the dependent projects to oslo.test | 13:00 |
| dhellmann | yeah, that takes care of making sure we don't break the projects | 13:01 |
| sdague | because zuul basically dynamically pivots on job name | 13:01 |
| dhellmann | for the other way around, though, what do you think about installing oslo.test from source instead of a package, in the tox jobs for the projects? | 13:01 |
| *** bingbu has joined #openstack-infra | 13:01 | |
| sdague | I think that get messy | 13:01 |
| dhellmann | that would put the test in the check jobs, instead of the gate, which is ok | 13:01 |
| dhellmann | yeah | 13:02 |
| sdague | just because we've avoided that so far | 13:02 |
| dhellmann | well, we have, except we've been copying versions of that code into the projects instead | 13:02 |
| sdague | yep, I know | 13:02 |
| *** jp_at_hp has quit IRC | 13:02 | |
| dhellmann | I'm trying to figure out how to stop copying it there, and still have some level of symmetric testing | 13:02 |
| sdague | I'm pondering a bit | 13:02 |
| dhellmann | ok | 13:02 |
| *** pdmars has joined #openstack-infra | 13:04 | |
| sdague | so there is really a lot less symmetry than you think in the gate, especially when you look at the 100+ dependencies coming in | 13:04 |
| dhellmann | true | 13:04 |
| dhellmann | do you think it's enough to gate changes to oslo.test against the unit tests of the other projects? | 13:04 |
| sdague | yeh, I'm looking at the code now | 13:04 |
| dhellmann | either gate or a check job, which amounts to the same thing | 13:04 |
| *** matsuhas_ has quit IRC | 13:04 | |
| sdague | it seems small enough that I'd honestly figure out if there was a way to do that | 13:05 |
| sdague | and take the risk of a firedrill if we wedge something | 13:05 |
| sdague | because it's such a small amount of code | 13:05 |
| dhellmann | I can script that entirely within oslo.test's repo, and set up a tox env for it | 13:05 |
| dhellmann | yeah, I expect it to grow, but not hugely | 13:05 |
| *** afazekas has quit IRC | 13:05 | |
| dhellmann | ok, and I realized this morning, I'm going to have to rename it oslotest instead of oslo.test, because of the devstack namespace package issue | 13:06 |
| *** matsuhashi has joined #openstack-infra | 13:06 | |
| sdague | yeh | 13:06 |
| *** jcooley_ has joined #openstack-infra | 13:06 | |
| dhellmann | I had been thinking we'd install this from within devstack to use in the existing gate job, but ... | 13:06 |
| dhellmann | I can rename the deploy package, and we can rename the repo some time later | 13:06 |
| dhellmann | ok, if you're comfortable without the symmetry, I'll just work on a check job for the new library | 13:07 |
| dhellmann | thanks! | 13:07 |
| *** sarob has joined #openstack-infra | 13:07 | |
| dhellmann | sdague: one more thing, do we have a list of integrated projects anywhere other than the PROJECTS list in devstack-gate-wrap.sh? | 13:09 |
| sdague | well, remember, the integrated pipeline is dynamic. What's in devstack-gate-wrap is a super set | 13:10 |
| sdague | but there are combinations of projects that can enter the gate and not be linked | 13:10 |
| dhellmann | I need a way to have a list of projects to test against, without having to keep the list in the oslo.test repo and somewhere else | 13:10 |
| sdague | well, which projects include it today? | 13:11 |
| *** jcooley_ has quit IRC | 13:11 | |
| dhellmann | it's brand new, so nothing, but I want to set up the job before I start introducing it | 13:11 |
| dhellmann | I mean, I guess a lot of the projects copy the classes in, but I haven't made that list -- I'm assuming we want to test against all integrated projects | 13:11 |
| *** jgallard has joined #openstack-infra | 13:12 | |
| dhellmann | hmm, I wouldn't need to test against tempest or the mirror job or some of those others so maybe a separate list does make sense | 13:13 |
| *** jp_at_hp has joined #openstack-infra | 13:13 | |
| sdague | yeh, realistically we should really just test against the projects that use it. Otherwise we're just burning cycles | 13:17 |
| dhellmann | yeah, I expect that to be all or most of them, but we can keep a separate list | 13:20 |
| *** jroovers|afk has joined #openstack-infra | 13:21 | |
| *** jroovers has quit IRC | 13:21 | |
| *** eharney has joined #openstack-infra | 13:22 | |
| *** afazekas has joined #openstack-infra | 13:23 | |
| *** dpyzhov has quit IRC | 13:24 | |
| *** dprince has joined #openstack-infra | 13:24 | |
| *** dpyzhov has joined #openstack-infra | 13:25 | |
| *** matrohon has quit IRC | 13:32 | |
| *** prad has joined #openstack-infra | 13:33 | |
| *** jcoufal has quit IRC | 13:34 | |
| *** jcoufal has joined #openstack-infra | 13:34 | |
| *** dcramer_ has quit IRC | 13:37 | |
| *** andreaf has joined #openstack-infra | 13:39 | |
| *** sarob has quit IRC | 13:39 | |
| *** gokrokve has joined #openstack-infra | 13:39 | |
| *** jroovers has joined #openstack-infra | 13:41 | |
| *** sandywalsh has joined #openstack-infra | 13:44 | |
| *** jroovers|afk has quit IRC | 13:44 | |
| *** bingbu has quit IRC | 13:44 | |
| *** gokrokve has quit IRC | 13:44 | |
| *** matrohon has joined #openstack-infra | 13:46 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 13:48 |
| *** leifmadsen has quit IRC | 13:48 | |
| *** jroovers has quit IRC | 13:49 | |
| *** saju_m has quit IRC | 13:50 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Auth support https://review.openstack.org/73219 | 13:51 |
| *** ryanpetrello has joined #openstack-infra | 13:51 | |
| *** jroovers has joined #openstack-infra | 13:53 | |
| *** jroovers|afk has joined #openstack-infra | 13:54 | |
| *** jroovers has quit IRC | 13:57 | |
| *** gordc has joined #openstack-infra | 14:03 | |
| *** dstufft has quit IRC | 14:06 | |
| *** dstufft has joined #openstack-infra | 14:06 | |
| *** changbl has quit IRC | 14:06 | |
| *** w_ has quit IRC | 14:07 | |
| *** sarob has joined #openstack-infra | 14:07 | |
| *** leifmadsen has joined #openstack-infra | 14:09 | |
| *** zul has quit IRC | 14:09 | |
| *** julim has joined #openstack-infra | 14:09 | |
| *** zul has joined #openstack-infra | 14:11 | |
| *** mriedem has joined #openstack-infra | 14:12 | |
| *** luqas has joined #openstack-infra | 14:12 | |
| *** jungleboyj has quit IRC | 14:13 | |
| *** mfer has joined #openstack-infra | 14:14 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Clean up puppet (deploy LAMP / setup app config) https://review.openstack.org/69636 | 14:17 |
| *** CaptTofu has quit IRC | 14:17 | |
| *** jeckersb_gone is now known as jeckersb | 14:20 | |
| *** w_ has joined #openstack-infra | 14:21 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add jasondunsmore to heat-core https://review.openstack.org/73570 | 14:21 |
| ArxCruz | ALL: I'm getting a problem with zuul and statsd, it says that module statsd.statsd can't be found does anyone having the same problem ? | 14:23 |
| SergeyLukjanov | ArxCruz, hey, have you checked installed requirements? | 14:26 |
| ArxCruz | SergeyLukjanov: yes, it says statsd >= 1.0.0 <3.0 | 14:26 |
| *** dims has quit IRC | 14:26 | |
| ArxCruz | but it fails in extras.try_import('statsd.statsd') | 14:26 |
| *** dstanek has joined #openstack-infra | 14:29 | |
| *** hashar has quit IRC | 14:29 | |
| SergeyLukjanov | ArxCruz, heh, interesting | 14:29 |
| *** jgallard has quit IRC | 14:31 | |
| fungi | ArxCruz: are you sure you have the correct statsd? there are several python modules named statsd unfortunately | 14:32 |
| ArxCruz | fungi: 2.0.1 | 14:32 |
| ArxCruz | fungi: that's not the problem, I see statsd it's only to generate the graphics right ? | 14:32 |
| ArxCruz | fungi: my zuul daemon is starting but isn't creating the git repo in /var/lib/zuul/git for the projects I want to listen | 14:33 |
| fungi | i believe the zuul documentation mentions which particular statsd you need to use, and if you have additionally installed one of the others it can shadow the one you need in the global namespace so you end up importing the wrong one | 14:33 |
| *** hashar has joined #openstack-infra | 14:33 | |
| fungi | try starting an interactive python session and 'import statsd' followed by 'help(statsd)' | 14:34 |
| ArxCruz | fungi: import statsd I can | 14:35 |
| ArxCruz | but extras.try_import('statsd.statsd') return nothing | 14:35 |
| fungi | on our zuul server i see the last couple lines of the help output say that the version is 2.0.1 | 14:35 |
| fungi | that should confirm whether the statsd you get when importing it is the one you think you're using | 14:36 |
| fungi | whereas pip freeze can lie to you, because it only knows package names not necessarily what names teh modules are presenting in the global namespace with the usual search path | 14:37 |
| *** dcramer_ has joined #openstack-infra | 14:37 | |
| sdague | fungi: how do you feel about this - https://review.openstack.org/#/c/74331/ ? | 14:37 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 14:38 |
| *** protux has joined #openstack-infra | 14:39 | |
| *** yamahata has quit IRC | 14:39 | |
| *** oubiwann has joined #openstack-infra | 14:39 | |
| *** sarob has quit IRC | 14:39 | |
| *** dims has joined #openstack-infra | 14:40 | |
| gordc | hi folks, if anyone has some time, i was hoping i could get a pair of eyes on https://review.openstack.org/#/c/73603/ ... it's blocking me from checking in a bug fix currently. apologies for begging. | 14:40 |
| *** gokrokve has joined #openstack-infra | 14:40 | |
| *** yamahata has joined #openstack-infra | 14:43 | |
| *** gokrokve has quit IRC | 14:45 | |
| ArxCruz | fungi: thanks | 14:48 |
| fungi | ArxCruz: was that the problem? | 14:48 |
| ArxCruz | fungi: I give up, statsd isn't necessary to zuul works, only to generate the graphics (at least was what I understand) | 14:48 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 14:48 |
| ArxCruz | right now my problem is zuul isn't creating git repository in /var/lib/zuul/git | 14:49 |
| fungi | ArxCruz: also, if you're using zuul's master branch, be aware we merged some pretty radical changes yesterday which will need adjustments to your config (see the news file for details) | 14:49 |
| ArxCruz | :/ | 14:49 |
| ArxCruz | fungi: do you have the change ? | 14:50 |
| fungi | they had been up for review for a while... not sure if you reviewed them though | 14:50 |
| * fungi gets you the link | 14:50 | |
| fungi | ArxCruz: https://review.openstack.org/#/c/71628/7/NEWS.rst | 14:51 |
| ArxCruz | fungi: probably don't | 14:51 |
| ArxCruz | too busy to turn reports on :( | 14:51 |
| fungi | ahh | 14:52 |
| *** banix has joined #openstack-infra | 14:52 | |
| *** jergerber has joined #openstack-infra | 14:52 | |
| fungi | well, the good news here is that if you're running a very busy zuul (like we are) you can distribute the merge reference calculations and git serving out to additional machines connected to zuul's gearman server over the network | 14:53 |
| fungi | and your job workers connect directly to the merge workers to retrieve your zuul refs | 14:53 |
| SergeyLukjanov | fungi, morning, do you already have any visible improvements of how zuul works with mergers? | 14:54 |
| fungi | SergeyLukjanov: not yet. the load won't probably get heavy enough to see the effects until later in the week | 14:54 |
| SergeyLukjanov | fungi, i3 will help us to see results ;) | 14:55 |
| ArxCruz | fungi: ohhhh.... now we have a zuul-merger daemon... duh! I didn't start this one lol | 14:55 |
| fungi | SergeyLukjanov: mostly what we should see is gate resets going more quickly when they happen, and less pileup in the event/result queue because zuul will be spending less time on merge calculations during gate resets | 14:55 |
| fungi | ArxCruz: yep, start the new daemon and make sure your configuration is adjusted for the new section (several confg options specific to the merge operation moved to the new section for the merger daemon) | 14:56 |
| SergeyLukjanov | fungi, that's nice | 14:56 |
| *** nosnos_ has quit IRC | 14:57 | |
| *** thomasbiege has quit IRC | 14:57 | |
| koolhead17 | hi all | 14:58 |
| openstackgerrit | gordon chung proposed a change to openstack-infra/config: make gate-pycadf-python33 non-voting https://review.openstack.org/73603 | 14:58 |
| *** jcooley_ has joined #openstack-infra | 14:59 | |
| ArxCruz | fungi: thanks, it's working now :) | 15:00 |
| openstackgerrit | gordon chung proposed a change to openstack-infra/config: drop gate-pycadf-python33 https://review.openstack.org/73603 | 15:00 |
| *** gokrokve has joined #openstack-infra | 15:01 | |
| *** jaypipes has joined #openstack-infra | 15:02 | |
| *** jungleboyj has joined #openstack-infra | 15:03 | |
| miqui | jeblair: hi , are you guys broken by zuul latest changes? cuz we are .... | 15:05 |
| *** jcooley_ has quit IRC | 15:05 | |
| anteaya | hi koolhead17 | 15:06 |
| fungi | miqui: see my discussion with ArxCruz above | 15:06 |
| miqui | jeblair: looking | 15:06 |
| fungi | miqui: also, we don't *think* we're broken by the latest zuul changes, but then again we were the ones who merged them | 15:07 |
| jaypipes | another great new phrase from sdague: "blind meatgrinder behavior" | 15:07 |
| sdague | :) | 15:07 |
| fungi | jaypipes: i'm not even sure i want to know the context | 15:07 |
| *** sarob has joined #openstack-infra | 15:07 | |
| miqui | fungi: so are you installing a tagged version of zuul that is stable or pullling the latest and greatest ( because we are ....) | 15:07 |
| jaypipes | fungi: :) | 15:07 |
| * fungi is clearly behind on his ml reading | 15:08 | |
| fungi | miqui: we're using/writing/maintaining the tip of the master branch of zuul | 15:08 |
| miqui | fungi: ok... | 15:08 |
| fungi | miqui: see the news file updates about needing to adjust your configuration for the new zuul merger daemon and needing to make sure that gets started (via its associated initscript) | 15:09 |
| fungi | miqui: the zuul documentation was also updated as part of the commits which changed its behavior | 15:10 |
| *** jgrimm has joined #openstack-infra | 15:11 | |
| *** esker has joined #openstack-infra | 15:11 | |
| dims | "blind meatgrinder behavior" - Nice! :) | 15:11 |
| *** dkliban has joined #openstack-infra | 15:11 | |
| *** mgagne has quit IRC | 15:12 | |
| annegentle | clarkb: or fungi: For the Operations Guide, based on https://wiki.openstack.org/wiki/GerritJenkinsGithub#Merge_Commits do I only get one "chance" to move changes from the feature/edits branch to master | 15:14 |
| annegentle | er, that's meant to be a question | 15:14 |
| *** lttrl has quit IRC | 15:15 | |
| miqui | fungi: ok... thanks rtfm.... | 15:15 |
| *** smarcet has left #openstack-infra | 15:15 | |
| *** beagles is now known as beagles_brb | 15:16 | |
| fungi | annegentle: no, you can incrementally merge back and forth between them if you like, but you should probably merge mostly in one direction if you're synchronizing, and have one last merge before deleting/replacing the feature branch | 15:16 |
| annegentle | fungi: okay yeah I was going to go first master > feature/edits | 15:17 |
| fungi | annegentle: normally with a feature branch you'd merge changes from master to the feature branch periodically to keep the feature branch from falling too far behind/diverging too much, and then merge the feature branch back to the master branch when you're ready to stop using it | 15:17 |
| annegentle | fungi: just didn't want to shoot myself in my future foot or some such | 15:17 |
| fungi | right | 15:17 |
| annegentle | fungi: thanks! Saved my morning heart attack :) | 15:18 |
| annegentle | fungi: now I will try this fancy-pants merging | 15:18 |
| fungi | fair warning, the review will look sort of wierd | 15:18 |
| annegentle | fungi: scared | 15:19 |
| annegentle | fungi: but ready :) | 15:19 |
| fungi | nothing to be scared about. the computer is your friend and protector. don't allow the mutants and secret societies tell you otherwise | 15:20 |
| mtreinish | sdague: so I've got a global requirements question for the grizzly/branch question | 15:24 |
| sdague | mtreinish: shoot | 15:24 |
| koolhead17 | jaypipes: anteaya :) | 15:24 |
| sdague | though I suspect the answer is ... global requirements doesn't work quite right on grizzly | 15:24 |
| sdague | because it happened post grizzly | 15:24 |
| *** luisg has joined #openstack-infra | 15:25 | |
| fungi | mtreinish: yeah, i gave up trying to sync it since we're about to turn it off anyway | 15:25 |
| mtreinish | sdague: yeah I think that's the answer too which wedged stable/havana grenade :( | 15:25 |
| sdague | got a failed run somewhere I can look at? | 15:25 |
| mtreinish | sdague: so I've got this https://review.openstack.org/#/c/74176/ change but it still tries to install the capped version here: http://logs.openstack.org/76/74176/1/check/check-devstack-dsvm-cells/979398e/console.html#_2014-02-17_21_59_17_185 | 15:26 |
| fungi | mtreinish: basically, we didn't have a consistent list of requirements for all the integrated projects during the grizzly cycle, so building that is the hardest part (i mostly got it working, but then we needed backports of various jobs to keep it in sync) | 15:26 |
| mtreinish | which leads to an exercises fail here: http://logs.openstack.org/76/74176/1/check/check-devstack-dsvm-cells/979398e/console.html#_2014-02-17_22_01_08_871 | 15:26 |
| mtreinish | fungi: ok maybe the answer here is to just skip the offending exercise for grenade | 15:27 |
| sdague | so, honestly, did cells test ever pass on grizzly? | 15:27 |
| mtreinish | sdague: it did when it just ran the exercises I think | 15:28 |
| mtreinish | but the failure is the same on grenade for stable/havana | 15:28 |
| sdague | on grizzly? | 15:28 |
| sdague | it only runs exercises | 15:28 |
| mtreinish | I thought so, but maybe I'm wrong | 15:28 |
| fungi | and are we afraid that in the next six weeks we're going to merge a backport to grizzly which makes cells less functional there than it already is? | 15:29 |
| mtreinish | fungi: yeah it's probably safe to rip it out for the stable/grizzly branch | 15:29 |
| sdague | so this is honestly more about the fact that the devstack exercises are super fragile | 15:30 |
| mtreinish | but it still doesn't fix the underlying issue with grenade on stable/havana | 15:30 |
| *** markwash has joined #openstack-infra | 15:30 | |
| sdague | mtreinish: do you have one of those reviews? | 15:30 |
| * fungi rechecks notes from the summit to confirm it's even 6 weeks | 15:30 | |
| mtreinish | sdague: no not really because if you look how it fails it just runs swiftclient and get's an import error starting the binary | 15:30 |
| sdague | because my fix for stable grizzly would be to remove the cells job from it completely, because it's not a cells issue that's happening | 15:30 |
| mtreinish | sdague: sure one sec let me dig it up | 15:30 |
| fungi | nope... not even... "When to deprecate grizzly -> at icehouse -3" https://etherpad.openstack.org/p/stable-havana-ideas | 15:31 |
| fungi | so ~3 weeks | 15:31 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Doc for project dependencies in gate https://review.openstack.org/66025 | 15:32 |
| fungi | and then grizzly is tagged eol and ripped out of everything anyway | 15:32 |
| fungi | so definitely don't spend too much time on testing improvements for grizzly 3 weeks from eol | 15:32 |
| mtreinish | sdague: http://logs.openstack.org/76/72576/1/check/check-grenade-dsvm-neutron/bf9977e/console.html#_2014-02-17_18_36_40_289 | 15:33 |
| mtreinish | fungi: I'm just curious where does that leave us for grenade on havana though? | 15:33 |
| *** markmcclain has joined #openstack-infra | 15:34 | |
| sdague | mtreinish: so give me the high level on what's up with swift client | 15:34 |
| sdague | I can probably work us backwards to a solution | 15:34 |
| ArxCruz | fungi: also, I'm getting Forbidden error when I try to access the status page, I'm using the same apache conf that you have in puppet | 15:34 |
| sdague | is the issue that we need an old version at this part of the process? | 15:34 |
| fungi | ArxCruz: that sounds like your apache configuration may not be quite right | 15:34 |
| sdague | or that we need a newer version | 15:35 |
| mtreinish | sdague: so grizzly req has a version cap of <2 but swiftclient just had a major release of 2 | 15:35 |
| *** jnoller has joined #openstack-infra | 15:35 | |
| mtreinish | we install swiftclient from master | 15:35 |
| fungi | ArxCruz: or else the files apache wants to serve you aren't readable by the user under which it's running | 15:35 |
| mtreinish | so things get weird when running swiftclient because the script is from 1.9 but the code is 2.0.2 | 15:35 |
| mtreinish | err the bin/swift script is the old version but the code in the swiftclient namespace is master | 15:35 |
| *** david-lyle has joined #openstack-infra | 15:36 | |
| ArxCruz | fungi: which is weird, apache starts fine :/ | 15:36 |
| sdague | because bin/swift doesn't get replaced when we pip install the old version | 15:36 |
| *** dolphm has joined #openstack-infra | 15:37 | |
| fungi | mtreinish: right, one of the reasons requirements wrangling on grizzly is a bear (pun intended) is that we were capping versions of clients in server reqs lists | 15:37 |
| mtreinish | sdague: the reverse but yeah | 15:37 |
| fungi | yet we want to test clients with tip of master, so it's somewhat incompatible with the new model | 15:37 |
| mtreinish | sdague: the pip install replaces bin/swift but not the other code | 15:38 |
| mtreinish | which leads to that import error exception because the Exception Class it tries to import doesn't exist in master | 15:38 |
| sdague | oh.... because pip stupidness | 15:38 |
| mtreinish | sdague: yep | 15:38 |
| mtreinish | fungi: yeah it's annoying | 15:39 |
| *** sarob has quit IRC | 15:39 | |
| fungi | mtreinish: that ended up being a big part of what led me to consider it relatively intractable for the projected gain | 15:39 |
| sdague | so how explody would things get if we uncapped glance? | 15:39 |
| sdague | because that's the only thing holding us down | 15:40 |
| *** virmitio has joined #openstack-infra | 15:40 | |
| mtreinish | sdague: it's not just glance, it's horizon, glance, glanceclient, and I think there is another 1 maybe 2 things that have the capped version | 15:41 |
| sdague | it looks like it's just glance and horizon | 15:42 |
| sdague | remember, glanceclient is on master | 15:42 |
| *** chandan_kumar has quit IRC | 15:43 | |
| mtreinish | sdague: yeah you're right I misread a glanceclient <2 requires as glanceclient requires swiftclient<2 (that's not confusing at all) | 15:43 |
| *** afazekas has quit IRC | 15:44 | |
| sdague | so, the question is, will that explode those projects? | 15:44 |
| *** sarob has joined #openstack-infra | 15:45 | |
| mtreinish | sdague: I have no idea and honestly I'm not sure it's worth the risk for the grizzly branch that expires in ~3 weeks | 15:45 |
| *** pcrews has joined #openstack-infra | 15:45 | |
| mtreinish | wouldn't just skipping the exercise in grenade fix it | 15:45 |
| mtreinish | because we don't use the swiftclient cli anywhere | 15:45 |
| mtreinish | except the exercises | 15:46 |
| sdague | sure, you could propose that fix | 15:47 |
| sdague | it's 2 exercises actually | 15:47 |
| sdague | that would let us see if it's a bigger issue | 15:47 |
| *** wenlock has joined #openstack-infra | 15:48 | |
| mtreinish | sdague: sure will do | 15:48 |
| *** gokrokve has quit IRC | 15:49 | |
| *** gokrokve has joined #openstack-infra | 15:49 | |
| *** mgagne has joined #openstack-infra | 15:49 | |
| mordred | ++ | 15:49 |
| mordred | I'm in support of that plan | 15:49 |
| mordred | also, we shoudl really resurrect the master-clients/stable-servers work - you know, next time we have a lul :) | 15:50 |
| fungi | next time we have a lulz | 15:51 |
| *** rcleere has joined #openstack-infra | 15:51 | |
| *** AaronGr_Zzz is now known as AaronGr | 15:51 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Clean up puppet (deploy LAMP / setup app config) https://review.openstack.org/69636 | 15:52 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add single-use py3k-precise nodes https://review.openstack.org/73846 | 15:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Run most non-sensitive jobs on single-use workers https://review.openstack.org/73732 | 15:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove obsolete static job workers https://review.openstack.org/73852 | 15:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Shift more nodepool nodes onto rax-dfw https://review.openstack.org/73853 | 15:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add more py3k-precise nodes https://review.openstack.org/73850 | 15:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Use py3k-precise nodes https://review.openstack.org/73851 | 15:53 |
| sdague | lul... this is a new and interesting word that I'm not familiar with | 15:53 |
| *** coolsvap has joined #openstack-infra | 15:54 | |
| *** jcooley_ has joined #openstack-infra | 15:54 | |
| *** gokrokve has quit IRC | 15:54 | |
| * fungi knows not this lul of which you speak | 15:56 | |
| *** zhiyan is now known as zhiyan_ | 15:56 | |
| * mordred throws a lul at fungi, misses by a derp | 15:57 | |
| fungi | 73732 there has a chance to merge-conflict on just about any job configuration change, so we probably should merge it soon if we want to merge it at all | 15:58 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 15:59 | |
| fungi | and we'll also want to keep an eye out for any pending change after that adding a job with one of the old reusable node labels | 15:59 |
| *** jcooley_ has quit IRC | 15:59 | |
| *** atiwari has joined #openstack-infra | 16:01 | |
| *** amotoki_ has joined #openstack-infra | 16:01 | |
| mtreinish | sdague: https://review.openstack.org/#/c/74419/ | 16:01 |
| sdague | ok, lets see how the tests run on that | 16:02 |
| sdague | I think separately proposing a remove of the devstack-cells job on grizzly would be a good idea | 16:03 |
| *** DinaBelova is now known as DinaBelova_ | 16:03 | |
| *** MIDENN_ has quit IRC | 16:03 | |
| sdague | because we won't be able to merge any grizzly code until that happens | 16:03 |
| jeblair | fungi: zuul merger sitrep? | 16:04 |
| jeblair | fungi: (which is my way of saying, 'i hope this morning finds you well!') | 16:05 |
| *** UtahDave has joined #openstack-infra | 16:05 | |
| fungi | jeblair: parsing error elided. YES! very well | 16:05 |
| fungi | things are going smashingly | 16:05 |
| *** ok_delta has joined #openstack-infra | 16:05 | |
| fungi | the only weird report i found was http://logs.openstack.org/63/71163/3/gate/gate-grenade-dsvm/6933a25/logs/devstack-gate-setup-workspace-old.txt | 16:06 |
| fungi | looks like a job failing to clone nova | 16:06 |
| fungi | but i have no direct evidence to suggest that it's related at all to the zuul changes | 16:06 |
| fungi | we've had a few zuul downstreams who do cd pop in and ask why their zuul suddenly broke, but they seem squared away now too | 16:07 |
| anteaya | jeblair: I'm thinking of getting zuul merger stirep? on a t-shirt | 16:08 |
| anteaya | *sitrep? | 16:08 |
| jeblair | fungi: that is weird; it should never have to clone nova; it should rsync it from /opt/git | 16:09 |
| *** nicedice has joined #openstack-infra | 16:09 | |
| jeblair | fungi: that doesn't seem to be universal though | 16:09 |
| fungi | which makes me think the rsync might have gone awry | 16:10 |
| jeblair | fungi: loks like it didn't do the rsync because /opt/git/openstack/nova didn't exist | 16:10 |
| sdague | there were actually quite a number of jobs that failed that way, it was the reason I proposed - https://review.openstack.org/#/c/74331/ this morning | 16:10 |
| jeblair | fungi: but the other projects did | 16:10 |
| *** thomasbiege has joined #openstack-infra | 16:10 | |
| fungi | right, which is extra strange | 16:11 |
| *** thomasbiege has quit IRC | 16:11 | |
| jeblair | sdague: do you have a link to another one? | 16:11 |
| mtreinish | sdague: actually looking at it the zuul layout says cells should be nonvoting for grizzly (and everything else too) | 16:11 |
| sdague | mtreinish: no, that's the tempest job | 16:11 |
| *** smarcet has joined #openstack-infra | 16:11 | |
| sdague | there is a devstack job | 16:11 |
| sdague | jeblair: not off the top of my head, I hit some in my review queue this morning | 16:12 |
| mtreinish | sdague: it's everything: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n316 | 16:12 |
| *** mdenny has joined #openstack-infra | 16:13 | |
| *** homeless has quit IRC | 16:14 | |
| *** luqas has quit IRC | 16:14 | |
| *** lcheng has joined #openstack-infra | 16:14 | |
| sdague | mtreinish: where is the job def for that? | 16:14 |
| mtreinish | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#n315 | 16:15 |
| sdague | mtreinish: so I think because it doesn't have a limitter, it's voting on everything | 16:15 |
| *** UtahDave has quit IRC | 16:15 | |
| jeblair | sdague: how about we have the main script say "check this log file" if it detects an error? | 16:15 |
| jeblair | sdague: that way the madness is hidden from view normally, but when there is an error, it's, uh, reported. :) | 16:16 |
| sdague | jeblair: honestly, my preference remains a single file. Because in my experience people don't read :) | 16:17 |
| sdague | but if we go down that path, then it also needs local timestamp wrapping and logstashing as well | 16:17 |
| *** pblaho has quit IRC | 16:17 | |
| *** homeless has joined #openstack-infra | 16:17 | |
| sdague | because not having this content in logstash is the reason I can't answer your question about what other jobs failed this way | 16:18 |
| *** jcoufal-mobile has joined #openstack-infra | 16:18 | |
| jeblair | sdague: i have a serious concern about all of the non-error messages that are in those logs. people often think that the jenkins file descriptor warning is the reason their job failed.... | 16:18 |
| *** ArxCruz has quit IRC | 16:18 | |
| jeblair | sdague: and the reason that we moved that into separate log files in the first place is because, well, if they think the jenkins warning is important, what do they make of this: | 16:19 |
| *** medieval1 has joined #openstack-infra | 16:19 | |
| jeblair | fatal: http://zm01.openstack.org/p/openstack-dev/grenade/info/refs not found: did you run git update-server-info on the server? | 16:19 |
| *** ArxCruz has joined #openstack-infra | 16:19 | |
| sdague | jeblair: right, I'm actually trying to get that sorted by making devstack able to run under errexit | 16:19 |
| sdague | so we'll fail early | 16:19 |
| sdague | the issue really is that the failure is far off screen by the end of the job | 16:20 |
| *** mrmartin has quit IRC | 16:20 | |
| jeblair | sdague: that's pretty obviously the cause of the failure, right? it says 'fatal'! :) | 16:20 |
| jeblair | sdague: devstack fail early is great, but i don't think that addresses this | 16:20 |
| sdague | so then why not filter out that log message instead | 16:20 |
| *** jcoufal-mobile has quit IRC | 16:20 | |
| *** medieval1 has quit IRC | 16:20 | |
| jeblair | sdague: because _i_ need it to diagnose failures | 16:20 |
| *** jcoufal-mobile has joined #openstack-infra | 16:20 | |
| sdague | because I think if that remains in the log, and we send people to the log | 16:20 |
| *** medieval1 has joined #openstack-infra | 16:21 | |
| jeblair | sdague: i think if we sent people to the log, we would only do it on an actual error, and presumably the error at the bottom of the lost would be it. | 16:21 |
| *** sandywalsh has quit IRC | 16:21 | |
| *** dstanek has quit IRC | 16:22 | |
| *** rossella-s has quit IRC | 16:22 | |
| sdague | my instinct is it won't work, based on the questions I see from folks. But if that's what you want to do, that's fine. | 16:23 |
| sdague | I would also clean up those git calls to not put the words fatal in there. Just wrap it if it's expected behavior. | 16:23 |
| jeblair | sdague: i'm more interested in making this bulletproof. this part _always_ needs to work.... | 16:24 |
| jeblair | sdague: patch git? | 16:24 |
| sdague | filter the output | 16:24 |
| jeblair | sdague: so when something goes wrong here, i'm the one who needs to read these log files | 16:24 |
| jeblair | sdague: i'm okay if someone comes in here and says "i think this broke and i don't understand why" | 16:24 |
| *** jcoufal-mobile has quit IRC | 16:24 | |
| jeblair | sdague: as long as i have the information i need to figure out what happened | 16:25 |
| sdague | so I think this is the crux of the issue | 16:25 |
| *** jcoufal-mobile has joined #openstack-infra | 16:25 | |
| jeblair | sdague: separate log files with unfiltered output are important for me to be able to diagnose that | 16:25 |
| jeblair | sdague: for the same reason that openstack devs would like individual nova/etc logs, and unfiltered | 16:25 |
| sdague | because we're in bash, segregating info between users / user cases is hard. | 16:26 |
| jeblair | sdague: i agree that timestamps would be helpful. i would love it if it were in logstash. and i think the main script should report errors instead of hiding them. | 16:26 |
| sdague | so honestly, I'd rather just have one big dump and let everyone figure out things that are or are not important in it | 16:26 |
| jeblair | sdague: i think we can do all of that without losing the benefits we currently have. | 16:26 |
| sdague | ok, I don't really see benefits with the current system. I mostly see it as me scratching my head for 5 minutes then remembering, oh right, there are other logs | 16:27 |
| *** UtahDave has joined #openstack-infra | 16:27 | |
| jeblair | sdague: so let's have it output the error. the benefits is that with the current system, i can diagnose the error. you are proposing that we should make it harder for me to do so. | 16:27 |
| *** mrodden1 has quit IRC | 16:27 | |
| sdague | it's harder if it's all in console? | 16:27 |
| jeblair | sdague: way harder | 16:28 |
| sdague | what if we tee it instead then? | 16:28 |
| sdague | so it doesn't impact you having targetted logs, but we'll also have them in console | 16:28 |
| jeblair | sdague: that addresses one problem but doesn't address the error-spam. | 16:28 |
| jeblair | sdague: i'd be okay with it, but i think it's a bad user experience. | 16:29 |
| jeblair | sdague: what about 'cat' on error? | 16:29 |
| *** jcoufal-mobile has quit IRC | 16:29 | |
| sdague | so, honestly, I think that's the least of our user experience issues with the devstack log | 16:29 |
| *** jcoufal-mobile has joined #openstack-infra | 16:29 | |
| sdague | jeblair: it doesn't let you see things as they are happening in jenkins, which I actually find pretty useful | 16:29 |
| *** jcoufal-mobile has quit IRC | 16:30 | |
| *** jcoufal-mobile has joined #openstack-infra | 16:30 | |
| *** markwash has quit IRC | 16:30 | |
| *** dstanek has joined #openstack-infra | 16:30 | |
| *** dkehn has quit IRC | 16:30 | |
| *** dkehn has joined #openstack-infra | 16:31 | |
| *** changbl has joined #openstack-infra | 16:32 | |
| *** matsuhashi has quit IRC | 16:32 | |
| *** dkehn has quit IRC | 16:33 | |
| *** sabari_ has joined #openstack-infra | 16:33 | |
| jeblair | sdague: my preference is that we timestamp and index the separate logs and have the main script output a real error message. i'm okay with teeing them, but i think another 3400 lines (5700 lines for grenade) of meaningless output isn't going to be considered helpful by developers. | 16:33 |
| jeblair | if i were reading this, i would much rather it just have one line that said "oops, i couldn't do the git checkout stuff; if you want to see why, look here: ..." | 16:34 |
| *** markwash has joined #openstack-infra | 16:34 | |
| jeblair | (and that line, btw, would be indexed in logstash) | 16:35 |
| *** markwash has quit IRC | 16:35 | |
| *** dkliban has quit IRC | 16:36 | |
| *** dkehn has joined #openstack-infra | 16:37 | |
| jeblair | i just logged into a random az3 devstack-precise machine, and its /opt/git/openstack/nova looks fine | 16:38 |
| *** sandywalsh has joined #openstack-infra | 16:38 | |
| sdague | so 3400 lines is only about a 15% add to the console. Realistically, I think having a better segmentation of phases in the console would be the best thing to do. I have some thoughts on that, but no real time on poking at it. | 16:39 |
| *** mrodden has joined #openstack-infra | 16:39 | |
| jaypipes | jeblair: I've got a job in gearman that doesn't have any workers, but has a job waiting. It's the merger:merge queue. In my Zuul debug log, I see a message like this: http://paste.openstack.org/show/66943/. If I run the job myself in jenkins, it completes successfully, but for some reason, I can't seem to get the upstream recheck comment to trigger the job, and the only thing weird I see is that merger:merge queue.. | 16:41 |
| jaypipes | . any ideas? | 16:41 |
| jeblair | jaypipes: read the top of the NEWS file; you need to start the zuul-merger process | 16:42 |
| jaypipes | jeblair: ah... | 16:43 |
| *** andreaf has quit IRC | 16:43 | |
| jaypipes | jeblair: oh... just added yesterday :) no wonder! | 16:43 |
| *** luqas has joined #openstack-infra | 16:44 | |
| fungi | i've started suggesting that people running cd from tip of master (for any project really) should be subscribing to the proposed changes so they know what's coming | 16:45 |
| jaypipes | fungi: good idea :) | 16:45 |
| *** beagles_brb is now known as beagles | 16:45 | |
| *** dprince has quit IRC | 16:45 | |
| fungi | even if you just skim the commit messages for the stuff which is in review, you'll have some inkling of what's lurking out there about to ruin your morning | 16:45 |
| *** jcooley_ has joined #openstack-infra | 16:46 | |
| jaypipes | jeblair: for the external testing platform, any reason I can't run the merger on the same VM as the other zuul process? | 16:46 |
| jaypipes | jeblair: I presume this is just a scaling thing? | 16:46 |
| mfer | fungi jeblair I was wondering if there's anything I can do to help https://review.openstack.org/#/c/71956/ along. It's about .NET stuff with OpenStack and we're starting to connect with others on this work. | 16:46 |
| jeblair | jaypipes: nope. it even says so in the NEWS file. :) | 16:46 |
| *** dpyzhov has quit IRC | 16:46 | |
| jeblair | jaypipes: nope == no reason you can not do that == yes you can do that. :) | 16:46 |
| jaypipes | jeblair: heh, once again you call me out on my lack of RTFM ;P | 16:47 |
| *** DinaBelova_ is now known as DinaBelova | 16:48 | |
| jeblair | mfer: we have a backlog of reviews and unfortunately new stackforge projects are not our highest priority | 16:48 |
| *** jcoufal has quit IRC | 16:48 | |
| mfer | jeblair i entirely understand that. | 16:49 |
| jeblair | mfer: i can promise we'll get to it (i review oldest first), so it won't get lost. but i couldn't say when. | 16:49 |
| *** jcoufal has joined #openstack-infra | 16:49 | |
| mfer | jeblair thanks. are we talking days or weeks? | 16:49 |
| jeblair | mfer: if i had to guess, probably not this week, maybe next. | 16:51 |
| *** sandywalsh has quit IRC | 16:52 | |
| mfer | thanks. i like guestimates so i know when to check back in :) | 16:52 |
| *** gokrokve has joined #openstack-infra | 16:52 | |
| *** boris-42_ has joined #openstack-infra | 16:54 | |
| *** gokrokve_ has joined #openstack-infra | 16:54 | |
| *** UtahDave has quit IRC | 16:56 | |
| *** gokrokve has quit IRC | 16:57 | |
| openstackgerrit | sahid proposed a change to openstack-infra/elastic-recheck: Adds query for bug #1278988 https://review.openstack.org/72713 | 16:57 |
| *** boris-42_ has quit IRC | 16:59 | |
| *** sabari_ has quit IRC | 16:59 | |
| *** sabari_ has joined #openstack-infra | 16:59 | |
| *** dangers_away is now known as dangers | 17:01 | |
| *** vkozhukalov has joined #openstack-infra | 17:02 | |
| *** sabari_ is now known as sabari | 17:02 | |
| *** jcoufal-mobile has quit IRC | 17:03 | |
| *** sarob has quit IRC | 17:04 | |
| *** sandywalsh has joined #openstack-infra | 17:04 | |
| *** boris-42_ has joined #openstack-infra | 17:04 | |
| *** sarob has joined #openstack-infra | 17:04 | |
| anteaya | we have a patch failing on git being unavailable: https://jenkins05.openstack.org/job/gate-grenade-dsvm/735/console | 17:05 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Don't check logs on stable/grizzly https://review.openstack.org/71285 | 17:05 |
| fungi | anteaya: yep, there's a bug open for that. we frequently see dns lookup failures against rackspace's recursive resolvers recently | 17:05 |
| anteaya | k | 17:05 |
| * anteaya looks for bug | 17:06 | |
| *** markmc has quit IRC | 17:06 | |
| jeblair | fungi: wow, that doesn't usually happen so early | 17:06 |
| fungi | yep--that one hardly had a chance to make it off the beach | 17:06 |
| anteaya | https://bugs.launchpad.net/openstack-ci/+bug/1270382 | 17:07 |
| *** dkliban has joined #openstack-infra | 17:07 | |
| anteaya | this the right one? | 17:07 |
| anteaya | shot in the boat | 17:07 |
| *** dkliban is now known as dliban_afk | 17:07 | |
| *** dliban_afk is now known as dkliban_afk | 17:07 | |
| anteaya | where art thou bug bot? | 17:07 |
| anteaya | is that soren's bot? | 17:07 |
| fungi | anteaya: yep, that one | 17:08 |
| jeblair | jnoller: we've noticed an increase in failures due to rax dns servers being unavailable. is there a rate limit, or quota we could be hitting at a host or account level? or are we noticing a rax service degredation? | 17:08 |
| anteaya | k thanks, back in it goes | 17:08 |
| jeblair | pvo: ^ ? | 17:08 |
| jnoller | jeblair: I noticed DNS issues too | 17:08 |
| jnoller | jeblair: let me look - IAD right? | 17:08 |
| pvo | jeblair: standby | 17:09 |
| fungi | jnoller: that last one was dfw | 17:09 |
| jnoller | ok | 17:09 |
| *** sarob has quit IRC | 17:09 | |
| dhellmann | are we publishing binary eggs for projects now? https://pypi.python.org/pypi/pyghmi | 17:09 |
| *** krtaylor has quit IRC | 17:09 | |
| fungi | jnoller: pvo: we're tracking it in https://launchpad.net/bugs/1270382 at the moment, but can run a query against logstash for regions and times if it will help | 17:09 |
| dhellmann | or is that a wheel, maybe? | 17:09 |
| *** amotoki_ has quit IRC | 17:10 | |
| fungi | dhellmann: i think that must have been manually uploaded | 17:10 |
| anteaya | fungi: adding a reverify apparently lets tests continue | 17:11 |
| dhellmann | fungi: ok, it's breaking the ironic unit tests in some cases | 17:11 |
| anteaya | I guess I need to snipe it and send it back in | 17:11 |
| dhellmann | fungi: https://bugs.launchpad.net/ironic/+bug/1281385 | 17:11 |
| *** coolsvap1 has joined #openstack-infra | 17:11 | |
| anteaya | will snipe and then reverify | 17:11 |
| fungi | dhellmann: Author: Jarrod Johnson Package Index Owner: jbjohnso (we don't have access to upload pyghmi releases even if we wanted it) | 17:12 |
| dhellmann | fungi: ah, ok, I saw it on stackforge so assumed releases went through openstackci | 17:12 |
| *** coolsvap1 has quit IRC | 17:12 | |
| fungi | dhellmann: seems not | 17:12 |
| dhellmann | fungi: I'll try to get in touch with him or devenanda, thanks | 17:12 |
| *** coolsvap1 has joined #openstack-infra | 17:12 | |
| *** coolsvap1 has quit IRC | 17:12 | |
| *** coolsvap1 has joined #openstack-infra | 17:13 | |
| jeblair | fungi: i think https://review.openstack.org/#/c/73840/ is ready for your aprv | 17:14 |
| *** coolsvap has quit IRC | 17:14 | |
| fungi | jeblair: nibalizer: wow--neat! | 17:14 |
| *** coolsvap1 is now known as coolsvap | 17:15 | |
| openstackgerrit | Jay Pipes proposed a change to openstack-infra/config: Adds ! defined() guards around a2mod declarations https://review.openstack.org/74443 | 17:15 |
| fungi | nibalizer: you're pretty comfortable that the postgresql module 3.0.0->3.1.0 change will be non-impacting for our use? | 17:16 |
| *** cadenzajon has joined #openstack-infra | 17:17 | |
| fungi | i'll do a quick check of the licenses on those additional modules | 17:17 |
| *** changbl has quit IRC | 17:19 | |
| ArxCruz | fungi: I'm getting these errors: 2014-02-18 17:19:20,484 DEBUG zuul.Merger: Unable to find commit for ref master/Z1965c208f | 17:19 |
| jeblair | on the nodepool front, i have found that the cleanupServer method in the image build causes rss to jump from 36380 to 293872; continuing to narrow down | 17:20 |
| ArxCruz | fungi: http://paste.openstack.org/show/66956/ | 17:20 |
| fungi | ArxCruz: that doesn't _say_ error (says debug) | 17:20 |
| ArxCruz | so, is that normal okay ? | 17:20 |
| jeblair | ArxCruz: yeah, that's normal | 17:20 |
| ArxCruz | ok, cool | 17:20 |
| *** matrohon has quit IRC | 17:20 | |
| jeblair | ArxCruz: basically it's looking at the git repo to see if it's already staged a commit for that zuul ref; if it has it will use it as a basis, if not, it will create it | 17:21 |
| *** marun has joined #openstack-infra | 17:22 | |
| *** yassine has quit IRC | 17:24 | |
| nibalizer | fungi: yea i reviewed the changelog | 17:24 |
| fungi | nibalizer: i figured you probably did! (i just have to ask these things anyway) | 17:25 |
| nibalizer | puppetlabs modules are strictly semver'd so 3.0.x -> 3.1.x should only be enhancements not any breaking changes | 17:25 |
| anteaya | fungi jeblair sdague so getting some datapoints on auto check | 17:25 |
| anteaya | markmcclain rebased this patch: https://review.openstack.org/#/c/69110/6 | 17:26 |
| anteaya | and it is back in the check queue | 17:26 |
| nibalizer | yea, plus puppet module install puppetlabs-puppetdb will be upgrading the postgres module to 3.1.0 anyways, which might have been a fun debugging experience for someone | 17:26 |
| fungi | anteaya: right, because that rebased patchset hasn't been confirmed working yet | 17:26 |
| anteaya | we are just curious about what behavour should be expected in this circumstance | 17:26 |
| anteaya | that was what I was wondering | 17:26 |
| anteaya | though it is a trivial change | 17:27 |
| anteaya | it is not? | 17:27 |
| fungi | anteaya: since it has an approval vote, once jenkins posts back a +1 from the recheck it will add the change into teh gate | 17:27 |
| anteaya | Automatically re-added by Gerrit trivial rebase detection script. | 17:27 |
| anteaya | okay | 17:27 |
| anteaya | just bringing it up for discussion | 17:27 |
| pleia2 | morning | 17:27 |
| anteaya | since the neutronclient release is waiting on it | 17:27 |
| anteaya | pleia2: morning | 17:27 |
| anteaya | so where I to snipe it out to get it back into the gate faster, it still would have to go through check again | 17:28 |
| fungi | anteaya: this makes sure that *every* change patchset passes check jobs before gating, even if it's just a trivial rebase | 17:28 |
| fungi | anteaya: correct | 17:28 |
| anteaya | okay | 17:28 |
| anteaya | just wanting to ensure this is intended behaviour | 17:29 |
| anteaya | and not something that isn't known | 17:29 |
| *** CaptTofu has joined #openstack-infra | 17:29 | |
| clarkb | morning | 17:31 |
| *** krtaylor has joined #openstack-infra | 17:32 | |
| anteaya | morning clarkb | 17:34 |
| jaypipes | hmm, after pulling in all new zuul puppet stuff (for zuul-merger, et al), now zuul-server doesn't want to stay up. when starting it, the process dies after a few seconds. doing an strace on it reveals that some lockfile (presumably the pidfile) cannot be acquired: http://paste.openstack.org/show/66963/ | 17:35 |
| *** hashar has quit IRC | 17:35 | |
| *** fcarpenter has joined #openstack-infra | 17:40 | |
| fcarpenter | mordred https://review.openstack.org/#/c/74173/ seems stuck (approved yesterday)? | 17:41 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Skip exercises that use swiftclient cli on grizzly https://review.openstack.org/74451 | 17:41 |
| clarkb | mtreinish: sdague: grizzly never passed cells + exercises | 17:41 |
| clarkb | mtreinish: sdague: it does however pass the tempest flavor for grizzly | 17:41 |
| clarkb | the d-g cells jobs are arranged to accomodate that | 17:41 |
| ArxCruz | fungi: I need a little help, I configure zuul, it's fine, it's starting the jobs, it's not fetching the changes, it says that http://my.zuul.server/p/openstac/nova doesn't exist... | 17:42 |
| fungi | fcarpenter: i think the approval for that happened in the middle of a zuul restart | 17:42 |
| *** thomasbiege has joined #openstack-infra | 17:42 | |
| *** luqas has quit IRC | 17:43 | |
| davidlenwell | fungi: how do we unstuck it ? | 17:43 |
| jeblair | AssertionError: len(["b1be38ffe282a36d5a0eae61a75facc037183a32\t\t'refs/changes/75/74175/3' of ssh://review.openstack.org:29418/stackforge/clouddocs-maven-plugin\n"]) != len(["RSA host key for IP address '2001:4800:780d:509:3bc3:d7f6:ff04:39f0' not in list of known hosts.", '', ' * branch refs/changes/75/74175/3 -> FETCH_HEAD']) | 17:43 |
| sdague | clarkb: ok, well there is a new issue given the swift version bump. But we'll see if mtreinish unsticks things first | 17:43 |
| jeblair | davidlenwell: approve it again | 17:43 |
| davidlenwell | I don't have +2 | 17:44 |
| fungi | davidlenwell: i reenqueued it | 17:44 |
| jeblair | fungi, clarkb: ^ it looks like we might want to add the ipv6 addr to the known_hosts file too | 17:44 |
| *** e0ne has quit IRC | 17:44 | |
| *** e0ne has joined #openstack-infra | 17:45 | |
| jeblair | fungi, clarkb: but that's a one-time warning; it will have failed that change, but zm02 should be good to go for subsequent changes | 17:45 |
| *** gyee has joined #openstack-infra | 17:45 | |
| davidlenwell | thanks guys | 17:46 |
| fcarpenter | yes thank you | 17:46 |
| jeblair | fungi: i think i may not have been clear in my comment on https://review.openstack.org/#/c/73732/ | 17:46 |
| jeblair | fungi: if multiple regex jobs match, then the later ones override the parameter function; so it doesn't call the 1st then the 2nd, but rather just the 2nd | 17:47 |
| fungi | jeblair: oh, so parameter functions aren't cumulative? | 17:47 |
| fungi | yeah, i'll rework it now, knowing that | 17:47 |
| fungi | thanks! | 17:47 |
| jeblair | fungi: right; so therefore later regex job parameter functions need to call earlier ones if you still want the behavior from them | 17:47 |
| fungi | so any job ends up with one and only one parameter function | 17:47 |
| jeblair | correct | 17:48 |
| fungi | fix on the way | 17:48 |
| jeblair | fungi: other than that lgtm, and if we nudge clarkb we can aprv it to avoid conflicts | 17:48 |
| * clarkb is caught up on sb looking at things to aprv now | 17:48 | |
| clarkb | 73732? | 17:48 |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Add Forrest Carpenter to launchpad/email https://review.openstack.org/74173 | 17:48 |
| *** e0ne has quit IRC | 17:49 | |
| *** fcarpenter has quit IRC | 17:49 | |
| *** dkliban_afk is now known as dkliban | 17:49 | |
| clarkb | oh looks like that needs changes | 17:50 |
| * clarkb waits for them patiently | 17:50 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add single-use py3k-precise nodes https://review.openstack.org/73846 | 17:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Run most non-sensitive jobs on single-use workers https://review.openstack.org/73732 | 17:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove obsolete static job workers https://review.openstack.org/73852 | 17:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Shift more nodepool nodes onto rax-dfw https://review.openstack.org/73853 | 17:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add more py3k-precise nodes https://review.openstack.org/73850 | 17:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Use py3k-precise nodes https://review.openstack.org/73851 | 17:53 |
| fungi | your patience is rewarded | 17:53 |
| *** ok___delta has joined #openstack-infra | 17:53 | |
| clarkb | fungi: jeblair: any chance https://review.openstack.org/#/c/72509/ can get reviewed today? I will probably self approve EOD today | 17:53 |
| fungi | grr... and it needs another rebase. hold on | 17:54 |
| jeblair | fungi: zm02 errored on an infra config change | 17:54 |
| clarkb | 72509 is the last piece and make elasticsearch + logstash as happy as possible again process | 17:54 |
| jeblair | it got the rsa host key warning again; that's weird | 17:54 |
| clarkb | jeblair: MitM attack >_> | 17:54 |
| fungi | jeblair: that was probably 73732. it said it needed a rebase when it doesn't seem to | 17:55 |
| *** jroovers|afk has quit IRC | 17:56 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:56 | |
| *** dpyzhov has joined #openstack-infra | 17:57 | |
| *** ok_delta has quit IRC | 17:57 | |
| jeblair | ohhh.... | 17:57 |
| jeblair | fungi, clarkb: ssh isn't updating known_hosts to add the ipv6 addr | 17:57 |
| jeblair | probably because it's using the old format and not the hash-based one | 17:57 |
| *** thomasbiege has quit IRC | 17:58 | |
| *** dpyzhov has quit IRC | 17:58 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add puppetdb to a new puppetdb host https://review.openstack.org/73840 | 17:58 |
| clarkb | jeblair: weird | 17:59 |
| clarkb | jeblair: can we exec ssh-keyscan in a non vulnerable way to add the key hashed | 18:00 |
| clarkb | keyscan is the wrong tool | 18:00 |
| *** dprince has joined #openstack-infra | 18:02 | |
| jeblair | clarkb: i think we could add the hashed version as a blob, but i think it includes the ip address, so there's the issue of having the ip address stored in hiera. that's not ideal. | 18:03 |
| jeblair | we could disable ip address matching | 18:03 |
| *** lcheng has quit IRC | 18:03 | |
| *** yamahata has quit IRC | 18:03 | |
| jeblair | clarkb, fungi: i wonder if we should put it in /etc/ssh/ssh_known_hosts instead | 18:04 |
| fungi | we could just add the ipv4 and ipv6 addresses (comma-separated, no space) after the fqdn | 18:04 |
| *** sarob has joined #openstack-infra | 18:05 | |
| fungi | jeblair: i'm a big proponent of managing /etc/ssh/known_hosts on server farms. did it that way for years | 18:05 |
| fungi | without a good dnssec/sshfp base to work from, it's the next best way to make sure all users on a system don't get mitm'd for hosts you already know about | 18:05 |
| jeblair | i would really like sshfp. :( | 18:05 |
| jeblair | fungi: so let's (1) add the ip address to the user file, then (2) longer term move to using /etc instead; how's that sound? | 18:06 |
| fungi | jeblair: swimmingly | 18:06 |
| clarkb | sshfp depends on dnssec for trust right? | 18:06 |
| clarkb | jeblair: sounds good | 18:06 |
| fungi | clarkb: in short, yes. there are workarounds, but they're dirty, dirty workarounds | 18:06 |
| jeblair | i'll work on change 1 | 18:07 |
| *** changbl has joined #openstack-infra | 18:07 | |
| *** mwagner_lap has joined #openstack-infra | 18:07 | |
| clarkb | fungi: does that mean your stack is ready for rereview? | 18:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add gerrit ip addrs to zuul's known_hosts https://review.openstack.org/74457 | 18:10 |
| SergeyLukjanov | folks, could you please confirm that [1] will make savanna and it's client gated together (I mean all patches will be simultaneously tested in gate pipeline) | 18:10 |
| SergeyLukjanov | [1] https://review.openstack.org/#/c/74310/1/modules/openstack_project/files/zuul/layout.yaml | 18:10 |
| fungi | clarkb: zuul's still testing teh rebase | 18:11 |
| *** mrmartin has joined #openstack-infra | 18:11 | |
| jeblair | SergeyLukjanov: yes; zuul will even tell you; see the test result here: http://logs.openstack.org/10/74310/1/check/gate-config-layout/bad34c5/console.html | 18:12 |
| jeblair | SergeyLukjanov: and look for this line: 2014-02-18 10:04:12.289 | INFO:zuul.DependentPipelineManager: <ChangeQueue gate: openstack/python-savannaclient, openstack/savanna> | 18:12 |
| SergeyLukjanov | jeblair, oh, awesome, thanks for the tip | 18:13 |
| *** vkozhukalov has quit IRC | 18:13 | |
| jeblair | SergeyLukjanov: (if you scroll up, you'll see the really big openstack integrated gate is the 2nd line of that section. the first line is the even bigger shared change queue of projects that call 'gate-noop') | 18:13 |
| *** Ajaeger has joined #openstack-infra | 18:14 | |
| SergeyLukjanov | jeblair, yup, see it | 18:15 |
| *** derekh has quit IRC | 18:15 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:15 | |
| *** jgallard has joined #openstack-infra | 18:15 | |
| SergeyLukjanov | shame on me - never see this section before :( | 18:15 |
| Ajaeger | Hi infra team! The Image API v2 link (http://docs.openstack.org/api/openstack-image-service/2.0/content/) is broken, and won't be fixed til this goes through: https://review.openstack.org/#/c/73690 . Is there a chance to get this reviewed and merged, please? | 18:15 |
| SergeyLukjanov | jeblair, looks like we have too many noop projects ;) | 18:16 |
| *** lcheng has joined #openstack-infra | 18:16 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 18:16 | |
| mordred | hahahaha | 18:16 |
| * mordred loves the shared gate-noop queue | 18:16 | |
| jeblair | :) | 18:17 |
| jeblair | zuul should probably have an internal noop, but it hasn't been a high priority | 18:17 |
| SergeyLukjanov | jeblair, oh, I thought that it's because some issues with internal approach :) | 18:18 |
| SergeyLukjanov | Ajaeger, I see that you converting some job from maven to freestyle, AFAIK it'll not work w/o manual removal of old job | 18:19 |
| SergeyLukjanov | Ajaeger, oh, I see you note in commit message too :) | 18:19 |
| *** thomasbiege has joined #openstack-infra | 18:20 | |
| *** sarob_ has joined #openstack-infra | 18:21 | |
| jeblair | Ajaeger: https://review.openstack.org/#/c/73690/ lgtm | 18:21 |
| *** markwash has joined #openstack-infra | 18:21 | |
| SergeyLukjanov | I have a useless manage-project fix attempt today TL;DR I have tried to find some issues in manage-projects script today by reading it and trying to test locally, but I don't see any possible errors except race that we all already seen | 18:22 |
| Ajaeger | SergeyLukjanov: only openstack-api-ref is changed that way, the rest not. | 18:22 |
| SergeyLukjanov | Ajaeger, yup, I just saw the note after reading the diff :) | 18:23 |
| *** sarob has quit IRC | 18:23 | |
| Ajaeger | SergeyLukjanov: Thanks for bringing it up - I wanted to say it here so that it won't be forgotten to do as well.. | 18:23 |
| *** sarob_ has quit IRC | 18:23 | |
| Ajaeger | Not everybody reads the commit message till the end ;) | 18:23 |
| *** sarob has joined #openstack-infra | 18:24 | |
| *** luqas has joined #openstack-infra | 18:25 | |
| clarkb | Ajaeger: gerrit does a wierd thing in my browser where it scrolls to the patchset area by default | 18:29 |
| clarkb | which ahs trained me to not read the commit message first :( | 18:29 |
| *** sarob has quit IRC | 18:29 | |
| *** jp_at_hp has quit IRC | 18:29 | |
| Ajaeger | clarkb: shall I paste it here for you ? | 18:29 |
| *** sarob has joined #openstack-infra | 18:29 | |
| clarkb | Ajaeger: no, I will read it :) | 18:29 |
| Ajaeger | clarkb: That's a weird browser experience you have ;( | 18:29 |
| * Ajaeger had to start chromium today for one page since firefox was hanging - chromium worked just fine ;( | 18:30 | |
| *** vkozhukalov has joined #openstack-infra | 18:30 | |
| *** dizquierdo has quit IRC | 18:31 | |
| *** hemna has joined #openstack-infra | 18:31 | |
| SergeyLukjanov | Ajaeger, your patch is lgtm for me too | 18:32 |
| clarkb | fungi: will Ajaeger's change interfere with your node swaps? | 18:32 |
| hemna | sdague, any chance I can get a +3 today for my low version number patch? https://review.openstack.org/#/c/73727/ | 18:32 |
| fungi | clarkb: probably--i haven't looked at it yet, but basically any change which adds a new job will | 18:32 |
| SergeyLukjanov | clarkb, fungi, yup, I think it will | 18:32 |
| *** ok___delta has quit IRC | 18:33 | |
| fungi | which is probably somewhere around 25% of all infra/config changes | 18:33 |
| SergeyLukjanov | old job use precise | 18:33 |
| SergeyLukjanov | Ajaeger, heh, I'm using safari @ osx for more than 3 years w/o any big issues ;) | 18:33 |
| clarkb | fungi: in that case I haven't approved Ajaeger;s change but it lgtm, I will let you offer guidance on rebase direction | 18:34 |
| clarkb | I am not going to rereview your stack | 18:34 |
| sdague | hemna: done | 18:34 |
| Ajaeger | fungi: my patch adds a new job. | 18:34 |
| Ajaeger | Is there anything I can do right now with the patch - or do I have to wait? | 18:34 |
| hemna | sdague, thank you | 18:35 |
| *** luqas has quit IRC | 18:35 | |
| fungi | sorry, sort of heads-down poking at jenkins masters all day | 18:35 |
| *** mrmartin has quit IRC | 18:35 | |
| *** coolsvap has quit IRC | 18:36 | |
| fungi | Ajaeger: if you want to rebase on https://review.openstack.org/73732 and switch your node types from precise to bare-precise that would help tremendously | 18:36 |
| Ajaeger | fungi: I'll give it a try. | 18:36 |
| clarkb | s/not/now/ | 18:36 |
| clarkb | silly fingers typing changing the meaning of things completely | 18:36 |
| fungi | funny what a difference one little letter makes | 18:37 |
| *** nati_ueno has joined #openstack-infra | 18:37 | |
| *** thomasbiege has quit IRC | 18:37 | |
| Ajaeger | will bare-precise have all the packages installed we need for docu jobs? | 18:37 |
| *** sarob_ has joined #openstack-infra | 18:37 | |
| Ajaeger | maven, fonts? | 18:37 |
| fungi | clarkb: yeah, this is a stack of things i think we want to cram through before the feature freeze change ramp up | 18:37 |
| fungi | Ajaeger: they're built from the same puppet manifests, so in theory yes | 18:38 |
| *** sarob has quit IRC | 18:38 | |
| *** khyati_ has joined #openstack-infra | 18:38 | |
| Ajaeger | fungi: Ok ;) | 18:38 |
| fungi | Ajaeger: the only major difference is that they're one-time-use slaves and we discard them after each job is run and use a fresh one | 18:38 |
| clarkb | fungi: yup, I hope to cram it in real soon | 18:38 |
| nibalizer | fungi: thanks for approving my change (puppetdb), whats the next step? How do we get that onto a machine? | 18:38 |
| clarkb | hopefully +A before the meeting if I can manage to review it properly | 18:38 |
| nibalizer | or will all that be covered at the meeting ? | 18:39 |
| clarkb | nibalizer: is it on the meeting agenda? if so we should be able to go over it there | 18:39 |
| nibalizer | clarkb: link to the agenda? | 18:39 |
| nibalizer | or is the agenda in a bot? | 18:39 |
| fungi | though we could also potentially just go over it in here unless it's something which really needs to be hashed out in the team meeting | 18:40 |
| clarkb | nibalizer: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 18:40 |
| clarkb | fungi: true | 18:40 |
| fungi | nibalizer: i can try to launch a new puppetdb.openstack.org server here in a bit once i come up for air | 18:41 |
| Ajaeger | What's the best way to rebase: git review -d 73732 (fungi's patch); git review -d 73690 (my patch); git rebase review/jeremy_stanley/single-use - and then send the patch for review? | 18:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Add gerrit ip addrs to zuul's known_hosts https://review.openstack.org/74457 | 18:41 |
| jeblair | DEBUG:nodepool.ProviderManager:Manager hpcloud-az1 running task <nodepool.provider_manager.ListKeypairsTask object at 0x2e88ed0> | 18:41 |
| jeblair | DEBUG:requests.packages.urllib3.connectionpool:"GET /v1.1/10409882459003/os-keypairs HTTP/1.1" 200 24692239 | 18:41 |
| fungi | Ajaeger: -x instead of -d on the second patch | 18:41 |
| nibalizer | fungi: awesome | 18:41 |
| *** sarob_ has quit IRC | 18:42 | |
| fungi | Ajaeger: that would cherry-pick your change from gerrit on top of mine | 18:42 |
| jeblair | fungi, clarkb: ^ that's what causes nodepool to use 250m of memory | 18:42 |
| fungi | jeblair: keypair management?!? srsly | 18:42 |
| jeblair | fungi: yep. still tracing; not sure if it's nodepool or novaclient, but we're getting really close. :) | 18:43 |
| Ajaeger | fungi: Ah, thanks! | 18:43 |
| *** e0ne has joined #openstack-infra | 18:43 | |
| jeblair | and that, at least, is something we can do without a 1-hour test/debug cycle | 18:43 |
| clarkb | jeblair: woot | 18:43 |
| *** sarob has joined #openstack-infra | 18:44 | |
| *** mrmartin has joined #openstack-infra | 18:44 | |
| clarkb | fungi: comment inline on your change, let me know what you think | 18:45 |
| *** uvirtbot has joined #openstack-infra | 18:46 | |
| fungi | clarkb: thanks, looking | 18:46 |
| fungi | clarkb: we could whitelist the -noop jobs in layout.yaml to have reusable_node | 18:47 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Remove maven usage from api-jobs.yaml https://review.openstack.org/73690 | 18:47 |
| fungi | jeblair: does that seem safe? | 18:47 |
| Ajaeger | Rebased on top-of fungi's patch - hope it is fine now ^. | 18:47 |
| clarkb | fungi: or leave them on precise and keep a few of those nodes around? | 18:47 |
| clarkb | basically just bit sinks | 18:48 |
| fungi | that gets us back to potential slave agent communication failures with those... | 18:49 |
| clarkb | oh right, hrm | 18:49 |
| fungi | though in theory we also have that with our long-term slaves runnign sensitive jobs too | 18:50 |
| clarkb | maybe we stick with what you have for now and update zuul to do noop jobs internally | 18:50 |
| fungi | that seems better all around, at the expense of some additional node waste for now | 18:50 |
| *** cadenzajon has quit IRC | 18:50 | |
| fungi | i contemplated it initially, but have a feeling the number of -noop jobs we run out of the total proportion of jobs run is probably not that huge | 18:51 |
| clarkb | I can update my vote/approve if jeblair doesn't have any better ideas | 18:51 |
| fungi | the majority of the changes we test are official openstack projects because they're far higher volume, and few of them use -noop | 18:51 |
| clarkb | give him a few minutes here before I do that | 18:51 |
| clarkb | ya | 18:51 |
| fungi | so it felt to me like premature optimization | 18:52 |
| *** cadenzajon has joined #openstack-infra | 18:52 | |
| *** pblaho has joined #openstack-infra | 18:52 | |
| *** beagles is now known as beagles_brb | 18:53 | |
| *** w_ is now known as olaph | 18:53 | |
| *** balar has quit IRC | 18:54 | |
| *** balar has joined #openstack-infra | 18:54 | |
| *** sarob has quit IRC | 18:54 | |
| jeblair | i'm torn, but i think overall we shouldn't let noop hold up the march of progress | 18:57 |
| *** hogepodge has joined #openstack-infra | 18:57 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Clean up puppet (deploy LAMP / setup app config) https://review.openstack.org/69636 | 18:58 |
| *** talluri has joined #openstack-infra | 18:58 | |
| *** lcheng has quit IRC | 18:58 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Enable docs for python-savannaclient https://review.openstack.org/74470 | 18:59 |
| *** sarob has joined #openstack-infra | 18:59 | |
| *** protux_ has joined #openstack-infra | 18:59 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Allow devstack-precise-check for non-voting jobs https://review.openstack.org/71103 | 19:00 |
| clarkb | jeblair: /me approves | 19:00 |
| fungi | meeting time! | 19:00 |
| jeblair | fungi: you are fast | 19:00 |
| fungi | heh | 19:00 |
| *** protux has quit IRC | 19:00 | |
| *** talluri has quit IRC | 19:02 | |
| *** talluri has joined #openstack-infra | 19:02 | |
| *** dstufft has quit IRC | 19:03 | |
| *** melwitt has joined #openstack-infra | 19:04 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/config: Add Eavesdrop bot to #openstack-keystone https://review.openstack.org/74472 | 19:05 |
| *** dstufft has joined #openstack-infra | 19:06 | |
| *** talluri has quit IRC | 19:06 | |
| pleia2 | morganfainberg: thanks re: reviewing the reviewday json \o/ | 19:07 |
| morganfainberg | pleia2, sure thing. | 19:08 |
| morganfainberg | pleia2, there are some odd choices, but i know they were actually choices | 19:08 |
| *** weshay has quit IRC | 19:08 | |
| *** thomasbiege has joined #openstack-infra | 19:08 | |
| morganfainberg | so, out of curiosity how do i get uvirtbot into a new channel? | 19:08 |
| *** weshay has joined #openstack-infra | 19:09 | |
| morganfainberg | #openstack-keystone would like the bot hanging around | 19:09 |
| *** sarob_ has joined #openstack-infra | 19:09 | |
| fungi | morganfainberg: you ask soren | 19:09 |
| SergeyLukjanov | morganfainberg, have you added openstack-infra to channel founders? (I've placed a comment toyour change) | 19:09 |
| morganfainberg | soren, ping ^ uvirbot is awesome and keystone would love it in #openstack-keystone | 19:09 |
| morganfainberg | SergeyLukjanov, not yet, doing all of that slowly | 19:10 |
| morganfainberg | SergeyLukjanov, but it is on the list todo in the next few minutes | 19:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Run most non-sensitive jobs on single-use workers https://review.openstack.org/73732 | 19:11 |
| *** sarob has quit IRC | 19:11 | |
| morganfainberg | SergeyLukjanov, done now | 19:11 |
| SergeyLukjanov | morganfainberg, I just duplicate my comment to irc ;) | 19:11 |
| morganfainberg | SergeyLukjanov, i actually was just looking for the "name" to add earlier. but yes, it is done | 19:12 |
| SergeyLukjanov | morganfainberg, thx | 19:13 |
| *** beagles_brb is now known as beagles | 19:13 | |
| morganfainberg | SergeyLukjanov, and if there is a way to kindof fast-track the eavesdrop bit, we would be very appreciative, since we've already moved conversation from -dev over | 19:16 |
| morganfainberg | and don't want to lose too much data | 19:16 |
| morganfainberg | SergeyLukjanov, :) | 19:16 |
| *** lcostantino has joined #openstack-infra | 19:17 | |
| SergeyLukjanov | morganfainberg, you should ping other infra core folks to review/approve it | 19:17 |
| *** jeckersb is now known as jeckersb_gone | 19:17 | |
| morganfainberg | jeblair, fungi, pleia2, clarkb, https://review.openstack.org/#/c/74472/ if you please, keystone-core will continue to love you guys (even more)... ok ok maybe the rest of them wont, but I will | 19:18 |
| SergeyLukjanov | morganfainberg, :) | 19:19 |
| *** amcrn has joined #openstack-infra | 19:20 | |
| *** che-arne has quit IRC | 19:24 | |
| *** ok___delta has joined #openstack-infra | 19:27 | |
| *** nati_ueno has quit IRC | 19:29 | |
| *** nati_ueno has joined #openstack-infra | 19:29 | |
| *** shardy is now known as shardy_afk | 19:32 | |
| *** jcooley_ has quit IRC | 19:33 | |
| *** ok___delta has quit IRC | 19:33 | |
| *** jcooley_ has joined #openstack-infra | 19:33 | |
| *** oubiwann has quit IRC | 19:35 | |
| *** ArxCruz has quit IRC | 19:35 | |
| soren | morganfainberg: Alrighty. | 19:36 |
| *** johnthetubaguy has quit IRC | 19:37 | |
| *** thomasbiege1 has joined #openstack-infra | 19:37 | |
| *** jeckersb_gone is now known as jeckersb | 19:37 | |
| *** fbo is now known as fbo_away | 19:37 | |
| morganfainberg | soren, awesome! :) tyvm | 19:38 |
| *** nati_uen_ has joined #openstack-infra | 19:38 | |
| *** jcooley_ has quit IRC | 19:38 | |
| *** thomasbiege has quit IRC | 19:41 | |
| *** boris-42_ has quit IRC | 19:42 | |
| gordc | SergeyLukjanov: if you have time, can you re-review this: https://review.openstack.org/#/c/73603/ to make sure i didn't miss/mess anything. | 19:42 |
| *** nati_ueno has quit IRC | 19:42 | |
| SergeyLukjanov | gordc, added to backlog | 19:42 |
| *** jgallard has quit IRC | 19:42 | |
| gordc | SergeyLukjanov: much appreciated. :) | 19:42 |
| *** boris-42_ has joined #openstack-infra | 19:46 | |
| *** pblaho has quit IRC | 19:46 | |
| soren | morganfainberg: No problem at all. | 19:48 |
| *** thomasem has joined #openstack-infra | 19:50 | |
| *** talluri has joined #openstack-infra | 19:50 | |
| jog0 | just saw my name pop up in the infra meeting, with regard to adding tripleo nodes into nodepool | 19:52 |
| *** moted has joined #openstack-infra | 19:53 | |
| *** talluri has quit IRC | 19:54 | |
| lifeless | jog0: yes. | 19:56 |
| *** thomasbiege1 has quit IRC | 19:56 | |
| *** markmc has joined #openstack-infra | 20:00 | |
| *** skraynev is now known as skraynev_afk | 20:01 | |
| *** mrda_away is now known as mrda | 20:01 | |
| clarkb | I am going to be headed north to seattle shortly so that I don't have to drive in the rainnight, will be afk for a while | 20:01 |
| anteaya | drive safely | 20:01 |
| jeblair | jog0: log is here http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-02-18-19.01.log.html | 20:03 |
| lifeless | jog0: read the discussion, then lets discuss (also remembering you're *one* of the cd-admins who should fix the ci-overcloud were it to barf) | 20:03 |
| *** ianw has quit IRC | 20:03 | |
| *** ianw has joined #openstack-infra | 20:04 | |
| *** gyee has quit IRC | 20:05 | |
| Ajaeger | jeblair, clarkb, SergeyLukjanov: I'd appreciate if you could double check my updated patch https://review.openstack.org/#/c/73690/ | 20:05 |
| jog0 | jeblair lifeless: so I am torn on this | 20:07 |
| jog0 | I see a very real benifit from using the tripleo cloud, its another from of CI/testing | 20:08 |
| jog0 | that will inevitably flesh out real OpenStack bugs before the release | 20:08 |
| *** sarob_ has quit IRC | 20:08 | |
| lifeless | jog0: already has :) | 20:09 |
| *** sarob has joined #openstack-infra | 20:09 | |
| jog0 | exactly | 20:09 |
| jog0 | I like the idea of a seperate queue for tripleo for now | 20:10 |
| jog0 | as long as a tripleo failure doesn't stop nodepuul/zuul I think its worth the risk of changing things so late in the cycle | 20:10 |
| jog0 | nodepool* | 20:10 |
| sdague | yeh, I'm good with separate queue | 20:11 |
| sdague | my big concern was stalling the experimental queue | 20:11 |
| sdague | but a separate queue alleviates that concern | 20:11 |
| jog0 | sdague: agreed | 20:11 |
| lifeless | https://review.openstack.org/#/c/73863/ adds the experimental queue | 20:12 |
| lifeless | I'm going to rebase and add the new check variant there in a minute | 20:12 |
| lifeless | jeblair: would you prefer the addition of the experimental-tripleo jobs to be in the same patch ? | 20:12 |
| lifeless | jeblair: I had it separate to let the semantic vs bulk work be clear | 20:12 |
| *** andre__ has quit IRC | 20:12 | |
| sdague | lifeless: if you put < on line 139 you can wrap without issues | 20:13 |
| sdague | like we do in er queries | 20:13 |
| lifeless | sdague: oh, I was following what I saw in the file :) | 20:13 |
| sdague | https://github.com/openstack-infra/elastic-recheck/blob/master/queries/1277495.yaml | 20:13 |
| lifeless | will give it a go | 20:13 |
| sdague | actually, it's > | 20:13 |
| sdague | I always forget | 20:14 |
| *** sarob has quit IRC | 20:14 | |
| lifeless | sdague: does it add a space across lines? | 20:16 |
| sdague | I think so | 20:16 |
| sdague | yeh, it must | 20:16 |
| med_ | jeblair, are things more backed-up today in zuul than usual, or is today usual? | 20:19 |
| clarkb | Ajaeger: lgtm thanks | 20:22 |
| Ajaeger | clarkb: Thanks! | 20:22 |
| *** dstanek has quit IRC | 20:22 | |
| clarkb | med proposal feature freeze is this week and feature freeze is next week stuff is probably going to back up | 20:23 |
| SergeyLukjanov | Ajaeger, lgtm for me too, thanks for rebaising | 20:24 |
| *** oubiwann has joined #openstack-infra | 20:24 | |
| *** e0ne_ has joined #openstack-infra | 20:25 | |
| *** sarob has joined #openstack-infra | 20:26 | |
| lifeless | jeblair: what happens if two pipe lines have the same comment_filter trigger? | 20:27 |
| lifeless | jeblair: they'll both run right, but report seperately? | 20:27 |
| *** cadenzajon has quit IRC | 20:27 | |
| jeblair | lifeless: yes | 20:28 |
| *** e0ne has quit IRC | 20:29 | |
| Ajaeger | Thanks, SergeyLukjanov! | 20:30 |
| *** denis_makogon_ has joined #openstack-infra | 20:30 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 20:30 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add an -tripleo pipelines. https://review.openstack.org/73863 | 20:30 |
| *** ociuhandu has quit IRC | 20:31 | |
| *** sarob has quit IRC | 20:31 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 20:31 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add -tripleo pipelines. https://review.openstack.org/73863 | 20:31 |
| *** sarob has joined #openstack-infra | 20:31 | |
| lifeless | jeblair: https://review.openstack.org/73863 should be reviewable. Per your suggestion I made check-tripleo not require a pass to enqueue to the gate. I hope I got the layout right. | 20:31 |
| *** denis_makogon_ is now known as denis_makogon | 20:32 | |
| *** denis_makogon has quit IRC | 20:36 | |
| *** denis_makogon has joined #openstack-infra | 20:36 | |
| *** denis_makogon_ has joined #openstack-infra | 20:37 | |
| *** denis_makogon has quit IRC | 20:41 | |
| *** yolanda has quit IRC | 20:41 | |
| *** oubiwann has quit IRC | 20:43 | |
| med_ | clarkb, thanks | 20:43 |
| *** talluri has joined #openstack-infra | 20:44 | |
| miqui | jeblair: our jobs are getting stuck in zuul.... | 20:46 |
| *** talluri has quit IRC | 20:48 | |
| *** morganfainberg is now known as morganfainberg_Z | 20:49 | |
| jeblair | miqui: did you see that a separate merger process was added? | 20:49 |
| miqui | yes, the zuul-merger daemon thingy.....yes | 20:50 |
| miqui | it is running, seems to be since nothing 'bad' shows up in the logs... | 20:50 |
| miqui | am sure have seen same.... | 20:53 |
| miqui | sorry, others seen same behaviour | 20:53 |
| *** thomasbiege has joined #openstack-infra | 20:54 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-dev/hacking: Enforce capitalization of cfg help strings https://review.openstack.org/74493 | 20:56 |
| *** mrmartin has quit IRC | 20:59 | |
| *** DinaBelova is now known as DinaBelova_ | 21:00 | |
| *** ok___delta has joined #openstack-infra | 21:01 | |
| miqui | jeblair: we had to roll back | 21:01 |
| *** denis_makogon_ is now known as denis_makogon | 21:03 | |
| jeblair | miqui: if you telnet to port 4730 and enter 'status', you should be able to see whether there are jobs waiting for the merger, and how many mergers you have on-line. | 21:03 |
| *** rfolco has quit IRC | 21:04 | |
| miqui | jeblair: 2 jobs | 21:05 |
| *** melwitt has quit IRC | 21:06 | |
| miqui | jeblair: how do we trubshoot queued jobs... | 21:07 |
| jeblair | miqui: how many mergers are connected? | 21:07 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Let infra manage pyghmi releases https://review.openstack.org/74499 | 21:07 |
| devananda | clarkb: ^ first attempt at creating a release job myself. I probably botched it, please comment when you have a minute :) | 21:08 |
| miqui | jeblair: thanks we are gonna punt for now, to version -8 days old or so.... we'll try later when we have more time.... | 21:08 |
| *** cadenzajon has joined #openstack-infra | 21:09 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add grenade-dsvm-partial-ncpu test https://review.openstack.org/64200 | 21:09 |
| jeblair | miqui: my guess is that the merger is not running or is unable to connect | 21:09 |
| jeblair | miqui: let me know if i can help when you have more time to debug | 21:09 |
| jeblair | mikal: hi, how can i help with storyboard? | 21:09 |
| jeblair | mikal: here's some background: https://etherpad.openstack.org/p/icehouse-summit-storyboard-basic-concepts-and-next | 21:10 |
| jeblair | mikal: (there was a summit session where we discussed it) | 21:10 |
| *** e0ne_ has quit IRC | 21:10 | |
| jeblair | mikal: and i think there's been some talk on the -dev and -infra lists... | 21:10 |
| jeblair | mikal: i think there hasn't been a big announcement or anything because it's really quite early. we don't want to say "hey everyone, we want to move to a new bug tracking system" prematurely. | 21:11 |
| *** morganfainberg_Z is now known as morganfainberg | 21:12 | |
| jeblair | mikal: rather the approach has been, "let's see if we can get a critical mass of people working on a proof of concept and see if we can produce something that works" | 21:13 |
| *** dstanek has joined #openstack-infra | 21:13 | |
| jeblair | and then go from there | 21:13 |
| *** melwitt has joined #openstack-infra | 21:13 | |
| miqui | jeblair: thanks jeb.... | 21:14 |
| *** dprince has quit IRC | 21:14 | |
| miqui | jeblair: will ping you back... | 21:14 |
| *** lcestari has quit IRC | 21:15 | |
| mattoliverau | morning all | 21:16 |
| anteaya | morning mattoliverau | 21:16 |
| jeblair | mattoliverau: good morning | 21:17 |
| *** nati_uen_ has quit IRC | 21:18 | |
| *** nati_ueno has joined #openstack-infra | 21:19 | |
| *** jungleboyj has quit IRC | 21:20 | |
| *** ok___delta has quit IRC | 21:21 | |
| jeblair | russellb: about gantt, do you think it will be part of an openstack program eventually? | 21:21 |
| fungi | okay, all 9 jenkins masters are now running jenkins 1.551 and zmq plugin 0.0.3 and have been restarted, with a complete nodepool node flush for each while i was at it | 21:22 |
| jeblair | russellb: i ask because (a) it's really a pain to rename repos, and (b) we have a certain amout of looking-forward in the repo orgs; even before programs, we added incubated projects to openstack/ ... | 21:22 |
| *** markmc has quit IRC | 21:22 | |
| zaro | jeblair: are you satisfied with | 21:22 |
| jeblair | russellb: so if you thought we'd move gantt back in 9 months, i'd just as soon leave it. :) | 21:22 |
| zaro | jeblair: are you satisfied with https://review.openstack.org/#/c/60348 ? | 21:23 |
| *** smarcet has left #openstack-infra | 21:23 | |
| jeblair | zaro: i gave it a +2 | 21:23 |
| SergeyLukjanov | jeblair, fungi, clarkb, I'll really appreciate you take a look at savanna jobs changes today if have some time - https://review.openstack.org/#/c/74310/ and https://review.openstack.org/#/c/74470/ | 21:23 |
| zaro | jeblair: cool. | 21:23 |
| SergeyLukjanov | fungi, awesome! | 21:24 |
| fungi | well, there were a ton of security fixes in 1.551, so it was fairly necessary | 21:25 |
| mikal | jeblair: I wasn't really saying I was confused. I was saying that lots of people I meet are confused. | 21:25 |
| fungi | mikal: we should just buy a jira license, right? ;) | 21:25 |
| *** lcostantino has quit IRC | 21:26 | |
| *** jhesketh has joined #openstack-infra | 21:26 | |
| mikal | fungi: heh | 21:27 |
| *** hogepodge_ has joined #openstack-infra | 21:28 | |
| *** afazekas has joined #openstack-infra | 21:28 | |
| *** hogepodge has quit IRC | 21:29 | |
| *** hogepodge_ is now known as hogepodge | 21:29 | |
| lifeless | SergeyLukjanov: replied on https://review.openstack.org/#/c/73863/ | 21:30 |
| jeblair | lifeless: hehe, i think SergeyLukjanov may have been pointing out that the actual code says "check tripleo" even though the docs say "check experimental" | 21:31 |
| jeblair | lifeless: so regardless of intent, there is a mismatch | 21:32 |
| lifeless | oh, duh. thanks | 21:32 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 21:32 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add -tripleo pipelines. https://review.openstack.org/73863 | 21:32 |
| lifeless | jeblair: there ^ | 21:32 |
| *** Ajaeger has quit IRC | 21:34 | |
| *** jhesketh has quit IRC | 21:35 | |
| *** jamielennox is now known as jamielennox|away | 21:35 | |
| *** jamielennox|away is now known as jamielennox | 21:35 | |
| *** talluri has joined #openstack-infra | 21:38 | |
| *** greghaynes has quit IRC | 21:41 | |
| *** melwitt has quit IRC | 21:42 | |
| *** melwitt has joined #openstack-infra | 21:42 | |
| *** talluri has quit IRC | 21:42 | |
| *** greghaynes has joined #openstack-infra | 21:44 | |
| *** markwash_ has joined #openstack-infra | 21:45 | |
| lifeless | fungi: when nodepool fails to start, do you get a backtrace or something ? | 21:45 |
| fungi | lifeless: i wasn't trying to start it last time... jeblair what did you witness? | 21:45 |
| fungi | i'm guessing this could be tested by configuring a local nodepool installation for a nonexistent provider and seeing what happens | 21:46 |
| *** tjones has joined #openstack-infra | 21:46 | |
| *** dkliban has quit IRC | 21:46 | |
| jeblair | lifeless: sorry, i don't have the output handy. it didn't get past whatever initialization happens, but i don't recall the details | 21:46 |
| *** markwash has quit IRC | 21:47 | |
| *** markwash_ is now known as markwash | 21:47 | |
| *** talluri has joined #openstack-infra | 21:47 | |
| jeblair | lifeless: i was specifically trying to use the cli; so my use of 'nodepool delete' was broken. | 21:48 |
| *** salv-orlando has joined #openstack-infra | 21:48 | |
| jeblair | which is also why we don't have logs | 21:48 |
| *** gyee has joined #openstack-infra | 21:49 | |
| lifeless | ok | 21:50 |
| lifeless | I will configure up a broken config and try both nodepoold and the cli | 21:50 |
| *** UtahDave has joined #openstack-infra | 21:51 | |
| *** gyee has quit IRC | 21:51 | |
| fungi | lifeless: one place where we've seen it struggle (and not sure whether we managed to catch all possible ways it can happen) is when image rebuilds start and one provider is offline | 21:51 |
| *** gyee has joined #openstack-infra | 21:51 | |
| fungi | so you might want to configure a tight loop around that too and see what nodepoold chokes up | 21:51 |
| *** rlandy has quit IRC | 21:51 | |
| markmcclain | I'm seeing some strange behavior for checks | 21:52 |
| lifeless | jeblair: you were deleting an image on provider A, but B was down, right ? | 21:52 |
| markmcclain | 69110 is saying a check is queued | 21:52 |
| markmcclain | but that check has completed | 21:52 |
| *** talluri has quit IRC | 21:52 | |
| markmcclain | https://jenkins04.openstack.org/job/check-tempest-dsvm-neutron/4566/console | 21:52 |
| jeblair | lifeless: s/image/node/ yes | 21:52 |
| lifeless | INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): cd-overcloud.tripleo.org | 21:52 |
| *** jcooley_ has joined #openstack-infra | 21:52 | |
| lifeless | sorry, didn't mean to paste that | 21:52 |
| jeblair | lifeless: 'nodepool delete' does a fuul initialization of all providers (which is insane but that's what it does) | 21:52 |
| anteaya | that same test check-tempest-dsvm-neutron was queued previously | 21:53 |
| anteaya | 69110 has been in check 4.5 hours and was waiting on that test to get a node and complete and now it is waiting on a node again | 21:53 |
| fungi | markmcclain: jeblair: i'm similarly seeing some strange looping of jobs. certain jobs complete and then get rerun, like zuul never got the completion status (or didn't recognise it) . i'm digging in the logs to see whether i can tell why | 21:53 |
| *** afazekas has quit IRC | 21:54 | |
| anteaya | 73770 has also been waiting on a node for ~ the last hour | 21:55 |
| lifeless | jeblair: so did it break entirely, or just so slow as to be unusuable? (I'm trying to make sure I fix this sufficiently hard to make it good for you) | 21:55 |
| *** tjones has quit IRC | 21:56 | |
| fungi | jeblair: AttributeError: 'str' object has no attribute '_ZuulGearmanClient__gearman_job' | 21:56 |
| fungi | jeblair: http://paste.openstack.org/show/67040/\ | 21:56 |
| *** tjones has joined #openstack-infra | 21:57 | |
| BobBall | How can I see what the overall failure rate for full tempest jobs in the gate is? | 21:57 |
| jeblair | lifeless: i can't say for certain. i believe it eventually timed out, so probably the latter. | 21:57 |
| fungi | jeblair: is that part of what's fixed by gear 0.5.1? | 21:57 |
| jeblair | fungi: i don't think so | 21:58 |
| lifeless | jeblair: sorry, I think I was unclear. What I meant was - if the 'fix' still takes 45s to startup when a cloud is down, but does the delete/starts daemon/whatever - would that be ok? | 21:58 |
| lifeless | jeblair: or, does it need to not just achieve the task, but also be fairly snappy about it | 21:58 |
| lifeless | ? | 21:58 |
| *** dcramer_ has quit IRC | 21:59 | |
| ekarlso | did somebody spill oil at the gates or ? :D | 21:59 |
| lifeless | I've isolated the place it breaks by, which is _getExtensions called from ProviderManager.__init__ | 21:59 |
| anteaya | ekarlso: what are you seeing? | 21:59 |
| anteaya | ekarlso: we are tracking some issues, do you have any datapoints? | 21:59 |
| ekarlso | just long lines ;) | 22:00 |
| ekarlso | at least I don't have to manually pick a line number ;) | 22:00 |
| anteaya | well the aforementioned issues, plus we are heading into feature freeze | 22:00 |
| anteaya | the season to queue has begun | 22:00 |
| anteaya | ekarlso: there's that | 22:00 |
| ekarlso | anteaya: FF meaning tons of reviews ? : p | 22:01 |
| anteaya | feature freeze is tons of patches being tested, yes | 22:01 |
| *** tjones has quit IRC | 22:01 | |
| *** tjones has joined #openstack-infra | 22:01 | |
| jeblair | lifeless: i think taking ~ 1 min to start when a provider is down is probably ok for the daemon; the cli needs to be snappier (but that can probably be achieved by avoiding unecessary inits) | 22:01 |
| anteaya | since any code not in by feature freeze doesn't make it into icehouse | 22:01 |
| anteaya | unless as a bug fix | 22:01 |
| lifeless | I think I have a fairly minimal approach, poking at it now | 22:02 |
| anteaya | or an exception | 22:02 |
| jeblair | lifeless: much better would be if the init was async and the daemon could just start using them when they showed up, but that be a future improvement | 22:02 |
| jeblair | s/be/could be/ | 22:02 |
| jeblair | fungi: thoughts? ^ | 22:02 |
| *** tjones has quit IRC | 22:02 | |
| *** tjones1 has joined #openstack-infra | 22:02 | |
| *** oubiwann has joined #openstack-infra | 22:02 | |
| jeblair | fungi: (and i'm starting to look into the zuul error) | 22:03 |
| *** mrodden has quit IRC | 22:03 | |
| *** dcramer_ has joined #openstack-infra | 22:03 | |
| fungi | jeblair: also these look wrong... http://paste.openstack.org/show/67041/ | 22:03 |
| jaypipes | Hi y'all... I'm getting the following when running a Jenkins job that runs devstack-gate scripts. The problem is that I've set stack user to NOPASSWD sudo rights... but it's still failing. Looking to see if you have some thoughts... | 22:03 |
| jaypipes | 22:00:31 Running devstack | 22:03 |
| jaypipes | 22:00:31 sudo: no tty present and no askpass program specified | 22:03 |
| jaypipes | dtroyer: hoping you might have some ideas for above ^^ :) | 22:05 |
| fungi | jeblair: lifeless: yeah, i think that's okay in general. ideally the client could also get smarter and check the database to find out which provider it needs to authenticate to for a delete rather than shotgun-blasting them all up front | 22:05 |
| *** dstanek has quit IRC | 22:06 | |
| jeblair | fungi: so that traceback suggests that what it expects to be a Build object is really a string. that is confusing. | 22:06 |
| jeblair | fungi: oh, one sec; checking something. | 22:07 |
| fungi | jeblair: right. curious whether i can find the details for it in the debug log. hunting there currently | 22:07 |
| fungi | as to what's in the string | 22:07 |
| *** nati_uen_ has joined #openstack-infra | 22:07 | |
| fungi | or is it possible the conversion from raw string to build object missed coverage? | 22:08 |
| lifeless | ahhh | 22:08 |
| dtroyer | jaypipes: I ran into this for the firt time ever recently…looking through notes to recall exactly what it was | 22:08 |
| lifeless | checkForMissingImages is a problem because it doesn't use threads | 22:08 |
| fungi | lifeless: that may be where i saw it blocking on image rebuilds when tripleo was offline | 22:09 |
| lifeless | yes | 22:09 |
| fungi | lifeless: and nodepool just flat stopped launching any new nodes | 22:09 |
| sdague | jaypipes: you on rhel? | 22:09 |
| lifeless | you put a catchall exception in | 22:09 |
| lifeless | but that actually violates the provider axioms | 22:09 |
| sdague | or something that looks like rhel? | 22:09 |
| lifeless | so I'm pulling that out | 22:09 |
| jaypipes | dtroyer: yeah... I mean https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L274 makes it seem like stack.sh is just being run by the stack user, so unless stack.sh changes privs or something, I don't understand why sudo would still be problematic. | 22:09 |
| jaypipes | sdague: ubuntu precise. | 22:09 |
| sdague | hmmmm | 22:09 |
| fungi | lifeless: fine by me--i noted at the time that it was probably not a very clean way to fix it, but i didn't grok the greater context there | 22:09 |
| *** geekinutah has joined #openstack-infra | 22:10 | |
| lifeless | def updateImage(self, session, provider, image): | 22:10 |
| lifeless | provider = self.config.providers[provider.name] | 22:10 |
| lifeless | image = provider.images[image.name] | 22:10 |
| lifeless | is the issue I believe | 22:10 |
| jeblair | fungi: okay, so actually it's iterating over a dictionary's (string) keys and expecting (Build object) values. so the puzzle now is how anything ever works. | 22:10 |
| lifeless | fungi: if you could dig back the log and see if the last line there is in the traceback that would be super useful | 22:10 |
| *** nati_ueno has quit IRC | 22:10 | |
| fungi | lifeless: sure thing | 22:11 |
| sdague | jaypipes: where did you set it to NOPASSWD? because I just did that via /etc/sudoers.d/ earlier today on this vagrant + puppet + devstack install, and it worked fine | 22:11 |
| jaypipes | sdague: ubuntu@slave:~$ sudo cat /etc/sudoers.d/jenkins-sudo | 22:11 |
| jaypipes | jenkins ALL=(root) NOPASSWD:ALL | 22:11 |
| jaypipes | stack ALL=(root) NOPASSWD:ALL | 22:11 |
| jeblair | fungi: hypothesis: that has never worked but doesn't affect anything. | 22:11 |
| *** pdmars has quit IRC | 22:12 | |
| fungi | jeblair: so perhaps not the cause of the looping jobs we're seeing now? | 22:12 |
| *** melwitt has quit IRC | 22:12 | |
| dtroyer | jaypipes: that is an attempt to force stack.sh into line-buffered mode on stdout/stderr. I think we're losing log output in some error cases. anyway, is that where the fail is at? | 22:12 |
| sdague | jaypipes: ALL=(ALL) ? | 22:12 |
| sdague | because you are actually changing to not root | 22:12 |
| jeblair | fungi: yeah, that shows up previously in the logs | 22:12 |
| fungi | jeblair: new theory... | 22:13 |
| fungi | jeblair: the "last reconfigured" is from 22 hours ago | 22:13 |
| jaypipes | dtroyer: not sure where the failure is unfortunately. doesn't indicate in the console log. | 22:13 |
| *** jhesketh_ has joined #openstack-infra | 22:13 | |
| fungi | but we've merged layout.yaml changes today | 22:13 |
| *** prad has quit IRC | 22:13 | |
| fungi | jeblair: are there maybe problems reloading the config? | 22:14 |
| *** prad has joined #openstack-infra | 22:14 | |
| jhesketh_ | Howdy | 22:14 |
| fungi | wondering whether something in the new commits has broken config reloads | 22:14 |
| jeblair | fungi: puppet is still off | 22:14 |
| fungi | ahh | 22:15 |
| lifeless | fungi: so yeah, that line should be fine - I'll need to see a traceback of that failure mode if possible | 22:15 |
| jeblair | fungi: i want to get it working correctly with zm02 then bring it back on on 01 and zuul | 22:15 |
| nibalizer | fungi: did you get a chance to fire up that vm yet? | 22:15 |
| fungi | jeblair: um, so my change to switch jobs to single-use nodes is maybe an issue then? zuul doesn't know it should change the parameters for them? | 22:15 |
| fungi | nibalizer: not yet. things went and got all breaky | 22:16 |
| *** mwagner_lap has quit IRC | 22:16 | |
| nibalizer | ah okay, stupid things, always breaking | 22:16 |
| *** CaptTofu has quit IRC | 22:17 | |
| jeblair | yeah, that's going to be a problem. | 22:17 |
| fungi | lifeless: http://paste.openstack.org/show/67046/ | 22:18 |
| jeblair | fungi: but that shouldn't cause looping jobs | 22:18 |
| *** vkozhukalov has quit IRC | 22:18 | |
| lifeless | fungi: that can't be the cause - there is a except there | 22:19 |
| *** mrodden has joined #openstack-infra | 22:19 | |
| fungi | lifeless: there wasn't back then | 22:19 |
| lifeless | s/a except/an except:/ | 22:19 |
| lifeless | fungi: ah! | 22:19 |
| lifeless | fungi: so,thats been fixed properly then | 22:19 |
| lifeless | but there may be other cases | 22:19 |
| geekinutah | jeblair, fungi: This is my third "FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel" | 22:19 |
| geekinutah | is that what you mean when you say looping jobs? | 22:20 |
| fungi | geekinutah: "this" what this? | 22:20 |
| *** denis_makogon has quit IRC | 22:20 | |
| geekinutah | I've been watching py27 tests run by on a review and keep getting this error | 22:20 |
| fungi | geekinutah: have a link? | 22:20 |
| geekinutah | https://jenkins04.openstack.org/job/gate-nova-python27/2499/console | 22:20 |
| *** tjones1 has quit IRC | 22:20 | |
| *** prad has quit IRC | 22:20 | |
| *** e0ne has joined #openstack-infra | 22:21 | |
| jeblair | fungi: puppet started on z.o.o and zuul reconfigured | 22:21 |
| jeblair | (ssh on zm02 checked out) | 22:21 |
| geekinutah | fungi: unfortunately neglected to collect the 2 before this | 22:22 |
| fungi | jeblair: thanks | 22:22 |
| jeblair | geekinutah: did zuul report that to gerrit, or are you just watching live? | 22:22 |
| fungi | geekinutah: yes, it looks like gate-nova-python27 has been restarted on that change | 22:22 |
| geekinutah | watching it live | 22:22 |
| *** tjones has joined #openstack-infra | 22:22 | |
| fungi | and it's back to pending again | 22:23 |
| geekinutah | pylint is another that has restarted several times | 22:23 |
| lifeless | fungi: actually that periodicCheck one - I think thats when an image definition is deleted but there were still nodes left. | 22:23 |
| fungi | i hadn't seen an abort requested exception on the ones i watched finish | 22:23 |
| geekinutah | I'm watching it again, see if it reports the same thing: https://jenkins04.openstack.org/job/gate-nova-pylint/1222/console | 22:23 |
| fungi | ahh, yep, here's one on the heat change at the head of the gate... https://jenkins05.openstack.org/job/gate-heat-python26/119/console | 22:24 |
| *** tjones has quit IRC | 22:24 | |
| *** dizquierdo has joined #openstack-infra | 22:24 | |
| *** prad has joined #openstack-infra | 22:24 | |
| jeblair | fungi: do you think geekinutah is seeing the same problem we're looking into? | 22:25 |
| *** dstanek has joined #openstack-infra | 22:25 | |
| fungi | jeblair: i believe so | 22:25 |
| fungi | something is asking for a job abort | 22:25 |
| jeblair | 2014-02-18 22:03:48,099 INFO zuul.Gearman: Build <gear.Job 0x7f331ae941d0 handle: H:127.0.0.1:476178 name: build:gate-nova-python27 unique: 68c845de139a45d498c1b59efd914594> complete, result None | 22:25 |
| *** e0ne has quit IRC | 22:26 | |
| anteaya | besides sitting here, is there anything I can do to help? | 22:26 |
| *** thomasbiege has quit IRC | 22:26 | |
| jeblair | fungi: so that's what zuul got back from jenkins: it indicated the job was complete but had no result | 22:26 |
| *** hogepodge has quit IRC | 22:26 | |
| *** melwitt has joined #openstack-infra | 22:27 | |
| jeblair | fungi: notes here: | 22:27 |
| jeblair | fungi: https://etherpad.openstack.org/p/VoXr3sXst8 | 22:28 |
| *** thomasem has quit IRC | 22:28 | |
| *** hogepodge has joined #openstack-infra | 22:28 | |
| *** esker has quit IRC | 22:29 | |
| *** tjones has joined #openstack-infra | 22:29 | |
| jeblair | fungi: ok | 22:29 |
| jeblair | fungi: it's related to the bare node change | 22:29 |
| jeblair | oh wait | 22:30 |
| jeblair | offline when complete was set... | 22:30 |
| fungi | gate-nova-python27 has been on bare nodes for weeks | 22:30 |
| fungi | so the old zuul config would still have been sending that | 22:30 |
| jeblair | fungi: yeah, i'm starting to wonder about the new jenkins version | 22:31 |
| jeblair | fungi: and whether offline on complet is broken | 22:31 |
| fungi | i've already started googling likely errors | 22:31 |
| fungi | we could try to switch to a jenkins "stable" release | 22:31 |
| jeblair | fungi: because basically it looks like that node ran a job, completed it, tried to mark it offline, got an exception, and then started running another job | 22:32 |
| jeblair | fungi: and so then eventually nodepool gets around to deleting it in the middle of a later job, thus the interrupted exception | 22:32 |
| *** jcoufal has quit IRC | 22:32 | |
| fungi | we could switch to their "lts" 1.532.2 | 22:33 |
| jeblair | fungi: gimme a sec to figure out how bad the problem is and if we can fix in gearman-plugin | 22:33 |
| fungi | which is not too terribly old... we only recently upgraded from 1.525 to 1.543 after all | 22:33 |
| jeblair | fungi: that may not be a bad idea, we used to be unable to run lts because it wasn't new enough. i don't think we care now. | 22:34 |
| jaypipes | sdague: that was it. thx man. | 22:34 |
| sdague | jaypipes: no worries | 22:34 |
| *** derekh has joined #openstack-infra | 22:35 | |
| jeblair | fungi: also, why are there always jenkins security updates near milestones? | 22:35 |
| fungi | jeblair: because they're secretly trying to undermine our operation | 22:35 |
| fungi | or maybe because we're always near milestones | 22:35 |
| jeblair | heh | 22:35 |
| fungi | there's really never a good time for any of this stuff to break any more | 22:36 |
| jeblair | fungi: do you want to install the lts on jenkins-dev? | 22:36 |
| jeblair | (i'll continue to look at gearman-plugin) | 22:36 |
| fungi | yep, was just logging in | 22:37 |
| fungi | thanks! | 22:37 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Make nodepool more robust to offline clouds. https://review.openstack.org/74521 | 22:40 |
| lifeless | derekh: ^ this one | 22:40 |
| *** andre__ has joined #openstack-infra | 22:41 | |
| *** talluri has joined #openstack-infra | 22:41 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/nodepool: Support offline providers on startup https://review.openstack.org/74523 | 22:43 |
| derekh | lifeless: so this is what I did ^^ | 22:44 |
| lifeless | jeblair: ask and ye shall receive. Twice over. | 22:44 |
| fungi | jeblair: Jenkins ver. 1.532.2 on jenkins-dev seems to start with our current plugin versions loaded and test-jenkins-api.py runs clean (minus that one cancel build test which has been failing for a while) | 22:45 |
| lifeless | derekh: cool, so - I'll see what approach the infra folk like most and if its yours with tweaks, will do them for them. | 22:45 |
| *** talluri has quit IRC | 22:46 | |
| jeblair | fungi: i just got the actual traceback for the error; it looks like something about the offline interface has changed | 22:46 |
| fungi | jeblair: and given that we've run similar versions from that timeframe, it seems like a safe alternative if you decide whatever's broken with the plugin isn't going to be easy to fix | 22:46 |
| derekh | lifeless: ok, sounds good, got a little side tracked adding support for the fedora clound image | 22:46 |
| *** gordc has left #openstack-infra | 22:46 | |
| fungi | jeblair: oh! well that would make sense | 22:46 |
| fungi | as an explanation for what we've seen | 22:47 |
| jeblair | fungi: i'm leaning toward downgrading jenkins not only because i think it is expedient but i think switching to lts is something we should do anyway | 22:47 |
| jeblair | clarkb, mordred, SergeyLukjanov: ^ thoughts? | 22:47 |
| fungi | i'll get spinning on it in that case | 22:47 |
| fungi | if there's concord | 22:47 |
| mordred | jeblair: I agree - downgrading jenkins to LTS | 22:47 |
| fungi | their lts these days isn't that far behind mainline | 22:47 |
| jeblair | fungi: yeah, lets get started on it. | 22:47 |
| fungi | unlike, say, a year ago | 22:47 |
| fungi | okay, i'll roll through a couple at a time overlapping | 22:48 |
| fungi | unless i should just do them all in parallel | 22:48 |
| anteaya | I like a couple at a time | 22:48 |
| anteaya | jsut in case | 22:48 |
| jeblair | fungi: overlap; the system is still moderately functional | 22:48 |
| lifeless | jeblair: so, we're now waiting on review I think :) | 22:48 |
| jeblair | lifeless: after firefighting | 22:49 |
| *** completely-despe has joined #openstack-infra | 22:49 | |
| lifeless | jeblair: I say that not to pressure you, but because I figure you want the change in early to avoid the FF period | 22:49 |
| lifeless | jeblair: ack, will get out of your way now :) | 22:49 |
| *** oubiwann has quit IRC | 22:49 | |
| completely-despe | Hi everyone | 22:49 |
| anteaya | how can we help completely-despe | 22:49 |
| completely-despe | i | 22:49 |
| completely-despe | i've been going up and own all over the place | 22:50 |
| completely-despe | and here's the story, I've setup some parts of the openstack-infra for using gerrit/zuul/jenkins you know to run the unit testing from openstack projects | 22:50 |
| jeblair | fungi: maybe do half at once? | 22:50 |
| fungi | jeblair: will do | 22:51 |
| fungi | that's a good, quick compromise | 22:51 |
| completely-despe | but I'm having a really hard time on how to setup the unit testing, I mean which commands do I need to execute for projects in neutron, swift, glance and heat | 22:51 |
| fungi | eek, and we have the staff call in <10 minutes | 22:51 |
| anteaya | completely-despe: are you trying to comply with third party testing requirements for neutron? | 22:52 |
| completely-despe | I went to the docs and followed everything I could find but for instance, glance breaks on setting up the unit testing because pysendfile can't be downloaded, then another example is that neutron breaks at Running `tools/with_venv.sh python -m neutron.openstack.common.lockutils python setup.py testr --slowest --testr-args='--subunit '` | 22:52 |
| *** jgrimm has quit IRC | 22:52 | |
| completely-despe | so i was wondering if you guys have a document where it states how to setup the unit testing for the projects other than ./run_tests.sh -P -V | 22:53 |
| anteaya | completely-despe: let me try to understand why you are doing the set up of parts of openstack-infra? | 22:53 |
| completely-despe | the openstack infra is setup, but the contents of the jobs in jenkins for running the tests is what I was wondering if you share them also | 22:54 |
| fungi | just noticed we have an older gearman plugin on jenkins.o.o but i'll deal with that later | 22:54 |
| jeblair | fungi: i think next step is to update the api test to exercise gearman; then update gearman-plugin for the change which will presumably be making it to lts eventually | 22:55 |
| anteaya | completely-despe: everything is opensourced | 22:55 |
| *** nati_ueno has joined #openstack-infra | 22:55 | |
| fungi | jeblair: agreed | 22:56 |
| *** mfer has quit IRC | 22:57 | |
| anteaya | completely-despe: here is the config for the jobs we run in the gate: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml | 22:57 |
| anteaya | completely-despe: we use jenkins job builder to build the jobs: http://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/ | 22:58 |
| *** oubiwann has joined #openstack-infra | 22:59 | |
| anteaya | completely-despe: I'm still lacking context for why you are setting up an infra | 22:59 |
| anteaya | you don't have to tell me but it might help me find more relevant information for you | 22:59 |
| *** nati_uen_ has quit IRC | 22:59 | |
| *** markwash_ has joined #openstack-infra | 22:59 | |
| *** jnoller has quit IRC | 23:00 | |
| *** scottsanchez has joined #openstack-infra | 23:00 | |
| completely-despe | I can tell you no worries | 23:01 |
| completely-despe | so the deal is that at ebay we're contributing code back from some modifications that we've done to some of the projects | 23:01 |
| rainya | i am unable to sign into review.openstack.org and was wondering if anyone else had noticed problems? | 23:01 |
| scottsanchez | rainya: me too! | 23:02 |
| *** markwash has quit IRC | 23:02 | |
| *** markwash_ is now known as markwash | 23:02 | |
| completely-despe | so we saw the openstack-ci and we setup it up also to help us out during the ci workflow | 23:02 |
| rainya | annegentle is already signed in and it's working fine for her | 23:02 |
| rainya | just can't sign into the thing | 23:02 |
| rainya | :D | 23:02 |
| fungi | okay, i've downloaded the latest jenkins lts to all masters, upgraded jenkins.o.o already, odd numbered masters are in shutdown quiescing now | 23:03 |
| jaypipes | completely-despe, anteaya: that's for devstack runs, not unit tests... for unit tests, here is the config definition: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml#n75 and https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/macros.yaml#n149, which runs https://git | 23:03 |
| jaypipes | .openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/run-unittests.sh | 23:03 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/nodepool: Catch key problems in ssh_connect https://review.openstack.org/74528 | 23:03 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/nodepool: Add fedora support https://review.openstack.org/74529 | 23:03 |
| anteaya | jaypipes: yay, I was hoping you would jump in | 23:03 |
| sdague | so.... gerrit login busted? | 23:03 |
| completely-despe | but few things is that we're running on RHEL instead of ubuntu, and we wanted to execute the unit tests of each openstack project through it | 23:03 |
| anteaya | sdague: what are you seeing? | 23:03 |
| mordred | completely-despe: we actually don't use run_tests.sh at all for anything in CI - you may want to read https://wiki.openstack.org/wiki/ProjectTestingInterface | 23:03 |
| sdague | loggout, then try to log back in | 23:03 |
| sdague | and it just hangs | 23:04 |
| mordred | completely-despe: also, the run-unittests.sh script that jay posted | 23:04 |
| scottsanchez | sign in button does nothing... just hangs | 23:04 |
| jaypipes | completely-despe: you might find this article useful (it covers the unit test jobs): https://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/run-unittests.sh | 23:04 |
| anteaya | there is jenkins and gearman plugin work happening | 23:04 |
| sdague | which probably means launchpad gone completely bonkers | 23:04 |
| anteaya | sdague: oh the timing | 23:04 |
| *** dims has quit IRC | 23:04 | |
| mordred | sdague: oh wow | 23:04 |
| sdague | so maybe don't logout if you want to still be able to use gerrit | 23:04 |
| sdague | yeh, kenichi just woke up and got on in .jp, and no one there could get in | 23:04 |
| jeblair | yeah, i'll try with another browser | 23:04 |
| anteaya | sdague: thanks for the heads up, I will ask in #launchpad | 23:04 |
| *** dims has joined #openstack-infra | 23:05 | |
| mordred | sdague: yeah. and like an idiot, I just followed your instructions to see if I could reproduce | 23:05 |
| mordred | I can | 23:05 |
| jeblair | mordred: haha | 23:05 |
| anteaya | ha ha ha | 23:05 |
| anteaya | find a cliff | 23:05 |
| * mordred hangs head in shame | 23:05 | |
| openstackgerrit | Chad Lung proposed a change to openstack-infra/config: Adding Barbican support for DevStack https://review.openstack.org/74530 | 23:05 |
| anteaya | mordred: can you ssh? | 23:05 |
| mordred | anteaya: yes | 23:05 |
| completely-despe | thanks a lot jaypipes i'll look into it right away | 23:05 |
| mordred | anteaya: I mean, I'm not dead in the water or anything - but DOH | 23:05 |
| anteaya | so perhaps it is as sdague says something with lauchpad | 23:05 |
| anteaya | d'oh | 23:06 |
| sdague | no, this is gerrit | 23:06 |
| completely-despe | thanks everyone i got another starting point I'll look into all the links | 23:06 |
| completely-despe | thanks a lot for the help i really appreciate it | 23:06 |
| sdague | openid login via launchpad works on wiki | 23:06 |
| *** sarob has quit IRC | 23:07 | |
| sdague | did gerrit go extra bonkers? | 23:07 |
| *** ken2ohmichi has joined #openstack-infra | 23:07 | |
| anteaya | I'm asking in #launchpad | 23:07 |
| jeblair | review-dev seems to work | 23:08 |
| sdague | anteaya: well the fact that login is fine in the openstack wiki, would seem to indicate lp was fine | 23:08 |
| *** dcramer__ has joined #openstack-infra | 23:08 | |
| anteaya | yes, I just added that bit to my question | 23:08 |
| anteaya | so far no response, perhaps people are checking | 23:08 |
| jeblair | anteaya: let's not bother them until we think it's really a lp problem | 23:09 |
| anteaya | ah too late | 23:09 |
| anteaya | how do I back out gracefully | 23:09 |
| anteaya | I asked how we assess if it is a lp issue | 23:09 |
| anteaya | not that I said I thought it was | 23:09 |
| jeblair | sshing into review.o.o is very slow | 23:10 |
| * anteaya looks at cacti | 23:10 | |
| *** dstanek_afk has joined #openstack-infra | 23:11 | |
| *** dstanek_afk has quit IRC | 23:11 | |
| jeblair | i'm suspecting a dns problem; it would affect both things | 23:11 |
| *** melwitt1 has joined #openstack-infra | 23:12 | |
| *** oubiwann has quit IRC | 23:12 | |
| jeblair | [2014-02-18 23:11:20,812] ERROR com.google.gerrit.httpd.auth.openid.OpenIdServiceImpl : Cannot discover OpenID https://login.launchp | 23:12 |
| jeblair | ad.net/+openid | 23:12 |
| jeblair | org.openid4java.discovery.yadis.YadisException: 0x704: I/O transport error: login.launchpad.net | 23:12 |
| jeblair | Caused by: java.net.UnknownHostException: login.launchpad.net | 23:12 |
| fungi | dns | 23:12 |
| fungi | i can't look that up with review's default resolvers | 23:13 |
| fungi | ;; connection timed out; no servers could be reached | 23:13 |
| *** dpyzhov has joined #openstack-infra | 23:13 | |
| jeblair | fungi: some thing works from review-dev | 23:13 |
| *** dcramer_ has quit IRC | 23:14 | |
| fungi | 72.3.128.240 and 72.3.128.241? | 23:14 |
| jeblair | fungi: yep, and instantly | 23:14 |
| fungi | yep | 23:14 |
| jeblair | pvo: if you're still around, i think we may have a more serious manifestation of rax dns issues | 23:15 |
| fungi | maybe massive dns packet loss on the same compute node | 23:15 |
| fungi | udp loss | 23:15 |
| jeblair | i will open a ticket | 23:15 |
| fungi | thanks! | 23:15 |
| *** dcramer__ has quit IRC | 23:16 | |
| *** tjones has quit IRC | 23:16 | |
| *** melwitt has quit IRC | 23:16 | |
| *** dstanek has quit IRC | 23:16 | |
| *** dizquierdo has quit IRC | 23:16 | |
| *** salv-orlando has quit IRC | 23:16 | |
| *** krtaylor has quit IRC | 23:16 | |
| *** unicell has quit IRC | 23:16 | |
| *** openstackgerrit has quit IRC | 23:16 | |
| *** portante has quit IRC | 23:16 | |
| *** gmoro has quit IRC | 23:16 | |
| *** lcostantino has joined #openstack-infra | 23:17 | |
| *** alexpilotti has quit IRC | 23:17 | |
| *** salv-orlando has joined #openstack-infra | 23:18 | |
| *** krtaylor has joined #openstack-infra | 23:18 | |
| *** unicell has joined #openstack-infra | 23:18 | |
| *** openstackgerrit has joined #openstack-infra | 23:18 | |
| *** gmoro has joined #openstack-infra | 23:18 | |
| *** portante has joined #openstack-infra | 23:18 | |
| anteaya | we just had two running jobs move back to queued on 72297 in the gate, I had their console logs open in tabs | 23:20 |
| anteaya | do you want them in the etherpad? | 23:20 |
| jeblair | anteaya: nah, i think we're done diagnosing this until jenkins is downgraded | 23:21 |
| anteaya | k | 23:21 |
| *** lcostantino has quit IRC | 23:21 | |
| jeblair | fungi: so, um, shall we configure 8.8.8.8? | 23:23 |
| *** andre__ has quit IRC | 23:24 | |
| *** amcrn has quit IRC | 23:24 | |
| *** sarob has joined #openstack-infra | 23:24 | |
| jeblair | i filed a ticket and pinged them in chat | 23:24 |
| *** protux_ has quit IRC | 23:25 | |
| *** tjones has joined #openstack-infra | 23:25 | |
| *** protux_ has joined #openstack-infra | 23:25 | |
| fungi | jeblair: heh, yeah i seem to be able to get responses from other nameservers than theirs | 23:25 |
| fungi | on review.o.o | 23:25 |
| *** scottsanchez has quit IRC | 23:25 | |
| fungi | i guess we can just do that, puppet doesn't manage resolv.conf | 23:26 |
| *** boris-42_ has quit IRC | 23:26 | |
| *** boris-42_ has joined #openstack-infra | 23:26 | |
| *** tjones has quit IRC | 23:26 | |
| *** flaper87 is now known as flaper87|afk | 23:26 | |
| fungi | not so sure about 8.8.8.8 (or google in general) | 23:26 |
| fungi | but we should see if another rackspace resolved is working there | 23:27 |
| fungi | jeblair: for example 69.20.0.196 is one in iad | 23:28 |
| fungi | works fine | 23:28 |
| jeblair | fungi: ha, switch regions. good idea. :) | 23:28 |
| *** tjones has joined #openstack-infra | 23:28 | |
| mikal | Is there some way I can help with this Rax DNS problem? | 23:28 |
| fungi | 69.20.0.164 is the other one in iad | 23:28 |
| fungi | mikal: if you have a really, really long cricket bat maybe | 23:28 |
| mikal | Which region is borkroken? | 23:28 |
| mikal | (heh, best typo ever) | 23:29 |
| jeblair | mikal: i'm chatting with a first level tech and filed a ticket | 23:29 |
| fungi | mikal: one of our hosts in dfw can't get responses from either resolver in dfw | 23:29 |
| mikal | Ok, cool | 23:29 |
| mikal | Only one host though? | 23:29 |
| fungi | mikal: but we have other hosts in dfw which can | 23:29 |
| mikal | Huh | 23:29 |
| fungi | mikal: and the problem host can get responses from rax resolvers in other regions just fine | 23:29 |
| mikal | Sounds like network voodoo to me | 23:30 |
| mikal | Well, let me know if there's anything I can do | 23:30 |
| fungi | might be anycast issues | 23:30 |
| fungi | if they're using anycast | 23:30 |
| fungi | because i'm not seeing signs of other packet loss, just no response from those resolvers | 23:31 |
| jeblair | replacing resolv.conf with hosts from iad | 23:31 |
| *** protux_ has quit IRC | 23:31 | |
| fungi | sounds goos | 23:32 |
| fungi | good | 23:32 |
| jeblair | sdague, mordred: should be, um, 'resolved' now. | 23:32 |
| anteaya | rainya: ^ | 23:32 |
| rainya | anteaya, thanks will check | 23:32 |
| anteaya | thx | 23:32 |
| jeblair | #status notice Gerrit login issues should be resolved. | 23:33 |
| openstackstatus | NOTICE: Gerrit login issues should be resolved. | 23:33 |
| rainya | anteaya, jeblair: all better | 23:33 |
| anteaya | \o/ | 23:33 |
| *** completely-despe is now known as miguelzuniga | 23:33 | |
| anteaya | nice work jeblair fungi and rax 1st level tech guy | 23:33 |
| rainya | we were doing a "so you want to be an openstack contributor" tech talk this afternoon when we discovered it; i'll let the attendees know you guys rocked it | 23:34 |
| fungi | anteaya: i don't think anyone at rax has likely picked up the ticket yet | 23:34 |
| fungi | just a workaround in place for now | 23:34 |
| anteaya | does that affect the looping issue | 23:34 |
| fungi | anteaya: completely separate | 23:34 |
| fungi | other than the karmic aspects i suppose | 23:34 |
| anteaya | thought i would be generous with the blanket thanks | 23:35 |
| rainya | fungi, was the launchpad issue related to the dns questions you were pinging pvo about? | 23:35 |
| fungi | rainya: yeah, seems to have nothing to do with launchpad per se | 23:35 |
| *** ryanpetrello has quit IRC | 23:35 | |
| fungi | rainya: just that our gerrit server was trying to resolve an address record for launchpad and was getting no response from the dns resolvers in its region (dfw) | 23:35 |
| *** talluri has joined #openstack-infra | 23:36 | |
| anteaya | rainya: I jumped the gun | 23:36 |
| rainya | anteaya, you did? | 23:36 |
| anteaya | fungi: maybe we are getting our perfect storm of syncronous failures over with early? | 23:36 |
| anteaya | she said hopefully but not believing it | 23:36 |
| anteaya | thinking it might be launchpad and asking them about it | 23:36 |
| ken2ohmichi | thanks for solving, I can login it now! | 23:37 |
| anteaya | ken2ohmichi: thanks for letting us know | 23:37 |
| *** masayukig has joined #openstack-infra | 23:37 | |
| fungi | anteaya: it's only the tip of the iceberg, i'm sure | 23:39 |
| anteaya | yay | 23:39 |
| anteaya | more fun to come | 23:39 |
| *** talluri has quit IRC | 23:40 | |
| *** cadenzajon_ has joined #openstack-infra | 23:41 | |
| *** jhesketh_ has quit IRC | 23:43 | |
| fungi | jenkins01 down^Hupgraded | 23:43 |
| *** cadenzajon has quit IRC | 23:43 | |
| *** nati_uen_ has joined #openstack-infra | 23:44 | |
| jog0 | dumb question: how can I make tox use a custom indexserver? | 23:45 |
| *** nati_ueno has quit IRC | 23:47 | |
| *** virmitio has quit IRC | 23:47 | |
| mordred | jog0: there's an option ... one sec | 23:47 |
| mordred | jog0: permanently? or for an invocation? | 23:48 |
| jog0 | mordred: permanently | 23:48 |
| jog0 | so I don't have to do 'tox -i' everytime | 23:48 |
| *** masayukig has quit IRC | 23:48 | |
| mordred | jog0: hrm. I _think_ it will honor thigns you put into pip.conf | 23:49 |
| jog0 | can I have a global pip.conf in my root dir? | 23:49 |
| mordred | you can have one in ~/.pip/pip.conf | 23:50 |
| *** ryanpetrello has joined #openstack-infra | 23:51 | |
| jog0 | I tried that | 23:51 |
| *** ken2ohmichi has quit IRC | 23:52 | |
| *** dpyzhov has quit IRC | 23:53 | |
| jeblair | BobBall: have you tried the latest novaclient with nodepool? | 23:53 |
| *** ryanpetrello has quit IRC | 23:56 | |
| anteaya | fungi: so far jobs that are looping will only stop looping if they are on jenkins_01? | 23:56 |
| fungi | anteaya: well, jenkins01 hasn't started getting any nodepool nodes built for it yet, which is worrying me now as well | 23:56 |
| anteaya | hmmm | 23:57 |
| jeblair | fungi, clarkb: hpcloud az1 has 48038 keys | 23:57 |
| fungi | jeblair: that sounds like rather more than we need | 23:58 |
| jeblair | fungi: one of the defects i've noticed with nodepool is that when it is very busy, its main loop is slowed by the backlog of jenkins requests. | 23:58 |
| jeblair | fungi: right now, it seems like the main loop is in a 40-minute interval | 23:58 |
| fungi | got it | 23:58 |
| fungi | less worried in that case | 23:58 |
| jeblair | fungi: should hit any minute now | 23:58 |
| *** jergerber has quit IRC | 23:59 | |
| jeblair | fungi: but also, there are a lot of ready nodes which are probably not ready, likely related to your jenkins work. that's going to make it think it needs to start fewer nodes | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!