| *** CaptTofu has quit IRC | 00:01 | |
| *** CaptTofu has joined #openstack-infra | 00:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add d-g jobs for python-savannaclient https://review.openstack.org/74310 | 00:02 |
|---|---|---|
| *** jerryz has quit IRC | 00:04 | |
| *** CaptTofu has quit IRC | 00:07 | |
| *** ryanpetrello has joined #openstack-infra | 00:10 | |
| jeblair | clarkb: nice... | 00:12 |
| jeblair | clarkb: https://review.openstack.org/#/c/73850/ did not get enqueued into the gate because it did not have a +1 from jenkins. it had a +2. | 00:12 |
| clarkb | uh | 00:13 |
| jeblair | clarkb: you unapproved it yesterday | 00:13 |
| jeblair | but jenkins still left a +2 | 00:13 |
| clarkb | right, so unapproving worked but created more trouble | 00:13 |
| clarkb | :/ | 00:13 |
| *** matsuhashi has joined #openstack-infra | 00:15 | |
| SergeyLukjanov | will recheck help? | 00:15 |
| jeblair | SergeyLukjanov: yep. though it goes to the back of the line. :( | 00:15 |
| *** dims has quit IRC | 00:16 | |
| SergeyLukjanov | :( | 00:16 |
| *** weshay has joined #openstack-infra | 00:17 | |
| anteaya | I dont' see the upstream-translation-upstream job running by looking at either post or periodic in status.o.o/zuul | 00:17 |
| anteaya | is that node available? | 00:17 |
| anteaya | the jenkins link i have says it is offline | 00:17 |
| SergeyLukjanov | heh, finally going to sleep | 00:18 |
| anteaya | rest well SergeyLukjanov | 00:18 |
| lifeless | jeblair: today was a fire-fight yeah? | 00:18 |
| SergeyLukjanov | anteaya, thx | 00:18 |
| *** julim has quit IRC | 00:18 | |
| SergeyLukjanov | and have a good day / evening ;) | 00:18 |
| clarkb | anteaya: looking | 00:18 |
| clarkb | anteaya: yes it is offline | 00:19 |
| jeblair | lifeless: yep. your nodepool changes are waiting for me in tabs. | 00:19 |
| anteaya | k | 00:19 |
| clarkb | jeblair: I guess that means we didn't fix it completely | 00:19 |
| jeblair | clarkb: 1 sec | 00:19 |
| michchap | hello infra, I've somehow concocted a mess that means people can't add me for review on gerrit. My working account is michaeltchapman, which associated with a cisco.com email address that I no longer have access to, but I have also added a personal address to. There is another account which I can't access that has one of these addresses which I think is causing the problems. | 00:19 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Allow changes into gate with +2 from jenkins https://review.openstack.org/74877 | 00:19 |
| jeblair | clarkb: ^ | 00:19 |
| clarkb | jeblair: looking | 00:19 |
| clarkb | michchap: I can take a look in a short bit | 00:19 |
| mgagne | Is there a way to abandon someone else's change if you are "core" ? | 00:19 |
| clarkb | michchap: can you provide your account id number here? | 00:20 |
| michchap | clarkb: thanks. | 00:20 |
| *** david_lyle has quit IRC | 00:20 | |
| michchap | clarkb: 6994 is the working one | 00:20 |
| clarkb | jeblair: zuul ORs those events together right? | 00:21 |
| *** matsuhashi has quit IRC | 00:21 | |
| *** jhesketh__ has joined #openstack-infra | 00:21 | |
| *** matsuhashi has joined #openstack-infra | 00:21 | |
| *** matsuhashi has quit IRC | 00:21 | |
| jeblair | clarkb: yeah, separate triggers are or'd; (require-approvals are and'ed so that won't work) | 00:21 |
| *** matsuhashi has joined #openstack-infra | 00:22 | |
| *** markwash has quit IRC | 00:22 | |
| clarkb | approved | 00:23 |
| clarkb | ok now to debug gerrit accounts | 00:23 |
| *** dcramer_ has joined #openstack-infra | 00:24 | |
| anteaya | clarkb: thanks for getting proposal.slave.o.o working again | 00:24 |
| *** che-arne has quit IRC | 00:25 | |
| clarkb | anteaya: it will probably offline again in the not so distant future due to nodepool | 00:25 |
| jeblair | wow! we're no longer running at capacity (actually haven't been for about 25 mins) so basically there's no build backlog now. | 00:25 |
| clarkb | anteaya: but I thought we had fixed that so will need to dig in | 00:25 |
| clarkb | jeblair: yup and logstash is way behind now :) | 00:25 |
| clarkb | 184k events, basically we overran it | 00:25 |
| jeblair | clarkb: it should be able to start catching up soon | 00:25 |
| clarkb | jeblair: ya I hope so | 00:25 |
| *** markmcclain1 has quit IRC | 00:26 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Doc for project dependencies in gate https://review.openstack.org/66025 | 00:26 |
| *** prad has quit IRC | 00:26 | |
| *** dangers is now known as dangers_away | 00:27 | |
| anteaya | clarkb: if I see it loitering again I will bring it to your atttention | 00:27 |
| *** zul has quit IRC | 00:28 | |
| clarkb | jeblair: that is pretty amazing looking at the numbers | 00:28 |
| clarkb | michchap: your other account is id 7616 | 00:28 |
| clarkb | username michchap | 00:28 |
| *** dcramer_ has quit IRC | 00:29 | |
| *** cadenzajon has quit IRC | 00:29 | |
| clarkb | michchap: the openid for michaeltchapman on launchpad matches 7616 and not 6994 | 00:30 |
| *** nati_ueno has quit IRC | 00:30 | |
| clarkb | michchap: did you merge launchpad accounts at some point? | 00:30 |
| michchap | clarkb: I tried to merge the two launchpad accounts I had | 00:30 |
| clarkb | ah so that is what happened | 00:30 |
| *** dkranz has joined #openstack-infra | 00:30 | |
| clarkb | michchap: you want to keep 6994 michaeltchapman and kill 7616 michchap? | 00:31 |
| *** nati_ueno has joined #openstack-infra | 00:31 | |
| *** nati_ueno has quit IRC | 00:31 | |
| michchap | clarkb: yes | 00:31 |
| *** ryanpetrello has quit IRC | 00:31 | |
| *** dims has joined #openstack-infra | 00:32 | |
| clarkb | jeblair: fungi: so how does this work, do I need to change the openid for michaeltchapman, then remove all trace of 7616? | 00:32 |
| *** nati_ueno has joined #openstack-infra | 00:32 | |
| michchap | clarkb: I don't think the launchpad accounts are properly merged either, but that's probably less of an issue. | 00:32 |
| clarkb | michchap: I may draw this up as a launchpad bug and have fungi walk me through it when he is not afk | 00:32 |
| clarkb | I really need to do one of these account merge fixes and he is the guru | 00:32 |
| jeblair | clarkb: http://ci.openstack.org/gerrit.html#duplicate-accounts-in-gerrit | 00:33 |
| clarkb | oh we documented it | 00:33 |
| michchap | clarkb: Thanks, no rus | 00:33 |
| michchap | *rush | 00:33 |
| clarkb | jeblair: so that doesn't address the openid problem | 00:33 |
| clarkb | jeblair: won't michchap just login as 7616? | 00:34 |
| michchap | I can still login using the @cisco.com openid, which is how I've been working | 00:34 |
| clarkb | michchap: ah ok, so 6994 should be fine then, I will null out the email on 7616 for you | 00:35 |
| *** tjones has quit IRC | 00:36 | |
| *** jeckersb_gone is now known as jeckersb | 00:37 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable docs for python-savannaclient https://review.openstack.org/74470 | 00:40 |
| *** masayukig has joined #openstack-infra | 00:40 | |
| *** coolsvap has quit IRC | 00:41 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add more py3k-precise nodes https://review.openstack.org/73850 | 00:41 |
| lifeless | jeblair: kk, its our top priority to do what you need there; please ping me when you've reviewed so I can get right on it | 00:41 |
| *** dcramer_ has joined #openstack-infra | 00:42 | |
| clarkb | michchap: I have done the things in the doc jeblair linked to | 00:42 |
| clarkb | michchap: can you check that folks can add you now | 00:43 |
| michchap | clarkb: thanks, I'll see how it goes | 00:43 |
| anteaya | wow look at the fall off in the graph | 00:43 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow changes into gate with +2 from jenkins https://review.openstack.org/74877 | 00:43 |
| anteaya | recovery from fire fighting is much faster | 00:43 |
| *** SpamapS has quit IRC | 00:44 | |
| clarkb | jeblair: logstash down to 181k events so slowly falling | 00:44 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Toplevel collection typecheck. https://review.openstack.org/70997 | 00:45 |
| michchap | clarkb: Michael Chapman <woppin@XXX> is neither a registered user nor a group. | 00:45 |
| *** jerryz has joined #openstack-infra | 00:46 | |
| clarkb | michchap: I am able to ad 7616... I think the directions we have documented are not sufficient for this case | 00:46 |
| jeblair | lifeless: i'm reviewing derek's change now (it's shorter) | 00:46 |
| michchap | clarkb: Shall I wait until fungi is back then? | 00:47 |
| clarkb | michchap: I think I will create a bug and ya we can poke at it when the gate is less on fire | 00:48 |
| *** atiwari has quit IRC | 00:49 | |
| michchap | clarkb: sure. | 00:49 |
| clarkb | and firefox has crashed woo | 00:51 |
| StevenK | clarkb: Feature. | 00:52 |
| clarkb | michchap: https://bugs.launchpad.net/openstack-ci/+bug/1282332 | 00:52 |
| uvirtbot | Launchpad bug 1282332 in openstack-ci "Gerrit account cleanup" [Medium,Triaged] | 00:52 |
| anteaya | clarkb: ha ha ha | 00:52 |
| jerryz | clarkb: If OFFLINE_NODE_WHEN_COMPLETE is set in the job setting, will that work for gearman plugin? | 00:52 |
| jerryz | clarkb: cuz gerrit trigger will not send that parameter to jenkins | 00:53 |
| clarkb | jerryz: right you need to use zuul | 00:54 |
| clarkb | and tell zuul to mark the node offline in the gearman payload | 00:54 |
| *** sarob has quit IRC | 00:56 | |
| jerryz | clarkb: so set OFFLINE_NODE_WHEN_COMPLETE fixed to 1 in job parameters won't work? if i list job parameters, it is there. but the slave doesn't seem to be taken offline when a job is finished or aborted. | 00:57 |
| jeblair | jerryz: the gearman-plugin is what takes the node offline | 00:58 |
| clarkb | jerryz: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/openstack_functions.py#n34 is how we do it but the job need s to be triggered through the gearman plugin | 00:58 |
| *** yamahata has joined #openstack-infra | 00:59 | |
| *** sarob has joined #openstack-infra | 00:59 | |
| *** matsuhashi has quit IRC | 00:59 | |
| *** gokrokve has quit IRC | 01:00 | |
| *** gokrokve has joined #openstack-infra | 01:00 | |
| *** matsuhashi has joined #openstack-infra | 01:00 | |
| *** sarob_ has joined #openstack-infra | 01:01 | |
| *** SpamapS has joined #openstack-infra | 01:02 | |
| *** tjones has joined #openstack-infra | 01:02 | |
| *** VijayT has quit IRC | 01:03 | |
| *** gokrokve has quit IRC | 01:03 | |
| *** sarob has quit IRC | 01:04 | |
| jerryz | clarkb: ok. i see. so if the job is not triggered through gearman, the job parameters won't be read by gearman. | 01:04 |
| *** gokrokve has joined #openstack-infra | 01:05 | |
| *** tjones has quit IRC | 01:05 | |
| *** zul has joined #openstack-infra | 01:06 | |
| *** sarob_ has quit IRC | 01:06 | |
| clarkb | woo down to 178k events | 01:08 |
| clarkb | jerryz: correct | 01:08 |
| *** tjones has joined #openstack-infra | 01:09 | |
| dims | clarkb, jeblair, fungi - when you get a chance, can you please take a peek at this review - https://review.openstack.org/#/c/74669/ - will help unblock oslo.vmware patches (bad requirements.txt in imported repo and old projects.txt in image causing doc build failure) | 01:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Make nodepool more robust to offline clouds. https://review.openstack.org/74521 | 01:11 |
| jeblair | lifeless: ^ (i fixed the pep8 error; lgtm) | 01:11 |
| jeblair | lifeless: i also reviewed derek's change | 01:12 |
| *** tjones has quit IRC | 01:12 | |
| *** mriedem has joined #openstack-infra | 01:12 | |
| jeblair | fungi, clarkb, mordred, SergeyLukjanov, mattoliverau: https://review.openstack.org/#/c/74521/ | 01:13 |
| *** wenlock has quit IRC | 01:13 | |
| *** packet has quit IRC | 01:13 | |
| clarkb | jeblair: looking | 01:14 |
| *** mriedem has quit IRC | 01:14 | |
| jeblair | also, i'd love to roll https://review.openstack.org/#/c/74773/ into the next nodepool restart, but it's not blocking anyone. | 01:14 |
| clarkb | jeblair: I take it your +2 on https://review.openstack.org/#/c/66173/12 indicates you are ok with the current state of that test? | 01:15 |
| *** mriedem has joined #openstack-infra | 01:15 | |
| jeblair | clarkb: regarding the az2 thing -- i think it only affects us on image creation, and there we haven't had an image created since feb 12 | 01:16 |
| *** jeckersb is now known as jeckersb_gone | 01:16 | |
| clarkb | jeblair: interesting | 01:16 |
| *** mriedem has quit IRC | 01:17 | |
| *** dstanek has quit IRC | 01:17 | |
| clarkb | I approved jhesketh_'s change | 01:17 |
| clarkb | now onto nodepool | 01:17 |
| *** mriedem has joined #openstack-infra | 01:17 | |
| jeblair | clarkb: yeah, i'm okay with it; i'd say it's up to jhesketh_ if he wants to update it with my suggestions https://review.openstack.org/#/c/66173/12 | 01:17 |
| jeblair | which he can now do in a follow-up patch if he decides to. :) | 01:17 |
| *** sarob has joined #openstack-infra | 01:17 | |
| jhesketh__ | right, I did see your comment, thanks jeblair, just didn't have a chance to reroll | 01:18 |
| jhesketh__ | since it's been merged I'll leave it as is for now | 01:18 |
| anteaya | ha ha ha | 01:18 |
| jhesketh__ | the loop method shouldn't be any slower than pushing the job through | 01:18 |
| *** gokrokve has quit IRC | 01:18 | |
| *** gokrokve has joined #openstack-infra | 01:19 | |
| jhesketh__ | clarkb: if you have time to look at my comments on this one that'd be great: https://review.openstack.org/#/c/73461/ | 01:19 |
| clarkb | jhesketh__: right that sleep(0) is quite the busy loop :) | 01:19 |
| openstackgerrit | David Johansen proposed a change to openstack-infra/jenkins-job-builder: Added support for least load plugin. https://review.openstack.org/71316 | 01:20 |
| jhesketh__ | heh, I did that in testing and never bothered to up it to anything somewhat sane, but it should only be for a second or so | 01:20 |
| *** tjones has joined #openstack-infra | 01:21 | |
| jhesketh__ | jeblair: and when you're less snowed under some feedback on the swift stuff here would be appreciated: https://review.openstack.org/#/c/68297/ | 01:21 |
| jeblair | jhesketh__: definitely; it's next on my list | 01:22 |
| jhesketh__ | thanks mate :-) | 01:22 |
| *** gokrokve has quit IRC | 01:23 | |
| lifeless | jeblair: thanks | 01:23 |
| clarkb | jeblair: lifeless' change lgtm and has been approved, looking at yours now | 01:24 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Make nodepool more robust to offline clouds. https://review.openstack.org/74521 | 01:26 |
| *** tjones has quit IRC | 01:28 | |
| *** khyati has joined #openstack-infra | 01:29 | |
| *** tjones has joined #openstack-infra | 01:29 | |
| openstackgerrit | A change was merged to openstack/requirements: Update neutron-client minimum to 2.3.4 https://review.openstack.org/74805 | 01:30 |
| openstackgerrit | A change was merged to openstack/requirements: Update openstack-doc-tools to 0.7.1 https://review.openstack.org/74827 | 01:30 |
| clarkb | jeblair: comment on your nodepool change | 01:30 |
| anteaya | clarkb: proposal.slave.o.o has a job to run and is offline again | 01:35 |
| clarkb | anteaya: gah | 01:35 |
| anteaya | yeah | 01:35 |
| anteaya | it was idle | 01:35 |
| *** ArxCruz has joined #openstack-infra | 01:35 | |
| anteaya | then I checked it when I saw a job waiting | 01:35 |
| lifeless | jeblair: so https://review.openstack.org/#/c/73863/ would be next then | 01:36 |
| clarkb | anteaya: ok I amgoing to look at this closely now because I am tired of kicking it | 01:36 |
| anteaya | k | 01:36 |
| anteaya | go you | 01:36 |
| *** jcoufal has quit IRC | 01:36 | |
| anteaya | so far just one queued job for it | 01:36 |
| *** tjones has quit IRC | 01:38 | |
| *** weshay has quit IRC | 01:38 | |
| *** nosnos has joined #openstack-infra | 01:39 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Fix reusable node regex for proposal node. https://review.openstack.org/74888 | 01:40 |
| *** elasticio has joined #openstack-infra | 01:40 | |
| clarkb | anteaya: jeblair ^ | 01:40 |
| *** melwitt has quit IRC | 01:40 | |
| jeblair | clarkb: i think that regex is too complicated and we should list the singleton jobs individually | 01:42 |
| jeblair | clarkb: but i aprvd | 01:42 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try Icehouse precise proposed UCA - Not for review https://review.openstack.org/74889 | 01:42 |
| jeblair | clarkb: and replied to your comment | 01:42 |
| clarkb | jeblair: ok | 01:43 |
| jhesketh__ | lifeless: why not add non-voting jobs in the experimental pipeline rather than creating new pipelines for tripleO? | 01:44 |
| *** dstanek has joined #openstack-infra | 01:44 | |
| clarkb | jeblair: are you worried about leaks like that? | 01:44 |
| clarkb | jeblair: I am happy to run it and see if it leaks in practice. I am guessing that openstack can't give us a 2 second turn around from build to delete | 01:45 |
| *** dolphm_503 is now known as dolphm | 01:45 | |
| lifeless | jhesketh__: because folk that look at the pipelines get worried when stuff stalls | 01:45 |
| lifeless | jhesketh__: new pipelines make it clear that if this happens the d-g set of jobs are not affected | 01:46 |
| lifeless | jhesketh__: we hope | 01:46 |
| jhesketh__ | Yep, okay :-) | 01:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix reusable node regex for proposal node. https://review.openstack.org/74888 | 01:46 |
| lifeless | jhesketh__: as for experimental vs check - these jobs are ready to vote, but we don't have 2+ regions yet. | 01:46 |
| jhesketh__ | right | 01:47 |
| anteaya | clarkb: well done | 01:48 |
| *** dstanek has quit IRC | 01:48 | |
| *** ryanpetrello has joined #openstack-infra | 01:48 | |
| mikal | Why https://review.openstack.org/#/c/63617/ no merge? | 01:52 |
| *** tjones has joined #openstack-infra | 01:52 | |
| *** banix has joined #openstack-infra | 01:58 | |
| *** sarob has quit IRC | 02:01 | |
| *** zul has quit IRC | 02:01 | |
| *** dstanek has joined #openstack-infra | 02:01 | |
| *** sarob has joined #openstack-infra | 02:01 | |
| anteaya | mikal: it might have gotten lost in the fire fighting | 02:02 |
| anteaya | jenkins never started the gate jobs | 02:02 |
| anteaya | can it be reapproved? | 02:02 |
| *** CaptTofu has joined #openstack-infra | 02:02 | |
| *** ryanpetrello has quit IRC | 02:03 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try Icehouse precise proposed UCA - Not for review https://review.openstack.org/74889 | 02:04 |
| mikal | anteaya: I can do that | 02:04 |
| mikal | I just wanted to make sure it wasn't lost in a world of pain or something | 02:04 |
| *** lttrl has joined #openstack-infra | 02:04 | |
| anteaya | right, from my seat here I think it got lost in the shuffle | 02:05 |
| anteaya | hopefully a small push should get it back in the swing of things | 02:05 |
| *** sarob has quit IRC | 02:05 | |
| *** CaptTofu has quit IRC | 02:07 | |
| *** dripton has quit IRC | 02:09 | |
| *** dcramer_ has quit IRC | 02:10 | |
| *** yamahata has quit IRC | 02:10 | |
| *** yamahata has joined #openstack-infra | 02:10 | |
| *** mgagne has quit IRC | 02:17 | |
| *** yamahata has quit IRC | 02:18 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try Icehouse precise proposed UCA - Not for review https://review.openstack.org/74889 | 02:20 |
| *** noorul has joined #openstack-infra | 02:21 | |
| noorul | hi | 02:21 |
| noorul | I see that a fix is released for https://bugs.launchpad.net/openstack-ci/+bug/1282136 | 02:21 |
| uvirtbot | Launchpad bug 1282136 in openstack-ci "Git problem: "Failed to resolve 'HEAD' as a valid ref."" [Undecided,Fix released] | 02:21 |
| noorul | But the problem still exists. | 02:21 |
| clarkb | noorul: do you have the latest nodepool scripts? | 02:21 |
| noorul | Oops, does that get committed just now? | 02:22 |
| *** gokrokve has joined #openstack-infra | 02:22 | |
| clarkb | noorul: its in the bug | 02:22 |
| *** tjones has quit IRC | 02:22 | |
| noorul | clarkb: How can I check whether a slave has the latest scripts? | 02:22 |
| clarkb | noorul: where did you see the failure? | 02:23 |
| *** gokrokve_ has joined #openstack-infra | 02:23 | |
| noorul | https://jenkins01.openstack.org/job/gate-solum-python27/555/console | 02:24 |
| clarkb | noorul: the image for az2 has not updated yet | 02:24 |
| clarkb | the cloud provider is having trouble, we have submitted a ticket to them and hopeflly that will get resolved | 02:24 |
| noorul | Oh I see | 02:25 |
| noorul | clarkb: Thanks for the clarification | 02:25 |
| clarkb | np | 02:25 |
| *** sarob has joined #openstack-infra | 02:25 | |
| *** gokrokv__ has joined #openstack-infra | 02:27 | |
| Shrews | For the record, fungi is a bad influence | 02:27 |
| Shrews | Just so everyone knows | 02:27 |
| clarkb | Shrews: are you all unable to walk a straight line now? | 02:27 |
| *** gokrokve has quit IRC | 02:27 | |
| *** sarob has quit IRC | 02:28 | |
| *** gokrokve_ has quit IRC | 02:28 | |
| Shrews | less so than 2 hours ago | 02:29 |
| *** sarob has joined #openstack-infra | 02:29 | |
| Shrews | nothing that bad. he's just a trouble maker | 02:30 |
| *** yaguang has joined #openstack-infra | 02:31 | |
| *** CaptTofu has joined #openstack-infra | 02:32 | |
| *** yamahata has joined #openstack-infra | 02:33 | |
| *** zul has joined #openstack-infra | 02:34 | |
| anteaya | Shrews: glad to hear it | 02:34 |
| *** david-lyle has joined #openstack-infra | 02:35 | |
| *** nati_ueno has quit IRC | 02:39 | |
| *** yamahata has quit IRC | 02:40 | |
| *** morganfainberg is now known as morganfainberg_Z | 02:42 | |
| jerryz | clarkb: just let you know that i used the old devstack-vm-inprogress.py script to re-label the node to used, ugly but it works. no more build on the old node | 02:43 |
| *** elasticio has quit IRC | 02:48 | |
| *** AaronGr is now known as AaronGr_Zzz | 02:49 | |
| *** esker has joined #openstack-infra | 02:54 | |
| lifeless | clarkb: https://review.openstack.org/#/c/73863/ | 02:55 |
| *** markwash has joined #openstack-infra | 02:56 | |
| *** yamahata has joined #openstack-infra | 02:58 | |
| *** wenlock has joined #openstack-infra | 02:58 | |
| jhesketh__ | hmm, is the git server flaky for anybody else? | 02:59 |
| *** mgagne has joined #openstack-infra | 03:00 | |
| *** sarob has quit IRC | 03:01 | |
| jhesketh__ | ah, seems like I'm hitting bug/1282136 | 03:01 |
| jhesketh__ | maybe my jobs keep running on servers with the old images | 03:02 |
| *** morganfainberg_Z is now known as morganfainberg | 03:02 | |
| jog0 | jhesketh__: http://status.openstack.org/elastic-recheck/data/uncategorized.html shows we don't know whats failing the gate | 03:03 |
| *** mriedem has quit IRC | 03:05 | |
| jog0 | is there a bug filed for http://logs.openstack.org/21/73921/4/gate/gate-tempest-dsvm-large-ops/160dc0f/console.html | 03:06 |
| jog0 | curl: (7) couldn't connect to host | 03:06 |
| *** AaronGr_Zzz is now known as AaronGr | 03:06 | |
| *** rcleere has joined #openstack-infra | 03:06 | |
| jog0 | is that the rax network issue? | 03:07 |
| noorul | jhesketh__: clarkb said there are some issues with az2 nodes | 03:07 |
| jhesketh__ | from bug 1282136: "this seems to be something wrong with the git repos that are cached on the slave images." | 03:08 |
| uvirtbot | Launchpad bug 1282136 in openstack-ci "Git problem: "Failed to resolve 'HEAD' as a valid ref."" [Undecided,Fix released] https://launchpad.net/bugs/1282136 | 03:08 |
| jog0 | http://logs.openstack.org/10/68410/9/gate/gate-tempest-dsvm-large-ops/e2ede17/console.html#_2014-02-19_00_59_15_788 | 03:09 |
| jog0 | WAT | 03:09 |
| *** wenlock_ has joined #openstack-infra | 03:09 | |
| jog0 | message:"You are already running a stack.sh session." | 03:10 |
| jog0 | that looks like the most frequent failure | 03:10 |
| jog0 | 424 hits | 03:11 |
| jog0 | in 48 hours | 03:11 |
| mordred | jog0: that seems like a strange error | 03:11 |
| jog0 | in both hp and rax | 03:11 |
| jog0 | mordred: it went away | 03:11 |
| jog0 | already | 03:11 |
| jog0 | ah bug 1103082 | 03:13 |
| uvirtbot | Launchpad bug 1103082 in openstack-ci "Build failed with " You are already running a stack.sh session"" [Low,Triaged] https://launchpad.net/bugs/1103082 | 03:13 |
| jog0 | clarkb: logstash is still running right? | 03:14 |
| jog0 | and getting new data taht is | 03:14 |
| clarkb | ish | 03:14 |
| clarkb | its way behind after dfw fell over | 03:14 |
| clarkb | and it has ETOOMUCHDATA again | 03:15 |
| *** wenlock has quit IRC | 03:16 | |
| clarkb | was watching it all day. its basically keeping up but not catching up | 03:16 |
| jog0 | asking so I can figure out if the query message:"You are already running a stack.sh session." AND filename:"console.html" is accurate | 03:16 |
| jog0 | as in bug fixed | 03:16 |
| jog0 | clarkb: speaking of which about to add that to e-r for bug ^^^ but wanted to see if you had any comments first | 03:17 |
| *** nati_ueno has joined #openstack-infra | 03:18 | |
| clarkb | not now. dinner now then probably afk | 03:18 |
| jog0 | clarkb: kk, in that case here it goes anyway | 03:18 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1103082 https://review.openstack.org/74913 | 03:20 |
| uvirtbot | Launchpad bug 1103082 in openstack-ci "Build failed with " You are already running a stack.sh session"" [Low,Triaged] https://launchpad.net/bugs/1103082 | 03:20 |
| *** nati_ueno has quit IRC | 03:22 | |
| clarkb | jog0 I think that is fixed | 03:26 |
| *** esker has quit IRC | 03:26 | |
| clarkb | it prompted the jenkins downgrade | 03:27 |
| *** esker has joined #openstack-infra | 03:27 | |
| jog0 | clarkb: o_O more jenkins issues sigh | 03:27 |
| *** dpyzhov has joined #openstack-infra | 03:27 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1103082 https://review.openstack.org/74913 | 03:28 |
| uvirtbot | Launchpad bug 1103082 in openstack-ci "Build failed with " You are already running a stack.sh session"" [Low,Triaged] https://launchpad.net/bugs/1103082 | 03:28 |
| *** dpyzhov has quit IRC | 03:29 | |
| *** basha has joined #openstack-infra | 03:31 | |
| *** lcheng has joined #openstack-infra | 03:31 | |
| *** khyati has quit IRC | 03:32 | |
| *** dpyzhov has joined #openstack-infra | 03:34 | |
| ianw | gerrit seems to have stopped sending me emails for watched projects (i'm pretty sure it's not an issue at my end). Anyone know if this is possible? | 03:41 |
| *** dpyzhov has quit IRC | 03:41 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1282262 https://review.openstack.org/74919 | 03:41 |
| uvirtbot | Launchpad bug 1282262 in openstack-ci "Permission denied errors on /opt during devstack" [Undecided,New] https://launchpad.net/bugs/1282262 | 03:41 |
| annegentle | fungi: this seems bad "Your change requires a recursive merge to resolve" on https://review.openstack.org/#/c/74777/, my first attempt at porting from master to feature/edits in the operations-guide... no need to look at it now but I'll be tackling tomorrow. | 03:42 |
| *** CaptTofu has quit IRC | 03:42 | |
| *** CaptTofu has joined #openstack-infra | 03:43 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1282262 https://review.openstack.org/74919 | 03:43 |
| uvirtbot | Launchpad bug 1282262 in openstack-ci "Permission denied errors on /opt during devstack" [Undecided,New] https://launchpad.net/bugs/1282262 | 03:43 |
| *** jeckersb_gone has quit IRC | 03:44 | |
| jog0 | mtreinish: strange bug ^ | 03:45 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Adds query for bug #1278988 https://review.openstack.org/72713 | 03:46 |
| uvirtbot | Launchpad bug 1278988 in openstack-ci "wrong fs type, bad option, bad superblock on /dev/loop2" [Undecided,New] https://launchpad.net/bugs/1278988 | 03:46 |
| *** matsuhashi has quit IRC | 03:46 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try Icehouse precise proposed UCA - Not for review https://review.openstack.org/74889 | 03:47 |
| jog0 | dims: woot ^ | 03:47 |
| *** CaptTofu has quit IRC | 03:47 | |
| *** jeckersb_gone has joined #openstack-infra | 03:48 | |
| dims | jog0, :) cross out collective fingers | 03:49 |
| *** sarob has joined #openstack-infra | 03:49 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1282262 https://review.openstack.org/74919 | 03:50 |
| uvirtbot | Launchpad bug 1282262 in openstack-ci "Permission denied errors on /opt during devstack" [Undecided,New] https://launchpad.net/bugs/1282262 | 03:50 |
| *** sandywalsh_ has quit IRC | 03:52 | |
| cyeoh | is anyone else having trouble getting to review.openstack.org? | 03:54 |
| mordred | cyeoh: works for me | 03:55 |
| cyeoh | mordred: hrm. Seems to be working again for me now :-) Must have just been a network hiccup between me and there for a bit.... | 03:56 |
| *** dpyzhov has joined #openstack-infra | 04:08 | |
| *** sandywalsh has joined #openstack-infra | 04:08 | |
| *** kiall has quit IRC | 04:08 | |
| *** rcleere has quit IRC | 04:09 | |
| *** pcrews has quit IRC | 04:10 | |
| *** Ryan_Lane has joined #openstack-infra | 04:10 | |
| *** kiall has joined #openstack-infra | 04:11 | |
| *** ryanpetrello has joined #openstack-infra | 04:12 | |
| *** pcrews has joined #openstack-infra | 04:15 | |
| *** ArxCruz has quit IRC | 04:17 | |
| *** chandan_kumar has joined #openstack-infra | 04:20 | |
| *** wenlock_ has quit IRC | 04:24 | |
| *** pcrews has quit IRC | 04:26 | |
| *** matsuhashi has joined #openstack-infra | 04:31 | |
| *** sarob has quit IRC | 04:32 | |
| *** banix has quit IRC | 04:34 | |
| *** chandan_kumar has quit IRC | 04:37 | |
| *** sarob has joined #openstack-infra | 04:46 | |
| *** miqui has joined #openstack-infra | 04:47 | |
| *** dpyzhov has quit IRC | 04:47 | |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack-infra/elastic-recheck: Change 1254872 for catching issues from libvirtd https://review.openstack.org/74925 | 04:47 |
| *** unicell has quit IRC | 04:52 | |
| *** unicell has joined #openstack-infra | 04:52 | |
| *** unicell has quit IRC | 04:56 | |
| *** unicell has joined #openstack-infra | 04:57 | |
| *** dpyzhov has joined #openstack-infra | 04:57 | |
| *** dpyzhov has quit IRC | 05:00 | |
| *** unicell has quit IRC | 05:01 | |
| *** chandan_kumar has joined #openstack-infra | 05:06 | |
| *** SpamapS has quit IRC | 05:06 | |
| *** dstanek_afk has joined #openstack-infra | 05:08 | |
| *** sarob has quit IRC | 05:10 | |
| *** dstanek has quit IRC | 05:10 | |
| *** gokrokv__ has quit IRC | 05:11 | |
| *** gokrokve has joined #openstack-infra | 05:11 | |
| *** khyati has joined #openstack-infra | 05:12 | |
| *** jcooley_ has joined #openstack-infra | 05:13 | |
| *** gokrokve_ has joined #openstack-infra | 05:16 | |
| *** gokrokve has quit IRC | 05:16 | |
| *** bradm has quit IRC | 05:18 | |
| *** zhiyan_ is now known as zhiyan | 05:19 | |
| *** gokrokve_ has quit IRC | 05:20 | |
| *** rockstar has quit IRC | 05:20 | |
| *** gokrokve has joined #openstack-infra | 05:20 | |
| *** unicell has joined #openstack-infra | 05:22 | |
| *** bradm has joined #openstack-infra | 05:22 | |
| *** rockstar has joined #openstack-infra | 05:23 | |
| *** rockstar has quit IRC | 05:23 | |
| *** rockstar has joined #openstack-infra | 05:23 | |
| *** SpamapS has joined #openstack-infra | 05:23 | |
| *** SpamapS has joined #openstack-infra | 05:23 | |
| *** ryanpetrello has quit IRC | 05:24 | |
| *** bhuvan has joined #openstack-infra | 05:24 | |
| *** gokrokve has quit IRC | 05:25 | |
| *** nicedice has quit IRC | 05:28 | |
| lifeless | clarkb: any chance of a review on https://review.openstack.org/#/c/73863/ | 05:31 |
| lifeless | clarkb: its a necessary condition for getting the ci-overcloud back online, so I'm rather interested in it :) | 05:31 |
| clarkb | lifeless: in the morning? | 05:33 |
| clarkb | I am being forced to watch old episodes of jeopardy | 05:33 |
| lifeless | ok | 05:34 |
| lifeless | once thats done we need to then reinstate the nodepool config for the tripleo cloud and the zuul config (but edited) with our check jobs | 05:34 |
| *** khyati has quit IRC | 05:42 | |
| *** ayoung is now known as ayoung-ZzZzZzZ | 05:43 | |
| *** CaptTofu has joined #openstack-infra | 05:44 | |
| *** inahandizha has joined #openstack-infra | 05:45 | |
| *** inahandizha has left #openstack-infra | 05:47 | |
| *** yolanda has joined #openstack-infra | 05:48 | |
| *** CaptTofu has quit IRC | 05:48 | |
| *** loquacities has quit IRC | 05:56 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Allow workers to send back metadata https://review.openstack.org/66173 | 05:56 |
| *** loquacities has joined #openstack-infra | 05:58 | |
| *** talluri has joined #openstack-infra | 05:58 | |
| *** DinaBelova_ is now known as DinaBelova | 05:59 | |
| *** noorul has left #openstack-infra | 06:04 | |
| SergeyLukjanov | jeblair, I'm late, but https://review.openstack.org/#/c/74521/2 lgtm too | 06:05 |
| *** dpyzhov has joined #openstack-infra | 06:13 | |
| *** persia has quit IRC | 06:14 | |
| *** persia has joined #openstack-infra | 06:15 | |
| *** persia is now known as Guest91627 | 06:15 | |
| *** bhuvan has quit IRC | 06:15 | |
| *** gokrokve has joined #openstack-infra | 06:16 | |
| *** dpyzhov has quit IRC | 06:18 | |
| *** skraynev_afk is now known as skraynev | 06:19 | |
| *** gokrokve has quit IRC | 06:21 | |
| *** gokrokve has joined #openstack-infra | 06:22 | |
| *** cadenzajon has joined #openstack-infra | 06:26 | |
| *** gokrokve has quit IRC | 06:27 | |
| *** dpyzhov has joined #openstack-infra | 06:27 | |
| *** nati_ueno has joined #openstack-infra | 06:28 | |
| openstackgerrit | YAMAMOTO Takashi proposed a change to openstack-infra/config: run-unittests.sh: fix a typo https://review.openstack.org/74941 | 06:29 |
| *** cadenzajon has quit IRC | 06:30 | |
| *** mrda is now known as mrda_away | 06:36 | |
| *** lcheng has quit IRC | 06:36 | |
| *** chandan_kumar has quit IRC | 06:38 | |
| mordred | jeblair: I found a fascinating edge case in nodepool | 06:39 |
| mordred | that I never expect infra to run in to | 06:39 |
| mordred | jeblair: and by I, I pretty much mean AaronGr, clarkb and jog0 | 06:40 |
| mordred | jeblair: and then I senselessly just started patching things | 06:41 |
| *** nati_ueno has quit IRC | 06:42 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Reset the client object before creating the image https://review.openstack.org/74943 | 06:45 |
| mordred | jeblair: that ^^ is not a general solution, but it does fix a symptom | 06:45 |
| mordred | jeblair: I'd love suggestions as to how to achieve a similar thing except in a way that might not be crazy | 06:46 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 06:48 | |
| *** jhesketh_ has quit IRC | 06:50 | |
| *** jhesketh__ has quit IRC | 06:50 | |
| *** jhesketh_ has joined #openstack-infra | 06:51 | |
| *** dpyzhov has quit IRC | 06:51 | |
| *** masayukig has quit IRC | 06:57 | |
| *** vkozhukalov has joined #openstack-infra | 06:58 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 07:00 | |
| *** matsuhashi has quit IRC | 07:01 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 07:02 |
| *** matsuhashi has joined #openstack-infra | 07:03 | |
| *** bhuvan has joined #openstack-infra | 07:03 | |
| *** jhesketh__ has joined #openstack-infra | 07:03 | |
| *** chandan_kumar has joined #openstack-infra | 07:03 | |
| *** DinaBelova is now known as DinaBelova_ | 07:08 | |
| *** bhuvan has quit IRC | 07:09 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 07:09 | |
| *** Guest91627 has quit IRC | 07:14 | |
| *** Guest91627 has joined #openstack-infra | 07:14 | |
| *** saju_m has joined #openstack-infra | 07:15 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 07:18 | |
| *** saju_m has quit IRC | 07:20 | |
| *** saju_m has joined #openstack-infra | 07:21 | |
| *** gokrokve has joined #openstack-infra | 07:22 | |
| *** mgagne has quit IRC | 07:23 | |
| *** mgagne has joined #openstack-infra | 07:23 | |
| *** hashar has joined #openstack-infra | 07:26 | |
| *** saju_m has quit IRC | 07:26 | |
| *** hashar has quit IRC | 07:26 | |
| *** saju_m has joined #openstack-infra | 07:26 | |
| *** gokrokve has quit IRC | 07:27 | |
| *** DinaBelova_ is now known as DinaBelova | 07:31 | |
| openstackgerrit | Peter Liljenberg proposed a change to openstack-infra/jenkins-job-builder: Added support for Jenkins Campfire plugin https://review.openstack.org/69315 | 07:32 |
| *** morganfainberg is now known as morganfainberg_Z | 07:32 | |
| *** rlandy has joined #openstack-infra | 07:38 | |
| *** CaptTofu has joined #openstack-infra | 07:44 | |
| *** david_lyle_ has joined #openstack-infra | 07:44 | |
| *** mayu has joined #openstack-infra | 07:45 | |
| *** flaper87|afk is now known as flaper87 | 07:47 | |
| *** david-lyle has quit IRC | 07:47 | |
| *** saju_m has quit IRC | 07:49 | |
| *** CaptTofu has quit IRC | 07:49 | |
| mayu | I want to apply an service count for third party ci, can anybody tell me how ? | 07:50 |
| amotoki | mayu: I see many folks requesting a service account on -infra list. I think it is the best way to request it. | 07:57 |
| *** e0ne has joined #openstack-infra | 07:58 | |
| *** nati_ueno has joined #openstack-infra | 07:58 | |
| mayu | thanks amotoki | 07:59 |
| *** nati_ueno has quit IRC | 08:00 | |
| *** afazekas has joined #openstack-infra | 08:02 | |
| *** marun has quit IRC | 08:05 | |
| *** dpyzhov has joined #openstack-infra | 08:09 | |
| *** luqas has joined #openstack-infra | 08:10 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 08:11 | |
| *** mrmartin has joined #openstack-infra | 08:16 | |
| mrmartin | morning | 08:16 |
| SergeyLukjanov | mayu, ack, you should ask for account in openstack-infra list, some instructions - http://ci.openstack.org/third_party.html#requesting-a-service-account | 08:18 |
| SergeyLukjanov | mrmartin, o/ | 08:18 |
| openstackgerrit | Shane Wang proposed a change to openstack-infra/storyboard: Fix misspellings in storyboard https://review.openstack.org/72053 | 08:18 |
| mrmartin | SergeyLukjanov: could you check me this openstack-infra/groups gating jobs: http://status.openstack.org/zuul/ ? | 08:19 |
| mrmartin | this seems to be the only stucked job currently, can we do something with that? It seems to be still queued. | 08:19 |
| *** zhiyan has quit IRC | 08:20 | |
| mayu | thanks SergeyLukjanov> | 08:21 |
| SergeyLukjanov | mrmartin, looking | 08:21 |
| mrmartin | thanks | 08:21 |
| *** gokrokve has joined #openstack-infra | 08:22 | |
| SergeyLukjanov | mrmartin, looks like lack of slaves for running your jobs | 08:23 |
| *** cody-somerville has quit IRC | 08:23 | |
| mrmartin | so what will be the solution? Am I need to wait or do we need to do some intervention here? This job sits there more than 24 hours... | 08:25 |
| mrmartin | could we retrigger it somehow? | 08:25 |
| *** gokrokve has quit IRC | 08:27 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Fix gate-groups-unittests to use bare-precise https://review.openstack.org/74959 | 08:27 |
| SergeyLukjanov | mrmartin, here is a fix for it ^^ | 08:28 |
| *** unicell has quit IRC | 08:28 | |
| SergeyLukjanov | mrmartin, I'll retrigger your job when it'll be landed | 08:28 |
| *** unicell has joined #openstack-infra | 08:28 | |
| SergeyLukjanov | mrmartin, ETA ~2-3h | 08:28 |
| mrmartin | SergeyLukjanov, do you think that this bare-precise will work well? We need the php5 packages for this platform. | 08:29 |
| mrmartin | and the same is true for openstackid gating | 08:29 |
| *** chandan_kumar has quit IRC | 08:29 | |
| *** markwash has quit IRC | 08:30 | |
| SergeyLukjanov | mrmartin, I think yes, because precise node was built using the same scripts, bare-precise is just a single-use analogue | 08:30 |
| SergeyLukjanov | mrmartin, I'll check it again | 08:30 |
| SergeyLukjanov | gtg | 08:31 |
| *** zhiyan has joined #openstack-infra | 08:31 | |
| mrmartin | https://github.com/openstack-infra/config/blob/master/modules/jenkins/manifests/slave.pp : if ($bare == false) { | 08:31 |
| mrmartin | $packages = [$common_packages, $standard_packages] | 08:31 |
| mrmartin | } else { | 08:31 |
| mrmartin | $packages = $common_packages | 08:31 |
| mrmartin | } | 08:31 |
| *** jamielennox is now known as jamielennox|away | 08:31 | |
| mrmartin | and php5-cli, php5-mcrypt packages were included in $standard_packages | 08:32 |
| *** jcoufal has joined #openstack-infra | 08:32 | |
| *** DinaBelova is now known as DinaBelova_ | 08:34 | |
| *** dpyzhov has quit IRC | 08:36 | |
| *** chandan_kumar has joined #openstack-infra | 08:37 | |
| *** thomasbiege has joined #openstack-infra | 08:37 | |
| *** cody-somerville has joined #openstack-infra | 08:37 | |
| *** Ryan_Lane has quit IRC | 08:38 | |
| *** mayu has quit IRC | 08:40 | |
| *** rossella-s has joined #openstack-infra | 08:42 | |
| *** vkozhukalov has quit IRC | 08:44 | |
| *** saju_m has joined #openstack-infra | 08:45 | |
| *** Hunner has quit IRC | 08:47 | |
| *** Hunner has joined #openstack-infra | 08:49 | |
| *** jroovers has joined #openstack-infra | 08:49 | |
| *** david_lyle_ has quit IRC | 09:04 | |
| *** yassine has joined #openstack-infra | 09:05 | |
| *** jpich has joined #openstack-infra | 09:07 | |
| *** jgallard has joined #openstack-infra | 09:08 | |
| *** masayukig has joined #openstack-infra | 09:11 | |
| *** dpyzhov has joined #openstack-infra | 09:11 | |
| *** markmc has joined #openstack-infra | 09:18 | |
| *** dpyzhov has quit IRC | 09:21 | |
| *** boris-42_ has quit IRC | 09:22 | |
| *** gokrokve has joined #openstack-infra | 09:22 | |
| *** gokrokve has quit IRC | 09:26 | |
| *** david-lyle has joined #openstack-infra | 09:27 | |
| *** dpyzhov has joined #openstack-infra | 09:29 | |
| *** matsuhashi has quit IRC | 09:29 | |
| *** matsuhashi has joined #openstack-infra | 09:30 | |
| *** fbo_away is now known as fbo | 09:35 | |
| *** dizquierdo has joined #openstack-infra | 09:39 | |
| *** salv-orlando has quit IRC | 09:40 | |
| *** salv-orlando has joined #openstack-infra | 09:40 | |
| *** CaptTofu has joined #openstack-infra | 09:45 | |
| *** che-arne has joined #openstack-infra | 09:46 | |
| *** luqas has quit IRC | 09:46 | |
| *** derekh has joined #openstack-infra | 09:47 | |
| *** thomasbiege has quit IRC | 09:49 | |
| *** Ryan_Lane has joined #openstack-infra | 09:49 | |
| *** jcoufal has quit IRC | 09:49 | |
| *** CaptTofu has quit IRC | 09:50 | |
| *** jcoufal has joined #openstack-infra | 09:50 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency https://review.openstack.org/62726 | 09:52 |
| *** ociuhandu has quit IRC | 09:53 | |
| *** Ryan_Lane has quit IRC | 09:54 | |
| *** jp_at_hp has joined #openstack-infra | 09:55 | |
| *** yamahata has quit IRC | 09:59 | |
| *** roz has left #openstack-infra | 10:01 | |
| *** luqas has joined #openstack-infra | 10:08 | |
| *** fbo is now known as fbo_away | 10:09 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 10:09 | |
| *** Hefeweizen has quit IRC | 10:12 | |
| *** Hefeweizen has joined #openstack-infra | 10:13 | |
| *** talluri has quit IRC | 10:14 | |
| *** talluri has joined #openstack-infra | 10:15 | |
| *** masayukig has quit IRC | 10:18 | |
| *** skolekonov has joined #openstack-infra | 10:18 | |
| *** max_lobur_afk is now known as max_lobur | 10:19 | |
| *** talluri has quit IRC | 10:19 | |
| *** gokrokve has joined #openstack-infra | 10:22 | |
| *** gokrokve has quit IRC | 10:27 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/nodepool: Add fedora support https://review.openstack.org/74529 | 10:27 |
| *** boris-42_ has joined #openstack-infra | 10:31 | |
| *** vkozhukalov has joined #openstack-infra | 10:33 | |
| *** dpyzhov has quit IRC | 10:36 | |
| *** dpyzhov_ has joined #openstack-infra | 10:36 | |
| *** andre__ has joined #openstack-infra | 10:39 | |
| *** DinaBelova_ is now known as DinaBelova | 10:40 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 10:46 | |
| *** masayukig_mob has joined #openstack-infra | 10:47 | |
| *** masayukig_mob has quit IRC | 10:47 | |
| *** johnthetubaguy has joined #openstack-infra | 10:51 | |
| *** yassine has quit IRC | 10:53 | |
| SergeyLukjanov | mrmartin, I see you comment, I'm afraid that we have no more precise slaves... | 10:57 |
| *** andreaf has joined #openstack-infra | 10:57 | |
| *** yamahata has joined #openstack-infra | 10:58 | |
| *** matsuhashi has quit IRC | 10:58 | |
| mrmartin | SergeyLukjanov, ok, but how the other jobs depending on standard_packages are running? | 11:00 |
| SergeyLukjanov | mrmartin, currently I see only your job that uses precise instead of bare-precise | 11:02 |
| *** ociuhandu has joined #openstack-infra | 11:05 | |
| SergeyLukjanov | mrmartin, confirm, there are no non-bare precise slaves and there are no such slaves in nodepool configs, so, you should wait for someone from root team to setup such slaves or to add some non-bare slaves to nodepool | 11:08 |
| mrmartin | SergeyLukjanov: - job-template: | 11:08 |
| mrmartin | name: '{name}-tarball' | 11:08 |
| mrmartin | node: bare-precise | 11:08 |
| mrmartin | ok | 11:09 |
| SergeyLukjanov | mrmartin, what is about template? | 11:09 |
| mrmartin | but in slave.pp you can find a lot of packages for xample, $::jenkins::params::mongodb_package, # for ceilometer unit tests | 11:10 |
| mrmartin | how the ceilometer unit tests are running now? | 11:10 |
| mrmartin | # Packages that all jenkins slaves need | 11:11 |
| mrmartin | $common_packages = [ | 11:11 |
| mrmartin | $::jenkins::params::jdk_package, # jdk for building java jobs | 11:11 |
| mrmartin | $::jenkins::params::ccache_package, | 11:11 |
| mrmartin | $::jenkins::params::python_netaddr_package, # Needed for devstack address_in_net() | 11:11 |
| mrmartin | ] | 11:11 |
| mrmartin | and if you see the comments of standard packages openstack docs, ceilometer, zeromq could releate somehow to this 'precise' node | 11:12 |
| *** yaguang has quit IRC | 11:13 | |
| openstackgerrit | A change was merged to openstack-infra/config: Temporary : Mark a few oslo.vmware jobs as non-voting https://review.openstack.org/74669 | 11:18 |
| *** matsuhashi has joined #openstack-infra | 11:18 | |
| *** gokrokve has joined #openstack-infra | 11:22 | |
| *** lcostantino has joined #openstack-infra | 11:23 | |
| *** CaptTofu has joined #openstack-infra | 11:24 | |
| *** matsuhashi has quit IRC | 11:26 | |
| *** gokrokve has quit IRC | 11:27 | |
| *** matsuhashi has joined #openstack-infra | 11:27 | |
| *** max_lobur is now known as max_lobur_afk | 11:31 | |
| *** matsuhashi has quit IRC | 11:31 | |
| *** bhuvan has joined #openstack-infra | 11:31 | |
| *** jgallard has quit IRC | 11:33 | |
| *** jcoufal has quit IRC | 11:35 | |
| *** jcoufal has joined #openstack-infra | 11:35 | |
| *** mrmartin has quit IRC | 11:40 | |
| *** fbo_away is now known as fbo | 11:42 | |
| *** bhuvan has quit IRC | 11:44 | |
| *** matsuhashi has joined #openstack-infra | 11:45 | |
| *** rfolco has joined #openstack-infra | 11:46 | |
| *** dstanek_afk has quit IRC | 11:50 | |
| *** gsamfira has joined #openstack-infra | 11:51 | |
| *** che-arne has quit IRC | 11:56 | |
| *** dstanek_afk has joined #openstack-infra | 11:58 | |
| derekh | 17:04:35 <SpamapS> what sort of hackery is needed to get our tripleo-rack-booted machines on the internet? | 12:00 |
| derekh | SpamapS: baremetal node? did you get sorted? | 12:00 |
| *** matsuhashi has quit IRC | 12:01 | |
| *** matsuhashi has joined #openstack-infra | 12:01 | |
| *** dpyzhov_ has quit IRC | 12:01 | |
| *** jroovers is now known as jroovers|afk | 12:05 | |
| *** matsuhashi has quit IRC | 12:05 | |
| *** matsuhashi has joined #openstack-infra | 12:07 | |
| *** gokrokve has joined #openstack-infra | 12:07 | |
| *** ArxCruz has joined #openstack-infra | 12:07 | |
| *** alexpilotti has joined #openstack-infra | 12:08 | |
| *** basha has quit IRC | 12:09 | |
| *** yassine has joined #openstack-infra | 12:12 | |
| *** matsuhashi has quit IRC | 12:14 | |
| *** matsuhashi has joined #openstack-infra | 12:15 | |
| *** matsuhashi has quit IRC | 12:18 | |
| *** matsuhashi has joined #openstack-infra | 12:18 | |
| *** gokrokve has quit IRC | 12:19 | |
| *** gokrokve has joined #openstack-infra | 12:19 | |
| *** jcoufal has quit IRC | 12:20 | |
| *** jcoufal has joined #openstack-infra | 12:20 | |
| *** gokrokve has quit IRC | 12:23 | |
| *** dstanek_afk is now known as dstanek | 12:25 | |
| *** ArxCruz has quit IRC | 12:26 | |
| *** derekh has quit IRC | 12:30 | |
| *** jhesketh_ has quit IRC | 12:39 | |
| *** yamahata has quit IRC | 12:39 | |
| *** jhesketh_ has joined #openstack-infra | 12:40 | |
| *** smarcet has joined #openstack-infra | 12:42 | |
| *** hashar has joined #openstack-infra | 12:44 | |
| *** lcostantino has quit IRC | 12:44 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: ArtifactDeployer Plugin support added https://review.openstack.org/75015 | 12:45 |
| openstackgerrit | Alexander Jones proposed a change to openstack-infra/git-review: Fix parsing of SCP-style URLs, as these are valid in Git itself https://review.openstack.org/72751 | 12:45 |
| *** mriedem has joined #openstack-infra | 12:46 | |
| *** yamahata has joined #openstack-infra | 12:47 | |
| *** ArxCruz has joined #openstack-infra | 12:50 | |
| *** gokrokve has joined #openstack-infra | 12:50 | |
| *** gokrokve_ has joined #openstack-infra | 12:52 | |
| *** weshay has joined #openstack-infra | 12:53 | |
| *** dizquierdo is now known as dizquierdo_afk | 12:54 | |
| *** gokrokve has quit IRC | 12:55 | |
| *** unicell has quit IRC | 12:55 | |
| *** unicell has joined #openstack-infra | 12:55 | |
| *** gokrokve_ has quit IRC | 12:56 | |
| *** derekh has joined #openstack-infra | 12:58 | |
| *** dkranz has quit IRC | 12:58 | |
| *** CaptTofu has quit IRC | 13:01 | |
| *** CaptTofu has joined #openstack-infra | 13:02 | |
| *** che-arne has joined #openstack-infra | 13:03 | |
| *** eharney has quit IRC | 13:04 | |
| *** portante has quit IRC | 13:05 | |
| *** CaptTofu has quit IRC | 13:06 | |
| *** jgallard has joined #openstack-infra | 13:07 | |
| *** lcostantino has joined #openstack-infra | 13:09 | |
| *** rossella-s has quit IRC | 13:15 | |
| *** david-lyle has quit IRC | 13:17 | |
| *** gokrokve has joined #openstack-infra | 13:22 | |
| *** portante has joined #openstack-infra | 13:23 | |
| *** esker has quit IRC | 13:26 | |
| *** gokrokve has quit IRC | 13:27 | |
| *** johnthetubaguy has quit IRC | 13:28 | |
| *** yamahata has quit IRC | 13:33 | |
| *** dstanek has quit IRC | 13:34 | |
| ihrachys | with https://review.openstack.org/#/c/75014/, I got check-requirements-integration-dsvm failure with: distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr>=0.5.16,<0.6') I've checked rechecks page but haven't found anything related. Any ideas? | 13:35 |
| *** dpyzhov has joined #openstack-infra | 13:36 | |
| *** dstanek has joined #openstack-infra | 13:43 | |
| *** yamahata has joined #openstack-infra | 13:47 | |
| *** matsuhas_ has joined #openstack-infra | 13:51 | |
| *** b3nt_pin has joined #openstack-infra | 13:52 | |
| *** unicell1 has joined #openstack-infra | 13:53 | |
| *** gordc has joined #openstack-infra | 13:53 | |
| *** ArxCruz has quit IRC | 13:53 | |
| *** gokrokve has joined #openstack-infra | 13:54 | |
| *** Shrews_ has joined #openstack-infra | 13:54 | |
| *** jgriffit2 has joined #openstack-infra | 13:55 | |
| *** tru_tru_ has joined #openstack-infra | 13:55 | |
| *** matel has joined #openstack-infra | 13:55 | |
| *** rcleere has joined #openstack-infra | 13:56 | |
| sdague | ihrachys: it looks like an issue with trove, though I'm not sure why that edge case is hit | 13:57 |
| ihrachys | sdague: should I just recheck no bug, or create a new one? | 13:58 |
| *** jroovers|afk is now known as jroovers | 13:58 | |
| *** ArxCruz has joined #openstack-infra | 13:59 | |
| *** che-arne has quit IRC | 13:59 | |
| *** unicell has quit IRC | 13:59 | |
| *** matsuhashi has quit IRC | 13:59 | |
| *** gsamfira has quit IRC | 13:59 | |
| *** loquacities has quit IRC | 13:59 | |
| *** beagles has quit IRC | 13:59 | |
| *** vrovachev has quit IRC | 13:59 | |
| *** ayoung-ZzZzZzZ has quit IRC | 13:59 | |
| *** exz has quit IRC | 13:59 | |
| *** DinaBelova has quit IRC | 13:59 | |
| *** bodepd has quit IRC | 13:59 | |
| *** tru_tru has quit IRC | 13:59 | |
| *** sileht has quit IRC | 13:59 | |
| *** psedlak has quit IRC | 13:59 | |
| *** julienvey has quit IRC | 13:59 | |
| *** ekarlso has quit IRC | 13:59 | |
| *** Adri2000 has quit IRC | 13:59 | |
| *** akscram has quit IRC | 13:59 | |
| *** jgriffith has quit IRC | 13:59 | |
| *** primeministerp has quit IRC | 13:59 | |
| *** milki has quit IRC | 13:59 | |
| *** Shrews has quit IRC | 13:59 | |
| *** jd__ has quit IRC | 13:59 | |
| *** med_ has quit IRC | 13:59 | |
| *** b3nt_pin is now known as beagles | 14:00 | |
| *** esker has joined #openstack-infra | 14:00 | |
| *** milki has joined #openstack-infra | 14:00 | |
| *** yamahata has quit IRC | 14:01 | |
| *** DinaBelova has joined #openstack-infra | 14:01 | |
| *** jd__ has joined #openstack-infra | 14:01 | |
| *** primeministerp has joined #openstack-infra | 14:01 | |
| *** ArxCruz has quit IRC | 14:01 | |
| *** yamahata has joined #openstack-infra | 14:02 | |
| *** dizquierdo_afk is now known as dizquierdo | 14:02 | |
| *** primeministerp has quit IRC | 14:11 | |
| *** saju_m has quit IRC | 14:11 | |
| sdague | ihrachys: recheck isn't going to help | 14:11 |
| sdague | this isn't a race condition, it's a bad interaction. We need to figure out why | 14:11 |
| *** markmcclain has joined #openstack-infra | 14:11 | |
| *** markmcclain has quit IRC | 14:11 | |
| *** dizquierdo is now known as dizquierdo_afk | 14:15 | |
| *** johnthetubaguy has joined #openstack-infra | 14:15 | |
| *** ArxCruz has joined #openstack-infra | 14:15 | |
| *** yamahata has quit IRC | 14:15 | |
| *** markmcclain has joined #openstack-infra | 14:15 | |
| *** dstanek has quit IRC | 14:15 | |
| *** nosnos_ has joined #openstack-infra | 14:15 | |
| *** markmcclain has quit IRC | 14:15 | |
| *** dprince has joined #openstack-infra | 14:15 | |
| *** ayoung-ZzZzZzZ has joined #openstack-infra | 14:15 | |
| *** yamahata has joined #openstack-infra | 14:15 | |
| *** julienvey has joined #openstack-infra | 14:16 | |
| *** gsamfira has joined #openstack-infra | 14:16 | |
| *** loquacities has joined #openstack-infra | 14:16 | |
| *** vrovachev has joined #openstack-infra | 14:16 | |
| *** exz has joined #openstack-infra | 14:16 | |
| *** bodepd has joined #openstack-infra | 14:16 | |
| *** sileht has joined #openstack-infra | 14:16 | |
| *** psedlak has joined #openstack-infra | 14:16 | |
| *** ekarlso has joined #openstack-infra | 14:16 | |
| *** Adri2000 has joined #openstack-infra | 14:16 | |
| *** akscram has joined #openstack-infra | 14:16 | |
| *** che-arne has joined #openstack-infra | 14:16 | |
| *** markmcclain has joined #openstack-infra | 14:16 | |
| *** saju_m has joined #openstack-infra | 14:16 | |
| *** yamahata has quit IRC | 14:16 | |
| *** julim has joined #openstack-infra | 14:16 | |
| *** primeministerp has joined #openstack-infra | 14:16 | |
| *** NikitaKonovalov has quit IRC | 14:16 | |
| *** nosnos has quit IRC | 14:16 | |
| *** matrohon has quit IRC | 14:16 | |
| *** changbl has quit IRC | 14:16 | |
| *** matrohon has joined #openstack-infra | 14:16 | |
| *** NikitaKonovalov has joined #openstack-infra | 14:16 | |
| *** jcooley_ has quit IRC | 14:17 | |
| *** matsuhas_ has quit IRC | 14:17 | |
| *** yamahata has joined #openstack-infra | 14:17 | |
| *** matsuhashi has joined #openstack-infra | 14:18 | |
| mordred | sdague: I believe it's because stable/havana pins pbr to <0.6, we released 0.6 and now client libs depend on it | 14:18 |
| mordred | or somethign similar | 14:18 |
| mordred | sdague: I think we might need to move to an unpinned pbr model and seriously just we can't change pbr's behavior ever | 14:19 |
| *** nprivalova has quit IRC | 14:20 | |
| *** che-arne has quit IRC | 14:20 | |
| *** gsamfira has quit IRC | 14:20 | |
| *** loquacities has quit IRC | 14:20 | |
| *** vrovachev has quit IRC | 14:20 | |
| *** exz has quit IRC | 14:20 | |
| *** bodepd has quit IRC | 14:20 | |
| *** sileht has quit IRC | 14:20 | |
| *** psedlak has quit IRC | 14:20 | |
| *** ekarlso has quit IRC | 14:20 | |
| *** Adri2000 has quit IRC | 14:20 | |
| *** akscram has quit IRC | 14:20 | |
| sdague | so why did it only die in trove? | 14:21 |
| *** jeckersb_gone is now known as jeckersb | 14:21 | |
| sdague | or is that just the first stable/havana hit? | 14:21 |
| *** jpeeler has quit IRC | 14:23 | |
| mordred | I think just first stable/havana hit | 14:24 |
| mordred | I believe I remember mtreinish pointing out a stable/havana problem a few days ago | 14:24 |
| mordred | or, wait- not, that was teh stable/grizzly thing | 14:24 |
| *** jpeeler has joined #openstack-infra | 14:25 | |
| *** rossella-s has joined #openstack-infra | 14:26 | |
| *** mrmartin has joined #openstack-infra | 14:26 | |
| *** alexpilotti has quit IRC | 14:27 | |
| *** saju_m has quit IRC | 14:27 | |
| *** enikanorov__ has joined #openstack-infra | 14:28 | |
| *** protux has joined #openstack-infra | 14:28 | |
| *** chandan_kumar has quit IRC | 14:29 | |
| *** tru_tru has joined #openstack-infra | 14:29 | |
| *** juice- has joined #openstack-infra | 14:29 | |
| *** jog0_ has joined #openstack-infra | 14:29 | |
| *** sandywalsh_ has joined #openstack-infra | 14:30 | |
| *** dcramer_ has joined #openstack-infra | 14:31 | |
| *** jasond`` has joined #openstack-infra | 14:31 | |
| sdague | yeh, sorry, I've been paying attension to some other things, so lost context on this one. Will take a stab later. | 14:31 |
| *** jgriffith has joined #openstack-infra | 14:32 | |
| ihrachys | sdague: am I right that pbr 0.6 is fine for us, so we should unpin it? | 14:33 |
| *** che-arne has joined #openstack-infra | 14:33 | |
| *** gsamfira has joined #openstack-infra | 14:33 | |
| *** loquacities has joined #openstack-infra | 14:33 | |
| *** vrovachev has joined #openstack-infra | 14:33 | |
| *** exz has joined #openstack-infra | 14:33 | |
| *** bodepd has joined #openstack-infra | 14:33 | |
| *** sileht has joined #openstack-infra | 14:33 | |
| *** psedlak has joined #openstack-infra | 14:33 | |
| *** ekarlso has joined #openstack-infra | 14:33 | |
| *** Adri2000 has joined #openstack-infra | 14:33 | |
| *** akscram has joined #openstack-infra | 14:33 | |
| *** ProfFalken has quit IRC | 14:34 | |
| ihrachys | hm, but wait, in global-requirements, we have pbr>=0.5.21,<1.0 | 14:34 |
| ihrachys | so this <0.6 requirement comes from some other source | 14:34 |
| *** max_lobur has joined #openstack-infra | 14:34 | |
| *** markmcclain has quit IRC | 14:35 | |
| sdague | ihrachys: right, because in havana .... trove isn't synced? | 14:35 |
| *** b3nt_pin has joined #openstack-infra | 14:35 | |
| sdague | that seems odd | 14:35 |
| sdague | there is also a grizzly grenade issue here which I'm confused by, as I thought mtreinish fixed that | 14:36 |
| *** tru_tru_ has quit IRC | 14:38 | |
| *** juice has quit IRC | 14:38 | |
| *** sandywalsh has quit IRC | 14:38 | |
| *** jog0 has quit IRC | 14:38 | |
| *** ianw has quit IRC | 14:38 | |
| *** yamahata has quit IRC | 14:38 | |
| *** dizquierdo_afk has quit IRC | 14:38 | |
| *** uvirtbot has quit IRC | 14:38 | |
| *** juice- is now known as juice | 14:38 | |
| *** jgriffit2 has quit IRC | 14:38 | |
| *** beagles has quit IRC | 14:38 | |
| *** yassine has quit IRC | 14:38 | |
| *** mgagne has quit IRC | 14:38 | |
| *** max_lobur_afk has quit IRC | 14:38 | |
| *** enikanorov_ has quit IRC | 14:38 | |
| *** jasond` has quit IRC | 14:38 | |
| *** yassine has joined #openstack-infra | 14:38 | |
| *** ianw has joined #openstack-infra | 14:38 | |
| *** dstanek has joined #openstack-infra | 14:39 | |
| *** dizquierdo_afk has joined #openstack-infra | 14:39 | |
| *** jpeeler has quit IRC | 14:40 | |
| *** yamahata has joined #openstack-infra | 14:40 | |
| *** jpeeler has joined #openstack-infra | 14:40 | |
| ihrachys | sdague: in trove:stable/havana, we have pbr>=0.5.16,<0.6 | 14:40 |
| *** jpeeler has quit IRC | 14:40 | |
| *** jpeeler has joined #openstack-infra | 14:40 | |
| ihrachys | so we need to update requirement there, in trove | 14:40 |
| sdague | right, but I'm confused why that's not getting overwritten | 14:40 |
| *** dizquierdo_afk is now known as dizquierdo | 14:41 | |
| *** prad has joined #openstack-infra | 14:41 | |
| sdague | that's kind of the point of global requirements :) | 14:42 |
| *** prad has quit IRC | 14:43 | |
| *** uvirtbot has joined #openstack-infra | 14:43 | |
| *** mgagne has joined #openstack-infra | 14:43 | |
| *** jhesketh_ has quit IRC | 14:44 | |
| *** jergerber has joined #openstack-infra | 14:44 | |
| *** mfer has joined #openstack-infra | 14:45 | |
| *** protux has quit IRC | 14:45 | |
| *** protux has joined #openstack-infra | 14:45 | |
| mordred | client libs | 14:46 |
| mordred | there are no stable/havana client libs | 14:46 |
| mordred | so transitive deps | 14:46 |
| *** jp_at_hp has quit IRC | 14:47 | |
| sdague | but the client libs should come from git as well, and get overwritten? | 14:47 |
| *** jp_at_hp has joined #openstack-infra | 14:48 | |
| ihrachys | sdague: the point of global requirements is that they are synced to each project (and https://wiki.openstack.org/wiki/Requirements tells that 'Once the change to openstack/requirements has merged, the local pypi mirror will be updated and then the requirements file for the OpenStack project may be updated.') | 14:48 |
| ihrachys | sdague: so if we don't sync requirements to trove, trove uses old requirements, that's how I expect it to work | 14:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix gate-groups-unittests to use bare-precise https://review.openstack.org/74959 | 14:49 |
| ihrachys | as per README in requirements master (not havana!): The continuous integration infrastructure will also sync up the requirements across all the official projects and will create reviews in the participating projects for any mis-matches. | 14:50 |
| fungi | SergeyLukjanov: mrmartin: if you're still around, see the comments on my approval for https://review.openstack.org/74959 | 14:50 |
| sdague | ihrachys: actually, in the gate, we ignore the project requirements | 14:50 |
| sdague | and use global requirements | 14:50 |
| *** jhesketh__ has quit IRC | 14:51 | |
| openstackgerrit | Nejc Saje proposed a change to openstack/requirements: Add croniter dependency https://review.openstack.org/75055 | 14:51 |
| ihrachys | ah, ok, but still, I haven't seen those 'continuous integration infrastructure' reviews before, so may be it's not even true | 14:51 |
| mrmartin | fungi: I'm here. I need also update the similar openstackid patch waiting for review | 14:51 |
| mordred | sdague: oh. hrm. you're right | 14:52 |
| *** boris-42_ has quit IRC | 14:52 | |
| mrmartin | fungi: what will happen with the stucked openstack-infra/groups job (71068): http://status.openstack.org/zuul/ ? | 14:53 |
| fungi | mrmartin: i think it will get dropped from the check pipeline and we can recheck it properly once zuul's config updates. that's a new behavior which only just got added in the past week or so i think | 14:54 |
| fungi | so it's possible that doesn't work yet | 14:54 |
| fungi | we'll see | 14:54 |
| mrmartin | ok, because this job spend more than a day there :) | 14:54 |
| fungi | not sure how i missed fixing it in my patch which changed all the other nodes from precise to bare-precise a few days ago, unless it just got approved since then | 14:55 |
| *** ryanpetrello has joined #openstack-infra | 14:55 | |
| fungi | if the former, i needed to be more thorough in my patch. if the latter, then we need to be more thorough in reviewing changes like that | 14:55 |
| mrmartin | fungi: it was just random chains of events. | 14:57 |
| fungi | mrmartin: i think i see why i missed it. i didn't think to look for quotes around node labels (because they shouldn't be needed). there may be more i missed in that case | 14:59 |
| fungi | checking now | 14:59 |
| fungi | okay, good... that job was the only one | 15:00 |
| *** devoid has joined #openstack-infra | 15:01 | |
| *** julim_ has joined #openstack-infra | 15:01 | |
| *** zul has quit IRC | 15:01 | |
| *** zul has joined #openstack-infra | 15:01 | |
| devoid | Hi folks, is there a specific reason why http://eavesdrop.openstack.org/irclogs/ does not cointain #openstack-nova | 15:02 |
| fungi | devoid: the channel founders or other responsible parties need to ask to have it logged | 15:02 |
| devoid | fungi, should I create a bug and have them comment on it? | 15:02 |
| fungi | devoid: we don't log all channels because the people who hang out in some of them may not want official logs | 15:03 |
| fungi | devoid: that would be a way to go about it. sounds goof | 15:03 |
| fungi | good | 15:03 |
| ihrachys | sdague: I see the following after requirements are synced: git clone /opt/stack/new/trove /tmp/tmp.a0d03VY4bO/projects/trove | 15:03 |
| sdague | yeh, me to, which is why I'm confused | 15:03 |
| ihrachys | and then we run setup.py from this tempdir, and I guess new reqs are not cloned, hence the failure | 15:04 |
| *** julim has quit IRC | 15:04 | |
| *** matsuhashi has quit IRC | 15:05 | |
| SergeyLukjanov | fungi, I'm here now, thanks for the comment, just was quite sleepy when looked at prepare_... | 15:05 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Basic gating and release jobs of openstackid project https://review.openstack.org/73549 | 15:05 |
| SergeyLukjanov | fungi, thanks for the explanation | 15:05 |
| *** dhellmann has quit IRC | 15:05 | |
| fungi | SergeyLukjanov: it's not your fault at all. i think as part of the puppet refactoring we should rename that variable to something like "minimal" | 15:05 |
| fungi | or rename the node types, but that would be a lot more involved | 15:06 |
| SergeyLukjanov | fungi, 'minimal' sounds better ;) | 15:06 |
| *** dhellmann has joined #openstack-infra | 15:06 | |
| *** nosnos_ has quit IRC | 15:06 | |
| SergeyLukjanov | fungi, yup, not fot end of cycle :) | 15:06 |
| *** skolekonov has quit IRC | 15:06 | |
| *** prad_ has joined #openstack-infra | 15:07 | |
| *** dhellmann_ has joined #openstack-infra | 15:07 | |
| *** dhellmann has quit IRC | 15:07 | |
| miqui | jeblair: have a min? | 15:08 |
| *** dhellmann_ is now known as dhellmann | 15:08 | |
| miqui | jeblair: i created an identical misc.yaml under jenkins job builder.... | 15:08 |
| miqui | jeblair: funny part it creates a gate-noop' job in jenkins.... which was unexpected.... | 15:09 |
| fungi | miqui: is it a copy of our misc.yaml then? | 15:10 |
| miqui | fungi: i removed the node entry since we do not use it | 15:10 |
| miqui | job: | 15:10 |
| miqui | name: gate-noop | 15:10 |
| miqui | thats it | 15:10 |
| miqui | indented correctly ofcourse... | 15:10 |
| jeblair | miqui: that tells jjb to create a job called gate-noop :) | 15:11 |
| miqui | jeblair: hahah yup | 15:11 |
| miqui | jeblair: k, so what is the right way of implementing gate-noop in review ? | 15:12 |
| mrmartin | fungi: I updated this patch also with bare-precise node setting, may I ask a review for that: https://review.openstack.org/#/c/73549/ ? | 15:12 |
| fungi | mrmartin: sure | 15:12 |
| ihrachys | sdague: where can I find the script called by the task? | 15:13 |
| *** jgrimm has joined #openstack-infra | 15:13 | |
| sdague | update.py in requirements repo | 15:13 |
| jeblair | mordred: i left some questions on https://review.openstack.org/#/c/74943/ | 15:14 |
| mordred | jeblair: thank you | 15:14 |
| mtreinish | sdague: the grizzly fix is still pending approval: https://review.openstack.org/#/c/74451/ | 15:14 |
| sdague | jeblair / mordred ^^^^ | 15:14 |
| ihrachys | sdague: no, I mean the whole check-requirements-integration-dsvm thing. I guess update.py is just to sync requirements | 15:14 |
| *** dmsimard has joined #openstack-infra | 15:15 | |
| sdague | ihrachys: ah, mordred would know | 15:15 |
| mordred | jeblair: I'll answer on the thing- but I believe it's inactivity timeout- and I believe it was being triggered by the image downloads | 15:15 |
| dmsimard | jeblair: Good morning ! Still have that git issue this morning. | 15:15 |
| sdague | I don't remember where the root job is | 15:15 |
| *** smarcet has left #openstack-infra | 15:15 | |
| mordred | specifically, the debian vmdk | 15:15 |
| jeblair | mordred: so then how does the reset help? wouldn't that just shift the window but still you have a, say, 10 min section of inacitivity? | 15:16 |
| jeblair | dmsimard: we learned yesterday that hpcloud az2 can't build images... let me check to see if that's been resolved | 15:16 |
| mordred | jeblair: oh - wait - sorry, I'm still caffinating | 15:17 |
| mordred | jeblair: the chattyness is on the ssh connection | 15:17 |
| *** alexpilotti has joined #openstack-infra | 15:17 | |
| jeblair | dmsimard, mordred, clarkb: i don't see anything new about hpcloud az2 | 15:17 |
| mordred | jeblair: the inactivity is on the http connection to the nova api | 15:17 |
| jeblair | all the other regions built images this morning | 15:18 |
| mordred | jeblair: let me go bug someone in our noc | 15:18 |
| jeblair | mordred: thanks | 15:18 |
| *** amcrn has joined #openstack-infra | 15:18 | |
| jeblair | dmsimard: if we don't hear something from mordred soon, we'll yank hpcloud-az2 from the config (which will hurt because it's a substantial amount of our nodes) | 15:18 |
| jeblair | dmsimard: so i can double check, can you link me to the latest failure? | 15:19 |
| *** oubiwann has joined #openstack-infra | 15:19 | |
| *** eharney has joined #openstack-infra | 15:19 | |
| *** arborism has joined #openstack-infra | 15:20 | |
| jeblair | mordred: oh, i see. yes i was mistaken then about http/ssh; how does the http failure manifest? do you have a traceback? because maybe that kind of exception should just get caught and then the client reset. | 15:20 |
| mordred | jeblair: I do - one sec | 15:20 |
| *** dims has quit IRC | 15:21 | |
| *** jroovers|afk has joined #openstack-infra | 15:21 | |
| *** oubiwann has quit IRC | 15:21 | |
| *** mgagne has quit IRC | 15:21 | |
| *** jroovers has quit IRC | 15:22 | |
| *** dkliban has joined #openstack-infra | 15:23 | |
| mordred | jeblair: https://etherpad.openstack.org/p/hSoC7QzrHI | 15:23 |
| mordred | jeblair: at the bottom | 15:23 |
| mordred | jeblair: so yeah, there is a ProxyError thrown, which could be caught in general | 15:23 |
| mordred | should that happen maybe in the task runner? | 15:24 |
| mordred | try: run_task except ProxyError: reset_client(); run_task ? | 15:24 |
| *** amcrn has quit IRC | 15:24 | |
| jeblair | mordred: well, the taskmanager isn't supposed to know anything about novaclient, only its subclass the providermanager is... | 15:25 |
| mordred | mmm. indeed | 15:26 |
| jeblair | mordred: (the taskmanager also runs jenkins tasks) | 15:26 |
| *** max_lobur is now known as max_lobur_afk | 15:26 | |
| jeblair | mordred: but assuming you can work out the oo separation there, yeah, that sounds like the way to go. :) | 15:26 |
| mordred | oy | 15:26 |
| jeblair | mordred: actually... | 15:27 |
| fungi | jeblair: one short-term alternative to removing az2 entirely is to delete the images. i've done that before to force a refresh... nodepool will just keep retrying to build each until one finally sticks. doesn't solve the long-term issue but at least either gets us reasonably recent images and keeps the old ones from spawning new nodes until that happens | 15:28 |
| jeblair | fungi: that's a good idea | 15:28 |
| jeblair | mordred: maybe subclass Task so that either run or main does the try/except | 15:29 |
| mordred | jeblair: I was just thinking the same thing | 15:29 |
| *** dhellman_ has joined #openstack-infra | 15:29 | |
| *** pdmars has joined #openstack-infra | 15:30 | |
| mordred | omg | 15:31 |
| mordred | it's an uncaught requests ProxyError | 15:31 |
| *** wenlock has joined #openstack-infra | 15:31 | |
| mordred | maybe? | 15:31 |
| mordred | who has a ProxyError? | 15:31 |
| mordred | gah | 15:31 |
| *** dhellman_ is now known as dhellmann_ | 15:32 | |
| *** dhellmann_ is now known as dhellman_ | 15:32 | |
| *** dims has joined #openstack-infra | 15:32 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Check server status in batch https://review.openstack.org/74773 | 15:32 |
| fungi | mordred: requests bubbles all sorts of errors up through novaclient i've noticed. it's fun stuff | 15:33 |
| jeblair | clarkb: ^ that should address the potential leak | 15:33 |
| *** dolphm is now known as dolphm_503 | 15:33 | |
| *** dhellmann has quit IRC | 15:33 | |
| *** dhellman_ is now known as dhellmann | 15:33 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use py3k-precise nodes https://review.openstack.org/73851 | 15:33 |
| * fungi is babysitting that ^ | 15:33 | |
| mordred | jeblair: looking | 15:34 |
| mordred | jeblair: also, I can't do it in Task, because I've lost context to reset client there | 15:34 |
| mordred | I'll come up with something | 15:34 |
| *** jgrimm has quit IRC | 15:35 | |
| jeblair | mordred: ok. maybe best thing is something like add a simple method to task_manager that just calls task.run; then override that in provider_manager | 15:35 |
| jeblair | mordred: that way you can override just the part of task_manager.run that is interesting. | 15:36 |
| *** dpyzhov has quit IRC | 15:36 | |
| mordred | yah | 15:36 |
| *** dpyzhov has joined #openstack-infra | 15:37 | |
| *** protux has quit IRC | 15:39 | |
| *** markmcclain has joined #openstack-infra | 15:40 | |
| *** sandywalsh_ has quit IRC | 15:40 | |
| *** banix has joined #openstack-infra | 15:41 | |
| openstackgerrit | A change was merged to openstack/requirements: Update pecan >= 0.4.5 in global requirements. https://review.openstack.org/74830 | 15:41 |
| *** vkozhukalov has quit IRC | 15:41 | |
| *** markmcclain1 has joined #openstack-infra | 15:42 | |
| *** zul has quit IRC | 15:42 | |
| *** zul has joined #openstack-infra | 15:42 | |
| *** david-lyle has joined #openstack-infra | 15:44 | |
| *** markmcclain has quit IRC | 15:44 | |
| *** devoid has left #openstack-infra | 15:46 | |
| *** smarcet has joined #openstack-infra | 15:46 | |
| *** jcoufal has quit IRC | 15:50 | |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-api-devstack job https://review.openstack.org/75078 | 15:52 |
| *** jcoufal has joined #openstack-infra | 15:52 | |
| ihrachys | sdague: ok, some findings here... when checking requirements, the following file from infra/config repo is used: modules/openstack_project/files/jenkins_job_builder/config/requirements.yaml There, we call bash -xe $BASE/new/pbr/tools/integration.sh $PROJECTS. In this script located in pbr repo, for each project, we do: git clone $REPODIR/$SHORT_PROJECT $shortprojectdir, then cd $shortprojectdir, then | 15:52 |
| ihrachys | $sdistvenv/bin/python setup.py sdist. This sdist call actually reinitiates venv package installations again, using non-global requirements. | 15:52 |
| *** boris-42_ has joined #openstack-infra | 15:52 | |
| *** sandywalsh_ has joined #openstack-infra | 15:54 | |
| *** dkranz has joined #openstack-infra | 15:54 | |
| *** DinaBelova is now known as DinaBelova_ | 15:55 | |
| *** mgagne has joined #openstack-infra | 15:55 | |
| *** flaper87 is now known as flaper87|afk | 15:55 | |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-api-devstack job https://review.openstack.org/75078 | 15:55 |
| *** DinaBelova_ is now known as DinaBelova | 15:56 | |
| fungi | Alex_Gaynor: does this pypy failure look familiar? https://jenkins04.openstack.org/job/gate-python-ceilometerclient-pypy/1/console | 15:56 |
| fungi | Alex_Gaynor: we're switching to a new method of building the workers where the pypy jobs run, and i just want to make sure there's nothing wrong at the system level | 15:56 |
| dmsimard | jeblair: Sorry, I had to step away for a sec, let me link you at the latest failure - hang on | 15:57 |
| dmsimard | jeblair: https://review.openstack.org/#/c/74082/ | 15:57 |
| ihrachys | sdague: I'm not sure what this means, but it looks like we at least attempt to install non-global requirements for each official project when updating requirements, and this explains why we didn't experience such failure before - because requirements were not touched | 15:58 |
| jeblair | dmsimard: all az2 as expected. let's give mordred's poking a bit longer, then we'll pull those images. | 15:59 |
| *** mgagne has quit IRC | 15:59 | |
| *** andreaf has quit IRC | 15:59 | |
| *** vkozhukalov has joined #openstack-infra | 15:59 | |
| *** mgagne has joined #openstack-infra | 15:59 | |
| fungi | jeblair: if/when we're ready, i'm all queued up with the image-delete commands | 15:59 |
| *** dpyzhov has quit IRC | 15:59 | |
| dmsimard | jeblair: Sure thing. Go mordred, I believe in you :) | 15:59 |
| fungi | also very close to being comfortable moving forward with deletion of all the static slaves no longer in use in rax-dfw so we can up the quota there | 16:00 |
| *** virmitio has joined #openstack-infra | 16:00 | |
| fungi | hopefully in another hour, give or take | 16:00 |
| *** oubiwann has joined #openstack-infra | 16:00 | |
| *** changbl has joined #openstack-infra | 16:00 | |
| *** Shrews_ is now known as Shrews | 16:01 | |
| fungi | so that should offset the loss of az2 some (still by less than a third, but every bit counts) | 16:01 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Basic gating and release jobs of openstackid project https://review.openstack.org/73549 | 16:01 |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-api-devstack job https://review.openstack.org/75078 | 16:02 |
| Alex_Gaynor | fungi: no, that error doesn't look super familiar, it looks like the kind of thing that happens if your setuptools is mis-installed or something? cc dstufft | 16:02 |
| *** pcrews has joined #openstack-infra | 16:03 | |
| *** dolphm_503 is now known as dolphm | 16:03 | |
| fungi | Alex_Gaynor: yeah, the "Found existing installation: setuptools 2.1 [...] Uninstalling setuptools" strikes me as a potential culprit | 16:03 |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-api-devstack job https://review.openstack.org/75078 | 16:04 |
| *** dpyzhov has joined #openstack-infra | 16:04 | |
| fungi | dstufft: for context (if you're around) https://jenkins04.openstack.org/job/gate-python-ceilometerclient-pypy/1/console | 16:04 |
| fungi | i'm waiting to catch another pypi job on one of the new nodes and see if it's a persistent failure | 16:05 |
| *** esker has quit IRC | 16:07 | |
| *** zhiyan is now known as zhiyan_ | 16:07 | |
| *** lcheng has joined #openstack-infra | 16:07 | |
| *** esker has joined #openstack-infra | 16:07 | |
| *** ravikumar_hp has quit IRC | 16:09 | |
| *** ravikumar_hp has joined #openstack-infra | 16:10 | |
| *** oubiwann has quit IRC | 16:11 | |
| annegentle | just one more core for fixing the image API doc builds: https://review.openstack.org/#/c/73690 | 16:11 |
| annegentle | Looks fine and rebased, clarkb and SergeyLukjanov already approved before that | 16:11 |
| *** esker has quit IRC | 16:11 | |
| fungi | annegentle: yep, approved. i'll delete the old maven-based jobs so the new freestyle ones get recreated correctly | 16:12 |
| *** dolphm is now known as dolphm_503 | 16:13 | |
| annegentle | fungi: thanks! | 16:13 |
| annegentle | fungi: I also nervously await a response on https://review.openstack.org/#/c/74777/ ... should I abandon and try try again, leaving master alone for a while? | 16:14 |
| *** thomasbiege has joined #openstack-infra | 16:14 | |
| ArxCruz | fungi: hey, I'm having a problem with my nodepool, hope you can help me | 16:16 |
| ArxCruz | when the job finishes, the gearman plugin disconnect the vm from jenkins, but it stays in ready status in nodepool | 16:16 |
| ArxCruz | i'm using the single_use_node | 16:16 |
| SergeyLukjanov | fungi, is "remote HEAD refers to nonexistent ref, unable to checkout" know issue? | 16:16 |
| dmsimard | SergeyLukjanov: https://bugs.launchpad.net/openstack-ci/+bug/1282136 | 16:17 |
| uvirtbot | Launchpad bug 1282136 in openstack-ci "Git problem: "Failed to resolve 'HEAD' as a valid ref."" [Undecided,Fix released] | 16:17 |
| fungi | SergeyLukjanov: if the node is in hpcloud-az2, then yes | 16:17 |
| fungi | annegentle: looking | 16:17 |
| SergeyLukjanov | fungi, ok, thanks | 16:17 |
| SergeyLukjanov | dmsimard, thx | 16:17 |
| *** thomasbiege has quit IRC | 16:17 | |
| ArxCruz | fungi: nevermind, iptables rules | 16:18 |
| SergeyLukjanov | fungi, dmsimard, anyway it still fails sometimes, example - https://review.openstack.org/#/c/74719/ | 16:19 |
| dmsimard | SergeyLukjanov: Yeah, they're looking into it - still occuring this morning. Something to do with a set of nodes (hpaz2) | 16:19 |
| *** oubiwann has joined #openstack-infra | 16:20 | |
| *** markvoelker has joined #openstack-infra | 16:20 | |
| SergeyLukjanov | dmsimard, ack | 16:20 |
| *** chuck_ has joined #openstack-infra | 16:21 | |
| *** chuck_ has quit IRC | 16:21 | |
| *** chuck_ has joined #openstack-infra | 16:21 | |
| mordred | fungi, jeblair L2 support is looking in to the issue, I have no new information right now | 16:21 |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-api-devstack job https://review.openstack.org/75078 | 16:22 |
| *** mrmartin has quit IRC | 16:23 | |
| fungi | mordred: jeblair: in that case shall i delete the images from az2 for now (so we at least start retrying to build new ones there more aggressively)? | 16:23 |
| SergeyLukjanov | mordred, thanks for info | 16:24 |
| *** zul has quit IRC | 16:24 | |
| *** spredzy has joined #openstack-infra | 16:24 | |
| *** oubiwann has quit IRC | 16:25 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:25 | |
| fungi | annegentle: did you see my second comment on 74777? it sounds to me like you wanted to merge master into feature/edits, when what you actually proposed for review was the opposite (merging feature/edits into master) | 16:25 |
| *** UtahDave has joined #openstack-infra | 16:25 | |
| *** VijayT has joined #openstack-infra | 16:26 | |
| *** dolphm_503 is now known as dolphm | 16:26 | |
| *** ayoung-ZzZzZzZ has quit IRC | 16:26 | |
| jeblair | fungi: ++ | 16:26 |
| jeblair | (az2) | 16:26 |
| annegentle | fungi: ohhhh. sigh. augh. sigh. | 16:26 |
| annegentle | fungi: can I talk through it for a minute? | 16:27 |
| fungi | annegentle: sure | 16:27 |
| annegentle | I checkout a new branch based on feature/edits | 16:27 |
| annegentle | I merge it with master | 16:27 |
| annegentle | with git merge origin master | 16:28 |
| fungi | annegentle: i think there are steps missing between those | 16:28 |
| annegentle | er git merge origin/master | 16:28 |
| *** gokrokve has quit IRC | 16:28 | |
| *** jgrimm has joined #openstack-infra | 16:28 | |
| *** gokrokve has joined #openstack-infra | 16:28 | |
| fungi | you check out feature/edits, you make changes to that branch, propose them for review, approve them, watch them land, and so on... then *eventually* you merge feature/edits to master | 16:28 |
| dstufft | fungi: yea that normally means setuptools got trashed somehow | 16:29 |
| fungi | annegentle: in the meantime, if you need feature/edits to have more recent changes which also landed on master, you can propose merges of master into feature/edits | 16:29 |
| *** VijayT has quit IRC | 16:30 | |
| *** tjones has joined #openstack-infra | 16:30 | |
| *** DinaBelova is now known as DinaBelova_ | 16:31 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove maven usage from api-jobs.yaml https://review.openstack.org/73690 | 16:31 |
| annegentle | fungi: yes that's what I mean to do, merges of master into feature/edits | 16:31 |
| *** mfer has quit IRC | 16:31 | |
| annegentle | fungi: like backporting cherry picks | 16:32 |
| fungi | annegentle: so yes, checkout feature/edits, git remote update, git merge origin/master, git review | 16:32 |
| *** mfer has joined #openstack-infra | 16:32 | |
| fungi | oh, wait | 16:32 |
| annegentle | fungi: but I'll get all at once from master? | 16:32 |
| annegentle | fungi: which might be fine, but I had been thinking more like cherry picking | 16:32 |
| fungi | your .gitreview file probably lacks a defaultbranch: feature/edits | 16:32 |
| annegentle | fungi: ohhh right | 16:32 |
| fungi | so git review feature/edits | 16:32 |
| *** gokrokve has quit IRC | 16:33 | |
| annegentle | fungi: that's it, I have had the luxury of someone else editing and checking in a .gitreview file when doing stable backports | 16:34 |
| fungi | you don't actually want to update defaultbranch in .gitreview on feature/edits because you'd keep having to redo it after every sync from master and remember to undo it again before merging it back into master | 16:34 |
| annegentle | fungi: heh right | 16:34 |
| *** cody-somerville has quit IRC | 16:34 | |
| annegentle | fungi: I just have to always do git review feature/edits right? | 16:34 |
| fungi | so instead, just make sure you provide the target branch name to git review when you run it, and double-check the branch field on the review page | 16:34 |
| fungi | annegentle: yes | 16:35 |
| fungi | dstufft: thanks! | 16:35 |
| ihrachys | sdague: fyi I've sent trove requirement update as: https://review.openstack.org/#/c/75092/ | 16:35 |
| *** jcoufal has quit IRC | 16:35 | |
| *** dolphm is now known as dolphm_503 | 16:35 | |
| *** jcoufal has joined #openstack-infra | 16:36 | |
| *** mdenny has joined #openstack-infra | 16:38 | |
| *** nicedice has joined #openstack-infra | 16:38 | |
| *** oubiwann has joined #openstack-infra | 16:39 | |
| markvoelker | jeblair: should the fix for https://bugs.launchpad.net/openstack-ci/+bug/1282136 have propagated by now? Still seeing it. | 16:40 |
| uvirtbot | Launchpad bug 1282136 in openstack-ci "Git problem: "Failed to resolve 'HEAD' as a valid ref."" [Undecided,Fix released] | 16:40 |
| markvoelker | jeblair: Example: http://logs.openstack.org/17/71117/3/check/gate-puppet-neutron-puppet-lint/20c6ba8/console.html | 16:41 |
| *** markmcclain1 has quit IRC | 16:41 | |
| fungi | markvoelker: not yet. we had stale images on hpcloud-az2 so any jobs run there could encounter it. i've just deleted the images in that az which will cause us to no longer build new nodes there. should subside shortly | 16:41 |
| markvoelker | fungi: Excellent, thanks! | 16:42 |
| fungi | markvoelker: the "Building remotely on bare-precise-hpcloud-az2-1469415" at the top of that log says that it was run on one of the nodes from that stale image | 16:42 |
| markvoelker | fungi: Ok, I'll check in again later today. Thanks! | 16:43 |
| *** dhellmann has quit IRC | 16:44 | |
| fungi | markvoelker: i think any jobs run starting now should no longer hit that error | 16:44 |
| * markvoelker goes to test one | 16:44 | |
| *** marun has joined #openstack-infra | 16:44 | |
| dmsimard | fungi: I'll give it a try as well | 16:44 |
| *** atiwari has joined #openstack-infra | 16:44 | |
| fungi | confirmed, we no longer have any ready nodepool nodes in hpcloud-az2, so shouldn't be a problem hopefully | 16:45 |
| *** Ryan_Lane has joined #openstack-infra | 16:45 | |
| *** VijayT has joined #openstack-infra | 16:45 | |
| dmsimard | Hope this doesn't generate a zuul congestion for you guys :( | 16:46 |
| mriedem | portante: ping | 16:46 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 16:46 |
| *** gokrokve has joined #openstack-infra | 16:48 | |
| *** sabari_ has joined #openstack-infra | 16:48 | |
| fungi | mordred: setuptools fun... apparently we're installing setuptools from our puppet currently on python 3.3 slaves in ways which work for pip under python 3 but not under pypy... https://jenkins03.openstack.org/job/gate-python-heatclient-pypy/1/console | 16:52 |
| mordred | fungi: lovely | 16:52 |
| fungi | i'm thinking we may need to either find a way to install setuptools consistently for both on the same system, or have separate pypy slaves... ideas? | 16:53 |
| mordred | fungi: well, we have the updated pip install puppet patches | 16:53 |
| mordred | it could be expanded to install for pypy too | 16:53 |
| fungi | this apparently worked on the static py3k slaves because they had cruft python 2.7 setuptools on them too, no longer reflected in our puppeting/bootstrap i think | 16:54 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Perform all deletes in threads https://review.openstack.org/75099 | 16:54 |
| *** VijayT has quit IRC | 16:54 | |
| *** sabari_ has quit IRC | 16:54 | |
| jeblair | fungi: ^ that's so we don't have to spend as much time telling nodepool to do things it should already know to be doing | 16:54 |
| *** sabari_ has joined #openstack-infra | 16:54 | |
| *** VijayT has joined #openstack-infra | 16:55 | |
| anteaya | this neutron patch is in the gate and all tests are skipped, it has no dependencies: https://review.openstack.org/#/c/75054/ | 16:55 |
| anteaya | what am I not seeing? | 16:55 |
| *** jp_at_hp has quit IRC | 16:55 | |
| fungi | anteaya: the black dot? | 16:55 |
| fungi | anteaya: hover your cursor over it for the tooltip | 16:56 |
| anteaya | yes | 16:56 |
| anteaya | ah | 16:56 |
| * anteaya hovers | 16:56 | |
| fungi | anteaya: says it merge-conflicts with another patch ahead of it | 16:56 |
| anteaya | hmmm, okay | 16:56 |
| *** dolphm_503 is now known as dolphm | 16:57 | |
| *** talluri has joined #openstack-infra | 16:57 | |
| fungi | anteaya: so once it gets to the top of the gate, unless the change it conflicts with fails, zuul will report back to the change with a comment saying it needs a rebase | 16:57 |
| anteaya | fungi: so snipe or leave it in? | 16:57 |
| anteaya | so leave it be? | 16:57 |
| fungi | anteaya: it's not doing any harm | 16:57 |
| anteaya | okay thanks | 16:57 |
| *** tjones has quit IRC | 16:58 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Perform all deletes in threads https://review.openstack.org/75099 | 16:58 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Check server status in batch https://review.openstack.org/74773 | 16:58 |
| *** tjones has joined #openstack-infra | 16:58 | |
| *** chuck_ is now known as zul | 16:59 | |
| *** zul has quit IRC | 16:59 | |
| *** zul has joined #openstack-infra | 16:59 | |
| *** oubiwann has quit IRC | 17:00 | |
| markvoelker | fungi: full of win. Stackforge patches are starting to clear the gate. Thanks! | 17:00 |
| *** tjones has quit IRC | 17:00 | |
| *** dolphm is now known as dolphm_503 | 17:00 | |
| *** tjones has joined #openstack-infra | 17:01 | |
| dmsimard | fungi: I got a success as well. Yay. | 17:01 |
| *** dolphm_503 is now known as dolphm | 17:02 | |
| *** gokrokve has quit IRC | 17:04 | |
| *** markvoelker has left #openstack-infra | 17:04 | |
| *** gokrokve has joined #openstack-infra | 17:04 | |
| *** jroovers|afk is now known as jroovers | 17:04 | |
| jpich | Hi! Any chance a Gerrit admin could restore this patch? https://review.openstack.org/#/c/71061/ The author is away on holiday and a colleague of his would like to pick it up | 17:06 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Delete all building nodes on daemon start https://review.openstack.org/75103 | 17:06 |
| fungi | jpich: done! | 17:07 |
| jpich | fungi: Awesome, thank you! | 17:07 |
| jeblair | fungi: ^ and that patch tackles the other thing we've been doing manually in nodepool restarts | 17:07 |
| *** jnoller has joined #openstack-infra | 17:08 | |
| *** jnoller has quit IRC | 17:08 | |
| *** gokrokve has quit IRC | 17:09 | |
| *** jnoller has joined #openstack-infra | 17:09 | |
| *** markmc has quit IRC | 17:09 | |
| *** UtahDave has quit IRC | 17:09 | |
| *** nati_ueno has joined #openstack-infra | 17:10 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Temporarily switch PyPy jobs back to precisepy3k https://review.openstack.org/75104 | 17:10 |
| jeblair | fungi: who's working on that? | 17:11 |
| fungi | that ^ should fix the currently failing pypy jobs while i put together a working solution | 17:11 |
| fungi | jeblair: i think i am | 17:11 |
| fungi | just wanted to stop the bleeding while i do so | 17:11 |
| fungi | hope to have something working in the next hour | 17:12 |
| *** DinaBelova_ is now known as DinaBelova | 17:12 | |
| *** dmsimard has left #openstack-infra | 17:13 | |
| *** cadenzajon has joined #openstack-infra | 17:14 | |
| mordred | fungi: are you looking at the pip patch? or do you want me to? | 17:15 |
| fungi | mordred: if you could that would be great, though if it's a potentially disruptive change then obviously we'll need to be careful implementing it | 17:16 |
| fungi | and will presumably need rebuilds of all nodepool images? | 17:16 |
| mordred | well, it changes how we install pip globally | 17:16 |
| mordred | so yeah, it's pretty suck | 17:17 |
| mordred | otoh - we could just put in the pip/setuptools cantrips into the nodepool build scripts | 17:17 |
| mordred | that might be safer for now | 17:17 |
| *** e0ne has quit IRC | 17:17 | |
| mordred | jeblair: any opinion/preference ^^ ? | 17:17 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 17:18 | |
| fungi | mordred: if you want to take a poke at that, i'll press forward with removing all the centos6/precise slaves (but not the handful of precisepy3k slaves) from rax-dfw so we can up the available quota there | 17:18 |
| *** sabari_ has quit IRC | 17:18 | |
| *** sabari_ has joined #openstack-infra | 17:18 | |
| *** dkliban is now known as dkliban_afk | 17:19 | |
| jeblair | mordred: i'm okay with that, but can you locally test the new versions of the scripts? | 17:20 |
| gordc | hi folks, hopefully gate issues from yesterday are clear. if not ignore this request... if anyone's free to take a quick look at https://review.openstack.org/#/c/73603/ that'd be great... just so i can stop it from blocking a merge. cheers. | 17:21 |
| *** jroovers is now known as jroovers|afk | 17:22 | |
| mordred | jeblair: yes I can | 17:24 |
| *** gokrokve has joined #openstack-infra | 17:25 | |
| *** jog0_ is now known as jog0 | 17:28 | |
| *** dpyzhov has quit IRC | 17:28 | |
| *** moted has quit IRC | 17:28 | |
| *** mrodden has quit IRC | 17:30 | |
| *** esker has joined #openstack-infra | 17:30 | |
| *** matrohon has quit IRC | 17:31 | |
| *** talluri has quit IRC | 17:32 | |
| *** luqas has quit IRC | 17:32 | |
| *** vkozhukalov has quit IRC | 17:32 | |
| *** talluri has joined #openstack-infra | 17:32 | |
| *** saju_m has joined #openstack-infra | 17:33 | |
| *** cadenzajon has quit IRC | 17:34 | |
| *** cadenzajon_ has joined #openstack-infra | 17:34 | |
| *** markwash has joined #openstack-infra | 17:34 | |
| *** sabari_ has quit IRC | 17:34 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Shift more nodepool nodes onto rax-dfw https://review.openstack.org/73853 | 17:36 |
| fungi | that ^ should allow us to move ahead with the additional quota | 17:36 |
| *** talluri has quit IRC | 17:36 | |
| openstackgerrit | A change was merged to openstack-infra/config: Temporarily switch PyPy jobs back to precisepy3k https://review.openstack.org/75104 | 17:37 |
| * fungi is almost done deleting the old nodes except the precisepy3k ones | 17:37 | |
| jeblair | fungi: +2; i'll let you aprv when done | 17:38 |
| fungi | thanks! | 17:38 |
| fungi | it's lovely how rackspace sometimes just completely ignores nova delete calls | 17:39 |
| jeblair | fungi: cloud | 17:40 |
| fungi | "eventually consistent (...so long as you keep trying!)" | 17:40 |
| *** nati_ueno has quit IRC | 17:42 | |
| *** jcoufal has quit IRC | 17:42 | |
| *** nati_ueno has joined #openstack-infra | 17:43 | |
| *** nati_ueno has quit IRC | 17:43 | |
| *** nati_ueno has joined #openstack-infra | 17:44 | |
| *** saju_m has quit IRC | 17:46 | |
| *** yassine has quit IRC | 17:46 | |
| *** jgallard has quit IRC | 17:48 | |
| *** lcostantino is now known as lcostantino_brb | 17:49 | |
| clarkb | morning | 17:51 |
| openstackgerrit | A change was merged to openstack-infra/config: drop gate-pycadf-python33 https://review.openstack.org/73603 | 17:52 |
| *** nati_ueno has quit IRC | 17:52 | |
| fungi | nom nom nom all gone | 17:53 |
| *** CaptTofu has joined #openstack-infra | 17:53 | |
| *** nati_uen_ has joined #openstack-infra | 17:54 | |
| zaro | SergeyLukjanov: ping | 17:56 |
| *** dkliban_afk is now known as dkliban | 17:56 | |
| SergeyLukjanov | clarkb, morning | 17:56 |
| SergeyLukjanov | zaro, hey, pong | 17:56 |
| zaro | SergeyLukjanov: question about this, https://review.openstack.org/#/c/60348/21/modules/openstack_project/files/jenkins_job_builder/config/gerrit.yaml | 17:56 |
| *** nati_uen_ has quit IRC | 17:57 | |
| SergeyLukjanov | zaro, it's not an issue, it's just not used anywhere else | 17:57 |
| zaro | SergeyLukjanov: i just grabbed that from macro.yaml | 17:57 |
| zaro | SergeyLukjanov: so is it ok to merge now? | 17:57 |
| SergeyLukjanov | zaro, I think that's ok | 17:58 |
| *** derekh has quit IRC | 17:58 | |
| SergeyLukjanov | zaro, gate is alive, so, apprv | 17:59 |
| zaro | SergeyLukjanov: Thanks! | 17:59 |
| *** banix has quit IRC | 17:59 | |
| openstackgerrit | A change was merged to openstack-infra/config: Shift more nodepool nodes onto rax-dfw https://review.openstack.org/73853 | 17:59 |
| *** oubiwann has joined #openstack-infra | 18:01 | |
| *** markmcclain has joined #openstack-infra | 18:03 | |
| *** oubiwann has quit IRC | 18:03 | |
| *** jroovers|afk is now known as jroovers | 18:03 | |
| *** moted has joined #openstack-infra | 18:03 | |
| *** dstanek has quit IRC | 18:03 | |
| *** oubiwann has joined #openstack-infra | 18:03 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 18:05 | |
| *** lcostantino_brb is now known as lcostantino | 18:06 | |
| *** markmcclain1 has joined #openstack-infra | 18:06 | |
| *** banix has joined #openstack-infra | 18:06 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:06 | |
| *** hashar has quit IRC | 18:07 | |
| *** markmcclain has quit IRC | 18:08 | |
| *** harlowja has joined #openstack-infra | 18:08 | |
| *** saju_m has joined #openstack-infra | 18:09 | |
| *** banix has quit IRC | 18:11 | |
| *** dizquierdo has quit IRC | 18:14 | |
| *** dripton has joined #openstack-infra | 18:14 | |
| *** oubiwann has quit IRC | 18:14 | |
| fungi | nodepool delete works waaay better now that it only hits the appropriate api endpoint rather than all the ones it knows about | 18:15 |
| jeblair | fungi: yaay! though i like my patch that makes it even faster. | 18:16 |
| *** dstanek has joined #openstack-infra | 18:16 | |
| fungi | yep, looking at that series next | 18:17 |
| *** saju_m has quit IRC | 18:17 | |
| *** gyee has joined #openstack-infra | 18:17 | |
| *** banix has joined #openstack-infra | 18:17 | |
| clarkb | so much code to review :) | 18:17 |
| fungi | in other good news, az2 has newly-minted bare-precise nodes again (from a fresh image) | 18:17 |
| *** gyee has quit IRC | 18:17 | |
| fungi | devstack-precise is still in the works though | 18:17 |
| jeblair | it's for a good cause. i wouldn't have proposed it if i didn't think it would significantly help us over the next couple weeks | 18:18 |
| *** banix has quit IRC | 18:18 | |
| *** gyee has joined #openstack-infra | 18:18 | |
| jeblair | fungi: that's good news indeed | 18:18 |
| *** cadenzajon_ has quit IRC | 18:18 | |
| fungi | so at least the quota there isn't wasted | 18:18 |
| jeblair | (i certainly think it would have significantly helped yesterday) | 18:18 |
| clarkb | jeblair: fungi: the last email I had from hp was that they were looking into az2. Did it miraculously work? or do we think they fixed something? | 18:20 |
| jeblair | clarkb: it fails most but not all of the time; fungi deleted the images so nodepool is in a tight loop trying to rebuild them | 18:20 |
| clarkb | gotcha | 18:21 |
| clarkb | so luck of the draw | 18:21 |
| *** saju_m has joined #openstack-infra | 18:21 | |
| fungi | increased our luck by as fast as nodepool was willing to retry, rather than once a day | 18:21 |
| jeblair | and when this is done, i'll see if we need to delete a bunch of keypairs. :) | 18:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 18:22 |
| *** fcarpenter has joined #openstack-infra | 18:22 | |
| zaro | clarkb: your review day? | 18:23 |
| clarkb | zaro: I have been trying to review stuff and keep getting distracted. I would like to sit down and review a bunch of stuff | 18:23 |
| clarkb | gate issues take precedence though | 18:23 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix changeid links in gerrit https://review.openstack.org/73771 | 18:23 |
| jeblair | fungi, clarkb: ^ we should restart gerrit later | 18:24 |
| *** e0ne has joined #openstack-infra | 18:24 | |
| fungi | yes, agreed | 18:24 |
| jeblair | (i find it significantly less usable without those links!) | 18:24 |
| *** mrodden has joined #openstack-infra | 18:25 | |
| zaro | yeah, +1 | 18:25 |
| fungi | deleting/recreating the openstack-api-ref job on all the masters now that 73690 has merged | 18:25 |
| *** jnoller has quit IRC | 18:28 | |
| *** saju_m has quit IRC | 18:28 | |
| *** cadenzajon has joined #openstack-infra | 18:28 | |
| annegentle | fungi: so this looks right, right? https://review.openstack.org/#/c/75106/ (sorry to be such a pest) :) | 18:30 |
| *** saju_m has joined #openstack-infra | 18:32 | |
| *** markwash_ has joined #openstack-infra | 18:32 | |
| fungi | annegentle: yep! look at http://git.openstack.org/cgit/openstack/operations-guide/log/?h=feature/edits | 18:32 |
| *** jpich has quit IRC | 18:33 | |
| fungi | annegentle: you should see changes mentioned there which are from your master branch over the past couple weeks since the feature branch was first created | 18:33 |
| *** mrmartin has joined #openstack-infra | 18:33 | |
| *** johnthetubaguy has quit IRC | 18:34 | |
| *** gsamfira has quit IRC | 18:35 | |
| *** dhellmann has joined #openstack-infra | 18:35 | |
| *** markwash has quit IRC | 18:36 | |
| *** markwash_ is now known as markwash | 18:36 | |
| *** oubiwann has joined #openstack-infra | 18:36 | |
| clarkb | ok quick check on logstash/ES shows it is improving. Down to 111k events in the backlog now | 18:37 |
| clarkb | I had to kick a few workers that fell over :( but I figured that may happen as I increased the workers counts to chew into the backlog | 18:37 |
| clarkb | I think 2 per host is safest. 3 runs into occasional problems and 4 is suicide | 18:37 |
| jog0 | clarkb: I assume you are aware of the occasional 502s from ES | 18:38 |
| SergeyLukjanov | jeblair, +1 about change id links | 18:38 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Basic gating and release jobs of openstackid project https://review.openstack.org/73549 | 18:38 |
| clarkb | jog0: I am not but doesn't surprise me | 18:38 |
| clarkb | jog0: the tl;dr is we have overrun it | 18:39 |
| clarkb | there is too much data and too many queries | 18:39 |
| clarkb | mostly too much data I think | 18:39 |
| jog0 | figured as much | 18:39 |
| *** tjones has quit IRC | 18:40 | |
| *** tjones has joined #openstack-infra | 18:40 | |
| *** afazekas_ has joined #openstack-infra | 18:41 | |
| clarkb | jog0: there are a lot of cache evictions happening | 18:42 |
| clarkb | which indicates to me that the problem space is now larger than ES can sanely keep in its head at one time | 18:42 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: add a new .gitreview file https://review.openstack.org/73126 | 18:43 |
| *** morganfainberg_Z is now known as morganfainberg | 18:44 | |
| jog0 | so I think geekinutah ca nget us igger boxes | 18:44 |
| jog0 | can get us* | 18:44 |
| clarkb | at rax? | 18:45 |
| jog0 | at bluehost | 18:45 |
| *** sarob has joined #openstack-infra | 18:45 | |
| clarkb | bluehost is not one of our gracious donated "hardware" providers though having more would be nice | 18:46 |
| dansmith | clarkb: they have some big boxes, | 18:46 |
| *** sabari_ has joined #openstack-infra | 18:46 | |
| dansmith | clarkb: and mentioned last week that they wouldn't mind helping with some one-off things like this | 18:46 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 18:46 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add -tripleo pipelines. https://review.openstack.org/73863 | 18:46 |
| dansmith | clarkb: whether they really want to when pressed is another question, but they seem pretty cool :) | 18:46 |
| *** gsamfira has joined #openstack-infra | 18:47 | |
| *** fcarpenter has left #openstack-infra | 18:47 | |
| lifeless | fungi: I've replied to your comment in 73863 | 18:47 |
| mrmartin | jeblair, SergeyLukjanov: may I ask a review on this patch: https://review.openstack.org/#/c/73549/ ? | 18:48 |
| *** geekinutah has joined #openstack-infra | 18:48 | |
| geekinutah | jog0: hallo | 18:48 |
| jog0 | geekinutah: o/ | 18:48 |
| jog0 | clarkb: ^ | 18:48 |
| jog0 | geekinutah: so our ElasticSearch boxes are too small | 18:48 |
| jeblair | i'd love for them to help. what would be great is to start using them for gate test nodes. we just need an openstack endpoint. | 18:48 |
| jog0 | geekinutah: meet jeblair, jeblair geekinutah (bluehost) | 18:49 |
| clarkb | jog0: the other possible way to tackle this is to add more nodes to the cluster and have more shards | 18:49 |
| clarkb | jog0: that doesn't solve the recovery time issues that we have as you still have to copy gigs (terrabytes) of data around, but should accomodate the search caching much better | 18:50 |
| *** pmathews1 has joined #openstack-infra | 18:50 | |
| jeblair | geekinutah: hi again. :) i feel like part of this conversation may have already happened. if you're interested in donating resources to openstack infra, that would be great. we run everything using openstack apis, so if you have that, it's easy for us to get started. | 18:50 |
| geekinutah | yeah, we did have this conversation :-) | 18:50 |
| jog0 | ahh I wasn't sure | 18:50 |
| clarkb | jog0: bigger boxes with more network bandwidth is the more ideal situation | 18:51 |
| jeblair | jog0, clarkb: the other other way to address this is to use bigger rax nodes. current ones are 30gb standard; we should definetely move them to performance nodes, and those go up to 120gb. | 18:51 |
| geekinutah | jog0: I'm basically doing what we talked about in Provo, fixing the API endpoint stuffs so we can use it | 18:51 |
| jog0 | geekinutah: nice! | 18:51 |
| *** gsamfira has quit IRC | 18:51 | |
| geekinutah | I started playing with NodePool, so far so good... | 18:52 |
| clarkb | jeblair: oh 120gb performance nodes would be nice. only potential issue there is disk IO would be to cinder volumes right? | 18:52 |
| jeblair | geekinutah: cool; i think maybe fungi mentioned which parts of it we use (mostly nova, a little cinder when needed, and we really wish we could use glance). | 18:52 |
| clarkb | jeblair: would need to double check that cinder volumes aren't a big IO hit | 18:52 |
| fungi | SergeyLukjanov: heads up, https://review.openstack.org/60348 added jobs which want precise nodes (not bare-precise) | 18:53 |
| clarkb | jeblair: but that certainly seems doable | 18:53 |
| geekinutah | jeblair, jog0: I will let you guys know when I have an endpoint so we can start playing with it | 18:53 |
| jog0 | geekinutah: neato | 18:53 |
| jeblair | clarkb: they come in ssd flavors, but it's probably worth trying spinning disks first for big volume. either should still be faster; if we consider the data ephemeral, we can use the local storage. | 18:54 |
| SergeyLukjanov | fungi, heh, shame on me, going to fix that | 18:54 |
| zaro | could someone please help with this, https://review.openstack.org/#/c/73126/ ? | 18:54 |
| fungi | SergeyLukjanov: cool--thanks! | 18:54 |
| jeblair | geekinutah: that's great! let us know if you have any questions too. i'm sorry nodepool is a bit lacking in documentation atm. i'm trying to fix that, slowly. | 18:54 |
| zaro | looks like the job gets lost? | 18:54 |
| clarkb | jeblair: local storage is too small though. we want about a TB on each node | 18:54 |
| SergeyLukjanov | fungi, should remove +2 marks from existing changes to check nodes again | 18:54 |
| fungi | zaro: SergeyLukjanov is fixing it | 18:54 |
| clarkb | jeblair: but 120GB nodes with cinder volumes should give us plenty of room to grow, I like this idea | 18:55 |
| jeblair | clarkb: 120gb performance has 1.2tb of ephemeral tsorage. not suggesting that's the best option. | 18:55 |
| *** marun has quit IRC | 18:55 | |
| clarkb | jeblair: oh they do. I thought all performance nodes were like 20GB | 18:55 |
| fungi | SergeyLukjanov: yeah, i'm trying to get time to go back through open changes looking for any which need updating based on the new node labels | 18:55 |
| zaro | also i noticed that jenkins02 doesn't have the 'check-gerrit-buck' job, while all of the other jenkins do have it. | 18:56 |
| *** marun has joined #openstack-infra | 18:56 | |
| jeblair | clarkb: it's more like hp -- small root and ehpemeral scales with node size | 18:56 |
| fungi | zaro: jjb seems to take longer to run on 02 for some reason, and frequently exceeds the puppet exec timeout | 18:56 |
| clarkb | jeblair: are performance node ephemeral disks more ephemeral than the standard flavor disk? | 18:56 |
| *** lcheng has quit IRC | 18:57 | |
| *** kfox1111 has joined #openstack-infra | 18:57 | |
| kfox1111 | getting an issue when trying to use disk-image-builder https://git.openstack.org/openstack/neutron.git/ checkout is failing. | 18:57 |
| kfox1111 | git clone says gnutls_handshake() failed: A TLS packet with unexpected length was received. | 18:57 |
| kfox1111 | curling it shows a 404. | 18:57 |
| fungi | clarkb: for some reason, yes. they switched to a much smaller root device | 18:57 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Use bare-precise nodes for gerrit buck jobs https://review.openstack.org/75137 | 18:57 |
| SergeyLukjanov | fungi, I'll check open changes for it later today | 18:57 |
| *** markmcclain1 has quit IRC | 18:58 | |
| SergeyLukjanov | fungi, https://review.openstack.org/75137 | 18:58 |
| fungi | kfox1111: have you tried http:// or git://? i'm guessing you may have a problem with a proxy | 18:58 |
| jeblair | clarkb: i think they are just not imaged with the server | 18:58 |
| clarkb | oh and we can use pvhvm images | 18:58 |
| *** markmcclain has joined #openstack-infra | 18:59 | |
| jeblair | clarkb: but i don't think they delete them as long as the server exists | 18:59 |
| jeblair | least, that's my understanding | 18:59 |
| jeblair | also, the local storage is all ssd | 18:59 |
| kfox1111 | fungi: ugg. your right. changing it to http made it work. :/ | 18:59 |
| kfox1111 | thanks. | 18:59 |
| jeblair | raid10 | 18:59 |
| jeblair | clarkb: http://www.rackspace.com/knowledge_center/article/what-is-new-with-performance-cloud-servers | 18:59 |
| clarkb | jeblair: ya I am reading that now | 18:59 |
| SergeyLukjanov | zaro, https://review.openstack.org/75137 | 19:00 |
| *** banix has joined #openstack-infra | 19:01 | |
| *** lcheng has joined #openstack-infra | 19:01 | |
| *** oubiwann has quit IRC | 19:02 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1282724 https://review.openstack.org/75138 | 19:02 |
| uvirtbot | Launchpad bug 1282724 in python-openstackclient "cliff.app Conflict occurred attempting to store user" [Undecided,New] https://launchpad.net/bugs/1282724 | 19:02 |
| SergeyLukjanov | fungi, thx | 19:02 |
| zaro | SergeyLukjanov: thanks. i think i is bare-precise | 19:02 |
| zaro | SergeyLukjanov: i think i've been told, but remind me what is bare-precise? | 19:02 |
| clarkb | jeblair: so I think this is worth a shot, when things settle down a bit we should sit down and plan a new cluster | 19:03 |
| SergeyLukjanov | zaro, it singe-use precise | 19:04 |
| SergeyLukjanov | zaro, single* | 19:04 |
| fungi | zaro: we replaced the old static "precise" slaves with "bare-precise" single-use nodes managed via nodepool | 19:04 |
| SergeyLukjanov | zaro, managed by nodepool | 19:04 |
| fungi | zaro: and then we deleted all the old static "precise" slaves once we confirmed that was working as intended | 19:04 |
| clarkb | jeblair: I have a hunch we can get away with the ephemeral disk then use pvhvm images for extra IOPs | 19:04 |
| jeblair | clarkb: cool, the hard thing is going to be finding the sweet spots for number and size of nodes, especially since we can't really use standard as a baseline. | 19:05 |
| clarkb | ya | 19:05 |
| clarkb | going straight to 120GB is tempting for the disk. I think 90GB probably has enough disk too | 19:06 |
| clarkb | just to avoid juggling volumes. But we could start small with 60GB images + volumes then if we need to upsize just reattach volumes on bigger nodes | 19:06 |
| jeblair | clarkb: i would dismiss cinder either... graphite is pretty io heavy and it's on cinder with rust. | 19:07 |
| jeblair | s/would/wouldn't/ | 19:07 |
| jeblair | gee | 19:07 |
| jeblair | clarkb: we can add one node to the cluster and see how it does, right? then iterate from there? | 19:07 |
| clarkb | jeblair: yes we can | 19:08 |
| jeblair | maybe one or two nodes, just to try to get a baseline | 19:08 |
| clarkb | jeblair: do you think we should do that now? or put it off for after feature freeze? | 19:08 |
| *** sarob has quit IRC | 19:08 | |
| clarkb | also I am tempted to have someone else do it so that other folks know how to massage es :) | 19:08 |
| clarkb | but I know everyone is busy so let me know | 19:10 |
| jeblair | clarkb: if you think we will catch up and have a useful system, let's leave it; but if you think we won't be able to use it until after ff anyway, let's start sooner. | 19:10 |
| lifeless | jeblair: https://review.openstack.org/#/c/73863/ is readty for re-review | 19:10 |
| clarkb | jeblair: I think that the current situation may be common. We will end up a few hours to half a day behind depending on load, catch up over the weekend then fall behind again | 19:10 |
| fungi | clarkb: i'm happy to be not you and dig into that, but need to dig myself back out from under stuff first | 19:11 |
| clarkb | fungi: awesome. Let me know if I can help with any digging | 19:11 |
| *** tjones has quit IRC | 19:12 | |
| jeblair | clarkb, fungi: ok, then let's at least add a couple new nodes soon as experiments to get a baseline (which will also help with the load) | 19:12 |
| fungi | clarkb: at the moment i'm trying to catch up on reviews. let's see what i can get knocked out today and then maybe go over details for building new ones tomorrow? | 19:12 |
| clarkb | sounds good | 19:12 |
| clarkb | I am going to dive into reviews today as wel | 19:12 |
| *** khyati_ has joined #openstack-infra | 19:13 | |
| fungi | trying to prioritize stuff like the nodepool improvements which will benefit us in the coming gitpocalypse | 19:13 |
| clarkb | yup yup | 19:13 |
| jeblair | https://review.openstack.org/#/c/71190/ is a good one that will help tripleo folks self-diagnose nodepool image problems | 19:15 |
| anteaya | lifeless: do you have a moment for -neutron? | 19:17 |
| sdague | jeblair, clarkb, fungi: https://review.openstack.org/#/c/74451/ - stable/havana is completely blocked until we approve that | 19:17 |
| lifeless | anteaya: sure | 19:17 |
| sdague | so a second set of eyes would be good | 19:17 |
| jeblair | lifeless: your change https://review.openstack.org/#/c/73863/5 lgtm; however your reply to fungi's comment illustrates a potential problem | 19:18 |
| *** mattray has joined #openstack-infra | 19:18 | |
| jeblair | lifeless: i think check and tripleo-check should be exclusive, otherwise they will race for the +1 vote. | 19:18 |
| clarkb | sdague: the check dsvm havana job that failed on that change failed for other reasons? | 19:18 |
| clarkb | sdague: I can look into it if you haven't yet, but checking if that was already diagnozed | 19:18 |
| fungi | lifeless: jeblair right, they will leave only one actual vote per change/patchset | 19:18 |
| jeblair | lifeless: either that, or make tripleo-check non-voting (but then you have the thing you described where people have to remember to look for the comments) | 19:19 |
| sdague | clarkb: that's got a jenkins +1 vote right now | 19:19 |
| lifeless | jeblair: I was going to have all the jobs in tripleo-check be non-voting; I presume I make jus tmake the pipeline nonvoting ? | 19:19 |
| openstackgerrit | A change was merged to openstack-infra/config: Basic gating and release jobs of openstackid project https://review.openstack.org/73549 | 19:19 |
| sdague | or is a recheck running now with failure? | 19:19 |
| clarkb | sdague: right the d-g non master jobs are non voting | 19:19 |
| clarkb | sdague: I mean this is in the grizzly branch so I think it must be not releated | 19:20 |
| lifeless | jeblair: This thing with people looking is a emergent fact of the same things making us have a separate pipeline for now I think | 19:20 |
| sdague | so the havana fail us unrelated, it's actually a tempest race condition I think | 19:20 |
| jeblair | lifeless: yeah, then that would be easier and you'll get a more meaningful 'Success' or 'Failure' message at the start of the comment. | 19:20 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Change 1254872 for catching issues from libvirtd https://review.openstack.org/74925 | 19:20 |
| clarkb | sdague: thanks and fungi has approved, beat me to it | 19:21 |
| jeblair | lifeless: do you want non-voting so that you aren't yourself blocked by a tripleo cloud failure? | 19:21 |
| sdague | cool, thanks folks | 19:21 |
| fungi | clarkb: yeah, that looks like a known nondeterministic failure on the tempext-havana branch job | 19:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 19:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add -tripleo pipelines. https://review.openstack.org/73863 | 19:21 |
| fungi | tempest | 19:21 |
| lifeless | jeblair: how does that look ? | 19:21 |
| SergeyLukjanov | fungi, I've find some open CRs with precise usage, -1'd all of them for correct node usage (https://review.openstack.org/#/c/71308/, https://review.openstack.org/#/c/68791/, https://review.openstack.org/#/c/73806/, https://review.openstack.org/#/c/73667/, https://review.openstack.org/#/c/73074/, https://review.openstack.org/#/c/70014/, https://review.openstack.org/#/c/69489/, | 19:21 |
| fungi | SergeyLukjanov: thank you very much! | 19:22 |
| jeblair | lifeless: i ask that last question to try to understand whether the nonvoting pipeline approach is better than two voting pipelines that are exclusive. | 19:22 |
| clarkb | jeblair: re image logs. I think you are most familiar with what gets logged there. Any worry of data leaks? eg cloud credentials? | 19:22 |
| lifeless | jeblair: I think a nonvoting pipeline is better because I had no intention of having voting check jobs in a separate pipeline | 19:23 |
| jeblair | clarkb: i can think of no case where that would happen -- that logger should only be used for the ssh session of the image build. | 19:23 |
| lifeless | jeblair: I think thats too hard to reason about | 19:23 |
| *** skraynev is now known as skraynev_afk | 19:23 | |
| clarkb | jeblair: roger thanks | 19:23 |
| lifeless | jeblair: since our goal is to be robust and fast enough that the other projects are happy with us being in the regular pipelines... this is ramp towards that goal | 19:24 |
| fungi | lifeless: jeblair: yeah, i hadn't seen the change which added the jobs back, and so assumed all jobs were going to get moved from the check pipeline to the check-tripleo pipeline (based on a misintrepretation of lifeless's reply in the review) | 19:24 |
| lifeless | fungi: they haven't been added back | 19:24 |
| *** vkozhukalov has joined #openstack-infra | 19:24 | |
| lifeless | fungi: I need to do that but wanted to let any kibbitzing on the plumbing happen first | 19:24 |
| *** oubiwann has joined #openstack-infra | 19:25 | |
| fungi | lifeless: right i know. talking about the theoretical change which didn't exist yet for adding them | 19:25 |
| lifeless | roger | 19:25 |
| fungi | anyway i think what you have here will work | 19:25 |
| *** dpyzhov has joined #openstack-infra | 19:25 | |
| *** zul has quit IRC | 19:26 | |
| jeblair | lifeless: hrm. i'm not following your reasoning there.... more typing follows... | 19:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Use bare-precise nodes for gerrit buck jobs https://review.openstack.org/75137 | 19:26 |
| *** zul has joined #openstack-infra | 19:26 | |
| clarkb | jeblair: zaro: btw jeblair's nodepool log change may conflict with https://review.openstack.org/#/c/61321/ I think the logging change is more important though so will be approving it if I can't find trouble with it | 19:27 |
| *** tjones has joined #openstack-infra | 19:27 | |
| fungi | oh, lifeless: jeblair: will that "verified: 0" potentially unset the vote being added by the real check pipeline? you probably want to leave out the start actions on check-tripleo | 19:27 |
| lifeless | doh yes | 19:27 |
| jeblair | lifeless: i think the reason to chose two exclusive voting pipelines is so that you can take advantage of the +1 requirement and it's basically no visible change for tripleo devs. | 19:27 |
| jeblair | lifeless: the reason to chose a nonvoting one is so that if the triplo cloud is unavailable, your own patches are not blocked. | 19:28 |
| lifeless | jeblair: I may misunderstand what two exclusive voting pipelines means | 19:28 |
| lifeless | jeblair: are you saying any one project would have either check xor check-tripleo jobs defined ? | 19:28 |
| *** gordc has left #openstack-infra | 19:28 | |
| jeblair | lifeless: yes | 19:28 |
| lifeless | ok | 19:29 |
| lifeless | but the reason we're putting in the pipeline is as a risk mitigation against the cloud having an operational problem | 19:29 |
| fungi | which i agree does sound better, and was what i assumed you were going for initially | 19:29 |
| lifeless | whether that be load or downtime | 19:29 |
| jeblair | lifeless: (i think either approach is fine, i just want to make sure we chose the right one for the right reasons) | 19:29 |
| lifeless | hmm, wish derekh were around. dprince - what do you think? | 19:30 |
| jeblair | lifeless: yes. i'm assuming that only tripleo projects would use the check-tripleo pipeline, and so you can make your own risk/reward tradeoff there. | 19:30 |
| *** dolphm is now known as dolphm_503 | 19:30 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1282724 https://review.openstack.org/75138 | 19:30 |
| uvirtbot | Launchpad bug 1282724 in python-openstackclient "cliff.app Conflict occurred attempting to store user" [Undecided,New] https://launchpad.net/bugs/1282724 | 19:30 |
| jeblair | lifeless: (it won't affect other projects, and the builds will be just as stuck in zuul regardless of which pipeline, so it's all the same to me :) | 19:30 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/config: Add the kite (KDS) project and kiteclient project to gerrit/jenkins https://review.openstack.org/73074 | 19:31 |
| clarkb | jeblair: nodepool logging change has been approved | 19:31 |
| *** mrmartin has quit IRC | 19:31 | |
| clarkb | I can keep an eye on that | 19:31 |
| jeblair | mattoliverau: ^ | 19:31 |
| *** thomasem has joined #openstack-infra | 19:33 | |
| *** oubiwann has quit IRC | 19:34 | |
| openstackgerrit | A change was merged to openstack-infra/config: Nodepool image logs accessible via HTTP https://review.openstack.org/71190 | 19:34 |
| *** _david_ has joined #openstack-infra | 19:35 | |
| lifeless | jeblair: ok on balance I don't think we should be more risk embracing than other projects here. | 19:37 |
| jeblair | lifeless: ok, so nonvoting pipeline then, and review your current patch? | 19:37 |
| lifeless | jeblair: its not a lack of confidence, its that folk need to get work done; we need the facility, but shit can happen and we don't want to make it an infra firedrill if it does. Even indirectly. | 19:37 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for swift bug 1274828 https://review.openstack.org/75154 | 19:37 |
| uvirtbot | Launchpad bug 1274828 in swift "grenade tests fail with error trying to connect Account server" [Undecided,Confirmed] https://launchpad.net/bugs/1274828 | 19:37 |
| lifeless | jeblair: please | 19:37 |
| jeblair | wfm | 19:38 |
| jeblair | lifeless: fungi has a thing that needs fixin | 19:38 |
| fungi | i'm going to manually re-run jjb on jenkins02. some jobs are trying to use the wrong node type still, because the puppet exec times out more frequently there | 19:39 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/reviewstats: Add murano core team https://review.openstack.org/75156 | 19:39 |
| clarkb | fungi: ok | 19:39 |
| _david_ | jeblair, fungi, clarkb : Have you noticed new submit_on_behalf_of feature in Gerrit: https://gerrit-review.googlesource.com/54153 ? | 19:39 |
| lifeless | jeblair: yeah, I paused while we spoke | 19:40 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 19:40 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add -tripleo pipelines. https://review.openstack.org/73863 | 19:40 |
| lifeless | fungi: ^ thanks! | 19:40 |
| *** rossella-s has quit IRC | 19:42 | |
| jeblair | _david_: no i haven't seen that. neat! | 19:42 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Reset the client object before creating the image https://review.openstack.org/74943 | 19:42 |
| _david_ | jeblair, Chad asked for it, I guess for their Zuul installation ;-) | 19:43 |
| mordred | jeblair: ^^ there's an updated version of the proxy-deal-with patch | 19:43 |
| mordred | _david_: VERY COOL | 19:43 |
| *** tjones has quit IRC | 19:43 | |
| _david_ | mordred, thanks, kudos go to Chad too, for asking for that... | 19:44 |
| jeblair | _david_: do you know if chad is on irc? | 19:44 |
| mordred | fungi: ok - I'm going to hack on the pip thing - do we know specifically why pypy was messed up otherwise? | 19:44 |
| *** smarcet has left #openstack-infra | 19:44 | |
| _david_ | jeblair, i don't see him now | 19:44 |
| jeblair | _david_: just thought i'd say hi and good idea. :) | 19:45 |
| _david_ | d^ or something like that was his nick name | 19:45 |
| jeblair | _david_: oh, i know him. i don't think i ever knew his real name. :) | 19:45 |
| *** dkliban is now known as dkliban_afk | 19:45 | |
| _david_ | jeblair, let me pint you on his discussion with Gerrit maintainer on the log | 19:45 |
| *** hogepodge has quit IRC | 19:47 | |
| fungi | mordred: it looks like the setuptools installation we get when setting python3=yes on the slave manifest works with pip in a python 3.3 tox virtualenv, but not with a pypy virtualenv (the log i liked you to shows setuptools being uninstalled in the virtualenv and then a subsequent package installation fails). it looks like the old hand-created slaves ended up with system-installed python 2.7 | 19:47 |
| fungi | setuptools which dodn't get removed later | 19:47 |
| mordred | fungi: oh - is it because pypy consumes libs installed in python2.7 land/ | 19:48 |
| mordred | ? | 19:48 |
| fungi | mordred: i believe so, yes | 19:48 |
| mordred | ok | 19:48 |
| _david_ | jeblair, http://echelog.com/logs/browse/gerrit/1391036400 | 19:49 |
| mordred | fungi: after we run puppet in prepare_node - we don't ever run it again, right? | 19:49 |
| fungi | and i think on our new slaves we don't install setuptools globally any longer, or perhaps only install it using python 3.3 | 19:49 |
| _david_ | <^d> Hi gerrit-ers. ACL question. I'm loving the "on behalf of" for review commands. I'm wondering if something similar is in the pipeline for Submit for 2.9? | 19:49 |
| fungi | mordred: i believe so | 19:49 |
| _david_ | <^d> I shall file a feature request then. My use case is possibly having Jenkins submit on behalf of the person who kicked off the jenkins job, rather than on behalf of itself. | 19:49 |
| _david_ | <^d> Well I still want jenkins to review for itself. | 19:49 |
| _david_ | <^d> But right now I'm having jenkins do the submit, which means we're ending up with merge commits by jenkins. | 19:49 |
| _david_ | <^d> People complain that it makes history ugly when the committer is jenkins. | 19:49 |
| *** hogepodge has joined #openstack-infra | 19:50 | |
| mordred | so in our case, we'd make the submitter be the person who voted +A yeah? | 19:50 |
| jeblair | mordred: i believe that's the idea. i'm not as convinced as ^d is that having jenkins as the author of record for merge commits is wrong or ugly. but i'm certainly happy to have the option in zuul and consider it for us. | 19:51 |
| mordred | fungi: and, just to be sure I'm up to speed - we're using the python3k nodes for pypy | 19:51 |
| mordred | jeblair: ++ | 19:51 |
| fungi | mordred: yes. i'm not entirely clear on why that was done, but Alex_Gaynor may have some more details | 19:52 |
| *** branen has joined #openstack-infra | 19:52 | |
| clarkb | fungi: iirc it was because py3k and pypy were both experimental at the time | 19:52 |
| clarkb | so killing one or both wouldn't really affect too muh | 19:52 |
| fungi | when we build py3k slaves we pass python3=true and include_pypy=true | 19:52 |
| jeblair | mordred: (i expect developers who would have their commit stats doubled would be strongly in favor of this :) | 19:52 |
| Alex_Gaynor | fungi: someone told me to, I don't remember why :P | 19:52 |
| mordred | jeblair: :) | 19:53 |
| fungi | Alex_Gaynor: good answer! | 19:53 |
| fungi | mordred: i'm entirely open to alternative suggestions besides colocating python3.3 and pypy interpreters on one node type | 19:53 |
| mordred | nope. fine by me - just making sure I was hacking in the right place | 19:54 |
| *** branen_ has quit IRC | 19:55 | |
| clarkb | so vim just segfaulted on me | 19:56 |
| fungi | that's because it loves you | 19:57 |
| clarkb | so this is amazing I can do it repeatedly | 19:57 |
| *** dkliban_afk is now known as dkliban | 19:57 | |
| fungi | bad plugin/extension? | 19:57 |
| mordred | open a .yaml.erb file did you? | 19:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Add -tripleo pipelines. https://review.openstack.org/73863 | 19:58 |
| clarkb | mordred: yes | 19:58 |
| clarkb | mordred: is this a known problem? | 19:58 |
| mordred | clarkb: I have had that happen to me before | 19:59 |
| fungi | i can edit .yaml.erb files with vim 7.4, no problem | 19:59 |
| clarkb | clearly I should've stuck with precise | 19:59 |
| *** b3nt_pin is now known as beagles | 19:59 | |
| mordred | clarkb: I can usually edit them | 19:59 |
| clarkb | the core file is pretty useless as there are no debugging symbols | 19:59 |
| mordred | but somewhere there is a combination of erb and yaml that can cause the segfault | 19:59 |
| mordred | so it's not all yaml erb - just s+something_ we happen to do in one of our files _sometimes_ | 19:59 |
| fungi | clarkb: i suspect you need a vim.debug deb or some such installed to get those | 19:59 |
| mordred | fungi: ubuntu moved those to a weird place because $reasons | 20:00 |
| fungi | seems i recall ubuntu builds those for most c-based applications | 20:00 |
| fungi | oh | 20:00 |
| mordred | they do build them | 20:00 |
| mordred | I can just never remember how to install them after ubuntu got too clever and broke things | 20:00 |
| anteaya | clarkb: what release are you using? | 20:00 |
| clarkb | anteaya: saucy xubuntu | 20:00 |
| fungi | clarkb: alternatively, apt-get source the requisite package, edit debian/rules to omit dh_strip, and then debuild it | 20:01 |
| *** zhiyan_ is now known as zhiyan | 20:01 | |
| clarkb | fungi: or maybe I should take up emacs | 20:01 |
| fungi | clarkb: emacs doesn't segfault. it just inflicts lisp condescension upon you ;) | 20:02 |
| anteaya | clarkb: is in beta yet? | 20:02 |
| anteaya | or alpha still | 20:02 |
| anteaya | oh I am thinking T | 20:02 |
| clarkb | anteaya: it is the latest release, I figured with trusty coming up soon I could upgrade to that from saucy | 20:02 |
| *** mrda_away is now known as mrda | 20:02 | |
| clarkb | I have also noticed that firefox + gerrit on saucy do not like each other | 20:02 |
| * clarkb cats file, just needed to read it anyways | 20:03 | |
| jeblair | clarkb: you could use emacs with vi-mode. | 20:03 |
| anteaya | does cat work? | 20:03 |
| jeblair | fungi: condescending enough? :) | 20:03 |
| clarkb | anteaya: yes, this appears to be a vim issue | 20:03 |
| clarkb | jeblair: :) | 20:03 |
| *** hashar has joined #openstack-infra | 20:03 | |
| clarkb | jeblair: fungi lifeless https://review.openstack.org/#/c/73886/7 needs to be rebased on top of a change to add tripleo back to nodepool right? | 20:04 |
| * clarkb looks for that change | 20:04 | |
| *** mattray has left #openstack-infra | 20:04 | |
| clarkb | I don't see one | 20:04 |
| lifeless | I'll prep nodepool + zuul changes for that now | 20:04 |
| lifeless | since we've go the framework in place | 20:04 |
| clarkb | lifeless: thanks | 20:05 |
| clarkb | fungi: mordred: the symbols hide in their own deb archive | 20:06 |
| fungi | jeblair: i'm not surprised emacs includes vi. after all, most operating systems do | 20:06 |
| clarkb | mordred: fwiw the nodepool.yaml.erb is doing it to me now. any cahnce you can check if it does it to you too? | 20:09 |
| fungi | clarkb: my vim can edit it fine (debian/sid). my nifty plugins tell me on the status line "[Git(master)]nodepool.yaml.erb [utf-8,unix][+][eruby]" | 20:10 |
| dkehn_ | fungi, I'm see an issue in the gate-python-neutronclient-pypy--> http://logs.openstack.org/66/49166/6/check/gate-python-neutronclient-pypy/e5852d8/console.html, which is a bad | 20:11 |
| dkehn_ | option when installing mccabe, are there known issues there? | 20:11 |
| fungi | clarkb: i wonder if it's a ruby syntax highlighter gone awry | 20:12 |
| *** cadenzajon has quit IRC | 20:12 | |
| fungi | dkehn_: what was the job where you saw that? a pypy job? | 20:12 |
| *** lcheng has quit IRC | 20:12 | |
| dkehn_ | fungi, http://logs.openstack.org/66/49166/6/check/gate-python-neutronclient-pypy/e5852d8/ | 20:12 |
| clarkb | fungi: yeah I am trying it with the vim-ruby package installed next | 20:12 |
| *** rcleere has quit IRC | 20:13 | |
| dkehn_ | fungi, https://review.openstack.org/#/c/49166/ | 20:13 |
| *** dizquierdo has joined #openstack-infra | 20:13 | |
| clarkb | that didn't help but I also installed vim-dbg so off to gdb | 20:13 |
| openstackgerrit | Matt Ray proposed a change to openstack-infra/config: Create new Chef cookbook-openstack-integration-test for Tempest support. https://review.openstack.org/68791 | 20:13 |
| fungi | dkehn_: yep. that's fixed now. we moved pypy jobs to a new node type but their success was apparently depending on unpuppeted setuptools cruft on the old slaves, so we've moved them back for now | 20:13 |
| clarkb | hrm that didn't help I must be doing something wrong | 20:14 |
| *** denis_makogon_ has joined #openstack-infra | 20:14 | |
| dkehn_ | fungi, so its still broke then, or rerun it | 20:14 |
| fungi | dkehn_: recheck it and it should pass | 20:14 |
| dkehn_ | fungi, love the sound of that | 20:15 |
| *** dprince has quit IRC | 20:16 | |
| dkehn_ | fungi, is that a recheck or recheck nobug? | 20:17 |
| fungi | recheck no bug | 20:17 |
| dkehn_ | fungi ouch | 20:17 |
| *** mrodden has quit IRC | 20:17 | |
| fungi | unless someone wants to open a resolved openstack-ci bug about it | 20:18 |
| clarkb | :) vim-ruby put vim-gtk in my path instead of vim.basic or was it tiny... | 20:18 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Revert "Temporarily stop running tripleo seed/undercloud" https://review.openstack.org/75173 | 20:19 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 20:20 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Revert "Remove tripleo from nodepool config" https://review.openstack.org/72035 | 20:20 |
| clarkb | in may_core_dump () at os_unix.c:3190 <- best function name ever | 20:20 |
| lifeless | clarkb: ^ | 20:20 |
| clarkb | lifeless: danke reviewing now | 20:20 |
| anteaya | post jobs are all queued, except for upstream-translation-update which is running | 20:21 |
| anteaya | way to go proposal.slave.olo | 20:21 |
| anteaya | o.o | 20:21 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 20:26 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try Icehouse precise proposed UCA - Not for review https://review.openstack.org/74889 | 20:26 |
| *** dizquierdo has quit IRC | 20:30 | |
| *** prad has joined #openstack-infra | 20:30 | |
| *** sarob has joined #openstack-infra | 20:32 | |
| *** prad_ has quit IRC | 20:33 | |
| *** rlandy has quit IRC | 20:33 | |
| clarkb | there is a lot of queueing in zuul | 20:33 |
| fungi | i just saw | 20:33 |
| * clarkb tails nodepool logs | 20:34 | |
| fungi | and the nodepool graph makes me think something's stuck | 20:34 |
| clarkb | ya | 20:34 |
| *** masayukig has joined #openstack-infra | 20:34 | |
| anteaya | whatever it is, it is not affecting proposal.slave apparently | 20:35 |
| clarkb | there appears to be long waits between errors happening for node deletion | 20:35 |
| clarkb | anteaya: proposal.slave is one of our few remaining non single use slaves so it should be happy | 20:35 |
| fungi | clarkb: proposed changes to nodepool should make that go faster | 20:35 |
| clarkb | oh that wasn't the debug log | 20:35 |
| clarkb | fungi: right | 20:35 |
| *** sarob has quit IRC | 20:36 | |
| anteaya | seems to be the only happy one atm | 20:36 |
| *** sarob has joined #openstack-infra | 20:36 | |
| *** gyee has quit IRC | 20:37 | |
| clarkb | someone just did a stack dump | 20:37 |
| jeblair | o/ | 20:37 |
| fungi | nodepool seems to think there are a fair number of ready and used nodes which i'm not seeing on the jenkins masters | 20:38 |
| clarkb | fungi: ya and it doesn't seem to be trying to launch many new nodes which could be explained by ^ | 20:39 |
| jeblair | the stacktraces tell me it's basically idle | 20:39 |
| clarkb | jeblair: yup | 20:39 |
| fungi | most of the used nodes seem to be in rax-iad by a fairly wide margin | 20:39 |
| *** mrodden has joined #openstack-infra | 20:39 | |
| fungi | though the ready nodes are fairly evenly distributed across providers and jenkins masters | 20:40 |
| clarkb | jeblair: fungi: I don't see any zmq connections | 20:40 |
| jeblair | clarkb: i was just noticing the same thing | 20:40 |
| *** ociuhandu has quit IRC | 20:40 | |
| jeblair | clarkb: also, spot checking an offline node in jenkins, nodepool never logged having received a zmq notification | 20:40 |
| jeblair | for that node | 20:40 |
| clarkb | jeblair: logstash's gearman client still has tcp connections for zmq | 20:41 |
| clarkb | so this doesn't appear to be an issue everywhere | 20:41 |
| jeblair | https://etherpad.openstack.org/p/GrlJrrgw1V | 20:42 |
| jeblair | that looks strange | 20:42 |
| clarkb | jeblair: it received a signal? | 20:43 |
| *** denis_makogon_ has quit IRC | 20:43 | |
| fungi | maybe refreshed from a config update? | 20:43 |
| clarkb | I wonder if the puppet change to add new logging borked it | 20:43 |
| jeblair | clarkb, fungi: hah, i think we may be witnessing a graceful nodepool shutdown. | 20:44 |
| clarkb | oh interesting | 20:44 |
| *** relaxdiego has joined #openstack-infra | 20:44 | |
| *** tjones has joined #openstack-infra | 20:44 | |
| *** mrodden has quit IRC | 20:44 | |
| jeblair | i think we need to redefine that a little bit... | 20:44 |
| fungi | it's not as beautiful as, say, witnessing a sunset | 20:44 |
| *** jerryz has quit IRC | 20:44 | |
| *** dpyzhov has quit IRC | 20:44 | |
| jeblair | basically, it'll wait until everything is deleted and launched. | 20:44 |
| clarkb | jeblair: instead it should probably stop launches, give up on deletes and die then start again | 20:45 |
| jeblair | i think with my stack of changes, we can probably make it exit a bit faster with the expectation that it will cleanup on restart. | 20:45 |
| *** mrodden has joined #openstack-infra | 20:45 | |
| * fungi gets back to reviewing those | 20:45 | |
| *** gordc has joined #openstack-infra | 20:45 | |
| *** dpyzhov has joined #openstack-infra | 20:45 | |
| jeblair | i think we ought to go ahead and hard-restart it, and i'll just go ahead and install the tip of my branch manually for now. | 20:46 |
| clarkb | ok | 20:47 |
| * clarkb goes back to reviewing | 20:47 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 20:47 | |
| clarkb | jeblair: when you are done maybe you can look at lifeless' stack here https://review.openstack.org/#/c/72035/ | 20:48 |
| *** tjones has quit IRC | 20:48 | |
| clarkb | since we should be ready for that I think | 20:48 |
| fungi | zomg... just spotted why jjb isn't updating on jenkins02 today. wasn't the exec timeout from before | 20:49 |
| mtreinish | clarkb: if you're reviewing I've got quite a few (at least for me) infra patches up :) | 20:49 |
| fungi | puppet agent is stuck with a child process doing "git-remote-https origin https://github.com/openstack-infra/jenkins-job-builder.git" | 20:49 |
| *** e0ne has quit IRC | 20:50 | |
| jeblair | fungi: yeah, i think we changed that in puppet but old git repos on old servers still have the old remote | 20:50 |
| fungi | precisely. i'm cleaning that up | 20:50 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add satori - Configuration Discovery https://review.openstack.org/73667 | 20:50 |
| clarkb | mtreinish: ya focusing on nodepool related things now, but hope to get through that and on to normal reviewing | 20:50 |
| clarkb | fungi: fun | 20:50 |
| *** prad_ has joined #openstack-infra | 20:51 | |
| jeblair | found a live bug in lifeless's nodepool patch; i'm going to see if i can fix quickly. | 20:52 |
| openstackgerrit | Gabriele Cerami proposed a change to openstack-infra/jenkins-job-builder: Added support for python virtualenv plugin https://review.openstack.org/71926 | 20:52 |
| *** prad has quit IRC | 20:53 | |
| *** prad_ is now known as prad | 20:53 | |
| fungi | yep, if i blow away the clone under /opt and rerun puppet manually it gets created using the updated remote urls. good | 20:53 |
| *** gsamfira has joined #openstack-infra | 20:54 | |
| jeblair | too much to do while the system is down; i'm reverting that patch in production then i'll fix it locally and push it for review | 20:57 |
| jeblair | fungi, clarkb: we still need to do something about the false-ready nodes | 20:58 |
| jeblair | fungi, clarkb: i propose we nodepool delete every ready node | 21:00 |
| jeblair | there's only one running job right now. | 21:00 |
| jeblair | so it's a pretty minimal impact | 21:00 |
| jeblair | fungi, clarkb: if that works for you, i'll just 'update node set state=4' in the db. | 21:01 |
| jeblair | and the new cleanup stuff should handle the rest. | 21:01 |
| *** sarob has quit IRC | 21:05 | |
| *** masayukig has quit IRC | 21:05 | |
| *** branen has quit IRC | 21:05 | |
| *** zul has quit IRC | 21:05 | |
| *** markwash has quit IRC | 21:05 | |
| *** harlowja has quit IRC | 21:05 | |
| *** pmathews1 has quit IRC | 21:05 | |
| *** dims has quit IRC | 21:05 | |
| *** alexpilotti has quit IRC | 21:05 | |
| *** ianw has quit IRC | 21:05 | |
| *** beagles has quit IRC | 21:05 | |
| *** jgriffith has quit IRC | 21:05 | |
| *** jog0 has quit IRC | 21:05 | |
| *** enikanorov__ has quit IRC | 21:05 | |
| *** NikitaKonovalov has quit IRC | 21:05 | |
| *** Hunner has quit IRC | 21:05 | |
| *** clarkb has quit IRC | 21:05 | |
| *** jesusaurus has quit IRC | 21:05 | |
| *** adarazs has quit IRC | 21:05 | |
| *** dstanek has quit IRC | 21:05 | |
| *** david-lyle has quit IRC | 21:05 | |
| *** julim_ has quit IRC | 21:05 | |
| *** uvirtbot has quit IRC | 21:05 | |
| *** markmcclain has quit IRC | 21:05 | |
| *** russellb has quit IRC | 21:05 | |
| *** adam_g has quit IRC | 21:05 | |
| *** Guest14741 has quit IRC | 21:05 | |
| *** stevebaker has quit IRC | 21:05 | |
| *** pvo has quit IRC | 21:05 | |
| *** rainya has quit IRC | 21:05 | |
| *** cyeoh has quit IRC | 21:05 | |
| *** tteggel has quit IRC | 21:05 | |
| *** EntropyWorks has quit IRC | 21:05 | |
| *** annegentle has quit IRC | 21:05 | |
| *** dragondm has quit IRC | 21:05 | |
| *** mattymo has quit IRC | 21:05 | |
| *** mikal has quit IRC | 21:05 | |
| *** mtreinish has quit IRC | 21:05 | |
| *** plomakin has quit IRC | 21:05 | |
| *** bnemec has quit IRC | 21:05 | |
| *** gaelL has quit IRC | 21:05 | |
| *** tristanC has quit IRC | 21:05 | |
| *** ttx has quit IRC | 21:05 | |
| *** msuriar has quit IRC | 21:05 | |
| *** dangers_away has quit IRC | 21:05 | |
| *** wendar has quit IRC | 21:05 | |
| *** StevenK has quit IRC | 21:05 | |
| *** jlk has quit IRC | 21:05 | |
| *** fungi has quit IRC | 21:05 | |
| *** jbryce has quit IRC | 21:05 | |
| *** mrodden has quit IRC | 21:05 | |
| *** mriedem has quit IRC | 21:05 | |
| *** bknudson has quit IRC | 21:05 | |
| *** pabelanger has quit IRC | 21:05 | |
| *** dstufft has quit IRC | 21:05 | |
| *** luisg has quit IRC | 21:05 | |
| *** bookwar has quit IRC | 21:05 | |
| *** sdague has quit IRC | 21:05 | |
| *** oubiwann-home has quit IRC | 21:05 | |
| *** sbadia has quit IRC | 21:05 | |
| *** jeremyb has quit IRC | 21:05 | |
| *** che-arne has quit IRC | 21:05 | |
| *** loquacities has quit IRC | 21:05 | |
| *** vrovachev has quit IRC | 21:05 | |
| *** exz has quit IRC | 21:05 | |
| *** bodepd has quit IRC | 21:05 | |
| *** sileht has quit IRC | 21:05 | |
| *** psedlak has quit IRC | 21:05 | |
| *** ekarlso has quit IRC | 21:05 | |
| *** Adri2000 has quit IRC | 21:05 | |
| *** akscram has quit IRC | 21:05 | |
| *** hashar has quit IRC | 21:05 | |
| *** hogepodge has quit IRC | 21:05 | |
| *** kfox1111 has quit IRC | 21:05 | |
| *** sabari_ has quit IRC | 21:05 | |
| *** CaptTofu has quit IRC | 21:05 | |
| *** esker has quit IRC | 21:05 | |
| *** ravikumar_hp has quit IRC | 21:05 | |
| *** changbl has quit IRC | 21:05 | |
| *** sandywalsh_ has quit IRC | 21:05 | |
| *** jasond`` has quit IRC | 21:05 | |
| *** tru_tru has quit IRC | 21:05 | |
| *** zhiyan has quit IRC | 21:05 | |
| *** miqui has quit IRC | 21:05 | |
| *** jaypipes has quit IRC | 21:05 | |
| *** gmoro has quit IRC | 21:05 | |
| *** khyati_ has quit IRC | 21:05 | |
| *** DinaBelova has quit IRC | 21:05 | |
| *** milki has quit IRC | 21:05 | |
| *** portante has quit IRC | 21:05 | |
| *** afazekas has quit IRC | 21:05 | |
| *** bradm has quit IRC | 21:05 | |
| *** ihrachys has quit IRC | 21:05 | |
| *** koolhead17 has quit IRC | 21:05 | |
| *** belliott has quit IRC | 21:05 | |
| *** tian has quit IRC | 21:05 | |
| *** jooools has quit IRC | 21:05 | |
| *** maurosr has quit IRC | 21:05 | |
| *** prad has quit IRC | 21:05 | |
| *** dpyzhov has quit IRC | 21:05 | |
| *** saju_m has quit IRC | 21:05 | |
| *** nicedice has quit IRC | 21:05 | |
| *** wenlock has quit IRC | 21:05 | |
| *** jergerber has quit IRC | 21:05 | |
| *** max_lobur_afk has quit IRC | 21:05 | |
| *** jd__ has quit IRC | 21:05 | |
| *** weshay has quit IRC | 21:05 | |
| *** SpamapS has quit IRC | 21:05 | |
| *** yongli has quit IRC | 21:05 | |
| *** amotoki has quit IRC | 21:05 | |
| *** homeless has quit IRC | 21:05 | |
| *** vipul has quit IRC | 21:05 | |
| *** locke1051 has quit IRC | 21:05 | |
| *** michchap has quit IRC | 21:05 | |
| *** nibalizer has quit IRC | 21:05 | |
| *** HenryG has quit IRC | 21:05 | |
| *** aude has quit IRC | 21:05 | |
| *** gothicmindfood has quit IRC | 21:05 | |
| *** vladan has quit IRC | 21:05 | |
| *** ilyashakhat_ has quit IRC | 21:05 | |
| *** gpocentek has quit IRC | 21:05 | |
| *** _david_ has quit IRC | 21:05 | |
| *** thomasem has quit IRC | 21:05 | |
| *** relaxdiego has quit IRC | 21:05 | |
| *** marun has quit IRC | 21:05 | |
| *** atiwari has quit IRC | 21:05 | |
| *** mdenny has quit IRC | 21:05 | |
| *** dkranz has quit IRC | 21:05 | |
| *** eharney has quit IRC | 21:05 | |
| *** ryanpetrello has quit IRC | 21:05 | |
| *** jpeeler has quit IRC | 21:05 | |
| *** yamahata has quit IRC | 21:05 | |
| *** dcramer_ has quit IRC | 21:05 | |
| *** juice has quit IRC | 21:05 | |
| *** ArxCruz has quit IRC | 21:05 | |
| *** rfolco has quit IRC | 21:05 | |
| *** kiall has quit IRC | 21:05 | |
| *** skraynev_afk has quit IRC | 21:05 | |
| *** yamahata_ has quit IRC | 21:05 | |
| *** mestery has quit IRC | 21:05 | |
| *** blamar has quit IRC | 21:05 | |
| *** Steap_ has quit IRC | 21:05 | |
| *** samalba has quit IRC | 21:05 | |
| *** chmouel has quit IRC | 21:05 | |
| *** mburned has quit IRC | 21:05 | |
| *** jcooley has quit IRC | 21:05 | |
| *** ogelbukh has quit IRC | 21:05 | |
| *** kmartin has quit IRC | 21:05 | |
| *** jang has quit IRC | 21:05 | |
| *** vkozhukalov has quit IRC | 21:05 | |
| *** afazekas_ has quit IRC | 21:05 | |
| *** dripton has quit IRC | 21:05 | |
| *** moted has quit IRC | 21:05 | |
| *** gokrokve has quit IRC | 21:05 | |
| *** VijayT has quit IRC | 21:05 | |
| *** primeministerp has quit IRC | 21:05 | |
| *** julienvey has quit IRC | 21:05 | |
| *** unicell1 has quit IRC | 21:05 | |
| *** andre__ has quit IRC | 21:05 | |
| *** Hefeweizen has quit IRC | 21:05 | |
| *** asadoughi has quit IRC | 21:05 | |
| *** medieval1 has quit IRC | 21:05 | |
| *** esmute has quit IRC | 21:05 | |
| *** davidlenwell has quit IRC | 21:05 | |
| *** gema has quit IRC | 21:05 | |
| *** mordred has quit IRC | 21:05 | |
| *** GheRivero has quit IRC | 21:05 | |
| *** mugsie has quit IRC | 21:05 | |
| *** gsamfira has quit IRC | 21:05 | |
| *** banix has quit IRC | 21:05 | |
| *** boris-42_ has quit IRC | 21:05 | |
| *** dkliban has quit IRC | 21:05 | |
| *** jroovers has quit IRC | 21:05 | |
| *** gordc has quit IRC | 21:05 | |
| *** matel has quit IRC | 21:05 | |
| *** yolanda has quit IRC | 21:05 | |
| *** rockstar has quit IRC | 21:05 | |
| *** jeckersb has quit IRC | 21:05 | |
| *** krtaylor has quit IRC | 21:05 | |
| *** hughsaunders has quit IRC | 21:05 | |
| *** vishy has quit IRC | 21:05 | |
| *** leifmadsen has quit IRC | 21:05 | |
| *** cp16net has quit IRC | 21:05 | |
| *** dklyle has quit IRC | 21:05 | |
| *** kevinbenton has quit IRC | 21:05 | |
| *** saper has quit IRC | 21:05 | |
| *** isviridov has quit IRC | 21:05 | |
| *** hemnafk has quit IRC | 21:05 | |
| *** fbo has quit IRC | 21:05 | |
| *** ChanServ has quit IRC | 21:05 | |
| *** morganfainberg has quit IRC | 21:05 | |
| *** Ryan_Lane has quit IRC | 21:05 | |
| *** mfer has quit IRC | 21:05 | |
| *** spredzy has quit IRC | 21:05 | |
| *** pcrews has quit IRC | 21:05 | |
| *** pdmars has quit IRC | 21:05 | |
| *** Shrews has quit IRC | 21:05 | |
| *** geekinutah has quit IRC | 21:05 | |
| *** lcostantino has quit IRC | 21:05 | |
| *** openstackgerrit has quit IRC | 21:05 | |
| *** dolphm_503 has quit IRC | 21:05 | |
| *** flaper87|afk has quit IRC | 21:05 | |
| *** nijaba has quit IRC | 21:05 | |
| *** mkoderer has quit IRC | 21:05 | |
| *** pleia2 has quit IRC | 21:05 | |
| *** EmilienM has quit IRC | 21:05 | |
| *** Alex_Gaynor has quit IRC | 21:05 | |
| *** spiffxp_ has quit IRC | 21:05 | |
| *** dachary has quit IRC | 21:05 | |
| *** westmaas has quit IRC | 21:05 | |
| *** saschpe_ has quit IRC | 21:05 | |
| *** dansmith has quit IRC | 21:05 | |
| *** dhellmann has quit IRC | 21:05 | |
| *** jgrimm has quit IRC | 21:05 | |
| *** virmitio has quit IRC | 21:05 | |
| *** salv-orlando has quit IRC | 21:05 | |
| *** lttrl has quit IRC | 21:05 | |
| *** dkehn_ has quit IRC | 21:05 | |
| *** guitarzan has quit IRC | 21:05 | |
| *** dkehn has quit IRC | 21:05 | |
| *** hemna has quit IRC | 21:05 | |
| *** Xurong has quit IRC | 21:05 | |
| *** lifeless has quit IRC | 21:05 | |
| *** SergeyLukjanov has quit IRC | 21:05 | |
| *** ruhe has quit IRC | 21:05 | |
| *** katyafervent_awa has quit IRC | 21:05 | |
| *** notmyname has quit IRC | 21:05 | |
| *** AaronGr has quit IRC | 21:05 | |
| *** mattoliverau has quit IRC | 21:05 | |
| *** ewindisch has quit IRC | 21:05 | |
| *** dtroyer has quit IRC | 21:05 | |
| *** Daviey has quit IRC | 21:05 | |
| *** hub_cap has quit IRC | 21:05 | |
| *** greghaynes has quit IRC | 21:05 | |
| *** olaph has quit IRC | 21:05 | |
| *** hdd has quit IRC | 21:05 | |
| *** rwsu has quit IRC | 21:05 | |
| *** antonym has quit IRC | 21:05 | |
| *** mancdaz has quit IRC | 21:05 | |
| *** rahmu has quit IRC | 21:05 | |
| *** BobBall has quit IRC | 21:05 | |
| *** mrda has quit IRC | 21:05 | |
| *** bogdando has quit IRC | 21:05 | |
| *** sdake has quit IRC | 21:05 | |
| *** shardy_afk has quit IRC | 21:05 | |
| *** jamielennox|away has quit IRC | 21:05 | |
| *** brosenberg has quit IRC | 21:05 | |
| *** mika has quit IRC | 21:05 | |
| *** NobodyCam has quit IRC | 21:05 | |
| *** devananda has quit IRC | 21:05 | |
| *** zaro has quit IRC | 21:05 | |
| *** Ng has quit IRC | 21:05 | |
| *** SlickNik has quit IRC | 21:05 | |
| *** jbryce has joined #openstack-infra | 21:08 | |
| *** fungi has joined #openstack-infra | 21:08 | |
| *** jlk has joined #openstack-infra | 21:08 | |
| *** StevenK has joined #openstack-infra | 21:08 | |
| *** wendar has joined #openstack-infra | 21:08 | |
| *** dangers_away has joined #openstack-infra | 21:08 | |
| *** msuriar has joined #openstack-infra | 21:08 | |
| *** ttx has joined #openstack-infra | 21:08 | |
| *** tristanC has joined #openstack-infra | 21:08 | |
| *** gaelL has joined #openstack-infra | 21:08 | |
| *** bnemec has joined #openstack-infra | 21:08 | |
| *** plomakin has joined #openstack-infra | 21:08 | |
| *** mtreinish has joined #openstack-infra | 21:08 | |
| *** mikal has joined #openstack-infra | 21:08 | |
| *** mattymo has joined #openstack-infra | 21:08 | |
| *** dragondm has joined #openstack-infra | 21:08 | |
| *** annegentle has joined #openstack-infra | 21:08 | |
| *** EntropyWorks has joined #openstack-infra | 21:08 | |
| *** tteggel has joined #openstack-infra | 21:08 | |
| *** cyeoh has joined #openstack-infra | 21:08 | |
| *** pvo has joined #openstack-infra | 21:08 | |
| *** stevebaker has joined #openstack-infra | 21:08 | |
| *** rainya has joined #openstack-infra | 21:08 | |
| *** Guest14741 has joined #openstack-infra | 21:08 | |
| *** adam_g has joined #openstack-infra | 21:08 | |
| *** russellb has joined #openstack-infra | 21:08 | |
| *** markmcclain has joined #openstack-infra | 21:08 | |
| *** melwitt has joined #openstack-infra | 21:08 | |
| *** NikitaKonovalov has joined #openstack-infra | 21:08 | |
| *** SlickNik has joined #openstack-infra | 21:08 | |
| *** mrda has joined #openstack-infra | 21:08 | |
| *** BobBall has joined #openstack-infra | 21:08 | |
| *** rahmu has joined #openstack-infra | 21:08 | |
| *** Ng has joined #openstack-infra | 21:08 | |
| *** hub_cap has joined #openstack-infra | 21:08 | |
| *** Daviey has joined #openstack-infra | 21:08 | |
| *** zaro has joined #openstack-infra | 21:08 | |
| *** devananda has joined #openstack-infra | 21:08 | |
| *** NobodyCam has joined #openstack-infra | 21:08 | |
| *** mika has joined #openstack-infra | 21:08 | |
| *** jamielennox|away has joined #openstack-infra | 21:08 | |
| *** shardy_afk has joined #openstack-infra | 21:08 | |
| *** dansmith has joined #openstack-infra | 21:08 | |
| *** ChanServ has joined #openstack-infra | 21:08 | |
| *** saschpe_ has joined #openstack-infra | 21:08 | |
| *** westmaas has joined #openstack-infra | 21:08 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 21:08 | |
| *** spiffxp_ has joined #openstack-infra | 21:08 | |
| *** dachary has joined #openstack-infra | 21:08 | |
| *** fbo has joined #openstack-infra | 21:08 | |
| *** mancdaz has joined #openstack-infra | 21:08 | |
| *** dtroyer has joined #openstack-infra | 21:08 | |
| *** mugsie has joined #openstack-infra | 21:08 | |
| *** GheRivero has joined #openstack-infra | 21:08 | |
| *** mattoliverau has joined #openstack-infra | 21:08 | |
| *** ewindisch has joined #openstack-infra | 21:08 | |
| *** Alex_Gaynor has joined #openstack-infra | 21:08 | |
| *** EmilienM has joined #openstack-infra | 21:08 | |
| *** isviridov has joined #openstack-infra | 21:08 | |
| *** AaronGr has joined #openstack-infra | 21:08 | |
| *** mordred has joined #openstack-infra | 21:08 | |
| *** gema has joined #openstack-infra | 21:08 | |
| *** davidlenwell has joined #openstack-infra | 21:08 | |
| *** esmute has joined #openstack-infra | 21:08 | |
| *** kmartin has joined #openstack-infra | 21:08 | |
| *** ilyashakhat_ has joined #openstack-infra | 21:08 | |
| *** saper has joined #openstack-infra | 21:08 | |
| *** vladan has joined #openstack-infra | 21:08 | |
| *** kevinbenton has joined #openstack-infra | 21:08 | |
| *** gothicmindfood has joined #openstack-infra | 21:08 | |
| *** dklyle has joined #openstack-infra | 21:08 | |
| *** ogelbukh has joined #openstack-infra | 21:08 | |
| *** cp16net has joined #openstack-infra | 21:08 | |
| *** pleia2 has joined #openstack-infra | 21:08 | |
| *** notmyname has joined #openstack-infra | 21:08 | |
| *** aude has joined #openstack-infra | 21:08 | |
| *** jcooley has joined #openstack-infra | 21:08 | |
| *** antonym has joined #openstack-infra | 21:08 | |
| *** HenryG has joined #openstack-infra | 21:08 | |
| *** morganfainberg has joined #openstack-infra | 21:08 | |
| *** mkoderer has joined #openstack-infra | 21:08 | |
| *** jang has joined #openstack-infra | 21:08 | |
| *** hemnafk has joined #openstack-infra | 21:08 | |
| *** mburned has joined #openstack-infra | 21:08 | |
| *** nibalizer has joined #openstack-infra | 21:08 | |
| *** katyafervent_awa has joined #openstack-infra | 21:08 | |
| *** maurosr has joined #openstack-infra | 21:08 | |
| *** nijaba has joined #openstack-infra | 21:08 | |
| *** locke1051 has joined #openstack-infra | 21:08 | |
| *** chmouel has joined #openstack-infra | 21:08 | |
| *** ruhe has joined #openstack-infra | 21:08 | |
| *** flaper87|afk has joined #openstack-infra | 21:08 | |
| *** SergeyLukjanov has joined #openstack-infra | 21:08 | |
| *** vipul has joined #openstack-infra | 21:08 | |
| *** sdake has joined #openstack-infra | 21:08 | |
| *** samalba has joined #openstack-infra | 21:08 | |
| *** jooools has joined #openstack-infra | 21:08 | |
| *** Steap_ has joined #openstack-infra | 21:08 | |
| *** bogdando has joined #openstack-infra | 21:08 | |
| *** blamar has joined #openstack-infra | 21:08 | |
| *** rwsu has joined #openstack-infra | 21:08 | |
| *** hdd has joined #openstack-infra | 21:08 | |
| *** brosenberg has joined #openstack-infra | 21:08 | |
| *** lifeless has joined #openstack-infra | 21:08 | |
| *** michchap has joined #openstack-infra | 21:08 | |
| *** Xurong has joined #openstack-infra | 21:08 | |
| *** leifmadsen has joined #openstack-infra | 21:08 | |
| *** olaph has joined #openstack-infra | 21:08 | |
| *** homeless has joined #openstack-infra | 21:08 | |
| *** medieval1 has joined #openstack-infra | 21:08 | |
| *** hemna has joined #openstack-infra | 21:08 | |
| *** greghaynes has joined #openstack-infra | 21:08 | |
| *** gmoro has joined #openstack-infra | 21:08 | |
| *** mestery has joined #openstack-infra | 21:08 | |
| *** dkehn has joined #openstack-infra | 21:08 | |
| *** tian has joined #openstack-infra | 21:08 | |
| *** yamahata_ has joined #openstack-infra | 21:08 | |
| *** amotoki has joined #openstack-infra | 21:08 | |
| *** vishy has joined #openstack-infra | 21:08 | |
| *** yongli has joined #openstack-infra | 21:08 | |
| *** skraynev_afk has joined #openstack-infra | 21:08 | |
| *** hughsaunders has joined #openstack-infra | 21:08 | |
| *** belliott has joined #openstack-infra | 21:08 | |
| *** asadoughi has joined #openstack-infra | 21:08 | |
| *** koolhead17 has joined #openstack-infra | 21:08 | |
| *** openstackgerrit has joined #openstack-infra | 21:08 | |
| *** ihrachys has joined #openstack-infra | 21:08 | |
| *** guitarzan has joined #openstack-infra | 21:08 | |
| *** gpocentek has joined #openstack-infra | 21:08 | |
| *** jaypipes has joined #openstack-infra | 21:08 | |
| *** krtaylor has joined #openstack-infra | 21:08 | |
| *** dkehn_ has joined #openstack-infra | 21:08 | |
| *** dolphm has joined #openstack-infra | 21:08 | |
| *** lttrl has joined #openstack-infra | 21:08 | |
| *** jeckersb has joined #openstack-infra | 21:08 | |
| *** kiall has joined #openstack-infra | 21:08 | |
| *** miqui has joined #openstack-infra | 21:08 | |
| *** bradm has joined #openstack-infra | 21:08 | |
| *** rockstar has joined #openstack-infra | 21:08 | |
| *** SpamapS has joined #openstack-infra | 21:08 | |
| *** yolanda has joined #openstack-infra | 21:08 | |
| *** afazekas has joined #openstack-infra | 21:08 | |
| *** zhiyan has joined #openstack-infra | 21:08 | |
| *** salv-orlando has joined #openstack-infra | 21:08 | |
| *** Hefeweizen has joined #openstack-infra | 21:08 | |
| *** andre__ has joined #openstack-infra | 21:08 | |
| *** weshay has joined #openstack-infra | 21:08 | |
| *** lcostantino has joined #openstack-infra | 21:08 | |
| *** portante has joined #openstack-infra | 21:08 | |
| *** unicell1 has joined #openstack-infra | 21:08 | |
| *** Shrews has joined #openstack-infra | 21:08 | |
| *** matel has joined #openstack-infra | 21:08 | |
| *** milki has joined #openstack-infra | 21:08 | |
| *** DinaBelova has joined #openstack-infra | 21:08 | |
| *** jd__ has joined #openstack-infra | 21:08 | |
| *** ArxCruz has joined #openstack-infra | 21:08 | |
| *** julienvey has joined #openstack-infra | 21:08 | |
| *** primeministerp has joined #openstack-infra | 21:08 | |
| *** tru_tru has joined #openstack-infra | 21:08 | |
| *** juice has joined #openstack-infra | 21:08 | |
| *** dcramer_ has joined #openstack-infra | 21:08 | |
| *** jasond`` has joined #openstack-infra | 21:08 | |
| *** max_lobur_afk has joined #openstack-infra | 21:08 | |
| *** yamahata has joined #openstack-infra | 21:08 | |
| *** jpeeler has joined #openstack-infra | 21:08 | |
| *** jergerber has joined #openstack-infra | 21:08 | |
| *** ryanpetrello has joined #openstack-infra | 21:08 | |
| *** eharney has joined #openstack-infra | 21:08 | |
| *** jroovers has joined #openstack-infra | 21:08 | |
| *** dkliban has joined #openstack-infra | 21:08 | |
| *** pdmars has joined #openstack-infra | 21:08 | |
| *** wenlock has joined #openstack-infra | 21:08 | |
| *** boris-42_ has joined #openstack-infra | 21:08 | |
| *** sandywalsh_ has joined #openstack-infra | 21:08 | |
| *** dkranz has joined #openstack-infra | 21:08 | |
| *** virmitio has joined #openstack-infra | 21:08 | |
| *** changbl has joined #openstack-infra | 21:08 | |
| *** pcrews has joined #openstack-infra | 21:08 | |
| *** ravikumar_hp has joined #openstack-infra | 21:08 | |
| *** spredzy has joined #openstack-infra | 21:08 | |
| *** jgrimm has joined #openstack-infra | 21:08 | |
| *** mfer has joined #openstack-infra | 21:08 | |
| *** mdenny has joined #openstack-infra | 21:08 | |
| *** nicedice has joined #openstack-infra | 21:08 | |
| *** atiwari has joined #openstack-infra | 21:08 | |
| *** Ryan_Lane has joined #openstack-infra | 21:08 | |
| *** VijayT has joined #openstack-infra | 21:08 | |
| *** gokrokve has joined #openstack-infra | 21:08 | |
| *** esker has joined #openstack-infra | 21:08 | |
| *** CaptTofu has joined #openstack-infra | 21:08 | |
| *** moted has joined #openstack-infra | 21:08 | |
| *** dripton has joined #openstack-infra | 21:08 | |
| *** saju_m has joined #openstack-infra | 21:08 | |
| *** dhellmann has joined #openstack-infra | 21:08 | |
| *** afazekas_ has joined #openstack-infra | 21:08 | |
| *** sabari_ has joined #openstack-infra | 21:08 | |
| *** geekinutah has joined #openstack-infra | 21:08 | |
| *** marun has joined #openstack-infra | 21:08 | |
| *** kfox1111 has joined #openstack-infra | 21:08 | |
| *** banix has joined #openstack-infra | 21:08 | |
| *** khyati_ has joined #openstack-infra | 21:08 | |
| *** vkozhukalov has joined #openstack-infra | 21:08 | |
| *** thomasem has joined #openstack-infra | 21:08 | |
| *** _david_ has joined #openstack-infra | 21:08 | |
| *** hogepodge has joined #openstack-infra | 21:08 | |
| *** hashar has joined #openstack-infra | 21:08 | |
| *** relaxdiego has joined #openstack-infra | 21:08 | |
| *** gordc has joined #openstack-infra | 21:08 | |
| *** dpyzhov has joined #openstack-infra | 21:08 | |
| *** prad has joined #openstack-infra | 21:08 | |
| *** enikanorov_ has joined #openstack-infra | 21:08 | |
| *** masayuki_ has joined #openstack-infra | 21:08 | |
| *** branen_ has joined #openstack-infra | 21:08 | |
| *** pafuent has joined #openstack-infra | 21:08 | |
| *** alexpilotti has joined #openstack-infra | 21:08 | |
| *** jgriffit1 has joined #openstack-infra | 21:08 | |
| *** jhesketh_ has joined #openstack-infra | 21:08 | |
| *** beagles has joined #openstack-infra | 21:08 | |
| *** adarazs_ has joined #openstack-infra | 21:08 | |
| *** mrodden has joined #openstack-infra | 21:08 | |
| *** mriedem has joined #openstack-infra | 21:08 | |
| *** bknudson has joined #openstack-infra | 21:08 | |
| *** pabelanger has joined #openstack-infra | 21:08 | |
| *** dstufft has joined #openstack-infra | 21:08 | |
| *** luisg has joined #openstack-infra | 21:08 | |
| *** bookwar has joined #openstack-infra | 21:08 | |
| *** sdague has joined #openstack-infra | 21:08 | |
| *** oubiwann-home has joined #openstack-infra | 21:08 | |
| *** sbadia has joined #openstack-infra | 21:08 | |
| *** jeremyb has joined #openstack-infra | 21:08 | |
| *** jog0_ has joined #openstack-infra | 21:08 | |
| *** dstanek has joined #openstack-infra | 21:08 | |
| *** david-lyle has joined #openstack-infra | 21:08 | |
| *** julim_ has joined #openstack-infra | 21:08 | |
| *** uvirtbot has joined #openstack-infra | 21:08 | |
| *** jog0_ is now known as Guest45106 | 21:08 | |
| *** pafuent has left #openstack-infra | 21:08 | |
| morganfainberg | hm. | 21:08 |
| jeblair | there was a netsplit | 21:08 |
| morganfainberg | not sure if this is something we should be worried about https://review.openstack.org/#/c/72808/12/etc/keystone.conf.sample | 21:08 |
| morganfainberg | https://review.openstack.org/#/c/72808/12/etc/keystone.conf.sample | 21:08 |
| morganfainberg | erm | 21:08 |
| jeblair | 2014-02-20T20:57:56 <jeblair> too much to do while the system is down; i'm reverting that patch in production then i'll fix it locally and push it for review | 21:08 |
| jeblair | 2014-02-20T20:58:56 <jeblair> fungi, clarkb: we still need to do something about the false-ready nodes | 21:08 |
| jeblair | 2014-02-20T21:00:05 <jeblair> fungi, clarkb: i propose we nodepool delete every ready node | 21:08 |
| jeblair | 2014-02-20T21:00:26 <jeblair> there's only one running job right now. | 21:08 |
| jeblair | 2014-02-20T21:00:31 <jeblair> so it's a pretty minimal impact | 21:08 |
| jeblair | 2014-02-20T21:01:21 <jeblair> fungi, clarkb: if that works for you, i'll just 'update node set state=4' in the db. | 21:08 |
| jeblair | 2014-02-20T21:01:34 <jeblair> and the new cleanup stuff should handle the rest. | 21:08 |
| morganfainberg | it's saying "intraline difference not available due to server error" | 21:09 |
| jeblair | fungi, clarkb: since then i have done that | 21:09 |
| jeblair | morganfainberg: it's a little worrying. | 21:09 |
| jeblair | gerrit is not logging an error | 21:10 |
| morganfainberg | jeblair, yeah. | 21:10 |
| fungi | jeblair: ahh, thanks! i didn't realize you'd gone... thought you were still working on it | 21:10 |
| morganfainberg | jeblair, not sure if it's just because it's a massive change... or ... if there is some legit issue | 21:11 |
| morganfainberg | jeblair, anyway figured i'd pass the error on | 21:11 |
| jeblair | morganfainberg: ah found the error (thanks to stackoverflow) | 21:12 |
| jeblair | [2014-02-20 21:04:27,427] WARN com.google.gerrit.server.patch.IntraLineLoader : 5000 ms timeout reached for IntraLineDiff in project openstack/keystone on commit 5c1662f75bf8896e26659a0dde679a6837bc4e9f for path etc/keystone.conf.sample comparing de3a8a8821ef58f3e85a208baaeb9a6242cf7ebf..156be573328008453bf7c81854e93b454179b234. Killing IntraLineDiff-3 | 21:12 |
| jeblair | morganfainberg: so yeah, it's cause it's huge. | 21:12 |
| jeblair | morganfainberg: otherwise seems harmless | 21:12 |
| lifeless | jeblair: whats wrong with nodepool; do you want me to work on a patch ? | 21:13 |
| *** cadenzajon has joined #openstack-infra | 21:13 | |
| jeblair | lifeless: i'll get it; should be a simple fix. novaclient isn't thread safe so we need to run those new calls through the taskmanager too | 21:14 |
| morganfainberg | jeblair, ok :) | 21:14 |
| *** dims has joined #openstack-infra | 21:14 | |
| *** zul has joined #openstack-infra | 21:14 | |
| *** jesusaurus has joined #openstack-infra | 21:14 | |
| *** che-arne has joined #openstack-infra | 21:15 | |
| *** loquacities has joined #openstack-infra | 21:15 | |
| *** vrovachev has joined #openstack-infra | 21:15 | |
| *** exz has joined #openstack-infra | 21:15 | |
| *** bodepd has joined #openstack-infra | 21:15 | |
| *** sileht has joined #openstack-infra | 21:15 | |
| *** psedlak has joined #openstack-infra | 21:15 | |
| *** ekarlso has joined #openstack-infra | 21:15 | |
| *** Adri2000 has joined #openstack-infra | 21:15 | |
| *** akscram has joined #openstack-infra | 21:15 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for swift bug 1274828 https://review.openstack.org/75154 | 21:15 |
| uvirtbot | Launchpad bug 1274828 in swift "grenade tests fail with error trying to connect Account server" [Undecided,Confirmed] https://launchpad.net/bugs/1274828 | 21:15 |
| *** Guest45106 has quit IRC | 21:26 | |
| *** dstanek has quit IRC | 21:26 | |
| *** david-lyle has quit IRC | 21:26 | |
| *** julim_ has quit IRC | 21:26 | |
| *** uvirtbot has quit IRC | 21:26 | |
| *** che-arne has quit IRC | 21:26 | |
| *** loquacities has quit IRC | 21:26 | |
| *** vrovachev has quit IRC | 21:26 | |
| *** exz has quit IRC | 21:26 | |
| *** bodepd has quit IRC | 21:26 | |
| *** sileht has quit IRC | 21:26 | |
| *** psedlak has quit IRC | 21:26 | |
| *** ekarlso has quit IRC | 21:26 | |
| *** Adri2000 has quit IRC | 21:26 | |
| *** akscram has quit IRC | 21:26 | |
| *** melwitt has quit IRC | 21:26 | |
| *** markmcclain has quit IRC | 21:26 | |
| *** russellb has quit IRC | 21:26 | |
| *** adam_g has quit IRC | 21:26 | |
| *** Guest14741 has quit IRC | 21:26 | |
| *** stevebaker has quit IRC | 21:26 | |
| *** pvo has quit IRC | 21:26 | |
| *** rainya has quit IRC | 21:26 | |
| *** cyeoh has quit IRC | 21:26 | |
| *** tteggel has quit IRC | 21:26 | |
| *** EntropyWorks has quit IRC | 21:26 | |
| *** annegentle has quit IRC | 21:26 | |
| *** dragondm has quit IRC | 21:26 | |
| *** mattymo has quit IRC | 21:26 | |
| *** mikal has quit IRC | 21:26 | |
| *** mtreinish has quit IRC | 21:26 | |
| *** plomakin has quit IRC | 21:26 | |
| *** bnemec has quit IRC | 21:26 | |
| *** gaelL has quit IRC | 21:26 | |
| *** tristanC has quit IRC | 21:26 | |
| *** ttx has quit IRC | 21:26 | |
| *** msuriar has quit IRC | 21:26 | |
| *** dangers_away has quit IRC | 21:26 | |
| *** wendar has quit IRC | 21:26 | |
| *** StevenK has quit IRC | 21:26 | |
| *** jlk has quit IRC | 21:26 | |
| *** fungi has quit IRC | 21:26 | |
| *** jbryce has quit IRC | 21:26 | |
| *** mrodden has quit IRC | 21:26 | |
| *** mriedem has quit IRC | 21:26 | |
| *** bknudson has quit IRC | 21:26 | |
| *** pabelanger has quit IRC | 21:26 | |
| *** dstufft has quit IRC | 21:26 | |
| *** luisg has quit IRC | 21:26 | |
| *** bookwar has quit IRC | 21:26 | |
| *** sdague has quit IRC | 21:26 | |
| *** oubiwann-home has quit IRC | 21:26 | |
| *** sbadia has quit IRC | 21:26 | |
| *** jeremyb has quit IRC | 21:26 | |
| *** enikanorov_ has quit IRC | 21:26 | |
| *** hashar has quit IRC | 21:26 | |
| *** hogepodge has quit IRC | 21:26 | |
| *** kfox1111 has quit IRC | 21:26 | |
| *** sabari_ has quit IRC | 21:26 | |
| *** CaptTofu has quit IRC | 21:26 | |
| *** esker has quit IRC | 21:26 | |
| *** ravikumar_hp has quit IRC | 21:26 | |
| *** changbl has quit IRC | 21:26 | |
| *** sandywalsh_ has quit IRC | 21:26 | |
| *** jasond`` has quit IRC | 21:26 | |
| *** tru_tru has quit IRC | 21:26 | |
| *** zhiyan has quit IRC | 21:26 | |
| *** miqui has quit IRC | 21:26 | |
| *** jaypipes has quit IRC | 21:26 | |
| *** gmoro has quit IRC | 21:26 | |
| *** jesusaurus has quit IRC | 21:26 | |
| *** jgriffit1 has quit IRC | 21:26 | |
| *** branen_ has quit IRC | 21:26 | |
| *** khyati_ has quit IRC | 21:26 | |
| *** DinaBelova has quit IRC | 21:26 | |
| *** milki has quit IRC | 21:26 | |
| *** portante has quit IRC | 21:26 | |
| *** afazekas has quit IRC | 21:26 | |
| *** bradm has quit IRC | 21:26 | |
| *** ihrachys has quit IRC | 21:26 | |
| *** koolhead17 has quit IRC | 21:26 | |
| *** belliott has quit IRC | 21:26 | |
| *** tian has quit IRC | 21:26 | |
| *** jooools has quit IRC | 21:26 | |
| *** maurosr has quit IRC | 21:26 | |
| *** dims has quit IRC | 21:26 | |
| *** prad has quit IRC | 21:26 | |
| *** dpyzhov has quit IRC | 21:26 | |
| *** saju_m has quit IRC | 21:26 | |
| *** nicedice has quit IRC | 21:26 | |
| *** wenlock has quit IRC | 21:26 | |
| *** jergerber has quit IRC | 21:26 | |
| *** max_lobur_afk has quit IRC | 21:26 | |
| *** jd__ has quit IRC | 21:26 | |
| *** weshay has quit IRC | 21:26 | |
| *** SpamapS has quit IRC | 21:26 | |
| *** yongli has quit IRC | 21:26 | |
| *** amotoki has quit IRC | 21:26 | |
| *** homeless has quit IRC | 21:26 | |
| *** vipul has quit IRC | 21:26 | |
| *** locke1051 has quit IRC | 21:26 | |
| *** michchap has quit IRC | 21:26 | |
| *** nibalizer has quit IRC | 21:26 | |
| *** HenryG has quit IRC | 21:26 | |
| *** aude has quit IRC | 21:26 | |
| *** gothicmindfood has quit IRC | 21:26 | |
| *** vladan has quit IRC | 21:26 | |
| *** ilyashakhat_ has quit IRC | 21:26 | |
| *** gpocentek has quit IRC | 21:26 | |
| *** cadenzajon has quit IRC | 21:26 | |
| *** adarazs_ has quit IRC | 21:26 | |
| *** alexpilotti has quit IRC | 21:26 | |
| *** _david_ has quit IRC | 21:26 | |
| *** thomasem has quit IRC | 21:26 | |
| *** relaxdiego has quit IRC | 21:26 | |
| *** marun has quit IRC | 21:26 | |
| *** atiwari has quit IRC | 21:26 | |
| *** mdenny has quit IRC | 21:26 | |
| *** dkranz has quit IRC | 21:26 | |
| *** eharney has quit IRC | 21:26 | |
| *** ryanpetrello has quit IRC | 21:26 | |
| *** jpeeler has quit IRC | 21:26 | |
| *** yamahata has quit IRC | 21:26 | |
| *** dcramer_ has quit IRC | 21:26 | |
| *** juice has quit IRC | 21:26 | |
| *** ArxCruz has quit IRC | 21:26 | |
| *** kiall has quit IRC | 21:26 | |
| *** skraynev_afk has quit IRC | 21:26 | |
| *** yamahata_ has quit IRC | 21:26 | |
| *** mestery has quit IRC | 21:26 | |
| *** blamar has quit IRC | 21:26 | |
| *** Steap_ has quit IRC | 21:26 | |
| *** samalba has quit IRC | 21:26 | |
| *** chmouel has quit IRC | 21:26 | |
| *** mburned has quit IRC | 21:26 | |
| *** jcooley has quit IRC | 21:26 | |
| *** ogelbukh has quit IRC | 21:26 | |
| *** kmartin has quit IRC | 21:26 | |
| *** jang has quit IRC | 21:26 | |
| *** jhesketh_ has quit IRC | 21:26 | |
| *** vkozhukalov has quit IRC | 21:26 | |
| *** afazekas_ has quit IRC | 21:26 | |
| *** dripton has quit IRC | 21:26 | |
| *** moted has quit IRC | 21:26 | |
| *** gokrokve has quit IRC | 21:26 | |
| *** VijayT has quit IRC | 21:26 | |
| *** primeministerp has quit IRC | 21:26 | |
| *** julienvey has quit IRC | 21:26 | |
| *** unicell1 has quit IRC | 21:26 | |
| *** andre__ has quit IRC | 21:26 | |
| *** Hefeweizen has quit IRC | 21:26 | |
| *** asadoughi has quit IRC | 21:26 | |
| *** medieval1 has quit IRC | 21:26 | |
| *** esmute has quit IRC | 21:26 | |
| *** davidlenwell has quit IRC | 21:26 | |
| *** gema has quit IRC | 21:26 | |
| *** mordred has quit IRC | 21:26 | |
| *** GheRivero has quit IRC | 21:26 | |
| *** mugsie has quit IRC | 21:26 | |
| *** zul has quit IRC | 21:26 | |
| *** beagles has quit IRC | 21:26 | |
| *** banix has quit IRC | 21:26 | |
| *** boris-42_ has quit IRC | 21:26 | |
| *** dkliban has quit IRC | 21:26 | |
| *** jroovers has quit IRC | 21:26 | |
| *** gordc has quit IRC | 21:26 | |
| *** matel has quit IRC | 21:26 | |
| *** yolanda has quit IRC | 21:26 | |
| *** rockstar has quit IRC | 21:26 | |
| *** jeckersb has quit IRC | 21:26 | |
| *** krtaylor has quit IRC | 21:26 | |
| *** hughsaunders has quit IRC | 21:26 | |
| *** vishy has quit IRC | 21:26 | |
| *** leifmadsen has quit IRC | 21:26 | |
| *** cp16net has quit IRC | 21:26 | |
| *** dklyle has quit IRC | 21:26 | |
| *** kevinbenton has quit IRC | 21:26 | |
| *** saper has quit IRC | 21:26 | |
| *** isviridov has quit IRC | 21:26 | |
| *** hemnafk has quit IRC | 21:26 | |
| *** fbo has quit IRC | 21:26 | |
| *** ChanServ has quit IRC | 21:26 | |
| *** morganfainberg has quit IRC | 21:26 | |
| *** NikitaKonovalov has quit IRC | 21:26 | |
| *** masayuki_ has quit IRC | 21:26 | |
| *** Ryan_Lane has quit IRC | 21:26 | |
| *** mfer has quit IRC | 21:26 | |
| *** spredzy has quit IRC | 21:26 | |
| *** pcrews has quit IRC | 21:26 | |
| *** pdmars has quit IRC | 21:26 | |
| *** Shrews has quit IRC | 21:26 | |
| *** geekinutah has quit IRC | 21:26 | |
| *** lcostantino has quit IRC | 21:26 | |
| *** openstackgerrit has quit IRC | 21:26 | |
| *** dolphm has quit IRC | 21:26 | |
| *** flaper87|afk has quit IRC | 21:26 | |
| *** nijaba has quit IRC | 21:26 | |
| *** mkoderer has quit IRC | 21:26 | |
| *** pleia2 has quit IRC | 21:26 | |
| *** EmilienM has quit IRC | 21:26 | |
| *** Alex_Gaynor has quit IRC | 21:26 | |
| *** spiffxp_ has quit IRC | 21:26 | |
| *** dachary has quit IRC | 21:26 | |
| *** westmaas has quit IRC | 21:26 | |
| *** saschpe_ has quit IRC | 21:26 | |
| *** dansmith has quit IRC | 21:26 | |
| *** dhellmann has quit IRC | 21:26 | |
| *** jgrimm has quit IRC | 21:26 | |
| *** virmitio has quit IRC | 21:26 | |
| *** salv-orlando has quit IRC | 21:26 | |
| *** lttrl has quit IRC | 21:26 | |
| *** dkehn_ has quit IRC | 21:26 | |
| *** guitarzan has quit IRC | 21:26 | |
| *** dkehn has quit IRC | 21:26 | |
| *** hemna has quit IRC | 21:26 | |
| *** Xurong has quit IRC | 21:26 | |
| *** lifeless has quit IRC | 21:26 | |
| *** SergeyLukjanov has quit IRC | 21:26 | |
| *** ruhe has quit IRC | 21:26 | |
| *** katyafervent_awa has quit IRC | 21:26 | |
| *** notmyname has quit IRC | 21:26 | |
| *** AaronGr has quit IRC | 21:26 | |
| *** mattoliverau has quit IRC | 21:26 | |
| *** ewindisch has quit IRC | 21:26 | |
| *** dtroyer has quit IRC | 21:26 | |
| *** Daviey has quit IRC | 21:26 | |
| *** hub_cap has quit IRC | 21:26 | |
| *** greghaynes has quit IRC | 21:26 | |
| *** olaph has quit IRC | 21:26 | |
| *** hdd has quit IRC | 21:26 | |
| *** rwsu has quit IRC | 21:26 | |
| *** antonym has quit IRC | 21:26 | |
| *** mancdaz has quit IRC | 21:26 | |
| *** rahmu has quit IRC | 21:26 | |
| *** BobBall has quit IRC | 21:26 | |
| *** mrda has quit IRC | 21:26 | |
| *** bogdando has quit IRC | 21:26 | |
| *** sdake has quit IRC | 21:26 | |
| *** shardy_afk has quit IRC | 21:26 | |
| *** jamielennox|away has quit IRC | 21:26 | |
| *** brosenberg has quit IRC | 21:26 | |
| *** mika has quit IRC | 21:26 | |
| *** NobodyCam has quit IRC | 21:26 | |
| *** devananda has quit IRC | 21:26 | |
| *** zaro has quit IRC | 21:26 | |
| *** Ng has quit IRC | 21:26 | |
| *** SlickNik has quit IRC | 21:26 | |
| *** niska has quit IRC | 21:26 | |
| *** jamespage has quit IRC | 21:28 | |
| *** sarob has joined #openstack-infra | 21:31 | |
| *** pmathews1 has joined #openstack-infra | 21:31 | |
| *** Hunner has joined #openstack-infra | 21:31 | |
| *** clarkb has joined #openstack-infra | 21:31 | |
| jeblair | i've stopped nodepool to investigate another error | 21:40 |
| *** sarob has quit IRC | 21:44 | |
| *** pmathews1 has quit IRC | 21:44 | |
| *** Hunner has quit IRC | 21:44 | |
| *** clarkb has quit IRC | 21:44 | |
| *** ttx has joined #openstack-infra | 21:50 | |
| *** markmcclain has joined #openstack-infra | 21:50 | |
| *** jlk has joined #openstack-infra | 21:50 | |
| *** russellb has joined #openstack-infra | 21:50 | |
| *** wendar has joined #openstack-infra | 21:50 | |
| *** lillie has joined #openstack-infra | 21:50 | |
| *** annegentle has joined #openstack-infra | 21:50 | |
| *** rainya has joined #openstack-infra | 21:50 | |
| *** jbryce has joined #openstack-infra | 21:50 | |
| *** pvo has joined #openstack-infra | 21:50 | |
| *** adam_g has joined #openstack-infra | 21:50 | |
| *** sarob has joined #openstack-infra | 21:50 | |
| *** Hunner has joined #openstack-infra | 21:50 | |
| *** EmilienM has joined #openstack-infra | 21:58 | |
| *** pleia2 has joined #openstack-infra | 22:06 | |
| *** mkoderer has joined #openstack-infra | 22:06 | |
| *** spiffxp has joined #openstack-infra | 22:06 | |
| *** zehicle_at_dell has joined #openstack-infra | 22:06 | |
| *** dansmith has joined #openstack-infra | 22:06 | |
| *** NikitaKonovalov has joined #openstack-infra | 22:06 | |
| *** oubiwann has joined #openstack-infra | 22:06 | |
| *** spredzy has joined #openstack-infra | 22:06 | |
| *** dachary has joined #openstack-infra | 22:06 | |
| *** dolphm has joined #openstack-infra | 22:06 | |
| *** cyeoh has joined #openstack-infra | 22:06 | |
| *** tjones has joined #openstack-infra | 22:06 | |
| *** westmaas_ has joined #openstack-infra | 22:06 | |
| *** saschpe has joined #openstack-infra | 22:06 | |
| *** nijaba_ has joined #openstack-infra | 22:06 | |
| *** Shrews_ has joined #openstack-infra | 22:06 | |
| *** pcrews_ has joined #openstack-infra | 22:06 | |
| *** geekinutah1 has joined #openstack-infra | 22:06 | |
| *** dangers_` has joined #openstack-infra | 22:06 | |
| *** dpyzhov has joined #openstack-infra | 22:06 | |
| *** flaper87 has joined #openstack-infra | 22:06 | |
| *** afrittoli has joined #openstack-infra | 22:06 | |
| *** pdmars_ has joined #openstack-infra | 22:06 | |
| *** Ryan_Lane1 has joined #openstack-infra | 22:06 | |
| *** masayukig has joined #openstack-infra | 22:06 | |
| *** 45PAAH8X1 has joined #openstack-infra | 22:06 | |
| *** gyee has joined #openstack-infra | 22:06 | |
| *** fungi has joined #openstack-infra | 22:06 | |
| *** mattymo has joined #openstack-infra | 22:06 | |
| *** tristanC has joined #openstack-infra | 22:06 | |
| *** stevebaker has joined #openstack-infra | 22:06 | |
| *** dragondm has joined #openstack-infra | 22:06 | |
| *** harlowja has joined #openstack-infra | 22:06 | |
| *** dizquierdo has joined #openstack-infra | 22:06 | |
| *** ianw has joined #openstack-infra | 22:06 | |
| *** jamespag` has joined #openstack-infra | 22:06 | |
| *** plomakin_ has joined #openstack-infra | 22:06 | |
| *** cadenzajon has joined #openstack-infra | 22:06 | |
| *** jcooley_ has joined #openstack-infra | 22:06 | |
| *** pmathews1 has joined #openstack-infra | 22:06 | |
| *** jamielennox|away has joined #openstack-infra | 22:06 | |
| *** mtreinish has joined #openstack-infra | 22:06 | |
| *** mika has joined #openstack-infra | 22:06 | |
| *** gaelL_ has joined #openstack-infra | 22:06 | |
| *** msuriar_ has joined #openstack-infra | 22:06 | |
| *** mikal_ has joined #openstack-infra | 22:06 | |
| *** tteggel has joined #openstack-infra | 22:06 | |
| *** StevenK_ has joined #openstack-infra | 22:06 | |
| *** bnemec_ has joined #openstack-infra | 22:06 | |
| *** niska` has joined #openstack-infra | 22:06 | |
| *** jog0 has joined #openstack-infra | 22:06 | |
| *** EntropyWorks has joined #openstack-infra | 22:06 | |
| *** david_lyle has joined #openstack-infra | 22:06 | |
| *** dstanek has joined #openstack-infra | 22:06 | |
| *** weshay has joined #openstack-infra | 22:06 | |
| *** clarkb has joined #openstack-infra | 22:06 | |
| *** akscram has joined #openstack-infra | 22:06 | |
| *** Adri2000 has joined #openstack-infra | 22:06 | |
| *** ekarlso has joined #openstack-infra | 22:06 | |
| *** psedlak has joined #openstack-infra | 22:06 | |
| *** sileht has joined #openstack-infra | 22:06 | |
| *** bodepd has joined #openstack-infra | 22:06 | |
| *** exz has joined #openstack-infra | 22:06 | |
| *** vrovachev has joined #openstack-infra | 22:06 | |
| *** loquacities has joined #openstack-infra | 22:06 | |
| *** che-arne has joined #openstack-infra | 22:06 | |
| *** jesusaurus has joined #openstack-infra | 22:06 | |
| *** zul has joined #openstack-infra | 22:06 | |
| *** dims has joined #openstack-infra | 22:06 | |
| *** SlickNik has joined #openstack-infra | 22:06 | |
| *** mrda has joined #openstack-infra | 22:06 | |
| *** BobBall has joined #openstack-infra | 22:06 | |
| *** rahmu has joined #openstack-infra | 22:06 | |
| *** Ng has joined #openstack-infra | 22:06 | |
| *** hub_cap has joined #openstack-infra | 22:06 | |
| *** Daviey has joined #openstack-infra | 22:06 | |
| *** zaro has joined #openstack-infra | 22:06 | |
| *** devananda has joined #openstack-infra | 22:06 | |
| *** NobodyCam has joined #openstack-infra | 22:06 | |
| *** shardy_afk has joined #openstack-infra | 22:06 | |
| *** ChanServ has joined #openstack-infra | 22:06 | |
| *** fbo has joined #openstack-infra | 22:06 | |
| *** mancdaz has joined #openstack-infra | 22:06 | |
| *** dtroyer has joined #openstack-infra | 22:06 | |
| *** mugsie has joined #openstack-infra | 22:06 | |
| *** GheRivero has joined #openstack-infra | 22:06 | |
| *** mattoliverau has joined #openstack-infra | 22:06 | |
| *** ewindisch has joined #openstack-infra | 22:06 | |
| *** isviridov has joined #openstack-infra | 22:06 | |
| *** AaronGr has joined #openstack-infra | 22:06 | |
| *** mordred has joined #openstack-infra | 22:06 | |
| *** gema has joined #openstack-infra | 22:06 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 22:06 | |
| *** davidlenwell has joined #openstack-infra | 22:06 | |
| *** esmute has joined #openstack-infra | 22:06 | |
| *** kmartin has joined #openstack-infra | 22:06 | |
| *** ilyashakhat_ has joined #openstack-infra | 22:06 | |
| *** saper has joined #openstack-infra | 22:06 | |
| *** vladan has joined #openstack-infra | 22:06 | |
| *** kevinbenton has joined #openstack-infra | 22:06 | |
| *** gothicmindfood has joined #openstack-infra | 22:06 | |
| *** dklyle has joined #openstack-infra | 22:06 | |
| *** ogelbukh has joined #openstack-infra | 22:06 | |
| *** cp16net has joined #openstack-infra | 22:06 | |
| *** notmyname has joined #openstack-infra | 22:06 | |
| *** aude has joined #openstack-infra | 22:06 | |
| *** jcooley has joined #openstack-infra | 22:06 | |
| *** antonym has joined #openstack-infra | 22:06 | |
| *** HenryG has joined #openstack-infra | 22:06 | |
| *** morganfainberg has joined #openstack-infra | 22:06 | |
| *** jang has joined #openstack-infra | 22:06 | |
| *** hemnafk has joined #openstack-infra | 22:06 | |
| *** mburned has joined #openstack-infra | 22:06 | |
| *** nibalizer has joined #openstack-infra | 22:06 | |
| *** katyafervent_awa has joined #openstack-infra | 22:06 | |
| *** maurosr has joined #openstack-infra | 22:06 | |
| *** locke1051 has joined #openstack-infra | 22:06 | |
| *** chmouel has joined #openstack-infra | 22:06 | |
| *** ruhe has joined #openstack-infra | 22:06 | |
| *** SergeyLukjanov has joined #openstack-infra | 22:06 | |
| *** vipul has joined #openstack-infra | 22:06 | |
| *** sdake has joined #openstack-infra | 22:06 | |
| *** samalba has joined #openstack-infra | 22:06 | |
| *** jooools has joined #openstack-infra | 22:06 | |
| *** Steap_ has joined #openstack-infra | 22:06 | |
| *** bogdando has joined #openstack-infra | 22:06 | |
| *** blamar has joined #openstack-infra | 22:06 | |
| *** rwsu has joined #openstack-infra | 22:06 | |
| *** hdd has joined #openstack-infra | 22:06 | |
| *** brosenberg has joined #openstack-infra | 22:06 | |
| *** lifeless has joined #openstack-infra | 22:06 | |
| *** michchap has joined #openstack-infra | 22:06 | |
| *** Xurong has joined #openstack-infra | 22:06 | |
| *** leifmadsen has joined #openstack-infra | 22:06 | |
| *** olaph has joined #openstack-infra | 22:06 | |
| *** homeless has joined #openstack-infra | 22:06 | |
| *** hemna has joined #openstack-infra | 22:06 | |
| *** greghaynes has joined #openstack-infra | 22:06 | |
| *** gmoro has joined #openstack-infra | 22:06 | |
| *** mestery has joined #openstack-infra | 22:06 | |
| *** dkehn has joined #openstack-infra | 22:06 | |
| *** tian has joined #openstack-infra | 22:06 | |
| *** yamahata_ has joined #openstack-infra | 22:06 | |
| *** amotoki has joined #openstack-infra | 22:06 | |
| *** vishy has joined #openstack-infra | 22:06 | |
| *** yongli has joined #openstack-infra | 22:06 | |
| *** skraynev_afk has joined #openstack-infra | 22:06 | |
| *** hughsaunders has joined #openstack-infra | 22:06 | |
| *** asadoughi has joined #openstack-infra | 22:06 | |
| *** koolhead17 has joined #openstack-infra | 22:06 | |
| *** ihrachys has joined #openstack-infra | 22:06 | |
| *** guitarzan has joined #openstack-infra | 22:06 | |
| *** gpocentek has joined #openstack-infra | 22:06 | |
| *** krtaylor has joined #openstack-infra | 22:06 | |
| *** dkehn_ has joined #openstack-infra | 22:06 | |
| *** lttrl has joined #openstack-infra | 22:06 | |
| *** jeckersb has joined #openstack-infra | 22:06 | |
| *** kiall has joined #openstack-infra | 22:06 | |
| *** miqui has joined #openstack-infra | 22:06 | |
| *** bradm has joined #openstack-infra | 22:06 | |
| *** rockstar has joined #openstack-infra | 22:06 | |
| *** afazekas has joined #openstack-infra | 22:06 | |
| *** zhiyan_ has joined #openstack-infra | 22:06 | |
| *** salv-orlando has joined #openstack-infra | 22:06 | |
| *** Hefeweizen has joined #openstack-infra | 22:06 | |
| *** andre__ has joined #openstack-infra | 22:06 | |
| *** portante has joined #openstack-infra | 22:06 | |
| *** unicell1 has joined #openstack-infra | 22:06 | |
| *** matel has joined #openstack-infra | 22:06 | |
| *** milki has joined #openstack-infra | 22:06 | |
| *** DinaBelova_ has joined #openstack-infra | 22:06 | |
| *** jd__ has joined #openstack-infra | 22:06 | |
| *** ArxCruz has joined #openstack-infra | 22:06 | |
| *** julienvey has joined #openstack-infra | 22:06 | |
| *** primeministerp has joined #openstack-infra | 22:06 | |
| *** tru_tru has joined #openstack-infra | 22:06 | |
| *** juice has joined #openstack-infra | 22:06 | |
| *** dcramer_ has joined #openstack-infra | 22:06 | |
| *** jasond`` has joined #openstack-infra | 22:06 | |
| *** max_lobur_afk has joined #openstack-infra | 22:06 | |
| *** yamahata has joined #openstack-infra | 22:06 | |
| *** jpeeler has joined #openstack-infra | 22:06 | |
| *** jergerber has joined #openstack-infra | 22:06 | |
| *** ryanpetrello has joined #openstack-infra | 22:06 | |
| *** eharney has joined #openstack-infra | 22:06 | |
| *** jroovers has joined #openstack-infra | 22:06 | |
| *** wenlock has joined #openstack-infra | 22:06 | |
| *** boris-42_ has joined #openstack-infra | 22:06 | |
| *** sandywalsh_ has joined #openstack-infra | 22:06 | |
| *** dkranz has joined #openstack-infra | 22:06 | |
| *** virmitio has joined #openstack-infra | 22:06 | |
| *** changbl has joined #openstack-infra | 22:06 | |
| *** ravikumar_hp has joined #openstack-infra | 22:06 | |
| *** jgrimm has joined #openstack-infra | 22:06 | |
| *** mdenny has joined #openstack-infra | 22:06 | |
| *** nicedice has joined #openstack-infra | 22:06 | |
| *** atiwari has joined #openstack-infra | 22:06 | |
| *** VijayT has joined #openstack-infra | 22:06 | |
| *** esker has joined #openstack-infra | 22:06 | |
| *** CaptTofu has joined #openstack-infra | 22:06 | |
| *** moted has joined #openstack-infra | 22:06 | |
| *** dripton has joined #openstack-infra | 22:06 | |
| *** dhellmann has joined #openstack-infra | 22:06 | |
| *** afazekas_ has joined #openstack-infra | 22:06 | |
| *** sabari_ has joined #openstack-infra | 22:06 | |
| *** marun has joined #openstack-infra | 22:06 | |
| *** kfox1111 has joined #openstack-infra | 22:06 | |
| *** banix has joined #openstack-infra | 22:06 | |
| *** khyati_ has joined #openstack-infra | 22:06 | |
| *** vkozhukalov has joined #openstack-infra | 22:06 | |
| *** hogepodge has joined #openstack-infra | 22:06 | |
| *** hashar has joined #openstack-infra | 22:06 | |
| *** relaxdiego has joined #openstack-infra | 22:06 | |
| *** prad has joined #openstack-infra | 22:06 | |
| *** enikanorov_ has joined #openstack-infra | 22:06 | |
| *** branen_ has joined #openstack-infra | 22:06 | |
| *** alexpilotti has joined #openstack-infra | 22:06 | |
| *** jgriffit1 has joined #openstack-infra | 22:06 | |
| *** jhesketh_ has joined #openstack-infra | 22:06 | |
| *** beagles has joined #openstack-infra | 22:06 | |
| *** adarazs_ has joined #openstack-infra | 22:06 | |
| *** mrodden has joined #openstack-infra | 22:06 | |
| *** pabelanger has joined #openstack-infra | 22:06 | |
| *** dstufft has joined #openstack-infra | 22:06 | |
| *** luisg has joined #openstack-infra | 22:06 | |
| *** bookwar has joined #openstack-infra | 22:06 | |
| *** sdague has joined #openstack-infra | 22:06 | |
| *** oubiwann-home has joined #openstack-infra | 22:06 | |
| *** sbadia has joined #openstack-infra | 22:06 | |
| *** jeremyb has joined #openstack-infra | 22:06 | |
| *** jeblair has quit IRC | 22:06 | |
| fungi | jog0: no idea if there's a bug open yet. i did see a couple failures along those lines, but no idea what could cause them. tox isn't finding the interpreter, and it's running on a centos box. not sure how tox can even run when there's no python interpreter | 22:06 |
| *** jeblair has joined #openstack-infra | 22:06 | |
| jeblair | fungi, clarkb: oh you're over here | 22:07 |
| *** forrest has joined #openstack-infra | 22:07 | |
| clarkb | jeblair: yes | 22:07 |
| jog0 | I'll ile a new bug | 22:07 |
| jeblair | eavesdrop is elsewhere | 22:07 |
| *** andreaf has joined #openstack-infra | 22:07 | |
| jeblair | what did i miss? | 22:07 |
| *** oubiwann has quit IRC | 22:07 | |
| *** forrest has quit IRC | 22:07 | |
| *** jaypipes has joined #openstack-infra | 22:07 | |
| jeblair | (i found another problem with nodepool and have hand-patched; will need to do some cleanup) | 22:07 |
| *** jeckersb is now known as jeckersb_gone | 22:07 | |
| *** lillie is now known as Guest38907 | 22:08 | |
| clarkb | jeblair: some interesting test failures | 22:08 |
| *** forrest has joined #openstack-infra | 22:08 | |
| fungi | jeblair: the openstack meetbot rejoined when you did, roughly | 22:08 |
| *** sabari_ is now known as sabari | 22:08 | |
| *** jgriffit1 is now known as jgriffith | 22:08 | |
| clarkb | I need to write up a thing about test matrixes | 22:08 |
| *** belliott has joined #openstack-infra | 22:08 | |
| *** david_lyle has quit IRC | 22:08 | |
| *** forrest has quit IRC | 22:08 | |
| *** Alex_Gaynor has joined #openstack-infra | 22:09 | |
| *** fcarpenter has joined #openstack-infra | 22:09 | |
| jeblair | jog0: so novaclient returns server ids as ints when talking to hpcloud, and strings when talking to rax | 22:09 |
| *** fcarpenter has quit IRC | 22:09 | |
| jeblair | jog0: that sound bug-like to you? | 22:09 |
| *** fcarpenter has joined #openstack-infra | 22:09 | |
| jog0 | WAT | 22:09 |
| jog0 | its a bug alright | 22:10 |
| *** fcarpenter has quit IRC | 22:10 | |
| *** bknudson has joined #openstack-infra | 22:10 | |
| jog0 | not sure where | 22:10 |
| fungi | a pretty awesome bug too | 22:10 |
| jog0 | have you compared the raw REST calls? | 22:10 |
| jog0 | I wonder what defcore thinks of this one haha | 22:10 |
| *** fcarpenter has joined #openstack-infra | 22:11 | |
| jeblair | jog0: no; have done very little with this so far | 22:11 |
| jeblair | jog0: (am fixing nodepool production probs) | 22:11 |
| fungi | jog0: you should ask. the refstack meeting is going on in #openstack-meeting-alt right now ;) | 22:11 |
| jog0 | jeblair: when you have some bandwidth I would be interested in helping you figure out where this bug belongs | 22:12 |
| jeblair | jog0: sounds good | 22:12 |
| jog0 | fungi: I should have brought a beer with me to the park (where I am working from) | 22:12 |
| jog0 | and jeblair, oh god sorry you have to deal with us | 22:13 |
| sdague | actually, jog0 if you have ids on both, it would be interesting to run one of the simple tempest tests against them to see. Given that we have our own client in tempest | 22:13 |
| *** relaxdiego has quit IRC | 22:13 | |
| jog0 | sdague: that would be interesting, oh wait I have access to both rax and hp | 22:14 |
| sdague | I only have rax id | 22:14 |
| sdague | right, I assumed :) | 22:14 |
| sdague | that's how I caught you in that trap.... | 22:14 |
| *** relaxdiego has joined #openstack-infra | 22:15 | |
| jog0 | id": 3585537 | 22:17 |
| jog0 | thats the json in hp | 22:17 |
| jog0 | an int | 22:17 |
| clarkb | I am going to reboot really quickly to pick up updates and remount filesystems with new options | 22:18 |
| fungi | clarkb: you could just feign another netsplit | 22:18 |
| *** dolphm is now known as dolphm_503 | 22:18 | |
| mordred | fungi: what's a thing I should run to make sure my fix script has made a node that can do the right thing? | 22:19 |
| jog0 | sdague: does rax work without racspace-novaclient? | 22:20 |
| jog0 | sdague: what does devstack do for id? | 22:20 |
| *** fcarpenter has left #openstack-infra | 22:20 | |
| *** jamespage_ has joined #openstack-infra | 22:20 | |
| jog0 | string or int? | 22:20 |
| mordred | jog0, jeblair: nova is diablo in hp | 22:20 |
| fungi | mordred: i think trying to tox -epypy any random project with working pypy tests should be sufficient. python-heatclient for example | 22:20 |
| *** sarob has quit IRC | 22:20 | |
| jog0 | mordred: both are API v2 | 22:20 |
| mordred | those are the server ids the cloud returns for hp cloud 1.0 | 22:20 |
| mordred | in 1.1, they are hex strings | 22:20 |
| jog0 | mordred: o_O | 22:21 |
| *** sarob has joined #openstack-infra | 22:21 | |
| mordred | so - just saying, I don't think it's a novaclient bug - I think it's a behavioral difference with a very old cloud | 22:21 |
| jog0 | me finds his hp cloud 1.1 creds | 22:21 |
| clarkb | and I am back | 22:21 |
| mordred | Alex_Gaynor: swiftclient has working pypy tests, yeah? | 22:22 |
| Alex_Gaynor | mordred: Uhh, I think so | 22:22 |
| jeblair | mordred: that's true if openstack wanted to make an api compatability breaking change when we went from ints to strings | 22:22 |
| jog0 | mordred: v13.5? | 22:22 |
| jeblair | mordred: if openstack said "they are opaque things that should be represented by strings" to start with, then it's a novaclient bug | 22:22 |
| anteaya | fungi: we lost about 100 nodes when we made the change on hp-az2? | 22:30 |
| anteaya | the graph appears to have about 100 nodes less than it did before | 22:32 |
| anteaya | dang top of the gate just failed | 22:32 |
| *** andreaf has quit IRC | 22:32 | |
| *** spiffxp has quit IRC | 22:32 | |
| *** nijaba_ has quit IRC | 22:32 | |
| *** Ryan_Lane1 has quit IRC | 22:32 | |
| *** 45PAAH8X1 has quit IRC | 22:32 | |
| *** clarkb has quit IRC | 22:32 | |
| *** Alex_Gaynor has quit IRC | 22:32 | |
| *** cyeoh has quit IRC | 22:32 | |
| *** dolphm_503 has quit IRC | 22:32 | |
| *** fungi has quit IRC | 22:32 | |
| *** StevenK_ has quit IRC | 22:32 | |
| *** che-arne has quit IRC | 22:32 | |
| *** loquacities has quit IRC | 22:32 | |
| *** vrovachev has quit IRC | 22:32 | |
| *** exz has quit IRC | 22:32 | |
| *** bodepd has quit IRC | 22:32 | |
| *** sileht has quit IRC | 22:32 | |
| *** psedlak has quit IRC | 22:32 | |
| *** ekarlso has quit IRC | 22:32 | |
| *** Adri2000 has quit IRC | 22:32 | |
| *** akscram has quit IRC | 22:32 | |
| *** bknudson has quit IRC | 22:32 | |
| *** mkoderer has quit IRC | 22:32 | |
| *** harlowja has quit IRC | 22:32 | |
| *** mrodden has quit IRC | 22:32 | |
| *** pabelanger has quit IRC | 22:32 | |
| *** dstufft has quit IRC | 22:32 | |
| *** luisg has quit IRC | 22:32 | |
| *** bookwar has quit IRC | 22:32 | |
| *** sdague has quit IRC | 22:32 | |
| *** oubiwann-home has quit IRC | 22:33 | |
| *** sbadia has quit IRC | 22:33 | |
| *** jeremyb has quit IRC | 22:33 | |
| *** dansmith has quit IRC | 22:33 | |
| *** pcrews_ has quit IRC | 22:33 | |
| *** mattymo has quit IRC | 22:33 | |
| *** bnemec_ has quit IRC | 22:33 | |
| *** niska` has quit IRC | 22:33 | |
| *** enikanorov_ has quit IRC | 22:33 | |
| *** hashar has quit IRC | 22:33 | |
| *** hogepodge has quit IRC | 22:33 | |
| *** kfox1111 has quit IRC | 22:33 | |
| *** sabari has quit IRC | 22:33 | |
| *** CaptTofu has quit IRC | 22:33 | |
| *** esker has quit IRC | 22:33 | |
| *** ravikumar_hp has quit IRC | 22:33 | |
| *** changbl has quit IRC | 22:33 | |
| *** sandywalsh_ has quit IRC | 22:33 | |
| *** jasond`` has quit IRC | 22:33 | |
| *** tru_tru has quit IRC | 22:33 | |
| *** zhiyan_ has quit IRC | 22:33 | |
| *** miqui has quit IRC | 22:33 | |
| *** gmoro has quit IRC | 22:33 | |
| *** pleia2 has quit IRC | 22:33 | |
| *** dachary has quit IRC | 22:33 | |
| *** dpyzhov has quit IRC | 22:33 | |
| *** flaper87 has quit IRC | 22:33 | |
| *** mika has quit IRC | 22:33 | |
| *** EntropyWorks has quit IRC | 22:33 | |
| *** jesusaurus has quit IRC | 22:33 | |
| *** jgriffith has quit IRC | 22:33 | |
| *** branen_ has quit IRC | 22:33 | |
| *** khyati_ has quit IRC | 22:33 | |
| *** DinaBelova_ has quit IRC | 22:33 | |
| *** milki has quit IRC | 22:33 | |
| *** portante has quit IRC | 22:33 | |
| *** afazekas has quit IRC | 22:33 | |
| *** bradm has quit IRC | 22:33 | |
| *** ihrachys has quit IRC | 22:33 | |
| *** koolhead17 has quit IRC | 22:33 | |
| *** tian has quit IRC | 22:33 | |
| *** jooools has quit IRC | 22:33 | |
| *** maurosr has quit IRC | 22:33 | |
| *** jeblair has quit IRC | 22:33 | |
| *** NikitaKonovalov has quit IRC | 22:33 | |
| *** Shrews_ has quit IRC | 22:33 | |
| *** afrittoli has quit IRC | 22:33 | |
| *** masayukig has quit IRC | 22:33 | |
| *** gyee has quit IRC | 22:33 | |
| *** mtreinish has quit IRC | 22:33 | |
| *** jog0 has quit IRC | 22:33 | |
| *** dims has quit IRC | 22:33 | |
| *** prad has quit IRC | 22:33 | |
| *** nicedice has quit IRC | 22:33 | |
| *** wenlock has quit IRC | 22:33 | |
| *** jergerber has quit IRC | 22:33 | |
| *** max_lobur_afk has quit IRC | 22:33 | |
| *** jd__ has quit IRC | 22:33 | |
| *** yongli has quit IRC | 22:33 | |
| *** amotoki has quit IRC | 22:33 | |
| *** homeless has quit IRC | 22:33 | |
| *** vipul has quit IRC | 22:33 | |
| *** locke1051 has quit IRC | 22:33 | |
| *** michchap has quit IRC | 22:33 | |
| *** nibalizer has quit IRC | 22:33 | |
| *** HenryG has quit IRC | 22:33 | |
| *** aude has quit IRC | 22:33 | |
| *** gothicmindfood has quit IRC | 22:33 | |
| *** vladan has quit IRC | 22:33 | |
| *** ilyashakhat_ has quit IRC | 22:33 | |
| *** gpocentek has quit IRC | 22:33 | |
| *** geekinutah1 has quit IRC | 22:33 | |
| *** pdmars_ has quit IRC | 22:33 | |
| *** dragondm has quit IRC | 22:33 | |
| *** plomakin_ has quit IRC | 22:33 | |
| *** cadenzajon has quit IRC | 22:33 | |
| *** pmathews1 has quit IRC | 22:33 | |
| *** adarazs_ has quit IRC | 22:33 | |
| *** alexpilotti has quit IRC | 22:33 | |
| *** marun has quit IRC | 22:33 | |
| *** atiwari has quit IRC | 22:33 | |
| *** mdenny has quit IRC | 22:33 | |
| *** dkranz has quit IRC | 22:33 | |
| *** eharney has quit IRC | 22:33 | |
| *** ryanpetrello has quit IRC | 22:33 | |
| *** jpeeler has quit IRC | 22:33 | |
| *** yamahata has quit IRC | 22:33 | |
| *** dcramer_ has quit IRC | 22:33 | |
| *** juice has quit IRC | 22:33 | |
| *** ArxCruz has quit IRC | 22:33 | |
| *** kiall has quit IRC | 22:33 | |
| *** skraynev_afk has quit IRC | 22:33 | |
| *** yamahata_ has quit IRC | 22:33 | |
| *** mestery has quit IRC | 22:33 | |
| *** blamar has quit IRC | 22:33 | |
| *** Steap_ has quit IRC | 22:33 | |
| *** samalba has quit IRC | 22:33 | |
| *** chmouel has quit IRC | 22:33 | |
| *** mburned has quit IRC | 22:33 | |
| *** jcooley has quit IRC | 22:33 | |
| *** ogelbukh has quit IRC | 22:33 | |
| *** kmartin has quit IRC | 22:33 | |
| *** jang has quit IRC | 22:33 | |
| *** belliott has quit IRC | 22:33 | |
| *** spredzy has quit IRC | 22:33 | |
| *** ianw has quit IRC | 22:33 | |
| *** jcooley_ has quit IRC | 22:33 | |
| *** jamielennox|away has quit IRC | 22:33 | |
| *** gaelL_ has quit IRC | 22:33 | |
| *** weshay has quit IRC | 22:33 | |
| *** jhesketh_ has quit IRC | 22:33 | |
| *** vkozhukalov has quit IRC | 22:33 | |
| *** afazekas_ has quit IRC | 22:33 | |
| *** dripton has quit IRC | 22:33 | |
| *** moted has quit IRC | 22:33 | |
| *** VijayT has quit IRC | 22:33 | |
| *** primeministerp has quit IRC | 22:33 | |
| *** julienvey has quit IRC | 22:33 | |
| *** unicell1 has quit IRC | 22:33 | |
| *** andre__ has quit IRC | 22:33 | |
| *** Hefeweizen has quit IRC | 22:33 | |
| *** asadoughi has quit IRC | 22:33 | |
| *** esmute has quit IRC | 22:33 | |
| *** davidlenwell has quit IRC | 22:33 | |
| *** gema has quit IRC | 22:33 | |
| *** mordred has quit IRC | 22:33 | |
| *** GheRivero has quit IRC | 22:33 | |
| *** mugsie has quit IRC | 22:33 | |
| *** jamespage_ has quit IRC | 22:33 | |
| *** zehicle_at_dell has quit IRC | 22:33 | |
| *** westmaas_ has quit IRC | 22:33 | |
| *** saschpe has quit IRC | 22:33 | |
| *** dangers_` has quit IRC | 22:33 | |
| *** tristanC has quit IRC | 22:33 | |
| *** msuriar_ has quit IRC | 22:33 | |
| *** mikal_ has quit IRC | 22:33 | |
| *** zul has quit IRC | 22:33 | |
| *** beagles has quit IRC | 22:33 | |
| *** banix has quit IRC | 22:33 | |
| *** boris-42_ has quit IRC | 22:33 | |
| *** jroovers has quit IRC | 22:33 | |
| *** matel has quit IRC | 22:33 | |
| *** rockstar has quit IRC | 22:33 | |
| *** jeckersb_gone has quit IRC | 22:33 | |
| *** krtaylor has quit IRC | 22:33 | |
| *** hughsaunders has quit IRC | 22:33 | |
| *** vishy has quit IRC | 22:33 | |
| *** leifmadsen has quit IRC | 22:33 | |
| *** cp16net has quit IRC | 22:33 | |
| *** dklyle has quit IRC | 22:33 | |
| *** kevinbenton has quit IRC | 22:33 | |
| *** saper has quit IRC | 22:33 | |
| *** isviridov has quit IRC | 22:33 | |
| *** hemnafk has quit IRC | 22:33 | |
| *** fbo has quit IRC | 22:33 | |
| *** ChanServ has quit IRC | 22:33 | |
| *** morganfainberg has quit IRC | 22:33 | |
| *** sarob has quit IRC | 22:33 | |
| *** jaypipes has quit IRC | 22:33 | |
| *** tjones has quit IRC | 22:33 | |
| *** dizquierdo has quit IRC | 22:33 | |
| *** jamespag` has quit IRC | 22:33 | |
| *** tteggel has quit IRC | 22:33 | |
| *** dstanek has quit IRC | 22:33 | |
| *** dhellmann has quit IRC | 22:33 | |
| *** jgrimm has quit IRC | 22:33 | |
| *** virmitio has quit IRC | 22:33 | |
| *** salv-orlando has quit IRC | 22:33 | |
| *** lttrl has quit IRC | 22:33 | |
| *** dkehn_ has quit IRC | 22:33 | |
| *** guitarzan has quit IRC | 22:33 | |
| *** dkehn has quit IRC | 22:33 | |
| *** hemna has quit IRC | 22:33 | |
| *** Xurong has quit IRC | 22:33 | |
| *** lifeless has quit IRC | 22:33 | |
| *** SergeyLukjanov has quit IRC | 22:33 | |
| *** ruhe has quit IRC | 22:33 | |
| *** katyafervent_awa has quit IRC | 22:33 | |
| *** notmyname has quit IRC | 22:33 | |
| *** AaronGr has quit IRC | 22:33 | |
| *** mattoliverau has quit IRC | 22:33 | |
| *** ewindisch has quit IRC | 22:33 | |
| *** dtroyer has quit IRC | 22:33 | |
| *** Daviey has quit IRC | 22:33 | |
| *** hub_cap has quit IRC | 22:33 | |
| *** stevebaker has quit IRC | 22:33 | |
| *** greghaynes has quit IRC | 22:33 | |
| *** olaph has quit IRC | 22:33 | |
| *** hdd has quit IRC | 22:33 | |
| *** rwsu has quit IRC | 22:33 | |
| *** antonym has quit IRC | 22:33 | |
| *** mancdaz has quit IRC | 22:33 | |
| *** rahmu has quit IRC | 22:33 | |
| *** BobBall has quit IRC | 22:33 | |
| *** mrda has quit IRC | 22:33 | |
| *** relaxdiego has quit IRC | 22:33 | |
| *** bogdando has quit IRC | 22:33 | |
| *** sdake has quit IRC | 22:33 | |
| *** shardy_afk has quit IRC | 22:33 | |
| *** brosenberg has quit IRC | 22:33 | |
| *** NobodyCam has quit IRC | 22:33 | |
| *** devananda has quit IRC | 22:33 | |
| *** zaro has quit IRC | 22:33 | |
| *** Ng has quit IRC | 22:33 | |
| *** SlickNik has quit IRC | 22:33 | |
| *** nati_ueno has joined #openstack-infra | 22:37 | |
| *** spiffxp_ has joined #openstack-infra | 22:37 | |
| *** chris_johnson has joined #openstack-infra | 22:37 | |
| *** openstackgerrit has joined #openstack-infra | 22:37 | |
| *** nijaba has joined #openstack-infra | 22:37 | |
| *** Ryan_Lane2 has joined #openstack-infra | 22:37 | |
| *** jamespage_ has joined #openstack-infra | 22:37 | |
| *** relaxdiego has joined #openstack-infra | 22:37 | |
| *** bknudson has joined #openstack-infra | 22:37 | |
| *** Alex_Gaynor has joined #openstack-infra | 22:37 | |
| *** belliott has joined #openstack-infra | 22:37 | |
| *** jaypipes has joined #openstack-infra | 22:37 | |
| *** jeblair has joined #openstack-infra | 22:37 | |
| *** pleia2 has joined #openstack-infra | 22:37 | |
| *** mkoderer has joined #openstack-infra | 22:37 | |
| *** zehicle_at_dell has joined #openstack-infra | 22:37 | |
| *** dansmith has joined #openstack-infra | 22:37 | |
| *** NikitaKonovalov has joined #openstack-infra | 22:37 | |
| *** spredzy has joined #openstack-infra | 22:37 | |
| *** dachary has joined #openstack-infra | 22:37 | |
| *** dolphm_503 has joined #openstack-infra | 22:37 | |
| *** cyeoh has joined #openstack-infra | 22:37 | |
| *** tjones has joined #openstack-infra | 22:37 | |
| *** westmaas_ has joined #openstack-infra | 22:37 | |
| *** saschpe has joined #openstack-infra | 22:37 | |
| *** Shrews_ has joined #openstack-infra | 22:37 | |
| *** pcrews_ has joined #openstack-infra | 22:37 | |
| *** geekinutah has joined #openstack-infra | 22:37 | |
| *** dangers_` has joined #openstack-infra | 22:37 | |
| *** dpyzhov has joined #openstack-infra | 22:37 | |
| *** flaper87 has joined #openstack-infra | 22:37 | |
| *** afrittoli has joined #openstack-infra | 22:37 | |
| *** pdmars_ has joined #openstack-infra | 22:37 | |
| *** masayukig has joined #openstack-infra | 22:37 | |
| *** gyee has joined #openstack-infra | 22:37 | |
| *** fungi has joined #openstack-infra | 22:37 | |
| *** mattymo has joined #openstack-infra | 22:37 | |
| *** tristanC has joined #openstack-infra | 22:37 | |
| *** stevebaker has joined #openstack-infra | 22:37 | |
| *** dragondm has joined #openstack-infra | 22:37 | |
| *** harlowja has joined #openstack-infra | 22:37 | |
| *** ianw has joined #openstack-infra | 22:37 | |
| *** jamespag` has joined #openstack-infra | 22:37 | |
| *** plomakin_ has joined #openstack-infra | 22:37 | |
| *** cadenzajon has joined #openstack-infra | 22:37 | |
| *** pmathews1 has joined #openstack-infra | 22:37 | |
| *** jamielennox|away has joined #openstack-infra | 22:37 | |
| *** mtreinish has joined #openstack-infra | 22:37 | |
| *** mika has joined #openstack-infra | 22:37 | |
| *** gaelL_ has joined #openstack-infra | 22:37 | |
| *** msuriar_ has joined #openstack-infra | 22:37 | |
| *** mikal_ has joined #openstack-infra | 22:37 | |
| *** tteggel has joined #openstack-infra | 22:37 | |
| *** StevenK_ has joined #openstack-infra | 22:37 | |
| *** bnemec_ has joined #openstack-infra | 22:37 | |
| *** niska` has joined #openstack-infra | 22:37 | |
| *** jog0 has joined #openstack-infra | 22:37 | |
| *** EntropyWorks has joined #openstack-infra | 22:37 | |
| *** dstanek has joined #openstack-infra | 22:37 | |
| *** weshay has joined #openstack-infra | 22:37 | |
| *** akscram has joined #openstack-infra | 22:37 | |
| *** Adri2000 has joined #openstack-infra | 22:37 | |
| *** ekarlso has joined #openstack-infra | 22:37 | |
| *** psedlak has joined #openstack-infra | 22:37 | |
| *** sileht has joined #openstack-infra | 22:37 | |
| *** bodepd has joined #openstack-infra | 22:37 | |
| *** exz has joined #openstack-infra | 22:37 | |
| *** vrovachev has joined #openstack-infra | 22:37 | |
| *** loquacities has joined #openstack-infra | 22:37 | |
| *** che-arne has joined #openstack-infra | 22:37 | |
| *** jesusaurus has joined #openstack-infra | 22:37 | |
| *** zul has joined #openstack-infra | 22:37 | |
| *** dims has joined #openstack-infra | 22:37 | |
| *** SlickNik has joined #openstack-infra | 22:37 | |
| *** mrda has joined #openstack-infra | 22:37 | |
| *** BobBall has joined #openstack-infra | 22:37 | |
| *** rahmu has joined #openstack-infra | 22:37 | |
| *** Ng has joined #openstack-infra | 22:37 | |
| *** hub_cap has joined #openstack-infra | 22:37 | |
| *** Daviey has joined #openstack-infra | 22:37 | |
| *** zaro has joined #openstack-infra | 22:37 | |
| *** devananda has joined #openstack-infra | 22:37 | |
| *** NobodyCam has joined #openstack-infra | 22:37 | |
| *** shardy_afk has joined #openstack-infra | 22:37 | |
| *** ChanServ has joined #openstack-infra | 22:37 | |
| *** fbo has joined #openstack-infra | 22:37 | |
| *** mancdaz has joined #openstack-infra | 22:37 | |
| *** dtroyer has joined #openstack-infra | 22:37 | |
| *** mugsie has joined #openstack-infra | 22:37 | |
| *** GheRivero has joined #openstack-infra | 22:37 | |
| *** mattoliverau has joined #openstack-infra | 22:37 | |
| *** ewindisch has joined #openstack-infra | 22:37 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 22:37 | |
| *** isviridov has joined #openstack-infra | 22:37 | |
| *** AaronGr has joined #openstack-infra | 22:37 | |
| *** mordred has joined #openstack-infra | 22:37 | |
| *** gema has joined #openstack-infra | 22:37 | |
| *** davidlenwell has joined #openstack-infra | 22:37 | |
| *** esmute has joined #openstack-infra | 22:37 | |
| *** kmartin has joined #openstack-infra | 22:37 | |
| *** ilyashakhat_ has joined #openstack-infra | 22:37 | |
| *** saper has joined #openstack-infra | 22:37 | |
| *** vladan has joined #openstack-infra | 22:37 | |
| *** kevinbenton has joined #openstack-infra | 22:37 | |
| *** gothicmindfood has joined #openstack-infra | 22:37 | |
| *** dklyle has joined #openstack-infra | 22:37 | |
| *** ogelbukh has joined #openstack-infra | 22:37 | |
| *** cp16net has joined #openstack-infra | 22:37 | |
| *** notmyname has joined #openstack-infra | 22:37 | |
| *** aude has joined #openstack-infra | 22:37 | |
| *** jcooley has joined #openstack-infra | 22:37 | |
| *** antonym has joined #openstack-infra | 22:37 | |
| *** HenryG has joined #openstack-infra | 22:37 | |
| *** morganfainberg has joined #openstack-infra | 22:37 | |
| *** jang has joined #openstack-infra | 22:37 | |
| *** hemnafk has joined #openstack-infra | 22:37 | |
| *** mburned has joined #openstack-infra | 22:37 | |
| *** nibalizer has joined #openstack-infra | 22:37 | |
| *** katyafervent_awa has joined #openstack-infra | 22:37 | |
| *** maurosr has joined #openstack-infra | 22:37 | |
| *** locke1051 has joined #openstack-infra | 22:37 | |
| *** chmouel has joined #openstack-infra | 22:37 | |
| *** ruhe has joined #openstack-infra | 22:37 | |
| *** SergeyLukjanov has joined #openstack-infra | 22:37 | |
| *** vipul has joined #openstack-infra | 22:37 | |
| *** sdake has joined #openstack-infra | 22:37 | |
| *** samalba has joined #openstack-infra | 22:37 | |
| *** jooools has joined #openstack-infra | 22:37 | |
| *** Steap_ has joined #openstack-infra | 22:37 | |
| *** bogdando has joined #openstack-infra | 22:37 | |
| *** blamar has joined #openstack-infra | 22:37 | |
| *** rwsu has joined #openstack-infra | 22:37 | |
| *** hdd has joined #openstack-infra | 22:37 | |
| *** brosenberg has joined #openstack-infra | 22:37 | |
| *** lifeless has joined #openstack-infra | 22:37 | |
| *** michchap has joined #openstack-infra | 22:37 | |
| *** Xurong has joined #openstack-infra | 22:37 | |
| *** leifmadsen has joined #openstack-infra | 22:37 | |
| *** olaph has joined #openstack-infra | 22:37 | |
| *** homeless has joined #openstack-infra | 22:37 | |
| *** hemna has joined #openstack-infra | 22:37 | |
| *** greghaynes has joined #openstack-infra | 22:37 | |
| *** gmoro has joined #openstack-infra | 22:37 | |
| *** mestery has joined #openstack-infra | 22:37 | |
| *** dkehn has joined #openstack-infra | 22:37 | |
| *** tian has joined #openstack-infra | 22:37 | |
| *** yamahata_ has joined #openstack-infra | 22:37 | |
| *** amotoki has joined #openstack-infra | 22:37 | |
| *** vishy has joined #openstack-infra | 22:37 | |
| *** yongli has joined #openstack-infra | 22:37 | |
| *** skraynev_afk has joined #openstack-infra | 22:37 | |
| *** hughsaunders has joined #openstack-infra | 22:37 | |
| *** asadoughi has joined #openstack-infra | 22:37 | |
| *** koolhead17 has joined #openstack-infra | 22:37 | |
| *** ihrachys has joined #openstack-infra | 22:37 | |
| *** guitarzan has joined #openstack-infra | 22:37 | |
| *** gpocentek has joined #openstack-infra | 22:37 | |
| *** krtaylor has joined #openstack-infra | 22:37 | |
| *** dkehn_ has joined #openstack-infra | 22:37 | |
| *** lttrl has joined #openstack-infra | 22:37 | |
| *** jeckersb_gone has joined #openstack-infra | 22:37 | |
| *** kiall has joined #openstack-infra | 22:37 | |
| *** miqui has joined #openstack-infra | 22:37 | |
| *** bradm has joined #openstack-infra | 22:37 | |
| *** rockstar has joined #openstack-infra | 22:37 | |
| *** afazekas has joined #openstack-infra | 22:37 | |
| *** zhiyan_ has joined #openstack-infra | 22:37 | |
| *** salv-orlando has joined #openstack-infra | 22:37 | |
| *** Hefeweizen has joined #openstack-infra | 22:37 | |
| *** andre__ has joined #openstack-infra | 22:37 | |
| *** portante has joined #openstack-infra | 22:37 | |
| *** unicell1 has joined #openstack-infra | 22:37 | |
| *** matel has joined #openstack-infra | 22:37 | |
| *** milki has joined #openstack-infra | 22:37 | |
| *** DinaBelova_ has joined #openstack-infra | 22:37 | |
| *** jd__ has joined #openstack-infra | 22:37 | |
| *** ArxCruz has joined #openstack-infra | 22:37 | |
| *** julienvey has joined #openstack-infra | 22:37 | |
| *** primeministerp has joined #openstack-infra | 22:37 | |
| *** tru_tru has joined #openstack-infra | 22:37 | |
| *** juice has joined #openstack-infra | 22:37 | |
| *** dcramer_ has joined #openstack-infra | 22:37 | |
| *** jasond`` has joined #openstack-infra | 22:37 | |
| *** max_lobur_afk has joined #openstack-infra | 22:37 | |
| *** yamahata has joined #openstack-infra | 22:37 | |
| *** jpeeler has joined #openstack-infra | 22:37 | |
| *** jergerber has joined #openstack-infra | 22:37 | |
| *** ryanpetrello has joined #openstack-infra | 22:37 | |
| *** eharney has joined #openstack-infra | 22:37 | |
| *** jroovers has joined #openstack-infra | 22:37 | |
| *** wenlock has joined #openstack-infra | 22:37 | |
| *** boris-42_ has joined #openstack-infra | 22:37 | |
| *** sandywalsh_ has joined #openstack-infra | 22:37 | |
| *** dkranz has joined #openstack-infra | 22:37 | |
| *** virmitio has joined #openstack-infra | 22:37 | |
| *** changbl has joined #openstack-infra | 22:37 | |
| *** ravikumar_hp has joined #openstack-infra | 22:37 | |
| *** jgrimm has joined #openstack-infra | 22:37 | |
| *** mdenny has joined #openstack-infra | 22:37 | |
| *** nicedice has joined #openstack-infra | 22:37 | |
| *** atiwari has joined #openstack-infra | 22:37 | |
| *** VijayT has joined #openstack-infra | 22:37 | |
| *** esker has joined #openstack-infra | 22:37 | |
| *** CaptTofu has joined #openstack-infra | 22:37 | |
| *** moted has joined #openstack-infra | 22:37 | |
| *** dripton has joined #openstack-infra | 22:37 | |
| *** dhellmann has joined #openstack-infra | 22:37 | |
| *** afazekas_ has joined #openstack-infra | 22:37 | |
| *** sabari has joined #openstack-infra | 22:37 | |
| *** marun has joined #openstack-infra | 22:37 | |
| *** kfox1111 has joined #openstack-infra | 22:37 | |
| *** banix has joined #openstack-infra | 22:37 | |
| *** khyati_ has joined #openstack-infra | 22:37 | |
| *** vkozhukalov has joined #openstack-infra | 22:37 | |
| *** hogepodge has joined #openstack-infra | 22:37 | |
| *** hashar has joined #openstack-infra | 22:37 | |
| *** prad has joined #openstack-infra | 22:37 | |
| *** enikanorov_ has joined #openstack-infra | 22:37 | |
| *** branen_ has joined #openstack-infra | 22:37 | |
| *** alexpilotti has joined #openstack-infra | 22:37 | |
| *** jgriffith has joined #openstack-infra | 22:37 | |
| *** jhesketh_ has joined #openstack-infra | 22:37 | |
| *** beagles has joined #openstack-infra | 22:37 | |
| *** adarazs_ has joined #openstack-infra | 22:37 | |
| *** mrodden has joined #openstack-infra | 22:37 | |
| *** pabelanger has joined #openstack-infra | 22:37 | |
| *** dstufft has joined #openstack-infra | 22:37 | |
| *** luisg has joined #openstack-infra | 22:37 | |
| *** bookwar has joined #openstack-infra | 22:37 | |
| *** sdague has joined #openstack-infra | 22:37 | |
| *** oubiwann-home has joined #openstack-infra | 22:37 | |
| *** sbadia has joined #openstack-infra | 22:37 | |
| *** jeremyb has joined #openstack-infra | 22:37 | |
| *** Shrews_ is now known as Shrews | 22:37 | |
| fungi | looks like nodepoold's flatlined again... i take it that's what 75206 is fixing? | 22:37 |
| jeblair | fungi: i've stopped it while i clean up from another bug (whose patch is forthcoming) | 22:38 |
| fungi | got it | 22:38 |
| jeblair | sorry, that may have been lost in the netsplit | 22:38 |
| *** pdmars_ is now known as pdmars | 22:38 | |
| fungi | entirely possible | 22:38 |
| fungi | or i may have just missed it | 22:38 |
| jeblair | (the str!=int bug is what made the mess i'm having to clean up) | 22:38 |
| mordred | gah | 22:38 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix pip on py3k/pypy nodes https://review.openstack.org/75213 | 22:39 |
| *** clarkb has joined #openstack-infra | 22:39 | |
| mordred | fungi: ok. that should take care of pypy nodes ^^ | 22:39 |
| jog0 | jeblair: you should push a patch to novaclient to break if its an int | 22:39 |
| jog0 | and watch mordred cry | 22:39 |
| jog0 | i'll +2 it too | 22:39 |
| mordred | I tested it by spinning up a node, running the nodepool scripts on it, and then cloning python-swiftclient and running tox -epypy | 22:40 |
| fungi | mordred: excellent. and you can confirm without that you're seeing the setuptools breakage we saw on our py3k-precise nodes? | 22:40 |
| *** markmcclain has quit IRC | 22:41 | |
| mordred | fungi: nope. but I can try to reproduce that real quick | 22:42 |
| fungi | mordred: and if you could also confirm a tox -epy33 of something isn't fouled up by that adjustment too, i think it would help | 22:42 |
| *** hemna has quit IRC | 22:42 | |
| mordred | kk | 22:42 |
| jeblair | fungi, clarkb, mordred: the int!=str problem might explain the keypair leak | 22:43 |
| fungi | ohhhh...*lightbulb* yes | 22:43 |
| fungi | keypair ids also returned as ints in diablo, betting | 22:44 |
| mordred | jeblair: yah | 22:44 |
| lifeless | jeblair: are you happy for us to move on up the stack - reenable the cloud, add jobs etc? | 22:44 |
| mordred | jeblair: I'm amazed that this worked as well as it did for so long | 22:44 |
| *** EmilienM has quit IRC | 22:46 | |
| *** adam_g has quit IRC | 22:46 | |
| *** pvo has quit IRC | 22:46 | |
| *** jbryce has quit IRC | 22:46 | |
| *** annegentle has quit IRC | 22:46 | |
| *** wendar has quit IRC | 22:46 | |
| *** Guest38907 has quit IRC | 22:46 | |
| *** rainya has quit IRC | 22:46 | |
| *** russellb has quit IRC | 22:46 | |
| *** jlk has quit IRC | 22:46 | |
| *** ttx has quit IRC | 22:46 | |
| *** bknudson has quit IRC | 22:46 | |
| *** mkoderer has quit IRC | 22:46 | |
| *** harlowja has quit IRC | 22:46 | |
| *** mrodden has quit IRC | 22:46 | |
| *** pabelanger has quit IRC | 22:46 | |
| *** dstufft has quit IRC | 22:46 | |
| *** luisg has quit IRC | 22:46 | |
| *** bookwar has quit IRC | 22:46 | |
| *** sdague has quit IRC | 22:46 | |
| *** oubiwann-home has quit IRC | 22:46 | |
| *** sbadia has quit IRC | 22:46 | |
| *** jeremyb has quit IRC | 22:46 | |
| *** Alex_Gaynor has quit IRC | 22:46 | |
| *** cyeoh has quit IRC | 22:46 | |
| *** dolphm_503 has quit IRC | 22:46 | |
| *** fungi has quit IRC | 22:46 | |
| *** StevenK_ has quit IRC | 22:46 | |
| *** che-arne has quit IRC | 22:46 | |
| *** loquacities has quit IRC | 22:46 | |
| *** vrovachev has quit IRC | 22:46 | |
| *** exz has quit IRC | 22:46 | |
| *** bodepd has quit IRC | 22:46 | |
| *** sileht has quit IRC | 22:46 | |
| *** psedlak has quit IRC | 22:46 | |
| *** ekarlso has quit IRC | 22:46 | |
| *** Adri2000 has quit IRC | 22:46 | |
| *** akscram has quit IRC | 22:46 | |
| *** dansmith has quit IRC | 22:46 | |
| *** pcrews_ has quit IRC | 22:46 | |
| *** mattymo has quit IRC | 22:46 | |
| *** bnemec_ has quit IRC | 22:46 | |
| *** niska` has quit IRC | 22:46 | |
| *** enikanorov_ has quit IRC | 22:46 | |
| *** hashar has quit IRC | 22:46 | |
| *** hogepodge has quit IRC | 22:46 | |
| *** kfox1111 has quit IRC | 22:46 | |
| *** sabari has quit IRC | 22:46 | |
| *** CaptTofu has quit IRC | 22:46 | |
| *** esker has quit IRC | 22:46 | |
| *** ravikumar_hp has quit IRC | 22:46 | |
| *** changbl has quit IRC | 22:46 | |
| *** sandywalsh_ has quit IRC | 22:46 | |
| *** jasond`` has quit IRC | 22:46 | |
| *** tru_tru has quit IRC | 22:46 | |
| *** zhiyan_ has quit IRC | 22:46 | |
| *** miqui has quit IRC | 22:46 | |
| *** gmoro has quit IRC | 22:46 | |
| *** spiffxp_ has quit IRC | 22:46 | |
| *** pleia2 has quit IRC | 22:46 | |
| *** dachary has quit IRC | 22:46 | |
| *** dpyzhov has quit IRC | 22:46 | |
| *** flaper87 has quit IRC | 22:46 | |
| *** mika has quit IRC | 22:46 | |
| *** EntropyWorks has quit IRC | 22:46 | |
| *** jesusaurus has quit IRC | 22:46 | |
| *** jgriffith has quit IRC | 22:46 | |
| *** branen_ has quit IRC | 22:46 | |
| *** khyati_ has quit IRC | 22:46 | |
| *** DinaBelova_ has quit IRC | 22:46 | |
| *** milki has quit IRC | 22:46 | |
| *** portante has quit IRC | 22:46 | |
| *** afazekas has quit IRC | 22:46 | |
| *** bradm has quit IRC | 22:46 | |
| *** ihrachys has quit IRC | 22:46 | |
| *** koolhead17 has quit IRC | 22:46 | |
| *** tian has quit IRC | 22:46 | |
| *** jooools has quit IRC | 22:46 | |
| *** maurosr has quit IRC | 22:46 | |
| *** clarkb has quit IRC | 22:46 | |
| *** jeblair has quit IRC | 22:46 | |
| *** NikitaKonovalov has quit IRC | 22:46 | |
| *** Shrews has quit IRC | 22:46 | |
| *** afrittoli has quit IRC | 22:46 | |
| *** masayukig has quit IRC | 22:46 | |
| *** gyee has quit IRC | 22:46 | |
| *** mtreinish has quit IRC | 22:46 | |
| *** jog0 has quit IRC | 22:46 | |
| *** dims has quit IRC | 22:46 | |
| *** prad has quit IRC | 22:46 | |
| *** nicedice has quit IRC | 22:46 | |
| *** wenlock has quit IRC | 22:46 | |
| *** jergerber has quit IRC | 22:46 | |
| *** max_lobur_afk has quit IRC | 22:46 | |
| *** jd__ has quit IRC | 22:46 | |
| *** yongli has quit IRC | 22:46 | |
| *** amotoki has quit IRC | 22:46 | |
| *** homeless has quit IRC | 22:46 | |
| *** vipul has quit IRC | 22:46 | |
| *** locke1051 has quit IRC | 22:46 | |
| *** michchap has quit IRC | 22:46 | |
| *** nibalizer has quit IRC | 22:46 | |
| *** HenryG has quit IRC | 22:46 | |
| *** aude has quit IRC | 22:46 | |
| *** gothicmindfood has quit IRC | 22:46 | |
| *** vladan has quit IRC | 22:46 | |
| *** ilyashakhat_ has quit IRC | 22:46 | |
| *** gpocentek has quit IRC | 22:46 | |
| *** geekinutah has quit IRC | 22:46 | |
| *** pdmars has quit IRC | 22:46 | |
| *** dragondm has quit IRC | 22:46 | |
| *** plomakin_ has quit IRC | 22:46 | |
| *** cadenzajon has quit IRC | 22:46 | |
| *** pmathews1 has quit IRC | 22:46 | |
| *** adarazs_ has quit IRC | 22:46 | |
| *** alexpilotti has quit IRC | 22:46 | |
| *** marun has quit IRC | 22:46 | |
| *** atiwari has quit IRC | 22:46 | |
| *** mdenny has quit IRC | 22:46 | |
| *** dkranz has quit IRC | 22:46 | |
| *** eharney has quit IRC | 22:46 | |
| *** ryanpetrello has quit IRC | 22:46 | |
| *** jpeeler has quit IRC | 22:46 | |
| *** yamahata has quit IRC | 22:46 | |
| *** dcramer_ has quit IRC | 22:46 | |
| *** juice has quit IRC | 22:46 | |
| *** ArxCruz has quit IRC | 22:46 | |
| *** kiall has quit IRC | 22:46 | |
| *** skraynev_afk has quit IRC | 22:46 | |
| *** yamahata_ has quit IRC | 22:46 | |
| *** mestery has quit IRC | 22:46 | |
| *** blamar has quit IRC | 22:46 | |
| *** Steap_ has quit IRC | 22:46 | |
| *** samalba has quit IRC | 22:46 | |
| *** chmouel has quit IRC | 22:46 | |
| *** mburned has quit IRC | 22:46 | |
| *** jcooley has quit IRC | 22:46 | |
| *** ogelbukh has quit IRC | 22:46 | |
| *** kmartin has quit IRC | 22:46 | |
| *** jang has quit IRC | 22:46 | |
| *** openstackgerrit has quit IRC | 22:46 | |
| *** Ryan_Lane2 has quit IRC | 22:46 | |
| *** belliott has quit IRC | 22:46 | |
| *** spredzy has quit IRC | 22:46 | |
| *** ianw has quit IRC | 22:46 | |
| *** jamielennox|away has quit IRC | 22:46 | |
| *** gaelL_ has quit IRC | 22:46 | |
| *** weshay has quit IRC | 22:46 | |
| *** jhesketh_ has quit IRC | 22:46 | |
| *** vkozhukalov has quit IRC | 22:46 | |
| *** afazekas_ has quit IRC | 22:46 | |
| *** dripton has quit IRC | 22:46 | |
| *** moted has quit IRC | 22:46 | |
| *** VijayT has quit IRC | 22:46 | |
| *** primeministerp has quit IRC | 22:46 | |
| *** julienvey has quit IRC | 22:46 | |
| *** unicell1 has quit IRC | 22:46 | |
| *** andre__ has quit IRC | 22:46 | |
| *** Hefeweizen has quit IRC | 22:46 | |
| *** asadoughi has quit IRC | 22:46 | |
| *** esmute has quit IRC | 22:46 | |
| *** davidlenwell has quit IRC | 22:46 | |
| *** gema has quit IRC | 22:46 | |
| *** mordred has quit IRC | 22:46 | |
| *** GheRivero has quit IRC | 22:46 | |
| *** mugsie has quit IRC | 22:46 | |
| *** jamespage_ has quit IRC | 22:46 | |
| *** zehicle_at_dell has quit IRC | 22:46 | |
| *** westmaas_ has quit IRC | 22:46 | |
| *** saschpe has quit IRC | 22:46 | |
| *** dangers_` has quit IRC | 22:46 | |
| *** tristanC has quit IRC | 22:46 | |
| *** msuriar_ has quit IRC | 22:46 | |
| *** mikal_ has quit IRC | 22:46 | |
| *** zul has quit IRC | 22:46 | |
| *** beagles has quit IRC | 22:46 | |
| *** banix has quit IRC | 22:46 | |
| *** boris-42_ has quit IRC | 22:46 | |
| *** jroovers has quit IRC | 22:46 | |
| *** matel has quit IRC | 22:46 | |
| *** rockstar has quit IRC | 22:46 | |
| *** jeckersb_gone has quit IRC | 22:46 | |
| *** krtaylor has quit IRC | 22:46 | |
| *** hughsaunders has quit IRC | 22:46 | |
| *** vishy has quit IRC | 22:46 | |
| *** leifmadsen has quit IRC | 22:46 | |
| *** cp16net has quit IRC | 22:46 | |
| *** dklyle has quit IRC | 22:46 | |
| *** kevinbenton has quit IRC | 22:46 | |
| *** saper has quit IRC | 22:46 | |
| *** isviridov has quit IRC | 22:46 | |
| *** hemnafk has quit IRC | 22:46 | |
| *** fbo has quit IRC | 22:46 | |
| *** ChanServ has quit IRC | 22:46 | |
| *** morganfainberg has quit IRC | 22:46 | |
| *** nati_ueno has quit IRC | 22:46 | |
| *** nijaba has quit IRC | 22:46 | |
| *** jaypipes has quit IRC | 22:46 | |
| *** tjones has quit IRC | 22:46 | |
| *** jamespag` has quit IRC | 22:46 | |
| *** tteggel has quit IRC | 22:46 | |
| *** dstanek has quit IRC | 22:46 | |
| *** dhellmann has quit IRC | 22:46 | |
| *** jgrimm has quit IRC | 22:46 | |
| *** virmitio has quit IRC | 22:46 | |
| *** salv-orlando has quit IRC | 22:46 | |
| *** lttrl has quit IRC | 22:46 | |
| *** dkehn_ has quit IRC | 22:46 | |
| *** guitarzan has quit IRC | 22:46 | |
| *** dkehn has quit IRC | 22:46 | |
| *** Xurong has quit IRC | 22:46 | |
| *** lifeless has quit IRC | 22:46 | |
| *** SergeyLukjanov has quit IRC | 22:46 | |
| *** ruhe has quit IRC | 22:46 | |
| *** katyafervent_awa has quit IRC | 22:46 | |
| *** notmyname has quit IRC | 22:46 | |
| *** AaronGr has quit IRC | 22:46 | |
| *** mattoliverau has quit IRC | 22:46 | |
| *** ewindisch has quit IRC | 22:46 | |
| *** dtroyer has quit IRC | 22:46 | |
| *** Daviey has quit IRC | 22:46 | |
| *** hub_cap has quit IRC | 22:46 | |
| *** stevebaker has quit IRC | 22:46 | |
| *** greghaynes has quit IRC | 22:46 | |
| *** olaph has quit IRC | 22:46 | |
| *** hdd has quit IRC | 22:46 | |
| *** rwsu has quit IRC | 22:46 | |
| *** antonym has quit IRC | 22:46 | |
| *** mancdaz has quit IRC | 22:46 | |
| *** rahmu has quit IRC | 22:46 | |
| *** BobBall has quit IRC | 22:46 | |
| *** mrda has quit IRC | 22:46 | |
| *** chris_johnson has quit IRC | 22:46 | |
| *** relaxdiego has quit IRC | 22:46 | |
| *** bogdando has quit IRC | 22:46 | |
| *** sdake has quit IRC | 22:46 | |
| *** shardy_afk has quit IRC | 22:46 | |
| *** brosenberg has quit IRC | 22:46 | |
| *** NobodyCam has quit IRC | 22:46 | |
| *** devananda has quit IRC | 22:46 | |
| *** zaro has quit IRC | 22:46 | |
| *** Ng has quit IRC | 22:46 | |
| *** SlickNik has quit IRC | 22:46 | |
| *** hogepodge_ has joined #openstack-infra | 22:48 | |
| *** clarkb has joined #openstack-infra | 22:48 | |
| *** ttx has joined #openstack-infra | 22:48 | |
| *** jlk has joined #openstack-infra | 22:48 | |
| *** russellb has joined #openstack-infra | 22:48 | |
| *** wendar has joined #openstack-infra | 22:48 | |
| *** Guest38907 has joined #openstack-infra | 22:48 | |
| *** annegentle has joined #openstack-infra | 22:48 | |
| *** rainya has joined #openstack-infra | 22:48 | |
| *** jbryce has joined #openstack-infra | 22:48 | |
| *** pvo has joined #openstack-infra | 22:48 | |
| *** adam_g has joined #openstack-infra | 22:48 | |
| *** EmilienM has joined #openstack-infra | 22:48 | |
| *** nati_ueno has joined #openstack-infra | 22:48 | |
| *** spiffxp_ has joined #openstack-infra | 22:48 | |
| *** chris_johnson has joined #openstack-infra | 22:48 | |
| *** openstackgerrit has joined #openstack-infra | 22:48 | |
| *** nijaba has joined #openstack-infra | 22:48 | |
| *** Ryan_Lane2 has joined #openstack-infra | 22:48 | |
| *** jamespage_ has joined #openstack-infra | 22:48 | |
| *** relaxdiego has joined #openstack-infra | 22:48 | |
| *** bknudson has joined #openstack-infra | 22:48 | |
| *** Alex_Gaynor has joined #openstack-infra | 22:48 | |
| *** belliott has joined #openstack-infra | 22:48 | |
| *** jaypipes has joined #openstack-infra | 22:48 | |
| *** jeblair has joined #openstack-infra | 22:48 | |
| *** pleia2 has joined #openstack-infra | 22:48 | |
| *** mkoderer has joined #openstack-infra | 22:48 | |
| *** zehicle_at_dell has joined #openstack-infra | 22:48 | |
| *** dansmith has joined #openstack-infra | 22:48 | |
| *** NikitaKonovalov has joined #openstack-infra | 22:48 | |
| *** spredzy has joined #openstack-infra | 22:48 | |
| *** dachary has joined #openstack-infra | 22:48 | |
| *** dolphm_503 has joined #openstack-infra | 22:48 | |
| *** cyeoh has joined #openstack-infra | 22:48 | |
| *** tjones has joined #openstack-infra | 22:48 | |
| *** westmaas_ has joined #openstack-infra | 22:48 | |
| *** saschpe has joined #openstack-infra | 22:48 | |
| *** Shrews has joined #openstack-infra | 22:48 | |
| *** pcrews_ has joined #openstack-infra | 22:48 | |
| *** geekinutah has joined #openstack-infra | 22:48 | |
| *** dangers_` has joined #openstack-infra | 22:48 | |
| *** dpyzhov has joined #openstack-infra | 22:48 | |
| *** flaper87 has joined #openstack-infra | 22:48 | |
| *** afrittoli has joined #openstack-infra | 22:48 | |
| *** pdmars has joined #openstack-infra | 22:48 | |
| *** masayukig has joined #openstack-infra | 22:48 | |
| *** gyee has joined #openstack-infra | 22:48 | |
| *** fungi has joined #openstack-infra | 22:48 | |
| *** mattymo has joined #openstack-infra | 22:48 | |
| *** tristanC has joined #openstack-infra | 22:48 | |
| *** stevebaker has joined #openstack-infra | 22:48 | |
| *** dragondm has joined #openstack-infra | 22:48 | |
| *** harlowja has joined #openstack-infra | 22:48 | |
| *** ianw has joined #openstack-infra | 22:48 | |
| *** jamespag` has joined #openstack-infra | 22:48 | |
| *** plomakin_ has joined #openstack-infra | 22:48 | |
| *** cadenzajon has joined #openstack-infra | 22:48 | |
| *** pmathews1 has joined #openstack-infra | 22:48 | |
| *** jamielennox|away has joined #openstack-infra | 22:48 | |
| *** mtreinish has joined #openstack-infra | 22:48 | |
| *** mika has joined #openstack-infra | 22:48 | |
| *** gaelL_ has joined #openstack-infra | 22:48 | |
| *** msuriar_ has joined #openstack-infra | 22:48 | |
| *** mikal_ has joined #openstack-infra | 22:48 | |
| *** tteggel has joined #openstack-infra | 22:48 | |
| *** StevenK_ has joined #openstack-infra | 22:48 | |
| *** bnemec_ has joined #openstack-infra | 22:48 | |
| *** niska` has joined #openstack-infra | 22:48 | |
| *** jog0 has joined #openstack-infra | 22:48 | |
| *** EntropyWorks has joined #openstack-infra | 22:48 | |
| *** dstanek has joined #openstack-infra | 22:48 | |
| *** weshay has joined #openstack-infra | 22:48 | |
| *** akscram has joined #openstack-infra | 22:48 | |
| *** Adri2000 has joined #openstack-infra | 22:48 | |
| *** ekarlso has joined #openstack-infra | 22:48 | |
| *** psedlak has joined #openstack-infra | 22:48 | |
| *** sileht has joined #openstack-infra | 22:48 | |
| *** bodepd has joined #openstack-infra | 22:48 | |
| *** exz has joined #openstack-infra | 22:48 | |
| *** vrovachev has joined #openstack-infra | 22:48 | |
| *** loquacities has joined #openstack-infra | 22:48 | |
| *** che-arne has joined #openstack-infra | 22:48 | |
| *** jesusaurus has joined #openstack-infra | 22:48 | |
| *** zul has joined #openstack-infra | 22:48 | |
| *** dims has joined #openstack-infra | 22:48 | |
| *** jeremyb has joined #openstack-infra | 22:48 | |
| *** sbadia has joined #openstack-infra | 22:48 | |
| *** oubiwann-home has joined #openstack-infra | 22:48 | |
| *** sdague has joined #openstack-infra | 22:48 | |
| *** bookwar has joined #openstack-infra | 22:48 | |
| *** luisg has joined #openstack-infra | 22:48 | |
| *** dstufft has joined #openstack-infra | 22:48 | |
| *** pabelanger has joined #openstack-infra | 22:48 | |
| *** mrodden has joined #openstack-infra | 22:48 | |
| *** adarazs_ has joined #openstack-infra | 22:48 | |
| *** beagles has joined #openstack-infra | 22:48 | |
| *** jhesketh_ has joined #openstack-infra | 22:48 | |
| *** jgriffith has joined #openstack-infra | 22:48 | |
| *** alexpilotti has joined #openstack-infra | 22:48 | |
| *** branen_ has joined #openstack-infra | 22:48 | |
| *** enikanorov_ has joined #openstack-infra | 22:48 | |
| *** prad has joined #openstack-infra | 22:48 | |
| *** hashar has joined #openstack-infra | 22:48 | |
| *** hogepodge has joined #openstack-infra | 22:48 | |
| *** vkozhukalov has joined #openstack-infra | 22:48 | |
| *** khyati_ has joined #openstack-infra | 22:48 | |
| *** banix has joined #openstack-infra | 22:48 | |
| *** kfox1111 has joined #openstack-infra | 22:48 | |
| *** marun has joined #openstack-infra | 22:48 | |
| *** sabari has joined #openstack-infra | 22:48 | |
| *** afazekas_ has joined #openstack-infra | 22:48 | |
| *** dhellmann has joined #openstack-infra | 22:48 | |
| *** dripton has joined #openstack-infra | 22:48 | |
| *** moted has joined #openstack-infra | 22:48 | |
| *** CaptTofu has joined #openstack-infra | 22:48 | |
| *** esker has joined #openstack-infra | 22:48 | |
| *** VijayT has joined #openstack-infra | 22:48 | |
| *** atiwari has joined #openstack-infra | 22:48 | |
| *** nicedice has joined #openstack-infra | 22:48 | |
| *** mdenny has joined #openstack-infra | 22:48 | |
| *** jgrimm has joined #openstack-infra | 22:48 | |
| *** ravikumar_hp has joined #openstack-infra | 22:48 | |
| *** changbl has joined #openstack-infra | 22:48 | |
| *** virmitio has joined #openstack-infra | 22:48 | |
| *** dkranz has joined #openstack-infra | 22:48 | |
| *** sandywalsh_ has joined #openstack-infra | 22:48 | |
| *** boris-42_ has joined #openstack-infra | 22:48 | |
| *** wenlock has joined #openstack-infra | 22:48 | |
| *** jroovers has joined #openstack-infra | 22:48 | |
| *** eharney has joined #openstack-infra | 22:48 | |
| *** ryanpetrello has joined #openstack-infra | 22:48 | |
| *** jergerber has joined #openstack-infra | 22:48 | |
| *** jpeeler has joined #openstack-infra | 22:48 | |
| *** yamahata has joined #openstack-infra | 22:48 | |
| *** max_lobur_afk has joined #openstack-infra | 22:48 | |
| *** jasond`` has joined #openstack-infra | 22:48 | |
| *** dcramer_ has joined #openstack-infra | 22:48 | |
| *** juice has joined #openstack-infra | 22:48 | |
| *** tru_tru has joined #openstack-infra | 22:48 | |
| *** primeministerp has joined #openstack-infra | 22:48 | |
| *** julienvey has joined #openstack-infra | 22:48 | |
| *** ArxCruz has joined #openstack-infra | 22:48 | |
| *** jd__ has joined #openstack-infra | 22:48 | |
| *** DinaBelova_ has joined #openstack-infra | 22:48 | |
| *** milki has joined #openstack-infra | 22:48 | |
| *** matel has joined #openstack-infra | 22:48 | |
| *** unicell1 has joined #openstack-infra | 22:48 | |
| *** portante has joined #openstack-infra | 22:48 | |
| *** andre__ has joined #openstack-infra | 22:48 | |
| *** Hefeweizen has joined #openstack-infra | 22:48 | |
| *** salv-orlando has joined #openstack-infra | 22:48 | |
| *** zhiyan_ has joined #openstack-infra | 22:48 | |
| *** afazekas has joined #openstack-infra | 22:48 | |
| *** rockstar has joined #openstack-infra | 22:48 | |
| *** bradm has joined #openstack-infra | 22:48 | |
| *** miqui has joined #openstack-infra | 22:48 | |
| *** kiall has joined #openstack-infra | 22:48 | |
| *** jeckersb_gone has joined #openstack-infra | 22:48 | |
| *** lttrl has joined #openstack-infra | 22:48 | |
| *** dkehn_ has joined #openstack-infra | 22:48 | |
| *** krtaylor has joined #openstack-infra | 22:48 | |
| *** gpocentek has joined #openstack-infra | 22:48 | |
| *** guitarzan has joined #openstack-infra | 22:48 | |
| *** ihrachys has joined #openstack-infra | 22:48 | |
| *** koolhead17 has joined #openstack-infra | 22:48 | |
| *** asadoughi has joined #openstack-infra | 22:48 | |
| *** hughsaunders has joined #openstack-infra | 22:48 | |
| *** skraynev_afk has joined #openstack-infra | 22:48 | |
| *** yongli has joined #openstack-infra | 22:48 | |
| *** vishy has joined #openstack-infra | 22:48 | |
| *** amotoki has joined #openstack-infra | 22:48 | |
| *** yamahata_ has joined #openstack-infra | 22:48 | |
| *** tian has joined #openstack-infra | 22:48 | |
| *** dkehn has joined #openstack-infra | 22:48 | |
| *** mestery has joined #openstack-infra | 22:48 | |
| *** gmoro has joined #openstack-infra | 22:48 | |
| *** greghaynes has joined #openstack-infra | 22:48 | |
| *** homeless has joined #openstack-infra | 22:48 | |
| *** olaph has joined #openstack-infra | 22:48 | |
| *** leifmadsen has joined #openstack-infra | 22:48 | |
| *** Xurong has joined #openstack-infra | 22:48 | |
| *** michchap has joined #openstack-infra | 22:48 | |
| *** lifeless has joined #openstack-infra | 22:48 | |
| *** brosenberg has joined #openstack-infra | 22:48 | |
| *** hdd has joined #openstack-infra | 22:48 | |
| *** rwsu has joined #openstack-infra | 22:48 | |
| *** blamar has joined #openstack-infra | 22:48 | |
| *** bogdando has joined #openstack-infra | 22:48 | |
| *** Steap_ has joined #openstack-infra | 22:48 | |
| *** jooools has joined #openstack-infra | 22:48 | |
| *** samalba has joined #openstack-infra | 22:48 | |
| *** sdake has joined #openstack-infra | 22:48 | |
| *** vipul has joined #openstack-infra | 22:48 | |
| *** SergeyLukjanov has joined #openstack-infra | 22:48 | |
| *** ruhe has joined #openstack-infra | 22:48 | |
| *** chmouel has joined #openstack-infra | 22:48 | |
| *** locke1051 has joined #openstack-infra | 22:48 | |
| *** maurosr has joined #openstack-infra | 22:48 | |
| *** katyafervent_awa has joined #openstack-infra | 22:48 | |
| *** nibalizer has joined #openstack-infra | 22:48 | |
| *** mburned has joined #openstack-infra | 22:48 | |
| *** hemnafk has joined #openstack-infra | 22:48 | |
| *** jang has joined #openstack-infra | 22:48 | |
| *** morganfainberg has joined #openstack-infra | 22:48 | |
| *** HenryG has joined #openstack-infra | 22:48 | |
| *** antonym has joined #openstack-infra | 22:48 | |
| *** jcooley has joined #openstack-infra | 22:48 | |
| *** aude has joined #openstack-infra | 22:48 | |
| *** notmyname has joined #openstack-infra | 22:48 | |
| *** cp16net has joined #openstack-infra | 22:48 | |
| *** ogelbukh has joined #openstack-infra | 22:48 | |
| *** dklyle has joined #openstack-infra | 22:48 | |
| *** gothicmindfood has joined #openstack-infra | 22:48 | |
| *** kevinbenton has joined #openstack-infra | 22:48 | |
| *** vladan has joined #openstack-infra | 22:48 | |
| *** saper has joined #openstack-infra | 22:48 | |
| *** ilyashakhat_ has joined #openstack-infra | 22:48 | |
| *** kmartin has joined #openstack-infra | 22:48 | |
| *** esmute has joined #openstack-infra | 22:48 | |
| *** davidlenwell has joined #openstack-infra | 22:48 | |
| *** gema has joined #openstack-infra | 22:48 | |
| *** mordred has joined #openstack-infra | 22:48 | |
| *** AaronGr has joined #openstack-infra | 22:48 | |
| *** isviridov has joined #openstack-infra | 22:48 | |
| *** ewindisch has joined #openstack-infra | 22:48 | |
| *** mattoliverau has joined #openstack-infra | 22:48 | |
| *** GheRivero has joined #openstack-infra | 22:48 | |
| *** mugsie has joined #openstack-infra | 22:48 | |
| *** dtroyer has joined #openstack-infra | 22:48 | |
| *** mancdaz has joined #openstack-infra | 22:48 | |
| *** fbo has joined #openstack-infra | 22:48 | |
| *** ChanServ has joined #openstack-infra | 22:48 | |
| *** shardy_afk has joined #openstack-infra | 22:48 | |
| *** NobodyCam has joined #openstack-infra | 22:48 | |
| *** devananda has joined #openstack-infra | 22:48 | |
| *** zaro has joined #openstack-infra | 22:48 | |
| *** Daviey has joined #openstack-infra | 22:48 | |
| *** hub_cap has joined #openstack-infra | 22:48 | |
| *** Ng has joined #openstack-infra | 22:48 | |
| *** rahmu has joined #openstack-infra | 22:48 | |
| *** BobBall has joined #openstack-infra | 22:48 | |
| *** mrda has joined #openstack-infra | 22:48 | |
| *** SlickNik has joined #openstack-infra | 22:48 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 22:48 | |
| *** hogepodge has quit IRC | 22:48 | |
| *** hogepodge_ is now known as hogepodge | 22:48 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Node deletion related fixes https://review.openstack.org/75217 | 22:48 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Coerce all ids from novaclient to str https://review.openstack.org/75218 | 22:49 |
| mordred | fungi: hrm | 22:49 |
| *** geekinutah has left #openstack-infra | 22:49 | |
| jeblair | okay, that last one is an untested fix to the int/str problem | 22:49 |
| mordred | fungi: nope. I cannot reproduce the breakage | 22:49 |
| fungi | mordred: i think 75213 won't work without a shebang line in fix_pip.sh or invoking it with an explicit shell in prepare_node_py3k.sh | 22:49 |
| mordred | fungi: I'm going to spin up a new node and run the scripts without the fix, then run the tests to see if the breakage is there | 22:50 |
| fungi | mordred: do you still have one of the failure logs from earlier, or do you need me to dig it up?> | 22:50 |
| * mordred should have done this in the first place | 22:50 | |
| mordred | fungi: I can find it | 22:50 |
| fungi | k, thanks! | 22:50 |
| *** ryanpetrello has quit IRC | 22:51 | |
| clarkb | https://etherpad.openstack.org/p/juno-test-maxtrices woo brain vomit | 22:52 |
| *** dhellmann has quit IRC | 22:52 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1282795 https://review.openstack.org/75220 | 22:52 |
| clarkb | note I don't feel strongly about anything in that pad, just needed to get that out of my head and into writing | 22:53 |
| fungi | clarkb: very nice dr strangelove reference there | 22:53 |
| dstufft | what does fix_pip.sh do | 22:53 |
| dstufft | is it a bug I can fix in pip | 22:53 |
| fungi | dstufft: we replace it with ruby bundler | 22:53 |
| dstufft | fungi: smart | 22:53 |
| fungi | s/bundler/bungler/ | 22:54 |
| jeblair | it was a nice setup :) | 22:54 |
| jog0 | clarkb: just skimmed it looks nice | 22:54 |
| jog0 | fungi: ^^ | 22:54 |
| fungi | dstufft: actually runs get-pip instead of using distro-preinstalled packages | 22:54 |
| jog0 | py26 bug query | 22:54 |
| mordred | what fungi said | 22:54 |
| dstufft | fungi: oh! | 22:54 |
| jeblair | starting nodepool with all those changes applied | 22:54 |
| mordred | dstufft: it's more "fix stupid distro broken old setuptools" - but that's too long | 22:55 |
| dstufft | I'm working on wheel 2.0 spec | 22:55 |
| fungi | dstufft: and in this case sets up setuptools for more than one installed interpreter | 22:55 |
| dstufft | it'll make wheel s executable | 22:55 |
| mordred | hrm | 22:55 |
| mordred | do we want that? | 22:55 |
| dstufft | so you can just do python pip-*.whl | 22:55 |
| dstufft | or something | 22:55 |
| dstufft | no more get-pip.py | 22:55 |
| clarkb | sdague: you may be interested in that etherpad too | 22:55 |
| fungi | jog0: thanks! | 22:55 |
| mordred | I think one of the main things I like about wheels is that they don't execute code | 22:55 |
| dstufft | mordred: no no not like that | 22:55 |
| dstufft | installing a wheel will still be completely code free | 22:56 |
| mordred | ok. phew | 22:56 |
| dstufft | it jut means you can directly execute a wheel without installing it | 22:56 |
| dstufft | if the wheel is setup to allow that | 22:56 |
| *** banix has quit IRC | 22:56 | |
| fungi | dstufft: probably better to have an alternative means of installing it rather than passing it to the interpreter the same way you pass python code to the interpreter, just to avoind replacing a wheel with a malicious script | 22:56 |
| dstufft | sort of like a jar | 22:56 |
| dstufft | I think | 22:56 |
| jeblair | clarkb: while you are right in theory, i'm hesitant to say that we have a problem with the check backlog at the moment. | 22:56 |
| *** arborism has quit IRC | 22:57 | |
| sdague | clarkb: yes, very interested, can't hit it up right this sec though :) | 22:57 |
| jeblair | clarkb: there's too much system interruption going on to really determine that. | 22:57 |
| clarkb | jeblair: good point | 22:57 |
| mordred | dstufft: nod | 22:57 |
| mordred | dstufft: as long as one of your error messages is "can't find pip. possible alternatives: pip" | 22:58 |
| mordred | I'll be great | 22:58 |
| fungi | dstufft: er, with better words... to avoid someone maliciously substituting a script for the wheel... so python --install-wheel mything.whl instead? | 22:58 |
| dstufft | fungi: oh | 22:58 |
| dstufft | fungi: well this would just be using stuff already built into Python | 22:58 |
| *** pdmars has quit IRC | 22:58 | |
| dstufft | python can execute a zip file | 22:58 |
| clarkb | jeblair: I am framing it within the context of zuul because I think that is the most immediately important potential issue, but we aren't testing a lot of that stuff today when I think we could be | 22:58 |
| dstufft | if you create a zip with a top level __main__.py, python will execute it | 22:59 |
| mordred | dstufft: right. but I think fungi is suggesting that, if I get a wheel, I'm going to be expecting a no-code-executed thing | 22:59 |
| clarkb | jeblair: and I think we can be testing stuff while using fewer resources if we are smart about it | 22:59 |
| clarkb | jeblair: but then the feature parity thing happens and it all falls apart | 22:59 |
| mordred | dstufft: and if someon wrote a script, then named it foo.whl | 22:59 |
| jeblair | clarkb: yep. i agree with that. | 22:59 |
| fungi | mordred: dstufft: yes, that's my concern | 22:59 |
| mordred | the usage "python foo.whl" would do a thing - but not a good one | 22:59 |
| mordred | whereas a script masquerading as a wheel | 22:59 |
| mordred | passed to python --wheel foo.whl | 23:00 |
| mordred | would bea n error | 23:00 |
| dstufft | there's nothing special about the .whl directory | 23:00 |
| dstufft | er tld | 23:00 |
| dstufft | it's just a zip file with a __main__.py | 23:00 |
| dstufft | you can rename the .whl to .foo | 23:00 |
| dstufft | and do python bar.foo | 23:00 |
| dstufft | and it'll execute it | 23:00 |
| mordred | right. but you're suggesting that people might want to start passing that to python directly | 23:00 |
| *** jcooley_ has joined #openstack-infra | 23:01 | |
| *** mriedem has joined #openstack-infra | 23:02 | |
| fungi | if a wheel is static content, then having the installer place that content into the right locations and that's all... but not automatically run anything embedded within it | 23:02 |
| lifeless | jeblair: presuming you got netsplit: 11:44 < lifeless> jeblair: are you happy for us to move on up the stack - reenable the cloud, add jobs etc? | 23:02 |
| dstufft | an installer woldn't run any code when installing the wheel | 23:02 |
| dstufft | even an executeable wheel | 23:02 |
| dstufft | and wheels themselves wouldn't be generally self installing | 23:02 |
| fungi | dstufft: however, if you pass the wheel to the python interpreter the same way you pass a script to the interpreter, how do you keep the interpreter from running the wheel when it's actually a script and not really a wheel at all | 23:03 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Add support for TestNG publisher. https://review.openstack.org/70157 | 23:03 |
| clarkb | is nodepool stuck again? | 23:03 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Remove unhelpful log message https://review.openstack.org/75223 | 23:03 |
| clarkb | jeblair: fungi ^ zuul status implies it is | 23:03 |
| *** westmaas_ is now known as westmaas | 23:04 | |
| fungi | clarkb: jeblair is still working on it--he's stopped it | 23:04 |
| clarkb | ah | 23:04 |
| dstufft | fungi: well you can't, but I don't understand the attack vector? if you can modify the wheel you can do lots of evil things already | 23:04 |
| clarkb | did I miss that in the netsplitting? | 23:04 |
| jeblair | lifeless: yes, i'm happy to proceed now; are there reviews i should look at? | 23:04 |
| lifeless | jeblair: , numbers are: | 23:04 |
| *** jamielennox|away is now known as jamielennox | 23:04 | |
| jeblair | clarkb, fungi: i very recently restart nodepool with the tip of my branch installed, it's recovering now | 23:04 |
| fungi | dstufft: installing the wheel as an account with write access to a common location, but wanting to later run code from it as a user without write access to that location, as a common use case | 23:05 |
| clarkb | jeblair: thanks | 23:05 |
| lifeless | jeblair: https://review.openstack.org/#/c/72035/ https://review.openstack.org/#/c/75173/ and https://review.openstack.org/#/c/73886/ | 23:05 |
| dstufft | fungi: but if you're doing that, why would you be directly executing the wheel? | 23:05 |
| jeblair | clarkb, fungi: the str!=int bug needed significant manual cleanup (alien-list/nova delete) | 23:05 |
| fungi | dstufft: oh, you're not talking about having 'python mywheel.whl' install the wheel anywhere? | 23:06 |
| fungi | dstufft: you're suggesting a mechanism to execute the contents of a wheel instead of installing it? | 23:06 |
| dstufft | fungi: no, it just executes some code that is dependent on the Wheel itself, the pip wheel would cause that to install itself, but some other wheel could do something totally different | 23:06 |
| *** mriedem has quit IRC | 23:06 | |
| dstufft | fungi: yes | 23:06 |
| clarkb | jeblair: is that something multiple people can assist on? I am happy to help once I have reviewed the code changes | 23:06 |
| jeblair | clarkb: nope, all done | 23:07 |
| dstufft | fungi: python foo.whl would basically be "setup the paths so I can import the stuff inside my wheel, and then execute this function from inside the wheel" | 23:07 |
| fungi | dstufft: okay, that seems safe enough then. i was more concerned that you were muddying the syntax for installing a wheel with the syntax for running arbitrary python code | 23:07 |
| jeblair | though i'm thinking of restarting it again to pick up that last change; the log msgs from that are insane. | 23:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1282772 https://review.openstack.org/75185 | 23:07 |
| dstufft | fungi: ah | 23:07 |
| dstufft | fungi: nah, just in pip's case because it just means we could essentially bundle a get-pip.py inside of the Wheel and instead of running get-pip.py, execute the Wheel | 23:08 |
| dstufft | possibly support executing without installing pip at all too | 23:08 |
| *** niska` is now known as niska | 23:08 | |
| fungi | dstufft: though certainly having a way to install a wheel by using things built into python, without needing to get pip first, would also be nice. but yeah that's still the old 3.4 backport problem | 23:09 |
| *** mikal_ is now known as mikal | 23:10 | |
| clarkb | jeblair: comment in https://review.openstack.org/#/c/75218/ for future improvement | 23:11 |
| openstackgerrit | David Lenwell proposed a change to openstack-infra/config: Adding refstack into stackforge.. yay! https://review.openstack.org/75226 | 23:11 |
| fungi | jeblair: the pending stack of nodepool fixes lgtm. also it seems we're starting to get workers assigned on pending jobs again | 23:11 |
| *** dcramer_ has quit IRC | 23:11 | |
| jeblair | clarkb: nod | 23:12 |
| clarkb | fungi: yup lgtm as well, I have approved the tsack | 23:12 |
| * fungi needs to step away and cook dinner. bbiab | 23:12 | |
| *** ociuhandu has joined #openstack-infra | 23:12 | |
| *** andreaf has joined #openstack-infra | 23:13 | |
| clarkb | jeblair: is now a bad time to approve changes like https://review.openstack.org/#/c/73073/ or are you happy with the current state of nodepool? | 23:13 |
| jeblair | clarkb: let me restart one more time with that logging fix, then let's call it stable enough | 23:14 |
| clarkb | jeblair: k | 23:15 |
| *** andreaf has quit IRC | 23:15 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added support for Jenkins Campfire plugin https://review.openstack.org/69315 | 23:16 |
| jeblair | clarkb: done | 23:16 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Remove tripleo from nodepool config" https://review.openstack.org/72035 | 23:16 |
| jeblair | and just in time :) | 23:16 |
| *** weshay has quit IRC | 23:16 | |
| *** andreaf has joined #openstack-infra | 23:16 | |
| jeblair | that's going to be building images anyway | 23:16 |
| *** sandywalsh_ has quit IRC | 23:17 | |
| *** dolphm_503 is now known as dolphm | 23:17 | |
| * clarkb approves the other build images change | 23:17 | |
| kevinbenton | what should i do in the case that jenkins comes back with jobs that are "LOST" | 23:21 |
| *** e0ne has joined #openstack-infra | 23:21 | |
| kevinbenton | oh, never mind. i see there is a bug for it on the rechecks page | 23:22 |
| clarkb | kevinbenton: hold on | 23:22 |
| clarkb | kevinbenton: I think this is a different problem | 23:22 |
| *** jcooley_ has quit IRC | 23:22 | |
| clarkb | jeblair: ^ do you expect lost jobs to have occurred due to the nodepool stuff? | 23:22 |
| kevinbenton | clarkb: ok. let me know when it's safe to try a recheck | 23:22 |
| clarkb | jeblair: like the check queue is almost empty now | 23:23 |
| clarkb | I think something went boom | 23:23 |
| * clarkb looks at zuul logs | 23:23 | |
| * anteaya was waiting for the check patches to be reenqued so didn't say anything | 23:24 | |
| * greghaynes just got the LOST deal to | 23:24 | |
| anteaya | you and about 190 other patches | 23:24 |
| greghaynes | Sounds like a LOST party | 23:24 |
| clarkb | jeblair: yeah looks like zuul marked hundreds of jobs LOST | 23:24 |
| anteaya | the check queue was 201 last I looked | 23:24 |
| jeblair | my guess would be it lost connection to gearman | 23:24 |
| anteaya | and only 13 back in check now | 23:25 |
| clarkb | starting at 23:16 ish and ending 23:17 ish | 23:25 |
| * bnemec_ plays wallflower at the LOST party | 23:25 | |
| dkehn_ | ouch | 23:25 |
| clarkb | jeblair: it being all of the jenkins? | 23:25 |
| *** SumitNaiksatam has joined #openstack-infra | 23:25 | |
| jeblair | clarkb: zuu | 23:25 |
| jeblair | l | 23:25 |
| *** CaptTofu has quit IRC | 23:25 | |
| anteaya | so what action to we advise the LOST patch owners to engage in? | 23:26 |
| *** e0ne has quit IRC | 23:26 | |
| jeblair | anteaya: recheck | 23:26 |
| anteaya | okay folks recheck no bug your LOST patches | 23:26 |
| dkehn_ | on it, get in line | 23:27 |
| kevinbenton | thanks | 23:27 |
| anteaya | yup | 23:27 |
| clarkb | jeblair: gearman log on zuul.o.o shows a bunch of broken pipes | 23:27 |
| clarkb | jeblair: so something caused all of those slaves to go away | 23:27 |
| bnemec_ | Is there a bug I can reverify against for approved patches, or should I just re-approve? | 23:27 |
| anteaya | bnemec_: reapprove | 23:27 |
| clarkb | jeblair: would nodepool have done that? | 23:27 |
| bnemec_ | anteaya: Okay, thanks | 23:27 |
| anteaya | there is no filed bug | 23:27 |
| anteaya | np | 23:27 |
| jeblair | clarkb: i'm pretty sure the queued jobs would only be removed if zuul itself lost the connection | 23:28 |
| jeblair | clarkb: (the workers may have as well) | 23:28 |
| clarkb | jeblair: you think zuul lost its localhost connection? | 23:29 |
| *** VijayT has quit IRC | 23:29 | |
| jeblair | clarkb: it's happened once recently | 23:29 |
| *** markwash has joined #openstack-infra | 23:29 | |
| clarkb | interesting | 23:29 |
| *** jcooley_ has joined #openstack-infra | 23:30 | |
| clarkb | jeblair: KeyError: 'H:127.0.0.1:1295830' is in there right at the beginning of it all | 23:30 |
| anteaya | lots of queued right now | 23:30 |
| *** markmcclain has joined #openstack-infra | 23:30 | |
| anteaya | are things moving in the right direction in nodepool? | 23:30 |
| *** eharney has quit IRC | 23:31 | |
| *** dkranz has quit IRC | 23:31 | |
| jeblair | clarkb: note that happened in "_lostConnection" | 23:31 |
| anteaya | basically nothing in check is running | 23:31 |
| anteaya | and half of gate is running | 23:32 |
| anteaya | 34 in check 14 in gate | 23:32 |
| jeblair | clarkb: this is part of why i wrote that change to gear to have it log connection names; it would be much easier if that said "Zuul" | 23:32 |
| lifeless | fedoraand precise are buildings | 23:32 |
| anteaya | okay | 23:32 |
| anteaya | building is good | 23:32 |
| clarkb | jeblair: yeah | 23:33 |
| anteaya | lifeless: is that just for tripleo? | 23:33 |
| clarkb | jeblair: on the zuul side there are a lot of jobs lost due to disconnect | 23:33 |
| clarkb | jeblair: which isn't clear to me if that indicates zuul and gearman disconnected or if the gearman workers disconnected | 23:33 |
| *** dims has quit IRC | 23:33 | |
| clarkb | guessing the former | 23:33 |
| jeblair | clarkb: the former; zuul only sees a work_fail if the worker disconnects | 23:33 |
| jeblair | clarkb: 2014-02-20 23:16:59,690 ERROR gear.Client: Connection <gear.Connection 0x3127dd0 host: 127.0.0.1 port: 4730> timed out waiting for a response to a submit job request: <gear.Job 0x7f33073a8b90 handle: None name | 23:34 |
| jeblair | : stop:jenkins01.openstack.org unique: e23d1acc73314ad0ab5a0fbf9515c234> | 23:34 |
| chris_johnson | How would one get a room added to http://eavesdrop.openstack.org/irclogs/ ? | 23:34 |
| *** ryanpetrello has joined #openstack-infra | 23:34 | |
| anteaya | chris_johnson: let me dig that up for you | 23:34 |
| *** sandywalsh has joined #openstack-infra | 23:34 | |
| chris_johnson | coolio | 23:35 |
| chris_johnson | thanks anteaya: | 23:35 |
| clarkb | jeblair: thanks. Not to completely jump off a cliff, but is that the sort of thing that might make us consider the C server? | 23:35 |
| jeblair | clarkb: i'd love to use the c server; it needs to support canceling jobs though | 23:35 |
| clarkb | oh right | 23:35 |
| anteaya | chris_johnson: look on this page at the very bottom: http://ci.openstack.org/irc.html follow that last instruction about what command to run | 23:36 |
| *** jhesketh__ has joined #openstack-infra | 23:36 | |
| chris_johnson | sweet | 23:36 |
| anteaya | chris_johnson: I will get you a patch to copy after you run that command | 23:36 |
| *** pdmars has joined #openstack-infra | 23:36 | |
| *** pdmars has quit IRC | 23:36 | |
| anteaya | chris_johnson: then use this patch as a template to add your channel name to eavesdrop to be logged: https://review.openstack.org/#/c/74472/1 | 23:37 |
| *** jcooley_ has quit IRC | 23:38 | |
| anteaya | then be patient, we are approving the most critical patches only right now | 23:38 |
| *** jcooley_ has joined #openstack-infra | 23:38 | |
| davidlenwell | anyone have a second to help me figure out what I did wrong here? https://review.openstack.org/#/c/75226/ | 23:39 |
| anteaya | building is happening | 23:39 |
| clarkb | davidlenwell: nothing | 23:39 |
| anteaya | I am not seeing many additional nodes being assigned jobs though | 23:39 |
| clarkb | zuul lost its connection to gearman which marked all jobs LOST | 23:39 |
| davidlenwell | ahh .. awsome | 23:39 |
| davidlenwell | okay.. | 23:39 |
| davidlenwell | so I don't need to do anything? | 23:40 |
| anteaya | recheck no bug | 23:40 |
| anteaya | or reapprove | 23:40 |
| jeblair | clarkb: so i think the thing to do there is: 1) log stuff; 2) think about (but not necessarily do) changing the timeout on submitjob; 3) have zuul re-enqueue in that case like a jenkins derp. | 23:40 |
| anteaya | and get in line with the others | 23:40 |
| clarkb | jeblair: ++ | 23:40 |
| davidlenwell | anteaya: its not a merge its a review that isn't approved yet .. I'm happy to wait as long as needed .. or help if you guys need that | 23:41 |
| jeblair | #status notice A transient error caused Zuul to report jobs as LOST; if you were affected, leave a comment with "recheck no bug" | 23:41 |
| openstackstatus | NOTICE: A transient error caused Zuul to report jobs as LOST; if you were affected, leave a comment with "recheck no bug" | 23:41 |
| *** bnemec_ is now known as bnemec | 23:41 | |
| anteaya | davidlenwell: jeblair and clarkb are hashing it out | 23:41 |
| davidlenwell | I caught that .. thanks.. | 23:41 |
| anteaya | once they decide on a course of action if they need anything they will ask | 23:42 |
| *** vkozhukalov has quit IRC | 23:42 | |
| bnemec | Hmm, if I +A a change that had failed a check job, does Jenkins ignore it until I recheck no bug too? | 23:42 |
| anteaya | it should send it back into check | 23:42 |
| anteaya | do you see it in the check queue? | 23:43 |
| anteaya | so far no jobs are running in the check queue | 23:43 |
| anteaya | so it will take some time to catch back up | 23:43 |
| jeblair | gate has priority | 23:43 |
| anteaya | only half of gate is running | 23:43 |
| bnemec | anteaya: No, and there's no comment from Jenkins that it started anything either. | 23:43 |
| bnemec | https://review.openstack.org/#/c/72318/ | 23:43 |
| anteaya | 15 patches | 23:43 |
| *** mdenny has quit IRC | 23:44 | |
| mordred | fungi: testing now on an unmodified py3k node | 23:44 |
| mordred | using the run_unittests.sh script from slave_Scripts | 23:44 |
| *** jcooley_ has quit IRC | 23:45 | |
| jeblair | bnemec: i think you'll need to 'recheck' that; a reapprove only works if it has a -2, but it was the check jobs that failed... | 23:45 |
| jeblair | we might want to add approval + verify=-1 to the check queue to handle this case | 23:46 |
| *** jergerber has quit IRC | 23:46 | |
| anteaya | jeblair: so far I am not seeing any new jobs starting to run | 23:46 |
| anteaya | since the hiccup | 23:46 |
| lifeless | jeblair: looks like something failed in the template build? There are two precise templates building atm | 23:46 |
| anteaya | 480 nodes in building state according to the graph at the bottom | 23:46 |
| jeblair | anteaya: my status page says a job started 51 seconds ago | 23:47 |
| bnemec | jeblair: Okay, thanks. I'll recheck no bug those too. Just didn't want to spam everyone with extra Gerrit mail if I didn't have to. :-) | 23:47 |
| jeblair | anteaya: and another just now | 23:47 |
| anteaya | can I see the same status page? | 23:47 |
| mordred | fungi: nope. I cannot reproduce | 23:47 |
| jeblair | lifeless: http://nodepool.openstack.org/image.log | 23:47 |
| mordred | wow. that sounded final and personal | 23:47 |
| jeblair | anteaya: http://status.openstack.org/zuul/ | 23:47 |
| anteaya | I'm looking at that | 23:47 |
| clarkb | jeblair: so it looks like gaerman was handling a workstatus, ran into a broken pipe then was cleaning out the jobs list when it hit the keyerror. so the keyerror isn't terribly helpful. I need to reread gear code but does the client close the connection if it runs into a timeout? | 23:47 |
| mordred | fungi: I cannot reproduce the pypy error from earlier on a py3k node | 23:47 |
| anteaya | where do I see where it says a job started 51 seconds ago? | 23:48 |
| * mordred blames Alex_Gaynor | 23:48 | |
| jeblair | clarkb: yeah, i think there's a 30 second timeout for all commands/responses | 23:48 |
| *** mdenny has joined #openstack-infra | 23:49 | |
| *** dims has joined #openstack-infra | 23:49 | |
| clarkb | jeblair: maybe gear should raise and exception but keep the connection alive to avoid the whole thing blowing up? or is blowing up desirable here to not mask trouble | 23:49 |
| jeblair | clarkb: the theory was the latter. 30 seconds is an eternity for gearman | 23:49 |
| harlowja | if anyone here uses, https://pypi.python.org/pypi/gerrit-view 0.3 released | 23:50 |
| harlowja | added zuul review detailing... | 23:50 |
| harlowja | all that fun | 23:50 |
| harlowja | so czuul looks like https://raw.github.com/harlowja/gerrit_view/master/screenshots/screen2.png now (when using --details) | 23:50 |
| *** fbo is now known as fbo_away | 23:51 | |
| harlowja | using clarkb suggestion for pygit usage | 23:51 |
| harlowja | *GitPython i mean | 23:51 |
| *** dolphm is now known as dolphm_503 | 23:51 | |
| jeblair | harlowja: this is to show the commit subject? | 23:52 |
| harlowja | right | 23:52 |
| jeblair | harlowja: instead of that, could you put the subject into the zuul status json; then you don't have to do (expensive) git operations? | 23:52 |
| chris_johnson | anteaya: So if I understand correctly, first step is to add the channel to "modules/gerritbot/files/gerritbot_channel_config.yaml" and go thru code review process? | 23:52 |
| anteaya | no | 23:52 |
| harlowja | jeblair i'm not sure, i would have liked that, but this was a way to get around that :-P | 23:52 |
| lifeless | jeblair: that looks ok | 23:52 |
| anteaya | just run the irc command at the bottom of that page | 23:52 |
| jeblair | harlowja: why are you trying to get around it? you don't want to submit a patch to zuul? | 23:52 |
| *** virmitio has quit IRC | 23:53 | |
| Alex_Gaynor | mordred: what'd I do? | 23:53 |
| harlowja | jeblair nah, just did this for now, thats all | 23:53 |
| lifeless | jeblair: I'm just confused then perhaps - there are two templates | 23:53 |
| anteaya | chris_johnson: that adds permissions to your channel | 23:53 |
| mordred | Alex_Gaynor: pypy | 23:53 |
| harlowja | jeblair it also allows me to get all other review information, author... | 23:53 |
| Alex_Gaynor | mordred: only some of it | 23:53 |
| anteaya | then if there is a problem with the channel and it is logged infra has an option | 23:53 |
| chris_johnson | ahh, so run that command WITHIN the channel? | 23:53 |
| anteaya | yes | 23:53 |
| lifeless | jeblair: http://paste.openstack.org/show/67798/ | 23:54 |
| chris_johnson | gotcha | 23:54 |
| anteaya | or within your irc client | 23:54 |
| *** Sukhdev has joined #openstack-infra | 23:54 | |
| jeblair | harlowja: okay, it's super simple to do so, and i think it would be better for you because it will be faster, and better for the git server because if lots of people run this it won't get hammered | 23:54 |
| harlowja | jeblair k, let me see here | 23:54 |
| mordred | harlowja: plus, then you get the "I submitted a patch to zuul" merit badge | 23:55 |
| harlowja | lol | 23:55 |
| harlowja | +3 | 23:55 |
| jeblair | harlowja: zuul gets every bit of information about a change, so it's pretty easy to just add that into the json | 23:55 |
| chris_johnson | anteaya: I dont have perms to run that command | 23:55 |
| anteaya | chris_johnson: you need to find the person who does | 23:55 |
| jeblair | harlowja: if you do it, base on this change since it's a somewhat heavy rework of the json: https://review.openstack.org/#/c/68828/ | 23:55 |
| chris_johnson | should I reach out took | 23:55 |
| anteaya | is the channel registered? | 23:55 |
| chris_johnson | I assume so | 23:55 |
| harlowja | jeblair ah | 23:55 |
| harlowja | cool | 23:55 |
| chris_johnson | #openstack-sdks | 23:55 |
| harlowja | wow, lots of new info coming in | 23:56 |
| chris_johnson | @dtroyer is the guy I need | 23:56 |
| *** SumitNaiksatam has quit IRC | 23:56 | |
| anteaya | do what you need to do, the command needs to be run before eveasdrop can be added | 23:56 |
| anteaya | chris_johnson: happy hunting | 23:56 |
| chris_johnson | thx | 23:56 |
| anteaya | np | 23:56 |
| *** SpamapS has joined #openstack-infra | 23:56 | |
| harlowja | so 'self.change = change # a changeish' | 23:58 |
| harlowja | is that the git change? | 23:58 |
| harlowja | lol | 23:58 |
| *** SumitNaiksatam has joined #openstack-infra | 23:59 | |
| jeblair | harlowja: if it's a change-related event (as opposed to ref-updated) it's the change in gerrit, yeah. | 23:59 |
| anteaya | the zuul jobs launched per hour graph has topped out | 23:59 |
| *** prad has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
