| dhellmann | jog0: did you open a bug? | 00:00 |
|---|---|---|
| clarkb | jeblair: queue is at 100k, I don't think that will catch up tonight | 00:00 |
| jog0 | got distacted doing it now | 00:00 |
| dhellmann | jog0: yeah, we may have to move that somewhere else to deal with circular dependencies | 00:00 |
| jeblair | clarkb: how do you feel about losing the queue then? especially since it's backed up, i'd really like to get the data visible to jog0 and sdague (not to mention to help you and fungi as you bring the new cluster up) | 00:01 |
| *** e0ne has quit IRC | 00:01 | |
| dhellmann | for now let's get this unjammed | 00:01 |
| dhellmann | if you opened a bug I can assign it to gordc | 00:01 |
| *** e0ne has joined #openstack-infra | 00:01 | |
| clarkb | jeblair: its fine by me | 00:01 |
| jog0 | dhellmann: https://bugs.launchpad.net/oslo/+bug/1284379 | 00:02 |
| jog0 | dhellmann: lets make py33 non-voting for now | 00:02 |
| dhellmann | jog0: I have a patch ready to fix the tests to they work | 00:03 |
| dhellmann | it's installing pycadf, but none of the tests are using it | 00:03 |
| dhellmann | give me 30 seconds | 00:03 |
| jog0 | kk | 00:03 |
| *** lyxus has quit IRC | 00:03 | |
| dhellmann | jog0: https://review.openstack.org/76047 | 00:03 |
| reed | clarkb, fungi: can you please check https://review.openstack.org/#/c/69636/ ? | 00:04 |
| *** jhesketh__ has joined #openstack-infra | 00:04 | |
| jog0 | dhellmann: what about tests/unit/middleware/test_audit.py | 00:05 |
| *** alexpilotti has quit IRC | 00:05 | |
| dhellmann | looking | 00:05 |
| dhellmann | gordc: those tests are not run as part of the py33 configuration | 00:05 |
| dhellmann | sorry, jog0 | 00:05 |
| dhellmann | we have a separate subset of the tests that work under python 3, so the gate job is to keep out regressions | 00:06 |
| jog0 | dhellmann: ohh sneaky | 00:06 |
| dhellmann | yeah | 00:06 |
| jog0 | in that case LGTM | 00:06 |
| *** e0ne has quit IRC | 00:06 | |
| dhellmann | I'll self-approve after the tests actually pass | 00:06 |
| jog0 | dhellmann: cool, thanks for the lightning quick response | 00:07 |
| dhellmann | jog0: thanks for bringing it to my attention! | 00:08 |
| *** sdake_ has quit IRC | 00:08 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Install and run elasticsearch on new workers https://review.openstack.org/76048 | 00:09 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Create new 30g heap elasticsearch workers https://review.openstack.org/76026 | 00:09 |
| fungi | clarkb: ^ (i'll wip the second patch until we're configured, and it can be split into multiples if we prefer) | 00:09 |
| *** mgagne has quit IRC | 00:11 | |
| clarkb | fungi: lgtm | 00:11 |
| jog0 | dhellmann: dependencies are fun | 00:13 |
| sdague | clarkb: https://review.openstack.org/75997 | 00:13 |
| dhellmann | jog0: -1 | 00:13 |
| sdague | that's the latest on the xtrace trim | 00:13 |
| sdague | I'm calling it a night | 00:13 |
| *** relaxdiego has quit IRC | 00:13 | |
| jog0 | dhellmann: ^_^ | 00:14 |
| fungi | clarkb: i'll try to --noop 76026 on elasticsearch.o.o and elasticsearch1.o.o to make sure the delta is empty | 00:15 |
| *** weshay has joined #openstack-infra | 00:16 | |
| dhellmann | jog0: looks like that change merged; I'm going to sign off so please update the bug report if you spot any other related issues | 00:17 |
| jog0 | dhellmann: thaks | 00:18 |
| jog0 | thanks | 00:18 |
| *** salv-orlando has quit IRC | 00:19 | |
| *** khyati has joined #openstack-infra | 00:19 | |
| *** salv-orlando has joined #openstack-infra | 00:19 | |
| reed | devananda, the joys of Launchpad UI :) | 00:19 |
| fungi | clarkb: notice: /Stage[main]/Openstack_project::Base/Package[virtualenv]/ensure: current_value 1.9.1, should be 1.10.1 (noop) | 00:20 |
| fungi | clarkb: oh, nevermind. that's virtualenv | 00:20 |
| * fungi misread | 00:21 | |
| *** yamahata has quit IRC | 00:21 | |
| *** zhiwei has joined #openstack-infra | 00:22 | |
| *** khyati has quit IRC | 00:22 | |
| *** cody-somerville has joined #openstack-infra | 00:25 | |
| *** CaptTofu has quit IRC | 00:25 | |
| *** khyati has joined #openstack-infra | 00:25 | |
| *** matsuhashi has joined #openstack-infra | 00:25 | |
| *** cadenzajon has quit IRC | 00:27 | |
| fungi | clarkb: looks clean. should be safe | 00:29 |
| *** bhuvan_ has joined #openstack-infra | 00:29 | |
| *** lnxnut has joined #openstack-infra | 00:30 | |
| *** bhuvan has quit IRC | 00:31 | |
| *** flaper87 is now known as flaper87|afk | 00:31 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Install and run elasticsearch on new workers https://review.openstack.org/76048 | 00:31 |
| clarkb | fungi: great thanks for checking | 00:31 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch workers https://review.openstack.org/76051 | 00:33 |
| *** sdake_ has joined #openstack-infra | 00:34 | |
| *** CaptTofu has joined #openstack-infra | 00:34 | |
| *** lnxnut has quit IRC | 00:34 | |
| *** lnxnut has joined #openstack-infra | 00:35 | |
| *** sdake_ has quit IRC | 00:38 | |
| fungi | clarkb: and i removed the dummy dns entries since i don't suppose we'll need them. we can keep puppet stopped on the new servers and start the agent one-by-one | 00:38 |
| *** sdake_ has joined #openstack-infra | 00:38 | |
| *** sdake_ has quit IRC | 00:38 | |
| *** sdake_ has joined #openstack-infra | 00:38 | |
| clarkb | fungi: but the old servers need rules too | 00:39 |
| openstackgerrit | A change was merged to openstack-infra/config: Use statsd in logstash client https://review.openstack.org/76042 | 00:39 |
| clarkb | fungi: not sure how that will work | 00:39 |
| fungi | i don't understand... | 00:40 |
| *** lnxnut has quit IRC | 00:40 | |
| clarkb | fungi: the list of servers is used on the existing machines to update their firewalls too | 00:40 |
| *** lcheng has joined #openstack-infra | 00:40 | |
| clarkb | so we need the list to be valid when ever puppet runs on new or old servers | 00:40 |
| fungi | so we merge 76026 first and build 6 new empty servers | 00:40 |
| clarkb | oh I see, we are controlling this at the elasticsearch level | 00:40 |
| fungi | then we stop puppet agent on the 6 new servers | 00:40 |
| clarkb | I grok now, that will work | 00:40 |
| *** Sukhdev has joined #openstack-infra | 00:41 | |
| fungi | then we merge 76048 which updates the firewall rules based on the dns entries for the new servers, which exist by then | 00:41 |
| fungi | and we add the new nodes one by one by applying/enabling puppet on them | 00:42 |
| fungi | stopping services on the old servers along the way as the migration progresses | 00:42 |
| clarkb | yup I grok now, sounds good | 00:42 |
| fungi | then we merge 76051 at the end after deleting the old servers | 00:42 |
| fungi | okay, cool | 00:42 |
| *** derekh has quit IRC | 00:43 | |
| *** david_lyle has quit IRC | 00:44 | |
| fungi | so anyway, i'll go ahead approving 76026 and start spinning up the 6 new base nodes | 00:44 |
| fungi | and add cinder volumes... how big and where should i mount them? | 00:45 |
| fungi | presumably we'll pvcreate them and use lvm | 00:46 |
| fungi | based on the current es servers, i'm going to guess 1t | 00:46 |
| clarkb | ya 1t | 00:47 |
| * clarkb looks for mount point its something in /var/lib iirc | 00:47 | |
| jeblair | restarting jenkins log client | 00:48 |
| clarkb | fungi: #DATA_DIR=/var/lib/elasticsearch | 00:48 |
| clarkb | fungi: so that should be the mount point, or optionally we can /mnt/someplace it and change the elasticsearch DATA_DIR | 00:48 |
| clarkb | fungi: that can be changed in /etc/default/elasticsearch | 00:49 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 00:49 |
| zaro | clarkb: in transit had a little time to do ^ | 00:49 |
| *** lnxnut has joined #openstack-infra | 00:50 | |
| *** lcheng has quit IRC | 00:51 | |
| *** hogepodge has quit IRC | 00:51 | |
| zaro | fungi: my change to set java alternatives to java 7 was working yesterday, but doesn't seem to work today. happen to know what's going on? | 00:51 |
| zaro | fungi: this build just ran and it's trying to use java6.. https://jenkins04.openstack.org/job/check-gerrit-buck/3/console | 00:52 |
| fungi | zaro: did the not-working run end up on an hpcloud-az2 node? | 00:52 |
| *** fifieldt has joined #openstack-infra | 00:53 | |
| fungi | zaro: yep, it did | 00:53 |
| fungi | zaro: if only you knew someone who works at hp ;) | 00:53 |
| *** sabari has quit IRC | 00:53 | |
| fungi | we still can't reliably rebuild server images there | 00:53 |
| fungi | (in az2 only) | 00:54 |
| jeblair | fungi, zaro: i believe i saw a "we're still looking into it" eamil | 00:54 |
| jeblair | 00:54 | |
| jeblair | from hpcloud | 00:54 |
| *** lnxnut has quit IRC | 00:54 | |
| fungi | jeblair: more than one | 00:54 |
| fungi | they've sent about one of those a day i think | 00:54 |
| *** lnxnut has joined #openstack-infra | 00:54 | |
| zaro | fungi: argh! do i just have to wait? | 00:55 |
| *** dangers_away has quit IRC | 00:55 | |
| fungi | zaro: or retry and hope you hit a node on one of the other providers/regions | 00:55 |
| *** relaxdiego has joined #openstack-infra | 00:56 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add logstash job queue graph https://review.openstack.org/76056 | 00:56 |
| zaro | ok. i guess i'm unlucky, tried 3 times, no dice. | 00:56 |
| fungi | zaro: did all 3 run on hpcloud-az2? | 00:56 |
| jeblair | clarkb, fungi: there's a graph ^ it's neat | 00:56 |
| clarkb | jeblair: correct there was a we are wroking on it update email this morning | 00:57 |
| *** dangers_away has joined #openstack-infra | 00:57 | |
| jeblair | clarkb, fungi: it has a red line that is nearly, but not quite, vertical | 00:57 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: make xtrace stateful https://review.openstack.org/75690 | 00:58 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: add indent function https://review.openstack.org/75697 | 00:58 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: suppress devstack console log https://review.openstack.org/75698 | 00:58 |
| *** relaxdiego has quit IRC | 00:58 | |
| *** esker has joined #openstack-infra | 00:59 | |
| *** lnxnut has quit IRC | 00:59 | |
| *** e0ne has joined #openstack-infra | 00:59 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 01:01 |
| *** wenlock has quit IRC | 01:01 | |
| clarkb | jeblair: AaronGr ^ I think that is a WIP. It passes existing tests but does not have any new tests | 01:01 |
| *** lnxnut has joined #openstack-infra | 01:01 | |
| zaro | fungi: yep, just tried again, came up az2 again. | 01:01 |
| *** zhiwei has quit IRC | 01:02 | |
| *** nati_ueno has quit IRC | 01:03 | |
| *** wchrisj has quit IRC | 01:03 | |
| *** e0ne has quit IRC | 01:04 | |
| *** nati_ueno has joined #openstack-infra | 01:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create new 30g heap elasticsearch workers https://review.openstack.org/76026 | 01:05 |
| clarkb | jeblair: I marked 76057 WIP, but your feedback is always valuable | 01:06 |
| *** CaptTofu has quit IRC | 01:09 | |
| *** mgagne has joined #openstack-infra | 01:09 | |
| *** CaptTofu has joined #openstack-infra | 01:11 | |
| *** khyati has quit IRC | 01:12 | |
| *** unicell has quit IRC | 01:13 | |
| *** unicell has joined #openstack-infra | 01:13 | |
| *** Sukhdev has quit IRC | 01:14 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: calculate remaining timeout locally https://review.openstack.org/75726 | 01:14 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: annotate stages of script https://review.openstack.org/75705 | 01:14 |
| *** salv-orlando has quit IRC | 01:15 | |
| *** salv-orlando has joined #openstack-infra | 01:15 | |
| *** atiwari has quit IRC | 01:18 | |
| *** ryanpetrello has quit IRC | 01:19 | |
| *** ryanpetrello_ has joined #openstack-infra | 01:19 | |
| *** rcleere has quit IRC | 01:19 | |
| *** sarob has joined #openstack-infra | 01:20 | |
| *** wchrisj has joined #openstack-infra | 01:22 | |
| *** relaxdiego has joined #openstack-infra | 01:22 | |
| *** relaxdiego has quit IRC | 01:23 | |
| *** pmathews has quit IRC | 01:27 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Roll up node stats https://review.openstack.org/76064 | 01:28 |
| *** yamahata has joined #openstack-infra | 01:29 | |
| *** wchrisj has quit IRC | 01:31 | |
| *** mgagne has quit IRC | 01:31 | |
| *** talluri has quit IRC | 01:31 | |
| *** melwitt has quit IRC | 01:32 | |
| *** zhiwei has joined #openstack-infra | 01:34 | |
| *** mgagne has joined #openstack-infra | 01:35 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/reviewday: Add JSON link and fix header https://review.openstack.org/76065 | 01:37 |
| *** lnxnut has quit IRC | 01:38 | |
| *** lnxnut has joined #openstack-infra | 01:39 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/reviewday: Add JSON link and fix header https://review.openstack.org/76065 | 01:39 |
| *** gokrokve has quit IRC | 01:40 | |
| *** gokrokve has joined #openstack-infra | 01:41 | |
| openstackgerrit | Tushar Gohad proposed a change to openstack/requirements: Add PyECLib as a requirement https://review.openstack.org/76068 | 01:42 |
| *** gokrokve_ has joined #openstack-infra | 01:42 | |
| *** gokrokve_ has quit IRC | 01:43 | |
| *** lnxnut has quit IRC | 01:44 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters. https://review.openstack.org/70636 | 01:44 |
| *** nosnos has joined #openstack-infra | 01:44 | |
| *** gokrokve has quit IRC | 01:46 | |
| *** mgagne has quit IRC | 01:47 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters. https://review.openstack.org/70636 | 01:49 |
| *** pmathews has joined #openstack-infra | 01:50 | |
| lifeless | tripleo overcloud controller failed again | 01:52 |
| lifeless | looks like network card driver issue | 01:52 |
| lifeless | and old Bios | 01:52 |
| *** bhuvan_ has quit IRC | 01:53 | |
| *** lnxnut has joined #openstack-infra | 01:57 | |
| kashyap | clarkb, Heya, when you're here: I pinged the sudo maintainer in Fedora and there's an update on this bug (you notified on the Fedora-on-Gate etherpad that it's one of the issues that's blocking to get Fedora on Gate)-- https://bugzilla.redhat.com/show_bug.cgi?id=1020147#c1 | 01:57 |
| *** chenxu has joined #openstack-infra | 01:58 | |
| openstackgerrit | daisy-ycguo proposed a change to openstack-infra/config: Job to push Horizon translation to Transifex https://review.openstack.org/68042 | 01:58 |
| clarkb | kashyap: awesome, we managed to work around it by allocating ttys for all our ssh connections which is a bit silly but works | 01:59 |
| *** esker has quit IRC | 01:59 | |
| kashyap | clarkb, Like that? -- ssh -t host 'sudo -n command' | 01:59 |
| clarkb | basically, but we are using paramiko | 01:59 |
| *** e0ne has joined #openstack-infra | 01:59 | |
| kashyap | I see. | 01:59 |
| clarkb | so its get_pty=True in the arg list to connections | 01:59 |
| kashyap | I see. I'm afraid, the sudo maintainer is not yet convinced about the default option there in the bug (comment #4) | 02:00 |
| *** pmathews has quit IRC | 02:00 | |
| clarkb | kashyap: ya | 02:01 |
| clarkb | fwiw fedora seems to break a lot of cloudy assumptions | 02:01 |
| clarkb | which is annoying | 02:01 |
| kashyap | clarkb, If you/infra folks think it'd be more elegant to have 'requiretty' disabled by default, then I can raise it with FESCO (Fedora Engineering Steering Comittee) | 02:01 |
| clarkb | meh I don't care super strongly about it | 02:02 |
| kashyap | clarkb, Agree that's annoying. Yep, I can see - you don't have to care too :-) | 02:02 |
| clarkb | kashyap: another issue we have run into is that the fedora user which you are supposed to use instead of root on fedora images isn't created by the time sshd starts | 02:04 |
| kashyap | Thanks for the quick response. clarkb Is there anything else that's blocking? /me just looking at this -- https://etherpad.openstack.org/p/fedora-on-gate . | 02:04 |
| clarkb | so you can't ssh in immediately | 02:04 |
| clarkb | kashyap: pleia2 would be the person to ask | 02:04 |
| *** e0ne has quit IRC | 02:04 | |
| kashyap | clarkb, IIRC, it's created _after_ sshd is started, right? | 02:04 |
| clarkb | ya | 02:04 |
| clarkb | when it should just be baked into the image imo | 02:05 |
| kashyap | Okay, I think I noticed it (fedora user related) in some gerrit review that I was going through some time ago. | 02:06 |
| lifeless | rebooted the node, we now are backish, going to give it a few minutes to stablise | 02:06 |
| lifeless | then start thinking how I can move this to a less FITH machine | 02:07 |
| kashyap | clarkb, That issue seems to be worked around by retrying SSH on auth failure patch -- https://review.openstack.org/#/c/74825/ | 02:07 |
| *** zhiyan_ is now known as zhiyan | 02:07 | |
| *** MarkAtwood has quit IRC | 02:07 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Gate python-glanceclient on PyPy https://review.openstack.org/76072 | 02:08 |
| clarkb | kashyap: correct | 02:08 |
| clarkb | which is clunky because an auth failure usually means you don't want to retry | 02:09 |
| kashyap | clarkb, Yeah, I can see that. I'll talk to Fedora cloud folks on IRC (and probably raise it on mailing list) later this day and see if this can be amended | 02:09 |
| fungi | "While testing ssh access: [Errno 111] Connection refused" | 02:10 |
| Alex_Gaynor | flaper87|afk: Not sure if you're involved in glanceclient, but if yes, https://review.openstack.org/#/c/76072/ may be of interest :-) | 02:10 |
| fungi | good to know that still works | 02:10 |
| *** Ryan_Lane1 has joined #openstack-infra | 02:11 | |
| kashyap | clarkb, Just to be clear, you're referring to these, when you say "fedora images", right? -- http://fedoraproject.org/en/get-fedora#clouds | 02:12 |
| clarkb | kashyap: I believe those are the images used by tripleo | 02:12 |
| * kashyap doesn't know what time-zone clarkb is in (probably EST/PST). If it's too late, just shoo me away | 02:12 | |
| clarkb | lifeless: can you confirm that the fedora images you are using are those ^ | 02:13 |
| clarkb | kashyap: PST, and I am about to walk home :) | 02:13 |
| * kashyap surprised "Valley" folks walk home at 18:00 :-) | 02:14 | |
| clarkb | kashyap: oh I am not one of them :) I live in seattle | 02:14 |
| clarkb | dinner is ready in 15 minutes I need to hurry | 02:15 |
| lifeless | clarkb: I think so | 02:15 |
| *** dkranz has joined #openstack-infra | 02:15 | |
| clarkb | kashyap: so yes, those are the images we are using (lifeless thinks so) | 02:15 |
| kashyap | clarkb, Ah, okay. Yeah, I worked from there (Vallye) a few years ago, just saying what I see :-) G'night! | 02:15 |
| kashyap | lifeless, Heya, who can confirm? Maybe Dan Prince? | 02:16 |
| *** salv-orlando_ has joined #openstack-infra | 02:16 | |
| *** salv-orlando has quit IRC | 02:16 | |
| *** salv-orlando_ is now known as salv-orlando | 02:16 | |
| *** mgagne has joined #openstack-infra | 02:16 | |
| lifeless | kashyap: hit the fedora webpage, download an image | 02:16 |
| lifeless | kashyap: if you're talking about nodepool stuff | 02:16 |
| kashyap | lifeless, Ok, then I know what you're referring to, these ones -- http://fedoraproject.org/en/get-fedora#clouds. Thanks for confirming. I'll just inspect with guestfish to confirm the behavior of fedora user | 02:17 |
| kashyap | Thank you :-) | 02:18 |
| lifeless | kashyap: but if you're talking about dib built imagesfor tripleo | 02:18 |
| lifeless | see the fedora element in dib | 02:18 |
| kashyap | lifeless, Yes, I remember the dib -- not talking about this. | 02:18 |
| kashyap | lifeless, Context is: | 02:18 |
| kashyap | To have Fedora on Gate, trying to identify what are the issues/blockers. It's being coordinated here: https://etherpad.openstack.org/p/fedora-on-gate | 02:19 |
| lifeless | ok, cool | 02:19 |
| kashyap | soren, getting it into nodepool would be step-1 I see there | 02:19 |
| kashyap | Err, didn't mean to ping :-( | 02:19 |
| *** gokrokve has joined #openstack-infra | 02:19 | |
| lifeless | yeah, we want it for triple check jobs | 02:19 |
| kashyap | (ping him*) | 02:19 |
| lifeless | which is similar but different to gate | 02:20 |
| lifeless | for d-g you'll need public cloud providers with images | 02:20 |
| lifeless | which may be different | 02:20 |
| kashyap | lifeless, Yeah, few steps at a time :-). First trying to get into nodepool /me has not much clue about infra, still learning | 02:21 |
| *** moted has quit IRC | 02:24 | |
| *** kiall has joined #openstack-infra | 02:29 | |
| *** unicell has quit IRC | 02:30 | |
| *** pcrews has quit IRC | 02:30 | |
| *** unicell has joined #openstack-infra | 02:31 | |
| *** Ryan_Lane1 has quit IRC | 02:33 | |
| *** reed has quit IRC | 02:34 | |
| *** markmcclain has quit IRC | 02:36 | |
| pleia2 | kashyap: we got the paramiko issue sorted, and https://review.openstack.org/#/c/74825/ *should* fix the user+ssh issue (I haven't tested it, tripleo cloud was down earlier when I was going to) | 02:36 |
| *** emagana has joined #openstack-infra | 02:37 | |
| pleia2 | kashyap: the rest of the etherpad should be up to date, I still have https://review.openstack.org/#/c/70007/ lingering, and I need to figure out why puppet3 on fedora20 doesn't like pip https://review.openstack.org/#/c/70031/ | 02:37 |
| kashyap | pleia2, Yeah, I'm _just_ looking through git commits, this seems to be it from dprince, where you use Paramiko? -- 1963731f7de0e6fc97e2771a043aba60d49a91e8 | 02:37 |
| * kashyap clicks on the review | 02:37 | |
| pleia2 | https://review.openstack.org/#/c/69510/ is my WIP change for other things (mostly networking) | 02:39 |
| kashyap | pleia2, Yes, 74825 is the what the commit I noted above corresponds to. Good to know. | 02:39 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Extend unittest jenkins builder https://review.openstack.org/70511 | 02:39 |
| *** lnxnut has quit IRC | 02:39 | |
| *** lnxnut has joined #openstack-infra | 02:40 | |
| *** krotscheck has quit IRC | 02:41 | |
| kashyap | pleia2, Thanks for the work and your quick response here | 02:41 |
| pleia2 | kashyap: no problem :) I was out a bit last week and then at a conference, so I could use the help catching up | 02:42 |
| pleia2 | so jump in anywhere | 02:42 |
| kashyap | pleia2, Sure. Yeah, it's been hectic here too, been pulled into several directions :-(. And, I'll be away next week traveling. So, just thought I'd check here | 02:43 |
| *** lnxnut has quit IRC | 02:44 | |
| fungi | clarkb: so close! novaclient.exceptions.OverLimit: Quota exceeded for ram: Requested 61440, but already used 983040 of 1024000 ram (HTTP 413) (Request-ID: req-1a4c46f8-90d4-43bf-a556-06388cfacd0a) | 02:46 |
| fungi | that was on the sixth server | 02:46 |
| clarkb | ha | 02:48 |
| clarkb | silly limits | 02:48 |
| fungi | clarkb: we have a few things we can probably clean out | 02:48 |
| fungi | like a couple extra zuuls for a start | 02:48 |
| fungi | i'll see what else we haven't gotten around to cleaning up | 02:49 |
| *** matsuhashi has quit IRC | 02:50 | |
| clarkb | ok thanks I am eating dinner now | 02:50 |
| *** bhuvan has joined #openstack-infra | 02:50 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: start first partition at 1M not 0 https://review.openstack.org/75703 | 02:51 |
| *** oubiwann_ has quit IRC | 02:52 | |
| *** tjones has quit IRC | 02:53 | |
| *** pmathews has joined #openstack-infra | 02:55 | |
| *** matsuhashi has joined #openstack-infra | 02:55 | |
| *** sarob has quit IRC | 02:57 | |
| *** pcrews has joined #openstack-infra | 02:57 | |
| *** sarob has joined #openstack-infra | 02:57 | |
| *** thomasem has joined #openstack-infra | 02:59 | |
| *** thomasem has quit IRC | 02:59 | |
| *** e0ne has joined #openstack-infra | 02:59 | |
| *** bhuvan has quit IRC | 03:00 | |
| *** talluri has joined #openstack-infra | 03:00 | |
| zhiwei | fungi: hi | 03:01 |
| *** talluri has quit IRC | 03:02 | |
| *** talluri has joined #openstack-infra | 03:02 | |
| *** sarob has quit IRC | 03:02 | |
| *** e0ne has quit IRC | 03:04 | |
| *** sarob has joined #openstack-infra | 03:04 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: annotate stages of script https://review.openstack.org/75705 | 03:05 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: calculate remaining timeout locally https://review.openstack.org/75726 | 03:05 |
| *** talluri has quit IRC | 03:06 | |
| *** wchrisj has joined #openstack-infra | 03:07 | |
| *** jhesketh__ has quit IRC | 03:08 | |
| *** jhesketh__ has joined #openstack-infra | 03:09 | |
| *** weshay has quit IRC | 03:12 | |
| lifeless | oh wow | 03:12 |
| lifeless | nodepool is now failing thusly: | 03:12 |
| lifeless | ERROR: Quota exceeded for instances: Requested 1, but already used 100 of 100 instances (HTTP 413) (Request-ID: req-0c96b3bc-dc37-4685-8227-02398b3bea6b) | 03:12 |
| lifeless | nova list | wc -l | 03:12 |
| lifeless | 42 | 03:12 |
| lifeless | so nova quota stats are -way- off | 03:12 |
| lifeless | we haven't done any DB surgery at any point, so this is bugs | 03:13 |
| *** jcooley_ has quit IRC | 03:16 | |
| *** wchrisj has quit IRC | 03:18 | |
| *** wenlock has joined #openstack-infra | 03:19 | |
| lifeless | https://bugs.launchpad.net/nova/+bug/1284424 | 03:19 |
| *** lnxnut has joined #openstack-infra | 03:19 | |
| *** vkozhukalov_ has joined #openstack-infra | 03:20 | |
| fungi | clarkb: really turned out just to be the two old zuul servers. alone they wont be enough, but along with deleting the first elasticsearch server it should be possible | 03:23 |
| fungi | zhiwei: hi there | 03:23 |
| *** matsuhashi has quit IRC | 03:23 | |
| lifeless | ah, compute agents are down. stab. | 03:24 |
| lifeless | https://bugs.launchpad.net/tripleo/+bug/1284431 | 03:24 |
| zhiwei | fungi: can you tell me the status of renaming the stackforge project? | 03:24 |
| lifeless | dstufft: around ? | 03:27 |
| *** pmathews has quit IRC | 03:32 | |
| lifeless | fungi: can you perhaps tickle nodepool? we had another outage, as listed above, everything is back but I'm not seeing guests | 03:33 |
| *** nati_ueno has quit IRC | 03:36 | |
| *** lcheng has joined #openstack-infra | 03:38 | |
| *** morganfainberg is now known as morganfainberg_Z | 03:39 | |
| *** rcleere has joined #openstack-infra | 03:40 | |
| *** sdake__ has joined #openstack-infra | 03:40 | |
| *** sdake__ has quit IRC | 03:40 | |
| *** sdake__ has joined #openstack-infra | 03:40 | |
| *** sdake_ has quit IRC | 03:42 | |
| *** basha has joined #openstack-infra | 03:45 | |
| fungi | lifeless: i see a bunch in a building state for around half an hour. are those illegitimate? i can delete them if so | 03:45 |
| openstackgerrit | Lin Tan proposed a change to openstack-infra/zuul: Use six.moves.urllib.parse instead of urlparse https://review.openstack.org/72824 | 03:46 |
| *** CaptTofu has quit IRC | 03:46 | |
| basha | hey I have a question around failing builds in https://review.openstack.org/#/c/23424/14 | 03:48 |
| fungi | zhiwei: i had to refresh my memory from last week's meeting minutes. we're waiting for official or incubating openstack projects to need renames, and will batch stackforge project renames up with those. it seems like savanna may be ready to rename soon--they've sent their name options off to the foundation for a trademark search already | 03:49 |
| clarkb | basha: go for it | 03:50 |
| clarkb | fungi: :/ about needing more room. We can defnitely start with teh first 5 then spin up the 6th after we free up some of the old es servers | 03:50 |
| basha | clarkb: if you look at the python 26, 27 logs, theres just one test running and failing. SUrprisingly its passing in the local boxes here | 03:51 |
| basha | also clarkb fromn the logs it looks like its failing somewhere inside the library | 03:51 |
| clarkb | mock is failing | 03:51 |
| clarkb | there should be a pip freeze at the end of the lgs showing you the version of mock used | 03:52 |
| clarkb | you will want to make sure that the same version is used in local testing | 03:52 |
| clarkb | also the test orders may be different locally and in jenkins | 03:52 |
| fungi | clarkb: tomorrow i'll put the fake dns rr back for 06 so we can merge that puppet change | 03:52 |
| clarkb | fungi: sounds good | 03:52 |
| clarkb | basha: I would try running just that test locally and see if it fails in isolation | 03:53 |
| basha | clarkb: it didnt | 03:54 |
| clarkb | basha: and the mock versions match? | 03:54 |
| basha | clarkb: you mean the requirements? yyup | 03:55 |
| *** lnxnut has quit IRC | 03:55 | |
| clarkb | not the requirements, the actual installed versions | 03:55 |
| *** lnxnut has joined #openstack-infra | 03:56 | |
| basha | yes thats what I meant | 03:56 |
| basha | :) | 03:56 |
| fungi | lifeless: on the assumption that new node builds should take less than half an hour, i'm deleting any in the state for that long | 03:56 |
| *** pcrews has quit IRC | 03:56 | |
| clarkb | basha: ok, so the other possibility as far as test order goes is that if some other test runs first it breaks that test | 03:57 |
| clarkb | basha: what you can do is grab the subunit log, load that into testr locally `testr load subunutlogfile`, then do `testr run --analyze-isolation` and it will do test bisection and try to determine why that test failed | 03:57 |
| basha | clarkb: sure will try that | 03:58 |
| basha | Thanks :) | 03:58 |
| *** e0ne has joined #openstack-infra | 03:59 | |
| *** markmcclain has joined #openstack-infra | 03:59 | |
| *** lnxnut has quit IRC | 04:00 | |
| * fungi offlines. it's getting late | 04:00 | |
| clarkb | basha: the other thing to note is that zuul will merge that change atop master before testing it | 04:00 |
| *** rcleere has quit IRC | 04:00 | |
| clarkb | basha: if you haven't done that locally it may explain the failure | 04:00 |
| clarkb | fungi: good night | 04:00 |
| *** emagana has quit IRC | 04:01 | |
| *** talluri has joined #openstack-infra | 04:03 | |
| lifeless | fungi: ok, bunch errors, but we do have nodes now Ithink | 04:03 |
| *** jp_at_hp1 has quit IRC | 04:03 | |
| *** rcleere has joined #openstack-infra | 04:04 | |
| *** markmcclain has quit IRC | 04:04 | |
| *** e0ne has quit IRC | 04:04 | |
| *** talluri has quit IRC | 04:07 | |
| *** jcooley_ has joined #openstack-infra | 04:08 | |
| *** sarob has quit IRC | 04:08 | |
| *** sarob has joined #openstack-infra | 04:09 | |
| basha | clarkb: hmmmm, i thought I'd rebased it with master. but I'll check once more | 04:09 |
| *** pmathews has joined #openstack-infra | 04:11 | |
| *** lnxnut has joined #openstack-infra | 04:11 | |
| dstufft | lifeless: I am now | 04:11 |
| lifeless | dstufft: hey cool, wondering if you can fix the bad loop issue I reported on distutils-sig | 04:11 |
| lifeless | dstufft: since it really wigs bandersnatch out :) | 04:12 |
| *** harlowja is now known as harlowja_away | 04:12 | |
| *** sarob has quit IRC | 04:13 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 04:13 |
| *** jeckersb is now known as jeckersb_gone | 04:14 | |
| *** pcrews has joined #openstack-infra | 04:15 | |
| openstackgerrit | Edward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better https://review.openstack.org/51425 | 04:16 |
| *** lnxnut has quit IRC | 04:16 | |
| dstufft | lifeless: probably someone renamed something | 04:16 |
| dstufft | which means a redirect got cached | 04:17 |
| lifeless | dstufft: for how long would such a cache persist? | 04:18 |
| lifeless | dstufft: note that it redirects *to itself* | 04:18 |
| *** wchrisj has joined #openstack-infra | 04:21 | |
| *** tjones has joined #openstack-infra | 04:21 | |
| dstufft | lifeless: normally 24 hours | 04:21 |
| dstufft | should be fixed now | 04:21 |
| lifeless | dstufft: thanks! | 04:22 |
| openstackgerrit | Edward Raigosa proposed a change to openstack-infra/config: fix some small syntax issues in puppet manifest. https://review.openstack.org/72496 | 04:23 |
| lifeless | now to stab this a few times until it does a run without timeouts | 04:23 |
| lifeless | and I should stop putting nuts load on the server | 04:23 |
| *** david_lyle_ has joined #openstack-infra | 04:23 | |
| *** pcrews has quit IRC | 04:24 | |
| lifeless | sigh there's always one package that times out | 04:26 |
| *** lnxnut has joined #openstack-infra | 04:26 | |
| *** ryanpetrello_ has quit IRC | 04:28 | |
| *** moted has joined #openstack-infra | 04:28 | |
| *** moted has quit IRC | 04:28 | |
| *** relaxdiego has joined #openstack-infra | 04:30 | |
| *** matsuhashi has joined #openstack-infra | 04:31 | |
| *** rcleere has quit IRC | 04:32 | |
| *** harlowja_away is now known as harlowja | 04:32 | |
| *** jcooley_ has quit IRC | 04:32 | |
| *** jcooley_ has joined #openstack-infra | 04:33 | |
| dstufft | lifeless: you know you can icnrease the timeout? | 04:33 |
| *** tjones has quit IRC | 04:36 | |
| *** sarob has joined #openstack-infra | 04:38 | |
| *** sdake__ has quit IRC | 04:39 | |
| lifeless | dstufft: are the defaults wrong? | 04:40 |
| *** lnxnut has quit IRC | 04:40 | |
| dstufft | lifeless: they might be for you :) | 04:40 |
| dstufft | I hardly ever get timeouts in bandersnatch | 04:41 |
| *** mayu has quit IRC | 04:41 | |
| *** jcooley_ has quit IRC | 04:41 | |
| zhiwei | fungi: thanks. | 04:41 |
| lifeless | dstufft: heh | 04:43 |
| lifeless | dstufft: so we're on a 10Gbps link | 04:43 |
| lifeless | dstufft: its not clear whats causing the timeouts :) | 04:43 |
| *** dkehn__ has joined #openstack-infra | 04:44 | |
| dstufft | lifeless: are you down under? or do I have you confused with someone else? | 04:45 |
| *** dkehn__ has quit IRC | 04:46 | |
| *** jcooley_ has joined #openstack-infra | 04:46 | |
| lifeless | I live in NZ, the server in question is in las vegas :) | 04:46 |
| *** dkehn__ has joined #openstack-infra | 04:46 | |
| *** dkehn has quit IRC | 04:47 | |
| *** dkehn_ has quit IRC | 04:47 | |
| dstufft | lifeless: ah | 04:48 |
| *** dkehn has joined #openstack-infra | 04:49 | |
| lifeless | I don't have 10gbps to the net in my house :) | 04:49 |
| *** talluri has joined #openstack-infra | 04:49 | |
| *** dkehn__ is now known as dkehn_ | 04:49 | |
| dstufft | :) | 04:51 |
| *** lcheng has quit IRC | 04:53 | |
| *** lcheng has joined #openstack-infra | 04:56 | |
| lifeless | dstufft: anyhow it finally got a complete run, and now has a proper serial, yay. | 04:57 |
| *** jcooley_ has quit IRC | 04:58 | |
| *** pmathews has quit IRC | 04:59 | |
| *** jcooley_ has joined #openstack-infra | 04:59 | |
| *** e0ne has joined #openstack-infra | 04:59 | |
| *** jcooley_ has quit IRC | 05:03 | |
| *** e0ne has quit IRC | 05:04 | |
| *** reed has joined #openstack-infra | 05:04 | |
| *** talluri has quit IRC | 05:05 | |
| *** jcooley_ has joined #openstack-infra | 05:11 | |
| *** amotoki has joined #openstack-infra | 05:18 | |
| *** sarob has quit IRC | 05:19 | |
| *** sarob has joined #openstack-infra | 05:20 | |
| *** talluri has joined #openstack-infra | 05:22 | |
| *** talluri has quit IRC | 05:22 | |
| *** talluri has joined #openstack-infra | 05:22 | |
| *** sarob has quit IRC | 05:24 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 05:32 |
| *** nicedice has quit IRC | 05:34 | |
| *** lnxnut has joined #openstack-infra | 05:36 | |
| *** lcheng has quit IRC | 05:38 | |
| *** chenxu has quit IRC | 05:39 | |
| *** lnxnut has quit IRC | 05:41 | |
| *** CaptTofu has joined #openstack-infra | 05:47 | |
| *** lcheng has joined #openstack-infra | 05:50 | |
| openstackgerrit | A change was merged to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 05:51 |
| *** CaptTofu has quit IRC | 05:52 | |
| *** coolsvap has joined #openstack-infra | 05:58 | |
| *** e0ne has joined #openstack-infra | 05:59 | |
| *** tian has joined #openstack-infra | 06:01 | |
| *** jcooley_ has quit IRC | 06:01 | |
| *** harlowja is now known as harlowja_away | 06:01 | |
| *** yaguang has joined #openstack-infra | 06:02 | |
| *** e0ne has quit IRC | 06:04 | |
| *** chandan_kumar has joined #openstack-infra | 06:07 | |
| *** nati_ueno has joined #openstack-infra | 06:10 | |
| *** wchrisj has quit IRC | 06:14 | |
| *** DinaBelova_ is now known as DinaBelova | 06:17 | |
| *** sarob has joined #openstack-infra | 06:20 | |
| *** saju_m has joined #openstack-infra | 06:21 | |
| openstackgerrit | A change was merged to openstack/requirements: Remove vim header https://review.openstack.org/75821 | 06:25 |
| *** sarob has quit IRC | 06:26 | |
| *** lnxnut has joined #openstack-infra | 06:26 | |
| *** reed has quit IRC | 06:26 | |
| *** yolanda_ has joined #openstack-infra | 06:27 | |
| *** saju_m has quit IRC | 06:29 | |
| *** ildikov_ has quit IRC | 06:29 | |
| *** lnxnut has quit IRC | 06:31 | |
| *** gyee has quit IRC | 06:42 | |
| *** saju_m has joined #openstack-infra | 06:42 | |
| *** vkozhukalov_ has quit IRC | 06:48 | |
| *** MarkAtwood has joined #openstack-infra | 06:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add Zuul Job Queue graph https://review.openstack.org/75706 | 06:49 |
| *** mrda is now known as mrda_away | 06:54 | |
| *** mkoderer has joined #openstack-infra | 06:54 | |
| *** e0ne has joined #openstack-infra | 06:59 | |
| *** yolanda_ has quit IRC | 06:59 | |
| *** CaptTofu has joined #openstack-infra | 07:00 | |
| *** denis_makogon has joined #openstack-infra | 07:01 | |
| *** jlibosva has joined #openstack-infra | 07:02 | |
| *** e0ne has quit IRC | 07:04 | |
| *** mwagner_wfh has quit IRC | 07:04 | |
| *** CaptTofu has quit IRC | 07:06 | |
| *** gokrokve has quit IRC | 07:06 | |
| *** skolekonov has joined #openstack-infra | 07:07 | |
| *** DinaBelova is now known as DinaBelova_ | 07:09 | |
| *** gokrokve has joined #openstack-infra | 07:09 | |
| *** marun has quit IRC | 07:11 | |
| *** gokrokve_ has joined #openstack-infra | 07:11 | |
| *** saju_m has quit IRC | 07:13 | |
| *** saju_m has joined #openstack-infra | 07:13 | |
| *** gokrokve has quit IRC | 07:14 | |
| *** afazekas has quit IRC | 07:17 | |
| *** sarob has joined #openstack-infra | 07:23 | |
| *** sabari has joined #openstack-infra | 07:25 | |
| *** lnxnut has joined #openstack-infra | 07:26 | |
| *** sarob has quit IRC | 07:30 | |
| *** lnxnut has quit IRC | 07:31 | |
| *** lcheng has quit IRC | 07:32 | |
| *** vkozhukalov_ has joined #openstack-infra | 07:37 | |
| *** jcooley_ has joined #openstack-infra | 07:44 | |
| *** zhiyan is now known as zhiyan_ | 07:48 | |
| *** rcarrillocruz1 has joined #openstack-infra | 07:53 | |
| *** pblaho has joined #openstack-infra | 07:53 | |
| *** flaper87|afk is now known as flaper87 | 07:54 | |
| *** rcarrillocruz has quit IRC | 07:55 | |
| *** salv-orlando_ has joined #openstack-infra | 07:56 | |
| *** salv-orlando has quit IRC | 07:59 | |
| *** salv-orlando_ is now known as salv-orlando | 07:59 | |
| *** e0ne has joined #openstack-infra | 07:59 | |
| *** sabari has quit IRC | 08:00 | |
| *** e0ne has quit IRC | 08:04 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Fix manila-tempest-job to use exported env var https://review.openstack.org/76115 | 08:06 |
| *** denis_makogon has quit IRC | 08:15 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 08:17 | |
| *** mrmartin has joined #openstack-infra | 08:17 | |
| *** rlandy has joined #openstack-infra | 08:19 | |
| *** malini_afk is now known as malini | 08:19 | |
| *** dizquierdo has joined #openstack-infra | 08:21 | |
| *** jcoufal has joined #openstack-infra | 08:21 | |
| *** malini has left #openstack-infra | 08:23 | |
| *** dstanek has quit IRC | 08:25 | |
| *** sarob has joined #openstack-infra | 08:26 | |
| *** MarkAtwood has quit IRC | 08:27 | |
| *** DinaBelova_ is now known as DinaBelova | 08:27 | |
| *** lnxnut has joined #openstack-infra | 08:27 | |
| *** gokrokve_ has quit IRC | 08:28 | |
| *** mwagner_lap has joined #openstack-infra | 08:28 | |
| *** lnxnut has quit IRC | 08:31 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 08:31 |
| *** sarob has quit IRC | 08:32 | |
| openstackgerrit | Fengqian.gao proposed a change to openstack-infra/jeepyb: Keep py3.X compatibility for urllib https://review.openstack.org/76122 | 08:34 |
| *** jcooley_ has quit IRC | 08:34 | |
| *** luqas has joined #openstack-infra | 08:35 | |
| *** masayukig has joined #openstack-infra | 08:36 | |
| *** luqas has quit IRC | 08:36 | |
| *** hashar has joined #openstack-infra | 08:37 | |
| *** ildikov_ has joined #openstack-infra | 08:38 | |
| *** afazekas has joined #openstack-infra | 08:38 | |
| *** jgallard has joined #openstack-infra | 08:39 | |
| openstackgerrit | Fengqian.gao proposed a change to openstack-infra/nodepool: Keep py3.X compatibility for urllib/urllib2 https://review.openstack.org/76124 | 08:40 |
| *** luqas has joined #openstack-infra | 08:41 | |
| *** DinaBelova is now known as DinaBelova_ | 08:44 | |
| *** vkozhukalov_ has quit IRC | 08:53 | |
| *** DinaBelova_ is now known as DinaBelova | 08:57 | |
| *** wenlock has quit IRC | 08:57 | |
| *** rossella-s has joined #openstack-infra | 08:58 | |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** salv-orlando has quit IRC | 08:59 | |
| *** CaptTofu has joined #openstack-infra | 09:02 | |
| *** CaptTofu has quit IRC | 09:06 | |
| *** protux has joined #openstack-infra | 09:09 | |
| *** jcooley_ has joined #openstack-infra | 09:10 | |
| *** e0ne has joined #openstack-infra | 09:11 | |
| *** yassine has joined #openstack-infra | 09:11 | |
| *** gokrokve has joined #openstack-infra | 09:12 | |
| *** gokrokve has quit IRC | 09:17 | |
| *** derekh has joined #openstack-infra | 09:17 | |
| *** zhiwei has quit IRC | 09:18 | |
| *** fbo_away is now known as fbo | 09:22 | |
| *** nati_ueno has quit IRC | 09:23 | |
| *** luqas has left #openstack-infra | 09:23 | |
| *** luqas has joined #openstack-infra | 09:25 | |
| *** lnxnut has joined #openstack-infra | 09:26 | |
| *** jcooley_ has quit IRC | 09:27 | |
| *** ociuhandu has quit IRC | 09:29 | |
| *** sarob has joined #openstack-infra | 09:30 | |
| *** ildikov_ has quit IRC | 09:30 | |
| *** lnxnut has quit IRC | 09:31 | |
| *** matsuhashi has quit IRC | 09:31 | |
| *** matsuhashi has joined #openstack-infra | 09:34 | |
| *** sarob has quit IRC | 09:36 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency https://review.openstack.org/62726 | 09:37 |
| *** zhiwei has joined #openstack-infra | 09:41 | |
| *** luqas has quit IRC | 09:47 | |
| *** jcoufal has quit IRC | 09:49 | |
| *** rossella-s has quit IRC | 09:50 | |
| *** salv-orlando has joined #openstack-infra | 09:50 | |
| *** yassine has quit IRC | 09:50 | |
| *** jp_at_hp has joined #openstack-infra | 09:51 | |
| *** salv-orlando_ has joined #openstack-infra | 09:54 | |
| *** davidhadas__ has quit IRC | 09:57 | |
| *** davidhadas_ has quit IRC | 09:57 | |
| *** zhiwei has quit IRC | 09:59 | |
| *** salv-orlando has quit IRC | 09:59 | |
| *** salv-orlando_ is now known as salv-orlando | 09:59 | |
| *** matsuhashi has quit IRC | 09:59 | |
| *** matsuhashi has joined #openstack-infra | 09:59 | |
| *** che-arne has joined #openstack-infra | 10:02 | |
| *** coolsvap has quit IRC | 10:06 | |
| *** gokrokve has joined #openstack-infra | 10:12 | |
| *** nosnos has quit IRC | 10:14 | |
| *** rcarrillocruz has joined #openstack-infra | 10:14 | |
| *** rossella has joined #openstack-infra | 10:16 | |
| *** rcarrillocruz1 has quit IRC | 10:17 | |
| *** matsuhashi has quit IRC | 10:17 | |
| *** gokrokve has quit IRC | 10:17 | |
| *** luqas has joined #openstack-infra | 10:19 | |
| *** protux has quit IRC | 10:20 | |
| *** david_lyle_ has quit IRC | 10:21 | |
| *** CaptTofu has joined #openstack-infra | 10:23 | |
| *** lnxnut has joined #openstack-infra | 10:26 | |
| *** salv-orlando has quit IRC | 10:29 | |
| *** vkozhukalov_ has joined #openstack-infra | 10:31 | |
| *** lnxnut has quit IRC | 10:31 | |
| *** sarob has joined #openstack-infra | 10:33 | |
| *** rlandy_ has joined #openstack-infra | 10:34 | |
| *** coolsvap has joined #openstack-infra | 10:36 | |
| *** rcarrillocruz1 has joined #openstack-infra | 10:36 | |
| *** boris-42_ has quit IRC | 10:37 | |
| *** rlandy has quit IRC | 10:38 | |
| *** rcarrillocruz has quit IRC | 10:38 | |
| *** sarob has quit IRC | 10:38 | |
| *** davidhadas has joined #openstack-infra | 10:39 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 10:50 | |
| *** rlandy_ has quit IRC | 10:51 | |
| *** yassine has joined #openstack-infra | 10:57 | |
| *** relaxdiego has quit IRC | 11:01 | |
| *** psedlak has quit IRC | 11:02 | |
| *** denis_makogon has joined #openstack-infra | 11:04 | |
| *** rlandy_ has joined #openstack-infra | 11:04 | |
| denis_makogon | hello, guys, could someone help me with gate failure explanation ? | 11:05 |
| denis_makogon | i'm seeing weired issue on jenkins gate http://logs.openstack.org/40/76140/1/check/gate-trove-python26/e449799/console.html (patchset https://review.openstack.org/#/c/76132/) | 11:05 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add script trigger. https://review.openstack.org/70971 | 11:06 |
| *** jp_at_hp has quit IRC | 11:08 | |
| *** dkranz has quit IRC | 11:09 | |
| denis_makogon | please, i really need help | 11:09 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: added logstash plugin support and fixed test typo https://review.openstack.org/70917 | 11:10 |
| *** dkranz has joined #openstack-infra | 11:12 | |
| *** jp_at_hp has joined #openstack-infra | 11:12 | |
| *** gokrokve has joined #openstack-infra | 11:12 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Add attachment pattern expression to email-ext. https://review.openstack.org/70148 | 11:12 |
| *** mrmartin has quit IRC | 11:14 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Content-Type can now be set for email-ext publisher https://review.openstack.org/75919 | 11:14 |
| *** gokrokve has quit IRC | 11:17 | |
| *** psedlak has joined #openstack-infra | 11:18 | |
| *** jhesketh has quit IRC | 11:21 | |
| *** jhesketh__ has quit IRC | 11:21 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin https://review.openstack.org/73601 | 11:22 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Test for email-ext publisher https://review.openstack.org/75872 | 11:25 |
| *** lnxnut has joined #openstack-infra | 11:26 | |
| *** jerryz has quit IRC | 11:26 | |
| *** lnxnut has quit IRC | 11:31 | |
| *** salv-orlando has joined #openstack-infra | 11:36 | |
| *** fifieldt has quit IRC | 11:36 | |
| *** sarob has joined #openstack-infra | 11:36 | |
| *** jgallard has quit IRC | 11:38 | |
| *** boris-42_ has joined #openstack-infra | 11:38 | |
| *** jcooley_ has joined #openstack-infra | 11:41 | |
| *** DinaBelova is now known as DinaBelova_ | 11:41 | |
| *** sarob has quit IRC | 11:42 | |
| *** hashar has quit IRC | 11:45 | |
| *** jcoufal has joined #openstack-infra | 11:46 | |
| *** jcooley_ has quit IRC | 11:48 | |
| zigo | mordred: Around? | 11:50 |
| zigo | p1otr uploaded SQLA 0.9 to Sid last Sunday, this broke python-migrate at least, and probably some more. | 11:50 |
| zigo | http://paste.debian.net/83949/ | 11:50 |
| zigo | I've tried fixing this, but I couldn't. | 11:50 |
| zigo | It'd be awesome if you can find out what the error is. | 11:51 |
| zigo | What's Sean Dague nickname? | 11:51 |
| sdague | zigo: what's up? | 11:54 |
| zigo | sdague: just read above! :) | 11:54 |
| * zigo is happy to have found sdague! | 11:54 | |
| sdague | I don't know the actual fix, I have a partial solution here - https://review.openstack.org/#/c/66156/ | 11:55 |
| sdague | but it's not the whole fix | 11:55 |
| zigo | sdague: I already include this fix in my code base! :) | 11:56 |
| sdague | yeh, well I haven't dug into the rest of it | 11:56 |
| zigo | sdague: https://code.google.com/p/sqlalchemy-migrate/issues/detail?id=127 <--- This one suggests to use return ret instead of return self.preparer.quote(ret, cons.quote) | 11:56 |
| zigo | It seems it works! :) | 11:56 |
| *** yamahata has quit IRC | 11:57 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 11:57 | |
| zigo | I don't understand why but it does. | 11:58 |
| *** DinaBelova_ is now known as DinaBelova | 12:00 | |
| openstackgerrit | Thomas Leaman proposed a change to openstack/requirements: Move to newer version of python-swiftclient https://review.openstack.org/76170 | 12:01 |
| sdague | so I'd be a little concerned about doing that blindly, because that assumes the value is already safely quoted | 12:02 |
| *** luqas has quit IRC | 12:03 | |
| *** salv-orlando has quit IRC | 12:03 | |
| zigo | sdague: If you have a better patch, then great! | 12:04 |
| sdague | though let me play with that idea | 12:04 |
| sdague | zigo: yeh, I'd just be concerned that you opened up a security hole by blindly dropping quoting | 12:04 |
| zigo | I got it. | 12:04 |
| *** markmc has joined #openstack-infra | 12:06 | |
| *** ArxCruz has joined #openstack-infra | 12:09 | |
| *** alexpilotti has joined #openstack-infra | 12:10 | |
| *** luqas has joined #openstack-infra | 12:11 | |
| *** luqas has quit IRC | 12:11 | |
| *** gokrokve has joined #openstack-infra | 12:12 | |
| *** che-arne has quit IRC | 12:14 | |
| *** alexpilotti has quit IRC | 12:15 | |
| *** rlandy__ has joined #openstack-infra | 12:17 | |
| openstackgerrit | Fengqian.gao proposed a change to openstack-infra/nodepool: Keep py3.X compatibility for urllib/urllib2 https://review.openstack.org/76124 | 12:17 |
| *** gokrokve has quit IRC | 12:17 | |
| *** rlandy_ has quit IRC | 12:18 | |
| *** basha has quit IRC | 12:19 | |
| *** rlandy__ is now known as rlandy | 12:20 | |
| *** che-arne has joined #openstack-infra | 12:22 | |
| *** lnxnut has joined #openstack-infra | 12:26 | |
| *** lnxnut has quit IRC | 12:31 | |
| *** hashar has joined #openstack-infra | 12:35 | |
| *** jang has quit IRC | 12:35 | |
| *** hashar_ has joined #openstack-infra | 12:38 | |
| *** sarob has joined #openstack-infra | 12:39 | |
| *** hashar has quit IRC | 12:41 | |
| *** hashar_ has quit IRC | 12:45 | |
| *** hashar has joined #openstack-infra | 12:45 | |
| *** sarob has quit IRC | 12:46 | |
| *** sandywalsh has quit IRC | 12:48 | |
| *** alexpilotti has joined #openstack-infra | 12:50 | |
| *** yaguang has quit IRC | 12:56 | |
| *** ilyashakhat has quit IRC | 12:57 | |
| *** talluri has quit IRC | 12:59 | |
| *** lcostantino has joined #openstack-infra | 12:59 | |
| *** mgagne1 has joined #openstack-infra | 13:00 | |
| *** ilyashakhat has joined #openstack-infra | 13:00 | |
| *** sandywalsh has joined #openstack-infra | 13:01 | |
| *** mgagne has quit IRC | 13:03 | |
| *** coolsvap has quit IRC | 13:04 | |
| *** weshay has joined #openstack-infra | 13:05 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Campfire had a malformed doc syntax https://review.openstack.org/76185 | 13:05 |
| *** afazekas has quit IRC | 13:06 | |
| *** markmcclain has joined #openstack-infra | 13:06 | |
| *** luqas has joined #openstack-infra | 13:07 | |
| zigo | sdague: Here's my unit test logs now: http://paste.debian.net/83979/ | 13:11 |
| zigo | Any idea where the "Failed for mysql://openstack_citest:openstack_citest@localhost/openstack_citest" is comming from? | 13:11 |
| zigo | And about these other FAIL? | 13:11 |
| sdague | so you aren't testing against mysql there | 13:12 |
| sdague | test_db.cfg will give you the ability to set connect strings for mysql engines | 13:12 |
| sdague | and realize that if you don't test against mysql, a lot of it isn't tested, because sqlite doesn't implement foreign keys, which is where most of the issues are | 13:13 |
| zigo | sdague: I *do* test against MySQL. | 13:13 |
| sdague | hmmm | 13:13 |
| zigo | sdague: That's my shell script to do the unit tests: http://paste.debian.net/83980/ | 13:14 |
| zigo | :) | 13:14 |
| *** gokrokve has joined #openstack-infra | 13:14 | |
| sdague | yeh, so realitistically this probably a bigger thing than I can sift through for a while. Got to get back to some other things. | 13:15 |
| zigo | Ok, no worries. | 13:16 |
| *** smarcet has joined #openstack-infra | 13:17 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Use venv to build documentation https://review.openstack.org/76190 | 13:17 |
| *** andreaf has joined #openstack-infra | 13:18 | |
| *** gokrokve has quit IRC | 13:18 | |
| *** dstanek has joined #openstack-infra | 13:19 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Move requirement files to root of repository https://review.openstack.org/76192 | 13:22 |
| *** dizquierdo is now known as dizquierdo_afk | 13:23 | |
| *** afazekas has joined #openstack-infra | 13:23 | |
| *** rcarrillocruz1 is now known as rcarrillocruz | 13:23 | |
| *** CaptTofu has quit IRC | 13:24 | |
| *** lnxnut has joined #openstack-infra | 13:26 | |
| sdague | fungi / SergeyLukjanov: so we need this to go through soon - https://review.openstack.org/#/c/75719/ as we no longer put devstack logs in the console.html | 13:29 |
| SergeyLukjanov | sdague, hey | 13:29 |
| *** lnxnut has quit IRC | 13:31 | |
| *** jcooley_ has joined #openstack-infra | 13:31 | |
| *** exz has quit IRC | 13:31 | |
| *** exz has joined #openstack-infra | 13:31 | |
| *** dolphm_503 is now known as dolphm | 13:34 | |
| *** salv-orlando has joined #openstack-infra | 13:34 | |
| *** jgallard has joined #openstack-infra | 13:34 | |
| *** mbacchi has joined #openstack-infra | 13:36 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove old queries https://review.openstack.org/76196 | 13:37 |
| *** wenlock has joined #openstack-infra | 13:38 | |
| *** jcooley_ has quit IRC | 13:38 | |
| *** dkliban has quit IRC | 13:38 | |
| *** lcostantino has quit IRC | 13:38 | |
| openstackgerrit | Björn Michaelsen proposed a change to openstack-infra/git-review: add query option https://review.openstack.org/76197 | 13:39 |
| *** wenlock has quit IRC | 13:42 | |
| *** valentinbud has joined #openstack-infra | 13:42 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/76200 | 13:43 |
| valentinbud | Hello all. I have setup a Logstash, Elasticsearch, Kibana logging infrastructure using the openstack-infra/config repository. | 13:44 |
| fungi | valentinbud: neat! how's it working out for you? | 13:45 |
| valentinbud | The multiline Logstash filters for TRACE messages don't work. | 13:45 |
| valentinbud | I have Logstash 1.3.3 and Elasticsearch 1.0.0 on Ubuntu 14.04. | 13:45 |
| valentinbud | The system is processing logs from HyperV instances. | 13:46 |
| *** yassine has quit IRC | 13:47 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update comment filter for gerrit 2.8 https://review.openstack.org/70864 | 13:47 |
| valentinbud | fungi it works quite, I mean the system is stable and the logs are gathered and indexed by elasticsearch, but the filters aren't working though. | 13:49 |
| *** saju_m has quit IRC | 13:50 | |
| *** dizquierdo_afk is now known as dizquierdo | 13:51 | |
| *** dims has quit IRC | 13:51 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update comment filter for gerrit 2.8 https://review.openstack.org/70864 | 13:51 |
| fungi | valentinbud: interesting... i haven't used it to look for tracebacks so haven't noticed whether we get combined multiline records for those | 13:53 |
| *** rcleere has joined #openstack-infra | 13:54 | |
| fungi | valentinbud: have you looked for similar latterns on http://logstash.openstack.org/ to see whether it's working on ours? | 13:54 |
| *** afazekas has quit IRC | 13:54 | |
| fungi | s/latterns/patterns/ | 13:54 |
| *** dims has joined #openstack-infra | 13:54 | |
| *** dkranz has quit IRC | 13:54 | |
| *** pdmars has joined #openstack-infra | 13:55 | |
| *** pdmars has quit IRC | 13:55 | |
| *** pdmars has joined #openstack-infra | 13:55 | |
| *** dprince has joined #openstack-infra | 13:56 | |
| openstackgerrit | Martin Mágr proposed a change to openstack-infra/config: Watch also havana branch fro packstack https://review.openstack.org/76206 | 13:57 |
| *** mburned_` is now known as mburned | 14:00 | |
| valentinbud | fungi have searched for TRACE at logstash.openstack.org now. http://logstash.openstack.org/#eyJzZWFyY2giOiJUUkFDRSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MzMzNjgyNTEwMn0=. | 14:02 |
| *** lnxnut has joined #openstack-infra | 14:02 | |
| *** afazekas has joined #openstack-infra | 14:02 | |
| valentinbud | I don't know if the link takes you there, but if you search for TRACE you have events consisting of multiline TRACEs. | 14:03 |
| *** sandywalsh has quit IRC | 14:03 | |
| tristanC | Hello folks, I seek guidance to fix #1281540 (stable/havana gate job fails because of partial swift client upgrade) | 14:03 |
| fungi | valentinbud: ahh, yep. it looks like we are able to glob those together. not sure why it's not working in yours, but clarkb might have some ideas when he's awake in a few hours | 14:04 |
| fungi | tristanC: yeah, sorry, i did say yesterday i'd look into that, didn't i | 14:04 |
| valentinbud | Ok, I'll wait for him and pose him a question regarding this problem. Thank you. | 14:05 |
| tristanC | fungi: no worry :) Someone posted a patch for that but he abandoned it: https://review.openstack.org/75624 | 14:05 |
| fungi | tristanC: i believe we just need to make grenade not run the swift exercises in grizzly when testing havana changes | 14:05 |
| fungi | looking | 14:05 |
| tristanC | It basically does the same thing that have been done for grizzly (see https://review.openstack.org/#/c/74451/ ) | 14:05 |
| tristanC | fungi: thanks :) | 14:06 |
| fungi | tristanC: yeah, it's close. we only want to apply that in grenade though, i think, so i'll throw it in an else after the if [ "$DEVSTACK_GATE_GRENADE" -ne "1" ] | 14:06 |
| fungi | patch coming in moments | 14:06 |
| *** yassine has joined #openstack-infra | 14:09 | |
| *** boris-42_ has quit IRC | 14:10 | |
| *** mgagne1 is now known as mgagne | 14:10 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/76200 | 14:10 |
| *** salv-orlando has quit IRC | 14:10 | |
| *** pmathews has joined #openstack-infra | 14:10 | |
| *** salv-orlando has joined #openstack-infra | 14:10 | |
| *** wchrisj has joined #openstack-infra | 14:11 | |
| *** nijaba has joined #openstack-infra | 14:11 | |
| *** nijaba has joined #openstack-infra | 14:11 | |
| *** gokrokve has joined #openstack-infra | 14:12 | |
| *** lcostantino has joined #openstack-infra | 14:12 | |
| *** jeckersb_gone is now known as jeckersb | 14:13 | |
| *** ociuhandu has joined #openstack-infra | 14:13 | |
| *** salv-orlando_ has joined #openstack-infra | 14:14 | |
| *** pmathews has quit IRC | 14:15 | |
| *** markmc has quit IRC | 14:16 | |
| *** salv-orlando has quit IRC | 14:16 | |
| *** changbl has quit IRC | 14:17 | |
| *** gokrokve has quit IRC | 14:17 | |
| *** sandywalsh has joined #openstack-infra | 14:18 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Skip Swift exercises on Grizzly for Havana Grenade https://review.openstack.org/75624 | 14:19 |
| fungi | tristanC: ^ | 14:19 |
| *** markmc has joined #openstack-infra | 14:19 | |
| fungi | oh, my indenting is full of fail. fixing | 14:19 |
| *** salv-orlando_ is now known as salv-orlando | 14:20 | |
| tristanC | fungi: you rock sir! :) | 14:20 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Skip Swift exercises on Grizzly for Havana Grenade https://review.openstack.org/75624 | 14:21 |
| fungi | better | 14:22 |
| *** jraim has quit IRC | 14:22 | |
| *** jraim_ has joined #openstack-infra | 14:22 | |
| *** beagles is now known as beagles_brb | 14:24 | |
| *** pblaho has quit IRC | 14:24 | |
| *** yamahata has joined #openstack-infra | 14:24 | |
| *** thomasem has joined #openstack-infra | 14:25 | |
| *** prad_ has joined #openstack-infra | 14:27 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/76200 | 14:28 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Move requirement files to root of repository https://review.openstack.org/76192 | 14:28 |
| *** jcooley_ has joined #openstack-infra | 14:28 | |
| hashar | rebasing is a good spam system | 14:28 |
| *** sandywalsh has quit IRC | 14:29 | |
| fungi | hashar: especially if you stack 30 changes as dependencies and then rebase the bottom | 14:29 |
| *** sarob has joined #openstack-infra | 14:29 | |
| fungi | though i tend to prefer the mergespam we get when a big stack of changes all actually make it through the gate. much more satisfying | 14:30 |
| *** persia has quit IRC | 14:30 | |
| hashar | yeah those one are lovely | 14:30 |
| openstackgerrit | ZhiQiang Fan proposed a change to openstack/requirements: Upgrade six to 1.5.2 https://review.openstack.org/68424 | 14:31 |
| *** dkliban has joined #openstack-infra | 14:31 | |
| *** persia has joined #openstack-infra | 14:31 | |
| *** lnxnut has quit IRC | 14:31 | |
| *** miqui has joined #openstack-infra | 14:31 | |
| *** persia is now known as Guest63393 | 14:31 | |
| fungi | sdague: i'm noticing that of the 14 backward-compat jobs we run against d-g changes, we don't have one which makes sure stable grenade still works... do you think it's warranted? | 14:31 |
| sdague | yeh, though we're about to drop grizzly, right? | 14:32 |
| fungi | sdague: we are, in 9 days, but we'll have a new stable grenade soon, right? ;) | 14:32 |
| sdague | yeh, stable grenade should be in there | 14:33 |
| fungi | sdague: the only reason i notice it is that i'm looking at the jobs for 75624 which should fix grenade on grizzly for the next week, but we don't actually have a job to confirm it works | 14:33 |
| sdague | yep | 14:34 |
| fungi | something to keep in mind for the future, where we all have time to make things work correctly ;) | 14:34 |
| sdague | when is that going to be again? :) | 14:35 |
| fungi | i never can remember if we measure that in geological time scales or cosmic time scales | 14:35 |
| *** julim has joined #openstack-infra | 14:36 | |
| *** jcooley_ has quit IRC | 14:36 | |
| *** salv-orlando has quit IRC | 14:38 | |
| *** flaper87 is now known as flaper87|afk | 14:39 | |
| *** yamahata has quit IRC | 14:39 | |
| *** sarob has quit IRC | 14:41 | |
| *** sandywalsh has joined #openstack-infra | 14:42 | |
| *** ok_delta has joined #openstack-infra | 14:42 | |
| *** yamahata has joined #openstack-infra | 14:43 | |
| *** zehicle_at_dell has quit IRC | 14:43 | |
| *** lnxnut has joined #openstack-infra | 14:48 | |
| *** lnxnut has quit IRC | 14:48 | |
| *** katyafervent_awa has quit IRC | 14:49 | |
| *** miqui_ has joined #openstack-infra | 14:50 | |
| *** eharney has joined #openstack-infra | 14:52 | |
| *** miqui has quit IRC | 14:52 | |
| *** dkranz has joined #openstack-infra | 14:52 | |
| *** lnxnut has joined #openstack-infra | 14:53 | |
| *** yolanda_ has joined #openstack-infra | 14:53 | |
| *** valentinbud has quit IRC | 14:54 | |
| *** valentinbud has joined #openstack-infra | 14:54 | |
| *** markmcclain has quit IRC | 14:56 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/jeepyb: Make welcome-message --verbose work like -v https://review.openstack.org/76222 | 14:56 |
| *** katyafervent_awa has joined #openstack-infra | 14:58 | |
| *** beagles_brb is now known as beagles | 14:59 | |
| *** spredzy_ is now known as spredzy | 14:59 | |
| *** ryanpetrello has joined #openstack-infra | 15:00 | |
| *** Guest50062 is now known as sbadia | 15:01 | |
| *** tjones has joined #openstack-infra | 15:01 | |
| *** johnthetubaguy has joined #openstack-infra | 15:01 | |
| *** gokrokve has joined #openstack-infra | 15:02 | |
| *** bknudson1 has joined #openstack-infra | 15:02 | |
| *** bknudson has quit IRC | 15:05 | |
| *** bknudson1 has quit IRC | 15:07 | |
| *** skolekonov has quit IRC | 15:07 | |
| *** bknudson has joined #openstack-infra | 15:10 | |
| openstackgerrit | A change was merged to openstack-infra/config: Clean up puppet (deploy LAMP / setup app config) https://review.openstack.org/69636 | 15:16 |
| *** igor_ has joined #openstack-infra | 15:17 | |
| *** pdmars has quit IRC | 15:17 | |
| fungi | smarcet: ^ my --noop tests of that looked okay, but that didn't include executing any of the scripts so you'll want to make sure it's deploying as you and mrmartin intend | 15:18 |
| smarcet | fungi: ok cool, i will chat with mrmartin about it, tnx | 15:19 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Auth support https://review.openstack.org/73219 | 15:22 |
| fungi | sdague: are you cool with approving 75624 to un-wedge stable/grizzly security fixes? | 15:24 |
| fungi | er, i mean stable/havana security fixes | 15:24 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Make welcome-message --verbose work like -v https://review.openstack.org/76222 | 15:24 |
| sdague | let me look | 15:24 |
| *** CaptTofu has joined #openstack-infra | 15:25 | |
| fungi | very easy review | 15:25 |
| sdague | sure | 15:25 |
| openstackgerrit | Sahdev Zala proposed a change to openstack-infra/config: New StackForge project heat-translator https://review.openstack.org/75988 | 15:25 |
| sdague | though... you sure that actually fixes things | 15:26 |
| sdague | I thought that just made the fail happen later | 15:26 |
| *** rcleere has quit IRC | 15:26 | |
| *** boris-42_ has joined #openstack-infra | 15:26 | |
| valentinbud | clarkb: are you around? | 15:26 |
| *** david_lyle_ has joined #openstack-infra | 15:26 | |
| fungi | sdague: oh, maybe? yeah without a backward-compat job to actually run it, this is probably the fastest way to find out | 15:27 |
| sdague | yep | 15:27 |
| fungi | sdague: i guess we may end up needing to skip tempest tests for swift in havana on grenade jobs too? | 15:27 |
| *** david_lyle_ is now known as david_lyle | 15:27 | |
| *** boris-42_ has quit IRC | 15:27 | |
| sdague | well, actually we'd have to disable swift | 15:28 |
| fungi | at least, any that end up interacting with swiftclient | 15:28 |
| sdague | because I think that glance backs to swift in those tests | 15:28 |
| fungi | oh, runs that deep huh? | 15:28 |
| sdague | yes | 15:28 |
| *** afazekas has quit IRC | 15:28 | |
| sdague | it's integration tests | 15:28 |
| fungi | glance uses swiftclient to interact with swift i guess | 15:28 |
| sdague | anyway, we'll see | 15:28 |
| *** basha has joined #openstack-infra | 15:28 | |
| *** markmcclain has joined #openstack-infra | 15:28 | |
| fungi | sure, this is a cheap way to find out what else we're missing | 15:28 |
| *** markmc has quit IRC | 15:29 | |
| sdague | yep | 15:29 |
| *** CaptTofu has quit IRC | 15:30 | |
| *** sandywalsh has quit IRC | 15:30 | |
| *** coolsvap has joined #openstack-infra | 15:30 | |
| *** david_lyle_ has joined #openstack-infra | 15:31 | |
| *** mrodden has joined #openstack-infra | 15:31 | |
| *** sarob has joined #openstack-infra | 15:34 | |
| *** salv-orlando has joined #openstack-infra | 15:34 | |
| *** salv-orlando has quit IRC | 15:34 | |
| *** david_lyle has quit IRC | 15:35 | |
| *** david-lyle is now known as dklyle | 15:35 | |
| *** david_lyle_ is now known as david_lyle | 15:35 | |
| *** miqui_ is now known as miqui | 15:36 | |
| *** david_lyle_ has joined #openstack-infra | 15:36 | |
| *** VINOD has joined #openstack-infra | 15:37 | |
| *** wenlock has joined #openstack-infra | 15:38 | |
| *** david_lyle has quit IRC | 15:39 | |
| *** david-lyle has joined #openstack-infra | 15:41 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Updated from global requirements https://review.openstack.org/73119 | 15:41 |
| *** davidhadas_ has joined #openstack-infra | 15:42 | |
| *** sarob has quit IRC | 15:42 | |
| *** relaxdiego has joined #openstack-infra | 15:43 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 15:43 | |
| *** sarob has joined #openstack-infra | 15:43 | |
| *** cody-somerville has quit IRC | 15:44 | |
| *** davidhadas has quit IRC | 15:45 | |
| *** david_lyle_ has quit IRC | 15:45 | |
| *** sandywalsh has joined #openstack-infra | 15:46 | |
| *** CaptTofu has joined #openstack-infra | 15:46 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Make sure lvm2 tools are installed https://review.openstack.org/76244 | 15:47 |
| fungi | something else ^ missing from rackspace's ubuntu 12.10 pvhvm images | 15:48 |
| fungi | er, 12.04 | 15:48 |
| *** basha has quit IRC | 15:49 | |
| *** amotoki_ has joined #openstack-infra | 15:51 | |
| *** atiwari has joined #openstack-infra | 15:52 | |
| *** mfer has joined #openstack-infra | 15:52 | |
| valentinbud | fungi: what OS is OpenStack Logs sitting on? | 15:52 |
| valentinbud | Ubuntu 12.04 or 12.10? | 15:52 |
| fungi | valentinbud: if you're still talking about logstash, kibana and elasticsearch we're running it all on ubuntu 12.04 lts (precise pangolin) | 15:53 |
| fungi | though actually the server where we're hosting our static log content also runs on the same ubuntu version | 15:53 |
| *** rcleere has joined #openstack-infra | 15:54 | |
| valentinbud | fungi: yes, about that. I am trying to identify potential issues with our deployment. In our case the system is on Ubuntu 14.04. | 15:54 |
| fungi | valentinbud: ahh, 14.04 lts hasn't been released yet, so we haven't tried it | 15:55 |
| fungi | valentinbud: also, all our servers run in public cloud providers who don't have snapshot images for 14.04 (as far as i've seen) and don't provide a way for us to upload our own base images | 15:56 |
| fungi | so to test it out that way we'd probably have to do an in-place upgrade from a released ubuntu version anyway | 15:56 |
| *** reed has joined #openstack-infra | 15:57 | |
| *** jgrimm has joined #openstack-infra | 15:59 | |
| *** reed has quit IRC | 15:59 | |
| *** sarob has quit IRC | 15:59 | |
| *** DinaBelova is now known as DinaBelova_ | 15:59 | |
| *** jcoufal has quit IRC | 15:59 | |
| valentinbud | Thanks fungi. I am going to test it on 12.04.4. On Debian Wheezy I get the same behaviour. | 16:00 |
| *** jcoufal has joined #openstack-infra | 16:02 | |
| *** skraynev is now known as skraynev_afk | 16:02 | |
| *** luqas has quit IRC | 16:02 | |
| *** pmathews has joined #openstack-infra | 16:03 | |
| *** sandywalsh has quit IRC | 16:04 | |
| *** BobBall is now known as BobBallAway | 16:04 | |
| openstackgerrit | sahid proposed a change to openstack-infra/config: Adds new project Warm https://review.openstack.org/76247 | 16:04 |
| openstackgerrit | Alexei Kornienko proposed a change to openstack/requirements: Added argcomplete requirement https://review.openstack.org/65863 | 16:05 |
| *** lcheng has joined #openstack-infra | 16:07 | |
| valentinbud | fungi: I don't know why Ubuntu 14.04 was chosen. I am working on the HyperV CI Team and have received the OS pre installed and the task to get the logs up and running. | 16:08 |
| *** mgagne has quit IRC | 16:08 | |
| *** changbl has joined #openstack-infra | 16:10 | |
| *** sergmelikyan has joined #openstack-infra | 16:13 | |
| sergmelikyan | Guys, can someone help me to debug issue with gate-job? | 16:13 |
| sergmelikyan | https://jenkins07.openstack.org/job/gate-murano-conductor-python26/3/console | 16:13 |
| sergmelikyan | Tests running perfectly on local machine but hang on gate | 16:14 |
| sergmelikyan | No idea why it may happen :(\ | 16:14 |
| dhellmann | good morning | 16:14 |
| sergmelikyan | Morning, Doug | 16:15 |
| sergmelikyan | Looks like hangs code related to running RPC server | 16:17 |
| sergmelikyan | dhellmann, did you encountered something like this? | 16:17 |
| sergmelikyan | https://review.openstack.org/#/c/75892/5/muranoconductor/app.py | 16:17 |
| dhellmann | sergmelikyan: looking | 16:18 |
| * sergmelikyan has no idea why test is looking like he is looking, but actual running code is a simple RPC server | 16:18 | |
| *** jcooley_ has joined #openstack-infra | 16:18 | |
| *** pcrews has joined #openstack-infra | 16:19 | |
| dhellmann | sergmelikyan: so the tests hang but work for you locally? | 16:20 |
| *** thedodd has joined #openstack-infra | 16:20 | |
| sergmelikyan | Yes | 16:20 |
| *** thedodd has quit IRC | 16:20 | |
| sergmelikyan | I am running tests locally by 'tox -e py27' command | 16:20 |
| *** sandywalsh has joined #openstack-infra | 16:20 | |
| dhellmann | do the unit tests start a copy of the server? | 16:22 |
| dhellmann | jeblair, fungi, clarkb: if you have a few minutes, I'd appreciate some feedback on the gating approach for oslotest - https://review.openstack.org/#/c/74408/ | 16:23 |
| *** sahumada has joined #openstack-infra | 16:24 | |
| *** jlibosva has quit IRC | 16:24 | |
| *** dhellmann is now known as dhellmann_ | 16:24 | |
| sergmelikyan | dhellmann, yes | 16:24 |
| *** rcarrillocruz1 has joined #openstack-infra | 16:25 | |
| *** rcarrillocruz has quit IRC | 16:26 | |
| jeblair | dhellmann_: so actually i think we should use the devstack-gate approach for this. most of what d-g does is actually setting up the git repos correctly for cross-project gating | 16:27 |
| *** mrodden has quit IRC | 16:27 | |
| *** sabari has joined #openstack-infra | 16:27 | |
| *** markmcclain has quit IRC | 16:28 | |
| *** dizquierdo has quit IRC | 16:28 | |
| *** dizquierdo has joined #openstack-infra | 16:28 | |
| *** DinaBelova_ is now known as DinaBelova | 16:28 | |
| jeblair | dhellmann_: probably the easiest way to do this at the moment is to do something like the swift functional test job | 16:29 |
| jeblair | dhellmann_: where we use the d-g scripts to set everything up and then use the hooks in that job to actually run the tests | 16:29 |
| *** jergerber has joined #openstack-infra | 16:30 | |
| jeblair | dhellmann_: so, at a really high level, remove the git repo setup stuff from run_cross_tests, and just leave the bits where you install the projects and run tests | 16:30 |
| jeblair | dhellmann_: then you call run_cross_tests from the devstack-gate gate_hook | 16:30 |
| *** branen has quit IRC | 16:30 | |
| jeblair | er, actually post_test_hook | 16:30 |
| *** pdmars has joined #openstack-infra | 16:31 | |
| jeblair | dhellmann_: does that make sense (at a high level)? | 16:31 |
| *** pdmars has quit IRC | 16:31 | |
| *** mgagne has joined #openstack-infra | 16:31 | |
| *** thuc has joined #openstack-infra | 16:32 | |
| *** pdmars has joined #openstack-infra | 16:32 | |
| *** dizquierdo has quit IRC | 16:36 | |
| *** gyee has joined #openstack-infra | 16:37 | |
| *** UtahDave has joined #openstack-infra | 16:37 | |
| jeblair | dhellmann_: left comment in review; i can work through the details with you if you want. | 16:38 |
| *** luqas has joined #openstack-infra | 16:39 | |
| *** mrodden has joined #openstack-infra | 16:41 | |
| *** markwash has joined #openstack-infra | 16:42 | |
| fungi | jeblair: types faster than i do. now to see if i left the same comments on it | 16:42 |
| *** mwagner_lap has quit IRC | 16:42 | |
| fungi | yep... *sigh* | 16:43 |
| anteaya | we just had an az-2 node go offline: https://jenkins01.openstack.org/job/gate-neutron-python27/7770/console | 16:43 |
| *** wchrisj has quit IRC | 16:43 | |
| anteaya | no slave log and it looks like the same java error jog0 was talking about last night | 16:44 |
| *** tjones has quit IRC | 16:44 | |
| anteaya | the java error is that there is no slave log | 16:44 |
| *** wchrisj has joined #openstack-infra | 16:45 | |
| anteaya | neutron patch https://review.openstack.org/#/c/67556/ currently in the gate | 16:45 |
| jeblair | jenkins is constantly finding new ways to annoy us | 16:45 |
| anteaya | it seems | 16:45 |
| *** rcarrillocruz has joined #openstack-infra | 16:46 | |
| *** rcarrillocruz1 has quit IRC | 16:46 | |
| fungi | jeblair: is this another incidence of the possible gearman job deregistering issue you were suggesting yesterday? | 16:47 |
| jeblair | fungi: yeah. my initial thought is that either the deregistration is having an adverse impact. or possibly it's just coincidence and just otherwise somehow related to job start (lots of things are going on then). | 16:48 |
| fungi | have we seen it anywhere besides hpcloud-az2 i wonder... | 16:48 |
| fungi | we already know we have some stability issues in there | 16:48 |
| *** VINOD has quit IRC | 16:49 | |
| *** wchrisj has quit IRC | 16:49 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Skip Swift exercises on Grizzly for Havana Grenade https://review.openstack.org/75624 | 16:51 |
| jeblair | oh good question | 16:51 |
| *** wchrisj has joined #openstack-infra | 16:52 | |
| jeblair | fungi: mostly az2 and a little az1 | 16:53 |
| fungi | tristanC: now that's ^ merged, you probably want to try rechecking one of your havana backports and see if/where it fails again | 16:53 |
| *** chandankumar_ has joined #openstack-infra | 16:53 | |
| jeblair | 2 out of 17 hits in last 24 hours are az1, 15 are az2 | 16:53 |
| *** VINOD has joined #openstack-infra | 16:53 | |
| *** cody-somerville has joined #openstack-infra | 16:54 | |
| *** chandan_kumar has quit IRC | 16:54 | |
| fungi | i smell a wumpus | 16:54 |
| jeblair | if there's one thing i don't miss about nc it's smelling wumpuses | 16:54 |
| anteaya | ha ha ha | 16:54 |
| tristanC | fungi: yep thanks, was watching the gate already ;) | 16:55 |
| VINOD | hi | 16:55 |
| VINOD | my name is vinod and i am currently working at CERN | 16:55 |
| VINOD | i have uploaded code called Domain Quota API driver to Gerrit | 16:55 |
| VINOD | i can see that some of the tests are failing | 16:56 |
| VINOD | the gerrit page is | 16:56 |
| VINOD | https://review.openstack.org/#/c/76256/ | 16:56 |
| VINOD | some of the tests failing are | 16:56 |
| VINOD | check-tempest-dsvm-cells-full | 16:56 |
| VINOD | check-tempest-dsvm-neutron-heat-slow | 16:57 |
| VINOD | i have no idea why these are failing as i have added the code to nova to manage the quota | 16:57 |
| *** oubiwann_ has joined #openstack-infra | 16:57 | |
| *** tjones has joined #openstack-infra | 16:57 | |
| VINOD | can anybody help me in this | 16:57 |
| anteaya | hi VINOD | 16:57 |
| VINOD | even a test "gate-nova-pep8" failing saying that the nova.conf.sample is out of date | 16:58 |
| anteaya | that patch is still running it's check tests | 16:58 |
| anteaya | once the check tests are finished the logs for the tests will be posted to the patch | 16:58 |
| VINOD | you mean jenkins will do a re-check | 16:58 |
| *** valentinbud has left #openstack-infra | 16:58 | |
| anteaya | no | 16:58 |
| VINOD | yes, i have got logs | 16:58 |
| anteaya | jenkins will post the urls for the test logs | 16:58 |
| anteaya | so after the test logs are posted to the patch | 16:58 |
| VINOD | but i could not understand the log error is by any means related to my code | 16:58 |
| VINOD | for example even a test "gate-nova-pep8" failing saying that the nova.conf.sample is out of date | 16:59 |
| anteaya | let's wait until the logs are posted to the patch | 16:59 |
| anteaya | it is easier then for me to go through them with you | 16:59 |
| *** valentinbud has joined #openstack-infra | 16:59 | |
| *** boris-42_ has joined #openstack-infra | 17:00 | |
| *** valentinbud has quit IRC | 17:02 | |
| *** mriedem1 has joined #openstack-infra | 17:02 | |
| VINOD | i have not touched anything in nova.conf.sample file and i downloaded the latest from the upstream | 17:03 |
| VINOD | the logs are available | 17:03 |
| VINOD | please go to https://review.openstack.org/#/c/75967/ | 17:03 |
| VINOD | and scroll down to see jenkins tests | 17:03 |
| *** mriedem has quit IRC | 17:04 | |
| *** davidhadas has joined #openstack-infra | 17:04 | |
| *** davidhadas_ has quit IRC | 17:04 | |
| VINOD | 6 tests have failed out of 16 | 17:04 |
| VINOD | also vmware mine sweeper build also failed giving the error | 17:04 |
| VINOD | FAIL: setUpClass (tempest.api.compute.images.test_image_metadata.ImagesMetadataTestJSON) | 17:04 |
| VINOD | Details: {"itemNotFound": {"message": "Image not found.", "code": 404}} | 17:04 |
| VINOD | again, i have done nothing to the images classes | 17:04 |
| fungi | jeblair: do you need anything off two old zuul servers? if not, i was thinking of (carefully) deleting them to regain some of our openstackci quota in dfw | 17:05 |
| *** oubiwann_ has quit IRC | 17:05 | |
| *** oubiwann_ has joined #openstack-infra | 17:06 | |
| anteaya | VINOD: hi as soon as I finish eating I will have a look | 17:06 |
| VINOD | thank you so much | 17:06 |
| jeblair | fungi: go for it. also, while you're there if you want to delete any of the non-production pbx servers | 17:06 |
| VINOD | i will be grateful of you...because i have just submitted one more set of code for novaclient at https://review.openstack.org/#/c/76256/ and i fear that the same problems may happen to it as well | 17:07 |
| VINOD | again thank you so much | 17:07 |
| fungi | jeblair: can do. i assume you still need your jeblair-{centos,precise,raring} vms | 17:07 |
| sdague | jeblair: you want to send this one in - https://review.openstack.org/#/c/75719/ | 17:08 |
| jeblair | fungi: oh, nope those can go too, sorry. | 17:08 |
| VINOD | so i am leaving currently...any comments you can post to me at vinod.kumar.boppanna@cern.ch | 17:08 |
| jeblair | sdague: devstack-gate stack landed? | 17:08 |
| sdague | jeblair: yes | 17:08 |
| sdague | look at the console logs this morning, they are very short | 17:08 |
| fungi | jeblair: oh, cool! i might actually have room to spin up the last elasticsearch replacement in that case without having to do part of the replacement incrementally | 17:09 |
| *** VINOD has quit IRC | 17:09 | |
| jeblair | sdague: done | 17:09 |
| sdague | thanks much | 17:09 |
| *** chandankumar_ has quit IRC | 17:10 | |
| *** pmathews has quit IRC | 17:10 | |
| *** pmathews has joined #openstack-infra | 17:10 | |
| *** hashar has quit IRC | 17:10 | |
| *** chandan_kumar has joined #openstack-infra | 17:10 | |
| *** lcheng has quit IRC | 17:11 | |
| *** relaxdiego has quit IRC | 17:11 | |
| openstackgerrit | A change was merged to openstack-infra/config: index sublog files https://review.openstack.org/75719 | 17:11 |
| *** jlibosva1 has joined #openstack-infra | 17:12 | |
| *** jlibosva1 has quit IRC | 17:12 | |
| *** jlibosva has joined #openstack-infra | 17:13 | |
| tristanC | fungi: the grenade havana check failed on swift there: https://jenkins01.openstack.org/job/check-grenade-dsvm/7960/console | 17:13 |
| fungi | tristanC: okay, sort of expected that, based on discussion with sdague. i can iterate on it a bit more to see what else it needs | 17:14 |
| *** marun has joined #openstack-infra | 17:14 | |
| *** jlibosva has quit IRC | 17:14 | |
| *** jlibosva has joined #openstack-infra | 17:14 | |
| fungi | oh! javelin next | 17:15 |
| fungi | fun | 17:15 |
| *** cadenzajon has joined #openstack-infra | 17:15 | |
| sdague | yeh, this is really a much deeper issue | 17:15 |
| fungi | time to see what i can configure in javelin | 17:15 |
| ekarlso | javeln ? | 17:16 |
| ekarlso | what's that | 17:16 |
| fungi | ekarlso: the "production-like" mock cloud built as part of grenade tests | 17:16 |
| sdague | fungi: so realistically you need to stop running with git swiftclient | 17:16 |
| fungi | grr | 17:16 |
| sdague | or disable swift for havana | 17:16 |
| sdague | there really aren't other options | 17:16 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Remove unused _parse_mailmap() https://review.openstack.org/74279 | 17:17 |
| fungi | sdague: just disable swift for havana grenade jobs though, right? | 17:17 |
| fungi | sdague: seems it runs okay for other havana jobs | 17:17 |
| sdague | well, I guess we don't run the swift cli then? | 17:18 |
| SergeyLukjanov | morning folks | 17:18 |
| SergeyLukjanov | http://logs.openstack.org/17/76217/1/gate/gate-tempest-dsvm-savanna-full/3ba8ab6/console.html#_2014-02-25_16_53_36_727 | 17:18 |
| sdague | I guess that's what's different on this job | 17:18 |
| SergeyLukjanov | is it known issue? | 17:18 |
| *** jlibosva has joined #openstack-infra | 17:18 | |
| *** hashar has joined #openstack-infra | 17:19 | |
| sdague | SergeyLukjanov: yeh it's in the er list somewhere | 17:19 |
| SergeyLukjanov | sdague, ok, thx, will search again | 17:19 |
| ekarlso | ah | 17:19 |
| SergeyLukjanov | sdague, heh, the easy way - reload the page :( https://bugs.launchpad.net/nova/+bug/1255627 | 17:20 |
| fungi | sdague: third (possibly even worse) option, we cease grenade testing of havana changes a week early | 17:21 |
| *** jergerber has quit IRC | 17:21 | |
| sdague | yeh, the unwind seems kind of time consuming for 9 more days | 17:21 |
| fungi | it feels like giving up, but then again the other options sort of do too | 17:22 |
| sdague | so what really happened is a hole in our approach for handling client major version changes | 17:22 |
| fungi | sdague: well, that and we didn't test swiftclient master at all | 17:23 |
| *** e0ne has quit IRC | 17:23 | |
| sdague | yeh, and that | 17:23 |
| fungi | for integration tests i mean | 17:23 |
| *** amcrn has joined #openstack-infra | 17:23 | |
| fungi | otherwise they probably would have seen and worked on solving the backward-compat issues there | 17:23 |
| * fungi wanders off to propose the patch for the distastefully simple solution | 17:24 | |
| *** lcheng has joined #openstack-infra | 17:25 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627 https://review.openstack.org/76265 | 17:25 |
| openstackgerrit | sahid proposed a change to openstack-infra/config: Adds new project Warm https://review.openstack.org/76247 | 17:25 |
| *** moted has joined #openstack-infra | 17:26 | |
| *** kfarr has joined #openstack-infra | 17:27 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627 https://review.openstack.org/76265 | 17:28 |
| *** hogepodge has joined #openstack-infra | 17:28 | |
| *** yassine has quit IRC | 17:28 | |
| *** markwash has quit IRC | 17:29 | |
| *** markmcclain has joined #openstack-infra | 17:29 | |
| *** relaxdiego has joined #openstack-infra | 17:30 | |
| *** mwagner_lap has joined #openstack-infra | 17:30 | |
| *** jcoufal has quit IRC | 17:30 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add queue-name parameter to projects https://review.openstack.org/76268 | 17:30 |
| jeblair | sdague, jog0: ^ that's step 1/2 to get us overall stats for the 'integrated' queue. | 17:31 |
| *** jcoufal has joined #openstack-infra | 17:31 | |
| *** dhellmann_ is now known as dhellmann | 17:31 | |
| openstackgerrit | Logan Paschke proposed a change to openstack-infra/config: Create gerrit-powered-agenda Stackforge Repo https://review.openstack.org/76270 | 17:31 |
| *** valentinbud has joined #openstack-infra | 17:31 | |
| dhellmann | jeblair: thanks, I was pulled away right as you started responding -- is there documentation on any of that? | 17:31 |
| * dhellmann reads comments on review | 17:31 | |
| anteaya | russellb: do you accept 3000+ line patches in nova? https://review.openstack.org/#/c/75967/ | 17:32 |
| russellb | O.O | 17:32 |
| russellb | no | 17:32 |
| fungi | wowie zowie | 17:32 |
| anteaya | russellb: I ask because the contributor asked for some guidance on how to understand the log output | 17:32 |
| fungi | herehavesomecode | 17:32 |
| anteaya | russellb: read scrollback for my interaction with VINOD | 17:32 |
| anteaya | he seemed... in a hurry | 17:33 |
| russellb | -2 though ... not approved for icehouse | 17:33 |
| russellb | will have to wait | 17:33 |
| anteaya | that saves some time | 17:33 |
| *** markmcclain1 has joined #openstack-infra | 17:33 | |
| fungi | perhaps reaching out to tim bell or others at cern to give a heads up on better new contributor integration would help | 17:33 |
| anteaya | russellb: that sounds more like you than me | 17:33 |
| anteaya | wouldn't you say? | 17:34 |
| *** markmcclain1 has quit IRC | 17:34 | |
| *** davidhadas has quit IRC | 17:34 | |
| jeblair | dhellmann: not so much documentation as examples (mentioned in review) plus some moderately documented code in d-g itself. | 17:35 |
| * anteaya assumes russellb will chat with tim bell and moves on | 17:35 | |
| *** oubiwan__ has joined #openstack-infra | 17:35 | |
| *** oubiwann_ has quit IRC | 17:35 | |
| dhellmann | jeblair: ok, I'll go study and come back with questions | 17:35 |
| *** markmcclain1 has joined #openstack-infra | 17:36 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add queue-name parameter to projects https://review.openstack.org/76268 | 17:36 |
| *** markmcclain1 has quit IRC | 17:36 | |
| jeblair | dhellmann: okay, i'm happy to help with whatever you need | 17:36 |
| *** dkliban is now known as dkliban_afk | 17:36 | |
| *** beagles is now known as beagles_brb | 17:36 | |
| *** sarob has joined #openstack-infra | 17:36 | |
| *** markmcclain has quit IRC | 17:37 | |
| *** dkehn_ has quit IRC | 17:37 | |
| *** chuck__ has joined #openstack-infra | 17:37 | |
| *** dkehn has quit IRC | 17:39 | |
| *** dkehn has joined #openstack-infra | 17:40 | |
| *** branen has joined #openstack-infra | 17:41 | |
| *** oubiwan__ has quit IRC | 17:41 | |
| *** dkehn_ has joined #openstack-infra | 17:42 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Stop running Grenade on Havana changes early https://review.openstack.org/76280 | 17:44 |
| *** derekh has quit IRC | 17:44 | |
| fungi | tristanC: sdague: ^ | 17:44 |
| *** markwash has joined #openstack-infra | 17:45 | |
| *** DuncanT- has quit IRC | 17:45 | |
| *** markmcclain has joined #openstack-infra | 17:46 | |
| wenlock | hi jeblair, clarkb: saw there was some activity yesterday on https://review.openstack.org/#/c/51425/ that needed another re-base, i took care of it last night, hopefully it gets a review again. Thanks! | 17:46 |
| anteaya | jaypipes: you around? | 17:47 |
| anteaya | jaypipes: I was hoping to discuss http://lists.openstack.org/pipermail/openstack-dev/2014-February/028110.html | 17:47 |
| anteaya | jaypipes: I have some concerns | 17:47 |
| *** krotscheck has joined #openstack-infra | 17:50 | |
| *** pdmars has quit IRC | 17:50 | |
| kfarr | Hello all! I would appreciate feedback on a patch I'm working on. | 17:50 |
| kfarr | This patch that adds scenario tests in Tempest for cinder volume encryption ( https://review.openstack.org/#/c/58018/ ) depends on having a value for fixed_key for the nova keymgr set. The patch to add a default value for fixed_key in devstack ( https://review.openstack.org/#/c/62481/ ) was merged back in January, so all the Jenkins tests for the Tempest patch *should* pass. | 17:51 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 17:51 | |
| kfarr | However, the check-grenade-dsvm Jenkins test fails because it first checks out and stacks the stable/havana branch of devstack, which was released before the fixed_key patch merged. Even though it upgrades the repos, the settings stay the same because devstack is not restacked nor the settings updated. | 17:51 |
| kfarr | The solution so far is to modify the grenade project and add a value for fixed_key when grenade runs through the upgrades, perhaps in this file: https://github.com/openstack-dev/grenade/blob/master/from-havana/upgrade-nova. This idea doesn't seem ideal, so if anybody had alternative suggestions, I would appreciate hearing them. Waiting for the next stable release of devstack is an option, but I would like to get the patch | 17:51 |
| jeblair | kfarr: #openstack-qa is probably the best place to ask about that | 17:51 |
| jeblair | anteaya, jaypipes: i do too. | 17:53 |
| kfarr | jeblair: ok, thanks! | 17:53 |
| anteaya | thought you might | 17:53 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 17:53 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Stop running Grenade on Havana changes early https://review.openstack.org/76280 | 17:54 |
| fungi | google hangouts? | 17:54 |
| jeblair | yeah | 17:54 |
| fungi | eep | 17:54 |
| *** harlowja_away is now known as harlowja | 17:55 | |
| *** dkehn__ has joined #openstack-infra | 17:55 | |
| *** valentinbud has quit IRC | 17:55 | |
| fungi | i mean, they're free to use whatever they want, but that choice pretty much guarantees i won't be participating, and probably neither will many (any?) other infra team people | 17:56 |
| jeblair | fungi: exactly. i would think that being able to ping infra folks would be very useful for this | 17:57 |
| *** dkehn has quit IRC | 17:57 | |
| anteaya | my concern is that the target audience is already isolated from the communication network | 17:57 |
| *** dkehn_ has quit IRC | 17:57 | |
| anteaya | this re-enforces their isolation | 17:57 |
| fungi | anteaya: that is an extremely valid concern | 17:57 |
| anteaya | thanks | 17:57 |
| jeblair | jaypipes: after all, i believe we provided considerable help to you as you were establishing this. i think we might be able to continue to provide it if you don't exclude us from it. | 17:57 |
| fungi | and we have seen quite a lot of evidence to this effect | 17:57 |
| jeblair | anteaya: agreed | 17:57 |
| *** mriedem has joined #openstack-infra | 17:58 | |
| *** dkehn has joined #openstack-infra | 17:58 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 17:59 |
| *** chuck__ has quit IRC | 17:59 | |
| *** dkehn__ is now known as dkehn_ | 17:59 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:59 | |
| *** yolanda_ has quit IRC | 18:00 | |
| anteaya | blah https://review.openstack.org/#/c/67556/ is failing in the gate again, on a test it passed last testing set | 18:00 |
| anteaya | so far I can't see in the log file where the problem is | 18:01 |
| *** beagles_brb is now known as beagles | 18:01 | |
| clarkb | morning | 18:01 |
| *** mriedem1 has quit IRC | 18:01 | |
| pleia2 | morning clarkb | 18:01 |
| anteaya | this might be it: BadRequest: No nw_info cache associated with instance | 18:01 |
| anteaya | morning clarkb pleia2 | 18:02 |
| *** gokrokve has quit IRC | 18:03 | |
| *** david_lyle_ has joined #openstack-infra | 18:03 | |
| *** david_lyle_ is now known as david_lyle | 18:04 | |
| *** dkehn__ has joined #openstack-infra | 18:04 | |
| *** mwagner_lap has quit IRC | 18:04 | |
| *** CaptTofu_ has joined #openstack-infra | 18:04 | |
| *** lcostantino has quit IRC | 18:04 | |
| *** CaptTofu has quit IRC | 18:04 | |
| anteaya | found it: https://bugs.launchpad.net/nova/+bug/1249065 | 18:04 |
| anteaya | and the gate reset | 18:05 |
| *** thomasem_ has joined #openstack-infra | 18:06 | |
| *** nati_ueno has joined #openstack-infra | 18:06 | |
| *** jgriffit2 has joined #openstack-infra | 18:07 | |
| *** david-lyle has quit IRC | 18:07 | |
| *** jgriffith has quit IRC | 18:07 | |
| *** thomasem has quit IRC | 18:07 | |
| *** dkehn_ has quit IRC | 18:07 | |
| *** luqas has quit IRC | 18:07 | |
| *** oubiwann_ has joined #openstack-infra | 18:08 | |
| mfer | clarkb fungi or anyone else... if you could take a look at https://review.openstack.org/#/c/71956/ | 18:08 |
| mfer | we'd love to be able to start coding in there | 18:09 |
| *** hashar has quit IRC | 18:09 | |
| *** dprince_ has joined #openstack-infra | 18:09 | |
| *** dkehn_ has joined #openstack-infra | 18:10 | |
| anteaya | to save time, currently 71956 will need to be monitored if it merges since it doesn't have a current repo to draw from | 18:11 |
| *** dkliban_afk is now known as dkliban | 18:13 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:14 | |
| *** reed has joined #openstack-infra | 18:16 | |
| *** che-arne has quit IRC | 18:16 | |
| *** rfolco has quit IRC | 18:17 | |
| clarkb | fungi: so ES stuff. It just occurred to me that we have a couple things that don't talk to the whole cluster but directly to elasticsearch.o.o so we will want to do that node after we have at least on of the new nodes up and can point kibana and apache proxy at a new node | 18:17 |
| clarkb | fungi: also I find it super useful to run a local copy of elasticsearch bigdesk and elasticsearch head in my browser pointed at a port forward to the cluster | 18:17 |
| clarkb | fungi: these tools need privs we don't want to expose publicly hence the local copies and port forward, I would recommend setting those up on your end before we start adding and removing nodes | 18:18 |
| *** kmartin has quit IRC | 18:19 | |
| *** pmathews1 has joined #openstack-infra | 18:19 | |
| *** kmartin has joined #openstack-infra | 18:19 | |
| *** rlandy has quit IRC | 18:19 | |
| *** rlandy has joined #openstack-infra | 18:20 | |
| *** changbl has quit IRC | 18:20 | |
| *** relaxdiego has quit IRC | 18:20 | |
| *** jgallard has quit IRC | 18:20 | |
| *** dkehn has quit IRC | 18:20 | |
| *** hogepodge has quit IRC | 18:20 | |
| *** jgriffit2 is now known as jgriffith | 18:21 | |
| *** jcoufal has quit IRC | 18:21 | |
| *** dprince has quit IRC | 18:22 | |
| *** chandan_kumar has quit IRC | 18:22 | |
| fungi | clarkb: awesome. well, i'm setting up the cinder volumes on the last couple now (was able to delete enough cruft to free up sufficient quota for the last one) | 18:22 |
| SergeyLukjanov | folks, could we use templates in templates in zuul? | 18:22 |
| SergeyLukjanov | layout.yaml I mean | 18:22 |
| *** chandankumar_ has joined #openstack-infra | 18:22 | |
| clarkb | fungi: that is great news | 18:22 |
| clarkb | SergeyLukjanov: good question | 18:23 |
| fungi | clarkb: one issue which sprang to mind though... the /var/lib/elasticsearch mount point may not have the right ownership depending on how the package sets that stuff up, so the first one could come up unable to write there... i'll keep an eye out for it when we add the module | 18:23 |
| *** che-arne has joined #openstack-infra | 18:23 | |
| *** mfer has quit IRC | 18:23 | |
| clarkb | fungi: hrm maybe we want to manage the elasticsearch user in puppet? | 18:24 |
| clarkb | probably safe to see what happens in the first go though | 18:24 |
| *** jpich has quit IRC | 18:24 | |
| clarkb | iirc the data actually ends up in /var/lib/elasticsearch/elasticsearch which may be a workaround for issues like that | 18:24 |
| fungi | clarkb: also discovered that the pvhvm images lack the lvm2 package by default (i've proposed a change to add that in template.pp from now on) | 18:24 |
| SergeyLukjanov | clarkb, yup, will take a look on how it works | 18:25 |
| *** pmathews has quit IRC | 18:25 | |
| *** kfarr has quit IRC | 18:25 | |
| SergeyLukjanov | heh, one more crazy question - is it quite crazy if we'd like to publish master / release image built by dib but there are about 12 images for 10-20 GB in total? | 18:26 |
| anteaya | in the coverage logs url: logs.o.o/2 digits from sha/sha/post/<project>-coverage/7 random digits/cover/name of file | 18:27 |
| *** valentinbud has joined #openstack-infra | 18:27 | |
| fungi | SergeyLukjanov: /dev/mapper/main-tarballs 197G 30G 168G 16% /srv/static/tarballs | 18:27 |
| anteaya | where do the 7 random digits come from? | 18:27 |
| fungi | SergeyLukjanov: looks like we probably have room | 18:27 |
| SergeyLukjanov | fungi, yay! | 18:27 |
| clarkb | anteaya: from the python random uuid generator | 18:27 |
| anteaya | clarkb: no wonder they looked random | 18:28 |
| fungi | anteaya: click the link in the sidebar for the job parameters if you're trying to figure out what a running job will use | 18:28 |
| anteaya | so how can one find the coverage logs without having fore knowledge of the 7 random digits? | 18:28 |
| SergeyLukjanov | fungi, my dream is to publish images for all savanna plugins - versions matrix ;) | 18:28 |
| anteaya | fungi: yes, how about a job that has finished running | 18:28 |
| *** lcostantino has joined #openstack-infra | 18:28 | |
| anteaya | I can find the ones for the running ironic job | 18:28 |
| fungi | anteaya: is the coverage job run in post? | 18:28 |
| anteaya | trying to track down logs for neutron patches | 18:29 |
| *** dstufft has quit IRC | 18:29 | |
| anteaya | going by markmcclain's question of me, I would go with yes | 18:29 |
| fungi | anteaya: if it's in the post pipeline, look at the gitsha... give me an example project and commit | 18:29 |
| anteaya | project is neutron | 18:29 |
| anteaya | I'll find you a commit | 18:29 |
| markmcclain | ec3dfc93da605b7c41682951f2fd1e27fb09d53b | 18:29 |
| fungi | keep in mind that it will be what jenkins merged to the project, so it could be a merge commit rather than the parent | 18:30 |
| *** thedodd has joined #openstack-infra | 18:30 | |
| fungi | yeah, that wasn't the actual merge commit, that was just the proposed commit | 18:30 |
| fungi | you'd need the merge commit sha instead | 18:30 |
| *** chandankumar_ has quit IRC | 18:30 | |
| fungi | since that's what actually gets the coverage run | 18:31 |
| *** chandan_kumar has joined #openstack-infra | 18:31 | |
| *** julim has quit IRC | 18:31 | |
| markmcclain | af1aacff158d0004c081b8d8ee77f93baa6244a4 | 18:31 |
| *** julim has joined #openstack-infra | 18:31 | |
| fungi | http://logs.openstack.org/af/af1aacff158d0004c081b8d8ee77f93baa6244a4/ | 18:31 |
| fungi | basically http://logs.openstack.org/<first two chars of sha>/<full sha> | 18:32 |
| fungi | then browse from there | 18:32 |
| *** cody-somerville has quit IRC | 18:32 | |
| *** rohitk has joined #openstack-infra | 18:32 | |
| anteaya | fungi: thank you | 18:32 |
| *** andreaf has quit IRC | 18:32 | |
| fungi | so the coverage report was http://logs.openstack.org/af/af1aacff158d0004c081b8d8ee77f93baa6244a4/post/neutron-coverage/610a9d8/cover/ | 18:32 |
| markmcclain | fungi: thanks! | 18:32 |
| fungi | yw! | 18:32 |
| *** gokrokve has joined #openstack-infra | 18:33 | |
| *** rfolco has joined #openstack-infra | 18:33 | |
| *** vkozhukalov_ has quit IRC | 18:34 | |
| markmcclain | dhellmann just told me he wrote a tool to locate these | 18:34 |
| dhellmann | https://pypi.python.org/pypi/git-os-job | 18:34 |
| fungi | neato! | 18:35 |
| *** changbl has joined #openstack-infra | 18:35 | |
| fungi | i probably already knew that existed, but have a mind like a sieve | 18:35 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract check/gate req jobs to check-requirements https://review.openstack.org/76312 | 18:35 |
| dhellmann | fungi: me, too, that's why I write stuff like that | 18:36 |
| rohitk | Would the Jenkins issue on stable/havana be fixed anytime soon? | 18:36 |
| *** mfer has joined #openstack-infra | 18:36 | |
| *** pmathews has joined #openstack-infra | 18:37 | |
| fungi | rohitk: probably not directly. instead i've proposed a change to just stop running that job since it becomes obsolete in a week when we drop grizzly support | 18:37 |
| jaypipes | jeblair, anteaya: I'm surprised to hear you think I am excluding the infra team from my proposed Q&A/workshop thing :( | 18:37 |
| rohitk | fungi: Ok, thanks | 18:37 |
| anteaya | jaypipes: I don't think that is your motivation | 18:38 |
| anteaya | but I think it might be one of the consequences | 18:38 |
| fungi | jaypipes: only indirectly by your choice of non-free (as in speech) software | 18:38 |
| *** oubiwann_ has quit IRC | 18:38 | |
| *** khyati has joined #openstack-infra | 18:38 | |
| anteaya | due to the fact that the target audience is coming from a place that doesn't understand opensource development | 18:38 |
| anteaya | and how we communicate | 18:38 |
| *** ianw has quit IRC | 18:38 | |
| *** ianw has joined #openstack-infra | 18:39 | |
| *** oubiwann_ has joined #openstack-infra | 18:39 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add support for TestNG publisher. https://review.openstack.org/70157 | 18:39 |
| anteaya | so you may inadvertently be sending a message to them by your choice of medium | 18:39 |
| fungi | yes, so any examples we set by using non-free software to interact with them will potentially send an inaccurate picture of what the project is about | 18:39 |
| *** jamielen- has joined #openstack-infra | 18:39 | |
| *** igor_ has quit IRC | 18:39 | |
| anteaya | and I just wanted to ensure it was a concious choice | 18:39 |
| *** dstufft has joined #openstack-infra | 18:39 | |
| jaypipes | anteaya, fungi, jeblair: i am more than happy to change the choice of medium. like I mentioned, I wanted something that I could share screens on. | 18:39 |
| clarkb | screen! | 18:40 |
| anteaya | I wonder if perhaps a short tutorial on the benefits of using paste might help | 18:40 |
| anteaya | including the trigger command | 18:40 |
| * jaypipes really not sure how he's gotten everyone so upset :( | 18:40 | |
| anteaya | and full prompt | 18:40 |
| fungi | jaypipes: i'm not upset at all | 18:40 |
| fungi | i don't *think* anyone else is either | 18:40 |
| anteaya | not upset | 18:40 |
| clarkb | and don't even know what is going on yet :) catching up on email now | 18:40 |
| *** pmathews1 has quit IRC | 18:41 | |
| anteaya | just aware of the messaging of using non-free software | 18:41 |
| anteaya | with a group struggling to understand open source | 18:41 |
| fungi | anteaya: paste.o.o doesn't cover issues with showing people a demo of configuring the jenkins webui, i think was his point | 18:41 |
| jaypipes | anteaya: what do you suggest as an alternative? | 18:41 |
| jeblair | jaypipes: i know the lack of screenshare is :( but i think it will be offset by being able to interact with us, and as anteaya says, pulling the new folks into the community more fully | 18:41 |
| anteaya | that is a fair point | 18:41 |
| jeblair | jaypipes: i'd like to use irc | 18:42 |
| jaypipes | jeblair: totally. so would you want me to propose changing to an interactive IRC meeting instead? | 18:42 |
| jaypipes | k, will do. | 18:42 |
| jaypipes | not a problem. | 18:42 |
| jeblair | jaypipes: cool, thanks | 18:42 |
| jaypipes | np | 18:42 |
| anteaya | then it will be logged | 18:42 |
| jeblair | jaypipes: we have 3 channels now, so there are some good slots open | 18:42 |
| anteaya | and links to images can be logged to | 18:42 |
| fungi | we even have 3 meeting channels now, and you can make use of the meeting minutes that way for a record to point people back at if they can't attend in their timezone | 18:42 |
| fungi | er, what jeblair said | 18:42 |
| *** nicedice has joined #openstack-infra | 18:42 | |
| *** jamielennox has quit IRC | 18:42 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix unused variable in ci-skip wrapper. https://review.openstack.org/75879 | 18:43 |
| jeblair | jaypipes: the time you proposed works, or if you want to get me after breakfast, 1 or 2 hours later, on either monday or tuesday. :) | 18:43 |
| *** johnthetubaguy has quit IRC | 18:43 | |
| *** kmartin has quit IRC | 18:43 | |
| jaypipes | jeblair: ok, thx. and guys, apologies for causing heartburn. was not intended at all. and I hope you all know how much I appreciate all of the help you have given me on this front. | 18:44 |
| jeblair | jaypipes: we _insist_ that we must continue to give you help! :) | 18:44 |
| jaypipes | hehe | 18:44 |
| fungi | solving problems/confusion for these people early means less work for us later | 18:45 |
| anteaya | thanks for understanding jaypipes | 18:45 |
| fungi | and probably also gets them integrated into the ecosystem more quickly too | 18:45 |
| * mordred hugs jaypipes | 18:45 | |
| jaypipes | ++ | 18:45 |
| anteaya | knew it was just a matter of talking about it | 18:45 |
| clarkb | out of curiousity why so early PST time? strong interest from EU? | 18:46 |
| *** kmartin has joined #openstack-infra | 18:46 | |
| clarkb | maybe its the time I am normally awake creating bias but there seems to be a lot of people west of PST interested in third party testing that may be exlcluded at the proposed time slot | 18:47 |
| jaypipes | clarkb: well, frankly, there's lots of interest from all over the place... | 18:47 |
| jaypipes | clarkb: I can certainly move it back... | 18:47 |
| fungi | we've seen a ton of third-party ci influx from eu and apac | 18:47 |
| jaypipes | never really a good time for everyone. I'll propose a staggered one week this time, next week diff time | 18:47 |
| clarkb | you probably have a better idea of what would work well than I do, hence my question | 18:47 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add satori as a new project https://review.openstack.org/73667 | 18:48 |
| * anteaya thinks of poor mayu in china | 18:48 | |
| * jaypipes scratches head trying to figure out how to delete a Google event.... | 18:48 | |
| anteaya | he got his system up btw | 18:48 |
| anteaya | he was very grateful | 18:48 |
| jeblair | jaypipes: i'm not sure you can delete anything from google :) | 18:48 |
| jaypipes | yes, anteaya, I was working with mayu on #openstack-neutron yesterday. | 18:49 |
| jaypipes | jeblair: heh. or the NSA. | 18:49 |
| jeblair | fungi, clarkb, mordred, SergeyLukjanov: i'm going to be afk thursday and friday this week | 18:49 |
| *** oubiwann_ has quit IRC | 18:49 | |
| fungi | jeblair: thanks for the heads up! | 18:49 |
| jaypipes | perhaps the NSA has advice on setting up third party CI? ;) | 18:49 |
| clarkb | jeblair: noted | 18:50 |
| SergeyLukjanov | jeblair, noted | 18:50 |
| *** markmcclain has quit IRC | 18:50 | |
| *** mkoderer has quit IRC | 18:51 | |
| *** thomasem_ has quit IRC | 18:52 | |
| sdague | so I'm not sure why a hangouts based bootstrapping approach isn't seen as complementary to irc based ones. Honestly, I was just excited to see someone else taking effort off the plate for the infra team :) | 18:52 |
| anteaya | nova test failing in the gate looks like a tempest test failure | 18:52 |
| jaypipes | aha! figured out how to delet the event. :) | 18:52 |
| anteaya | but the fails on the requirements patch might be infra related | 18:52 |
| anteaya | https://jenkins06.openstack.org/job/gate-devstack-dsvm-cells/867/console | 18:53 |
| clarkb | sdague: in particular it creates a divide in participation. Third party testers should be here but the majority are not | 18:53 |
| *** valentinbud has quit IRC | 18:53 | |
| clarkb | but it also doesn't come with recorded text that is searchable | 18:53 |
| sdague | clarkb: well, I think there are stages of getting them here | 18:53 |
| clarkb | for the folks on the other side of the planet that may not be able to participate due to scheduling | 18:54 |
| sdague | sure, but I think there already exists a ton of useful text on the subject, so more archived text is not something I expect will change things | 18:54 |
| sdague | so if training wheels included another medium that they felt more comfortable with to then start engaging more regularly in infra, I see that as a good thing | 18:55 |
| clarkb | fungi: I am reviewing stuff now, please do ping when you are ready to do more ES stuff | 18:56 |
| fungi | clarkb: okay, i've got all 6 of the new 60g performance ubuntu 12.04 pvhvm elasticsearch servers in dns with 1tb cinder mounted on each. if you want to look one over (they should all be identical) go for it. puppet's not set to run on any of them, so next it should be safe to merge https://review.openstack.org/76048 right? | 18:56 |
| clarkb | lol timing | 18:56 |
| *** pdmars has joined #openstack-infra | 18:56 | |
| *** jcoufal has joined #openstack-infra | 18:56 | |
| *** mwagner_lap has joined #openstack-infra | 18:56 | |
| fungi | also, almost meeting times | 18:57 |
| *** pdmars has quit IRC | 18:57 | |
| clarkb | oh right | 18:57 |
| *** markmcclain has joined #openstack-infra | 18:57 | |
| *** pdmars has joined #openstack-infra | 18:58 | |
| clarkb | fungi: yes, I think it should be safe to merge that change then we can toggle puppet on nodes one at a time as we iterate through adding node then deleting node | 18:58 |
| clarkb | sdague: I might just be stubborn, but I think getting folks comfortable in the existing mediums is the greatest set of training wheels | 18:59 |
| *** SumitNaiksatam has quit IRC | 18:59 | |
| fungi | okay, let's allow it to merge and update while the meeting is underway and then i can get started with migration once the emeting ends | 18:59 |
| clarkb | teach a person to fish and all that | 19:00 |
| clarkb | fungi: sounds good | 19:00 |
| anteaya | so in http://logs.openstack.org/39/75539/9/gate/gate-devstack-dsvm-cells/ae77fc5/logs/devstacklog.txt.gz | 19:00 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract tarballs and pypi upload to pypi-release https://review.openstack.org/76322 | 19:00 |
| anteaya | I see: There are problems and -y was used without --force-yes | 19:00 |
| jeblair | it's meeting time over in #openstack-meeting | 19:00 |
| *** thomasem has joined #openstack-infra | 19:00 | |
| anteaya | looks like during the apt-get stage | 19:01 |
| *** jp_at_hp has quit IRC | 19:01 | |
| sdague | clarkb: sure, it's all about activation energy. Much like summits and mid cycle meetups, google hangouts provides a way to bring more folks into the fold. | 19:01 |
| anteaya | two more failing tests on a neutron patch, looking the same: https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-pg/664/console | 19:02 |
| *** jamielen- is now known as jamielennox | 19:02 | |
| *** dkehn_ is now known as dkehn | 19:02 | |
| *** dkehn__ is now known as dkehn_ | 19:03 | |
| anteaya | Failed to fetch http://mirror.rackspace.com/ubuntu/dists/precise-security/multiverse/binary-amd64/Packages | 19:03 |
| anteaya | is mirror.rackspace.com/ubuntu ours? | 19:03 |
| anteaya | from what I am seeing in the logs, devstack is unable to fetch ubuntu dependencies from it | 19:04 |
| *** markmcclain has quit IRC | 19:04 | |
| *** rohitk has quit IRC | 19:04 | |
| fungi | anteaya: rackspace manages that (or canonical may on behalf of rackspace--i'm somewhat unclear on the relationship there) | 19:04 |
| *** cadenzajon_ has joined #openstack-infra | 19:04 | |
| anteaya | okay, well from what I think I am seeing there is a problem with that mirror | 19:05 |
| anteaya | and patches are starting to fail as a result | 19:05 |
| *** hogepodge has joined #openstack-infra | 19:05 | |
| anteaya | anyone in channel whom we can ping? | 19:05 |
| *** markmcclain has joined #openstack-infra | 19:05 | |
| fungi | pvo: do you happen to know anything about rackspace ubuntu apt mirrors having problems? | 19:05 |
| sdague | anteaya: is this persistent? or just a one off? | 19:06 |
| *** pblaho has joined #openstack-infra | 19:06 | |
| anteaya | it started with the requirements patch | 19:06 |
| anteaya | and has crept into the neutron patch | 19:06 |
| anteaya | I haven't looked at teh failing nova patch yet | 19:06 |
| anteaya | it looks like it is expanding to me | 19:07 |
| sdague | anteaya: right, I was more curious if yuo have a logstash query for it | 19:07 |
| anteaya | I haven't taken a full look | 19:07 |
| *** cadenzajon has quit IRC | 19:07 | |
| anteaya | no no logstash yet | 19:07 |
| sdague | that's typically a good starting point, to figure out if it's localized, or systematic | 19:07 |
| *** esker has joined #openstack-infra | 19:07 | |
| anteaya | k, trying to do meeting to | 19:08 |
| anteaya | There are problems and -y was used without --force-yes | 19:08 |
| anteaya | looks like a key line | 19:08 |
| *** DinaBelova has quit IRC | 19:08 | |
| *** NikitaKonovalov_ has quit IRC | 19:08 | |
| *** ruhe has quit IRC | 19:08 | |
| *** SergeyLukjanov has quit IRC | 19:08 | |
| anteaya | in devstacklog.txt.gz | 19:08 |
| *** DinaBelova_ has joined #openstack-infra | 19:08 | |
| *** DinaBelova_ is now known as DinaBelova | 19:08 | |
| *** tteggel has quit IRC | 19:09 | |
| *** NikitaKonovalov_ has joined #openstack-infra | 19:09 | |
| *** ruhe has joined #openstack-infra | 19:09 | |
| anteaya | sdague: no events matched | 19:09 |
| *** SumitNaiksatam has joined #openstack-infra | 19:09 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:09 | |
| anteaya | logstash hasn't got the logs yet, since it is just in running tests | 19:09 |
| sdague | ok | 19:09 |
| sdague | so then it's new | 19:09 |
| *** SergeyLukjanov has joined #openstack-infra | 19:10 | |
| *** oubiwann_ has joined #openstack-infra | 19:10 | |
| *** valentinbud has joined #openstack-infra | 19:10 | |
| *** SergeyLukjanov2 has joined #openstack-infra | 19:10 | |
| anteaya | seems to be, yes | 19:10 |
| SpamapS | soren: I wonder.. could we maybe move uvirtbot under openstack-infra's control? | 19:11 |
| SpamapS | soren: it seems to go missing quite often... | 19:11 |
| SpamapS | and it is a really invaluable tool for easing conversations | 19:11 |
| *** oubiwann_ has quit IRC | 19:11 | |
| *** SergeyLukjanov has quit IRC | 19:11 | |
| *** SergeyLukjanov2 is now known as SergeyLukjanov | 19:11 | |
| anteaya | it is affecting the nova patch: http://logs.openstack.org/18/75918/1/gate/gate-tempest-dsvm-postgres-full/42a02b8/logs/devstacklog.txt.gz | 19:12 |
| anteaya | and ceilometer: http://logs.openstack.org/69/76069/1/gate/gate-tempest-dsvm-neutron-large-ops/b9a7a7a/logs/devstacklog.txt.gz | 19:13 |
| *** igor_ has joined #openstack-infra | 19:13 | |
| *** ryanpetrello has quit IRC | 19:13 | |
| anteaya | and this ceilometer: http://logs.openstack.org/87/66687/13/gate/gate-tempest-dsvm-full/495c46f/logs/devstacklog.txt.gz | 19:13 |
| *** tteggel has joined #openstack-infra | 19:14 | |
| anteaya | so yes this mirror.rackspace.com ubuntu issue is affecting us | 19:14 |
| anteaya | fungi: whom can I ping about this? | 19:14 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Merger: log non Git exceptions on merge https://review.openstack.org/70977 | 19:16 |
| *** rossella has quit IRC | 19:17 | |
| *** igor_ has quit IRC | 19:18 | |
| fungi | anteaya: it looks like they may have changed how they're mirroring that, or have updated from an upstream mirror in a way such that only the compressed sources and packages files are present for a couple of suites. i can open a trouble ticket with rackspace to give them a heads up about that | 19:19 |
| anteaya | okay | 19:19 |
| *** ryanpetrello has joined #openstack-infra | 19:19 | |
| anteaya | shall I open a bug on our end for those who will need to recheck/reverify? | 19:19 |
| jaypipes | anteaya, jeblair: lemme know if OK: http://lists.openstack.org/pipermail/openstack-dev/2014-February/028124.html | 19:22 |
| anteaya | jaypipes: looks good to me, thank you | 19:24 |
| *** mwagner_lap has quit IRC | 19:24 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract common dsvm to integrated-gate https://review.openstack.org/76333 | 19:25 |
| kashyap | clarkb, pleia2 Heya, just a note -- "Defaults requiretty" from sudo will be removed from Fedora's sudo. The maintainer just confirmed it on the bug | 19:27 |
| kashyap | I updated the etherpad here -- https://etherpad.openstack.org/p/fedora-on-gate | 19:27 |
| clarkb | kashyap: great news | 19:27 |
| * kashyap hits the hay, too late here. Just wanted to note the above. | 19:27 | |
| pleia2 | kashyap: great :) | 19:28 |
| *** sdake_ has joined #openstack-infra | 19:28 | |
| *** cody-somerville has joined #openstack-infra | 19:29 | |
| anteaya | https://bugs.launchpad.net/openstack-ci/+bug/1284786 | 19:30 |
| anteaya | that is the mirror.rackspace.com bug report | 19:30 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273908 https://review.openstack.org/76335 | 19:30 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract tarballs and pypi upload to pypi-release https://review.openstack.org/76322 | 19:32 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract common dsvm to integrated-gate https://review.openstack.org/76333 | 19:32 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract check/gate req jobs to check-requirements https://review.openstack.org/76312 | 19:32 |
| anteaya | so far just 5 patches that I see taken out by the mirror issue | 19:32 |
| *** valentinbud has quit IRC | 19:33 | |
| anteaya | 75539,9 73086,5 75918,1 76969,1 66687,13 | 19:34 |
| openstackgerrit | A change was merged to openstack-infra/config: Install and run elasticsearch on new workers https://review.openstack.org/76048 | 19:34 |
| *** che-arne has quit IRC | 19:35 | |
| reed | moto-g or nexus 5? | 19:36 |
| anteaya | reed hi what are you asking about? | 19:36 |
| reed | anteaya, random stuff :) I lost ... my phone was stolen in a cab | 19:37 |
| clarkb | reed: nexus 5 | 19:37 |
| *** fbo is now known as fbo_away | 19:37 | |
| clarkb | moto g is decent if you want cheap but nexus 5 is much better | 19:37 |
| anteaya | reed: :( | 19:37 |
| reed | clarkb, I figured... is that worth +$200 more and +7days wait? | 19:38 |
| fungi | anteaya: i think that apt mirror issue must have been short-lived. by the time i got enough of a break to log into a system and confirm it by hand, it seems to not be happening | 19:38 |
| reed | ah, the 1st world problems | 19:38 |
| reed | thanks anteaya | 19:38 |
| anteaya | fungi: I am thinking the same | 19:38 |
| anteaya | some sort of transference hiccup maybe | 19:39 |
| clarkb | fungi: I see the logs in the bug, were those the only logs available? gerrit lib logs at debug level a lot of additional info | 19:39 |
| clarkb | fungi: including return code and stderr | 19:39 |
| clarkb | and stdout | 19:39 |
| clarkb | which would probably help us debug | 19:39 |
| anteaya | fungi: but at least those 5 patches have a bug number | 19:39 |
| *** esker has quit IRC | 19:40 | |
| fungi | clarkb: yes, that's all puppet dropped into the syslog for us | 19:40 |
| pleia2 | jaypipes: not sure if anyone else has asked, but would it be possible for you to convert your blog posts into creating a better http://ci.openstack.org/third_party.html ? | 19:40 |
| soren | SpamapS: Yeah, it's a bit of a POS. | 19:41 |
| soren | SpamapS: There's not much to it, really. It's a standard Supybot with a plugin I stole from ubottu half a decade ago. | 19:41 |
| jaypipes | pleia2: most certainly -- all content on my blog is creative commons share-alike licensed. I was thinking of letting the blog posts bake on joinfu.com for at least another week or two (as I'm still updating the posts as folks send comments and corrections). But after that, I'd be happy to push up content patches to ci.openstack.org with some or all of that content. | 19:43 |
| pleia2 | jaypipes: awesome, thanks :) I was reading through the posts last night and they're really great | 19:44 |
| jaypipes | cheers | 19:44 |
| *** igor_ has joined #openstack-infra | 19:44 | |
| fungi | soren: SpamapS: if someone wanted to whip up a similar bot which did link mapping for the services we use, shortening, all those sorts of things, i expect infra would be happy to run it | 19:44 |
| jaypipes | pleia2: very much eager for criticism, comments, corrections, bugs,e tc :) | 19:44 |
| *** atiwari has quit IRC | 19:45 | |
| pleia2 | jaypipes: I don't like maintaining the os-ext-testing separate from our infrastructure, but I don't have a better suggestion at the moment | 19:46 |
| *** atiwari has joined #openstack-infra | 19:46 | |
| *** esker has joined #openstack-infra | 19:46 | |
| jaypipes | pleia2: me neither :) happy to push it to stackforge or somewhere in openstack-infra/ eventually... | 19:46 |
| * pleia2 nods | 19:47 | |
| mestery | Hi folks: I'm seeing some bizzare failures for some patches similar to the one here: https://review.openstack.org/#/c/73372/4 | 19:47 |
| pleia2 | probably openstack-infra at some point, or maybe include it as another part of config (since there is overlap) | 19:47 |
| mestery | Looks like the devstack node never even made it up. Have others seen this? I don't see a recheck bug for this, thus asking here. | 19:47 |
| jaypipes | pleia2: of course, the idea behind os-ext-testing was to reuse absolutely as much as possible the stock puppet manifests from openstack-infra/config and only create new manifests where needed | 19:47 |
| pleia2 | yeah | 19:47 |
| jaypipes | pleia2: the os-ext-testing-data repository was an attempt to introduce folks setting up these CI systems to the notion of keeping all config data in source control. I know that I could have used Hiera for the data store, too. | 19:49 |
| *** mwagner_lap has joined #openstack-infra | 19:49 | |
| anteaya | mestery: we just got hit with this: https://bugs.launchpad.net/openstack-ci/+bug/1284786 | 19:49 |
| mestery | anteaya: Thank you, that's what I was looking for! | 19:50 |
| anteaya | affected 5 gate patches before it went away and I don't know how many in check | 19:50 |
| anteaya | I checked your devstacklog file I didn't see this error | 19:50 |
| anteaya | mestery: is this the correct bug? | 19:50 |
| mestery | anteaya: I saw it yesterday. | 19:50 |
| *** shashank_ has joined #openstack-infra | 19:51 | |
| anteaya | mestery: did you | 19:51 |
| *** bhuvan has joined #openstack-infra | 19:51 | |
| anteaya | do you have the logs? | 19:51 |
| anteaya | we just spotted it within the last hour | 19:51 |
| sdague | question, a while ago someone had this need javascript library that let you do markup language for things like sequence diagrams | 19:52 |
| sdague | and I've since lost it | 19:52 |
| mestery | anteaya: http://logs.openstack.org/72/73372/15/check/check-tempest-dsvm-neutron/10663b0/ | 19:52 |
| mestery | anteaya: Not sure it's the same as the one you pointed me at. | 19:52 |
| fungi | mestery: looks like it died while or immediately after setting up the metadata agent? http://logs.openstack.org/72/73372/15/check/check-tempest-dsvm-neutron/10663b0/logs/devstacklog.txt.gz | 19:52 |
| mestery | anteaya: But I saw this same problem as the logs I just gave you yesterday with another Neutron patch. | 19:52 |
| *** hogepodge has quit IRC | 19:52 | |
| sdague | mestery: yeh, we split out logs | 19:53 |
| mestery | fungi: I think you're right, and then it hit a timeout and was reaped, is that it? | 19:53 |
| fungi | mestery: yeah, i don't see any screen-n logs for it... | 19:54 |
| anteaya | mestery: it is not looking like the same one to me, in 1284786 devstack dies because it can't apt-get from mirror.rackspace.com | 19:56 |
| anteaya | it appears your logs get past that step | 19:56 |
| *** pblaho has quit IRC | 19:57 | |
| mordred | jaypipes: AaronGr is working on some refactor ideas that should help with the amount of needed divergence | 19:57 |
| jaypipes | mordred: yup, I saw that email. ++! | 19:57 |
| mordred | jaypipes: cause we're tracking an exact copy right now and there are ... more than one thing that _must_ be forked :) | 19:57 |
| morganfainberg | jeblair, oooooo | 19:57 |
| morganfainberg | i mean | 19:57 |
| bhuvan | sdague: are you looking for something like http://plantuml.sourceforge.net/index.html? | 19:57 |
| morganfainberg | jaypipes, mordred, really? | 19:57 |
| mordred | morganfainberg: yah | 19:57 |
| morganfainberg | jaypipes, mordred, because i was about to toss up my hands and use chef (or the like) to setup something very similar | 19:58 |
| clarkb | sdague: I can dig a couple up shortly, I looked at them for my LCA talk before giving up on js | 19:58 |
| morganfainberg | mordred, the current puppet manifests were/are very hard-coded o.o infra specific | 19:58 |
| mestery | thanks anteaya, I agree with that assessment | 19:58 |
| shashank_ | Hi All... I see that my devstack is failing because it is unable to clone nova repository | 19:58 |
| clarkb | sdague: http://bramp.github.io/js-sequence-diagrams/ | 19:58 |
| *** boris-42_ is now known as boris-42 | 19:58 | |
| sdague | clarkb: thanks | 19:58 |
| shashank_ | 2014-02-25 11:25:30.076 | + git clone git://git.openstack.org/openstack/nova.git /opt/stack/nova | 19:59 |
| shashank_ | 2014-02-25 11:25:36.643 | Cloning into '/opt/stack/nova'... | 19:59 |
| shashank_ | 2014-02-25 11:26:36.645 | fatal: The remote end hung up unexpectedly | 19:59 |
| shashank_ | 2014-02-25 11:26:36.648 | fatal: early EOF | 19:59 |
| mordred | morganfainberg: they still are - but it's totally possible to do - and we're actively working on making it way less/not hardcoded infra specific | 19:59 |
| shashank_ | 2014-02-25 11:26:36.651 | fatal: recursion detected in die handler | 19:59 |
| shashank_ | Any idea what could be wrong? | 19:59 |
| morganfainberg | mordred, seriously awesome to hear | 19:59 |
| jeblair | morganfainberg: there's a lot of work being staged to fix that. most of it's waiting on puppetboard to be up and running, but we're getting pretty close to that | 19:59 |
| anteaya | shashank_: instead of pasting error codes into the channel can you instead post the url to the patch in question? | 19:59 |
| jeblair | morganfainberg: then i expect we'll start merging puppet refactoring changes quickly | 19:59 |
| morganfainberg | jeblair, ++ nice. i might need to stand stuff up on my own then still (i need to have an infra-like env up by next weeks end) | 20:00 |
| * fungi is reminded to see if the tweak to the jetty config is up and can be merged | 20:00 | |
| morganfainberg | which limits what i can do, but very very happy to hear this | 20:00 |
| shashank_ | anteaya: sorry, which patch? | 20:00 |
| morganfainberg | i'll def. contribute where i can once things are moving. | 20:00 |
| sdague | mestery: actually something failed in devstack setup, but something is weird about our cleanup, so it doesn't seem to have gotten logged | 20:00 |
| shashank_ | I am using devstack and haven't specified any nova branch or patch in the localrc | 20:00 |
| anteaya | shashank_: yes, which patch has the error you are posting about? | 20:01 |
| jeblair | morganfainberg: yeah, if you can stand it, maybe muddle through as best you can for now, but don't invest heavily in alternate infrastructure and instead, come back and contribute when you aren't rushed and we're hading in the right direction | 20:01 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 20:01 | |
| *** rlandy has quit IRC | 20:01 | |
| morganfainberg | jeblair, yep | 20:02 |
| clarkb | shashank_: are you just running localhost locally? | 20:02 |
| clarkb | s/localhost/devstack/ | 20:02 |
| shashank_ | clarkb: yes | 20:02 |
| clarkb | if so you might be having trouble with git protocol through a proxy/firewall | 20:02 |
| morganfainberg | jeblair, i already have the "just do it outside of CMS due to timing" so i'll use the puppet stuff you guys are doing as the template in either case so i'm not massively diverged. | 20:02 |
| morganfainberg | jeblair, and i'll look at making things much more in line/leverage the o.o stuff as it becomes viable to do so / i have time to contribute to that | 20:03 |
| mestery | sdague: So it will be impossible to track down the exact reason for the failure here? | 20:03 |
| sdague | unfortunately, yes, we've been trying to sort out these flush issues which are creeping up from time to time | 20:03 |
| shashank_ | clarkb: My localrc is as follows: http://paste.openstack.org/show/69528/ | 20:04 |
| jeblair | morganfainberg: sounds like a plan! | 20:04 |
| clarkb | shashank_: are you behind a firewall or proxy? | 20:04 |
| shashank_ | yes, I am | 20:05 |
| shashank_ | but other repositories are cloned successfully | 20:05 |
| *** davidhadas has joined #openstack-infra | 20:05 | |
| clarkb | shashank_: try changing the repo root to use https | 20:05 |
| anteaya | sdague: do we have a bug number for mestery? | 20:06 |
| shashank_ | 2014-02-25 11:24:53.331 | + git clone git://git.openstack.org/openstack/neutron.git /opt/stack/neutron | 20:06 |
| shashank_ | 2014-02-25 11:24:54.695 | Cloning into '/opt/stack/neutron'... | 20:06 |
| shashank_ | 2014-02-25 11:25:17.522 | + cd /opt/stack/neutron | 20:06 |
| shashank_ | 2014-02-25 11:25:17.525 | + git fetch git://git.openstack.org/openstack/neutron.git refs/changes/81/67281/10 | 20:06 |
| shashank_ | 2014-02-25 11:25:19.645 | From git://git.openstack.org/openstack/neutron | 20:06 |
| shashank_ | 2014-02-25 11:25:19.649 | * branch refs/changes/81/67281/10 -> FETCH_HEAD | 20:06 |
| shashank_ | 2014-02-25 11:25:19.652 | + git checkout FETCH_HEAD | 20:06 |
| anteaya | shashank_: please stop doing that | 20:06 |
| shashank_ | That works! | 20:06 |
| anteaya | please use paste | 20:06 |
| *** denis_makogon_ has joined #openstack-infra | 20:06 | |
| *** mrmartin has joined #openstack-infra | 20:06 | |
| clarkb | I forget the localrc setting for that but there is one you should see in stackrc that can be overridden in localrc | 20:06 |
| shashank_ | sure... sorry | 20:06 |
| anteaya | thank you | 20:06 |
| *** vishy has joined #openstack-infra | 20:07 | |
| *** igor_ has quit IRC | 20:07 | |
| shashank_ | hmmm... but I am curious why cloning nova repository fails consistently | 20:07 |
| shashank_ | while others succeed | 20:07 |
| fungi | shashank_: nova is a lot larger than some other repositories. you may be hitting proxy limits on your originating network | 20:08 |
| *** denis_makogon has quit IRC | 20:08 | |
| *** denis_makogon_ is now known as denis_makogon | 20:08 | |
| shashank_ | oh... | 20:08 |
| *** dmakogon_ has joined #openstack-infra | 20:09 | |
| mrmartin | fungi: hi, can I see somehow a server's puppet log without logging in through ssh? | 20:09 |
| fungi | mrmartin: i can get it for you... just a few minutes | 20:09 |
| *** dcramer_ has joined #openstack-infra | 20:09 | |
| mrmartin | fungi: allright, thanks, I need the puppet logs of openstackid-dev.openstack.org | 20:09 |
| *** boris-42 has quit IRC | 20:10 | |
| *** boris-42 has joined #openstack-infra | 20:10 | |
| *** dcramer_ has quit IRC | 20:10 | |
| fungi | mrmartin: i assumed that was what you wanted. i tested the change with --noop before i approved it, but that's not entirely comprehensive of course and i hadn't gone back to see how it ended up | 20:10 |
| *** dcramer_ has joined #openstack-infra | 20:11 | |
| * anteaya goes for a walk | 20:11 | |
| openstackgerrit | A change was merged to openstack-infra/config: Set puppetdb server to listen on all interfaces https://review.openstack.org/75950 | 20:11 |
| *** vkozhukalov_ has joined #openstack-infra | 20:12 | |
| mrmartin | fungi: I guess we have a failed deployment due the missing distro tarball, at tarballs.openstack.org/openstackid, but need to check the exact reason. | 20:12 |
| *** oubiwann_ has joined #openstack-infra | 20:13 | |
| *** mbacchi has quit IRC | 20:13 | |
| *** fbo_away is now known as fbo | 20:15 | |
| *** ianw has quit IRC | 20:15 | |
| *** ianw has joined #openstack-infra | 20:16 | |
| *** coolsvap has quit IRC | 20:16 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: add some helpfulness to our log output https://review.openstack.org/76349 | 20:17 |
| *** dcramer_ has quit IRC | 20:17 | |
| *** cadenzajon has joined #openstack-infra | 20:17 | |
| *** valentinbud has joined #openstack-infra | 20:17 | |
| *** boris-42 has quit IRC | 20:19 | |
| fungi | mrmartin: i think you're right... http://paste.openstack.org/show/69537/ | 20:19 |
| valentinbud | Hello clarkb, I have a problem with a logging infrastructure built from the openstack-infra/config bits. Multiline filters don't work as expected. | 20:19 |
| *** boris-42 has joined #openstack-infra | 20:19 | |
| valentinbud | I have been pointed in your direction. | 20:19 |
| *** bhuvan has quit IRC | 20:19 | |
| clarkb | valentinbud: hey | 20:19 |
| *** cadenzajon_ has quit IRC | 20:20 | |
| mrmartin | fungi: may I ask you to do an rm -rf /srv/openstackid ? | 20:20 |
| clarkb | valentinbud: what version of logstash are you suing with the ruleset? | 20:20 |
| valentinbud | Hey clarkb. I am working on the HyperV CI and using the same log processing infra. | 20:20 |
| fungi | mrmartin: sure, doing that now | 20:20 |
| valentinbud | 1.3.3 | 20:20 |
| mrmartin | fungi: how much time do we need to wait for another puppet run? | 20:20 |
| clarkb | valentinbud: ok so logtash version matches | 20:21 |
| *** ociuhandu has quit IRC | 20:21 | |
| clarkb | valentinbud: how are you testing/noticing the problem? and does it manifest itself as every line in the mulitline event being a distinct event? | 20:21 |
| *** rcarrillocruz1 has joined #openstack-infra | 20:21 | |
| fungi | mrmartin: it refreshes every 10 minutes | 20:21 |
| mrmartin | ok | 20:21 |
| *** oubiwann_ has quit IRC | 20:22 | |
| valentinbud | clarkb: I am feeding logstash via stdin and/or file inputs and tag the events with oslofmt. | 20:22 |
| *** rcarrillocruz has quit IRC | 20:22 | |
| valentinbud | The logs come from nova-compute. | 20:22 |
| fungi | mrmartin: the last update completed just before i removed that directory, at 20:21:11, so we've got about 9 minutes before it fires again | 20:22 |
| *** tjones has quit IRC | 20:23 | |
| *** bhuvan has joined #openstack-infra | 20:23 | |
| clarkb | valentinbud: and you are using the json_lines codec on stdin? | 20:23 |
| *** boris-42 has quit IRC | 20:23 | |
| *** boris-42_ has joined #openstack-infra | 20:23 | |
| valentinbud | clarkb: no, I am not, that wouldn't work because the lines are raw text, or is my assumption wrong? | 20:24 |
| clarkb | valentinbud: give me a second to reread what goes on there | 20:24 |
| *** oubiwann_ has joined #openstack-infra | 20:25 | |
| *** shashank_ has quit IRC | 20:25 | |
| valentinbud | Thank you clarkb. | 20:25 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Require openstack-doc-tools 0.8.2 https://review.openstack.org/76351 | 20:25 |
| clarkb | valentinbud: yeah json lines is what we use because the daemon we have talking to the logstash input spits out json | 20:25 |
| *** tomhe has joined #openstack-infra | 20:25 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627 https://review.openstack.org/76265 | 20:26 |
| *** mrmartin has quit IRC | 20:26 | |
| *** bhuvan_ has joined #openstack-infra | 20:27 | |
| jog0 | jeblair: nice | 20:28 |
| *** boris-42_ is now known as boris-42 | 20:29 | |
| valentinbud | The log-gearman-worker which also outputs to CRM114. | 20:29 |
| clarkb | valentinbud: correct | 20:29 |
| *** tjones has joined #openstack-infra | 20:29 | |
| *** bhuvan has quit IRC | 20:30 | |
| valentinbud | I like how you've done the logging infrastructure, robust and easily scalable. | 20:30 |
| *** dcramer_ has joined #openstack-infra | 20:30 | |
| valentinbud | I have noticed something strange with logstash. | 20:30 |
| *** gyee has quit IRC | 20:30 | |
| tomhe | Our Zuul setup has recently had a problem with the embedded Gearman server not registering a Jenkins job. I had to delete and recreate the job a couple of times until it listed as registered. Any ideas to what could be causing this? (On Zuul from master as of Fri Feb 21 08:28:58) | 20:30 |
| clarkb | valentinbud: thanks, making it scalable was definitely one of the goals | 20:30 |
| valentinbud | If I feed him only TRACE lines it filters the message as multiline. | 20:31 |
| valentinbud | But if I feed him one INFO and one DEBUG line, before or after the TRACE, it doesn't do it. | 20:31 |
| tomhe | On a side note: The new Zuul merger architecture really improved our performance. Great work you guys! | 20:31 |
| clarkb | valentinbud: interesting, so the way multiline works in logstash is a bit of a hack | 20:31 |
| *** dmsimard has joined #openstack-infra | 20:32 | |
| clarkb | valentinbud: logtash keeps a buffer with the previous event in it, if multiline matches it updates that buffer by appending the new data until something doesn't match the multiline condition | 20:32 |
| clarkb | valentinbud: when that happens the item in the buffer is flushed and the item that didn't match the condition goes into the buffer | 20:32 |
| dmsimard | Any issues with paste.openstack.org ? Unresponsive here. | 20:32 |
| clarkb | its possible that something around matching conditions is causing that buffer to flush when we don't expect it to | 20:32 |
| *** bhuvan has joined #openstack-infra | 20:33 | |
| *** tjones has quit IRC | 20:33 | |
| dmsimard | Looks like paste is back, nevermind about that. | 20:33 |
| valentinbud | clarkb: it might be that. How can I debug further. I have downloaded a nova-compute log from a Jenkins build failure and feed it via input file with start beginning. | 20:34 |
| valentinbud | clarkb: I output everything on stdout with codec rubydebug. | 20:34 |
| valentinbud | clarkb: For what you say I must go deeper. How can I do that? | 20:34 |
| *** bhuvan_ has quit IRC | 20:34 | |
| clarkb | valentinbud: I do testing locally but spinning up a geard server and a logstash-gearman-worker pointed at localhost 9999 and have a logstash instance with tcp input listening there | 20:35 |
| *** hogepodge has joined #openstack-infra | 20:35 | |
| clarkb | then I have a simple script that adds a job to the geard server causing logs to be fetched and go through the whole process | 20:35 |
| clarkb | valentinbud: give me a few minutes and I can share that small script with you | 20:35 |
| clarkb | but mocking up the whole pipeline like that would probably help | 20:36 |
| *** mrmartin has joined #openstack-infra | 20:36 | |
| *** bhuvan_ has joined #openstack-infra | 20:38 | |
| *** bhuvan_ has joined #openstack-infra | 20:38 | |
| *** dhellman_ has joined #openstack-infra | 20:40 | |
| *** dhellman_ has quit IRC | 20:40 | |
| clarkb | fungi: have you got elasticsearch head and bigdesk running locally? | 20:40 |
| clarkb | fungi: they run better in chromium fwiw | 20:40 |
| fungi | clarkb: took a detour to catch openstackid and puppetdb work up so i'm not blocking people. getting started on bigdesk installation now | 20:40 |
| clarkb | great | 20:40 |
| clarkb | no rush, I think once we start iterating on nodes will have lots of time in between steps | 20:40 |
| clarkb | because data needs to be copied around in order to not lose it | 20:40 |
| fungi | is there any need to sandbox this stuff? just wondering if i should be sticking it into a vm of some sort rather than running straight on my workstation | 20:40 |
| fungi | i assume it works in debian | 20:40 |
| clarkb | it is all js so sandboxed by browser | 20:40 |
| fungi | oh, wfm | 20:40 |
| *** oubiwann_ has quit IRC | 20:42 | |
| *** bhuvan has quit IRC | 20:42 | |
| *** dhellman_ has joined #openstack-infra | 20:42 | |
| *** bhuvan has joined #openstack-infra | 20:42 | |
| *** bhuvan has joined #openstack-infra | 20:42 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable puppetdb from puppetmaster https://review.openstack.org/74612 | 20:42 |
| clarkb | fungi: I just clone their github repos and open index.html in chromium | 20:42 |
| *** jlibosva has quit IRC | 20:42 | |
| *** bhuvan__ has joined #openstack-infra | 20:43 | |
| fungi | oh, no installing needed. great | 20:43 |
| *** dcramer_ has quit IRC | 20:43 | |
| *** ekarlso has quit IRC | 20:43 | |
| *** bhuvan has quit IRC | 20:43 | |
| *** bhuvan__ has quit IRC | 20:43 | |
| fungi | https://github.com/lukas-vlcek/bigdesk.git ? | 20:43 |
| *** bhuvan has joined #openstack-infra | 20:44 | |
| *** DinaBelova is now known as DinaBelova_ | 20:44 | |
| *** ekarlso has joined #openstack-infra | 20:44 | |
| clarkb | yup | 20:44 |
| *** bhuvan_ has quit IRC | 20:45 | |
| valentinbud | clarkb: I got to this after checking that all the other components of the system work. The gearman client subscribes to Jenkins ZMQ, receives the message, pass it to the workers that pull the logs via HTTP and feed it to CRM114 and to Logstash which indexes in Elasticsearch. | 20:45 |
| clarkb | correct | 20:45 |
| *** mrda_away is now known as mrda | 20:46 | |
| valentinbud | clarkb: I didn't want to modify substantially the indexer.conf file to be able to test it locally so I tag input with oslofmt and type jenkins. That means they are processed in the else if oslofmt in [tags] filter. | 20:48 |
| clarkb | correct | 20:48 |
| clarkb | my current hunch is that the input codecs are doing smoething subtly different in ways that break multiline | 20:49 |
| valentinbud | Both the multiline filters make sense to me. Match lines that have TRACE and the ones that don't begin with TIMESTAMP_ISO8601. | 20:50 |
| *** ildikov_ has joined #openstack-infra | 20:50 | |
| valentinbud | I even tried to use multiline codec - http://logstash.net/docs/1.3.3/codecs/multiline with the same pattern. Had the same results. | 20:51 |
| *** weshay has quit IRC | 20:51 | |
| *** lcostantino has quit IRC | 20:51 | |
| clarkb | yeah, maybe they are all getting a timestamp in some weird way? | 20:51 |
| fungi | Bigdesk may not work correctly! Found ES node version: 0.90.9; Requires ES node version: >= 1.0.0.RC1 | 20:52 |
| fungi | i guess i should checkout an older tag | 20:52 |
| valentinbud | clarkb: http://64.119.130.115/39998/19/Hyper-V/hv-compute1/nova-compute.log.gz do this timestamps look weird? | 20:53 |
| clarkb | fungi: oh yeah, their readme should tell you which versions support other versions | 20:53 |
| clarkb | fungi: I can also give you my git sha if it helps | 20:53 |
| *** dangers_away is now known as dangers | 20:53 | |
| *** jergerber has joined #openstack-infra | 20:54 | |
| *** Ajaeger has joined #openstack-infra | 20:54 | |
| fungi | clarkb: v2.2.2 works, later tags give that warning | 20:54 |
| fungi | (for future reference) | 20:54 |
| *** dcramer_ has joined #openstack-infra | 20:55 | |
| Ajaeger | Infra-team, any ideas why Zuul is not picking up this review https://review.openstack.org/#/c/67223/ ? | 20:55 |
| clarkb | valentinbud: they look ok to me, but I would compare them to upstream jenkins logs | 20:55 |
| fungi | clarkb: so what specifically should i be looking at now if anything? (i see what you mean now when you say "the cluster is green") | 20:55 |
| *** jraim_ is now known as jraim | 20:56 | |
| valentinbud | I would do that first then. Thank you. | 20:56 |
| clarkb | fungi: under nodes, you can click individual nodes and see a bunch of metrics | 20:56 |
| clarkb | fungi: we will be itnerested in indexing requests per second, ram and cpu use | 20:56 |
| clarkb | fungi: elasticsearch head is the other tool you want to grab, same process as bigdesk | 20:56 |
| fungi | Ajaeger: was it wip when it got approved? | 20:56 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/requirements: Bump python-savannaclient to 0.5.0 https://review.openstack.org/76357 | 20:56 |
| clarkb | elasticsearch head is helpful because it shows you shard state in an easy to view way | 20:56 |
| *** mbacchi has joined #openstack-infra | 20:57 | |
| Ajaeger | fungi: yes, it was. And then I approved it again | 20:57 |
| fungi | Ajaeger: we may have a hole in our layout.yaml gate enqueuing logic there, if the change already has +2 vrfy and it not merged | 20:58 |
| fungi | since that happens rarely | 20:58 |
| Ajaeger | so, what can I do? | 20:59 |
| fungi | Ajaeger: i just rechecked it... it will re-gate after the current check run finishes. please open a bug about this against openstack-ci so we remember to address it | 20:59 |
| Ajaeger | fungi: will do! | 20:59 |
| clarkb | fungi: let me know when you have elasticsearch head up to and have had a chance to poke around with the tools | 21:00 |
| *** dcramer_ has quit IRC | 21:00 | |
| fungi | clarkb: what's the plan for upgrading elasticsearch? (presumably after we get the cluster migrated to the new nodes) | 21:00 |
| clarkb | fungi: note head in particular gives you the ability to delete stuff so be a little careful :) | 21:00 |
| *** dhellman_ has quit IRC | 21:00 | |
| clarkb | fungi: that is a tough one since there are breaking changes between 0.90.9 | 21:01 |
| fungi | clarkb: ohhh... you're suggesting i run a copy of elasticsearch too, not just elasticsearch bigdesk connected to the existing cluster? i totally misunderstood what you meant there | 21:01 |
| clarkb | fungi: no no | 21:01 |
| clarkb | 'elasticsearch head' | 21:01 |
| clarkb | it is another tool like bigdesk runs in your browser shows you stuff | 21:01 |
| *** hashar has joined #openstack-infra | 21:01 | |
| fungi | is not the same thing as the head of elasticsearch's master branch | 21:01 |
| dhellmann | sdague: about https://review.openstack.org/#/c/74408/ -- I need to read up, but I wanted to make sure you saw the comments before I start re-implementing anything | 21:01 |
| fungi | ahhhhhhhhhhhhhhh | 21:01 |
| fungi | *lightbulb* | 21:01 |
| *** david_lyle is now known as david-lyle | 21:01 | |
| clarkb | fungi: in particular it shows you shard state | 21:02 |
| fungi | getting | 21:02 |
| clarkb | which is really handy when we start adding and removing nodes | 21:02 |
| *** thedodd has quit IRC | 21:02 | |
| fungi | clarkb: so https://github.com/mobz/elasticsearch-head | 21:02 |
| clarkb | ya | 21:03 |
| sdague | dhellmann: yeh, ok, so the road is really a functional test suite? | 21:03 |
| Ajaeger | fungi: filed - https://bugs.launchpad.net/openstack-ci/+bug/1284842 | 21:03 |
| fungi | clarkb: neat, i've got it up in another tab too | 21:04 |
| fungi | Ajaeger: thanks! | 21:04 |
| Ajaeger | fungi: I have to say thanks! | 21:04 |
| *** rcarrillocruz has joined #openstack-infra | 21:04 | |
| clarkb | fungi: feel free to poke around in both tools and get a feel for what they are showing you | 21:04 |
| *** Ajaeger has quit IRC | 21:05 | |
| clarkb | fungi: as I think having that mental model is a good thing | 21:05 |
| nibalizer | fungi: so looks like puppetdb fix + connector landed | 21:05 |
| clarkb | particularly for the shard layout | 21:05 |
| fungi | nibalizer: yes, i'm about to go back and see what the logs have to say now | 21:05 |
| *** rcarrillocruz1 has quit IRC | 21:05 | |
| dhellmann | sdague: I want to make sure we don't commit anything to oslo.test that breaks unit tests in the projects using it | 21:05 |
| dhellmann | sdague: I don't know if that counts as "functional" or not, but it seems like the way to test it is to run the unit tests of those projects | 21:06 |
| nibalizer | tail -f /var/log/puppetdb/puppetdb.log should say things like 'executed "replaced facts" for node foo.o.o" | 21:06 |
| *** dprince_ has quit IRC | 21:06 | |
| sdague | right, so you'll have a job which jumps into the various unit test directories and runs the tests | 21:07 |
| fungi | nibalizer: the syslog says... | 21:07 |
| fungi | nibalizer: Feb 25 20:49:20 ci-puppetmaster puppet-agent[32308]: (/Stage[main]/Puppetdb::Master::Config/Service[puppetmaster]) Failed to call refresh: Could not find init script or upstart conf file for 'puppetmaster' | 21:07 |
| sdague | so... have you thought about the duration of said job? | 21:07 |
| jeblair | dhellmann: so this is really (test + incubator), (test + config), (test + messaging)? | 21:08 |
| sdague | because neutron 2.7 unit tests come in at 40 minutes | 21:08 |
| jeblair | dhellmann: and not (test + incubator + config + messaging) | 21:08 |
| dhellmann | jeblair: right, the former | 21:08 |
| *** eharney has quit IRC | 21:08 | |
| dhellmann | sdague: I'd be happy if we could do it as multiple parallel jobs | 21:08 |
| fungi | nibalizer: but regardless, looking at the puppetdb log we seem to be getting events, so should hopefully be good to go for slapping puppetboard atop of this? | 21:08 |
| sdague | dhellmann: well that's already parallel using all the codes | 21:08 |
| dhellmann | sdague: this was a first stab, so we could start using the lib in oslo without breaking ourselves :-) | 21:08 |
| sdague | core | 21:09 |
| sdague | cores | 21:09 |
| dhellmann | sdague: I mean neutron in one job and nova in another, on different hosts | 21:09 |
| sdague | ok | 21:09 |
| dhellmann | like we do with py27 and py26 tests | 21:09 |
| sdague | sure | 21:09 |
| jeblair | dhellmann: you're also running neutron and nova tests? | 21:09 |
| nibalizer | fungi: I assume we run puppet under apache/mod_passsenger/rack ? | 21:09 |
| dhellmann | jeblair: we will, when they start consuming the library | 21:09 |
| tomhe | Our Zuul setup has recently had a problem with the embedded Gearman server not registering a Jenkins job. I had to delete and recreate the job a couple of times until it listed as registered. Any ideas to what could be causing this? (On Zuul from master as of Fri Feb 21 08:28:58) | 21:09 |
| nibalizer | i'll send a review your way | 21:09 |
| dhellmann | jeblair: *all* integrated projects | 21:09 |
| sdague | jeblair: yeh, this is about forward testing | 21:09 |
| sdague | so this library can't break those | 21:10 |
| dhellmann | I want to add the job to prevent us breaking before we land the change to start using the new lib | 21:10 |
| fungi | clarkb: in bigdesk, what feats has elasticsearch4 performed to earn a medal of honor? | 21:10 |
| jeblair | yeah, i just didn't realize the scope extended outside oslo. got it. | 21:10 |
| fungi | it looks plenty important | 21:10 |
| clarkb | fungi: it was elected the master | 21:10 |
| fungi | oic | 21:10 |
| dhellmann | jeblair: sure, that wasn't clear from the review | 21:10 |
| jeblair | dhellmann: so in the future, you might have a test that's testing (oslo.test + oslo.messaging + nova) together? | 21:11 |
| jeblair | dhellmann: or would you still be doing (oslo.test + nova) ? | 21:11 |
| dhellmann | jeblair: I don't think so, no | 21:11 |
| dhellmann | jeblair: just oslo.test + foo | 21:11 |
| clarkb | fungi: other points of interest, on the GC graph, if the pink line goes above the blue line it usually means big trouble | 21:12 |
| *** rcarrillocruz1 has joined #openstack-infra | 21:12 | |
| clarkb | fungi: that implies a few GC runs are taking massive amounts of CPU time | 21:12 |
| dhellmann | clarkb: don't cross the streams | 21:12 |
| clarkb | which usually leads to the cluster falling over beacuse intercluster pings stop getting processed | 21:12 |
| jeblair | dhellmann: nice! :) | 21:12 |
| *** smarcet has quit IRC | 21:13 | |
| *** rcarrillocruz has quit IRC | 21:13 | |
| fungi | clarkb: noted | 21:13 |
| clarkb | fungi: other points of interest, in elasticsearc-head if yo uclick on a green shard box you get a pop up thingy with lots of info | 21:13 |
| *** mrmartin has quit IRC | 21:14 | |
| clarkb | this is handy when you watch shards get restored as you can see progress in the size of the recovering shard | 21:14 |
| clarkb | fungi: when you are comfortable I think we can go ahead and turn on puppet on the first node and see what happens | 21:15 |
| fungi | okay, i think 67223 must not be an enqueuing logic issue in zuul... probably need to go digging in the log | 21:15 |
| sdague | dhellmann: so look at solum, barbican, they have gate_hook scripts to do this | 21:15 |
| sdague | and the swift functional test | 21:15 |
| clarkb | fungi: just ping me and I will pay attention wen you are ready for that | 21:16 |
| sdague | which gives you the d-g environment, but you can do anythign you want from there | 21:16 |
| dhellmann | sdague: ok, sounds good -- I went this direction based on our previous conversation so I wanted to make sure you knew about the change | 21:16 |
| fungi | clarkb: will do--thanks@ | 21:16 |
| clarkb | fungi: agree re 67223 | 21:16 |
| jeblair | dhellmann: ok. i think that still the combination of two projects is still worth using the d-g code to set up the repos. also, i think we can template that so you can do this as 3 separate jobs | 21:16 |
| sdague | yeh, I just didn't fully grok it | 21:16 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1255627 https://review.openstack.org/76265 | 21:16 |
| dhellmann | jeblair: that sounds great | 21:16 |
| sdague | appologies for sending you down a bad path | 21:16 |
| jeblair | dhellmann: that way it's just a matter of substituting in 'foo' to add a new one | 21:16 |
| dhellmann | sdague: not a big deal, just trying to keep everyone on the same page | 21:16 |
| dhellmann | jeblair: I'll see if I can take a stab at implementing it that way, unless you think you'll have time | 21:17 |
| jeblair | someday we'll genericize this a bit more so you don't have to use something called "devstack-gate" to get your repos set up... hashar wants that too. :) | 21:17 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Set apache as the puppet service name https://review.openstack.org/76366 | 21:17 |
| *** dolphm is now known as dolphm_503 | 21:18 | |
| nibalizer | fungi: that ^^ should deal with the 'no service puppetmaster' problem' | 21:18 |
| *** dmsimard has left #openstack-infra | 21:18 | |
| *** miqui has quit IRC | 21:18 | |
| jeblair | dhellmann: i'll take a stab at setting up the jjb template to call your script | 21:18 |
| jeblair | (via devstack-gate) | 21:18 |
| *** jhesketh has joined #openstack-infra | 21:18 | |
| dhellmann | jeblair: ok, let me know what args I'll need to handle (I need to look at how those hooks work still, today has been all meetings) | 21:19 |
| jhesketh | Morning | 21:19 |
| *** jhesketh_ has joined #openstack-infra | 21:19 | |
| *** tjones has joined #openstack-infra | 21:19 | |
| jeblair | jhesketh: good morning! | 21:21 |
| hashar | jeblair: dhellmann I even have a patch pending somewhere that reimplements devstach shell script in pure python. Nicknamed that zuul-cloner. | 21:21 |
| jhesketh | that stuff was cool hashar :-) | 21:21 |
| jeblair | hashar: yeah, i can see a future where we install that on all our slaves, and then devstack-gate calls it, and any other job can too | 21:21 |
| *** thedodd has joined #openstack-infra | 21:21 | |
| hashar | dhellmann: the idea is to pass a list of repositories clone them then for each repository attempt to apply the Zuul ref it exists, fallback to the branch if it exists, else master. | 21:21 |
| mordred | hashar: ++ | 21:22 |
| jeblair | hashar: also, since it's in python, we can have it produce sensible debug/error messages which we don't get in bash | 21:22 |
| *** julim has quit IRC | 21:23 | |
| *** oubiwann_ has joined #openstack-infra | 21:23 | |
| hashar | I resurrected the patchset which is https://review.openstack.org/#/c/70373/ | 21:23 |
| fungi | jeblair: https://launchpad.net/bugs/1284842 may be of interest to you | 21:24 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Openstackid deployment symlink override fix https://review.openstack.org/76368 | 21:24 |
| hashar | I am busy integrating Selenium tests to be triggered with Gerrit. So I am unlikely to work on that zuul cloner anytime soon :-/ | 21:24 |
| jeblair | hashar: cool, thanks | 21:24 |
| *** mrmartin has joined #openstack-infra | 21:24 | |
| anteaya | tomhe: the ones who can answer your question are jeblair fungi and clarkb and all are currently in the middle of something | 21:25 |
| jeblair | fungi: ack | 21:25 |
| anteaya | tomhe: I suggest patience and try again once one of them appears free | 21:25 |
| mrmartin | fungi: a very trivial fix for the openstackid deployment problem: https://review.openstack.org/#/c/76368/1/modules/openstackid/files/functions if you have some time plz. approve | 21:26 |
| mrmartin | then we also need an rm -rf /srv/openstackid to cleanup the application directory. | 21:26 |
| fungi | mrmartin: okay, can do | 21:26 |
| mrmartin | thnx | 21:26 |
| hashar | mordred: jeblair: Turbo-hipster author might be interested as well though I forgot to follow up. | 21:26 |
| jeblair | hashar: that's jhesketh | 21:26 |
| jeblair | hashar: he just said it's cool. ;) | 21:27 |
| jog0 | hoping to get an infra review on this e-r fingerprint https://review.openstack.org/#/c/76043/1/queries/1284371.yaml | 21:27 |
| fungi | mrmartin: i see only some of your symlinks use force. do the others not need it? | 21:27 |
| mrmartin | I guess no | 21:27 |
| fungi | that answer works for me then ;) | 21:27 |
| jog0 | being its an ifra bug | 21:27 |
| *** chandan_kumar has quit IRC | 21:27 | |
| mrmartin | we can enhance those scripts later, but lets deliver something minimal that works | 21:28 |
| *** zul has quit IRC | 21:28 | |
| fungi | nibalizer: thanks! | 21:28 |
| hashar | jeblair: jhesketh hipster thing might well let me replace Jenkins one day :-] If manage to add some plugins to turbo-hipster hehe. | 21:28 |
| fungi | nibalizer: is puppet_service_name (apache) going to be different from the initscript service name (apache2) or is that a typo? | 21:29 |
| jog0 | jeblair: thanks! | 21:29 |
| jeblair | hashar: that's the idea and what we're working toward. the swift log upload is the next step in that (replaces jenkins scp artifact archiving) | 21:29 |
| hashar | jeblair: swift is a good idea. I guess it is more scalable than a 2TB disk | 21:30 |
| jeblair | tomhe: i don't have an immediate suggestion for a fix, but to help debug, you can look in the jenkins logs (gearman-plugin is fairly verbose) and you can enable DEBUG level logging for the zuul gear server | 21:30 |
| nibalizer | fungi: typo for sure | 21:30 |
| nibalizer | i've been in 'httpd' (ungh) land for tooo long | 21:30 |
| jeblair | tomhe: with that you should be able to at least find out if the problem is gearman-plugin or gear server | 21:31 |
| fungi | nibalizer: thanks--no worries | 21:31 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Set apache as the puppet service name https://review.openstack.org/76366 | 21:31 |
| nibalizer | fungi: updated | 21:31 |
| fungi | nibalizer: lgtm. thanks again | 21:33 |
| *** alexpilotti has quit IRC | 21:33 | |
| *** gyee has joined #openstack-infra | 21:34 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1284371 https://review.openstack.org/76043 | 21:34 |
| nibalizer | happy to do it | 21:34 |
| nibalizer | i'll start working on the puppetboard change | 21:34 |
| nibalizer | probably wont give it to you today, perhaps tomorrow | 21:34 |
| fungi | nibalizer: awesome... i can't wait to look it over | 21:34 |
| *** dizquierdo has joined #openstack-infra | 21:34 | |
| nibalizer | we were thinking puppetboard on the puppetdb host at first for simplicity right? | 21:35 |
| fungi | nibalizer: that was my understanding | 21:35 |
| nibalizer | okay cool | 21:35 |
| fungi | clarkb: okay, i'm starting (and configuring to start on boot) the puppet agent on elasticsearch01 unless you have objections | 21:35 |
| *** cody-somerville has quit IRC | 21:36 | |
| clarkb | no objections 01 is a fine place to start | 21:37 |
| *** flaper87|afk is now known as flaper87 | 21:37 | |
| * fungi always likes to start at the beginning | 21:37 | |
| openstackgerrit | A change was merged to openstack-infra/config: Openstackid deployment symlink override fix https://review.openstack.org/76368 | 21:38 |
| fungi | mrmartin: in another 7 minutes that ^ will be on our puppetmaster, at which point i'll clear away /srv/openstackid again in preparation for the next update cycle | 21:39 |
| mrmartin | fungi, great, with this patch it was working in my vargant environment | 21:39 |
| fungi | clarkb: so, we've got an elasticsearch01 already appearing in bigdesk. that was quick | 21:39 |
| *** amcrn has quit IRC | 21:40 | |
| fungi | clarkb: also in es-head | 21:40 |
| clarkb | fungi: cool, the heap mem numbers bother me though | 21:40 |
| clarkb | fungi: it should have mlockall'd 30g of memory | 21:40 |
| clarkb | but doesn't appear to have done that | 21:40 |
| fungi | clarkb: i'll check its config | 21:41 |
| clarkb | fungi: you should be able to safely stop the service if you need to since that node doesn't have any data on it yet | 21:41 |
| jog0 | heh jenkins will rerun if a 3rd party CI server comments | 21:43 |
| jog0 | hopefully 3rd party wont rerun if jenkins comments otherwise we may get a loop | 21:44 |
| fungi | clarkb: the es-head jvm.mem is where you were looking, right? | 21:44 |
| fungi | "heap_max: 989.8mb" | 21:44 |
| clarkb | fungi: I was looking at bigdsek but similar numbers | 21:45 |
| anteaya | jog0: where are you seeing that? | 21:45 |
| *** thomasem_ has joined #openstack-infra | 21:45 | |
| clarkb | fungi: committed heap mem is just over 200MB | 21:45 |
| clarkb | that should be like 29.9GB | 21:45 |
| *** dhellman_ has joined #openstack-infra | 21:46 | |
| *** luis_ has joined #openstack-infra | 21:47 | |
| fungi | clarkb: as seen in /etc/default/elasticsearch... | 21:47 |
| fungi | # Heap Size (defaults to 256m min, 1g max) | 21:47 |
| fungi | ES_HEAP_SIZE=30g | 21:47 |
| clarkb | ok so that went in properly, I am going to blame mlockall then | 21:47 |
| fungi | maybe the file was puppeted in after the package was installed/service was started? | 21:47 |
| clarkb | fungi: ohhhh | 21:47 |
| *** oubiwann_ has quit IRC | 21:47 | |
| *** thomasem has quit IRC | 21:47 | |
| *** lnxnut_ has joined #openstack-infra | 21:48 | |
| clarkb | fungi: lets double check the mlockall settings too then restart the service | 21:48 |
| * fungi always suspects puppet order of operations logic first | 21:48 | |
| *** eharney has joined #openstack-infra | 21:48 | |
| clarkb | fungi: in /etc/elasticsearch/configsomething.yml there should be an mlockall setting set to true | 21:48 |
| *** fbo is now known as fbo_away | 21:48 | |
| clarkb | but it will only work if ES_HEAP_SIZE is properly set | 21:48 |
| *** ok_delta has quit IRC | 21:48 | |
| fungi | bootstrap.mlockall: true | 21:49 |
| fungi | that? | 21:49 |
| fungi | if so, then yes it's there | 21:49 |
| *** luisg has quit IRC | 21:50 | |
| fungi | "Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set to the same value...make sure that the ElasticSearch process is allowed to lock the memory, eg. by using `ulimit -l unlimited`." | 21:50 |
| fungi | do we need to check any of that? | 21:50 |
| clarkb | no, ES_HEAP_SIZE sets it for us | 21:50 |
| clarkb | so I think your theory is sound | 21:50 |
| *** lnxnut has quit IRC | 21:51 | |
| clarkb | just do an service stop then start | 21:51 |
| fungi | okay, i'll restart the service and see whether it changes | 21:51 |
| clarkb | and let us see how that goes | 21:51 |
| *** kynes has quit IRC | 21:51 | |
| *** cody-somerville has joined #openstack-infra | 21:51 | |
| *** cody-somerville has quit IRC | 21:51 | |
| *** cody-somerville has joined #openstack-infra | 21:51 | |
| fungi | "heap_max: 29.8gb" | 21:52 |
| clarkb | much better | 21:53 |
| *** cadenzajon has quit IRC | 21:53 | |
| fungi | so i'll see about fixing that in the puppetingness | 21:53 |
| *** yolanda_ has joined #openstack-infra | 21:53 | |
| *** dkliban has quit IRC | 21:53 | |
| fungi | mrmartin: btw your updated deploy has happened, looks like | 21:53 |
| clarkb | though I think the cluster is crankier about the flap than I anticipated... oh well it should eventually become appy again | 21:53 |
| mrmartin | oh great, we just need to setup the membership db with some fake test data | 21:54 |
| *** jcoufal has quit IRC | 21:54 | |
| mrmartin | fungi: thanks for the help | 21:54 |
| fungi | mrmartin: sure thing. if you need me to source a mysqldump into it or something, let me know | 21:54 |
| *** apevec has joined #openstack-infra | 21:54 | |
| *** gokrokve_ has joined #openstack-infra | 21:55 | |
| clarkb | fungi: ok it is happier now, over in elasticsearch-head you can see two purple shards, they are relocating to the new node | 21:55 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: add some helpfulness to our log output https://review.openstack.org/76349 | 21:56 |
| clarkb | it will relocate shards to balance things out, I think we should let it do that until it is appy then we can turn off es6 | 21:56 |
| clarkb | it will then move more data around, then we can add 02 | 21:56 |
| mrmartin | fungi: I think we need some real x509 SSL certs, because for dev we are currently using self-signed certificates, but I'll open a blueprint about that | 21:56 |
| clarkb | rince and repeat | 21:56 |
| *** jcoufal has joined #openstack-infra | 21:56 | |
| *** Sukhdev has joined #openstack-infra | 21:56 | |
| fungi | clarkb: sounds great. i'm going to pop out for dinner shortly and then resume with the fun | 21:57 |
| fungi | mrmartin: okay, we can get those for you when needed for production--no need for a blueprint | 21:57 |
| *** gokrokve has quit IRC | 21:57 | |
| fungi | mrmartin: jeblair usually takes care of them | 21:57 |
| *** dcramer_ has joined #openstack-infra | 21:57 | |
| *** esker has quit IRC | 21:58 | |
| clarkb | fungi: sounds good, thank you for the help | 21:58 |
| mrmartin | fungi: ok, I did some testing yesterday with a local gerrit installation, and I had problems with self-signed certs, needed to tweak the java cert store | 21:58 |
| fungi | mrmartin: very good point. it may be easier in this case for us to add one for dev as well, given the nature of that particular project | 21:59 |
| fungi | rather than trying to hunt down all the ways we need to set other consuming dev instances of applications to not validate or manually update their trusts | 21:59 |
| *** dhellman_ has quit IRC | 21:59 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove old queries https://review.openstack.org/76196 | 22:00 |
| *** thomasem_ has quit IRC | 22:02 | |
| *** bhuvan_ has joined #openstack-infra | 22:03 | |
| *** lnxnut_ has quit IRC | 22:03 | |
| *** lnxnut has joined #openstack-infra | 22:04 | |
| mrmartin | jeblair: could you make a valid x509 certs for openstackid-dev.openstack.org domain? | 22:04 |
| jeblair | mrmartin: yes, i'll do that this week; i have another cert to get too. | 22:05 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct elasticsearch config dependency ordering https://review.openstack.org/76376 | 22:05 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch workers https://review.openstack.org/76051 | 22:05 |
| *** jeckersb is now known as jeckersb_gone | 22:06 | |
| fungi | clarkb: merge or fiddle with 76376 as needed while i'm out at dinner, and then we can test it when we bring up the next cluster member | 22:06 |
| mrmartin | jeblair, please let me know when the cert is ready, and I'll prepare the config for that. What is the usual way, will we inject the private key / cert through hiera variables? | 22:06 |
| *** bhuvan has quit IRC | 22:07 | |
| fungi | mrmartin: that is precisely how we do it, yes. you can see examples of that throughout the openstack-infra/config repository | 22:07 |
| mrmartin | great, thank you guys for the help. | 22:07 |
| *** bhuvan_ has quit IRC | 22:08 | |
| *** bhuvan has joined #openstack-infra | 22:08 | |
| fungi | mrmartin: i believe i already configured the openstackid module that way | 22:08 |
| fungi | so we just update what's in hiera if so | 22:08 |
| *** lnxnut has quit IRC | 22:08 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 22:08 |
| mrmartin | fungi: we need to enlist the ssl params at the head of openstack_project::openstackid_dev class, but openstackid supports the ssl path variables. | 22:10 |
| *** rossella has joined #openstack-infra | 22:11 | |
| fungi | okay, great | 22:12 |
| * fungi heads to dinner... back in a while | 22:12 | |
| tomhe | jeblair: Ok, thanks | 22:14 |
| jhesketh | jeblair: your queue-name patch... would it be more sensible for the queue name to be on the job description in the layout.yaml? Specifically it is the job that joins queues together, not projects | 22:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1273908 https://review.openstack.org/76335 | 22:15 |
| *** oubiwann_ has joined #openstack-infra | 22:16 | |
| *** hashar has quit IRC | 22:16 | |
| mrmartin | fungi: could you send me the latest puppet run of openstackid? | 22:16 |
| *** dcramer__ has joined #openstack-infra | 22:17 | |
| *** jcoufal has quit IRC | 22:17 | |
| apevec | sdague, I need one more devstack core review for https://review.openstack.org/76189 (issue blocks stable/grizzly and havana) | 22:17 |
| *** dcramer___ has joined #openstack-infra | 22:19 | |
| sdague | apevec: done | 22:19 |
| apevec | thanks! | 22:19 |
| jeblair | jhesketh: that's true. that may be more correct and not significantly less convenient to configure. presumable we would set that for 'gate-mumble-tempest-mumble'... | 22:20 |
| *** jcoufal has joined #openstack-infra | 22:20 | |
| jhesketh | yeah, that's my thinking | 22:20 |
| *** dcramer___ has quit IRC | 22:20 | |
| *** dcramer___ has joined #openstack-infra | 22:21 | |
| *** flaper87 has quit IRC | 22:22 | |
| *** flaper87 has joined #openstack-infra | 22:22 | |
| *** dcramer_ has quit IRC | 22:22 | |
| *** dcramer__ has quit IRC | 22:23 | |
| *** shashank_ has joined #openstack-infra | 22:24 | |
| jhesketh | jeblair: I've added a few more thoughts in my review | 22:25 |
| jeblair | jhesketh: thx | 22:25 |
| *** lnxnut has joined #openstack-infra | 22:26 | |
| *** bhuvan has quit IRC | 22:26 | |
| *** bhuvan has joined #openstack-infra | 22:27 | |
| *** dcramer___ has quit IRC | 22:27 | |
| *** thomasem has joined #openstack-infra | 22:28 | |
| *** salv-orlando has joined #openstack-infra | 22:29 | |
| *** dolphm_503 is now known as dolphm | 22:29 | |
| *** denis_makogon has quit IRC | 22:30 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add oslo.test integration test https://review.openstack.org/76381 | 22:30 |
| *** mfer has quit IRC | 22:30 | |
| jeblair | dhellmann, sdague: ^ there's a rough cut of what that looks like. i realize i have been operating under the assumption that we want a symmetric gate between oslo.test and foo.... | 22:31 |
| *** dcramer___ has joined #openstack-infra | 22:32 | |
| jeblair | dhellmann, sdague: however, if that's not the case, and we only want to test proposed oslo.test changes with foo master in an asymmetric gate, then the original approach (with some changes to use cached git repos and the pypi mirror) would probably be ok. | 22:32 |
| *** ociuhandu has joined #openstack-infra | 22:33 | |
| jeblair | dhellmann, sdague: (this would work for that too, but is slightly overkill) | 22:33 |
| *** valentinbud has quit IRC | 22:33 | |
| *** miqui has joined #openstack-infra | 22:34 | |
| *** yolanda_ has quit IRC | 22:35 | |
| *** jergerber has quit IRC | 22:36 | |
| *** CaptTofu_ has quit IRC | 22:38 | |
| *** mwagner_lap has quit IRC | 22:39 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add logstash job queue graph https://review.openstack.org/76056 | 22:40 |
| *** thomasem has quit IRC | 22:42 | |
| *** jergerber has joined #openstack-infra | 22:43 | |
| *** mriedem has quit IRC | 22:44 | |
| *** thedodd has quit IRC | 22:44 | |
| *** jcoufal has quit IRC | 22:46 | |
| *** weshay has joined #openstack-infra | 22:46 | |
| *** cody-somerville has quit IRC | 22:49 | |
| *** jerryz has joined #openstack-infra | 22:49 | |
| *** mbacchi has quit IRC | 22:50 | |
| *** shashank_ has quit IRC | 22:51 | |
| *** rcleere has quit IRC | 22:53 | |
| *** vkozhukalov_ has quit IRC | 22:57 | |
| pleia2 | can we get this nudged along? https://review.openstack.org/#/c/70007/ (would help me out to carry fewer local patches right now) | 22:58 |
| pleia2 | clarkb: you mentioned you wanted to test it in case of weird firewall things | 22:58 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Update trigger script for new zuul url parameter https://review.openstack.org/73392 | 22:59 |
| clarkb | pleia2: ah yes, I can do a quick application of that somewhere | 22:59 |
| pleia2 | thanks | 22:59 |
| *** shashank_ has joined #openstack-infra | 22:59 | |
| clarkb | jeblair: jhesketh: mordred: https://review.openstack.org/#/c/76057/1 trying to decide if I actually like that | 23:00 |
| openstackgerrit | A change was merged to openstack-infra/meetbot: Fix misspellings in meetbot https://review.openstack.org/72048 | 23:01 |
| clarkb | jeblair: jeblair mordred part of me thinks that the zuul merger should be trusted then it can push to untrusted git replicas which the jobs consume instead of the zuul merger directly | 23:01 |
| clarkb | jhesketh: ^ tab complete fail sorry | 23:01 |
| openstackgerrit | A change was merged to openstack-infra/os-loganalyze: Fix misspellings in os-loganalyze https://review.openstack.org/72052 | 23:01 |
| mordred | I like that you talk to jeblair twice | 23:01 |
| jhesketh | his opinion matters twice as much ;-) | 23:01 |
| lifeless | or he listens half as much :P | 23:02 |
| clarkb | what do you guys think? should I continue to follow the train of thought in that WIP'd change or make mordred finish the zuul merger replication work | 23:02 |
| *** bhuvan_ has joined #openstack-infra | 23:02 | |
| *** dcramer___ has quit IRC | 23:02 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Correct the example of zuul enqueue command https://review.openstack.org/72308 | 23:02 |
| clarkb | jhesketh: your comment makes me think this ins't a terrible idea, but at the same time it seems clunky. Liek a more generic thing would be to have zuul mergers hand off a url to an arbitrary place then they can be trusted | 23:02 |
| *** dcramer___ has joined #openstack-infra | 23:03 | |
| *** ryanpetrello has quit IRC | 23:05 | |
| jhesketh | clarkb: not sure what you mean by the arbitrary place that they can be trusted? | 23:05 |
| *** openstack has joined #openstack-infra | 23:06 | |
| clarkb | at a network level they do | 23:06 |
| clarkb | which is what we are working around, if there is a btter term I can use let me know | 23:07 |
| jeblair | clarkb: zuul already supports a completely private access model -- don't run a public http server, and require your workers to ssh into the zuul mergers to fetch. | 23:07 |
| *** bhuvan_ has quit IRC | 23:07 | |
| *** bhuvan has joined #openstack-infra | 23:07 | |
| jeblair | clarkb: these changes are instead both just to work around arbitrary firewall rules. "don't allow incoming connections" is not actually a security model based on trust. | 23:07 |
| jhesketh | so I kind of like the thought process that a merger should be trusted. Separating out the mergers was only ever a way of scaling horizontally and therefore should sit next to zuul... with mordred's change to push refs elsewhere that could be enough for workers to grab their refs from? | 23:08 |
| jeblair | that's kind of beside the point other than i would suggest that talking about "trust" isn't the right way to think about this. | 23:08 |
| clarkb | thats fair | 23:08 |
| *** lcostantino has joined #openstack-infra | 23:09 | |
| *** changbl has quit IRC | 23:09 | |
| *** dcramer__ has joined #openstack-infra | 23:09 | |
| *** ryanpetrello has joined #openstack-infra | 23:10 | |
| *** dcramer___ has quit IRC | 23:10 | |
| mordred | yeah. I can agree withthat | 23:10 |
| clarkb | so to reframe, slaves can only talk to a similarly hosted git mirror (either zuul merger serving out its refs or a replica) in "external" space and they cannot talk to an internal gerrit | 23:11 |
| clarkb | at least not from external -> internal. We can have internal push out to external all we want | 23:12 |
| clarkb | from zuul's perspective does making the zuul merger push to an external replica make more sense than running an external zuul merger that can serve refs without talking to gerrit? | 23:12 |
| jeblair | clarkb: in either case you are doubling the number of times you transfer the refs. the overall performance is probably similar. | 23:13 |
| jeblair | clarkb: from an application engineering perspective, you've got the mergers on either the inside or outside of the evil corporate firewall... | 23:14 |
| *** dcramer__ has quit IRC | 23:14 | |
| *** markmcclain has quit IRC | 23:14 | |
| *** ryanpetrello has quit IRC | 23:14 | |
| jeblair | clarkb: so one thing to consider is: where do you really want them? inside the firewall for stability? outside the firewall for scalability? | 23:14 |
| jeblair | clarkb: because basically the use case here is that you're running most of the app internally but using external cloud resources for some of the computation. | 23:15 |
| clarkb | correct | 23:15 |
| jeblair | clarkb: of course, if you put the mergers inside, you still need git replicas outside. | 23:15 |
| *** dstanek has quit IRC | 23:15 | |
| *** dkliban has joined #openstack-infra | 23:15 | |
| clarkb | mordred: ^ you are in a btter position to answer these questions than I am | 23:16 |
| *** jgrimm has quit IRC | 23:16 | |
| mordred | well, although I agree with jeblair on the general idiocy of the points of view of evil corporate overlords | 23:17 |
| *** dkranz has quit IRC | 23:17 | |
| mordred | the current situation is that outbound ssh is "trusted" - so replicating out to a git replicate is "more better safer ESTUPID" | 23:17 |
| *** pdmars has quit IRC | 23:17 | |
| jeblair | clarkb: anyway, i'm okay with both patches in principle. as for which approach would be better in this case: i'm available for implementation consulting at extremely unreasonable rates. ;) | 23:17 |
| mordred | but ... I think I'd rather that we don't just put ina crutch for this - but rather if one of the approaches is actually a better fit in some way for zuul | 23:18 |
| mordred | jeblair: :) | 23:18 |
| clarkb | right which is why I ask, I don't think we need both patches | 23:18 |
| clarkb | we here == zuul upstream | 23:18 |
| mordred | yah | 23:18 |
| jhesketh | agreed | 23:19 |
| jeblair | clarkb: oh yay welcome back. :) | 23:19 |
| clarkb | zuul shouln't need to have N firewall workarounds. Just one sufficiently useable one | 23:20 |
| jhesketh | my vote for the two options is to expect mergers to sit next to zuul and therefore use zuul/gerrit's repos and push references arbitrarily. I also think this gives zuul more flexibility. It allows zuul to place refs wherever is required as opposed to requiring a zuul-merger to be installed at the destination | 23:20 |
| jeblair | okay, i'm trying to strain to think of a use for each of these patches other than as a firewall workaround | 23:20 |
| jeblair | clarkb: yours is a general 'put something in front of gerrit' approach which might help people in seriously large gerrit situations (like where there is no direct gerrit access) | 23:21 |
| jeblair | mordred: jhesketh makes a good point about yours, though it's hard for me to see a concrete application of it other than as a firewall workaround. | 23:23 |
| *** ociuhandu has quit IRC | 23:23 | |
| jeblair | (like _why_ would you want to push refs to samething other than the merger? maybe you want really big permanent git servers somewhere and ephemeral mergers? not sure why, but your patch would support that) | 23:23 |
| clarkb | yeah I like mordreds too | 23:23 |
| jeblair | clarkb: oddly enough, i think yours is the most likely to be useful by openstack-infra. because it's possible we may someday decide we want to have zuul hit the apache mirror. | 23:24 |
| clarkb | oh good point | 23:24 |
| fungi | i was leaning the same direction | 23:25 |
| clarkb | especially if the number of zuul mergers becomes quite large | 23:25 |
| * fungi also caught up on scrollback now | 23:25 | |
| jeblair | clarkb: indeed. | 23:25 |
| mordred | yeah. I think I agree with jeblair | 23:26 |
| jhesketh | as in instead of gerrit, fetch from git.o.o? | 23:26 |
| mordred | we've hit the scaling wall of needing to move off of a single location for cloning a few times now :) | 23:26 |
| jeblair | clarkb, mordred: if you want to twist my arm and force me to care about this, i think we should seriously consider both. | 23:27 |
| jeblair | jhesketh: that, but also we run an apache mirror at git.o.o that overlays gerrit | 23:28 |
| jeblair | gah | 23:28 |
| jeblair | jhesketh: that, but also we run an apache mirror at review.o.o that overlays gerrit | 23:28 |
| clarkb | jeblair: I am open to considering both too, my initial reaction is only one should be necessary but we seem to be coming up with decent argumetns for both that don't overlap | 23:28 |
| jeblair | jhesketh: so if you fetch from https://review/p/foo | 23:28 |
| mordred | jeblair: as in, finishing/applying both? or consider their relative merits further? | 23:28 |
| jeblair | jhesketh: that's actually served by apache from a location gerrit replicates to | 23:29 |
| jhesketh | right, yes | 23:29 |
| fungi | it just *happens* to reside on the same machine | 23:29 |
| jeblair | (to reduce load on gerrit itself which as it turns out isn't the worlds best static content server) | 23:29 |
| jeblair | mordred: apply both. | 23:30 |
| fungi | yaknow... "java" | 23:30 |
| mordred | jeblair: interesting | 23:31 |
| jeblair | mordred: cause if the goal is implementation flexibility to deal with evil firewalls, i could see two different sites deciding they wanted to solve that two different ways. | 23:31 |
| jeblair | mordred: but hey, you're the only one with a use case for this so far. | 23:31 |
| jeblair | mordred: so if you and clarkb only want to push one of them that's fine. | 23:32 |
| *** weshay has quit IRC | 23:32 | |
| mordred | jeblair: that we know about - apparently jaypipes it making an army of infra cloners out there ... | 23:32 |
| jaypipes | lol | 23:32 |
| fungi | some behind a very great firewall indeed | 23:32 |
| fungi | (great in the sense of its size, though from what i've seen not so great in terms of comprehensivity) | 23:33 |
| jeblair | clarkb's change has a very small but non-zero chance of being applied independently of your requirement. i don't foresee infra's need for mordred's patch, atm, but i certainly did when i wrote it the first time so who knows. :) | 23:33 |
| *** bhuvan has quit IRC | 23:33 | |
| fungi | clarkb: the cluster view in bigdesk, while amusing, does suggest that es01 is now fairly shardy (though not terrifically). does this mean downing es6 is safe at this point? | 23:33 |
| fungi | and i agree es-head provides a better functional view of that | 23:34 |
| *** zul has joined #openstack-infra | 23:34 | |
| clarkb | fungi: its technically safe to down es6 at any point beacuse we can lose one node with our redundancy | 23:34 |
| fungi | s/safe/convenient/ | 23:34 |
| *** dims has quit IRC | 23:34 | |
| clarkb | fungi: but by waiting longer and allowing as much data to copy to es01 now we reduce the time period we are vulnerable to data loss due to losing a node | 23:35 |
| *** bhuvan has joined #openstack-infra | 23:35 | |
| *** oubiwann_ has quit IRC | 23:35 | |
| fungi | clarkb: how do i tell when it's copied all it wants to? when it has a shard from each logstash worker? | 23:35 |
| clarkb | fungi: no, when there are no remaining purple shards | 23:35 |
| clarkb | purple means copying | 23:35 |
| fungi | nice. like in python where whitespace is significant, in elasticsearch colors seem to have significance | 23:36 |
| * fungi will wait for the purple problem to subside | 23:36 | |
| jeblair | man i wish i was on whatever you guys are on | 23:36 |
| clarkb | jeblair: its good stuff, made with 100% real java | 23:37 |
| dstufft | java, not even once | 23:37 |
| fungi | jeblair: it's like computers, but with pictures | 23:37 |
| jeblair | speaking of colors, reload http://status.openstack.org/zuul/ | 23:37 |
| jeblair | that sort of brown line is waiting and total drawing on top of each other for the logstash queue | 23:38 |
| fungi | job queue! | 23:38 |
| *** david-lyle has quit IRC | 23:38 | |
| clarkb | jeblair: in that case I will clean up my existing change | 23:38 |
| fungi | with my java goggles on, i think that sort-of-brown is called "brick" | 23:38 |
| fungi | or maybe "rust" | 23:38 |
| * fungi might need to get out the pantone catalog | 23:39 | |
| jeblair | sdague, jog0: http://status.openstack.org/zuul/ shows the logstash queue now so you can see how backed up it is | 23:39 |
| *** dizquierdo has quit IRC | 23:39 | |
| *** bhuvan has quit IRC | 23:39 | |
| *** bhuvan has joined #openstack-infra | 23:40 | |
| fungi | clarkb: do we need to try https://review.openstack.org/76376 from a dev env when we're ready to bring up es02, or are you cool with it as-is? | 23:41 |
| *** sandywalsh has quit IRC | 23:41 | |
| clarkb | looking | 23:42 |
| fungi | also, i note only two "purple" shards in es-head... are we just waiting for those to clear, or does the cluster start new copies at a set rate so as to not swamp intercommunication channels between cluster members? | 23:43 |
| clarkb | it is rate limited | 23:43 |
| clarkb | that change lgtm I don't think you need to test it on es02 | 23:43 |
| fungi | okay, so when one of these finishes another shard may go from green to purple? | 23:44 |
| clarkb | correct | 23:44 |
| clarkb | so this can take a while :) | 23:44 |
| *** bhuvan has quit IRC | 23:44 | |
| fungi | does the number of workers or cluster members allow us to predict the number of shards we should expect es01 to accumulate? | 23:44 |
| clarkb | fungi: so we could just add all of the nodes to the cluster first if you want to get over that hurdle, then remove old ones as things settle | 23:45 |
| clarkb | fungi: roughly. we have an index per day, 10 shards per index and 2 weeks of data. that is 140 shards. divide that by te number of cluster members to get a rough estimate | 23:45 |
| clarkb | in this case 20 | 23:45 |
| fungi | it looks to me like each elasticsearch cluster member has at least one shard for each logstash worker, in come cases more than one | 23:46 |
| *** prad_ has quit IRC | 23:46 | |
| fungi | er, some cases | 23:46 |
| clarkb | fungi: the shards don't map to logstash workers | 23:46 |
| clarkb | each worker can index on any shard | 23:46 |
| fungi | oh, wait, they map to dates | 23:46 |
| clarkb | correct | 23:46 |
| clarkb | since we create a new index each day | 23:46 |
| fungi | "logstash-2014.02.15" wasn'y just a truncated worker name and date. it's per-day shards | 23:47 |
| fungi | more lightbulbs | 23:47 |
| *** masayukig has quit IRC | 23:47 | |
| *** dims has joined #openstack-infra | 23:47 | |
| pmathews | hmmm | 23:47 |
| *** bhuvan has joined #openstack-infra | 23:47 | |
| clarkb | fungi: jeblair pleia2 https://review.openstack.org/#/c/70007/1 was noop applied to zuul-dev and the files updated by the change were not modified on zuul-dev | 23:47 |
| pmathews | I don't think versionlock is working on the live controllers as expected | 23:47 |
| fungi | so, given that all the new servers are built, and replication is slowish, i'm in favor of adding more new cluster members and then scaling off the old ones later. should i go particularly slowly with that? | 23:48 |
| clarkb | fungi: jeblair pleia2 happy with my approving that change? | 23:48 |
| fungi | clarkb: you bet | 23:48 |
| clarkb | fungi: that sounds fine with me, I would start each one and let it join the cluster before doing the next | 23:48 |
| fungi | clarkb: okay. in that case i'll approve 76376 and give it a chance to get in place first | 23:49 |
| clarkb | keep in mind we can only delete one server at a time regardless of how many total servers we have | 23:49 |
| fungi | yep | 23:49 |
| fungi | i don't intend to delete any of the old ones until you're good with it | 23:50 |
| clarkb | and wait for the cluster to recover before deleting the next | 23:50 |
| fungi | the old ones, and the outer gods | 23:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Correct elasticsearch config dependency ordering https://review.openstack.org/76376 | 23:51 |
| *** mspreitz has joined #openstack-infra | 23:52 | |
| clarkb | fungi: you ready for me to approve 70007? making sure I understand your response above. Do you think we shuold have jeblair weigh in first? I am always paranoid about firewall rule changes | 23:52 |
| mspreitz | Has anybody succeeded in doing a `get fetch` from an Ubuntu 13.10 (saucy) machine? | 23:53 |
| fungi | clarkb: yes, if you're satisfied with your testing on 70007, i'm cool with it | 23:53 |
| clarkb | pmathews: I don't have enough context to understand thsoe comments. wrong window or do I need to catch up on mail | 23:53 |
| *** bhuvan_ has joined #openstack-infra | 23:53 | |
| anteaya | mspreitz: hi, can you share some context? | 23:53 |
| clarkb | fungi: approved | 23:53 |
| *** sandywalsh has joined #openstack-infra | 23:53 | |
| *** bhuvan has quit IRC | 23:54 | |
| mspreitz | anteaya: I am trying to cherry pick a patch, and the `git fetch` fails with a connection problem | 23:55 |
| anteaya | mspreitz: would you be able to paste the output of the error and post the url of the paste in here? | 23:55 |
| clarkb | mspreitz: where are you fetching from and are you behind a firewall/proxy? | 23:55 |
| openstackgerrit | A change was merged to openstack-infra/config: Update some deprecated Puppet variable references https://review.openstack.org/70007 | 23:55 |
| fungi | mspreitz: i strongly suspect at least several people in this channel run ubuntu saucy (13.10?) and use the git fetch subcommand very often, but if you have a specific error message you wish to share (better pasted into http://paste.openstack.org/ with some additional contaxt and the url provided here), we're pretty good at interpreting git errors | 23:56 |
| mspreitz | See http://paste.openstack.org/show/69602/ | 23:56 |
| fungi | s/contaxt/context/ | 23:56 |
| *** Sukhdev has quit IRC | 23:56 | |
| fungi | sometimes running git is like throwing bones | 23:57 |
| fungi | and that error does indeed look like a broken transparent https proxy | 23:57 |
| anteaya | mspreitz: do you have an account on review.openstack.org? | 23:57 |
| *** ociuhandu has joined #openstack-infra | 23:57 | |
| *** lcostantino has quit IRC | 23:58 | |
| mspreitz | yes | 23:58 |
| anteaya | great | 23:58 |
| anteaya | ssh key on it and you can upload to it? | 23:58 |
| mspreitz | yes | 23:58 |
| *** bhuvan has joined #openstack-infra | 23:58 | |
| anteaya | wonderful | 23:58 |
| mspreitz | I copied my command line from Gerrit | 23:58 |
| fungi | mspreitz: also, try http:// or git:// to see if it helps | 23:58 |
| *** zehicle_at_dell has joined #openstack-infra | 23:58 | |
| *** ociuhandu has quit IRC | 23:58 | |
| *** flaper87 is now known as flaper87|afk | 23:58 | |
| mspreitz | Wow! Same complaint when I try http | 23:59 |
| *** bhuvan_ has quit IRC | 23:59 | |
| cyeoh | mspreitz: don't know if this might help or not: http://stackoverflow.com/questions/13524242/error-gnutls-handshake-failed-git-repository (the rebuild git answer looks like a lot of work) | 23:59 |
| fungi | gnutls should definitely not be complaining about malformed tls packets on http | 23:59 |
| mspreitz | BTW, you all know that there are some middle-boxes that abort SSH connections if either Key Exchange Init message is long, right? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!