| *** shashankhegde has joined #openstack-infra | 00:14 | |
| *** shashankhegde has quit IRC | 00:20 | |
| *** baoli has joined #openstack-infra | 00:20 | |
| *** pcrews has joined #openstack-infra | 00:25 | |
| *** dims_ has quit IRC | 00:34 | |
| *** mahito has joined #openstack-infra | 00:35 | |
| *** yamada-h has joined #openstack-infra | 00:37 | |
| *** zhiwei has joined #openstack-infra | 00:39 | |
| *** abramley has quit IRC | 00:39 | |
| tchaypo | morning | 00:41 |
|---|---|---|
| *** yamada-h has quit IRC | 00:41 | |
| *** salv-orlando has joined #openstack-infra | 00:43 | |
| *** abramley has joined #openstack-infra | 00:45 | |
| * lifeless looks around for pbr cores | 00:49 | |
| *** salv-orlando has quit IRC | 00:50 | |
| *** hichihara has joined #openstack-infra | 00:51 | |
| lifeless | who are working today | 00:52 |
| *** sputnik13 has quit IRC | 00:53 | |
| lifeless | https://review.openstack.org/#/c/177457/ mikal jhesketh <- bottom of a green stack of all innocuous patches making pip firedrills less likely | 00:54 |
| *** jaypipes has joined #openstack-infra | 00:56 | |
| jhesketh | lifeless: thanks, I'll take a look | 00:56 |
| lifeless | well, mostly green top two aren't, but any assists on the lower end would be grand | 00:57 |
| *** stevemar has quit IRC | 00:57 | |
| *** ddieterl_ has quit IRC | 01:01 | |
| mordred | lifeless: I'm assuming that PBR_PIP_VERSION and PIPVERSION are intended to be not the same | 01:04 |
| *** ddieterly has joined #openstack-infra | 01:04 | |
| mordred | lifeless: (It reads like it does - but just making sure) | 01:05 |
| *** ddieterl_ has joined #openstack-infra | 01:05 | |
| lifeless | mordred: thats correct | 01:05 |
| lifeless | mordred: one says the one to test with and is documented as a parameter in the next patch | 01:05 |
| lifeless | mordred: the other is used in the mkvenv function, and is altered within the scrpt | 01:05 |
| lifeless | hmm pip verbose really is verbose. I shall fix that next week | 01:07 |
| jhesketh | lifeless: I have some nits on https://review.openstack.org/#/c/177500/, but I don't want to block due to them | 01:08 |
| jhesketh | do you want to fix them, or shall I approve? | 01:08 |
| lifeless | let me have a look | 01:08 |
| *** ddieterly has quit IRC | 01:08 | |
| mordred | lifeless: on 177503 - you've verified that that doesn't stop-on-first-failure? | 01:09 |
| lifeless | mordred: I plan to go through the log with a fine tooth comb | 01:09 |
| mordred | kk | 01:09 |
| lifeless | mordred: (not that it matters, we weren't using the wheels *anyway* | 01:09 |
| mordred | :) | 01:09 |
| mordred | lifeless: so ... the other thing I just rhough of of why we did that | 01:09 |
| *** achanda has joined #openstack-infra | 01:09 | |
| mordred | a single pass will get you one wheel for each requirement | 01:10 |
| mordred | whereas, because of depsolver things - the while loop _may_ get you more than one version of something in the wheel house | 01:10 |
| mordred | which might be needed to satisfy different sequence combinations | 01:10 |
| mordred | I'm not saying this holds up | 01:10 |
| mordred | but as you sift through the logs | 01:11 |
| mordred | something for your brainhole | 01:11 |
| lifeless | jhesketh: one I agree with, easy to fix | 01:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 01:13 |
| mordred | worst latency way to test that ... | 01:13 |
| mordred | but it works locally | 01:13 |
| mordred | so - you know - adding print statement and pushing to gate to see why gate is different ... | 01:13 |
| lifeless | mordred: yeah, its a nice theory | 01:14 |
| lifeless | mordred: but because we do them all in isolation what it actually means is that you'll get the wrong wheel. | 01:14 |
| lifeless | mordred: foo <=3.4 in global requirements -> wheel of 3.4. | 01:14 |
| lifeless | mordred: bar deps foo <=3.3 somewhere, 3.3 is what we end up using. | 01:14 |
| lifeless | mordred: eitherway, the auto wheel building in 7.0.0 will address that | 01:15 |
| *** neoice_ is now known as neoice | 01:15 | |
| lifeless | mordred: and we can rip out this wheel stuff entirely | 01:15 |
| *** sandywalsh has joined #openstack-infra | 01:15 | |
| *** sigmavirus24_awa has quit IRC | 01:16 | |
| *** Trozz_ has quit IRC | 01:16 | |
| *** Trozz has joined #openstack-infra | 01:17 | |
| *** odyssey4me has quit IRC | 01:17 | |
| *** fungi has quit IRC | 01:18 | |
| *** ttx has quit IRC | 01:18 | |
| *** btully has joined #openstack-infra | 01:18 | |
| *** sandywalsh_ has quit IRC | 01:19 | |
| *** ctracey has quit IRC | 01:20 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Stop testing setup.py install https://review.openstack.org/177505 | 01:20 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Test pip install -e of projects. https://review.openstack.org/177504 | 01:20 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Build all the wheels in one pass rather than many. https://review.openstack.org/177503 | 01:20 |
| *** odyssey4me has joined #openstack-infra | 01:20 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Improve integration.sh. https://review.openstack.org/177500 | 01:20 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Use /opt/git directly https://review.openstack.org/177629 | 01:20 |
| lifeless | mordred: how big is /opt/git? | 01:20 |
| *** fungi has joined #openstack-infra | 01:21 | |
| *** ttx has joined #openstack-infra | 01:22 | |
| *** sigmavirus24_awa has joined #openstack-infra | 01:22 | |
| lifeless | ttx: hi | 01:22 |
| *** ctracey has joined #openstack-infra | 01:22 | |
| lifeless | ttx: [is that really you?] | 01:22 |
| *** yamamoto_ has joined #openstack-infra | 01:30 | |
| *** jamesmcarthur has joined #openstack-infra | 01:31 | |
| *** yamamoto_ has quit IRC | 01:32 | |
| *** sdake has joined #openstack-infra | 01:32 | |
| *** yamada-h has joined #openstack-infra | 01:33 | |
| *** yamamoto has quit IRC | 01:33 | |
| *** armax has joined #openstack-infra | 01:33 | |
| *** jyuso1 has quit IRC | 01:36 | |
| *** kaisers has quit IRC | 01:36 | |
| *** kaisers1 has quit IRC | 01:36 | |
| *** jyuso1 has joined #openstack-infra | 01:36 | |
| *** btully has quit IRC | 01:37 | |
| *** kaisers has joined #openstack-infra | 01:39 | |
| *** kaisers1 has joined #openstack-infra | 01:39 | |
| *** btully has joined #openstack-infra | 01:39 | |
| *** sdake_ has joined #openstack-infra | 01:48 | |
| *** reazem_ is now known as reazem | 01:50 | |
| *** sdake has quit IRC | 01:51 | |
| *** sputnik13 has joined #openstack-infra | 01:52 | |
| *** stevemar has joined #openstack-infra | 01:53 | |
| *** baoli has quit IRC | 01:57 | |
| *** erikmwilson_ is now known as erikmwilson | 01:59 | |
| *** asettle is now known as asettle-afk | 02:02 | |
| *** achanda has quit IRC | 02:02 | |
| *** mjordan has quit IRC | 02:02 | |
| *** jlvillal has quit IRC | 02:08 | |
| *** amitgandhinz has joined #openstack-infra | 02:09 | |
| *** jlvillal has joined #openstack-infra | 02:09 | |
| *** amitgandhinz has quit IRC | 02:17 | |
| *** amitgandhinz has joined #openstack-infra | 02:18 | |
| *** armax has quit IRC | 02:24 | |
| *** amitgandhinz has quit IRC | 02:25 | |
| *** yamada-h has quit IRC | 02:25 | |
| *** amitgandhinz has joined #openstack-infra | 02:26 | |
| mordred | lifeless: I mean ... it's not small? | 02:26 |
| mordred | lifeless: but I don't know the size off the top of my head | 02:26 |
| *** amitgandhinz has quit IRC | 02:28 | |
| *** Longgeek has quit IRC | 02:28 | |
| *** yamada-h has joined #openstack-infra | 02:31 | |
| *** Longgeek has joined #openstack-infra | 02:33 | |
| *** yamada-h_ has joined #openstack-infra | 02:33 | |
| *** baoli has joined #openstack-infra | 02:35 | |
| *** sdake_ has quit IRC | 02:35 | |
| *** sputnik13 has quit IRC | 02:36 | |
| *** yamada-h has quit IRC | 02:37 | |
| morganfainberg | sdague: I'm happy to co-lead the Service Catalog Standardization cross-project thing if no one else wants to. but I'd rather have someone else lead and make sure keystone folks are there. | 02:40 |
| morganfainberg | sdague: but i for sure wouldn't want to be the sole leader for it. | 02:40 |
| tchaypo | lifeless: looking at 177505, I’m not sure how this is testing pip more thoroughly - unless that just means that we check for migrate.cfg in $pipenv rather than $installenv | 02:43 |
| *** salv-orlando has joined #openstack-infra | 02:46 | |
| *** shashankhegde has joined #openstack-infra | 02:51 | |
| *** jamesmcarthur has quit IRC | 02:51 | |
| *** sdake has joined #openstack-infra | 02:52 | |
| *** wuhg has joined #openstack-infra | 02:52 | |
| *** salv-orlando has quit IRC | 02:56 | |
| *** asettle-afk is now known as asettle | 02:59 | |
| *** salv-orlando has joined #openstack-infra | 03:02 | |
| *** salv-orlando has quit IRC | 03:04 | |
| *** woodster_ has joined #openstack-infra | 03:05 | |
| *** baoli has quit IRC | 03:08 | |
| *** otter768 has quit IRC | 03:20 | |
| *** armax has joined #openstack-infra | 03:21 | |
| *** btully has quit IRC | 03:22 | |
| *** armax has quit IRC | 03:23 | |
| *** sdake_ has joined #openstack-infra | 03:28 | |
| *** ddieterl_ has quit IRC | 03:29 | |
| *** sdake has quit IRC | 03:32 | |
| *** yamada-h has joined #openstack-infra | 03:34 | |
| *** yamada-h has quit IRC | 03:39 | |
| *** mjordan has joined #openstack-infra | 03:40 | |
| *** ildikov has quit IRC | 03:40 | |
| *** tjones1 has joined #openstack-infra | 03:45 | |
| *** yamada-h has joined #openstack-infra | 03:47 | |
| *** yamada-h has quit IRC | 03:47 | |
| *** yamada-h has joined #openstack-infra | 03:50 | |
| *** ildikov has joined #openstack-infra | 03:54 | |
| *** tjones1 has quit IRC | 03:57 | |
| *** jtriley has joined #openstack-infra | 04:01 | |
| *** sarob has joined #openstack-infra | 04:01 | |
| *** jtriley has quit IRC | 04:05 | |
| *** yamada-h has joined #openstack-infra | 04:10 | |
| *** yamada-h has quit IRC | 04:10 | |
| *** mjordan has left #openstack-infra | 04:16 | |
| *** Longgeek has quit IRC | 04:17 | |
| *** Longgeek has joined #openstack-infra | 04:18 | |
| *** davideagnello has joined #openstack-infra | 04:24 | |
| *** davideagnello has quit IRC | 04:28 | |
| *** sarob has quit IRC | 04:28 | |
| *** sarob has joined #openstack-infra | 04:29 | |
| *** ddieterly has joined #openstack-infra | 04:30 | |
| *** yamamoto has joined #openstack-infra | 04:31 | |
| *** ildikov has quit IRC | 04:32 | |
| *** sarob has quit IRC | 04:34 | |
| *** ddieterly has quit IRC | 04:34 | |
| *** yamada-h has joined #openstack-infra | 04:40 | |
| *** yamada-h has quit IRC | 04:46 | |
| *** coolsvap|afk is now known as coolsvap | 04:47 | |
| *** ildikov has joined #openstack-infra | 04:49 | |
| *** ildikov has quit IRC | 04:53 | |
| *** AJaeger has joined #openstack-infra | 04:54 | |
| *** yfried__ has quit IRC | 04:56 | |
| AJaeger | fungi, clarkb, jeblair, pleia2, jhesketh, SergeyLukjanov: Gerrit is stuck, no no events are processed at all. | 04:56 |
| AJaeger | ttx: if nobody can help at this time, could you send an alert, please? | 04:57 |
| *** yamada-h has joined #openstack-infra | 04:59 | |
| *** yamada-h has quit IRC | 04:59 | |
| jhesketh | AJaeger: I'm here, but I have no powers :-( | 05:02 |
| jhesketh | trying to see what merged that might have stuck things | 05:02 |
| AJaeger | jhesketh: nothing seem to have merged in the last 24 hours but this is since 3 hours or so | 05:03 |
| AJaeger | at least my latest emails are from that time | 05:04 |
| jhesketh | yeah, nothing is standing out to me | 05:04 |
| jhesketh | looking at graphite indicates that zuul has lost connection with gerrit | 05:05 |
| *** Trozz has quit IRC | 05:11 | |
| jhesketh | AJaeger: looks like gerrit isn't streaming any events (to my zuul/user either) | 05:11 |
| *** Trozz has joined #openstack-infra | 05:12 | |
| AJaeger | so, we need somebody to restart gerrit? | 05:15 |
| AJaeger | SergeyLukjanov: are you around to do that? | 05:15 |
| jhesketh | well possibly | 05:16 |
| jhesketh | I cna't find any other hints in cacti or graphite though | 05:18 |
| *** jamespage_ has joined #openstack-infra | 05:19 | |
| AJaeger | So, nobody around - in that case, let me send an email to openstack-dev to inform people | 05:20 |
| *** otter768 has joined #openstack-infra | 05:21 | |
| *** deepakcs has joined #openstack-infra | 05:23 | |
| * AJaeger sent mail | 05:26 | |
| *** otter768 has quit IRC | 05:26 | |
| *** jamespage_ has quit IRC | 05:29 | |
| *** subscope_ has joined #openstack-infra | 05:30 | |
| *** hdd has joined #openstack-infra | 05:30 | |
| *** subscope_ has quit IRC | 05:31 | |
| *** BharatK has joined #openstack-infra | 05:31 | |
| jhesketh | AJaeger: thanks :-) | 05:33 |
| *** btully has joined #openstack-infra | 05:33 | |
| *** wolsen_ is now known as wolsen | 05:33 | |
| AJaeger | I have to go now, will be back later | 05:34 |
| AJaeger | thanks, jhesketh for your investigation! | 05:35 |
| *** AJaeger has quit IRC | 05:35 | |
| *** abregman has joined #openstack-infra | 05:35 | |
| jhesketh | no trouble, sorry I can't help more! | 05:36 |
| *** achanda has joined #openstack-infra | 05:39 | |
| ianw | :( | 05:42 |
| *** ajo has joined #openstack-infra | 05:43 | |
| *** masayukig has quit IRC | 05:44 | |
| *** sweston has quit IRC | 05:44 | |
| *** persia has quit IRC | 05:44 | |
| *** sweston has joined #openstack-infra | 05:45 | |
| *** masayukig has joined #openstack-infra | 05:46 | |
| *** persia has joined #openstack-infra | 05:46 | |
| *** persia has quit IRC | 05:46 | |
| *** persia has joined #openstack-infra | 05:46 | |
| *** salv-orlando has joined #openstack-infra | 05:46 | |
| *** cpallares has joined #openstack-infra | 05:47 | |
| *** [HeOS] has quit IRC | 05:47 | |
| *** tnovacik has joined #openstack-infra | 05:51 | |
| *** salv-orlando has quit IRC | 05:54 | |
| *** sdake_ has quit IRC | 05:56 | |
| *** sdake has joined #openstack-infra | 05:56 | |
| *** dannywilson has joined #openstack-infra | 05:57 | |
| *** yamada-h has joined #openstack-infra | 05:59 | |
| *** dannywilson has quit IRC | 06:02 | |
| *** dannywilson has joined #openstack-infra | 06:02 | |
| *** ildikov has joined #openstack-infra | 06:03 | |
| *** yamada-h has quit IRC | 06:04 | |
| *** morgabra has quit IRC | 06:04 | |
| *** morgabra has joined #openstack-infra | 06:04 | |
| *** mrunge has joined #openstack-infra | 06:05 | |
| *** mrunge has quit IRC | 06:06 | |
| *** kaisers1 has quit IRC | 06:06 | |
| *** kaisers has quit IRC | 06:06 | |
| *** Longgeek has quit IRC | 06:07 | |
| *** dtantsur|afk is now known as dtantsur | 06:07 | |
| *** Longgeek has joined #openstack-infra | 06:09 | |
| *** zul has joined #openstack-infra | 06:11 | |
| *** _nadya_ has joined #openstack-infra | 06:14 | |
| *** kaisers has joined #openstack-infra | 06:16 | |
| *** mrunge has joined #openstack-infra | 06:18 | |
| *** mrunge has joined #openstack-infra | 06:18 | |
| *** kiseok7_ has joined #openstack-infra | 06:19 | |
| *** _nadya_ has quit IRC | 06:19 | |
| *** dalgaaf has joined #openstack-infra | 06:20 | |
| *** masayukig has quit IRC | 06:21 | |
| *** afazekas has joined #openstack-infra | 06:21 | |
| *** vlaza has joined #openstack-infra | 06:22 | |
| *** liam__ has joined #openstack-infra | 06:22 | |
| *** liam__ has quit IRC | 06:22 | |
| *** masayukig has joined #openstack-infra | 06:23 | |
| *** dalgaaf has quit IRC | 06:24 | |
| *** e0ne has joined #openstack-infra | 06:24 | |
| *** dalgaaf has joined #openstack-infra | 06:24 | |
| *** dannywilson has quit IRC | 06:25 | |
| *** Qiming_ has joined #openstack-infra | 06:27 | |
| *** stevemar has quit IRC | 06:27 | |
| *** e0ne has quit IRC | 06:28 | |
| *** Sukhdev has joined #openstack-infra | 06:29 | |
| *** scheuran has joined #openstack-infra | 06:30 | |
| *** scheuran has quit IRC | 06:30 | |
| *** scheuran has joined #openstack-infra | 06:30 | |
| *** ddieterly has joined #openstack-infra | 06:31 | |
| *** Longgeek_ has joined #openstack-infra | 06:34 | |
| *** Sukhdev has quit IRC | 06:34 | |
| *** AJaeger has joined #openstack-infra | 06:35 | |
| *** ddieterly has quit IRC | 06:35 | |
| *** yfried__ has joined #openstack-infra | 06:36 | |
| *** Longgeek has quit IRC | 06:37 | |
| *** kaisers has quit IRC | 06:39 | |
| *** woodster_ has quit IRC | 06:40 | |
| *** zul has quit IRC | 06:40 | |
| *** salv-orlando has joined #openstack-infra | 06:42 | |
| *** Yingxin has joined #openstack-infra | 06:43 | |
| *** ssam2 has joined #openstack-infra | 06:44 | |
| *** salv-orlando has quit IRC | 06:44 | |
| *** Qiming has joined #openstack-infra | 06:46 | |
| *** ttx has quit IRC | 06:47 | |
| *** ttx has joined #openstack-infra | 06:47 | |
| *** Qiming_ has quit IRC | 06:48 | |
| *** AJaeger has quit IRC | 06:50 | |
| *** AJaeger has joined #openstack-infra | 06:51 | |
| *** camunoz is now known as camunoz_gone | 06:51 | |
| *** AJaeger has quit IRC | 06:53 | |
| * ttx catches up | 06:54 | |
| *** AJaeger has joined #openstack-infra | 06:54 | |
| jhesketh | ttx: basically we just need soembody with root to investigate/punch review.o.o | 06:55 |
| ttx | Ack, my powers are limited to giving status updates through the statusbot though | 06:55 |
| ttx | so, Zuul not processing events ? | 06:55 |
| jhesketh | ttx: okay, so the status is that zuul isn't receiving events from gerrit therefore nothing is queued for testing or merging | 06:55 |
| jhesketh | correct | 06:55 |
| *** Bsony has joined #openstack-infra | 06:56 | |
| AJaeger | ttx, so could you give status updates again, please? | 06:56 |
| ttx | yes | 06:56 |
| * AJaeger sent already an email to openstack-dev but let's spread the word | 06:56 | |
| AJaeger | good morning, ttx! | 06:56 |
| ttx | #status alert Zuul is currently not processing any event, so no check / gate tests are being run. The issue is being investigated. | 06:58 |
| ttx | hah, no statusbot | 06:58 |
| *** sdake has quit IRC | 06:58 | |
| ttx | We also need someone to punch the statusbot | 06:59 |
| ttx | One of those days, I guess | 06:59 |
| AJaeger | ;( | 06:59 |
| ttx | lifeless: yes it is me | 07:00 |
| AJaeger | So, options are: SergeyLukjanov is around and can help - unlikely, hasn't reacted yet - or waiting for fungi to wake up | 07:01 |
| *** ildikov has quit IRC | 07:01 | |
| rcarrillocruz | we need moar infra cores/root, just sayin'... | 07:02 |
| *** ivar-lazzaro has quit IRC | 07:02 | |
| *** yamada-h has joined #openstack-infra | 07:03 | |
| rcarrillocruz | in EMEA that is | 07:04 |
| yolanda | rcarrillocruz, +1 on that. Set a +1 in my topic for the sprint, please :) | 07:05 |
| * jhesketh wishes he could help more | 07:05 | |
| *** jcoufal has joined #openstack-infra | 07:06 | |
| yolanda | we know that system in our team from memory, we do the same as infra every day | 07:06 |
| rcarrillocruz | jhesketh: it must be quite late for you right? | 07:06 |
| yolanda | and will be glad to help | 07:06 |
| rcarrillocruz | which tz are you in? | 07:06 |
| *** jcoufal is now known as 7JTAAIHSL | 07:06 | |
| *** jcoufal has joined #openstack-infra | 07:06 | |
| *** jcoufal has quit IRC | 07:06 | |
| *** jcoufal has joined #openstack-infra | 07:06 | |
| *** jcoufal has quit IRC | 07:06 | |
| rcarrillocruz | i know AU has a handful of different TZs | 07:06 |
| jhesketh | rcarrillocruz: nope, just past 5pm | 07:07 |
| jhesketh | (it would be 6pm if we were still in daylight savings) | 07:07 |
| jhesketh | UTC+10 | 07:07 |
| *** kaisers has joined #openstack-infra | 07:08 | |
| jhesketh | western australia (of which I'm not sure we have any devs from) is UTC+7, so earlier for them | 07:08 |
| *** mpaolino has joined #openstack-infra | 07:08 | |
| StevenK | I thought WA was always UTC+8, because daylight savings isn't a thing there | 07:09 |
| *** ildikov has joined #openstack-infra | 07:09 | |
| jhesketh | StevenK: err, yes, sorry, my mistake | 07:09 |
| *** shardy_ has joined #openstack-infra | 07:09 | |
| StevenK | They've experimented with it three times in the last n years, but it keeps getting shot down | 07:09 |
| *** shardy has quit IRC | 07:11 | |
| *** hyakuhei has joined #openstack-infra | 07:11 | |
| rcarrillocruz | openstack is so mainstream now that i'd assume we have devs in every possible TZ these days | 07:11 |
| *** gema has joined #openstack-infra | 07:12 | |
| *** gema has quit IRC | 07:12 | |
| *** gema has joined #openstack-infra | 07:13 | |
| *** shardy_ has quit IRC | 07:14 | |
| *** apmelton_away has quit IRC | 07:15 | |
| *** shardy has joined #openstack-infra | 07:15 | |
| *** apmelton has joined #openstack-infra | 07:16 | |
| *** Nakato has quit IRC | 07:16 | |
| *** Nakato has joined #openstack-infra | 07:16 | |
| *** freyes has quit IRC | 07:16 | |
| *** freyes has joined #openstack-infra | 07:18 | |
| *** MaxV has joined #openstack-infra | 07:21 | |
| lifeless | ttx: it was mor itming t t | 07:21 |
| *** viktors has joined #openstack-infra | 07:22 | |
| viktors | hi folks! Can somebody look at patch https://review.openstack.org/#/c/156530/ (Add Ironic python3.4 experimental job) ? | 07:22 |
| *** otter768 has joined #openstack-infra | 07:22 | |
| *** saggi has joined #openstack-infra | 07:24 | |
| yolanda | jhesketh, thx for your quick review, i amended the text according to your comments | 07:24 |
| *** markvoelker has joined #openstack-infra | 07:26 | |
| *** otter768 has quit IRC | 07:27 | |
| jhesketh | yolanda: cool, I'll take a look later | 07:28 |
| yolanda | i think that Clark's suggestions to remove gearman, and just execute a simple test script, simplified that spec. I'm worried about future needs , specially if we need to reuse some jobs already declared on jenkins, but i prefer keeping that simple, and iterate later if we have more needs | 07:29 |
| SergeyLukjanov | AJaeger, hey | 07:29 |
| SergeyLukjanov | AJaeger, /me just woke up :) | 07:29 |
| AJaeger | morning, SergeyLukjanov ! Didn't want to wake you up ;) | 07:30 |
| *** markvoelker has quit IRC | 07:30 | |
| AJaeger | SergeyLukjanov, Zuul is not processing any events, seems the gerrit queue is stuck - could you revive it again, please? | 07:30 |
| SergeyLukjanov | AJaeger, looking | 07:31 |
| AJaeger | And bring us back the statusbot here, plesae? | 07:31 |
| AJaeger | thanks, SergeyLukjanov | 07:31 |
| *** ibiris_away is now known as ibiris | 07:31 | |
| SergeyLukjanov | AJaeger, it looks like event stream is dead, yeah | 07:31 |
| *** ddieterly has joined #openstack-infra | 07:32 | |
| SergeyLukjanov | AJaeger, jhesketh , I think I'll just restart gerrit, zuul and statusbot | 07:34 |
| jhesketh | SergeyLukjanov: if the logs aren't helpful that seems like a reasonable step | 07:36 |
| *** yamada-h has quit IRC | 07:36 | |
| lifeless | wow ok so | 07:36 |
| lifeless | 20G in one chromium process | 07:36 |
| *** ddieterly has quit IRC | 07:36 | |
| *** salv-orlando has joined #openstack-infra | 07:36 | |
| saggi | AJaeger: I'm part of the dragonflow team. We want to change the submission policy to fast forward only. Is it just a matter of setting `mergeContent = false` in `project-config/gerrit/acls/stackforge/dragonflow.config`? | 07:37 |
| lifeless | ttx: so I wanted to ask you | 07:37 |
| lifeless | ttx: do we have a timeslot for the dependency version management session | 07:37 |
| AJaeger | saggi, sorry, can't help with that. | 07:37 |
| saggi | AJaeger: Do you happen to know who can? | 07:38 |
| lifeless | ttx: and if so, is it against ops / 3rd-party CI ops / infra / packager sessions | 07:38 |
| AJaeger | saggi, I hope the other infra cores can help - please ask again during US time | 07:38 |
| *** BobBall_ is now known as BobBall | 07:39 | |
| *** gillliard is now known as gilliard | 07:39 | |
| *** hyakuhei has quit IRC | 07:39 | |
| saggi | AJaeger: They wake up so late :). We also have an issue with the GitHub mirror description not syncing. Is that someone else as well? | 07:39 |
| AJaeger | saggi, AFAIK github description is only synced at initial creation, so if you change it later, it won't be updated. | 07:40 |
| *** jlanoux has joined #openstack-infra | 07:40 | |
| *** samuelBartel has joined #openstack-infra | 07:41 | |
| AJaeger | saggi, but double check with the US based part of the team for this, please | 07:41 |
| SergeyLukjanov | saggi, I think mergeContent is a bit different config, if you'd like to produce strictly linear history the submit type "Fast Forward Only" needed | 07:41 |
| *** yamada-h has joined #openstack-infra | 07:42 | |
| SergeyLukjanov | instead of our defultt "Merge If Necessary" | 07:42 |
| *** sdake has joined #openstack-infra | 07:42 | |
| *** sdake has quit IRC | 07:42 | |
| *** sdake has joined #openstack-infra | 07:42 | |
| ttx | lifeless: we were planning to abuse the Friday Infra/QA/RelMgt sprint | 07:43 |
| *** mpavone has joined #openstack-infra | 07:45 | |
| samuelBartel | hi all! | 07:45 |
| samuelBartel | i have a question about the process http://docs.openstack.org/infra/manual/creators.html | 07:45 |
| SergeyLukjanov | tons of errors: | 07:45 |
| SergeyLukjanov | [2015-04-27 07:42:48,506] WARN com.google.gerrit.sshd.GerritServerSession : Exception caught | 07:45 |
| SergeyLukjanov | java.io.IOException: Connection reset by peer | 07:45 |
| SergeyLukjanov | jhesketh, ^^ | 07:45 |
| *** Bsony has quit IRC | 07:46 | |
| *** yamada-h has quit IRC | 07:46 | |
| samuelBartel | what is the correct workflow to be add to the gerrit groups after creation of the a new project? | 07:47 |
| samuelBartel | bug in launnchpad? request by email? | 07:47 |
| SergeyLukjanov | jhesketh, and some errors with hooks exec http://paste.openstack.org/show/207889/ | 07:47 |
| AJaeger | samuelBartel, Asking here or email to openstack-infra. Give the gerrit group and your email address or gerrit login. Once you've been added, you can use review.openstack.org to add more | 07:48 |
| samuelBartel | AJaeger, great thank you | 07:48 |
| *** hyakuhei has joined #openstack-infra | 07:48 | |
| samuelBartel | AJaeger, can I give you the informations (group name and email address)? | 07:49 |
| *** Hal has joined #openstack-infra | 07:49 | |
| AJaeger | samuelBartel, just give it here and one of the admins will read the backlog and do it later. | 07:49 |
| *** Hal is now known as Guest11290 | 07:50 | |
| *** skolekonov has joined #openstack-infra | 07:50 | |
| jhesketh | SergeyLukjanov: hmm, is that after a restart? | 07:50 |
| saggi | SergeyLukjanov: Can't find any project that uses "Fast Forward Only". Is this what I'm supposed to do? http://paste.fedoraproject.org/215712/12104414/ | 07:51 |
| SergeyLukjanov | jhesketh, nope, I'm still reading logs before restart | 07:51 |
| jhesketh | okay | 07:51 |
| SergeyLukjanov | jhesketh, will do it for another 5 mins and do restart | 07:51 |
| jhesketh | SergeyLukjanov: can you test the credentials that gerrit has to launchpad ? | 07:51 |
| SergeyLukjanov | jhesketh, I don't see any really bad stuff in the logs | 07:52 |
| SergeyLukjanov | jhesketh, there is only one such error in logs | 07:52 |
| AJaeger | saggi, since no other project is doing this, I wonder whether you really need it - if you change it, make a verbose commit message explaining the reasoning. | 07:52 |
| *** arxcruz has joined #openstack-infra | 07:52 | |
| *** Longgeek_ has quit IRC | 07:54 | |
| SergeyLukjanov | jhesketh, there is a branch of another exceptions in logs - http://paste.openstack.org/show/207892/ | 07:56 |
| SergeyLukjanov | and http://paste.openstack.org/show/207893/ | 07:57 |
| *** hashar has joined #openstack-infra | 07:57 | |
| *** jistr has joined #openstack-infra | 07:57 | |
| *** freyes has quit IRC | 07:59 | |
| *** rm_work has quit IRC | 07:59 | |
| *** rm_work|away has joined #openstack-infra | 07:59 | |
| *** rm_work|away is now known as rm_work | 07:59 | |
| *** rm_work has joined #openstack-infra | 07:59 | |
| *** persia has quit IRC | 08:00 | |
| jhesketh | SergeyLukjanov: hard to tell much off those pastes :-( | 08:00 |
| *** andrey-mp has joined #openstack-infra | 08:00 | |
| *** persia has joined #openstack-infra | 08:00 | |
| *** persia has quit IRC | 08:00 | |
| *** persia has joined #openstack-infra | 08:00 | |
| *** zul has joined #openstack-infra | 08:00 | |
| SergeyLukjanov | jhesketh, /me will send notice: Restarting gerrit because it stopped sending events | 08:01 |
| *** freyes has joined #openstack-infra | 08:01 | |
| SergeyLukjanov | (ETA 15 mins) | 08:01 |
| jhesketh | SergeyLukjanov: cool | 08:02 |
| AJaeger | SergeyLukjanov, did you restart the statusbot so that you can send notices? | 08:02 |
| *** btully has quit IRC | 08:02 | |
| SergeyLukjanov | AJaeger, yup | 08:02 |
| AJaeger | Great! | 08:02 |
| SergeyLukjanov | AJaeger, fighting slow internet connection :) | 08:02 |
| *** Longgeek has joined #openstack-infra | 08:03 | |
| *** shashankhegde has quit IRC | 08:03 | |
| *** hashar is now known as hasharMeeting | 08:03 | |
| *** fhubik has joined #openstack-infra | 08:04 | |
| *** Guest11290 has quit IRC | 08:05 | |
| *** flashgordon has quit IRC | 08:06 | |
| *** yamada-h has joined #openstack-infra | 08:06 | |
| *** openstackstatus has joined #openstack-infra | 08:06 | |
| *** ChanServ sets mode: +v openstackstatus | 08:06 | |
| *** zul has quit IRC | 08:08 | |
| SergeyLukjanov | #status notice Restarting gerrit because it stopped sending events (ETA 15 mins) | 08:09 |
| openstackstatus | SergeyLukjanov: sending notice | 08:09 |
| SergeyLukjanov | jhesketh, AJaeger ^^ | 08:09 |
| jhesketh | coolio | 08:09 |
| -openstackstatus- NOTICE: Restarting gerrit because it stopped sending events (ETA 15 mins) | 08:09 | |
| *** persia has quit IRC | 08:10 | |
| *** yamada-h has quit IRC | 08:10 | |
| *** persia has joined #openstack-infra | 08:11 | |
| openstackstatus | SergeyLukjanov: finished sending notice | 08:11 |
| *** sergsh has joined #openstack-infra | 08:11 | |
| *** _nadya_ has joined #openstack-infra | 08:12 | |
| *** openstackgerrit has quit IRC | 08:13 | |
| SergeyLukjanov | goodbye gerrit | 08:13 |
| *** Ala has joined #openstack-infra | 08:14 | |
| mkoderer | ok, Gerrit is back again :) | 08:16 |
| *** dtantsur is now known as dtantsur|brb | 08:16 | |
| *** achanda has quit IRC | 08:16 | |
| *** openstackgerrit has joined #openstack-infra | 08:16 | |
| jhesketh | SergeyLukjanov: cool, events appear to be streaming again | 08:17 |
| SergeyLukjanov | jhesketh, yeah, I already see firs vents | 08:17 |
| SergeyLukjanov | first events* | 08:17 |
| jhesketh | SergeyLukjanov: all 3rd parties will probably need to restart their zuul's | 08:17 |
| SergeyLukjanov | jhesketh, I will send notice to -dev | 08:18 |
| *** freyes has quit IRC | 08:18 | |
| SergeyLukjanov | jhesketh, yeah, we have first jobs running in zuul - http://zuul.openstack.org | 08:18 |
| jhesketh | SergeyLukjanov: and possibly the 3rd party ops announce list? | 08:18 |
| SergeyLukjanov | jhesketh, yeah | 08:18 |
| AJaeger | SergeyLukjanov, do we need to recheck all jobs? | 08:19 |
| *** sarob has joined #openstack-infra | 08:19 | |
| *** derekh has joined #openstack-infra | 08:19 | |
| mkoderer | AJaeger: seems so | 08:19 |
| AJaeger | SergeyLukjanov, shall I send out an email? | 08:20 |
| jhesketh | SergeyLukjanov: I have a script that gives you a set of rechecks to run | 08:21 |
| jhesketh | http://paste.openstack.org/show/207915/ | 08:21 |
| *** vponomaryov has quit IRC | 08:21 | |
| *** freyes has joined #openstack-infra | 08:21 | |
| *** liam__ has joined #openstack-infra | 08:22 | |
| *** liam__ has quit IRC | 08:23 | |
| *** vponomaryov has joined #openstack-infra | 08:23 | |
| *** vponomaryov has left #openstack-infra | 08:23 | |
| *** vponomaryov has joined #openstack-infra | 08:23 | |
| SergeyLukjanov | AJaeger, oops, I already send an email, if you'd like to add details | 08:23 |
| AJaeger | jhesketh, that is only nova, isn't it? | 08:24 |
| AJaeger | SergeyLukjanov, let me read you email and decide then;) | 08:24 |
| jhesketh | AJaeger: should be all projects? | 08:24 |
| *** sarob has quit IRC | 08:24 | |
| SergeyLukjanov | jhesketh, I see only nova in a list ;) | 08:24 |
| SergeyLukjanov | AJaeger, it's a sleepy morning notice of the hell :) | 08:24 |
| *** [HeOS] has joined #openstack-infra | 08:24 | |
| *** dguitarbite has joined #openstack-infra | 08:25 | |
| SergeyLukjanov | jhesketh, it looks like project name is hardcoded, I checked a few of the CR ids and it's not nova | 08:25 |
| jhesketh | please hold | 08:26 |
| jhesketh | ah, the recheck line is nova only, but I think the changes should be everything | 08:27 |
| *** markvoelker has joined #openstack-infra | 08:27 | |
| AJaeger | So, SergeyLukjanov do you want to run it? In that case I can edit my email and say we run rechecks | 08:29 |
| jhesketh | SergeyLukjanov: okay, trying again: http://paste.openstack.org/show/207927/ | 08:30 |
| SergeyLukjanov | AJaeger, jhesketh, do you remember, does we run rechecks previously when gerrit failed? | 08:30 |
| jhesketh | however that's just changes missing any votes, not any that should be running in gate | 08:30 |
| SergeyLukjanov | AJaeger, jhesketh honestly I don't remember | 08:30 |
| jhesketh | SergeyLukjanov: I'm not sure what has been done sorry | 08:30 |
| AJaeger | I don't remember. | 08:30 |
| AJaeger | But would be nice to do it... | 08:30 |
| jhesketh | SergeyLukjanov: bash history? | 08:31 |
| AJaeger | If there's a planned downtime, we normally requeue them... | 08:31 |
| *** xnox has joined #openstack-infra | 08:31 | |
| SergeyLukjanov | there is a short list | 08:31 |
| SergeyLukjanov | we could schedule it w/o any issues | 08:31 |
| *** markvoelker has quit IRC | 08:32 | |
| SergeyLukjanov | jhesketh, thx for the list, I'll run in a second | 08:32 |
| *** freyes has quit IRC | 08:32 | |
| SergeyLukjanov | AJaeger, ^^ | 08:32 |
| AJaeger | SergeyLukjanov, ok, will send out an updated email | 08:33 |
| *** ddieterly has joined #openstack-infra | 08:33 | |
| jhesketh | SergeyLukjanov: there will still be a disconnect between those and ones that should be in gate | 08:33 |
| *** shardy_ has joined #openstack-infra | 08:33 | |
| *** shardy has quit IRC | 08:34 | |
| *** freyes has joined #openstack-infra | 08:35 | |
| SergeyLukjanov | jhesketh, at least all changes will have votes | 08:35 |
| AJaeger | jhesketh, it covers most ;) My email states that we tried ;) | 08:35 |
| jhesketh | I'm working on a more complete list | 08:35 |
| *** Ala has quit IRC | 08:36 | |
| AJaeger | thanks, SergeyLukjanov and jhesketh ! | 08:36 |
| jhesketh | SergeyLukjanov: the api limits my queries to 500, so it's probably missing quite a few :-( | 08:37 |
| *** zhiwei has quit IRC | 08:37 | |
| SergeyLukjanov | jhesketh, I think it's ok, thanks! | 08:37 |
| openstackgerrit | Saggi Mizrahi proposed openstack-infra/project-config: Change dragonflow commit type https://review.openstack.org/177680 | 08:37 |
| *** ddieterly has quit IRC | 08:37 | |
| *** Ala has joined #openstack-infra | 08:38 | |
| *** shardy_ has quit IRC | 08:38 | |
| *** zul has joined #openstack-infra | 08:38 | |
| *** andrey-mp has quit IRC | 08:38 | |
| *** fhubik is now known as fhubik_afk | 08:39 | |
| *** shardy has joined #openstack-infra | 08:39 | |
| *** zul has quit IRC | 08:41 | |
| *** samuelBartel has quit IRC | 08:41 | |
| AJaeger | saggi, thanks for the verbose explanation. I'll abstain from commenting for now, others need to review this. | 08:41 |
| saggi | AJaeger: about GitHub description. Is there someone that can manually change it for us? Can we do it? We want the project to get traction so we try to get our info online to be helpful and accurate. | 08:41 |
| *** zul has joined #openstack-infra | 08:41 | |
| *** ildikov has quit IRC | 08:44 | |
| AJaeger | saggi, please ask the US based part of the team later | 08:44 |
| saggi | AJaeger: OK, thanks for all the help | 08:44 |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed https://review.openstack.org/173408 | 08:44 |
| openstack | bug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] https://launchpad.net/bugs/1444806 - Assigned to Bharat Kumar Kobagana (bharat-kobagana) | 08:44 |
| *** mfmcdonagh has quit IRC | 08:44 | |
| *** jamespage has quit IRC | 08:46 | |
| *** jamespage has joined #openstack-infra | 08:46 | |
| *** hyakuhei has quit IRC | 08:50 | |
| *** zhiwei has joined #openstack-infra | 08:50 | |
| *** sdake_ has joined #openstack-infra | 08:51 | |
| *** Darkwan has quit IRC | 08:52 | |
| *** fhubik_afk is now known as fhubik | 08:54 | |
| *** notnownikki has joined #openstack-infra | 08:55 | |
| *** sdake has quit IRC | 08:55 | |
| *** ildikov has joined #openstack-infra | 08:56 | |
| *** achanda has joined #openstack-infra | 08:59 | |
| *** e0ne has joined #openstack-infra | 08:59 | |
| *** mahito has quit IRC | 09:00 | |
| *** Darkwan has joined #openstack-infra | 09:01 | |
| *** _nadya_ has quit IRC | 09:03 | |
| *** Hal has joined #openstack-infra | 09:03 | |
| *** Hal is now known as Guest82649 | 09:04 | |
| jhesketh | the gerrit documentation seems to lie | 09:04 |
| jhesketh | I can't do a query with an offset | 09:04 |
| * jhesketh is probably failing at something obvious | 09:04 | |
| *** mpavone has quit IRC | 09:05 | |
| *** skolekonov has quit IRC | 09:05 | |
| *** flashgordon has joined #openstack-infra | 09:06 | |
| *** dalgaaf has quit IRC | 09:07 | |
| jhesketh | hmm, it doesn't look like you can search with an offset in gerrit 2.8 :-( | 09:09 |
| *** e0ne has quit IRC | 09:10 | |
| jhesketh | SergeyLukjanov: since the nodepool is exausted there is no point in figuring out the missing reviews due to the offset... I'll run my tool again later to see if there are more rechecks to be done, but for now we should catch most and others will catch the system up too | 09:10 |
| jhesketh | (others being people) | 09:10 |
| *** sdake_ has quit IRC | 09:13 | |
| *** e0ne has joined #openstack-infra | 09:13 | |
| *** Qiming has left #openstack-infra | 09:14 | |
| *** sdake has joined #openstack-infra | 09:15 | |
| *** pelix has joined #openstack-infra | 09:16 | |
| *** mpavone has joined #openstack-infra | 09:17 | |
| *** dtantsur|brb is now known as dtantsur | 09:19 | |
| *** fhubik is now known as fhubik_afk | 09:25 | |
| *** markvoelker has joined #openstack-infra | 09:27 | |
| *** dmellado has quit IRC | 09:28 | |
| *** fhubik_afk is now known as fhubik | 09:29 | |
| *** teran has quit IRC | 09:29 | |
| *** sks has joined #openstack-infra | 09:29 | |
| *** alexpilotti has joined #openstack-infra | 09:30 | |
| *** zul has quit IRC | 09:30 | |
| *** jamespage_ has joined #openstack-infra | 09:32 | |
| *** AJaeger has quit IRC | 09:32 | |
| *** Longgeek has quit IRC | 09:33 | |
| *** zul has joined #openstack-infra | 09:33 | |
| *** ddieterly has joined #openstack-infra | 09:33 | |
| *** Longgeek has joined #openstack-infra | 09:34 | |
| *** rossella_ has joined #openstack-infra | 09:34 | |
| *** ociuhandu has joined #openstack-infra | 09:35 | |
| *** markvoelker has quit IRC | 09:36 | |
| *** ddieterly has quit IRC | 09:37 | |
| *** yamamoto has quit IRC | 09:41 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/requirements: Add python-manilaclient to global requirements. https://review.openstack.org/135007 | 09:41 |
| *** AJaeger has joined #openstack-infra | 09:44 | |
| *** fhubik is now known as fhubik_afk | 09:44 | |
| *** fhubik_afk is now known as fhubik | 09:46 | |
| *** BobBall_ has joined #openstack-infra | 09:50 | |
| *** flip214 has quit IRC | 09:51 | |
| *** alexpilotti has quit IRC | 09:52 | |
| *** yamada-h has joined #openstack-infra | 09:53 | |
| *** BobBall has quit IRC | 09:54 | |
| *** mpavone has quit IRC | 09:54 | |
| *** achanda has quit IRC | 09:55 | |
| *** dguitarbite has quit IRC | 09:59 | |
| *** achanda has joined #openstack-infra | 10:00 | |
| *** mpavone has joined #openstack-infra | 10:01 | |
| *** dguitarbite has joined #openstack-infra | 10:01 | |
| *** yfried__ is now known as yfried|afk | 10:03 | |
| *** yfried|afk is now known as yfried__ | 10:06 | |
| *** alexpilotti has joined #openstack-infra | 10:07 | |
| *** sks has quit IRC | 10:08 | |
| *** sarob has joined #openstack-infra | 10:08 | |
| *** achanda has quit IRC | 10:11 | |
| *** mrunge has quit IRC | 10:11 | |
| *** sarob has quit IRC | 10:13 | |
| *** dims has joined #openstack-infra | 10:14 | |
| *** Longgeek has quit IRC | 10:14 | |
| *** Guest82649 has quit IRC | 10:17 | |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed https://review.openstack.org/173408 | 10:18 |
| openstack | bug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] https://launchpad.net/bugs/1444806 - Assigned to Bharat Kumar Kobagana (bharat-kobagana) | 10:18 |
| *** mattymo_ is now known as mattymo | 10:21 | |
| *** Longgeek has joined #openstack-infra | 10:26 | |
| *** markvoelker has joined #openstack-infra | 10:28 | |
| *** hichihara has quit IRC | 10:30 | |
| *** samueldmq has joined #openstack-infra | 10:32 | |
| *** markvoelker has quit IRC | 10:33 | |
| *** ddieterly has joined #openstack-infra | 10:34 | |
| *** dalgaaf has joined #openstack-infra | 10:36 | |
| *** ddieterly has quit IRC | 10:38 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 10:39 |
| *** yamada-h has joined #openstack-infra | 10:40 | |
| *** yamada-h has quit IRC | 10:40 | |
| *** sdake has quit IRC | 10:41 | |
| *** zul has quit IRC | 10:41 | |
| *** jamespage_ has quit IRC | 10:41 | |
| *** mrunge has joined #openstack-infra | 10:45 | |
| *** ociuhandu has quit IRC | 10:50 | |
| *** sdake has joined #openstack-infra | 10:51 | |
| *** lxsli0 is now known as lxsli | 10:52 | |
| *** u_glide has joined #openstack-infra | 10:54 | |
| *** ildikov has quit IRC | 10:56 | |
| *** ildikov has joined #openstack-infra | 10:56 | |
| *** alexpilotti has quit IRC | 10:58 | |
| *** fhubik is now known as fhubik_afk | 10:59 | |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| *** teran has joined #openstack-infra | 11:00 | |
| *** pc_m has joined #openstack-infra | 11:00 | |
| *** yamada-h has joined #openstack-infra | 11:01 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 11:02 |
| *** yamamoto has quit IRC | 11:02 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** _nadya_ has joined #openstack-infra | 11:02 | |
| *** teran has quit IRC | 11:03 | |
| *** teran has joined #openstack-infra | 11:03 | |
| *** sks has joined #openstack-infra | 11:03 | |
| *** zhiwei has quit IRC | 11:03 | |
| *** pc_m has quit IRC | 11:04 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 11:04 |
| *** pc_m has joined #openstack-infra | 11:04 | |
| *** afazekas has quit IRC | 11:05 | |
| *** yfried has joined #openstack-infra | 11:06 | |
| *** yfried__ has quit IRC | 11:07 | |
| *** e0ne is now known as e0ne_ | 11:07 | |
| *** e0ne_ is now known as e0ne | 11:07 | |
| *** pc_m has quit IRC | 11:09 | |
| *** pc_m_ has joined #openstack-infra | 11:09 | |
| *** sdake has quit IRC | 11:10 | |
| *** marcusvrn has joined #openstack-infra | 11:11 | |
| *** dprince has joined #openstack-infra | 11:11 | |
| *** marcusvrn1 has joined #openstack-infra | 11:12 | |
| *** yamada-h has quit IRC | 11:12 | |
| *** jamespage_ has joined #openstack-infra | 11:13 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Log heat-coe-templates to the openstack-containers channel https://review.openstack.org/171807 | 11:14 |
| *** marcusvrn has quit IRC | 11:15 | |
| *** pc_m_ has quit IRC | 11:18 | |
| *** e0ne is now known as e0ne_ | 11:19 | |
| *** pc_m has joined #openstack-infra | 11:19 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 11:19 |
| *** afazekas has joined #openstack-infra | 11:21 | |
| *** cdent has joined #openstack-infra | 11:23 | |
| *** salv-orlando has quit IRC | 11:23 | |
| *** dhritishikhar has joined #openstack-infra | 11:24 | |
| *** otter768 has joined #openstack-infra | 11:24 | |
| *** dkranz has quit IRC | 11:25 | |
| *** hdd_ has joined #openstack-infra | 11:25 | |
| deepakcs | Can someone help me understand the significance of "if [ "$BRANCH_OVERRIDE" != "default" ] ; then ..." in jenkins/jobs/devstack-gate.yaml ? | 11:25 |
| *** dkranz has joined #openstack-infra | 11:25 | |
| *** Mithrandir_ is now known as Mithrandir | 11:25 | |
| *** hdd has quit IRC | 11:27 | |
| *** coolsvap is now known as coolsvap|afk | 11:27 | |
| *** e0ne_ has quit IRC | 11:29 | |
| *** otter768 has quit IRC | 11:29 | |
| *** markvoelker has joined #openstack-infra | 11:29 | |
| rcarrillocruz | deepakcs: i guess it's because you can run different jobs per branch? | 11:29 |
| rcarrillocruz | different logic depending if it's default or not | 11:29 |
| *** mpaolino has quit IRC | 11:30 | |
| deepakcs | rcarrillocruz, branch of what ? not clear to me | 11:30 |
| *** jamespage_ has quit IRC | 11:31 | |
| *** jamespage__ has joined #openstack-infra | 11:32 | |
| *** zul has joined #openstack-infra | 11:32 | |
| rcarrillocruz | of the project being tested | 11:32 |
| rcarrillocruz | in this case, devstack-gate | 11:32 |
| rcarrillocruz | check it out projects.yaml: | 11:32 |
| AJaeger | deepakcs, jenkins/jobs/projects.yaml - line 2001 | 11:32 |
| *** pc_m has quit IRC | 11:32 | |
| AJaeger | "branch-override: stable/icehouse" | 11:33 |
| rcarrillocruz | http://paste.openstack.org/show/208157/ | 11:33 |
| rcarrillocruz | you pass branch-override per project | 11:33 |
| deepakcs | AJaeger, ah ok, so its the branch of project (cinder, glance etc) to test | 11:33 |
| *** markvoelker has quit IRC | 11:33 | |
| *** pc_m has joined #openstack-infra | 11:34 | |
| deepakcs | AJaeger, so if my job-template in devstack-gate.yaml does not use "if [ "$BRANCH_OVERRIDE" != "default" ] ..." - what does it mean ? Does it mean i cannot test anything but latest/master of the projects ? | 11:34 |
| *** ddieterly has joined #openstack-infra | 11:35 | |
| *** pc_m has quit IRC | 11:35 | |
| deepakcs | AJaeger, rcarrillocruz for eg: devstack-gate.yaml line: 1911 - devstack plugin template does not use branch-override at all | 11:35 |
| AJaeger | deepakcs, let's better ask sdague and fungi... | 11:35 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Add cloud-config-hosts element for non-puppet jobs https://review.openstack.org/177722 | 11:35 |
| AJaeger | branch-designator allows also to have different conditions (voting/non-voting) for one job | 11:36 |
| *** pblaho has joined #openstack-infra | 11:37 | |
| *** markvoelker has joined #openstack-infra | 11:38 | |
| *** ddieterly has quit IRC | 11:39 | |
| *** pc_m has joined #openstack-infra | 11:41 | |
| deepakcs | AJaeger, ya, i know about branch-designator, asking about the branch-override not being used in the devstack plugin job template | 11:43 |
| *** Ala has quit IRC | 11:43 | |
| AJaeger | deepakcs, those are often used together - but yeah, I also have some questions here ;) | 11:44 |
| deepakcs | AJaeger, ok :) i wanted to understand the relation between designator and override if any.. bcos i am seeing soem strange failure in one of the patch (it says duplicate job check-tempest-dsvm-glusterfs) exists | 11:44 |
| *** ldnunes has joined #openstack-infra | 11:44 | |
| deepakcs | AJaeger, http://logs.openstack.org/08/173408/9/check/gate-project-config-layout/65cbff1/console.html | 11:44 |
| *** salv-orlando has joined #openstack-infra | 11:45 | |
| *** mrmartin has joined #openstack-infra | 11:45 | |
| *** dkranz has quit IRC | 11:47 | |
| AJaeger | deepakcs, duplicate via projects.yaml, line 3549 on project devstack-plugin-glusterfs: '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}' | 11:47 |
| *** alexpilotti has joined #openstack-infra | 11:47 | |
| deepakcs | AJaeger, inspite of no duplicate jobs being present there ( I have 2 jobs -nv and one without -nv) but it seems its seeing both of them as same, hence the dup | 11:47 |
| AJaeger | '{pipeline}-tempest-dsvm-full-glusterfs{branch-designator}' and '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}' gives the duplicate | 11:48 |
| deepakcs | AJaeger, '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}' and '{pipeline}-tempest-dsvm-full-glusterfs{branch-designator}' are _not_ duplicates, right ? from a job template perspective ? | 11:48 |
| AJaeger | with plugin=glusterfs | 11:48 |
| deepakcs | AJaeger, why ? | 11:48 |
| *** mwagner_lap has quit IRC | 11:48 | |
| deepakcs | AJaeger, but there is no job with plugin: glusterfs | 11:48 |
| *** pc_m has quit IRC | 11:48 | |
| AJaeger | deepakcs, they are duplicates, both the same name, aren't they? | 11:48 |
| *** chmouel has quit IRC | 11:48 | |
| *** tnovacik has quit IRC | 11:48 | |
| deepakcs | AJaeger, but the name matchign happens before the yaml params are expanded, right ? | 11:48 |
| *** pc_m has joined #openstack-infra | 11:48 | |
| deepakcs | AJaeger, both are not same, one has {plugin} and other has glusterfs in the same place | 11:49 |
| openstackgerrit | Merged openstack-infra/project-config: Log heat-coe-templates to the openstack-containers channel https://review.openstack.org/171807 | 11:49 |
| AJaeger | deepakcs, the name matchin is after expansion | 11:49 |
| *** dkranz has joined #openstack-infra | 11:49 | |
| AJaeger | You have '{pipeline}-tempest-dsvm-full-glusterfs{branch-designator} which is check-tempest-dsvm-full-glusterfs" | 11:49 |
| AJaeger | and you have '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}' which gives the same name | 11:50 |
| *** chmouel has joined #openstack-infra | 11:50 | |
| deepakcs | AJaeger, if so, then there is no job with plugin: glusterfs in the yaml file (the patch replaces plugin: glusterfs as we are creating a specific glusterfs job) | 11:50 |
| AJaeger | deepakcs, scroll back: "line 3549 in projects.yaml" | 11:50 |
| deepakcs | AJaeger, thats the line the patch under Q changes :) | 11:51 |
| AJaeger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n3554 | 11:51 |
| deepakcs | AJaeger, and remove plugin: glusterfs | 11:51 |
| AJaeger | let me double check your patch... | 11:51 |
| deepakcs | AJaeger, sure, thanks | 11:51 |
| *** baoli has joined #openstack-infra | 11:51 | |
| deepakcs | AJaeger, also a related Q... if the name matching happens _after_ expansion, how does it know what to match, bcos job-templates in devstack-gate.yaml has parameterized names (not with expanded ones) | 11:51 |
| deepakcs | AJaeger, are u sure that the name matchin is after expansion ? | 11:52 |
| openstackgerrit | Merged openstack-infra/project-config: Adding create permissions to networking-l2gw https://review.openstack.org/177637 | 11:52 |
| openstackgerrit | Merged openstack-infra/project-config: Add Ironic python3.4 experimental job https://review.openstack.org/156530 | 11:52 |
| *** fhubik_afk is now known as fhubik | 11:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Elastic Recheck to docs.openstack.org/infra https://review.openstack.org/177077 | 11:53 |
| *** fhubik is now known as fhubik_afk | 11:53 | |
| AJaeger | deepakcs, just rename your job ;) | 11:53 |
| deepakcs | AJaeger, some black magic happenign there ? | 11:54 |
| AJaeger | AFAIK you can say both: "gate-{name}-pep8" with "name: bla" and "gate-bla-pep8" | 11:54 |
| *** afazekas has quit IRC | 11:54 | |
| deepakcs | AJaeger, you couldn't figure why its giving dup ? | 11:54 |
| AJaeger | jjb will expand it - and the same is here, jjb could expand your job in two different ways | 11:54 |
| *** dkranz has quit IRC | 11:55 | |
| deepakcs | AJaeger, jjb should do that only if plugin: glusterfs was defined for some job,but that isn't the case, since the patch remove that line | 11:55 |
| deepakcs | AJaeger, so still unclear on how/why it sees 2 dup jobs | 11:55 |
| openstackgerrit | Merged openstack-infra/project-config: Check more Puppet modules with tripleo-f20puppet-nonha https://review.openstack.org/177392 | 11:55 |
| *** dkranz has joined #openstack-infra | 11:56 | |
| AJaeger | deepakcs, devstack-gate.yaml still has two jobs that can be used for your line. | 11:56 |
| *** chmouel_ has joined #openstack-infra | 11:57 | |
| deepakcs | AJaeger, Hmm interesting | 11:57 |
| AJaeger | deepakcs, I suggest to discuss with clarkb or fungi, they might be able to give you a better explanation. | 11:57 |
| *** chmouel has quit IRC | 11:57 | |
| deepakcs | AJaeger, ok one quick observation. I removed line 2044 from jenkins/jobs/devstack-gate.yaml and then it no longer gives dup job error. Any idea how removing that makes it work ? | 11:57 |
| *** zhiwei has joined #openstack-infra | 11:58 | |
| *** sileht has quit IRC | 11:59 | |
| AJaeger | deepakcs, interesting ;) no quick idea | 11:59 |
| *** e0ne has joined #openstack-infra | 11:59 | |
| AJaeger | Please do rename your job, we know it will cause problems, so let's name them differently. And if the patch removes the usage of the old template, let's do it properly and remove it everywhere. | 12:00 |
| *** Longgeek has quit IRC | 12:00 | |
| *** ociuhandu has joined #openstack-infra | 12:00 | |
| *** ociuhandu has quit IRC | 12:00 | |
| AJaeger | And in that case I wonder why you cannot just change the old template? | 12:01 |
| *** alexpilotti has quit IRC | 12:01 | |
| *** dizquierdo has joined #openstack-infra | 12:01 | |
| deepakcs | AJaeger, change old template meaning ? | 12:01 |
| deepakcs | AJaeger, old template was the generic devstack plugin template | 12:01 |
| *** woodster_ has joined #openstack-infra | 12:01 | |
| AJaeger | deepakcs, ah, you're right - that shouldn't be touched | 12:01 |
| deepakcs | AJaeger, we tried to add tempest specific regex in that old template for 'glusterfs' and sdague didn't like it.. he suggested to add new job, hence this new patch :) | 12:02 |
| deepakcs | AJaeger, i hopeu remember giving +2 and then removign it after sdague comment :) | 12:02 |
| AJaeger | deepakcs, I have to leave now for a meeting... but yes, there was something... | 12:02 |
| deepakcs | AJaeger, ok thanks for ur time | 12:02 |
| *** yfried_ has joined #openstack-infra | 12:03 | |
| *** liam__ has joined #openstack-infra | 12:04 | |
| *** Ala has joined #openstack-infra | 12:05 | |
| *** Longgeek has joined #openstack-infra | 12:05 | |
| *** openstackgerrit has quit IRC | 12:06 | |
| *** yfried has quit IRC | 12:06 | |
| *** openstackgerrit has joined #openstack-infra | 12:06 | |
| *** pc_m has quit IRC | 12:07 | |
| *** afazekas has joined #openstack-infra | 12:07 | |
| *** sileht has joined #openstack-infra | 12:09 | |
| *** dkranz has quit IRC | 12:09 | |
| *** dkranz has joined #openstack-infra | 12:11 | |
| deepakcs | AJaeger, fwiw, we found the root cause, adding a job entry under devstack-jobs ( line 2044 from jenkins/jobs/devstack-gate.yaml ) causes check-tempest-dsvm-full-glusterfs job to be defined and we are also defining it under our job definition, hence the dup | 12:12 |
| *** bswartz has quit IRC | 12:12 | |
| *** u_glide has left #openstack-infra | 12:12 | |
| deepakcs | AJaeger, the correct way to fix it is to remove line 2044 from jenkins/jobs/devstack-gate.yaml - its not needed there, we create our jobs so no need to add our job-template under devstack-jobs section | 12:12 |
| *** weshay has joined #openstack-infra | 12:13 | |
| mordred | saggi: morning | 12:14 |
| saggi | mordred: morning | 12:14 |
| *** deepakcs has quit IRC | 12:15 | |
| *** esker has quit IRC | 12:15 | |
| AJaeger | deepakcs, I don't think that removing is the proper fix, please rename | 12:15 |
| mordred | in reverse order - one of us can change your github description - but it's not a thing we want to update a lot :) | 12:16 |
| mordred | saggi: also - as for ff-merge - let me look at your patch | 12:16 |
| openstackgerrit | Julien Danjou proposed openstack-dev/hacking: tests: allow to run the same test for different code https://review.openstack.org/152108 | 12:16 |
| saggi | mordred: Thanks, We don't intend to change it a lot :). | 12:17 |
| *** mpaolino has joined #openstack-infra | 12:18 | |
| *** mpaolino has quit IRC | 12:18 | |
| mordred | sdague: cool. just let me know what to change it to | 12:18 |
| AJaeger | fungi, mordred : I'm not seeing the latest merges to project-config in the post queue using status.openstack.org/zuul | 12:18 |
| mordred | gah | 12:18 |
| mordred | AJaeger: yay! | 12:18 |
| AJaeger | any idea what's broken? OTher jobs go in.. | 12:19 |
| *** dprince has quit IRC | 12:19 | |
| *** fhubik_afk is now known as fhubik | 12:20 | |
| mordred | AJaeger: I see things in the post queue? | 12:20 |
| mordred | saggi: cool. just let me know what to change it to | 12:20 |
| saggi | mordred: Whatever it is now. We already updated the project-config | 12:21 |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed https://review.openstack.org/173408 | 12:21 |
| openstack | bug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] https://launchpad.net/bugs/1444806 - Assigned to Bharat Kumar Kobagana (bharat-kobagana) | 12:21 |
| mordred | saggi: also - as for your change to use FF-only - I believe that change to your project-config will work - but I'll warn you it's gonna be super annoying to have gerrit set up that way | 12:21 |
| mordred | saggi: gotcha | 12:21 |
| mordred | saggi: what are you trying to accomplish with ff-only? | 12:22 |
| *** BobBall_ is now known as BobBall | 12:22 | |
| *** dkranz has quit IRC | 12:22 | |
| *** fhubik is now known as fhubik_afk | 12:25 | |
| *** liam__ has quit IRC | 12:25 | |
| openstackgerrit | Merged openstack-infra/project-config: New Fuel plugin project for Calico https://review.openstack.org/174763 | 12:25 |
| saggi | mordred: As I said in my commit message, Gerrit+Merge commits are a bad idea. They break most of gits helpful features. We currently don't have a lot of tests and do a lot of infra changes so we want to make sure developers test against master before we commit. We will probably change to cherry-pick later. If this is an issue we could just move to cherry-pick instead. | 12:25 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Add cloud-init-hosts element for non-puppet jobs https://review.openstack.org/177722 | 12:26 |
| *** fhubik_afk is now known as fhubik | 12:27 | |
| saggi | mordred: I honestly don't understand why I am the first to bring this up. Doesn't anybody care that the git history is a mess. But I'm not here to start a crusade, I just want my master branch to be readable. | 12:27 |
| *** Hal has joined #openstack-infra | 12:29 | |
| *** Hal is now known as Guest20374 | 12:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Support feature/development for askbot-theme https://review.openstack.org/176958 | 12:29 |
| *** jamespage__ has quit IRC | 12:30 | |
| *** gordc has joined #openstack-infra | 12:30 | |
| *** bknudson has quit IRC | 12:31 | |
| *** claudiub has joined #openstack-infra | 12:31 | |
| *** chmouel_ is now known as chmouel | 12:33 | |
| mordred | saggi: I hear that - it's just an unworkable option on most of our projects due to velocity | 12:35 |
| *** rfolco has joined #openstack-infra | 12:35 | |
| saggi | mordred: Even cherry-pick? | 12:36 |
| mordred | saggi: also, cherry-pick in gerrit gets very confusing for end users | 12:36 |
| mordred | yah | 12:36 |
| *** dprince has joined #openstack-infra | 12:36 | |
| *** samuelBartel has joined #openstack-infra | 12:36 | |
| *** ddieterly has joined #openstack-infra | 12:36 | |
| *** kgiusti has joined #openstack-infra | 12:36 | |
| mordred | it doesn't _quite_ work the way people think it will - we had it turned on for sevearl of the infra projects for a while | 12:36 |
| *** Longgeek has quit IRC | 12:36 | |
| mordred | it was more of a headache than it was worth | 12:36 |
| mordred | I'd try ff-only if you feel strongly about not seeing merge commits and see how it works for you | 12:37 |
| openstackgerrit | Roman Vasilets proposed openstack-infra/project-config: Using Neutron network by default in gate-rally-dsvm-murano-rally https://review.openstack.org/177746 | 12:37 |
| *** openstackgerrit has quit IRC | 12:37 | |
| mordred | however - you'll find that 100% of our infrastructure around here is built on the assumption that we're running merge commits | 12:37 |
| *** openstackgerrit has joined #openstack-infra | 12:37 | |
| mordred | so it's entirely possible that things will break in unexpected ways | 12:37 |
| saggi | mordred: I worked with gerrit+cherry-pick for years. Had no issues. | 12:38 |
| saggi | mordred: hmmmm | 12:38 |
| mordred | awesome | 12:38 |
| *** Longgeek has joined #openstack-infra | 12:39 | |
| *** ddieterly has quit IRC | 12:40 | |
| openstackgerrit | Roman Vasilets proposed openstack-infra/project-config: Using Neutron network in gate-rally-dsvm-murano-rally https://review.openstack.org/177746 | 12:41 |
| mordred | saggi: we have approx 2k developers who have very basic git knowledge - most of them perform the one or two commands we tell them to perform - so "confusing for end users" for us is always trying to look through that lens. _we_ can handle the ramifications of gerrit cherry-pick - but when running it it required enough git smarts on the user side that it seemed pretty unworkable for our general | 12:41 |
| mordred | population | 12:41 |
| mordred | there are a few places where we explicitly don't do "nicer" things for that reason - and it can be annoying for folks with deeper git-fu | 12:42 |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta https://review.openstack.org/177747 | 12:42 |
| *** ayoung has joined #openstack-infra | 12:43 | |
| *** xylan has joined #openstack-infra | 12:43 | |
| *** rossella_s has quit IRC | 12:44 | |
| *** rossella_ has quit IRC | 12:44 | |
| *** rossella_s has joined #openstack-infra | 12:44 | |
| *** miqui has joined #openstack-infra | 12:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 12:45 |
| *** BharatK has quit IRC | 12:47 | |
| openstackgerrit | Renat Akhmerov proposed openstack-infra/project-config: Add acl rule for Mistral to allow creating branches https://review.openstack.org/177750 | 12:47 |
| xylan | hi there, i proposed a new stackforge project, https://review.openstack.org/177747, but i didn't know whether or not everything is ok, i just did this according to http://docs.openstack.org/infra/manual/creators.html | 12:47 |
| *** fhubik is now known as fhubik_afk | 12:48 | |
| *** bswartz has joined #openstack-infra | 12:49 | |
| *** abregman has quit IRC | 12:49 | |
| *** jamespage_ has joined #openstack-infra | 12:50 | |
| saggi | mordred: I understand, I do like my git history. It's so useful. But I don't want to go against the grain if that means it might break something in the workflow. | 12:51 |
| *** sileht has quit IRC | 12:52 | |
| AJaeger | mordred, there are things in the post queue but project-config is not in it at all ;( | 12:55 |
| mordred | AJaeger: oh! project-config patches are not doing post queue | 12:55 |
| mordred | WEIRD | 12:55 |
| AJaeger | mordred, indeed | 12:56 |
| *** bknudson has joined #openstack-infra | 12:56 | |
| mordred | AJaeger: I blame jaypipes, but I have no reason to do so | 12:56 |
| * jaypipes hides | 12:57 | |
| openstackgerrit | Renat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral https://review.openstack.org/175328 | 12:58 |
| AJaeger | xylan, you did not setup irc access properly, see ci.openstack.org/irc.html | 12:59 |
| xylan | AJaeger: thanks for feedback...I see that. but, how could i create a channel? | 13:00 |
| AJaeger | xylan, you setup #terracota in your change - if you don't want it, then don't add it ;) | 13:01 |
| *** jamesmcarthur has joined #openstack-infra | 13:02 | |
| xylan | AJaeger: i really need it. but i didn't know how to create it | 13:02 |
| *** jamespage_ has quit IRC | 13:03 | |
| AJaeger | xylan, check the URL for comments on how to setup a new channel - and read the freenode docs | 13:03 |
| *** jamespage_ has joined #openstack-infra | 13:03 | |
| xylan | AJaeger: ok, thanks very much! | 13:03 |
| AJaeger | xylan, patch looks fine otherwise | 13:03 |
| *** fhubik_afk is now known as fhubik | 13:03 | |
| xylan | AJaeger: :) | 13:04 |
| *** jamespage_ has quit IRC | 13:04 | |
| * morganfainberg waves at jeblair. | 13:05 | |
| fungi | AJaeger: which change merging should have triggered which post jobs for project-config? | 13:05 |
| AJaeger | https://review.openstack.org/#/c/177077/ is the one that triggered my research | 13:05 |
| AJaeger | But then checking status.o.o/zuul I say not a single job in the post queue for project-config - and none entering it. | 13:06 |
| AJaeger | While other projects entered easily | 13:06 |
| AJaeger | so, http://docs.openstack.org/infra/ is still not created | 13:07 |
| fungi | AJaeger: i believe the only current post jobs for it are scoped to changes for specific file patterns. unless changes touch files in the docs-site or specs directories or the tox.ini file, there's nothing to be triggered | 13:07 |
| *** jtriley has joined #openstack-infra | 13:07 | |
| AJaeger | ah, you're right - post jobs is only specs site and docs-site. | 13:08 |
| fungi | AJaeger: see the files scoping for ^(check-infra-docs-index|publish-infra-docs-index)$ and ^(check-generate-specs-site|publish-specs-site)$ in the jobs list of layout.yaml | 13:08 |
| AJaeger | But docs-site did not trigger - or did not work as expected | 13:08 |
| AJaeger | fungi, yeah, just double checked | 13:08 |
| fungi | yeah, that job might also be broken | 13:08 |
| *** ZZelle has joined #openstack-infra | 13:08 | |
| AJaeger | 177077 should have triggered. The check-infra-docs-index was run | 13:09 |
| * jeblair waves at morganfainberg | 13:09 | |
| *** alexpilotti has joined #openstack-infra | 13:10 | |
| AJaeger | so, why did publish-specs-site not trigger? Is there a problem running this in post? | 13:10 |
| * mordred waves at morganfainberg and jeblair | 13:10 | |
| fungi | AJaeger: i'm looking for logs now | 13:10 |
| morganfainberg | mordred: ohai! | 13:10 |
| fungi | hrm, http://logs.openstack.org/f1/f13903df6ca7c8fade20024e01e0095e1c717297/ gives me a 404 | 13:10 |
| AJaeger | I meant: publish-infra-docs-index. Thanks, fugni for looking | 13:10 |
| AJaeger | fungi, or should it be: http://logs.openstack.org/57/570cf7ced999833afc2f2735dcb21b8cb18a340b/ ? Also 404 | 13:11 |
| fungi | oh, good point. i needed to dig out the merge commit sha | 13:11 |
| AJaeger | merge commit sha is the one above | 13:12 |
| AJaeger | if it's really the merge commit that'S the key: will checking for files really work in the post queue? | 13:13 |
| AJaeger | The match for 'docs-site/.*' will not work for the merge sha, or does it? | 13:13 |
| fungi | that's a good point too. the specs site used it, but... | 13:13 |
| *** dustins has joined #openstack-infra | 13:14 | |
| fungi | it's entirely possible we've also never used that job either since that got restricted | 13:15 |
| AJaeger | might be | 13:15 |
| mordred | sdague: hey - so, you use devstack-vagrant, yeah? can I use it to get a single-node devstack? | 13:16 |
| sdague | mordred: yes | 13:16 |
| sdague | there is a config.yaml (sample included) | 13:16 |
| sdague | if you only define hostname_manager, and not hostname_compute, you get a single node | 13:16 |
| fungi | AJaeger: looks like you restricted that last month with https://review.openstack.org/159512 | 13:17 |
| fungi | so we might just need to take that file restriction off for the publish versions | 13:17 |
| *** dims has quit IRC | 13:17 | |
| mordred | sdague: cool | 13:17 |
| sdague | it should probably be documented better, but it's a pretty spare time thing, so I haven't gotten around to it | 13:17 |
| * AJaeger hides | 13:17 | |
| fungi | AJaeger: and let it run for every change post merge | 13:17 |
| AJaeger | fungi, ok, will change both jobs | 13:17 |
| *** dims has joined #openstack-infra | 13:18 | |
| mordred | sdague: also - and I'm about to show my extreme ignorance here | 13:18 |
| mordred | sdague: I just got an error in trying to stand up a normal vm that something about not being able to run kvm and virtualbox | 13:18 |
| fungi | AJaeger: now that you mention it, i do recall files sections only being valid for proper change pipelines (i.e. those which act on gerrit change ids) | 13:18 |
| mordred | sdague: do I just rmmod kvm to make kvm go away right now? | 13:18 |
| sdague | hmmmm.... that seems weird, I think I have both running on my boxes | 13:18 |
| fungi | AJaeger: so that is almost certainly the problem | 13:18 |
| sdague | I suppose, but I didn't think I'd run into that issue before | 13:19 |
| AJaeger | fungi, thanks! Learned something new... | 13:19 |
| mordred | sdague: VT-x is being used by another hypervisor | 13:19 |
| mordred | sdague: virtualbox can't operate in VMX root mode. Please disable the KVM kernel extension, recompile your kernel and reboot. | 13:20 |
| sdague | hmm... weird | 13:20 |
| mordred | sdague: :) | 13:20 |
| sdague | because I don't get that error :) | 13:20 |
| mordred | sdague: have I mentioned that I've never gotten vagrant to boot a machine ever | 13:20 |
| sdague | kernel 3.19.0 | 13:20 |
| mordred | yup | 13:20 |
| sdague | mordred: you have not mentioned that fact | 13:20 |
| mordred | sdague: I'm going to make it happen today | 13:20 |
| sdague | vagrant 1:1.6.3 | 13:20 |
| AJaeger | our gerritbot is dead, can anybody restart it? | 13:21 |
| *** Ala has quit IRC | 13:21 | |
| fungi | AJaeger: i'll whip up an addition to http://ci.openstack.org/zuul/zuul.html#jobs to that effect too | 13:21 |
| AJaeger | fungi, https://review.openstack.org/177762 is the change I just pushed | 13:21 |
| fungi | AJaeger: restarting it now | 13:21 |
| mordred | sdague: 1.6.5+dfsg1-2 | 13:21 |
| *** yamahata has quit IRC | 13:21 | |
| *** tiswanso has joined #openstack-infra | 13:21 | |
| sdague | I'm trying to remember if there was anything I needed to once tweek in virtual box | 13:21 |
| mordred | sdague: with 4.3.26-dfsg-2ubuntu1 of virtualbox | 13:21 |
| AJaeger | fungi, let's get that updated content to http://docs.openstack.org/infra/zuul ;) | 13:21 |
| *** openstackgerrit has quit IRC | 13:21 | |
| *** openstackgerrit has joined #openstack-infra | 13:22 | |
| sdague | virtualbox-4.3 4.3.26-98988~Ubuntu~raring | 13:22 |
| sdague | so... I'm running virtualbox from oracle packages | 13:22 |
| *** esker has joined #openstack-infra | 13:22 | |
| fungi | AJaeger: okay, openstackgerrit has been restarted. hopefully its silence is not a sign that gerrit stream-events is dead again | 13:22 |
| AJaeger | fungi, should we add a comment to the beginning of the jobs section in zuul/layout.yaml | 13:22 |
| sdague | I do no know if that's important to this or not | 13:23 |
| AJaeger | "Queue lengths: 5282 events, 77 results. " in zuul | 13:23 |
| *** rkukura has quit IRC | 13:23 | |
| *** russellb has quit IRC | 13:24 | |
| fungi | last stream event seen by zuul was 13:09 utc (15 minutes ago) | 13:24 |
| fungi | hopefully that's just the queue length | 13:24 |
| sdague | mordred: I am happy to compare config values if you find things aren't working, I don't think I've had an issue recently getting vagrant / vbox up on ubunut | 13:24 |
| AJaeger | my project-config change is not in the queue ;( | 13:25 |
| *** jtriley_ has joined #openstack-infra | 13:25 | |
| AJaeger | fungi, seems we have a problem again | 13:25 |
| *** otter768 has joined #openstack-infra | 13:25 | |
| jeblair | AJaeger: wait until zuul's queue is empty to make that determination | 13:25 |
| AJaeger | "Queue lengths: 0 events, 0 results." | 13:25 |
| jeblair | AJaeger: okay then :) my wifi is slow | 13:26 |
| fungi | unfortunately, i'm not on a fast enough computer to load the zuul status page yet | 13:26 |
| jeblair | AJaeger, fungi: yeah, looks stuck | 13:26 |
| *** pc_m has joined #openstack-infra | 13:26 | |
| *** 7JTAAIHSL has quit IRC | 13:26 | |
| jeblair | 01861a4b waiting .... 13:10:07.270 com.google.gerrit.sshd.commands.StreamEv | 13:27 |
| jeblair | lots of those | 13:27 |
| jeblair | i guess my patch still isn't deployed :/ | 13:27 |
| *** tnovacik has joined #openstack-infra | 13:27 | |
| *** eharney has quit IRC | 13:27 | |
| fungi | i'll get a good set of thread dumps and then restart | 13:27 |
| TheJulia | w/win 9 | 13:27 |
| TheJulia | doh | 13:27 |
| *** russellb has joined #openstack-infra | 13:28 | |
| *** jtriley has quit IRC | 13:28 | |
| *** peristeri has joined #openstack-infra | 13:28 | |
| *** Ala has joined #openstack-infra | 13:28 | |
| *** mriedem has joined #openstack-infra | 13:29 | |
| *** otter768 has quit IRC | 13:29 | |
| jeblair | fungi: when you're done, what's the status of the gerrit build with the debugging patch? | 13:29 |
| *** marcusvrn1 has quit IRC | 13:30 | |
| *** superdan is now known as dansmith | 13:31 | |
| *** devvesa has joined #openstack-infra | 13:32 | |
| fungi | jeblair: zaro was unable to reproduce (or even get gerrit running normally) with the version of gerrit we're running now, but the patch seems to be working when picked onto 2.10 and since it's not long until the kilo release and we're nearing the gerrit upgrade i was reluctant to schedule a lengthy downtime for gerrit (since testing that again will result in another lucene reindex) | 13:32 |
| *** sileht has joined #openstack-infra | 13:32 | |
| *** mestery_ is now known as mestery | 13:32 | |
| fungi | i've got jstack thread dumps in my homedir, named jstack.1 through 3 | 13:32 |
| fungi | and gerrit is restarted again | 13:32 |
| *** BobH has joined #openstack-infra | 13:34 | |
| *** arielb has joined #openstack-infra | 13:34 | |
| *** Ala has quit IRC | 13:34 | |
| *** ddieterly has joined #openstack-infra | 13:34 | |
| *** dboik has quit IRC | 13:34 | |
| *** Ala has joined #openstack-infra | 13:34 | |
| jeblair | yeah, but if this is going to keep happening every few hours for the next two weeks, i'd love it if we could make progress on debugging it | 13:35 |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta https://review.openstack.org/177747 | 13:35 |
| AJaeger | thanks, fungi! | 13:35 |
| mordred | sdague: ok - we're one step closer to having achieved this tool working for me | 13:35 |
| mordred | sdague: I now have an instance booted, but ssh keys don't seem to be set up? | 13:35 |
| fungi | agreed. i've not seen it happen this close together before today | 13:35 |
| mordred | sdague: mordred@camelot:~/vms$ ssh -p 2222 vagrant@localhost | 13:35 |
| mordred | vagrant@localhost's password: | 13:35 |
| *** jamesmcarthur has quit IRC | 13:35 | |
| jeblair | fungi: just so we're clear -- i _ran_ the patch locally | 13:36 |
| mordred | sdague: (this isn't using devstack-vagrant yet - I did a 'vagrant init ubuntu/trusty64 ; vagrant up' - just to make sure I could get that far | 13:36 |
| jeblair | there is no reason i can think of that there is something wrong with the patch. we have an undiagnosed problem with the build | 13:36 |
| sdague | mordred: does "vagrant ssh" work ? | 13:36 |
| mordred | sdague: AH! | 13:36 |
| *** marcusvrn1 has joined #openstack-infra | 13:36 | |
| mordred | never would have thought of that | 13:36 |
| mordred | sdague: yes | 13:37 |
| mordred | it does | 13:37 |
| sdague | yeh, so vagrant ssh is for built in connection to all gutets | 13:37 |
| sdague | guests | 13:37 |
| sdague | however... | 13:37 |
| sdague | config.yaml supports adding stack_sshkey | 13:37 |
| mordred | gotcha. | 13:37 |
| sdague | so you can | 13:37 |
| fungi | jeblair: do you have a different build which works? i just also don't want to have gerrit offline for hours while we try to figure out what about that build is broken. but you're right if it's going to start continually breaking now then better that we make progress while it's offline than have it not triggering ci jobs | 13:37 |
| mordred | well, I'll try devstack-vagrant now | 13:37 |
| sdague | ssh stack@.... | 13:37 |
| AJaeger | fungi, do we need to send a note out to ask people to requeue jobs - or will you do it automatically somehow? | 13:37 |
| jeblair | fungi: yeah, i think we should download gerrit-v2.8.4.19.4548330.war and run it locally on some machine to make sure the problem is reproduced, then create another build and run it locally to make sure it does not appear, then upload that manually to tarballs and deploy it | 13:39 |
| fungi | jeblair: zaro says he downloaded that and it ran fine | 13:39 |
| fungi | for him | 13:39 |
| jeblair | fungi: i'll see if i can find the build i used for testing | 13:39 |
| fungi | AJaeger: yes, i'll switch to another room and work out the details | 13:40 |
| jeblair | fungi: oh, i thought he never got it working. i must have bisunderstood | 13:40 |
| *** Guest20374 has quit IRC | 13:40 | |
| AJaeger | fungi, thanks! Are you travelling? | 13:40 |
| mordred | sdague: do I put the filename of the ssh key? or the contents? | 13:40 |
| fungi | jeblair: he was never able to get any build of gerrit to start with contactstore enabled | 13:40 |
| *** Hal has joined #openstack-infra | 13:40 | |
| fungi | for reasons he's unable to identify yet | 13:40 |
| sdague | mordred: the pub key data | 13:40 |
| *** Hal is now known as Guest89094 | 13:40 | |
| sdague | so, contents | 13:40 |
| fungi | AJaeger: no, i just made the mistake of looking at irc from bed when i first woke up, and have been trapped there ever since | 13:40 |
| sdague | oh, hmm.... that's not in the sample documentaiton | 13:41 |
| fungi | waiting on a lull in the crazy | 13:41 |
| AJaeger | fungi, take a break, drink a coffee ;) | 13:41 |
| sdague | I should fix that | 13:41 |
| sdague | oh, no, it's there, it's just not as clearly docced as it could be | 13:42 |
| *** Guest89094 has quit IRC | 13:42 | |
| openstackgerrit | Merged openstack-dev/pbr: Honour new test variable PIPVERSION https://review.openstack.org/177457 | 13:42 |
| *** pc_m has quit IRC | 13:43 | |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Add os-testr to global requirements list https://review.openstack.org/170268 | 13:43 |
| mtreinish | sdague: if you get a sec https://review.openstack.org/#/c/177306/ just adds the tempest-lib kilo cap | 13:44 |
| sdague | mtreinish: +2 | 13:44 |
| *** morganfainberg has quit IRC | 13:44 | |
| *** jroll has quit IRC | 13:44 | |
| *** jroll has joined #openstack-infra | 13:44 | |
| AJaeger | sdague, mordred could you review https://review.openstack.org/177762 , please? | 13:45 |
| mtreinish | sdague: cool, thanks | 13:45 |
| *** xianghui has quit IRC | 13:45 | |
| fungi | #status notice gerrit has been restarted to clear a problem with its event stream. change events between 13:09 and 13:36 utc should be rechecked or have approval votes reapplied as needed to trigger jobs | 13:46 |
| openstackstatus | fungi: sending notice | 13:46 |
| *** fhubik is now known as fhubik_afk | 13:46 | |
| *** sergsh has quit IRC | 13:46 | |
| *** xianghui has joined #openstack-infra | 13:46 | |
| -openstackstatus- NOTICE: gerrit has been restarted to clear a problem with its event stream. change events between 13:09 and 13:36 utc should be rechecked or have approval votes reapplied as needed to trigger jobs | 13:47 | |
| *** morganfainberg has joined #openstack-infra | 13:47 | |
| AJaeger | thanks, sdague, fungi | 13:47 |
| anteaya | jhesketh: when you say you can't do a query with an offset, what query were you trying to run? you probably are correct, I'm just curious | 13:47 |
| openstackstatus | fungi: finished sending notice | 13:48 |
| *** bookwar has joined #openstack-infra | 13:49 | |
| *** fhubik_afk is now known as fhubik | 13:51 | |
| *** xylan is now known as xylan_ | 13:51 | |
| *** xylan_ is now known as xylan | 13:52 | |
| *** Ala has quit IRC | 13:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix project-config post jobs https://review.openstack.org/177762 | 13:55 |
| *** mkoderer has quit IRC | 13:55 | |
| *** dboik has joined #openstack-infra | 13:55 | |
| *** vishy has quit IRC | 13:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 13:56 |
| *** mwagner_lap has joined #openstack-infra | 13:57 | |
| *** amitgandhinz has joined #openstack-infra | 13:57 | |
| *** vishy has joined #openstack-infra | 13:58 | |
| *** mpaolino has joined #openstack-infra | 13:58 | |
| *** eharney has joined #openstack-infra | 13:58 | |
| *** mkoderer has joined #openstack-infra | 13:58 | |
| *** funzo has joined #openstack-infra | 13:59 | |
| *** rossella_s has quit IRC | 13:59 | |
| *** liam__ has joined #openstack-infra | 14:01 | |
| *** tonytan4ever has joined #openstack-infra | 14:01 | |
| *** ibiris is now known as ibiris_away | 14:01 | |
| *** liam__ has quit IRC | 14:02 | |
| *** alexpilotti has quit IRC | 14:02 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through https://review.openstack.org/172284 | 14:02 |
| *** ibiris_away is now known as ibiris | 14:03 | |
| *** wznoinsk has joined #openstack-infra | 14:03 | |
| *** sergsh has joined #openstack-infra | 14:04 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:05 | |
| yolanda | hi, i need some clarification for usage of zookeeper on the tests, is that still actively used? | 14:05 |
| *** hodos|2 has joined #openstack-infra | 14:05 | |
| *** dkranz has joined #openstack-infra | 14:06 | |
| fungi | i've been working on the assumption that it is | 14:06 |
| fungi | for tooz at least? | 14:07 |
| yolanda | fungi, i wasn't aware of that, so i wanted to know more, as i had some problems with that | 14:07 |
| mordred | yeah. I mean, it's not used for most of the things, right? | 14:07 |
| *** samuelBartel has quit IRC | 14:07 | |
| mordred | tooz and maybe ceilometer uses tooz at this point for something? | 14:07 |
| *** xylan has left #openstack-infra | 14:07 | |
| fungi | yolanda: problems installing the packages, or problems using zookeeper? | 14:07 |
| yolanda | well, zookeeper not booting, looked as a permissions problem | 14:07 |
| yolanda | but i got surprised because i had no idea that we had zookeeper on the slaves | 14:08 |
| yolanda | i fixed the issue but was a surprising discover | 14:08 |
| *** fhubik is now known as fhubik_afk | 14:08 | |
| *** marcusvrn1 has quit IRC | 14:09 | |
| *** stevemar has joined #openstack-infra | 14:10 | |
| *** dkliban has joined #openstack-infra | 14:10 | |
| *** rkukura has joined #openstack-infra | 14:11 | |
| *** btully has joined #openstack-infra | 14:11 | |
| *** fhubik_afk is now known as fhubik | 14:13 | |
| *** zhiwei has quit IRC | 14:13 | |
| *** dmellado has joined #openstack-infra | 14:14 | |
| jhesketh | anteaya: against the gerrit api, it seems that the --start param was introduced after 2.8 | 14:15 |
| jhesketh | (it's in the head documentation, not sure if that means 2.9 or 2.10) | 14:15 |
| anteaya | jhesketh: ah okay thanks | 14:15 |
| *** jtriley_ has quit IRC | 14:16 | |
| *** jtriley has joined #openstack-infra | 14:17 | |
| *** mtanino has joined #openstack-infra | 14:17 | |
| anteaya | jhesketh: I can't find docs for the --start param anywhere so far | 14:18 |
| nibalizer | yolanda: mordred fungi so maybe 'apt-get install' zookeeper could move out of the slave definition and into the test that needs it? | 14:19 |
| yolanda | if it's only used in a small subset of patches, it should be goog removing from slaves | 14:19 |
| yolanda | we have that on some static jenkins slaves, it's a process running there forever, that may not be even used | 14:20 |
| mordred | nibalizer: yup | 14:20 |
| mordred | nibalizer: we have a general desire to do that for everything | 14:20 |
| nibalizer | yea so lets do that | 14:21 |
| mordred | nibalizer: so, we'd like to have very few things in the base slave definition, one slave type with caching, and install appropriate things in the tests | 14:21 |
| nibalizer | yea, swinging of the pendulum etc etc | 14:21 |
| mordred | yup | 14:21 |
| nibalizer | i like it | 14:21 |
| nibalizer | big tent needs a really pared down node | 14:21 |
| fungi | nibalizer: i'm working on that for _all_ packages | 14:21 |
| fungi | not just zookeeper | 14:21 |
| nibalizer | fungi: ++ | 14:21 |
| fungi | so yes, zookeeper can be scoped back to just specific jobs soon, along with a lot of other packages too | 14:22 |
| *** salv-orl_ has joined #openstack-infra | 14:22 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot https://review.openstack.org/177787 | 14:23 |
| *** salv-orlando has quit IRC | 14:23 | |
| *** mase_x200 has joined #openstack-infra | 14:24 | |
| *** tiswanso has quit IRC | 14:24 | |
| *** marcusvrn has joined #openstack-infra | 14:25 | |
| *** ChuckC_ has joined #openstack-infra | 14:25 | |
| *** ChuckC has quit IRC | 14:26 | |
| *** wenlock has joined #openstack-infra | 14:26 | |
| *** wenlock_ has joined #openstack-infra | 14:26 | |
| AJaeger | jaypipes, you can unhide again from mordred - it was my own error that caused the problem I run into ;( | 14:28 |
| openstackgerrit | Merged openstack-infra/project-config: Add acl rule for Mistral to allow creating branches https://review.openstack.org/177750 | 14:28 |
| jaypipes | AJaeger: :) | 14:28 |
| * jaypipes crawls out from hole | 14:28 | |
| *** michchap has quit IRC | 14:28 | |
| AJaeger | Sorry, jaypipes ! Hope you didn't hurt... | 14:28 |
| *** mattfarina has joined #openstack-infra | 14:29 | |
| * mordred hands jaypipes a towel | 14:30 | |
| *** ChuckC_ has quit IRC | 14:30 | |
| jeblair | nibalizer, yolanda: it's a constituent of the dib-nodepool infra priority effort | 14:30 |
| gordc | dtroyer: just curious, but do you know of any clients outputting json rather than typical print_dict/list formatting? | 14:30 |
| *** sputnik13 has joined #openstack-infra | 14:30 | |
| mordred | gordc: like, command line clients? | 14:30 |
| gordc | dtroyer: i see this but it seems like it was abandoned: https://bugs.launchpad.net/python-novaclient/+bug/1254351 | 14:30 |
| openstack | Launchpad bug 1254351 in python-novaclient "client should have an option to output json" [Undecided,Won't fix] - Assigned to Abhishek Chanda (abhishek-i) | 14:30 |
| yolanda | jeblair, is that the work that is driven by fungi? | 14:30 |
| gordc | mordred: yeah | 14:30 |
| mordred | gordc: python-openstackclient has an option for json output | 14:31 |
| mordred | gordc: occasionally glanceclient outputs json - but when it does, it's a bug :) | 14:31 |
| *** dhritishikhar has quit IRC | 14:31 | |
| *** dhritishikhar_ has joined #openstack-infra | 14:31 | |
| gordc | mordred: :) bugs are hidden features. | 14:31 |
| mordred | :) | 14:31 |
| gordc | mordred: thanks for pointer. i'll take a look. | 14:32 |
| fungi | yolanda: yes | 14:32 |
| *** Ala has joined #openstack-infra | 14:32 | |
| anteaya | mordred: a towel, that might the first useful thing I've ever seen you hand to anyone | 14:32 |
| yolanda | fungi, i need to take a deeper look at your job, is one of the parts that is black magic to me | 14:32 |
| *** dhritishikhar_ has quit IRC | 14:32 | |
| mordred | anteaya: well, he _was_ in a hole | 14:33 |
| anteaya | he was | 14:33 |
| *** dhritishikhar_ has joined #openstack-infra | 14:33 | |
| *** achanda has joined #openstack-infra | 14:33 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Make oslo.messaging py3 job voting https://review.openstack.org/177534 | 14:34 |
| *** tiswanso has joined #openstack-infra | 14:34 | |
| fungi | yolanda: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/experimental-workers.yaml is a preview/proof of concept. i'm presently working on making bindep solve that "base-packages-trusty" builder in a more cross-platform friendly manner and reusing it to also determine what to cache on our worker images | 14:35 |
| *** weshay has quit IRC | 14:35 | |
| *** Longgeek has quit IRC | 14:35 | |
| fungi | if things calm down today, i've got a bunch of held workers in rackspace and hpcloud representative of all our devstack-.* and corresponding bare-.* types for some final bindep testing before i tag its current state up to pypi | 14:35 |
| *** Longgeek has joined #openstack-infra | 14:36 | |
| mordred | fungi: woot | 14:36 |
| * yolanda needs to dig more about bindep | 14:37 | |
| *** tjones1 has joined #openstack-infra | 14:37 | |
| fungi | once that part works, the package installer macro can look for a "other-requirements.txt" in the top-level directory of a project's git tree, and if found use that otherwise fall back to using http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt | 14:38 |
| *** dangers_away is now known as dangers | 14:38 | |
| *** weshay has joined #openstack-infra | 14:39 | |
| fungi | and we can tackle some of the less common entries in that fallback list piecemeal, for example getting a usable other-requirements.txt into ceilometer and tooz, and then removing entries from the fallback | 14:39 |
| *** jtriley has quit IRC | 14:40 | |
| *** achanda has quit IRC | 14:41 | |
| *** dmellado has quit IRC | 14:42 | |
| *** jamesmcarthur has joined #openstack-infra | 14:42 | |
| *** saggi has quit IRC | 14:43 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Make oslo.messaging py3 job voting https://review.openstack.org/177534 | 14:46 |
| *** spzala has joined #openstack-infra | 14:48 | |
| *** zz_jgrimm is now known as jgrimm | 14:48 | |
| *** hemnafk is now known as hemna | 14:49 | |
| *** jtriley has joined #openstack-infra | 14:50 | |
| sdague | grr... pypi mirrors in rax randomly dropping connections | 14:50 |
| *** annegentle has joined #openstack-infra | 14:51 | |
| mordred | sdague: it's a 'feature' | 14:51 |
| mordred | sdague: zomg. vagrant is actually doing things on my box without dying | 14:52 |
| mordred | anteaya: ^^ you'll be amazed | 14:52 |
| sdague | heh | 14:52 |
| anteaya | mordred: I am amazed | 14:52 |
| mordred | sdague: seriously. this is the first time it has ever done anything other than a crazy cryptic error | 14:52 |
| anteaya | I never thought I would see the day | 14:52 |
| *** tkelsey has joined #openstack-infra | 14:52 | |
| *** ibiris is now known as ibiris_away | 14:53 | |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta https://review.openstack.org/177747 | 14:53 |
| anteaya | I'm impressed you were able to work through your rage and get it working | 14:53 |
| mordred | sdague: also - if you clean up docs on the ssh key parameter - it's worth noting that it's just the key contents, not the leading ssh-rsa or the trailing comment | 14:53 |
| *** samuelBartel has joined #openstack-infra | 14:53 | |
| *** salv-orl_ has quit IRC | 14:54 | |
| sdague | oh, yeh. Good point | 14:55 |
| *** jtriley_ has joined #openstack-infra | 14:56 | |
| *** ibiris_away is now known as ibiris | 14:56 | |
| greghaynes | mordred: in config-drive? | 14:57 |
| *** _nadya_ has quit IRC | 14:58 | |
| *** ildikov has quit IRC | 14:58 | |
| *** jtriley has quit IRC | 14:59 | |
| *** zul has quit IRC | 14:59 | |
| *** marcusvrn_ has joined #openstack-infra | 14:59 | |
| *** zul has joined #openstack-infra | 14:59 | |
| *** sarob has joined #openstack-infra | 14:59 | |
| *** yfried_ is now known as yfried|afk | 15:00 | |
| jhesketh | anteaya: http://review.cyanogenmod.org/Documentation/rest-api-changes.html#list-changes after the big example (or find 'start' inline) | 15:00 |
| mordred | greghaynes: aroo? | 15:00 |
| *** Hal has joined #openstack-infra | 15:01 | |
| *** asselin has joined #openstack-infra | 15:01 | |
| greghaynes | mordred: the ssh key only having the asn.1 data | 15:01 |
| mordred | greghaynes: in vagrant-devstack | 15:01 |
| *** ChuckC_ has joined #openstack-infra | 15:01 | |
| greghaynes | ah | 15:01 |
| *** Hal is now known as Guest51666 | 15:01 | |
| *** ociuhandu has joined #openstack-infra | 15:01 | |
| mordred | greghaynes: I reached the point where not having a local devstack was detrimental to me getting anything done | 15:02 |
| *** dkranz has quit IRC | 15:04 | |
| *** tiswanso has quit IRC | 15:04 | |
| fungi | AJaeger: oh, after your discovery of whichever package that was that somehow failed to be listed in the pypi json and so bandersnatch helpfully deleted it on all our mirrors, i did a full refresh on all of them overnight. should be there again now | 15:06 |
| *** fhubik has quit IRC | 15:06 | |
| *** mase_x200 has quit IRC | 15:06 | |
| *** thinrichs has joined #openstack-infra | 15:06 | |
| *** nelsnelson has joined #openstack-infra | 15:07 | |
| *** yfried|afk is now known as yfried_ | 15:07 | |
| fungi | dstufft: thanks again for fixing that yesterday | 15:07 |
| *** dkranz has joined #openstack-infra | 15:07 | |
| *** arxcruz has quit IRC | 15:08 | |
| *** davideagnello has joined #openstack-infra | 15:09 | |
| jeblair | fungi, zaro: i just spun up the .19 war on a trusty vm with our puppet and did not see the problem | 15:09 |
| jeblair | fungi, zaro: it did start | 15:09 |
| zxiiro | you folks gonna be at OpenStack summit? | 15:10 |
| fungi | zxiiro: you mean all 432 people in this channel? | 15:10 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot https://review.openstack.org/177787 | 15:10 |
| zxiiro | sure :) | 15:10 |
| zxiiro | open question | 15:10 |
| fungi | jeblair: yeah, one of zaro's theories is that there's something about the puppet manifest's routine to unpack the libraries from the war which isn't doing what it should | 15:11 |
| *** jamesmcarthur has quit IRC | 15:11 | |
| *** jamesmcarthur has joined #openstack-infra | 15:11 | |
| *** xylan_kong has joined #openstack-infra | 15:12 | |
| *** tiswanso has joined #openstack-infra | 15:12 | |
| fungi | jeblair: which might also explain why rolling it back to the war we had been using worked... it might have still failed to unpack something correctly, but what was already present had been unpacked properly from that war long-ish ago | 15:12 |
| *** annegentle has quit IRC | 15:12 | |
| *** ivar-lazzaro has joined #openstack-infra | 15:12 | |
| jeblair | fungi, zaro: hrm. i did it first with .17, then edited the manifest to install .19 | 15:12 |
| fungi | okay, so the puppet manifest is even tested working for you. mostly rules out that theory too | 15:12 |
| jeblair | so it should be _roughly_ similar... though i suppose the prod server was probably built with << .17 | 15:13 |
| jklare | AJaeger: clarkb: hi, would be great if you could recheck this when you have a minute https://review.openstack.org/#/c/176674/ (we just talked about it in our core meeting and would love to move forward) | 15:13 |
| mordred | sdague: ok - more questions - devstack-vagrant worked -so thank you | 15:13 |
| *** davideagnello has quit IRC | 15:13 | |
| mordred | sdague: is there a "good" way to bind-mount directories from my host computer into the vagrant vm? like, if I wanted a local git repo in there? | 15:14 |
| *** ildikov has joined #openstack-infra | 15:14 | |
| *** sabeen1 has joined #openstack-infra | 15:14 | |
| *** ivar-lazzaro has quit IRC | 15:14 | |
| jeblair | fungi: yeah, i'm running out of ideas other than 'futz with it in prod' :/ | 15:14 |
| *** afazekas has quit IRC | 15:15 | |
| *** ivar-lazzaro has joined #openstack-infra | 15:15 | |
| *** salv-orlando has joined #openstack-infra | 15:16 | |
| jeblair | zaro: did you give puppet a real gpg key for the contactstore? that's probably necessary, i think it parses it and loads it in. should be fine to use any valid public gpg key | 15:16 |
| sdague | yes | 15:16 |
| jeblair | zaro: (it wants ascii armored public key export) | 15:16 |
| sdague | mordred: local_openstack_tree: /home/sdague/code/openstack | 15:17 |
| sdague | in config.yaml | 15:17 |
| fungi | jeblair: zaro: maybe we can neuter the puppet manifest to not run a lucene reindex (this minor change difference in the war shouldn't require one i wouldn't think?) to make turn-around on testing and rollback a lot faster | 15:17 |
| sdague | that will mount that up to /home/vagrant/openstack | 15:17 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Bump up delorean to allow for openstack-dashboard installation https://review.openstack.org/177176 | 15:17 |
| sdague | I should probably add magic that puts that also in /home/stack, but haven't yet | 15:18 |
| jeblair | fungi: yeah, i think we could do that temporarily | 15:18 |
| mordred | sdague: so - does that expect flat structure or group/repo structure? | 15:18 |
| sdague | it's just mounting up the entire subtree | 15:18 |
| sdague | basically it can be anything you want it to be | 15:19 |
| sdague | it's a directory locally that you would like to be /home/vagrant/openstack in the guest | 15:19 |
| mordred | ok. so it's not necessarily going to tell devstack to use repos from there | 15:19 |
| *** dannywilson has joined #openstack-infra | 15:19 | |
| sdague | mordred: oh, yes, it will be | 15:19 |
| *** dizquierdo has quit IRC | 15:19 | |
| *** dguitarbite has quit IRC | 15:19 | |
| sdague | sorry, I never built that next level of logic to automatically point devstack at it | 15:20 |
| mordred | yah - so - does /home/vagrant/openstack want nova to be in /home/vagrant/openstack/openstack/nova or /home/vagrant/openstack/nova ? | 15:20 |
| jeblair | fungi: so maybe we can stage this so that it's easy to try next time it gets stuck. so we don't schedule downtime, but if downtime schedules us, we can slip it in? | 15:20 |
| mordred | sdague: wait- I'm confused | 15:20 |
| sdague | mordred: ... what are you trying to do | 15:20 |
| mordred | sdague: I actually don't need devstack to know anything about these repos | 15:20 |
| sdague | maybe I'm confused | 15:20 |
| *** rossella_s has joined #openstack-infra | 15:20 | |
| mordred | sdague: I have a shade repo, and I want to hack on it adjacent to a running devstack | 15:20 |
| mordred | I just want to make sure I don't mess something else up | 15:21 |
| sdague | yep, you can use this then | 15:21 |
| mordred | neat | 15:21 |
| fungi | jeblair: oh, good point, that can be done in a puppet dev env, so don't actually need to temporarily patch it out of master | 15:21 |
| mordred | sdague: thanks! | 15:21 |
| sdague | note... it mounts into the guest as the vagrant user | 15:21 |
| sdague | which is sort of how vagrant is designed | 15:21 |
| jeblair | fungi: ah yeah -- and if it succeeds, you'll need to disable puppet; merge prod to match the environment; enable puppet | 15:22 |
| mordred | sdague: sweet. I think this is going to make my life much easier | 15:22 |
| *** yfried_ has quit IRC | 15:22 | |
| sdague | so if you pip install -e it, I think you can then just hack from host and life is good | 15:22 |
| fungi | jeblair: right, getting that set up in my "fungi" dev env on the puppetmaster now | 15:22 |
| *** marun has joined #openstack-infra | 15:22 | |
| mordred | sdague: last question - I did a vagrant reload to get that mount to happen - and the moutn happened and was happy | 15:23 |
| jeblair | fungi: probably don't need to even merge the neuter change, only the version bump | 15:23 |
| *** dboik_ has joined #openstack-infra | 15:23 | |
| mordred | sdague: but openstack is no longer running | 15:23 |
| fungi | jeblair: i concur. that's what i'm planning for | 15:23 |
| *** scheuran has quit IRC | 15:23 | |
| mordred | sdague: I want to run something in the devstack dir, right? | 15:23 |
| *** sabeen1 has quit IRC | 15:23 | |
| *** dannywilson has quit IRC | 15:23 | |
| *** dkranz has quit IRC | 15:23 | |
| sdague | mordred: just stack.sh again | 15:23 |
| mordred | sdague: cool | 15:24 |
| sdague | reload takes the guest down / up | 15:24 |
| mordred | sdague: as the stack user, yeah? | 15:24 |
| sdague | yes | 15:24 |
| mordred | cool | 15:24 |
| mordred | look at all the things I'm learning today :) | 15:24 |
| *** dtantsur is now known as dtantsur|afk | 15:24 | |
| sdague | there are many warts in this whole system, because it's sort of a "works for me" state, so suggestions on changes to make it better for other folks are very welcome | 15:24 |
| *** dannywilson has joined #openstack-infra | 15:25 | |
| jeblair | fungi: i'm heading into more intense conferencing now; ttyl | 15:25 |
| *** thinrichs has quit IRC | 15:25 | |
| mordred | sdague: cool. well, it works well for me at the moment | 15:25 |
| fungi | hrm... jeblair how do you go about testing other modules besides openstack_project using a dev-env? | 15:25 |
| sdague | I appreciate the new eyes looking at this and asking good questions | 15:25 |
| fungi | jeblair: i'll get it worked out and let you know later. happy conferencing | 15:25 |
| mordred | sdague: with the network bridging - is it possible for me to point a REST client at the thing in the vagrant vm from my host? | 15:25 |
| jeblair | fungi: something something environemnt module path mumble? | 15:25 |
| jeblair | :) | 15:26 |
| sdague | mordred: yes | 15:26 |
| *** otter768 has joined #openstack-infra | 15:26 | |
| fungi | jeblair: thanks ;) | 15:26 |
| sdague | that's what I do, I network bridge it and specify a stable mac | 15:26 |
| *** dboik has quit IRC | 15:26 | |
| *** sabeen1 has joined #openstack-infra | 15:26 | |
| sdague | then it means it always ends up on the same IP on my home network | 15:26 |
| sdague | so it's accessible from other machines on the network as well | 15:27 |
| rcarrillocruz | yah, that or forward the port from the host to the guest | 15:27 |
| mordred | ah - so I'm simpler - there is no local network really that's usable taht way - localhost is fine | 15:27 |
| sdague | I think even without the bridge though, you should be able to hit it on the hostonly network from your host | 15:27 |
| rcarrillocruz | but i agree with sdague, bridgining is easier and faster | 15:27 |
| mordred | so I may just need to do port forwarding | 15:27 |
| *** hdd_ has quit IRC | 15:27 | |
| rcarrillocruz | *bridging* | 15:28 |
| * mordred is imagining using this on a plane too | 15:28 | |
| mordred | so expecting the vm to get an IP from an environment is probably bad | 15:28 |
| *** MaxV has quit IRC | 15:28 | |
| sdague | mordred: yeh, so for the plane case, I think just hit the host only network | 15:28 |
| mordred | cool | 15:28 |
| *** sks has quit IRC | 15:28 | |
| *** MaxV has joined #openstack-infra | 15:28 | |
| mordred | I'll figure that out next | 15:28 |
| sdague | but... I don't do it that way, so it might require some futzing | 15:29 |
| mordred | ssh-ing to the vm is not particularly onerous :) | 15:29 |
| phschwartz | Giving you guys a heads up, I found an issue with a commit to JJB that was done in the past 4 months that breaks usage of versions of Jenkins < 1.575. I am testing a fix, but so you guys know incase anyone else has an issue. | 15:29 |
| *** spredzy_ is now known as spredzy_|afk | 15:29 | |
| *** harlowja_at_home has joined #openstack-infra | 15:29 | |
| mordred | phschwartz: cool, thanks! | 15:30 |
| *** thinrichs has joined #openstack-infra | 15:30 | |
| sdague | oh, I think "use_bridge: false" | 15:30 |
| sdague | will give you what you want | 15:30 |
| sdague | someone else added that bit | 15:30 |
| *** otter768 has quit IRC | 15:30 | |
| *** adrian_otto has joined #openstack-infra | 15:31 | |
| *** MaxV has quit IRC | 15:32 | |
| *** tjones1 has quit IRC | 15:32 | |
| adrian_otto | good morning. I am working on creating a stabe/kilo branch in Gerrit for Magnum, and I am looking for the "Admin", "Projects" menu while signed in, and I'm unable to find it. Is there a different way to create branches in Gerrit? | 15:33 |
| anteaya | jhesketh: thank you | 15:34 |
| adrian_otto | I can't find the word "Admin" in any of the Gerrit menus | 15:34 |
| fungi | adrian_otto: can you link to the documentation where it says there's a menu named "admin"? you're the second person i've seen ask about that | 15:34 |
| adrian_otto | fungi: I actually found it in the drivers docs… here: http://docs.openstack.org/infra/manual/drivers.html in the "End of Release" section | 15:35 |
| *** nelsnelson has quit IRC | 15:35 | |
| *** rkukura has quit IRC | 15:36 | |
| fungi | adrian_otto: assuming your permissions are correct, you select projects->list at the top of the webui, then find and click on the repo you want by using the filter there, then go to "branches" linked at the top of the page | 15:36 |
| adrian_otto | Yes, I get a list of branches | 15:36 |
| fungi | adrian_otto: i'll try to update the drivers guide for better clarity | 15:36 |
| adrian_otto | but I don't see a way to add one | 15:36 |
| *** dboik_ has quit IRC | 15:36 | |
| adrian_otto | I got to here: https://review.openstack.org/#/admin/projects/openstack/python-magnumclient,branches | 15:36 |
| *** dboik has joined #openstack-infra | 15:37 | |
| adrian_otto | but for me, no action buttons are present on that page | 15:37 |
| fungi | adrian_otto: if your account is a member of a group with "create" privileges in the refs/heads/* section of the acl for that repo, then there should be a "create branch" button and some fields to enter a branch name and initial revision | 15:37 |
| *** dkranz has joined #openstack-infra | 15:37 | |
| *** sputnik13 has quit IRC | 15:37 | |
| adrian_otto | https://review.openstack.org/#/admin/projects/openstack/python-magnumclient,access | 15:38 |
| adrian_otto | I am a member of magnum-core, but maybe I need to extend the ACL? | 15:38 |
| *** armax has joined #openstack-infra | 15:39 | |
| fungi | adrian_otto: yeah, looks like you don't have any groups delegated for branch creation. add a line like "create = group magnum-milestone" to your acl file in the openstack-infra/project-config repo | 15:39 |
| mordred | sdague: WOOT | 15:39 |
| mordred | sdague: works like a charm | 15:39 |
| fungi | adrian_otto: it'll need to be in alpha order between the abandon and label lines in the refs/heads/* section | 15:39 |
| adrian_otto | fungi: ok, will give that a try. | 15:39 |
| fungi | adrian_otto: also, only gerrit admins can delete branches, so you should probably restrict create to a subset of your core team to avoid accidental branch creation | 15:40 |
| fungi | adrian_otto: which is why i suggested your magnum-milestone group there | 15:40 |
| *** mpavone has quit IRC | 15:40 | |
| adrian_otto | fungi: there is a magnum-milestone group, so I can use that I suppose | 15:40 |
| adrian_otto | right | 15:40 |
| *** Swami has joined #openstack-infra | 15:41 | |
| *** vlaza has quit IRC | 15:41 | |
| *** flashgordon is now known as jogo | 15:42 | |
| *** jamesmcarthur has quit IRC | 15:42 | |
| *** jamesmcarthur has joined #openstack-infra | 15:43 | |
| *** jroll has quit IRC | 15:44 | |
| ttx | jeblair: random Depends-On feature: make sure a backport patch can't merge in a branch before the master patch gets in. | 15:44 |
| ttx | would remove a lot of release / security manual checking | 15:45 |
| fungi | i think that would require extending the depends-on syntax to be optionally project and branch specific | 15:45 |
| fungi | but we've seen other potential use cases for that as well | 15:46 |
| *** hasharMeeting has quit IRC | 15:46 | |
| *** hashar has joined #openstack-infra | 15:46 | |
| fungi | for example, depending on a master branch patch where the stable branch patch of the same change-id has been abandoned | 15:46 |
| *** jroll has joined #openstack-infra | 15:46 | |
| *** sdake has joined #openstack-infra | 15:46 | |
| *** hashar is now known as hasharAway | 15:47 | |
| fungi | or when a change to another project using the same change-id has been abandoned or isn't a legitimate dependency of the change proposed | 15:47 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: WiP: Add Neutron/Nova Floating IP support https://review.openstack.org/177036 | 15:47 |
| *** hasharAway is now known as hasharAway- | 15:47 | |
| openstackgerrit | Adrian Otto proposed openstack-infra/project-config: Added branch creation to Magnum ACL https://review.openstack.org/177833 | 15:47 |
| adrian_otto | fungi: okay, proposed that edit | 15:47 |
| fungi | adrian_otto: yep, lgtm | 15:47 |
| *** yamamoto has quit IRC | 15:48 | |
| *** sabeen1 has left #openstack-infra | 15:48 | |
| *** nelsnelson has joined #openstack-infra | 15:49 | |
| adrian_otto | thanks fungi for the guidance | 15:49 |
| *** sabeen1 has joined #openstack-infra | 15:49 | |
| fungi | adrian_otto: any time! | 15:49 |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| *** kgiusti has left #openstack-infra | 15:51 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot https://review.openstack.org/177787 | 15:53 |
| *** jroll has quit IRC | 15:53 | |
| *** jroll has joined #openstack-infra | 15:53 | |
| *** salv-orlando has quit IRC | 15:53 | |
| *** e0ne is now known as e0ne_ | 15:53 | |
| *** e0ne_ is now known as e0ne | 15:54 | |
| *** tjones1 has joined #openstack-infra | 15:54 | |
| zaro | morning | 15:55 |
| anteaya | morning zaro | 15:55 |
| *** gyee has joined #openstack-infra | 15:55 | |
| zaro | fungi, jeblair : upate to .19 does not require a reindex. | 15:56 |
| zaro | fungi: when are you thinking about trying the gerrit update? | 15:56 |
| *** Sukhdev has joined #openstack-infra | 15:56 | |
| fungi | zaro: the next time stream-events hangs | 15:56 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Correct output path for infra-site index https://review.openstack.org/177839 | 15:58 |
| fungi | AJaeger: ^ see http://logs.openstack.org/b2/b2388798925910fa061dc3341739fbd7e8d8b220/post/publish-infra-docs-index/ca0e93a/console.html for details | 15:58 |
| fungi | mordred: ^ | 15:58 |
| *** asselin has quit IRC | 15:58 | |
| zaro | fungi: ok. i guessing i'm mising that key jeblair was refering on my VM to be able to get contact store to work. I'm not sure how to create/get that key though. | 15:58 |
| *** AJaeger_ has joined #openstack-infra | 16:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix publish-infra-docs-index https://review.openstack.org/177842 | 16:00 |
| *** afazekas has joined #openstack-infra | 16:00 | |
| zaro | pelix: thanks for the rebase to 75514. i ran my tests again but am not confident yet. | 16:00 |
| AJaeger_ | fungi, the publish job was still wrong, I expect that change 177842 fixes it finally | 16:00 |
| pleia2 | good morning | 16:01 |
| *** david-ly_ is now known as david-lyle | 16:01 | |
| fungi | AJaeger_: that looks identical (minus your typo) to teh change i was pointing out to you | 16:02 |
| *** MaxV has joined #openstack-infra | 16:02 | |
| zaro | pelix: i got the same error I referenced in PS 19 on one of my test runs. | 16:02 |
| *** nelsnelson has quit IRC | 16:03 | |
| fungi | zaro: install gnupg and then run `gpg --gen-key` and follow the prompts (picking defaults if you want) and after that's done do `gpg --armor --output some_file --export whatever_the_key_id_is` and then the contents of some_file are the key you want. alternatively just borrow someone's published key | 16:03 |
| AJaeger_ | fungi: just came home and checked why it was not working - should have read backscroll... | 16:03 |
| *** alexpilotti has joined #openstack-infra | 16:04 | |
| AJaeger_ | Ok, let me review yours ;) | 16:04 |
| *** nelsnelson has joined #openstack-infra | 16:04 | |
| zaro | anybody know why zuul test would fail when run with 'tox -epy27' but not fail when run with testr inside of the virtualenv? | 16:04 |
| fungi | zaro: mine is linked at http://security.openstack.org/#how-to-report-security-issues-to-openstack if you want to just use it as a test | 16:04 |
| *** sergsh has quit IRC | 16:05 | |
| *** jlanoux has quit IRC | 16:05 | |
| zaro | fungi: thnaks | 16:05 |
| *** ildikov has quit IRC | 16:06 | |
| *** dkranz has quit IRC | 16:06 | |
| *** Guest51666 has quit IRC | 16:06 | |
| AJaeger_ | fungi, ah, you beat my by two minutes ;) | 16:06 |
| zaro | the zuul test failure is "process-returncode [ multipart" with "returncode 1420" | 16:07 |
| fungi | AJaeger_: i guess i'm just fast like that | 16:07 |
| fungi | you usually beat me to these with a wide margin | 16:07 |
| *** MaxV has quit IRC | 16:07 | |
| *** dmellado has joined #openstack-infra | 16:08 | |
| AJaeger_ | fungi: you're really fast - and if we start at the same time, I guess you always beat me ;) | 16:08 |
| AJaeger_ | fungi: you're often firefighting several fires at once ;) | 16:08 |
| fungi | AJaeger_: anyway, it's just as well. it was my mistake after all ;) | 16:08 |
| pelix | zaro: I was thinking it might be worth investing in writing some tests to exercise the parallel section of update_job either by mocking more of the python-jenkins api and letting it be called via threads or by creating a simple HTTP server from python and handling testing requests that way to shake out socket exceptions? | 16:08 |
| AJaeger_ | fungi: we're getting it fixed ;) | 16:09 |
| *** salv-orlando has joined #openstack-infra | 16:09 | |
| *** ChuckC_ has quit IRC | 16:09 | |
| *** ildikov has joined #openstack-infra | 16:09 | |
| *** mpaolino has quit IRC | 16:09 | |
| *** salv-orl_ has joined #openstack-infra | 16:10 | |
| *** devvesa has quit IRC | 16:10 | |
| *** rbradfor has joined #openstack-infra | 16:11 | |
| *** thinrichs has left #openstack-infra | 16:11 | |
| anteaya | morning pleia2 | 16:11 |
| *** SpamapS has quit IRC | 16:12 | |
| *** SpamapS has joined #openstack-infra | 16:12 | |
| *** tqtran has joined #openstack-infra | 16:12 | |
| *** salv-orlando has quit IRC | 16:13 | |
| *** AJaeger_ has quit IRC | 16:14 | |
| *** ildikov has quit IRC | 16:14 | |
| *** Sukhdev has quit IRC | 16:15 | |
| *** jcoufal has joined #openstack-infra | 16:15 | |
| *** sabeen2 has joined #openstack-infra | 16:15 | |
| *** sabeen1 has quit IRC | 16:15 | |
| *** unicell1 has quit IRC | 16:15 | |
| *** zul has quit IRC | 16:16 | |
| *** MaxV has joined #openstack-infra | 16:16 | |
| sdague | so is dfw in rax having issues this morning? | 16:17 |
| *** dkliban has left #openstack-infra | 16:17 | |
| sdague | http://logs.openstack.org/40/177840/1/check/check-tempest-dsvm-neutron-full/3b31ad2//logs/devstacklog.txt.gz#_2015-04-27_16_06_51_079 is the second time I've had a patch this morning get killed there | 16:17 |
| *** dkranz has joined #openstack-infra | 16:19 | |
| sdague | fungi: oh, wait, those are *our* mirrors | 16:19 |
| sdague | any idea why our mirrors are broken? | 16:19 |
| anteaya | sdague: your prior message was the only complaint I have seen on our mirrors | 16:20 |
| anteaya | 2015-04-27T14:50:32 <sdague> grr... pypi mirrors in rax randomly dropping connections | 16:20 |
| samueldmq | hi, I want to use the job defined here ( http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1264 ) | 16:20 |
| samueldmq | the only thing I need to do is to include it on the experimental part of the project I want to run it agains at zuul/layout, right ? | 16:21 |
| sdague | so, I've had at least 3 test fails this morning from - pypi.DFW.openstack.org disconnecting me | 16:21 |
| *** amotoki_ has joined #openstack-infra | 16:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Added branch creation to Magnum ACL https://review.openstack.org/177833 | 16:21 |
| zaro | pelix: i think the latter might be more useful that we we can simply just add update and delete tests as well. | 16:21 |
| *** amotoki has quit IRC | 16:21 | |
| *** jistr has quit IRC | 16:21 | |
| *** amotoki_ is now known as amotoki | 16:21 | |
| zaro | pelix: something like this? https://github.com/knalli/jenkins-api-mock | 16:22 |
| *** armax has quit IRC | 16:22 | |
| anteaya | samueldmq: it is on the tempest repo right now: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n3110 | 16:22 |
| anteaya | samueldmq: which repo do you want it on? | 16:23 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot https://review.openstack.org/177787 | 16:23 |
| samueldmq | anteaya, devstack | 16:23 |
| fungi | sdague: load average looks high-ish there http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2488&rra_id=all | 16:23 |
| zaro | pelix: regarding that change, '--workers' param is a noop for 'test' cmd right? | 16:23 |
| zaro | pelix: should it be a sub-menu of 'test'? | 16:23 |
| samueldmq | anteaya, so that we'll make changes on devstack, and then run it against that | 16:23 |
| *** armax has joined #openstack-infra | 16:23 | |
| *** viktors is now known as viktors|afk | 16:23 | |
| samueldmq | anteaya, using 'check experimental' | 16:24 |
| anteaya | samueldmq: well it might be a good idea to ask mtreinish if he wants that | 16:24 |
| anteaya | samueldmq: and then yes, adding the job name to the tempest experimental pipeline is the code to make it happen | 16:24 |
| anteaya | sorry devstack experimental pipeline | 16:24 |
| samueldmq | anteaya, right, but for now we just want tot test | 16:25 |
| samueldmq | anteaya, yes, experimental pipeline | 16:25 |
| adrian_otto | hey fungi, that patch merged https://review.openstack.org/177833 but even after logging out of Gerrit, and logging back in, I'm still not seeing a way to add a branch on the https://review.openstack.org/#/admin/projects/openstack/magnum,branches page. Is there more that needs to be done? | 16:25 |
| fungi | sdague: oh, exciting. bandersnatch seems to be running there in a loop again. i wonder if the way i file locked during my refresh last night didn't prevent it from conflicting somehow. i'll disable the cron job and get one loop going | 16:25 |
| fungi | adrian_otto: it takes about 30 minutes or so for that to get applied | 16:26 |
| adrian_otto | oh, thanks. I'll check back on that later then. | 16:26 |
| sdague | yeh, I've found fails 5 and 6 on that | 16:26 |
| fungi | sdague: just dfw? it's doing it on all four mirrors from what i can see | 16:26 |
| *** ociuhandu has quit IRC | 16:26 | |
| sdague | so, I only seem to notice it in DFW | 16:26 |
| fungi | maybe dfw is just on an especially unhappy compute node | 16:27 |
| *** liam__ has joined #openstack-infra | 16:27 | |
| *** mpaolino has joined #openstack-infra | 16:27 | |
| sdague | fungi: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVhZFRpbWVvdXRFcnJvcjogSFRUUENvbm5lY3Rpb25Qb29sXCIgQU5EIHRhZ3M6Y29uc29sZSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzMDE1MjA2Njc3M30= | 16:27 |
| sdague | yeh, it looks like DFW is the only one that we're actually failing on | 16:28 |
| sdague | those are basically all job fails | 16:28 |
| * anteaya is going to get a walk in | 16:28 | |
| *** hyakuhei has joined #openstack-infra | 16:29 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: WiP: Add Neutron/Nova Floating IP support https://review.openstack.org/177036 | 16:30 |
| *** dboik_ has joined #openstack-infra | 16:30 | |
| *** dizquierdo has joined #openstack-infra | 16:30 | |
| *** markvan has quit IRC | 16:32 | |
| *** sabeen1 has joined #openstack-infra | 16:32 | |
| *** tiswanso has quit IRC | 16:32 | |
| *** markvan has joined #openstack-infra | 16:32 | |
| *** tiswanso_ has joined #openstack-infra | 16:33 | |
| fungi | okay, puppet agent disabled, crontab edited and current bandersnatch runs killed on all our pypi mirrors, then fresh resyncs started in a root screen session on each of them | 16:33 |
| fungi | in a few hours i'll check back in on them to see whether they've finished | 16:34 |
| *** tonytan4ever has quit IRC | 16:34 | |
| fungi | and reenable cron/puppet if so | 16:34 |
| *** ociuhandu has joined #openstack-infra | 16:34 | |
| *** liam__ has quit IRC | 16:34 | |
| *** dboik has quit IRC | 16:34 | |
| *** sabeen2 has quit IRC | 16:35 | |
| *** dhritishikhar_ has quit IRC | 16:35 | |
| *** gordc is now known as gordc_afk | 16:35 | |
| *** afazekas has quit IRC | 16:35 | |
| *** dhritishikhar_ has joined #openstack-infra | 16:36 | |
| *** shashankhegde has joined #openstack-infra | 16:36 | |
| *** afazekas has joined #openstack-infra | 16:36 | |
| *** ashleighfarnham has joined #openstack-infra | 16:36 | |
| openstackgerrit | Tim Hinrichs proposed openstack-infra/project-config: ACL changes for python-congressclient https://review.openstack.org/177864 | 16:37 |
| openstackgerrit | Tim Hinrichs proposed openstack-infra/project-config: ACL changes for python-congressclient https://review.openstack.org/177864 | 16:38 |
| *** hyakuhei has quit IRC | 16:40 | |
| *** ashleighfarnham has quit IRC | 16:41 | |
| clarkb | fungi what prompted the refresh last night? | 16:41 |
| *** ssam2 has quit IRC | 16:42 | |
| openstackgerrit | Tim Hinrichs proposed openstack-infra/project-config: ACL changes for python-congressclient https://review.openstack.org/177864 | 16:43 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/project-config: Add experimental Identity v3 jobs to devstack https://review.openstack.org/177867 | 16:43 |
| *** hyakuhei has joined #openstack-infra | 16:43 | |
| samueldmq | morganfainberg, experimental devstack identity v3 jobs for devstack ^ | 16:44 |
| samueldmq | dolphm, cc ^ | 16:44 |
| fungi | clarkb: dstufft thinks the glusterfs backend for pypi had a fit, but anyway we noticed at least one package bandersnatch deleted from all our mirrors for not being listed in the pypi json. he corrected the problem at the pypi side but that's not covered by a new serial so bandersnatch continued to be out of the loop | 16:44 |
| dolphm | samueldmq: \0/ | 16:44 |
| *** MaxV has quit IRC | 16:44 | |
| samueldmq | dolphm, well, that's andreaf good job ! | 16:45 |
| dstufft | I'm not a big fun of glusterfs | 16:45 |
| dstufft | fan* | 16:45 |
| fungi | i can see why | 16:45 |
| *** ashleighfarnham has joined #openstack-infra | 16:45 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Move server outside of puppetmaster class https://review.openstack.org/172485 | 16:45 |
| samueldmq | dolphm, the goal now is to get morganfainberg's changes on devstack (disabling v2 for tests) and run 'check experimental' on them o/ | 16:45 |
| dstufft | sometimes our web nodes randomly lose connectivity, where attempting to CD or anything into the directory returns a not exist | 16:45 |
| dstufft | and the only way to fix it is to restart the glusterfs netfs service (sometimes... once I had to restart the whole server) | 16:46 |
| morganfainberg | samueldmq: thanks | 16:46 |
| morganfainberg | samueldmq: but there is likey more needed than just that: ds-gate updates among other things | 16:46 |
| sdague | fungi: where should I file a bug here to track the ER query? | 16:47 |
| mtreinish | sdague: https://bugs.launchpad.net/openstack-gate | 16:48 |
| samueldmq | morganfainberg, I thought it was only that, I asked anteaya and she confirmed | 16:48 |
| samueldmq | morganfainberg, the job template is already created .. so isn't jsut that ? | 16:48 |
| morganfainberg | samueldmq: oh hm... i am not sure that will do what you think it does | 16:49 |
| *** armax has quit IRC | 16:49 | |
| *** derekh has quit IRC | 16:49 | |
| samueldmq | morganfainberg, that makes tempest to use v3 tokens to its tests | 16:49 |
| *** hdd has joined #openstack-infra | 16:50 | |
| samueldmq | morganfainberg, so that we can make the devstack changes (disabling v2 for eg) and running 'check experimental' | 16:50 |
| samueldmq | morganfainberg, that's what I think | 16:50 |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 16:50 |
| *** _nadya_ has joined #openstack-infra | 16:50 | |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Add query for pypi mirrors disconnecting runs https://review.openstack.org/177871 | 16:50 |
| sdague | mtreinish: thanks | 16:50 |
| morganfainberg | samueldmq: that only changes tempest to use v3 | 16:50 |
| morganfainberg | samueldmq: that doesn't actually do much else. | 16:50 |
| morganfainberg | samueldmq: a bunch of other things will need to land | 16:51 |
| sdague | that's 144 fails in the last 7 days, so that should be position 2 in elastic recheck | 16:51 |
| *** Sukhdev has joined #openstack-infra | 16:51 | |
| samueldmq | morganfainberg, yeah, and after that, we just need to change devstack, no ? | 16:51 |
| morganfainberg | samueldmq: possibly more | 16:52 |
| morganfainberg | we might want an extra DS-GATE toggle to disable v2 | 16:52 |
| morganfainberg | but this is a good enough start | 16:52 |
| fungi | sdague: all dfw? if so, then this is likely unrelated to the bandersnatch refresh from last night and more likely that there's something wrong in dfw, perhaps with the compute node running that instance | 16:52 |
| *** afazekas has quit IRC | 16:52 | |
| samueldmq | morganfainberg, yeah, I am planning to use this just to test the devstakc changes | 16:52 |
| *** deepakcs has joined #openstack-infra | 16:53 | |
| *** unicell has joined #openstack-infra | 16:53 | |
| sdague | fungi: from scrolling through that list it all looks like dfw | 16:53 |
| sdague | 144 fails | 16:53 |
| sdague | maybe it's 142 fails and 2 succeeds actually on that query | 16:53 |
| sdague | but everything is dfw | 16:53 |
| samueldmq | morganfainberg, ++ for a new devstack gate with v2 disabled | 16:54 |
| sdague | fungi: https://bugs.launchpad.net/openstack-gate/+bug/1449136 query, logstash url, and example fail fail in there | 16:54 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 16:54 |
| fungi | sdague: yeah, this seems like maybe network issues, perhaps at the hypervisor level (we've seen this before when a neighbor on the same compute node is flooding/flooded by network traffic beyond what the physical interfaces can accommodate) | 16:54 |
| *** unicell has quit IRC | 16:54 | |
| *** unicell has joined #openstack-infra | 16:54 | |
| cinerama | pleia2: do we have a list handy of what repositories the translate jobs run on? i want to create a list of what locales we're currently using | 16:54 |
| pleia2 | cinerama: I believe it's in transifex, let's see... | 16:55 |
| *** unicell has quit IRC | 16:55 | |
| *** unicell has joined #openstack-infra | 16:55 | |
| pleia2 | cinerama: https://www.transifex.com/organization/openstack | 16:55 |
| sdague | anyway, I need to walk away for a bit | 16:55 |
| cinerama | pleia2: cool thanks | 16:56 |
| pleia2 | cinerama: currently, I believe when we need to add a new project or language, it's all done via the transifex interface (new projects are a customer support ticket) | 16:56 |
| pleia2 | so sadly no spiffy config file to pull from | 16:56 |
| cinerama | pleia2: yeah we're going to need to add the languages we use manually on zanata for now. i also need to check whether the locale names we're using match up with what zanata has | 16:57 |
| pleia2 | cinerama: ah, good point | 16:57 |
| *** vhoward has joined #openstack-infra | 16:58 | |
| *** erlon has joined #openstack-infra | 16:58 | |
| cinerama | pleia2: we can create a mapping in the zanata.xml we're autogenerating from zanata name to other name so our files in the repo will still have the same location | 16:59 |
| pleia2 | cinerama: good to know! | 17:00 |
| cinerama | pleia2: i was thinking i will just grab all the current locales and have that section as boilerplate in the xml | 17:00 |
| * pleia2 nods | 17:00 | |
| pleia2 | I don't actually know where the "73 languages" are listed in transifex | 17:00 |
| *** afazekas has joined #openstack-infra | 17:00 | |
| pleia2 | ah, on each project page you can "show all languages" | 17:01 |
| *** harlowja_at_home has quit IRC | 17:01 | |
| *** dhritishikhar_ has quit IRC | 17:01 | |
| *** dhritishikhar_ has joined #openstack-infra | 17:01 | |
| *** patrickeast has joined #openstack-infra | 17:02 | |
| *** bswartz has quit IRC | 17:03 | |
| *** alexpilotti has quit IRC | 17:03 | |
| *** Ala has quit IRC | 17:04 | |
| *** davideagnello has joined #openstack-infra | 17:05 | |
| *** Krinkle|detached is now known as Krinkle | 17:06 | |
| *** yamahata has joined #openstack-infra | 17:07 | |
| *** dalgaaf has quit IRC | 17:07 | |
| *** wuhg has quit IRC | 17:09 | |
| fungi | sdague: i've opened a fanatical support ticket for pypi.dfw tcp read timeouts. hopefully that turns something up. logs on the instance itself don't show any obvious problems | 17:10 |
| sdague | fungi: coolio, thanks | 17:10 |
| fungi | thanks for spotting the issue | 17:10 |
| clarkb | fungi: sdague is it reproduceable from your local end? | 17:10 |
| *** tnovacik has quit IRC | 17:11 | |
| sdague | clarkb: *shrug*? | 17:11 |
| sdague | the bug has what I have, a logstash signature, over 140 failed jobs, and an example to what a fail looks like | 17:11 |
| fungi | clarkb: it seems to be needle-in-a-haystack occurrences. e.g. one random pip download out of the many that happen during any given job | 17:11 |
| sdague | yeh, it's not the same package over and over again | 17:11 |
| fungi | so we'd need to do some heavy looping of requests to it to reproduce manually, i expect | 17:12 |
| clarkb | SpamapS: had a question about https://git.openstack.org/cgit/openstack-infra/shade/tree/shade/__init__.py#n165 late friday, not sure you saw it. Any idea why line 165 uses the __get__ lookup of func but 169 does not? is that a bug? | 17:12 |
| clarkb | fungi: sdague gotcha | 17:12 |
| clarkb | fungi: sdague one thing I would check is the eth0 bw useage | 17:13 |
| clarkb | fungi: sdague the git haproxy nodes saw tcp resets when they hit their bw max | 17:13 |
| clarkb | I think its 400Mbps for the 8GB Performance flavor | 17:13 |
| tqtran | https://review.openstack.org/#/c/176565/ following procedure to create a new repo but its not passing. Can someone guide me in the right direction? | 17:13 |
| *** ociuhandu has quit IRC | 17:13 | |
| sdague | clarkb: it's pypi mirrors | 17:13 |
| clarkb | sdague: I know, I am saying it could be bw limiting | 17:14 |
| mordred | one could imagine those hitting BW caps | 17:14 |
| sdague | ok | 17:14 |
| marun | Apologies for having asked this before, but what's the timeline on storyboard usurping launchpad? | 17:14 |
| *** ociuhandu has joined #openstack-infra | 17:14 | |
| sdague | clarkb: is that our side, or theirs? | 17:14 |
| clarkb | sdague: rax does the limiting for us | 17:14 |
| clarkb | sdague: and it results in really weird tcp resets where our side doesn't send it | 17:14 |
| marun | We're looking at some process changes in Neutron around feature requests and I'm wondering if we should be considering the process storyboard will impose. | 17:14 |
| mordred | marun: unknown - there is a larger question floating at the moment as to whether being in teh business of writing storyboard is the business we shoudl be in | 17:14 |
| clarkb | tqtran: http://logs.openstack.org/65/176565/3/check/gate-project-config-layout/241fdcf/console.html#_2015-04-24_18_26_59_818 is the errors, looks like you added a few jobs to the zuul layout that are not defined in jenkins job builder | 17:15 |
| sdague | clarkb: ok | 17:15 |
| dougwig | anteaya: (or anyone) - sanity check, is it correct to assume that using stackforge to implement a third-party CI is not allowed? | 17:15 |
| asselin_ | dougwig, correct | 17:15 |
| mordred | marun: I supposed I should say "I threw a wrench in the works by raising that question" - in any case - I do not believe there is any chance we'll be off of launchpad before tokyo | 17:15 |
| marun | dougwig: if you have to ask... ;) | 17:15 |
| sdague | ok, stepping away for a bit, hopefully someone can look into that, because I'm recheck grinding a tempest patch series because of it | 17:16 |
| marun | mordred: ok, thanks for the info | 17:16 |
| asselin_ | dougwig, we're putting it here: openstack-infra/puppet-openstackci | 17:16 |
| mordred | marun: sure nuff! I'm pretty sure it'll be an important topic for discussion in vancouver | 17:16 |
| clarkb | sdague: looking at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2497&rra_id=all I don't think it is BW rate limiting | 17:16 |
| marun | mordred: ++ for being wary of scope creep | 17:16 |
| clarkb | sdague: we are no where near the 400Mbps I expect to get throttled at | 17:16 |
| *** shashankhegde has quit IRC | 17:16 | |
| asselin_ | dougwig, http://specs.openstack.org/openstack-infra/infra-specs/specs/openstackci.html | 17:16 |
| dougwig | asselin_: i'm more noodling if there's some hybrid possible where one group runs a common jenkins/zuul/nodepool, since maintaining that is where all the failures seem to occur. overlays and virtualization mean that i don't need anything onsite. | 17:16 |
| *** EmilienM is now known as EmilienM|afk | 17:17 | |
| dougwig | asselin_: as opposed to us all running the same scripts. | 17:17 |
| fungi | clarkb: also it looks like socket timeouts, not reset by peer | 17:17 |
| clarkb | fungi: gotcha | 17:17 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Reduce number of services running in largeops-src https://review.openstack.org/177885 | 17:17 |
| *** afazekas has quit IRC | 17:18 | |
| fungi | a friend just ran into https://tickets.puppetlabs.com/browse/PUP-2931 | 17:18 |
| fungi | looks super fun | 17:18 |
| asselin_ | dougwig, that's an interesting idea. main issue I see is regarding trust/security... | 17:18 |
| clarkb | dougwig: I can imagine that convincing corporate network security to allow ssh into wherever your slaves are will be near impossible | 17:18 |
| clarkb | asselin_: yup thats where I expect to see the problems too | 17:18 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/elastic-recheck: Update query for bug 1403284 https://review.openstack.org/177886 | 17:19 |
| openstack | bug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] https://launchpad.net/bugs/1403284 - Assigned to Joe Gordon (jogo) | 17:19 |
| *** xyang1 has joined #openstack-infra | 17:19 | |
| *** achanda has joined #openstack-infra | 17:20 | |
| dougwig | i got around that by having my slaves the same as infra: single-use devstack nodes. on AWS, no less, and they use standard hooks to spin up a virtual appliance at the same place (which could be spun up across the internet somewhere via a hook). it'd literally be 100 lines of yaml/script in a stackforge project to replicate, with no security concerns that I | 17:21 |
| dougwig | don't already have. note that I am NOT suggesting community resources go to companies here, just exploring if we can maybe get some common maintenance around the parts not relevant to the test at hand. | 17:21 |
| *** sputnik13 has joined #openstack-infra | 17:21 | |
| dhellmann | fungi, clarkb, anteaya: can I get you to look at https://review.openstack.org/#/c/177885/1 for us? problems with that job are blocking us from merging an unrelated change in oslo.messaging that we need for the stable release | 17:21 |
| clarkb | dougwig: most third party CI requires special hardware local to company right? | 17:21 |
| *** e0ne has quit IRC | 17:21 | |
| clarkb | dougwig: in your example is there anything preventing you from just testing upstream? (possibly non open source code?) | 17:21 |
| dougwig | clarkb: i'd imagine that in neutron land, overlay networks and virtualization can get around that. in others, a DMZ piece of hardware. | 17:22 |
| dougwig | clarkb: do you mean me testing master vanilla? no. do you mean me testing in the same environment as master? no. not sure i'm parsing that properly. | 17:22 |
| asselin_ | dougwig, what are you testing? | 17:23 |
| clarkb | dougwig: I mean if you are just going to use virtual machines anyway (eg no hardware dependency) why not run your tests with our nodepool/jenkins/zuul | 17:23 |
| clarkb | dougwig: since we already provide this service you are looking for | 17:23 |
| dougwig | clarkb: that was my opening question: "sanity check, is it correct to assume that using stackforge to implement a third-party CI is not allowed?" :) | 17:23 |
| *** sergmelikyan has joined #openstack-infra | 17:23 | |
| clarkb | dougwig: I missed that | 17:23 |
| clarkb | dougwig: glusterfs and ceph and drbd for cinder are all doing it | 17:24 |
| clarkb | though ceph isn't in stackforge thats in tree iirc | 17:24 |
| dhellmann | fungi: thanks | 17:24 |
| dougwig | asselin_: a10 appliance with lbaas is what we're testing, to answer your question. | 17:24 |
| clarkb | but glusterfs and drbd have devstack plugins in stackforge that test their thing against openstack upstream | 17:24 |
| clarkb | dougwig: the key there is they are all open source | 17:24 |
| sergmelikyan | Hi folks, what is timeout to sync our mirrors with PyPI? We released new version of YAQL library in order to fix critical issue but it is already 3h are not found in gate: https://review.openstack.org/177811 | 17:24 |
| fungi | dougwig: your question was unclear to me too... totally depends on what you mean by "using stackforge" and "implement a third-party ci | 17:24 |
| clarkb | and don't rely on special hardware | 17:24 |
| *** mpaolino has quit IRC | 17:24 | |
| *** deepakcs has quit IRC | 17:24 | |
| *** jamesmcarthur has quit IRC | 17:24 | |
| dougwig | ahh, yes, i meant to explore if closed-source third-party can somehow figure out a way to not replicate all the parts. | 17:25 |
| fungi | dougwig: how are you attaching your a10 appliance to amazon ec2 instances? | 17:25 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/elastic-recheck: Make the query for bug 1403284 more generic https://review.openstack.org/177888 | 17:26 |
| openstack | bug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] https://launchpad.net/bugs/1403284 - Assigned to Joe Gordon (jogo) | 17:26 |
| *** salv-orlando has joined #openstack-infra | 17:26 | |
| dougwig | fungi: it just needs an ip/user/password to connect anywhere. today, i'm spinning up a soft appliance in parallel with the devstack node. it doesn't need hardware. | 17:26 |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: add what-broke.py analayzer https://review.openstack.org/177889 | 17:27 |
| *** otter768 has joined #openstack-infra | 17:27 | |
| *** salv-orl_ has quit IRC | 17:27 | |
| mriedem | sdague: ^ backport to stable/juno | 17:28 |
| fungi | dougwig: mostly trying to figure out if you're describing something like https://git.openstack.org/cgit/stackforge/sahara-ci-config/tree/ (i.e. hosting within stackforge the scripts and configuration which make up your ci) | 17:28 |
| fungi | dougwig: or whether you're talking about having a third-party ci report on changes proposed to a repo in stackforge | 17:28 |
| SpamapS | clarkb: regarding the cache_on_arguments thing, IIRC, it has to do with binding.. I didn't take notes when I was making it work unfortunately. | 17:28 |
| *** dhritishikhar_ has quit IRC | 17:29 | |
| *** dhritishikhar_ has joined #openstack-infra | 17:29 | |
| *** mpaolino has joined #openstack-infra | 17:29 | |
| clarkb | SpamapS: right I htink you want the __get__ lookup to make sure you always get the correct func regardless of inheritance, so I am wondering why invalidate doesn't do the same | 17:29 |
| *** marun has quit IRC | 17:30 | |
| fungi | dougwig: so i have no idea still what you mean by "find a way not to replicate all the parts" | 17:30 |
| dougwig | fungi: actually, i'm wondering if it'd provide stability gains if third-party jobs could use a common jenkins/zuul/nodepool/devstack-gate, with hooks to connect in their stuff, rather than everyone running all of that stuff. | 17:30 |
| SpamapS | clarkb: it has something to do with the fact that invalidate happens after binding. | 17:30 |
| *** yfried_ has joined #openstack-infra | 17:30 | |
| fungi | dougwig: oh, so you mean multiple third parties connecting their workers into a centrally-run ci system? | 17:31 |
| SpamapS | clarkb: morganfainberg pointed out that we also should memorize the wrap in invalidate since we already have func bound.. would save re-wrapping inside dogpile.cache.cache_on_arguments every time invalidate gets called | 17:31 |
| *** otter768 has quit IRC | 17:31 | |
| dougwig | fungi: right. instead of project-config, third-party-config, or whatnot. | 17:31 |
| clarkb | SpamapS: oh! I see ya that would explain it since func should be correct if its post binding | 17:31 |
| fungi | dougwig: one challenge you're going to find is that currently zuul is only capable of leaving one vote on a proposed patchset. it can't leave votes under multiple accounts. so all jobs running via that coop zuul would have to share a single vote | 17:32 |
| SpamapS | clarkb: but it's entirely possible that we should be doing a __get__ in invalidate too.. I can make a test right now by extending OpenStackCloud .. | 17:32 |
| clarkb | SpamapS: I think __get__ will work for invalidate it just isn't necessary if your note about binding is correct | 17:32 |
| *** notnownikki has quit IRC | 17:32 | |
| clarkb | SpamapS: whereas it would be ncessary on line 165 | 17:32 |
| morganfainberg | SpamapS: you could use functools partial to solve the issue with the late bind | 17:32 |
| SpamapS | clarkb: I went a little cross-eyed writing _cache_on_arguments so it is quite helpful to have more eyes on it. | 17:33 |
| openstackgerrit | Merged openstack-infra/infra-manual: Reverse dependency relationship in project creation https://review.openstack.org/173485 | 17:33 |
| fungi | dougwig: we've discussed the possibility of allowing third parties to slave their custom workers to zuul.openstack.org but at this point any planning toward that capability probably needs to happen as part of the zuul v3 redesign | 17:33 |
| SpamapS | the whole thing has soured me on the pattern really. | 17:33 |
| SpamapS | I'm quite inclined to just write my own cache_on_arguments method that doesn't have to bend over backwards like that. | 17:34 |
| clarkb | SpamapS: ya it is a bit complicated to follow | 17:34 |
| *** sdake_ has joined #openstack-infra | 17:34 | |
| greghaynes | ah, I see you are memoizing in a decorator | 17:34 |
| clarkb | greghaynes: ya | 17:34 |
| greghaynes | that is how that usually ends | 17:34 |
| greghaynes | pain and suffering | 17:34 |
| *** mattfarina has quit IRC | 17:34 | |
| dougwig | fungi: i'm not even sure you need to slave their workers, since many just run dsvm nodes with hooks. though maybe to spin up a pool in a different cloud. | 17:34 |
| SpamapS | clarkb: I'd be ok with the complicated flow if I knew how it worked and could document it a little more. I'm kind of like "yeah sure it works see the tests passing?" :-/ | 17:34 |
| openstackgerrit | Russell Bryant proposed openstack/requirements: global-requirements: Add ovs https://review.openstack.org/177891 | 17:34 |
| fungi | dougwig: then i'm back to not understanding why those don't just use our current workers connected to zuul.openstack.org | 17:35 |
| SpamapS | greghaynes: all we need is a metaclass and we'll have a sufficiently complex piece of syntactic sugar that python devs everywhere will cower in fear of our decorators. | 17:35 |
| dougwig | fungi: because it's testing a closed-source solution. | 17:35 |
| fungi | dougwig: if what they need is "to just run dsvm nodes with hooks" then we have that capability already | 17:35 |
| morganfainberg | SpamapS: use descriptors. | 17:35 |
| SpamapS | morganfainberg: I think I am. Sort of. :) | 17:35 |
| * greghaynes cowers in fear | 17:35 | |
| morganfainberg | SpamapS: people will cower in the Java like syntax. | 17:36 |
| SpamapS | morganfainberg: we have reached the frontier of my understanding, so I'm learning as I go here. | 17:36 |
| fungi | dougwig: oh, so you mean testing software you can't publish. that sounds complicated to engineer as a shared solution | 17:36 |
| *** sdake has quit IRC | 17:36 | |
| *** armax has joined #openstack-infra | 17:36 | |
| clarkb | morganfainberg: we are using descriptors | 17:36 |
| fungi | dougwig: i'm not sure how you would manage to hide the source code for your closed-source software if it's being tested on a community-run ci | 17:36 |
| clarkb | morganfainberg: but I think you mean implement descriptors such that the existing methods work without decoration? | 17:37 |
| dougwig | fungi: well, i'm testing the driver that is published (in neutron), and the appliance is a static widget. so my changes/maintenance are all in the CI plumbing itself. i don't think i'm alone in that. | 17:37 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for pypi mirrors disconnecting runs https://review.openstack.org/177871 | 17:37 |
| dougwig | fungi: i suspect we have a communication disconnect. :) | 17:37 |
| fungi | dougwig: yes, i have a feeling you and i mean completely different things with many of the words we're using | 17:37 |
| *** marun has joined #openstack-infra | 17:37 | |
| clarkb | morganfainberg: sorry without the special sauce decoration. iirc dogpile uses decoration out of the box | 17:37 |
| SpamapS | clarkb: oh thats an interesting thought. | 17:38 |
| greghaynes | no, no | 17:38 |
| fungi | dougwig: for me closed-source software is software for which the source code is not (legally) available to the general public. perhaps you're talking about non-free open-source software (i.e. software for which the source is public but which comes with restricted licenses that may hinder its reuse or derivation) | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add a method for getting an endpoint https://review.openstack.org/177314 | 17:39 |
| fungi | (or hinder its redistribution) | 17:40 |
| SpamapS | fungi: dude, you're so Debian right now | 17:40 |
| SpamapS | ;) | 17:40 |
| *** hdd has quit IRC | 17:41 | |
| fungi | SpamapS: well, it's germane to the problem and solution being discussed | 17:41 |
| fungi | dougwig: or are you talking about testing some free/open-source software's interactions with some other system (which may be not be generally available for various reasons)? | 17:41 |
| *** shashankhegde has joined #openstack-infra | 17:41 | |
| SpamapS | fungi: entirely, it was a compliment. :) | 17:43 |
| fungi | dougwig: probably concrete examples of what you're trying to solve with your proposal would help clarify things, rather than speaking in abstract generalities | 17:43 |
| fungi | SpamapS: as long as i'm not too debian-devel mailing list kilothread flametastic i'm cool with that ;) | 17:44 |
| dougwig | fungi: i'm talking open-source openstack driver with closed-source backend, which tests every gerrit submission for a given openstack project (aka third-party CI.) which is usually accomplished by running a copy of infra's setup, running vanilla devstack-gate, subbing in your driver in local.conf, and using a gate hook to create the connection to the | 17:45 |
| dougwig | proprietary backend. it can all be run trivially on infra's setup today, but it's a closed-source backend. so all of the vendors run a copy of the infra setup, and almost all of our issues are unrelated to our drivers (it's almost all canonical and pypi download problems.) i'd love to submit 10 lines of yaml to project-config and a gate hook in a repo and | 17:45 |
| dougwig | be done. so, i was just asking if something along those lines were feasible and lower-cost than the community pays for all the errors today. i expect the answer is, "we don't want to touch proprietary stuff with a 10' pole", which is totally ok. i'm just pondering based on where my CI has been flaky, the cost that has incurrent for folks, and how can we | 17:45 |
| dougwig | reduce it. the common puppet scripts are a great start, but that ongoing stuff that breaks isn't covered by that. | 17:45 |
| *** sdake has joined #openstack-infra | 17:45 | |
| SpamapS | clarkb: so I think what morganfainberg is suggesting is that we would make the methods we want to "decorate" non-data descriptors that would define a __get__ that would do the caching.. I think. | 17:45 |
| dougwig | sorry for the wall. :) | 17:45 |
| morganfainberg | SpamapS: basically | 17:45 |
| *** rkukura has joined #openstack-infra | 17:47 | |
| *** amotoki has quit IRC | 17:47 | |
| SpamapS | morganfainberg: and thats where you get into dogpile.cache having a metaclass that gives you the ability to do that more or less easily. Yes? | 17:47 |
| morganfainberg | That was my thought. But it might be the wrong approach since I haven't tried it out. | 17:47 |
| asselin_ | dougwig, if you can test your proprietary hardward using the -infra setup, just by adding a 10 line job, then that's probably what you should do (and what clarkb said earlier) | 17:47 |
| clarkb | SpamapS: morganfainberg gotcha so no decoration needed either just implement __get__ and __set__ such that caching happens | 17:47 |
| asselin_ | dougwig, the assumption is that there are no security issues on your end. | 17:48 |
| morganfainberg | clarkb: aye. | 17:48 |
| fungi | dougwig: okay, so you're asking how to test some otherwise inaccessible system/device's interaction with software running on workers coordinated by a central system... we already do a lot of that. i think it depends entirely on what kind of interaction you're needing to test | 17:48 |
| clarkb | asselin_: except that we explicitly don't want to be testing proprietary things | 17:48 |
| *** sdake_ has quit IRC | 17:48 | |
| morganfainberg | clarkb: that would be my approach probably now. But I haven't tried it. So beware of potential landmines. | 17:48 |
| SpamapS | well.. you'd still decorate I think, but your decorator would just add the __get__ and __set__ to the method. | 17:48 |
| clarkb | asselin_: anything we run upstream we intend to have be reproduceable by any dev. eg if you get a jenkins -1 that shouldn't mean that I have to go and talk to $company to help you sort it out with them | 17:48 |
| SpamapS | So it would all happen at the right time (parse time) and then you wouldn't have to rewrap at runtime like my method does. | 17:49 |
| SpamapS | I may dive down this rabbit hole a bit... it's quite interesting. | 17:49 |
| *** marcusvrn_ has quit IRC | 17:49 | |
| asselin_ | clarkb, yes...so that assumes then that the propreitary hardware is publicly accessible to everyone | 17:49 |
| * SpamapS wishing he hadn't taken a puck to the head yesterday... | 17:49 | |
| dougwig | asselin_, fungi - right, what clarkb said is the reason we run a billion CI's now. | 17:50 |
| dougwig | i may be exaggerating slightly. | 17:50 |
| *** dprince has quit IRC | 17:50 | |
| clarkb | well | 17:50 |
| fungi | asselin_: clarkb: dougwig: job running in the openstack ci making http calls to a random web service are probably a bit of a grey area | 17:50 |
| clarkb | fungi: we already don't allow PPAs and the like for the most part | 17:50 |
| fungi | if it needed custom credentials, we definitely wouldn't support it | 17:50 |
| fungi | we do allow ppas | 17:51 |
| fungi | we don't preconfigure them, but jobs can certainly modify the sources.list and install additional packages | 17:51 |
| *** cdent has quit IRC | 17:51 | |
| clarkb | thats true | 17:51 |
| dougwig | clarkb, fungi, asselin_: history tells us that infra is *way better* at running stable openstack CI's than the vendors. and there is a big cost to those false positives. is it possible/desired to leverage infra's awesomeness to help reduce that cost? that's my question. | 17:52 |
| *** harlowja_away is now known as harlowja_ | 17:52 | |
| *** mpaolino has quit IRC | 17:52 | |
| greghaynes | SpamapS: have you looked at the functools lru_cache btw? | 17:52 |
| mordred | greghaynes: I looked at it a little while ago before we went down the dogpile route | 17:54 |
| fungi | dougwig: so, the reasons we envisioned third-party testing originall were for cases where your jobs specifically needed workers to locally interact with specific sorts of hardware or have access to non-public services or install restricted/licensed software or... | 17:54 |
| greghaynes | even if you dont want to just use it, the source is there and it should be close :) | 17:54 |
| fungi | er, originally | 17:54 |
| fungi | also for the general case where the worker configuration was going to be too complex for us to maintain | 17:55 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Update query for bug 1403284 https://review.openstack.org/177886 | 17:56 |
| openstack | bug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] https://launchpad.net/bugs/1403284 - Assigned to Joe Gordon (jogo) | 17:56 |
| SpamapS | greghaynes: if we just want a global cache region, we can get that level of simplicity with dogpile.cache too. | 17:56 |
| SpamapS | greghaynes: the complexity is that we have one _cache per OpenStackCloud. | 17:57 |
| fungi | dougwig: asselin_: unfortunately a lot of projects who wanted to ensure various backends were thoroughly tested mis-stated their desires as wanting all the people responsible for those drivers and plugins to maintain separate external ci systems | 17:57 |
| greghaynes | SpamapS: ah! | 17:57 |
| greghaynes | SpamapS: namespaced_lru_cache | 17:57 |
| greghaynes | ;) | 17:57 |
| clarkb | greghaynes: mordred for ansible synchronize do you know how to give a different dest_path per node? eg if I say dest=/foo/bar/$VAR-thing can I have $VAR change per node? | 17:57 |
| clarkb | greghaynes: mordred if running in pull mode | 17:57 |
| mordred | clarkb: yes | 17:57 |
| SpamapS | greghaynes: did you just invent that? | 17:58 |
| greghaynes | SpamapS: yes | 17:58 |
| mordred | clarkb: although I do not know anything about pull mode other than it seems to destroy the main awesome thing about ansible | 17:58 |
| SpamapS | greghaynes: well done | 17:58 |
| greghaynes | SpamapS: naming it is the hard part | 17:58 |
| SpamapS | no the invalidation is | 17:58 |
| mordred | clarkb: however, I do not think that pull mode should break it eitehr way | 17:58 |
| SpamapS | :) | 17:58 |
| *** hdd has joined #openstack-infra | 17:58 | |
| clarkb | mordred: d-g needs to copy logs from all of the subnodes onto the primary node for jenkins scp to work | 17:58 |
| mordred | yah. | 17:59 |
| clarkb | mordred: its possible we can optimize that away when we go completely to swift | 17:59 |
| mordred | yah | 17:59 |
| *** jamesmcarthur has quit IRC | 17:59 | |
| clarkb | (but I like the main job script to not know how to deal with its logs so that its flexible) | 17:59 |
| *** claudiub has quit IRC | 18:00 | |
| clarkb | mordred: so what would that look like? iirc ansible does all of its variable stuff on the local side not the remote side? I suppose it still knows about the node | 18:00 |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: Add heat-translator to global requirments projects https://review.openstack.org/177904 | 18:01 |
| clarkb | mordred: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n517 is the todo I am trying to fix | 18:01 |
| mordred | clarkb: it depends - but you're using a few words in a way that I'm goign to have to ask annoying questions to make sure we're talking abot the same thing | 18:01 |
| mordred | clarkb: AH! excellent. let me look at that | 18:01 |
| pleia2 | can I get an infra-core quickly look at https://review.openstack.org/#/c/134008/ ? it's old, simple, and the author abandoned it last week (gave up?), but it's a good doc fix that we need so I restored | 18:02 |
| *** gordc_afk is now known as gordc | 18:02 | |
| mordred | clarkb: ok. so what you were thinking was a thing to run on each remote node that would scp the files back to the master node, yeah? | 18:03 |
| mordred | clarkb: rather tahn a thing that ran on the master node and scp'd from each node | 18:03 |
| *** jogo has quit IRC | 18:03 | |
| clarkb | mordred: no, I want to use ansible synchronize (which is rsync) to do it in parallel | 18:03 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Make the query for bug 1403284 more generic https://review.openstack.org/177888 | 18:04 |
| openstack | bug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] https://launchpad.net/bugs/1403284 - Assigned to Joe Gordon (jogo) | 18:04 |
| clarkb | mordred: I could also have nasible run asnible directly in parallel but thats kinda silly since syncrhonize works | 18:04 |
| clarkb | mordred: er ansible run rsync directly | 18:04 |
| mordred | clarkb: sure ... one sec | 18:04 |
| pelix | @zaro yeap something like jenkins-api-mock but could opt for just a simple HTTPServer that handles requests and you indicate either based on the present of request headers what the response should be, or a Queue of responses to be served irrespective of the request. | 18:04 |
| *** dhritishikhar_ has quit IRC | 18:04 | |
| *** dhritishikhar_ has joined #openstack-infra | 18:05 | |
| clarkb | mordred: then each node should go into a known subdr | 18:05 |
| mordred | clarkb: yeah. - that part is the easy part | 18:05 |
| *** EmilienM|afk is now known as EmilienM | 18:06 | |
| dhellmann | clarkb, anteaya: pinging again on https://review.openstack.org/#/c/177885/ -- problems with that job are blocking us from merging an unrelated change in oslo.messaging that we need for the stable/kilo release | 18:08 |
| pelix | zaro: indeed --workers probably not needed test at all unless the write out to files is put inside the function called in parallel | 18:08 |
| openstackgerrit | Levi Blackstone proposed openstack-infra/project-config: Enable gate jobs for remaining StackTach libraries https://review.openstack.org/177909 | 18:09 |
| *** davideagnello has quit IRC | 18:10 | |
| mordred | clarkb: ok - so I undersatnd what you want | 18:10 |
| clarkb | dhellmann: doesn't that need swift enabled for glance? | 18:10 |
| *** jtriley_ has quit IRC | 18:11 | |
| dhellmann | clarkb: good question. jogo put the patch together based on an earlier patch for the same bug | 18:11 |
| dhellmann | and now I don't see jogo on irc to ask | 18:11 |
| clarkb | sdague may know | 18:11 |
| clarkb | sdague: on https://review.openstack.org/#/c/177885/ is my -1 valid? | 18:12 |
| clarkb | I am pretty sure it is | 18:12 |
| clarkb | the only thing that would make it invalid is devstack magic aiui | 18:12 |
| *** jamesmcarthur has joined #openstack-infra | 18:12 | |
| *** davideagnello has joined #openstack-infra | 18:12 | |
| dhellmann | clarkb: I don't see swift enabled in any other place where that same line is in the same file | 18:13 |
| *** yamahata has quit IRC | 18:14 | |
| clarkb | oh the other large ops job does this too | 18:14 |
| clarkb | I am probably wrong then I can change my vote | 18:14 |
| clarkb | (that seems odd that we test glance differently here) | 18:14 |
| dhellmann | yeah, it does, but maybe that's part of the way we collapsed some alternate case jobs together? | 18:15 |
| mordred | clarkb: OH! I knew there was somewhere we were missing each other | 18:16 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 18:16 |
| mordred | clarkb: when you said "pull mode"- you meant "synchronize's pull mode" - I thought you meant ansible pull mode | 18:17 |
| mordred | clarkb: which is why I was confused | 18:17 |
| mordred | clarkb: one sec - it's easy | 18:17 |
| clarkb | mordred: yes synchronize pull mode. I have no idea what ansible pull mode is | 18:17 |
| mordred | clarkb: you never should | 18:17 |
| mordred | clarkb: it's a dumb idea | 18:17 |
| mordred | clarkb: (it's the thing that turns ansible into puppet agent) | 18:17 |
| *** dboik_ has quit IRC | 18:17 | |
| dhellmann | clarkb: ty | 18:17 |
| *** dboik has joined #openstack-infra | 18:18 | |
| clarkb | mordred: that sounds amazing (I am guessing that its easier for network security in some orgs to have a bunch of things connect to one place nevermind that its spofy and scales poorly) | 18:20 |
| *** Sukhdev has quit IRC | 18:20 | |
| pleia2 | clarkb: thanks | 18:20 |
| clarkb | dhellmann: pleia2 np | 18:20 |
| openstackgerrit | Alexander Tivelkov proposed openstack-infra/project-config: Added branch creation to yaql ACL https://review.openstack.org/177917 | 18:20 |
| *** Hal has joined #openstack-infra | 18:20 | |
| mordred | clarkb: http://paste.openstack.org/show/208690 | 18:20 |
| *** pelix has quit IRC | 18:20 | |
| *** jcoufal has quit IRC | 18:20 | |
| *** Hal is now known as Guest61628 | 18:21 | |
| mordred | clarkb: I tested a similar thing locally against a remote host and copied things back | 18:21 |
| mordred | clarkb: but consider that untested in terms of $BASE expansion, etc | 18:21 |
| openstackgerrit | Merged openstack-infra/nodepool: Update Readme config -> system-config repo https://review.openstack.org/134008 | 18:21 |
| clarkb | mordred: except we want a known constant dir | 18:21 |
| mordred | clarkb: sure | 18:22 |
| clarkb | mordred: because it makes other tooling that looks for logs simpler | 18:22 |
| mordred | clarkb: you explicitly want the numbered subdir, rather than one based on hostname, yeah? | 18:22 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Puppet: move from Stackforge to OpenStack https://review.openstack.org/176326 | 18:22 |
| clarkb | mordred: ya (or some other arbitrary but constant set of iterations) | 18:22 |
| clarkb | mordred: so a b c d etc is fine too | 18:23 |
| clarkb | mordred: but $random-nodename or $random-ip is not so good | 18:23 |
| mordred | nod | 18:23 |
| *** tiswanso_ has quit IRC | 18:23 | |
| openstackgerrit | Merged openstack-infra/project-config: Reduce number of services running in largeops-src https://review.openstack.org/177885 | 18:23 |
| mordred | clarkb: so we should, for this iteration, put a counter into the inventory | 18:23 |
| mordred | with a variable name | 18:23 |
| mordred | for each host | 18:23 |
| clarkb | mordred: ah! ok I can do that | 18:23 |
| mordred | and refernece that instead of inventory_hostname | 18:23 |
| *** tiswanso has joined #openstack-infra | 18:23 | |
| clarkb | mordred: thanks this helps | 18:23 |
| mordred | clarkb: woot! | 18:23 |
| * mordred was helpful today | 18:23 | |
| *** Guest61628 has quit IRC | 18:23 | |
| *** tnovacik has joined #openstack-infra | 18:24 | |
| *** AJaeger_ has joined #openstack-infra | 18:25 | |
| AJaeger_ | pleia2: reading the scrollback: Should we add you as admin to the openstack organization in transifex so that you have full access? | 18:25 |
| *** unicell has quit IRC | 18:26 | |
| *** unicell1 has joined #openstack-infra | 18:26 | |
| clarkb | mordred: I think this will help with the network setup too since I can use the same counter to set IPs | 18:26 |
| mordred | clarkb: ++ | 18:27 |
| clarkb | mordred: do I have to do anything specail to refer to an inventory var or would it just be {{ host_counter }} if host_counter=X is what I put in the inventory? | 18:28 |
| mordred | yup | 18:29 |
| pleia2 | AJaeger_: that would probably be helpful, I won't touch anything :) | 18:29 |
| mordred | clarkb: just host_counter=X | 18:29 |
| mordred | clarkb: and then {{ host_counter }} | 18:29 |
| clarkb | mordred: so inventory vars are "global" for each node? | 18:30 |
| mordred | clarkb: yes | 18:31 |
| mordred | clarkb: you can also do complexer variable data structures and namespacing - but then you're talking multiple files and stuff | 18:31 |
| *** rkukura has quit IRC | 18:33 | |
| *** zz_zz_zz_ja is now known as ja | 18:33 | |
| *** signed8bit has joined #openstack-infra | 18:33 | |
| *** tonytan4ever has joined #openstack-infra | 18:33 | |
| AJaeger_ | pleia2: what'y your login? | 18:34 |
| pleia2 | AJaeger_: good question! checking | 18:34 |
| otherwiseguy | anteaya: I assume infra uses a PyPI mirror for the CI stuff? I just added the first release of the 'ovs' python package to PyPI. When does that mirror update (if it exists)? | 18:34 |
| AJaeger_ | clarkb, mordred, do you agree with adding pleia2 as transifex admin? | 18:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Add host counter to ansible inventory https://review.openstack.org/177920 | 18:34 |
| clarkb | AJaeger_: ++ especially since she is doing a lot of the zanata work | 18:34 |
| mordred | AJaeger_: ++ | 18:35 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Add host counter to ansible inventory https://review.openstack.org/177920 | 18:35 |
| mordred | otherwiseguy: the mirror runs every ... I can't remember, 5 minutes? | 18:35 |
| clarkb | still no jogo but he will likely be interested in ^ too | 18:35 |
| clarkb | mordred: ya 5 minutes | 18:36 |
| AJaeger_ | clarkb, mordred: Thanks for the quick answer. I'll add her once she found her login ;) | 18:36 |
| pleia2 | AJaeger_: pleia2 | 18:36 |
| fungi | otherwiseguy: normally within about 5 minutes, but at the moment i'm having to refresh the mirrors so it could still be a couple hours until that completes | 18:36 |
| AJaeger_ | pleia2: that was too easy ;) | 18:36 |
| otherwiseguy | mordred: ah, so we were just *really* fast with adding the networking-ovn/global-requirements patches. :) | 18:36 |
| clarkb | mordred: otherwiseguy but fungi kicked off some full syncs to catch up after pypi glusterfs troubles | 18:36 |
| otherwiseguy | fungi: ah. ok. just making sure it wasn't a completely manual process I needed to poke someone about. :) Thanks! | 18:37 |
| *** _nadya_ has quit IRC | 18:37 | |
| *** [HeOS] has quit IRC | 18:38 | |
| AJaeger_ | pleia2: you should be setup now | 18:39 |
| fungi | btw, still no response from fantastical support, so i've got downloads of the latest setuptools wheel from pypi.dfw going in a continuous loop and validating the checksums. as soon as it breaks on a failure i'll see if i can get some more details from our logs and a tcpdump | 18:39 |
| *** teran has quit IRC | 18:39 | |
| pleia2 | AJaeger_: thanks | 18:40 |
| *** teran has joined #openstack-infra | 18:40 | |
| AJaeger_ | pleia2: the admin UI is not always intuitive, you might get lost - like I'm currently trying to find a page that shows that you're an admin ;) | 18:41 |
| AJaeger_ | pleia2: https://www.transifex.com/organization/openstack/edit/ includes you... | 18:41 |
| *** ildikov has joined #openstack-infra | 18:42 | |
| AJaeger_ | but you're not seen here: https://www.transifex.com/organization/openstack - strange ;( | 18:42 |
| AJaeger_ | pleia2: now you're added, forgot to save ;( | 18:43 |
| AJaeger_ | pleia2: if you have questions, feel free to email or catch me on IRC... | 18:43 |
| pleia2 | AJaeger_: will do, thanks for your help | 18:43 |
| *** teran has quit IRC | 18:44 | |
| AJaeger_ | cinerama: btw. to get the list of projects with translations, you can either check https://www.transifex.com/organization/openstack or check in project-config the file zuul/layout.yaml | 18:44 |
| *** ddieterly has quit IRC | 18:44 | |
| cinerama | AJaeger_: cool, thanks | 18:45 |
| AJaeger_ | repos that are translated have translation-jobs or openstack-doc-translation added | 18:45 |
| AJaeger_ | and adding a new project to transifex means additing it in project-config, additing it in transifex - after creation of the initial file po file in the repo itself | 18:46 |
| *** e0ne has joined #openstack-infra | 18:46 | |
| *** ddieterl_ has joined #openstack-infra | 18:46 | |
| AJaeger_ | Our tools only work if there's an existing po file, can be empty for a start and then we import we the toolchain | 18:46 |
| pleia2 | oh yes, that's a clever way of finding them in configs | 18:47 |
| sdague | clarkb: https://review.openstack.org/#/c/177885 - the -1 was invalid. Devstack sets up glance to use swift if it's enabled, otherwise it works with a file system backend. | 18:47 |
| *** Longgeek has quit IRC | 18:48 | |
| *** jamesmcarthur has quit IRC | 18:48 | |
| *** unicell has joined #openstack-infra | 18:48 | |
| *** unicell1 has quit IRC | 18:49 | |
| zaro | clarkb: do you know which version of virtualenv jenkins machine uses when running 'tox -epy27'? | 18:50 |
| *** hyakuhei has quit IRC | 18:50 | |
| *** AJaeger_ has quit IRC | 18:52 | |
| *** andrey-mp has joined #openstack-infra | 18:53 | |
| *** rkukura has joined #openstack-infra | 18:54 | |
| pleia2 | not sure how to respond to this, we kind of make the rules here, particularly as bots leave our controlled #openstack-* namespace https://review.openstack.org/#/c/177787/ | 18:54 |
| pleia2 | (and we gate on this, so it can't land) | 18:54 |
| *** Rockyg has joined #openstack-infra | 18:55 | |
| *** marun_ has joined #openstack-infra | 18:55 | |
| *** rbradfor has quit IRC | 18:56 | |
| *** tonytan4ever has quit IRC | 18:56 | |
| *** bswartz has joined #openstack-infra | 18:57 | |
| *** Sukhdev has joined #openstack-infra | 18:58 | |
| *** marun has quit IRC | 18:58 | |
| andrey-mp | hi! who can help me with rally experimental job for project stackforge/ec2api? It worked two weeks ago but not now... https://jenkins06.openstack.org/job/gate-rally-dsvm-ec2-api-fakevirt/ it fails becuase flag ERROR_ON_CLONE now is True in this job. I don't understand when/where it has changed last two weeks. And I don't understand how to fix it. | 19:00 |
| *** baoli has quit IRC | 19:01 | |
| *** dkranz has quit IRC | 19:01 | |
| *** pabelanger has quit IRC | 19:01 | |
| anteaya | otherwiseguy: I was out for a walk, looks like you got your response, which is better than I could have given you anyway | 19:01 |
| otherwiseguy | anteaya: thanks. :) | 19:02 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Puppet: move from Stackforge to OpenStack https://review.openstack.org/176326 | 19:02 |
| pleia2 | thanks fungi | 19:03 |
| *** dizquierdo has quit IRC | 19:04 | |
| fungi | pleia2: also your answer was great. running a separate gerritbot should be trivial for them (they can even set up a separate account for it to authenticate against if they want) | 19:04 |
| *** davideagnello has quit IRC | 19:04 | |
| *** sputnik13 has quit IRC | 19:04 | |
| * pleia2 nods | 19:05 | |
| pleia2 | I had a look at the readme to confirm first, it's easy stuff | 19:05 |
| openstackgerrit | Renat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral https://review.openstack.org/175328 | 19:06 |
| *** sputnik13 has joined #openstack-infra | 19:06 | |
| *** davideagnello has joined #openstack-infra | 19:06 | |
| fungi | yep, it is. apt-get install gerritbot, tweak config, copy initscript from our puppet module if you want, and you're off to the races | 19:06 |
| fungi | er, not apt-get install but pip install | 19:06 |
| *** kgiusti has joined #openstack-infra | 19:06 | |
| *** pabelanger has joined #openstack-infra | 19:07 | |
| fungi | hrm, we tagged 0.2.0 a year ago but it never landed on pypi | 19:07 |
| openstackgerrit | Renat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral https://review.openstack.org/175328 | 19:08 |
| fungi | ahh, it has no pypi jobs! | 19:08 |
| jklare | AJaeger: ping | 19:08 |
| *** jog0 has joined #openstack-infra | 19:09 | |
| fungi | no, nevermind. misread. it does now, but maybe it lacked them when 0.2.0 was tagged a year ago | 19:09 |
| sdague | did we have *another* gerrit restart? | 19:09 |
| *** AL34N1X has joined #openstack-infra | 19:09 | |
| sdague | oh, never mind, stale gerrit page | 19:09 |
| openstackgerrit | Renat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral https://review.openstack.org/175328 | 19:09 |
| *** dprince has joined #openstack-infra | 19:11 | |
| fungi | looking at the clustering of pypi.dfw read timeouts in logstash, it seems to come in spurts | 19:11 |
| *** dguerri is now known as _dguerri | 19:11 | |
| *** Somay has joined #openstack-infra | 19:12 | |
| fungi | we had none from ~16:40-18:30 but then 7 within a second of one another | 19:12 |
| clarkb | sdague right amd we test swift + glance everywhere but here | 19:12 |
| *** tonytan4ever has joined #openstack-infra | 19:12 | |
| clarkb | is that intentional? | 19:12 |
| sdague | yeh, there is nothing useful that swift does in largeops | 19:13 |
| sdague | except take up memory | 19:13 |
| sdague | the hypervisor is fake | 19:13 |
| *** hyakuhei has joined #openstack-infra | 19:13 | |
| *** hyakuhei has quit IRC | 19:14 | |
| *** spredzy_|afk is now known as spredzy_ | 19:16 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific https://review.openstack.org/176674 | 19:16 |
| jeblair | fungi: i strongly suspect i will not be able to attend infra meeting tomorrow | 19:16 |
| fungi | jeblair: no worries. i'll be happy to chair. anything you need added to the agenda? | 19:17 |
| jklare | could someone help me a bit and tell me if this patch works as decribed in the commit (especially the pacemaker part) https://review.openstack.org/#/c/176674/ | 19:17 |
| *** _dguerri is now known as dguerri | 19:17 | |
| fungi | jeblair: i'll also lurk tc and cross-project as usual tomorrow in case anything comes up | 19:17 |
| openstackgerrit | Merged openstack-infra/project-config: networking-nec: add ability to create and manage branches https://review.openstack.org/176466 | 19:17 |
| jeblair | fungi: don't think so -- remind folks about the summit prep etherpad? i think i'll collate that end of week/early next week. | 19:18 |
| jeblair | fungi: cool, thanks | 19:18 |
| fungi | jeblair: yep, will definitely do | 19:18 |
| jeblair | ttx: i'll probably miss the tc meeting too; will vote on tc changes today. | 19:18 |
| *** jtriley has joined #openstack-infra | 19:19 | |
| *** Somay has quit IRC | 19:19 | |
| *** shashankhegde has quit IRC | 19:19 | |
| andrey-mp | could this patch https://review.openstack.org/#/c/168796/2 break behavior of ERROR_ON_CLONE for this https://jenkins06.openstack.org/job/gate-rally-dsvm-ec2-api-fakevirt/ ? | 19:19 |
| *** jcoufal has joined #openstack-infra | 19:21 | |
| *** yamahata has joined #openstack-infra | 19:21 | |
| *** Somay has joined #openstack-infra | 19:22 | |
| *** sdake_ has joined #openstack-infra | 19:22 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through https://review.openstack.org/172284 | 19:23 |
| zaro | fungi, clarkb : do you know which version of virtualenv jenkins machine uses when running 'tox -epy27'? i don't see it in the logs. | 19:23 |
| *** jog0 is now known as jogo | 19:23 | |
| clarkb | zaro should be whatever latest tox includes | 19:23 |
| pleia2 | off to lunch + weekday errands, bbiab | 19:24 |
| clarkb | ya I'm mostly not here atm | 19:24 |
| zaro | hmm, then i'm stumped as to why zuul test only fail when run with tox. | 19:25 |
| *** AL34N1X has quit IRC | 19:25 | |
| lifeless | morning | 19:26 |
| *** sdake has quit IRC | 19:26 | |
| *** annegentle has joined #openstack-infra | 19:27 | |
| greghaynes | mordred: interesting dib regression for youall (I think) - we are going to be using default apt/yum mirror info now where as before the rax/hp mirror info was baked in to be a localish mirror | 19:28 |
| *** otter768 has joined #openstack-infra | 19:28 | |
| greghaynes | this is currently the case in hpcloud, obviously | 19:28 |
| *** rbradfor has joined #openstack-infra | 19:28 | |
| jeblair | greghaynes: i think that's a desired feature, actually :) | 19:29 |
| *** armax has quit IRC | 19:29 | |
| *** ociuhandu has quit IRC | 19:29 | |
| jeblair | greghaynes: when phrased as "do not use broken local mirror..." :) | 19:29 |
| greghaynes | hah, ok then. The downside being default mirrors tend to be slow | 19:30 |
| greghaynes | but as long as were aware | 19:30 |
| jeblair | greghaynes: yeah, we might want to run our own mirrors. we talk about that occasionally. | 19:30 |
| *** armax has joined #openstack-infra | 19:31 | |
| *** MaxV has joined #openstack-infra | 19:31 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Use /opt/git directly https://review.openstack.org/177629 | 19:32 |
| jeblair | greghaynes: but also, hopefully we will use the mirrors very little -- on-node cache should help a lot | 19:32 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Stop testing setup.py install https://review.openstack.org/177505 | 19:32 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Test pip install -e of projects. https://review.openstack.org/177504 | 19:32 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Build all the wheels in one pass rather than many. https://review.openstack.org/177503 | 19:32 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Improve integration.sh. https://review.openstack.org/177500 | 19:32 |
| greghaynes | jeblair: ah, good point | 19:32 |
| *** dguerri is now known as _dguerri | 19:32 | |
| andrey-mp | fungi, jeblair, sdague: who can help me with rally experimental job for project stackforge/ec2api? It worked two weeks ago but not now... https://jenkins06.openstack.org/job/gate-rally-dsvm-ec2-api-fakevirt/ it fails becuase flag ERROR_ON_CLONE now is True in this job. I don't understand when/where it has changed last two weeks. And I don't understand how to fix it. I think that this patch changed behavior - https://rev | 19:32 |
| andrey-mp | or maybe I should file a bug for devstack? | 19:33 |
| *** otter768 has quit IRC | 19:33 | |
| lifeless | could I beg two x +2 and a +A for https://review.openstack.org/#/c/177500 <- it was green and will be again - that push was just to tweak a comment for jhesketh | 19:33 |
| *** ja is now known as zz_ja | 19:33 | |
| *** hdd has quit IRC | 19:34 | |
| sdague | andrey-mp: I don't know, we never set that to true | 19:34 |
| jeblair | andrey-mp: i can't dig into it now, but in general, you should add stackforge/ec2api to the PROJECTS variable in the job | 19:34 |
| mordred | greghaynes: also - AFS solves this ... | 19:34 |
| *** Somay has quit IRC | 19:35 | |
| *** dkranz has joined #openstack-infra | 19:35 | |
| *** sdake_ has quit IRC | 19:35 | |
| sdague | andrey-mp: so, ERROR_ON_CLONE=True is true for all gate jobs | 19:35 |
| sdague | you are not allowed to do that differently | 19:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: Put input variables into ansible inventory https://review.openstack.org/177943 | 19:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: Move all the ansible calls into playbooks https://review.openstack.org/177944 | 19:35 |
| *** sdake has joined #openstack-infra | 19:35 | |
| *** Somay has joined #openstack-infra | 19:35 | |
| mordred | clarkb: I couldn't help myself - so I did that ^^ feel free to -2/abandon the if you hate it - no emotional attachment whatsoever | 19:35 |
| andrey-mp | sdague: yeah, i saw this in all files. but previous gating had a line '+ [[ False = \T\r\u\e ]]' and error didn't raise | 19:35 |
| mordred | clarkb: it was more a "I started poking and couldn't stop poking" | 19:36 |
| greghaynes | hahaah | 19:36 |
| greghaynes | the "AFS solves this" | 19:37 |
| andrey-mp | jeblair: ok. (ec2api already in PROJECTS) | 19:37 |
| nibalizer | yep | 19:37 |
| *** isviridov_away is now known as isviridov | 19:37 | |
| sdague | andrey-mp: this is why it doesn't work - http://logs.openstack.org/70/177570/10/experimental/gate-rally-dsvm-ec2-api-fakevirt/2f85849//logs/devstack-gate-setup-workspace-new.txt.gz#_2015-04-27_18_54_50_643 | 19:37 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific https://review.openstack.org/176674 | 19:38 |
| *** isviridov is now known as isviridov_away | 19:38 | |
| andrey-mp | sdague: thank you very much ) i will dig into this now | 19:39 |
| mordred | greghaynes: (so, it's unlikely to be how we solve it - but if we had apt/yum mirrors in AFS, and we had an AFS fileserver per cloud region, then the config for the nodes could be a completely static apt source of file:///afs/openstack.org/mirror/apt and the AFS infrastructure would direct the cache misses to the closest file server) | 19:39 |
| fungi | sdague: andrey-mp: rally moved from stackforge to openstack namespace. that's likely what caused it to start failing? | 19:39 |
| sdague | fungi: yes, that's what it looks like | 19:40 |
| *** marun_ is now known as marun | 19:40 | |
| jeblair | andrey-mp: change stackforge/rally to openstack/rally in that job defn | 19:40 |
| jeblair | oh yep. :) | 19:40 |
| * jeblair goes back to laggy conf wifi | 19:40 | |
| mordred | lifeless: I see you're back to teh while loop -- did we learn something? | 19:41 |
| andrey-mp | wow, thank you all very much )) | 19:41 |
| * mordred wants to make sure the pip hole in his brain is appropriately filled | 19:41 | |
| lifeless | mordred: no, its a stack | 19:41 |
| mordred | lifeless: okie | 19:41 |
| mordred | so I just haven't gotten back to the "remove the while loop" patch | 19:42 |
| * jeblair wonders if internet3 conference wifi will be better | 19:42 | |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: fix ec2-api rally job https://review.openstack.org/177947 | 19:43 |
| nibalizer | jeblair: can you just get on wifi2 ? | 19:43 |
| jeblair | nibalizer: i should totally set up an ap with that ssid here | 19:44 |
| *** hdd has joined #openstack-infra | 19:44 | |
| lifeless | internet3? I didn't think 2 was finished yet | 19:44 |
| mordred | lifeless: stack looks good | 19:44 |
| *** ibiris is now known as ibiris_away | 19:45 | |
| mordred | lifeless: all +2 from from me | 19:45 |
| lifeless | danke | 19:45 |
| openstackgerrit | Michael Krotscheck proposed openstack/requirements: Updated oslo.config to 1.11.0 https://review.openstack.org/173449 | 19:48 |
| mordred | clarkb: when you get a sec - could you relook at 176024 - you've got a -1 on it which SpamapS addresses in teh follow up patch | 19:49 |
| *** hdd has quit IRC | 19:49 | |
| *** [HeOS] has joined #openstack-infra | 19:51 | |
| *** emagana has joined #openstack-infra | 19:52 | |
| *** ildikov has quit IRC | 19:53 | |
| *** dhritishikhar_ has quit IRC | 19:53 | |
| *** jamesmcarthur has joined #openstack-infra | 19:54 | |
| lifeless | mordred: so - the q about failed wheels is moot now | 19:54 |
| lifeless | http://logs.openstack.org/03/177503/4/check/check-pbr-installation-dsvm-upstream/84e8663/console.html#_2015-04-27_01_42_27_716 | 19:54 |
| lifeless | mordred: everything succeeds anyway | 19:55 |
| lifeless | total time 6 minutes | 19:55 |
| *** jcoufal has quit IRC | 19:55 | |
| *** andrey-mp has quit IRC | 19:56 | |
| lifeless | vs 11 minutes | 19:56 |
| lifeless | so its a nice improvement | 19:56 |
| mordred | lifeless: woot | 19:57 |
| *** armax has quit IRC | 19:58 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/puppet-openstackci: Create a jenkins server like http://jenkins.openstack.org https://review.openstack.org/175571 | 19:58 |
| openstackgerrit | Michael Krotscheck proposed openstack/requirements: Updated oslo.config to 1.11.0 https://review.openstack.org/173449 | 19:58 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Stop testing setup.py easy_install behaviour https://review.openstack.org/177505 | 19:58 |
| nibalizer | so what state is https://review.openstack.org/#/c/153819 in ? I can't tell a good reason why it hasn't landed | 19:58 |
| nibalizer | i even tried reverify which im pretty sure isnt a thing | 19:59 |
| mordred | nibalizer: I nudged it with a second +A | 19:59 |
| mordred | let's see what happens | 19:59 |
| nibalizer | ty | 20:00 |
| fungi | btw, i manually triggered the release (tarball/wheel, pypi uploads) jobs got the gerritbot 0.2.0 release from a year ago | 20:00 |
| mordred | I also +2'd your readme change - but is there a reason we have README.md files? | 20:00 |
| mordred | fungi: neat | 20:01 |
| mordred | fungi: we wanted that, right? | 20:01 |
| *** ildikov has joined #openstack-infra | 20:01 | |
| fungi | mordred: i assume so. we're running 0.2.0.post6 on review.o.o for a while now | 20:01 |
| fungi | might even be time for a 0.3.0 or something | 20:01 |
| *** dhritishikhar_ has joined #openstack-infra | 20:02 | |
| fungi | so the 3-year-old 0.1.2 sdist on pypi was sort of misleading | 20:02 |
| *** marun has quit IRC | 20:03 | |
| *** dhritishikhar_ has quit IRC | 20:04 | |
| jesusaurus | fungi: jeblair: clarkb: can you add 74943 to your to-review list? | 20:05 |
| *** dhritishikhar_ has joined #openstack-infra | 20:05 | |
| *** _nadya_ has joined #openstack-infra | 20:05 | |
| mordred | fungi: ++ | 20:06 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 20:06 | |
| mordred | fungi: one MIGHT even go so far as to say it might be time for a 1.0 | 20:06 |
| fungi | indeed it might at that | 20:07 |
| fungi | jesusaurus: mordred: i take it our requests socket timeouts for provider api connections from nodepoold don't obviate the need for 74943? | 20:07 |
| fungi | for example, http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n200 | 20:08 |
| *** doude has quit IRC | 20:09 | |
| *** doude has joined #openstack-infra | 20:09 | |
| mordred | fungi: uhm | 20:09 |
| fungi | oic, the concern is that if you timeout the api during an image update, it may be lost | 20:09 |
| fungi | so this is explicit to proxyerror exceptions then | 20:11 |
| fungi | er, specific to | 20:11 |
| *** Swami has quit IRC | 20:11 | |
| *** dhritishikhar_ has quit IRC | 20:11 | |
| mrmartin | fungi: may I ask you to do a pip freeze on ask.o.o and paste the output to paste.o.o ? | 20:12 |
| fungi | mrmartin: `pip list` output http://paste.openstack.org/show/208853/ | 20:14 |
| *** jamesmcarthur has quit IRC | 20:14 | |
| *** teran has joined #openstack-infra | 20:14 | |
| mrmartin | thanks | 20:14 |
| lifeless | mordred: so I'm wondering | 20:15 |
| fungi | mordred: jesusaurus: got it, this is not solving a problem of the api endpoint ceasing to respond and the socket apparently going dead. we get an actual exception raised in this case... going back over it now with an eye to that as the problem statement | 20:15 |
| lifeless | mordred: is there enough memory to run integration.sh against tmpfs? | 20:15 |
| mordred | fungi: yah - it's a different thing | 20:15 |
| mordred | lifeless: there is 8G of ram | 20:15 |
| lifeless | mordred: and then partition it per cpu ? | 20:15 |
| lifeless | so yes | 20:15 |
| lifeless | I'll get the devstack-gate process removed first | 20:16 |
| *** dprince has quit IRC | 20:16 | |
| lifeless | and then potter with that I think, as 90m test runs are uhm, poor. | 20:16 |
| jesusaurus | fungi: yeah, the ProxyError is orthogonal to the api timeout, the error may be raised immediately | 20:16 |
| jesusaurus | fungi: in which case we should create a new client object (which reconnects through the proxy) and try again | 20:17 |
| lifeless | will have to see what its like with the easy_install stuff removed, which hopefully is the only truely uncached run | 20:18 |
| *** _dguerri is now known as dguerri | 20:20 | |
| fungi | i do like that trick we're using in nodepool for raising an exception from a subthread in the main thread. that's nifty | 20:23 |
| *** shashankhegde has joined #openstack-infra | 20:26 | |
| mrmartin | fungi: what is the base image we are using for ask.o.o in rax? | 20:26 |
| jesusaurus | fungi: im glad you like that, it took me a while to realize that the original patch was raising the exception in the wrong thread | 20:29 |
| *** MarkAtwood has quit IRC | 20:29 | |
| *** Rockyg has quit IRC | 20:29 | |
| fungi | mrmartin: Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM) | 20:29 |
| mrmartin | thnx | 20:29 |
| *** yfried_ is now known as yfried|afk | 20:30 | |
| fungi | mrmartin: and flavor is "4 GB Performance" | 20:30 |
| *** hdd has joined #openstack-infra | 20:31 | |
| mrmartin | yeah, great, and is it a standard 12.04LTS cloudimg? | 20:31 |
| fungi | yep, and then we apply our puppet manifest for ask.openstack.org | 20:31 |
| fungi | mrmartin: well, depends on what you mean by "standard 12.04LTS cloudimg" | 20:32 |
| mrmartin | the puppet part is ok. I'm playing with ask.o.o upgrade, and it works well in my vagrant box based on 12.04lts, but when i try to deploy it into a HP Helion public cloud | 20:32 |
| *** tiswanso has quit IRC | 20:32 | |
| fungi | mrmartin: it's whatever rackspace is calling "Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM)" but it's not the image provided from ubuntu.com for precise | 20:32 |
| mrmartin | things start to broke, and it seems to be a lot of extra python packages included by default | 20:32 |
| fungi | rackspace does things to make images work for their custom cloud environment | 20:32 |
| mrmartin | something like this: http://paste.openstack.org/show/208746/ | 20:33 |
| *** hdd has quit IRC | 20:33 | |
| mrmartin | so much-much more that we have in production ask.o.o | 20:33 |
| fungi | mrmartin: hpcloud uses images from ubuntu.com, which i think have cloud-init preinstalled, and it's python-based with a fairly lengthy set of python module dependencies | 20:34 |
| mrmartin | yeap | 20:34 |
| mrmartin | and one of the python packages are killing the new askbot deployment | 20:35 |
| mrmartin | I don't whether a custom virtualenv deployment could solve my issue | 20:35 |
| mrmartin | and deploy only those packages there that we required | 20:35 |
| fungi | possible you could uninstall cloud-init and its dependencies | 20:35 |
| mrmartin | I don't = I don't know | 20:35 |
| mrmartin | ok, but if I uninstall cloud-init, what will happen after a reboot? | 20:36 |
| *** dizquierdo has joined #openstack-infra | 20:36 | |
| lifeless | dstufft: https://github.com/pypa/pip/issues/2731 may interest you | 20:36 |
| *** annegentle has quit IRC | 20:36 | |
| fungi | mrmartin: not sure, but we definitely run instances in hpcloud without cloud-init installed | 20:37 |
| *** mwagner_lap has quit IRC | 20:37 | |
| mrmartin | ok, I'll play with that tomorrow, and try to build up a similar image | 20:37 |
| openstackgerrit | Merged openstack-infra/nodepool: Reset the client object after proxy timeouts https://review.openstack.org/74943 | 20:38 |
| fungi | mrmartin: i take that back, we seem to have cloud-init installed even on our dib-built images in hpcloud, but we may be installing it via pip instead of debs | 20:38 |
| *** dguerri is now known as _dguerri | 20:39 | |
| *** _dguerri is now known as dguerri | 20:39 | |
| *** whoops has joined #openstack-infra | 20:39 | |
| *** _nadya_ has quit IRC | 20:39 | |
| *** mrunge has quit IRC | 20:42 | |
| jogo | clarkb fungi: http://logs.openstack.org/14/166214/15/check/check-tempest-dsvm-multinode-full/d6c2efd/console.html#_2015-04-27_11_07_24_519 not sure if that is a one off or not | 20:43 |
| *** miqui has quit IRC | 20:43 | |
| fungi | jogo: ssh connectivity failure to a subnode? | 20:44 |
| jogo | fungi: yup | 20:44 |
| *** eharney has quit IRC | 20:44 | |
| fungi | following the logstash link from elastic-recheck for the pypi.dfw tcp read timeouts, i see some rally jobs which look like they're not being indexed into logstash correctly. anybody know what the cause is there? line splitting seems to be broken | 20:46 |
| fungi | maybe the console log format for those doesn't match similar jobs | 20:46 |
| mrmartin | fungi: if the infra approve this ask-staging patch: https://review.openstack.org/#/c/172880/1 tomorrow, what is the earliest date when we can use that? | 20:46 |
| *** purp has quit IRC | 20:47 | |
| *** purp has joined #openstack-infra | 20:47 | |
| *** tqtran has quit IRC | 20:48 | |
| *** MaxV has quit IRC | 20:48 | |
| fungi | mrmartin: i'll probably be fairly busy since meetings tend to take up most of my tuesdays, but i should have time to try launching the new server for it on wednesday | 20:48 |
| mrmartin | ok, great I try to bring the patch into a proper form | 20:48 |
| fungi | mrmartin: thanks--clarkb makes some good points in his comments | 20:49 |
| mrmartin | so it'll deploy a ask application environment where we can play | 20:49 |
| mrmartin | yeah I see that | 20:49 |
| fungi | and i guess we want it deploying from the feature/development branch | 20:49 |
| jklare | is there currently a known issue with the gates? | 20:49 |
| jklare | https://review.openstack.org/#/c/176674/ | 20:50 |
| *** yfried|afk is now known as yfried_ | 20:50 | |
| jklare | timed out two times in a row at different points | 20:50 |
| mrmartin | so clarkb do you suggest to rename the staging site to ask-dev.o.o or rename the manifest to ask-staging ? | 20:50 |
| fungi | jklare: that second one doesn't look like a timeout. looks like a git fetch error | 20:51 |
| *** turul has joined #openstack-infra | 20:51 | |
| fungi | jklare: in a different job | 20:51 |
| *** turul is now known as afazekas | 20:52 | |
| jklare | fungi: yeah, i thought it might be due to a connection issue (which lead to a timeout in the first case) | 20:52 |
| fungi | jklare: and i can repeat that fetch successfully now, so chances are there was some intermittent network issue between that worker in hpcloud and the zm07 zuul merger in rackspace | 20:53 |
| *** MaxV has joined #openstack-infra | 20:53 | |
| jklare | so just recheck? | 20:53 |
| fungi | jklare: the timeout in the previous check run on the puppet apply job for centos6 is a known intermittent issue we've been unable to reliably reproduce to be able to troubleshoot further | 20:53 |
| *** shardy is now known as shardy_z | 20:53 | |
| fungi | jklare: yes, looks like rechecking should be fine | 20:53 |
| *** hichihara has joined #openstack-infra | 20:54 | |
| fungi | as for centos6, apparently we sometimes see git remote calls fail just on that platform. and the puppet apply job tries to update some 50+ git repos, so it's exposed there more often | 20:54 |
| jklare | mmh... kk | 20:55 |
| *** yamahata has quit IRC | 20:55 | |
| fungi | but they fail in such a way that the socket is open blocked on read and eventually the job times out at 30 minutes | 20:55 |
| *** yamahata has joined #openstack-infra | 20:55 | |
| fungi | i've tried desperately to reproduce teh condition manually, to no avail | 20:56 |
| fungi | at this rate, we'll drop centos 6 as a testing platform before we identify the issue | 20:56 |
| jklare | sounds reasonable | 20:56 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Convert hiera configuration to support public data https://review.openstack.org/177320 | 20:57 |
| *** Sukhdev has quit IRC | 20:58 | |
| *** bswartz has quit IRC | 20:59 | |
| jklare | fungi: thanks for the fast response | 20:59 |
| *** yfried_ is now known as yfried|afk | 21:00 | |
| *** tkelsey has quit IRC | 21:00 | |
| sdake | do infra have sesions at smmit? | 21:00 |
| sdake | the kolla upstream would like to have a session on adding a docker registry to the openstack infrastructure if possible | 21:00 |
| *** amotoki has joined #openstack-infra | 21:01 | |
| *** MaxV has quit IRC | 21:01 | |
| SpamapS | yes | 21:01 |
| sdake | is there an etherpad for it spamaps? | 21:01 |
| fungi | sdake: https://etherpad.openstack.org/p/infra-liberty-summit-planning | 21:01 |
| sdake | thanks fungi | 21:01 |
| SpamapS | sdake: a whole session might be a bit much.. but I'd bring that up now, and get the discussion going now, so we can just hammer out face to face details. :) | 21:01 |
| *** Sukhdev has joined #openstack-infra | 21:01 | |
| mordred | yea | 21:01 |
| mordred | docker registry should not be a hard task | 21:02 |
| SpamapS | Seems like a pretty straight forward thing to run though. | 21:02 |
| mordred | yup | 21:02 |
| SpamapS | and might help with things like running devstack in docker | 21:02 |
| *** armax has joined #openstack-infra | 21:03 | |
| sdake | SpamapS right - I have no idea what it would take | 21:04 |
| sdake | fungi thanks for the link, I added the topic there | 21:04 |
| sdake | we can just discuss face to face if that is the infra cores preference | 21:04 |
| sdake | but right now a docker push inside our gate would take about 5-8 hours, so we can't do that with the 1 hour timelimit. | 21:04 |
| sdake | a local registry would be much faster - 30 minutes or so | 21:04 |
| mordred | wow | 21:04 |
| mordred | sdake: don't forget that we have nodes that span clouds | 21:05 |
| *** marun has joined #openstack-infra | 21:05 | |
| mordred | so a push may not be local | 21:05 |
| sdake | mordred good point | 21:05 |
| mordred | but I still think it's a useful thing to put on the list | 21:06 |
| sdake | thanks - its on the list | 21:06 |
| clarkb | ok mostly back now, catching upafter long lunch then finiahing capstone prep stuff | 21:07 |
| *** annegentle has joined #openstack-infra | 21:07 | |
| *** sdake_ has joined #openstack-infra | 21:07 | |
| *** annegent_ has joined #openstack-infra | 21:07 | |
| *** annegentle is now known as Guest92214 | 21:07 | |
| mordred | clarkb: I made you a couple of patches | 21:08 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add dstat to subnode in multinode mode https://review.openstack.org/177970 | 21:09 |
| jogo | clarkb: ^ | 21:09 |
| *** stevemar has quit IRC | 21:10 | |
| *** sdake has quit IRC | 21:11 | |
| *** Swami has joined #openstack-infra | 21:11 | |
| *** esker has quit IRC | 21:12 | |
| *** rfolco has quit IRC | 21:12 | |
| clarkb | jogo: commented | 21:13 |
| clarkb | mordred: they fail all the tests :P | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| *** jogo has quit IRC | 21:15 | |
| EmilienM | rubygems server is *too* often down, which slow down Puppet CI (we download gems from the official server directly). Would os-infra agree to host a mirror ( the tools are already in place : https://github.com/rubygems/rubygems-mirror ) | 21:15 |
| EmilienM | nibalizer: ^ | 21:15 |
| *** sdake has joined #openstack-infra | 21:15 | |
| EmilienM | we can pay with beers. | 21:16 |
| *** esker has joined #openstack-infra | 21:18 | |
| clarkb | any opposition to me self approving 173603? | 21:19 |
| clarkb | mordred: ^ you may be intersted in the set of slides there | 21:19 |
| clarkb | EmilienM: anyone can propose the change to add the server | 21:19 |
| *** sdake_ has quit IRC | 21:19 | |
| *** Guest92214 has quit IRC | 21:19 | |
| clarkb | EmilienM: it would probably be best if someone familiar with gems and its mirror mechanism did that | 21:20 |
| *** tonytan4ever has quit IRC | 21:20 | |
| *** dustins has quit IRC | 21:20 | |
| *** gyee has quit IRC | 21:20 | |
| EmilienM | clarkb: will look | 21:22 |
| nibalizer | clarkb: so are you thinking one mirror, or one mirror per az? or jsut colocating gem mirrors on the pypi mirrors ? | 21:22 |
| nibalizer | EmilienM: im happy to help you get the rubygems mirrors up if -infra is cool with the overall idea | 21:23 |
| clarkb | nibalizer: probably colocating on the pypi mirrors unless the dfw pypi mirror is derping due to load | 21:23 |
| jeblair | EmilienM, nibalizer: ++ | 21:24 |
| jeblair | clarkb: ++; should probably rename those "mirror." and shift things down a path level. but we can do that later. | 21:24 |
| jeblair | (and dust off afs) | 21:25 |
| *** tjones1 has quit IRC | 21:25 | |
| jeblair | (so we only have to run the mirror jobs in once place) | 21:25 |
| greghaynes | Speaking of mirrors, I have some patches to add wheels to the mirrors that coule use review https://review.openstack.org/#/q/status:open+branch:master+topic:feature/wheel-mirror,n,z | 21:25 |
| clarkb | mordred: SpamapS re 176024 there is still the issue in the tests | 21:26 |
| clarkb | mordred: SpamapS can we confirm its enforcing bug behavior before ew let that slip in? | 21:26 |
| clarkb | mordred: SpamapS it looks to me more like Mock assert truee because thats what it do | 21:26 |
| clarkb | thouh on closer reading args is not what is passed to the cretion it is used to assert what creation was clled with | 21:29 |
| *** otter768 has joined #openstack-infra | 21:29 | |
| clarkb | so I may be wrong about the mock being wrong | 21:29 |
| lifeless | ttx: mordred: https://rbtcollins.wordpress.com/2015/04/28/dealing-with-deps-in-openstack/ | 21:32 |
| *** mrmartin has quit IRC | 21:33 | |
| *** otter768 has quit IRC | 21:34 | |
| *** dkranz has quit IRC | 21:34 | |
| *** ZZelle_ has joined #openstack-infra | 21:35 | |
| sigmavirus24 | lifeless: you live in the future | 21:37 |
| lifeless | sigmavirus24: arguably :) | 21:38 |
| *** jgrimm is now known as zz_jgrimm | 21:38 | |
| *** vhoward has quit IRC | 21:41 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Let kolla-release create branches https://review.openstack.org/177980 | 21:42 |
| *** vhoward has joined #openstack-infra | 21:42 | |
| *** mriedem has quit IRC | 21:43 | |
| *** sdake_ has joined #openstack-infra | 21:46 | |
| *** jtriley has quit IRC | 21:46 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add Neutron/Nova Floating IP support https://review.openstack.org/177036 | 21:46 |
| clarkb | lifeless: I may have missed it but I don't think you address how proposal.3 and proposal.4 are supposed to help goals.3 and goals.4. basically what do distros do if we only every assert one set of things that no distro will ever support together | 21:46 |
| fungi | okay, i've figured out why bandersnatch wanted to redo its refresh after my manual refresh completed. it's not creating a /srv/static/mirror/status file, though it does create .lock and generation just fine | 21:47 |
| clarkb | granted its not like we actually test the full spread of deps today anyways, but ew do seem to at least consider it a bug if say debian and fedora can't both do something reasonable with say python-libvirt | 21:47 |
| *** isviridov_away is now known as isviridov | 21:47 | |
| *** sdake has quit IRC | 21:49 | |
| fungi | also there's a fairly lengthy /srv/static/mirror/todo file | 21:49 |
| fungi | after the resync completes | 21:49 |
| *** armax has quit IRC | 21:50 | |
| *** sarob has quit IRC | 21:52 | |
| clarkb | fungi: file permissions problem? | 21:52 |
| clarkb | fungi: or some other issue writing the status? maybe we ran out of disk? | 21:52 |
| fungi | clarkb: doesn't look like it. we run it as root and the files are owned by root | 21:52 |
| fungi | plenty of disk | 21:52 |
| *** sarob has joined #openstack-infra | 21:52 | |
| clarkb | huh | 21:53 |
| greghaynes | clarkb: SpamapS replied on https://review.openstack.org/#/c/176024/, Im not sure its really a bug | 21:53 |
| fungi | clarkb: look on pypi.ord for an example (root screen session is still running there, untouched from when the manual resync completed) | 21:54 |
| lifeless | clarkb: the distros either don't have this problem (nixos), use containers to avoid it (juju), or don't support using things during upgrades | 21:54 |
| fungi | unfortunately before i noticed the missing status file on pypi.dfw i tried rerunning bandersnatch just to make sure it updated successfully... and so it's redoing a complete refresh again | 21:55 |
| *** hasharAway- has quit IRC | 21:55 | |
| clarkb | lifeless: the last case assumes we are the only python thing asserting deps on the system | 21:55 |
| clarkb | lifeless: in the case of red hat that is really not true (yum) | 21:55 |
| *** marun has quit IRC | 21:55 | |
| lifeless | clarkb: in the context of the gate and devstack ? | 21:56 |
| clarkb | lifeless: no in the context of our users | 21:56 |
| *** gordc has quit IRC | 21:56 | |
| lifeless | maybe I don't understand your statement | 21:56 |
| lifeless | you're saying that redhat ignore our list of what works together and do their own thing? | 21:56 |
| clarkb | lifeless: people want to use our software, it makes sense that they would like to deploy the asserted list of what works | 21:56 |
| clarkb | lifeless: yes what other choice do they have? | 21:57 |
| clarkb | if we hvae pinned 300 python deps and half of them don't match what rhel ships | 21:57 |
| lifeless | clarkb: they can contribute a third party CI that tests with their shipping deps | 21:57 |
| lifeless | clarkb: and see community buyin on working on that set of deps | 21:57 |
| lifeless | clarkb: or they can seek community consensus that that should be the default test and shift the burden on other sets off to other people | 21:58 |
| pleia2 | cinerama: camunoz of redhat/zanata will be at the summit, so I'll be sure to send along our feature requests/bugs prior to so I can have some productive sit down time in person :) | 21:58 |
| *** isviridov is now known as isviridov_away | 21:58 | |
| lifeless | clarkb: smokestack was basically that, no? | 21:58 |
| clarkb | lifeless: ok so we are basically telling them its their thing to solve | 21:58 |
| lifeless | [the former choice] | 21:58 |
| clarkb | lifeless: no smokestack was test on fedora with xen | 21:58 |
| cinerama | pleia2: that's a great idea. i think we can work around some of the more critical things for now but it would be great to get some of those fixes | 21:58 |
| lifeless | clarkb: but also packaged deps | 21:59 |
| lifeless | clarkb: I remember danprince being quite specific about that catching issues | 21:59 |
| clarkb | lifeless: we for the most part did mostly packaged deps until recently | 21:59 |
| pleia2 | cinerama: indeed, I think I owe this project some theming research too, need to sift through bugs and things to see where we're at | 21:59 |
| lifeless | clarkb: so I'm not saying its 'their thing to solve' | 21:59 |
| lifeless | clarkb: I had a whole algebra for discussing this that I ripped out because ETOOLONG | 22:00 |
| *** arielb has quit IRC | 22:00 | |
| clarkb | (it was this cycle that devstack started pushing more things to pypi packages and not system packages) | 22:00 |
| lifeless | clarkb: in short though, their packaged versions are another constraint list. | 22:00 |
| lifeless | We can run any number of constraint lists in parallel with the same infrastructure | 22:00 |
| lifeless | one set of jobs per list | 22:00 |
| lifeless | testing known good is important for CI stability and us getting work done. Testing latest is important to keep us from ending up stale and stuck on unsupported or insecure versions. | 22:01 |
| clarkb | yes | 22:01 |
| lifeless | Testing whats in $distribution is important if we're going to change our code to work with that. | 22:01 |
| lifeless | I mean - if the consensus is 'we should work on whats out there today', then we should accept such bug reports | 22:02 |
| lifeless | and testing with such a constraint set would be a good way to avoid them ever becoming bugs. | 22:02 |
| lifeless | OTOH if the consensus is to require distros to upgrade to reasonable versions, then such bugs would be rejected, etc. | 22:03 |
| clarkb | lifeless: well its less about upgrading necessarily and mor eabout we are asserting one version | 22:03 |
| lifeless | I think technically its better to work with as old versions as can be done with moderate effort. | 22:03 |
| clarkb | lifeless: that may mean downgrades for some distros and upgrades for others | 22:03 |
| *** bknudson has quit IRC | 22:03 | |
| lifeless | Don't kill ourselves to work with old broken. | 22:03 |
| lifeless | But don't force upgrades spuriously. | 22:03 |
| clarkb | but it will most definitely not match anything that is out there | 22:04 |
| lifeless | clarkb: so *we* assert version X works. Redistributors have a choice: a) work with the version we tested. b) run tests themselves with a different version. | 22:04 |
| *** yamamoto has quit IRC | 22:04 | |
| lifeless | clarkb: Adding more upstream version tests (e.g. a set of 'lowest supported versions' as a parallel job) | 22:04 |
| lifeless | clarkb: would be one way of sharing the effort of b across the community | 22:04 |
| *** e0ne has quit IRC | 22:05 | |
| *** kgiusti has left #openstack-infra | 22:06 | |
| *** amitgandhinz has quit IRC | 22:07 | |
| clarkb | hrm jogo keeps disappearing, the failur ehe linked earlier is really weird | 22:07 |
| clarkb | ssh was working then stopped, so something funky happened | 22:08 |
| clarkb | (rax network issue? VM stopped ssh daemon? I dunno) | 22:08 |
| *** vhoward has quit IRC | 22:09 | |
| *** salv-orlando has quit IRC | 22:09 | |
| *** peristeri has quit IRC | 22:09 | |
| *** vhoward has joined #openstack-infra | 22:10 | |
| fungi | apparently if i run bandersnatch mirror instead of using the run-bandersnatch wrapper, it picks up with the todo file where the initial refresh left off. i sort of patched dfw back into shape that i can continue it that way | 22:10 |
| clarkb | fungi: interesting, maybe we broke compat with the wrapper script then? | 22:11 |
| fungi | hopefully this creates a proper status file at the end | 22:11 |
| *** jamesmcarthur has joined #openstack-infra | 22:11 | |
| clarkb | lifeless: in general though looks good. We definitely have to fix pip before we can consider much of this | 22:12 |
| clarkb | lifeless: so good to see you diving in there | 22:12 |
| clarkb | lifeless: any idea if "fixing" pip will lead to backward incompatible installs for pypi ackages and is that a problem? | 22:12 |
| jesusaurus | the log retention in logstash is 2 week, yes? how much disk space does that use up? | 22:12 |
| clarkb | jesusaurus: 10 days, looking at cacti now to get numbers | 22:13 |
| clarkb | hrm doesn't seem to be in cacti | 22:14 |
| clarkb | thats odd | 22:14 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Enhance error message in update_patch https://review.openstack.org/177985 | 22:14 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update recent Ironic exceptions https://review.openstack.org/177986 | 22:14 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: WIP: node_set_provision_state to task https://review.openstack.org/177987 | 22:14 |
| fungi | clarkb: does cloning puppet modules into /etc/puppet/environments/clarkb/modules on the puppetmaster work for you with your current environment.conf there? | 22:14 |
| clarkb | jesusaurus: looks like ~260GB per node and there are 6 nodes. We can probably bump back to 2 weeks based on those numbers | 22:15 |
| clarkb | fungi: yes, but you have to make sure you clone them to the right name, eg puppet-foo cloned to foo | 22:15 |
| fungi | clarkb: perfect--thanks! | 22:15 |
| clarkb | fungi: feel free to use my env too if you need to | 22:15 |
| clarkb | fungi: I am not needing it for anything | 22:15 |
| jesusaurus | clarkb: thanks | 22:16 |
| fungi | clarkb: nah, i think this ought to get me on the right track. going to take a stab at serving a custom gerrit module from my env that doesn't rerun the lucene reindex when replacing a warfile, so we can retest the patched 2.8 war without excessive downtime | 22:17 |
| *** vishy has quit IRC | 22:17 | |
| *** cyeoh has quit IRC | 22:17 | |
| lifeless | clarkb: can I have a +A on https://review.openstack.org/#/c/177500/5 | 22:18 |
| lifeless | clarkb: bug 2687 is the one I expect to make us fall over in a heap | 22:19 |
| openstack | bug 2687 in scummvm (Ubuntu) "ScummVM depends on libflac6 which is not avalaible with Breezy" [Medium,Fix released] https://launchpad.net/bugs/2687 - Assigned to MOTU (motu) | 22:19 |
| lifeless | clarkb: we'll need to add --ignore-installed while we fix stuff up | 22:19 |
| *** cyeoh has joined #openstack-infra | 22:19 | |
| *** vishy has joined #openstack-infra | 22:20 | |
| lifeless | or fungi: https://review.openstack.org/#/c/177500/5 | 22:22 |
| clarkb | lifeless: why did the old wheelhouse stuff not work? | 22:22 |
| *** dizquierdo has quit IRC | 22:23 | |
| clarkb | lifeless: is PIP_FIND_LINKS not valid and pip ignores the env var? | 22:23 |
| *** tnovacik has quit IRC | 22:23 | |
| *** marun has joined #openstack-infra | 22:23 | |
| clarkb | pretty sure devstack has similar problems i so | 22:23 |
| *** esker has quit IRC | 22:23 | |
| clarkb | lifeless: also should pipflags apply to the wheelhouse build commands? | 22:24 |
| lifeless | clarkb: the only hits for that variable in the pip codebase are in tests. | 22:24 |
| *** mriedem has joined #openstack-infra | 22:25 | |
| clarkb | lifeless: pip is supposed to treat all command line flags as valid env vars too | 22:25 |
| clarkb | magic around argparse maybe? | 22:25 |
| lifeless | let me cross check | 22:25 |
| *** amotoki has quit IRC | 22:27 | |
| lifeless | clarkb: ah, I see - found it. Yes it does, so it probably worked | 22:27 |
| lifeless | clarkb: OTOH I think this is clearer | 22:28 |
| *** Somay has quit IRC | 22:28 | |
| lifeless | clarkb: because you don't need as much context to understand whats going on | 22:28 |
| *** salv-orlando has joined #openstack-infra | 22:29 | |
| clarkb | lifeless: PIP_FIND_LINKS is much easier to understand than pip -f | 22:29 |
| lifeless | clarkb: only if you know its exported in the script | 22:30 |
| clarkb | lifeless: and it allows anyone to add pip commands without needing to know the magic sauce | 22:30 |
| clarkb | lifeless: which is already an issu because pipargs isn't used on some pip commands | 22:30 |
| clarkb | sorry PIPFLAGS | 22:30 |
| lifeless | sure | 22:31 |
| lifeless | knowing that there might be environment variables affecting things *is* magic sauce | 22:31 |
| clarkb | yes but its not necessarily knowledge | 22:31 |
| clarkb | *necessary | 22:32 |
| lifeless | well it is - because I misunderstood the script because I didn't know that thing | 22:32 |
| clarkb | anyways left comment about PIPFLAGS | 22:33 |
| *** bswartz has joined #openstack-infra | 22:33 | |
| *** mriedem has quit IRC | 22:33 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:33 | |
| *** mriedem has joined #openstack-infra | 22:34 | |
| *** nelsnelson has quit IRC | 22:34 | |
| *** tjones1 has joined #openstack-infra | 22:35 | |
| *** rbradfor has quit IRC | 22:35 | |
| *** dboik_ has joined #openstack-infra | 22:36 | |
| *** ZZelle_ has quit IRC | 22:37 | |
| *** camunoz_gone is now known as camunoz | 22:38 | |
| *** dangers is now known as dangers_away | 22:38 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Use /opt/git directly https://review.openstack.org/177629 | 22:38 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Stop testing setup.py easy_install behaviour https://review.openstack.org/177505 | 22:38 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Test pip install -e of projects. https://review.openstack.org/177504 | 22:38 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Build all the wheels in one pass rather than many. https://review.openstack.org/177503 | 22:38 |
| *** mriedem has quit IRC | 22:38 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Improve integration.sh. https://review.openstack.org/177500 | 22:38 |
| lifeless | clarkb: mordred: ^ more +2's up the whole stack please ... all will fail the diff test and need reapplying | 22:38 |
| *** dboik has quit IRC | 22:39 | |
| *** mriedem has joined #openstack-infra | 22:39 | |
| clarkb | lifeless: have we tested 177503 with a requirements list that fails to build a wheel in the middle? | 22:40 |
| *** dboik_ has quit IRC | 22:41 | |
| clarkb | I am trying t oremember which deps were problematic so I can test | 22:41 |
| clarkb | oh the comment says httpretty | 22:41 |
| *** marun has quit IRC | 22:41 | |
| lifeless | clarkb: httpretty delivers its own wheels now | 22:42 |
| lifeless | Downloading http://pypi.ORD.openstack.org/packages/2.7/h/httpretty/httpretty-0.8.6-py2-none-any.whl | 22:42 |
| clarkb | lifeless: thats not the question though, question is, if a wheel fails to build do we build all of the remaining wheels? | 22:43 |
| clarkb | lifeless: in the past pip did not do that | 22:43 |
| *** ddieterl_ has quit IRC | 22:43 | |
| lifeless | clarkb: pip 6.1.1 and 7.0.0 both look for errors after attempting all wheels | 22:43 |
| lifeless | clarkb: I do not know if we observed a bug in that loop, or if the loop did not exist in the past. | 22:44 |
| clarkb | lifeless: the loop did not exist at one point because -r did not work | 22:44 |
| clarkb | (and we had confirmed that) | 22:44 |
| lifeless | clarkb: you misunderstand. I know the symptom. | 22:44 |
| lifeless | clarkb: I am saying it could have been a) the loop was buggy, and stopped early. Or b) there was no loop. | 22:44 |
| lifeless | clarkb: if it was a), I do not know if the loop has been fixed. | 22:44 |
| clarkb | sure, and I am asking have you tested latest pip? | 22:45 |
| lifeless | clarkb: if it was b), there is now a loop. | 22:45 |
| *** jog0 has joined #openstack-infra | 22:45 | |
| lifeless | clarkb: none of our deps fail at all now. I have not dug around for one that fails to test manually. | 22:45 |
| clarkb | ok | 22:45 |
| clarkb | I am tethering now, but will try to test this evening when not tethering | 22:46 |
| lifeless | I don't think this matters though, since right now everything works and pip 7 auto caches its own wheels on demand | 22:46 |
| lifeless | [and if a wheel fails to build for 7 in a way that would break the loop, it will also break pip install] | 22:46 |
| clarkb | lifeless: it does matter if it breaks tomorrow and yesterdays code would've worked | 22:46 |
| lifeless | if that happens the -upstream job will also break | 22:47 |
| lifeless | even if we use a loop rather than -r | 22:47 |
| clarkb | I don't think that makes it ok | 22:48 |
| *** evgeny has joined #openstack-infra | 22:49 | |
| clarkb | particularly if you want ot isolate reasons for failure | 22:49 |
| lifeless | I do | 22:49 |
| clarkb | (one test failing is easier to debug than two) | 22:49 |
| *** evgeny has left #openstack-infra | 22:49 | |
| lifeless | this is a known unknown | 22:49 |
| clarkb | my known is that your change does not work | 22:50 |
| lifeless | there may be any number of fails to build wheels in some special way that throws an exception | 22:50 |
| clarkb | until I have evidence of the contrary I will not approve it | 22:50 |
| clarkb | but I will look for the evidence as soon as I have the ability to do so | 22:50 |
| lifeless | I am completely lost | 22:50 |
| lifeless | here is what I know | 22:50 |
| lifeless | there is some common infrastructure inside pip | 22:50 |
| clarkb | lifeless: pip wheel -r somefile will bail out early if a wheel fails to build | 22:50 |
| lifeless | clarkb: no, it won't | 22:51 |
| *** whoops has quit IRC | 22:51 | |
| clarkb | this was a known as of the lsat time I did anything with wheels and pip (which is admittedly some time) | 22:51 |
| lifeless | that infrastructure is used in pip wheel | 22:51 |
| clarkb | lifeless: ok please show me evidence of that | 22:51 |
| lifeless | and its used in pip install in 7.0 to autocache wheels | 22:51 |
| clarkb | lifeless: that is all I am asking for and will go find it myself as soon as I am able | 22:51 |
| lifeless | there is a test in 7.0 for pip install with a thing that will fail to build a wheel | 22:51 |
| *** annegent_ has quit IRC | 22:51 | |
| lifeless | so I know that with a vanila fails-to-build-a-wheel that the loop won't be interrupted | 22:52 |
| lifeless | and all the wheels will be attempted | 22:52 |
| clarkb | latest pip is still 6.1.1 | 22:52 |
| clarkb | not 7.0 | 22:52 |
| lifeless | 6.1.1 has the same code | 22:52 |
| lifeless | and it has not changed | 22:52 |
| *** jog0 is now known as jogo | 22:52 | |
| clarkb | ok so now all I need is to test it | 22:52 |
| lifeless | it just doesn't have the second user - the autocached | 22:52 |
| lifeless | but | 22:52 |
| clarkb | because 6.something definitely did fail | 22:52 |
| mtreinish | mordred: if you wanted big numbers about the gate for slides: http://paste.openstack.org/show/209007/ | 22:52 |
| lifeless | there may be ways that a wheel can fail to build that will provoke a failure in the loop | 22:52 |
| lifeless | I don't know of any | 22:53 |
| mtreinish | since we're basically at the end of the cycle that's basically a cycle's worth of tempest tests | 22:53 |
| lifeless | but its a big universe :) | 22:53 |
| clarkb | lifeless: the previous case was httpretty | 22:53 |
| clarkb | lifeless: which works now because they ship a wheel | 22:53 |
| lifeless | the httpretty case will work | 22:53 |
| clarkb | lifeless: but not becaues of pip | 22:53 |
| lifeless | because of pip | 22:53 |
| clarkb | no because they ship a hweel | 22:53 |
| clarkb | so pip wheel is a noop | 22:54 |
| lifeless | if you use a requirements file that clamps down to an earlier httppretty | 22:54 |
| lifeless | it will work | 22:54 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add dstat to subnode in multinode mode https://review.openstack.org/177970 | 22:54 |
| *** shashankhegde has quit IRC | 22:54 | |
| *** mwagner_lap has joined #openstack-infra | 22:54 | |
| lifeless | someting that sends SIGINT to its parent process wouldn't | 22:56 |
| *** jamesmcarthur has quit IRC | 22:56 | |
| *** sdake_ has quit IRC | 22:57 | |
| clarkb | ok confirmed that latest pip 6.1.1 with httpretty==0.8.0 will fail to build an httpretty wheel but then continue to build a wheel for tox | 22:57 |
| *** jtriley has joined #openstack-infra | 22:58 | |
| jhesketh | Morning | 22:59 |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:01 | |
| SpamapS | So, google is no help in squelching these, anybody: | 23:01 |
| *** Sukhdev has quit IRC | 23:01 | |
| SpamapS | /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. | 23:01 |
| SpamapS | time to upgrade to utopic/vivid ? | 23:02 |
| clarkb | SpamapS: 1) cry 2) you can get a new platform with latest python2.7 3) install requests[security] #note this requires a bunch of C headers for pyopenssl | 23:02 |
| greghaynes | SpamapS: There is a logging.conf filter | 23:02 |
| greghaynes | oh | 23:02 |
| greghaynes | I know ive seen a filter on a patch somewhere | 23:02 |
| greghaynes | I think it was in gertty | 23:02 |
| clarkb | ya I think it was gertty | 23:02 |
| lifeless | SpamapS: https://github.com/pypa/pip/issues/2681 | 23:03 |
| clarkb | because it broke curses | 23:03 |
| lifeless | which will tell you that you need pyopenssl ndg-httpsclient pyasn1 | 23:03 |
| SpamapS | http://stackoverflow.com/questions/29099404/ssl-insecureplatform-error-when-using-requests-package | 23:04 |
| SpamapS | found this one too | 23:04 |
| lifeless | yes | 23:04 |
| lifeless | aka this is why 'releases' are bad | 23:04 |
| clarkb | SpamapS: also note that about 80% of requests installs are affected based on rough graph provided by dstufft | 23:04 |
| SpamapS | seems like https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning should be updated to explain the libs to silence that | 23:05 |
| SpamapS | clarkb: osssuummm | 23:05 |
| clarkb | also requests bundles urllib3 so this has to be fixed in two places | 23:05 |
| clarkb | * the spamming must be fixed in two places I think its reasonable to provide one warning per process invocation | 23:06 |
| clarkb | but every request is a bit over the top | 23:06 |
| *** shashankhegde has joined #openstack-infra | 23:06 | |
| dstufft | clarkb: I think 2.6.1 only does it once | 23:06 |
| SpamapS | yeah keystoneclient had the same insanity | 23:06 |
| SpamapS | logs filling for days becuase of the premature v2 deprecation | 23:07 |
| *** yamamoto has joined #openstack-infra | 23:07 | |
| *** armax has joined #openstack-infra | 23:07 | |
| *** hemna is now known as hemnafk | 23:11 | |
| lifeless | clarkb: many places - see the pip bug I linked | 23:11 |
| fungi | clarkb: nibalizer: was there a question at some point about whether puppet assumes file ownership from the puppetmaster's filesystem when not specified in the file resource itself? if so, i have specific evidence i can share that it definitely does this | 23:12 |
| clarkb | fungi: not from me at least (its totally viewable when you use different puppet envs because perms change) | 23:12 |
| fungi | yep, okay... just checking | 23:13 |
| lifeless | mordred: / fungi: can you review this stack - https://review.openstack.org/#/c/177500/6 ? | 23:13 |
| jogo | lifeless: reading https://rbtcollins.wordpress.com/2015/04/28/dealing-with-deps-in-openstack/ enjoying it so far | 23:14 |
| nibalizer | fungi: yep | 23:15 |
| lifeless | jogo: cool | 23:15 |
| nibalizer | this is why File [ owner => root, group => root,] ends up at the top of site.pp a lot | 23:15 |
| clarkb | jogo: that test fail you linked is crazy fwiw, ssh works then stops working | 23:15 |
| clarkb | jogo: guesses are cloud networking issue, sshd dies, or keys are deleted | 23:16 |
| jogo | clarkb: yeah, I am wondering if that is a really strange one off or not | 23:16 |
| jogo | only 1 hit in 61 or so jobs | 23:16 |
| jogo | clarkb: mainly shared it so you are aware | 23:16 |
| mtreinish | clarkb: or someone sneezed in the data center and the network cable fell out | 23:16 |
| jogo | lifeless: sounds like you have some serious pipping to do | 23:16 |
| fungi | providers need to start investing in ieee sneezeproof cabling | 23:17 |
| clarkb | possible race in network setup maybe? iptables/ovs | 23:18 |
| lifeless | how do you get unprocessed logs? | 23:18 |
| clarkb | but I think it happens before that | 23:18 |
| clarkb | lifeless: accept txt/plain and there is a parameter too | 23:18 |
| clarkb | lifeless: wget willget nuprocessed fwiw. let me dig up param | 23:19 |
| lifeless | ah cool | 23:19 |
| lifeless | I am wgetting | 23:19 |
| lifeless | its just 116M | 23:19 |
| lifeless | {something assploded} | 23:19 |
| lifeless | clarkb: don't worry, wget DTRT is enough for me | 23:19 |
| lifeless | jogo: there's two conceptual lines of work | 23:19 |
| lifeless | jogo: three bugs, and it all folds into one specific interaction | 23:20 |
| *** sabeen1 has quit IRC | 23:20 | |
| *** dimtruck is now known as zz_dimtruck | 23:20 | |
| *** wenlock_ has quit IRC | 23:20 | |
| *** yamamoto has quit IRC | 23:21 | |
| jogo | lifeless: right | 23:21 |
| jogo | lifeless: to be clear, the goal is: a single constraints file for all of openstack | 23:22 |
| jogo | that uses the '==' notation | 23:22 |
| jogo | what do you mean loosen up the constraints in all our release branches | 23:22 |
| lifeless | jogo: so we have olso.messaging >= 1.4 < 1.5 | 23:23 |
| lifeless | jogo: it needs to be at minimum >= 1.4 < 1.6 | 23:23 |
| lifeless | jogo: otherwise we can't upgrade without violating the constraint | 23:23 |
| lifeless | and once pip 2687 is fixed, that will error | 23:24 |
| jogo | ok, how does that tie into requirements.txt vs setup.cfgs install_requires or whatever that was? | 23:24 |
| lifeless | so requirements.txt in each repo goes away | 23:25 |
| jogo | lifeless: also so your saying teach pip to support a requirements file and a seperate constraints file? | 23:25 |
| lifeless | the install_requires it contained move into setup.cfg | 23:25 |
| lifeless | this can be done independently though, its really just aligning with upstreams expectations | 23:25 |
| lifeless | we could perhaps have a constraints in each repo with ==, but I really question the point of doing that since we would have to be using the central one in all CI anyhow | 23:26 |
| fungi | lifeless: out of curiosity, once we have olso.messaging>=1.4,<1.6 how do we handle olso.messaging jumping from 1.4.7 to 2.0.0? | 23:26 |
| lifeless | fungi: so thats a true API break happening | 23:26 |
| lifeless | fungi: from a deployment perspective, that means you can't do an incremental deployment across that boundary | 23:27 |
| *** kaisers has quit IRC | 23:27 | |
| lifeless | fungi: unless you can upgrade all the users of it first to accept it | 23:27 |
| fungi | got it, so assume a requirements list update to accommodate those situations | 23:27 |
| lifeless | and point releases of the stable servers | 23:27 |
| *** emagana has quit IRC | 23:28 | |
| lifeless | e.g. liberty .4 release with < 2.1 in its deps | 23:28 |
| lifeless | you deploy those, that loosens the constraints, you deploy the thing that will bring in 2.0.0 | 23:28 |
| jogo | we also need to make some guarantees about compatibility with our prior release in our library development | 23:29 |
| lifeless | You can only do that once you know that liberty is compatible with 2.0 of course | 23:29 |
| lifeless | yup. Sorry devs, inconvenient truth. :) | 23:29 |
| jogo | lifeless: that line could be expanded on I think. I am not convinced that will work 100% of the time | 23:29 |
| *** otter768 has joined #openstack-infra | 23:30 | |
| lifeless | jogo: go on | 23:30 |
| jogo | lifeless: well what fungi said is part | 23:30 |
| jogo | lifeless: but what about non-openstack owned libraries | 23:30 |
| jogo | I also question the requirement actually | 23:30 |
| jogo | being able to partially upgrade a single box? | 23:30 |
| lifeless | we do this in grenade | 23:31 |
| lifeless | AIUI | 23:31 |
| clarkb | yes | 23:31 |
| fungi | sort of | 23:31 |
| openstackgerrit | Merged openstack-infra/system-config: Log #openstack-neutron-ovn. https://review.openstack.org/173876 | 23:31 |
| clarkb | but its a turn it all off then upgrade upgrade | 23:31 |
| fungi | we upgrade the services while keeping the requirements as they were | 23:31 |
| fungi | and the configuration as it was | 23:32 |
| lifeless | fungi: since the requirements are partitions, I don't understand that statement | 23:32 |
| jogo | fungi: we actaully don't keep the old requirements anymore AFAIK | 23:33 |
| fungi | oh, poop | 23:33 |
| *** annegentle has joined #openstack-infra | 23:33 | |
| fungi | i missed when that changed | 23:33 |
| * jogo double checks | 23:33 | |
| *** wenlock has quit IRC | 23:33 | |
| fungi | used to be install a stable/last devstack, then shut it down, then reinstall current git trees with stable/last requirements list | 23:34 |
| lifeless | thats the ultimate form of backwards comapt | 23:34 |
| fungi | though i also may have simply dreamt that | 23:34 |
| lifeless | I think its harder than needed | 23:34 |
| clarkb | fungi: there are a few jobs in the check queue that have been running for hours | 23:34 |
| fungi | awesome! | 23:34 |
| lifeless | jogo: what about the partial things where we bring up new nova conductor, old compute | 23:34 |
| jogo | lifeless: we don't restart the old service | 23:34 |
| *** otter768 has quit IRC | 23:35 | |
| fungi | clarkb: different jenkins masters too | 23:35 |
| jogo | so it uses the old copies | 23:35 |
| jogo | its a hack | 23:35 |
| clarkb | fungi: ya looks like maybe related to the swift uploads? | 23:35 |
| lifeless | jogo: and hope that when its starts new processes via rootwrap, that it all works | 23:35 |
| clarkb | https://jenkins05.openstack.org/job/check-dg-tempest-dsvm-full/262/console | 23:35 |
| jogo | lifeless: yup | 23:35 |
| lifeless | jogo: aiee | 23:35 |
| clarkb | ERROR:root:File posting error | 23:35 |
| clarkb | which I think is the get console log from jenkins post? | 23:35 |
| *** annegentle has quit IRC | 23:35 | |
| fungi | clarkb: yeah, but for example https://jenkins05.openstack.org/job/check-dg-tempest-dsvm-full/262/console last tried at 19:33 utc and has just been sitting there ever since | 23:36 |
| *** kaisers has joined #openstack-infra | 23:36 | |
| clarkb | oh wait no thats swift | 23:36 |
| clarkb | I wonder if that is what happens when we exceed the time limit on the tempurl? | 23:36 |
| clarkb | jhesketh: ^ you may be interested in this | 23:36 |
| fungi | looks like we _could_ abort these if we want, but being able to catch them in action seems like a useful reason to not do that | 23:37 |
| *** yamada-h has joined #openstack-infra | 23:37 | |
| clarkb | though it specifically says network unreachable | 23:37 |
| clarkb | hopefully we would get not authorized if tempurl is no longer valid | 23:38 |
| clarkb | notmyname: mattoliverau ^ | 23:38 |
| jhesketh | clarkb: hmm, I'll take a look in a bit | 23:38 |
| jogo | lifeless fungi: http://logs.openstack.org/20/177920/2/check/check-grenade-dsvm/111f5eb//logs/grenade.sh.txt.gz#_2015-04-27_19_11_27_470 | 23:38 |
| notmyname | clarkb: tempurl expires time should be at the time of the request, not the time the request completes | 23:39 |
| clarkb | notmyname: right but would we get "network unreachable" as the error if it had expired? | 23:40 |
| notmyname | that doesn't sound right to me | 23:40 |
| fungi | hrm, nodepool's getting EOFError testing ssh to devstack-precise nodes in hpcloud. not having a problem with bare-precise there, nor is rax having issues with devstack-precise | 23:40 |
| notmyname | if it had expired, you'd see a 401 response | 23:40 |
| clarkb | notmyname: thanks | 23:40 |
| fungi | something must be not quite right with those images | 23:41 |
| *** erlon has quit IRC | 23:41 | |
| lifeless | jogo: right so that is doing precisely what I said | 23:41 |
| lifeless | jogo: its loosening the constraints | 23:41 |
| lifeless | jogo: but we can't do that for the things we're consuming as releases | 23:41 |
| lifeless | jogo: so we have to make releases of them with looser constraints | 23:42 |
| *** yamada-h has quit IRC | 23:42 | |
| jogo | lifeless: its syncing constraints with g-r | 23:42 |
| jogo | lifeless: from master | 23:42 |
| jogo | lifeless: some of that is loosening some isn't that is besides the point here | 23:42 |
| jogo | hmm although it is using pip install -e | 23:44 |
| jogo | will that upgrade deps that conflict? | 23:45 |
| lifeless | yes | 23:45 |
| jogo | lifeless: kk thought so | 23:45 |
| lifeless | and with 2687 fixed, fail if it can't resolve | 23:45 |
| jogo | lifeless: so going back, I don't think supporting stable/N services + stable/N+1 services on the same node is a reasonable requirement | 23:46 |
| *** dims has quit IRC | 23:46 | |
| clarkb | jogo: thats how you upgrade a service at a time | 23:46 |
| jogo | its a nice one to have but we don't aim for that today | 23:46 |
| clarkb | eg upgrade keystone first | 23:47 |
| clarkb | and work your way up the stack | 23:47 |
| clarkb | (or down, I don't actually know what the appropriate direction would be) | 23:47 |
| jogo | clarkb: we upgrade one at a time by knowing if you don't restart the service it uses the old version already in memory | 23:47 |
| clarkb | jogo: what happens if you have to reboot | 23:47 |
| clarkb | jogo: thats not an acceptable method | 23:47 |
| jogo | clarkb: your fucked | 23:47 |
| jogo | clarkb: my point is we don't suppor that now | 23:48 |
| lifeless | jogo: ok so in summary: this thing I propose to avoid users getting fucked we currently say they can get fucked on? | 23:48 |
| jogo | I don't think we should add that as a hard req for fixing pip | 23:48 |
| lifeless | jogo: can we assess it on its merits rather than what we do today ? :) | 23:48 |
| clarkb | +1 | 23:48 |
| jogo | lifeless: merits and cost yes | 23:48 |
| lifeless | certainly | 23:48 |
| *** jtriley has quit IRC | 23:48 | |
| jogo | lifeless: if we can support this with minimal overhead and risk by all means lets do it | 23:49 |
| jogo | just don't see that being small | 23:49 |
| lifeless | so previously we didn't cap the libraries AIUI | 23:49 |
| jogo | anyway I believe sdague and dtroyer know more about this then I do | 23:49 |
| jogo | so I defer to them for further insight | 23:49 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Update to search UI. https://review.openstack.org/178003 | 23:49 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Renamed result-set-size directive https://review.openstack.org/178004 | 23:49 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Result set paging update. https://review.openstack.org/178005 | 23:49 |
| lifeless | my memory is that we started capping them because we made mistakes | 23:49 |
| lifeless | and broke ourselves | 23:49 |
| lifeless | not that we intended to do so | 23:49 |
| jogo | lifeless: correct, other things also broke us as well | 23:49 |
| jogo | besides ourselves | 23:49 |
| lifeless | right | 23:49 |
| *** armax has quit IRC | 23:49 | |
| jogo | btw for your blog post an example of the final result would be neat | 23:50 |
| jogo | something like the different reqs files and how they get used | 23:50 |
| lifeless | so the design here is to avoid that harming CI by having complete centralised control | 23:50 |
| jogo | lifeless: ++ | 23:50 |
| *** sarob has quit IRC | 23:51 | |
| lifeless | and if we have something that would break stable get out into the wild, we can find and fix without everything wedging | 23:51 |
| jogo | right | 23:51 |
| jogo | so I think the possible downside of requiring backwards compat with old servers is: | 23:51 |
| clarkb | lifeless: so master would remain open? | 23:51 |
| clarkb | if so that wasn't clear in the blog | 23:51 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Allow JJB to set connection timeout to jenkins server https://review.openstack.org/151472 | 23:51 |
| lifeless | clarkb: what do you mean by open ? | 23:52 |
| jogo | requiring extra work and planning to prevent backwards incompatible breaks | 23:52 |
| clarkb | lifeless: I mean master wouldn't pin everything and instead plumb the depths looking for broken | 23:52 |
| jogo | for openstack owned and 3rd party libraries | 23:52 |
| clarkb | then at rc time we can pin | 23:52 |
| jogo | clarkb: you see this https://rbtcollins.wordpress.com/2015/04/28/dealing-with-deps-in-openstack/ ? | 23:52 |
| clarkb | jogo: yes that is what I am referring to | 23:52 |
| lifeless | clarkb: I think that is something we could play around with | 23:52 |
| lifeless | clarkb: we could constrain master precisely to known good as the blog post shows | 23:53 |
| *** sarob has joined #openstack-infra | 23:54 | |
| *** yamamoto has joined #openstack-infra | 23:56 | |
| lifeless | clarkb: or we could unconstraint master and ride the wave | 23:56 |
| lifeless | clarkb: I think having master constrained and reporting on potential upgrades is better | 23:56 |
| *** armax has joined #openstack-infra | 23:56 | |
| lifeless | clarkb: because we spend a lot of time in firedrills | 23:56 |
| clarkb | except they are often real bugs | 23:56 |
| clarkb | so by ignoring them on master we just shift the work | 23:57 |
| lifeless | I didn't say ignore | 23:57 |
| clarkb | so I guess it is a question of whether or not shifting that work is beneficial | 23:57 |
| clarkb | lifeless: I know | 23:57 |
| clarkb | but history has shown it will be ignored. I will be pleasantly surprised if we do otherwise | 23:57 |
| lifeless | clarkb: interrupting 1000 developers is very expensive | 23:57 |
| clarkb | lifeless: but its rarely a real interrupt | 23:57 |
| lifeless | clarkb: except when it is | 23:57 |
| jogo | clarkb lifeless: OpenStack has grown up a lot since 'history' | 23:57 |
| clarkb | code got -1 because openstack is broken | 23:57 |
| *** Somay has joined #openstack-infra | 23:57 | |
| clarkb | thats a real interrupt | 23:58 |
| jogo | not sure if we can use said data as a useful reference anymore | 23:58 |
| lifeless | clarkb: code can't land is also an interrupt | 23:58 |
| clarkb | jogo: I don't think it has really learned its lessons | 23:58 |
| jogo | either way, this is something we can easily adjust over time | 23:58 |
| clarkb | jogo: we still can't turn on ipv6 in devstack for example | 23:58 |
| lifeless | indeed, its flexible | 23:58 |
| jogo | clarkb: that is a very different type of issue | 23:58 |
| clarkb | jogo: not really, we are talking about not putting off bugs to make now work | 23:58 |
| jogo | but yeah we are really arguing over what would amount to a few lines of configuration that is easy to cahnge over time | 23:58 |
| clarkb | jogo: its the same type of bug | 23:59 |
| clarkb | if we just turned on ipv6 and interrupted everyone then this would've been fixed in kilo | 23:59 |
| clarkb | instead nova ignored it | 23:59 |
| jogo | clarkb: most of the dependency bugs are fairly isolated in nature, is IPv6 thsame | 23:59 |
| clarkb | and now its another cycle until we can tet this | 23:59 |
| jogo | also part of the issue is rallying a team around a problem | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!